After years of wrestling with this issue in a number of applications, it appears that Microsoft has finally accepted it as a bug in .NET 4 CLR that causes this to occur. http://support.microsoft.com/kb/2640103.
I had previously been "fixing" it by forcing the garbage collector to run in server mode (gcServer enabled="true" in app.config) as described in the Microsoft article linked to by Think Before Coding. This in essence forces all threads in the application to pause during the collection removing the possibility of other threads accessing the memory being manipulated by the GC. I am happy to find that my years of searching in vain for a "bug" in my code or other 3rd party unmanaged libraries were only fruitless because the bug lay in Microsoft's code, not mine.