[.net] Application Crashes With "Internal Error In The .NET Runtime"

We have an application written against .NET 4.0 which over the weekend crashed, putting the following message into the event log:

Application: PnrRetrieverService.exe Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 791F9AAA (79140000) with exit code 80131506.

This is on a Windows Server 2003 R2 Standard Edition box. Googling this error hasn't turned up anything pertinent. For example, this isn't occurring in VS Studio, but instead on a production box; when the service was eventually restarted, it experienced no further problems.

How does one go about diagnosing a bug in the .NET Runtime?

I've experienced "internal errors" in the .NET runtime that turned out to be caused by bugs in my code; don't think that just because it was an "internal error" in the .NET runtime that there isn't a bug in your code as the root cause. Always always always blame your own code before you blame someone else's.

Hopefully you have logging and exception/stack trace information to point you where to start looking, or that you can repeat the state of the system before the crash.

In my case this exception was occured when disk space was over and .NET can't allocate memory in Windows Virtual Memory.

In event log I saw this error:

Application popup: Windows - Virtual Memory Minimum Too Low : Your system is low on virtual memory. Windows is increasing the size of your virtual memory paging file. During this process, memory requests for some applications may be denied.

And previous error:

The C: disk is at or near capacity. You may need to delete some files.

I am not sure it may help everyone, but I could get around this by running

devenv.exe /ResetSettings 

...in the path {Visual_Studio_root}\Common7\Ide

I had the following errors in the event log and VS was just crashing and restarting all the time:

Faulting application name: devenv.exe, version: 14.0.25123.0, time stamp: 0x56f22f32
Faulting module name: clr.dll, version: 4.7.2115.0, time stamp: 0x59af88f2
Exception code: 0xc0000005
Fault offset: 0x0015f90e
Faulting process id: 0x3a7c
Faulting application start time: 0x01d353463eaf0c36
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Report Id: a232f984-6e80-4f61-9003-e18a035c8f93
Faulting package full name: 
Faulting package-relative application ID: 

Had the same exact error on WinXP box with latest build of my .NET 4 code. Checked previous builds - now they crash too! Ok, so it's not me :). No suggestions here/above helped.

Much more recent (2018-05-09) report of the same problem: Application Crash with exit code 80131506.

A: We were receiving a similar error, but we believe ours was caused by the Citrix memory optimizer.
The resolution was to force a regeneration of the .Net core libraries on the host(s) where the issue was occurring:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe update /force

Root cause is still unknown (machine is not being updated and has little use), but that did it for me!

with exit code 80131506

That's a nasty one, ExecutionEngineException. Starting with .NET 4.0, this exception immediately terminates the program. The generic cause is corruption of the state of the garbage collected heap. Which in turn is invariably caused by unmanaged code. The exact location in code at which this exception is raised isn't helpful, the corruption usually occurred well before the damage is detected.

Finding the exact cause for this is going to be difficult. Review any unmanaged code your service might be using. Suspect environmental problems if there is no obvious candidate, misbehaving malware scanners are notorious. If it repeats very poorly then suspect hardware problems like soft RAM errors.

For those arriving here from google, I've eventually come across this SO question, and this specific answer solved my problem. I've contacted Microsoft for the hotfix through the live chat on support.microsoft.com and they sent me a link to the hotfix by email.

In my case the problem was a C++/CLI library in which there was a call to the NtQuerySystemInformation; for some kind of reason sometimes (and under mysterious circumstances), when it was called the CLR heap got corrupted and the application crashed.

I've resolved the problem using a "custom heap" created with HeapCreate and allocating there the buffers used by that function.

This might be an exception occurring in the finalizer. If you are doing the Pattern of ~Class(){ Dispose(false); } check what are you disposing as an un-managed resource. Just put a try..catch there and you should be fine.

We found the issue as we had this mysterious failure with no logs We did the usual recommended pattern of using a "void Dispose(bool disposing)".

Looking at the answers on this question about the finalizer we found a possible place where the Disposal of the unmanaged resources could throw an exception.

It turns out somewhere we did not dispose the object properly thus the finalizer took over the diposal of unmanaged resources thus behold an exception occurred.

In this case was using the Kafka Rest API to clean up the client from Kafka. Seems that it did threw exception at some point then this issue occurred.

I never figured out why this was happening for me. It was consistently reproducible for one of my applications, but went away after simply rebooting.

I am running Windows 2004 Build 19582.1001 (Insider Preview) with .net-4.8 and I also would not be surprised if this were due to something like a hardware memory error. Also, my application does load some unmanaged code and initialize it, so I can’t prove that the crash didn’t come from that.

In my case this error occured when logging in SAP Business One 9.1 application. In Windows events I could find also another error event in addition to the one reported by the OP:

Nome dell'applicazione che ha generato l'errore: SAP Business One.exe, versione: 9.10.160.0, timestamp: 0x551ad316
Nome del modulo che ha generato l'errore: clr.dll, versione: 4.0.30319.34014, timestamp: 0x52e0b784
Codice eccezione: 0xc0000005
Offset errore 0x00029f55
ID processo che ha generato l'errore: 0x1d7c
Ora di avvio dell'applicazione che ha generato l'errore: 0x01d0e6f4fa626e78
Percorso dell'applicazione che ha generato l'errore: C:\Program Files (x86)\SAP\SAP Business One\SAP Business One.exe
Percorso del modulo che ha generato l'errore: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
ID segnalazione: 3fd8e0e7-52e8-11e5-827f-74d435a9d02c
Nome completo pacchetto che ha generato l'errore: 
ID applicazione relativo al pacchetto che ha generato l'errore: 

The machine run Windows 8.1, with .NET Framework 4.0 installed and without the 4.5 version. As it seemed from the internet that it could be also a bug in .NET 4, I tried installing .NET Framework 4.5.2 and I solved the issue.

A bug in the concurrent implementation of the Garbage Collection on x64 .Net 4 can cause this as stated in the following microsoft KB entry:

ExecutionEngineException occurs during Garbage Collection

You should first make a deep minidump exploration to be sure that the problem occured during a Garbage collection.

The minidump location can usually be found in a Windows Error Reporting entry in the event log following the crash entry. Then, have fun with WinDbg !

The latest documentation on the use of the <gcConcurrent/> configuration element, to disable concurrent or (in .NET 4 and later) background garbage collection, can be found here.

In my case the problem was related to "Referencing .NET standard library in classic asp.net projects" and these two issues

https://github.com/dotnet/standard/issues/873

https://github.com/App-vNext/Polly/issues/628

and downgrading to Polly v6 was enough to workaround it

Every 5-10 minutes my application pool kept crashing with this exit code. I do not want to ruin your trust of the Garbage Collector, but the following solution worked for me.

I added a Job that calls GC.GetTotalMemory(true) every minute.

I suppose that, for some reason, the GC is not automatically inspecting the memory often enough for the high number of disposable objects that I use.

Could be a bug with concurrent GC http://support.microsoft.com/kb/2679415

After years of wrestling with this issue in a number of applications, it appears that Microsoft has finally accepted it as a bug in .NET 4 CLR that causes this to occur. http://support.microsoft.com/kb/2640103.

I had previously been "fixing" it by forcing the garbage collector to run in server mode (gcServer enabled="true" in app.config) as described in the Microsoft article linked to by Think Before Coding. This in essence forces all threads in the application to pause during the collection removing the possibility of other threads accessing the memory being manipulated by the GC. I am happy to find that my years of searching in vain for a "bug" in my code or other 3rd party unmanaged libraries were only fruitless because the bug lay in Microsoft's code, not mine.

Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.Reflection.TargetInvocationException

I have faced this Error, Application was working fine on some PCs and on some PCs giving the above error. I uninstall the Framework 4.5 and re-install this resolved my problem.

Cheer.

In my case the issue was due to duplicate binding redirects in my web.config. More info here.

I assume it was because of NuGet modifying the binding redirects, but for example it was looking like this:

  <dependentAssembly>
    <assemblyIdentity name="Lucene.Net" publicKeyToken="85089178b9ac3181"/>
    <bindingRedirect oldVersion="0.0.0.0-2.9.4.0" newVersion="3.0.3.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed"/>
    <bindingRedirect oldVersion="0.0.0.0-11.0.0.0" newVersion="11.0.0.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
    <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.0.0.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="Lucene.Net" publicKeyToken="85089178b9ac3181"/>
    <bindingRedirect oldVersion="0.0.0.0-2.9.4.0" newVersion="3.0.3.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed"/>
    <bindingRedirect oldVersion="0.0.0.0-11.0.0.0" newVersion="11.0.0.0"/>
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
    <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.0.0.0"/>
  </dependentAssembly>

Removing all the duplicates solved the problem.