System OutOfMemoryException was thrown when there is still plenty of memory free

Question

This is my code   int size   100000000  double sizeInMegabytes    size   8 0    1024 0   1024 0    762 mb double   randomNumbers   new double size     Exception  Exception of type  System OutOfMemoryException  was thrown   I have 4GB memory on this machine 2 5GB is free when I start this running  there is clearly enough space on the PC to handle the 762mb of 100000000 random numbers  I need to store as many random numbers as possible given available memory  When I go to production there will be 12GB on the box and I want to make use of it   Does the CLR constrain me to a default max memory to start with  and how do I request more   Update  I thought breaking this into smaller chunks and incrementally adding to my memory requirements would help if the issue is due to memory fragmentation  but it doesn t I can t get past a total ArrayList size of 256mb regardless of what I do tweaking blockSize   private static IRandomGenerator rnd   new MersenneTwister    private static IDistribution dist   new DiscreteNormalDistribution 1048576   private static List lt double gt  ndRandomNumbers   new List lt double gt      private static void AddNDRandomNumbers int numberOfRandomNumbers        for  int i   0  i  lt  numberOfRandomNumbers  i            ndRandomNumbers Add dist ICDF rnd nextUniform                              From my main method   int blockSize   1000000   while  true      try         AddNDRandomNumbers blockSize                             catch  System OutOfMemoryException ex          break                    double arrayTotalSizeInMegabytes    ndRandomNumbers Count   8 0    1024 0   1024 0

User · Answer

Well  I got a similar problem with large data set and trying to force the application to use so much data is not really the right option  The best tip I can give you is to process your data in small chunk if it is possible  Because dealing with so much data  the problem will come back sooner or later  Plus  you cannot know the configuration of each machine that will run your application so there s always a risk that the exception will happens on another pc

User · Answer

As you probably figured out  the issue is that you are trying to allocate one large contiguous block of memory  which does not work due to memory fragmentation   If I needed to do what you are doing I would do the following   int sizeA   10000      sizeB   10000  double sizeInMegabytes    sizeA   sizeB   8 0    1024 0   1024 0    762 mb double     randomNumbers   new double sizeA     for  int i   0  i  lt  randomNumbers Length  i          randomNumbers i    new double sizeB       Then  to get a particular index you would use randomNumbers i   sizeB  i   sizeB    Another option if you always access the values in order might be to use the overloaded constructor to specify the seed   This way you would get a semi random number  like the DateTime Now Ticks  store it in a variable  then when ever you start going through the list you would create a new Random instance using the original seed   private static int randSeed    int DateTime Now Ticks     Must stay the same unless you want to get different random numbers  private static Random GetNewRandomIterator         return new Random randSeed         It is important to note that while the blog linked in Fredrik M  rk s answer indicates that the issue is usually due to a lack of address space it does not list a number of other issues  like the 2GB CLR object size limitation  mentioned in a comment from ShuggyCoUk on the same blog   glosses over memory fragmentation  and fails to mention the impact of page file size  and how it can be addressed with the use of the CreateFileMapping function    The 2GB limitation means that randomNumbers  must be less than 2GB   Since arrays are classes and have some overhead them selves this means an array of double will need to be smaller then 2 31   I am not sure how much smaller then 2 31 the Length would have to be  but Overhead of a  NET array  indicates 12 - 16 bytes   Memory fragmentation is very similar to HDD fragmentation   You might have 2GB of address space  but as you create and destroy objects there will be gaps between the values   If these gaps are too small for your large object  and additional space can not be requested  then you will get the System OutOfMemoryException   For example  if you create 2 million  1024 byte objects  then you are using 1 9GB   If you delete every object where the address is not a multiple of 3 then you will be using  6GB of memory  but it will be spread out across the address space with 2024 byte open blocks in between   If you need to create an object which was  2GB you would not be able to do it because there is not a block large enough to fit it in and additional space cannot be obtained  assuming a 32 bit environment    Possible solutions to this issue are things like using smaller objects  reducing the amount of data you store in memory  or using a memory management algorithm to limit prevent memory fragmentation   It should be noted that unless you are developing a large program which uses a large amount of memory this will not be an issue   Also  this issue can arise on 64 bit systems as windows is limited mostly by the page file size and the amount of RAM on the system   Since most programs request working memory from the OS and do not request a file mapping  they will be limited by the system s RAM and page file size   As noted in the comment by N  stor S  nchez  N  233 stor S  225 nchez  on the blog  with managed code like C  you are stuck to the RAM page file limitation and the address space of the operating system     That was way longer then expected   Hopefully it helps someone   I posted it because I ran into the System OutOfMemoryException running a x64 program on a system with 24GB of RAM even though my array was only holding 2GB of stuff

User · Answer

32bit windows has a 2GB process memory limit  The  3GB boot option others have mentioned will make this 3GB with just 1gb remaining for OS kernel use  Realistically if you want to use more than 2GB without hassle then a 64bit OS is required  This also overcomes the problem whereby although you may have 4GB of physical RAM  the address space requried for the video card can make a sizeable chuck of that memory unusable - usually around 500MB

User · Answer

I had a similar problem  it was due to a StringBuilder ToString

User · Answer

If you do not need the Visual Studio Hosting Process   Uncheck the option  Project- Properties- Debug- Enable the Visual Studio Hosting Process  And then build   If you still face the problem   Go to Project- Properties- Build Events- Post-Build Event Command line and paste the following   call    DevEnvDir       vc vcvarsall bat  x86    DevEnvDir       vc bin EditBin exe     TargetPath     LARGEADDRESSAWARE   Now  build the project

User · Answer

You don t have a continuous block of memory in order to allocate 762MB  your memory is fragmented and the allocator cannot find a big enough hole to allocate the needed memory      You can try to work with  3GB  as others had suggested  Or switch to 64 bit OS  Or modify the algorithm so it will not need a big chunk of memory  maybe allocate a few smaller  relatively  chunks of memory

User · Answer

Rather than allocating a massive array  could you try utilizing an iterator  These are delay-executed  meaning values are generated only as they re requested in an foreach statement  you shouldn t run out of memory this way   private static IEnumerable lt double gt  MakeRandomNumbers int numberOfRandomNumbers         for  int i   0  i  lt  numberOfRandomNumbers  i                  yield return randomGenerator GetAnotherRandomNumber                      Hooray  we won t run out of memory  foreach var number in MakeRandomNumbers int MaxValue         Console WriteLine number       The above will generate as many random numbers as you wish  but only generate them as they re asked for via a foreach statement  You won t run out of memory that way   Alternately  If you must have them all in one place  store them in a file rather than in memory

User · Answer

Increase the Windows process limit to 3gb   via boot ini or Vista boot manager

User · Answer

Convert your solution to x64  If you still face an issue  grant max length to everything that throws an exception like below     var jsSerializer   new JavaScriptSerializer     jsSerializer MaxJsonLength   Int32 MaxValue

User · Answer

If you need such large structures  perhaps you could utilize Memory Mapped Files  This article could prove helpful  http   www codeproject com KB recipes MemoryMappedGenericArray aspx  LP  Dejan

User · Answer

Check that you are building a 64-bit process  and not a 32-bit one  which is the default compilation mode of Visual Studio  To do this  right click on your project  Properties -  Build -  platform target   x64  As any 32-bit process  Visual Studio applications compiled in 32-bit have a virtual memory limit of 2GB   64-bit processes do not have this limitation  as they use 64-bit pointers  so their theoretical maximum address space  the size of their virtual memory  is 16 exabytes  2 64   In reality  Windows x64 limits the virtual memory of processes to 8TB  The solution to the memory limit problem is then to compile in 64-bit   However  object   s size in Visual Studio is still limited to 2GB  by default  You will be able to create several arrays whose combined size will be greater than 2GB  but you cannot by default create arrays bigger than 2GB  Hopefully  if you still want to create arrays bigger than 2GB  you can do it by adding the following code to you app config file    lt configuration gt     lt runtime gt       lt gcAllowVeryLargeObjects enabled  true    gt     lt  runtime gt   lt  configuration gt

User · Answer

Changing from 32 to 64 bit worked for me - worth a try if you are on a 64 bit pc and it doesn t need to port

User · Answer

I d advise against the  3GB windows boot option  Apart from everything else  it s overkill to do this for one badly behaved application  and it probably won t solve your problem anyway   it can cause a lot of instability   Many Windows drivers are not tested with this option  so quite a few of them assume that user-mode pointers always point to the lower 2GB of the address space  Which means they may break horribly with  3GB   However  Windows does normally limit a 32-bit process to a 2GB address space  But that doesn t mean you should expect to be able to allocate 2GB   The address space is already littered with all sorts of allocated data  There s the stack  and all the assemblies that are loaded  static variables and so on  There s no guarantee that there will be 800MB of contiguous unallocated memory anywhere   Allocating 2 400MB chunks would probably fare better  Or 4 200MB chunks  Smaller allocations are much easier to find room for in a fragmented memory space   Anyway  if you re going to deploy this to a 12GB machine anyway  you ll want to run this as a 64-bit application  which should solve all the problems

User · Answer

You may want to read this      Out Of Memory    Does Not Refer to Physical Memory  by Eric Lippert   In short  and very simplified   Out of memory  does not really mean that the amount of available memory is too small  The most common reason is that within the current address space  there is no contiguous portion of memory that is large enough to serve the wanted allocation  If you have 100 blocks  each 4 MB large  that is not going to help you when you need one 5 MB block   Key Points     the data storage that we call    process memory    is in my opinion best visualized as a massive file on disk  RAM can be seen as merely a performance optimization Total amount of virtual memory your program consumes is really not hugely relevant to its performance  running out of RAM  seldom results in an    out of memory    error  Instead of an error  it results in bad performance because the full cost of the fact that storage is actually on disk suddenly becomes relevant

[c#] 'System.OutOfMemoryException' was thrown when there is still plenty of memory free

Examples related to c#

Examples related to memory-management

Examples related to out-of-memory