Node js heap out of memory

Question

Today I ran my script for filesystem indexing to refresh RAID files index and after 4h it crashed with following error    md5    241613 241627 97 5     md5    241614 241627 97 5     md5    241625 241627 98 1  Creating missing list     79570 files missing  Creating new files list     241627 new files    lt --- Last few GCs --- gt   11629672 ms  Mark-sweep 1174 6  1426 5  - gt  1172 4  1418 3  MB  659 9   0 ms  allocation failure   GC in old space requested   11630371 ms  Mark-sweep 1172 4  1418 3  - gt  1172 4  1411 3  MB  698 9   0 ms  allocation failure   GC in old space requested   11631105 ms  Mark-sweep 1172 4  1411 3  - gt  1172 4  1389 3  MB  733 5   0 ms  last resort gc   11631778 ms  Mark-sweep 1172 4  1389 3  - gt  1172 4  1368 3  MB  673 6   0 ms  last resort gc      lt --- JS stacktrace --- gt        JS stack trace                                            Security context  0x3d1d329c9e59  lt JS Object gt  1  SparseJoinWithSeparatorJS aka SparseJoinWithSeparatorJS   native array js  84   pc 0x3629ef689ad0   this 0x3d1d32904189  lt undefined gt  w 0x2b690ce91071  lt JS Array 241627  gt  L 241627 M 0x3d1d329b4a11  lt JS Function ConvertToString  SharedFunctionInfo 0x3d1d3294ef79  gt  N 0x7c953bf4d49  lt String 4      n   gt   2  Join aka Join   native array js 143   pc 0x3629ef616696   this 0x3d1d32904189  lt undefin     FATAL ERROR  CALL AND RETRY LAST Allocation failed - JavaScript heap out of memory  1  node  Abort     usr bin node   2  0xe2c5fc   usr bin node   3  v8  Utils  ReportApiFailure char const   char const     usr bin node   4  v8  internal  V8  FatalProcessOutOfMemory char const   bool    usr bin node   5  v8  internal  Factory  NewRawTwoByteString int  v8  internal  PretenureFlag    usr bin node   6  v8  internal  Runtime SparseJoinWithSeparator int  v8  internal  Object    v8  internal  Isolate     usr bin node   7  0x3629ef50961b   Server is equipped with 16gb RAM and 24gb SSD swap  I highly doubt my script exceeded 36gb of memory  At least it shouldn t  Script creates index of files stored as Array of Objects with files metadata  modification dates  permissions  etc  no big data   Here s full script code  http   pastebin com mjaD76c3  I ve already experiend weird node issues in the past with this script what forced me eg  split index into multiple files as node was glitching when working on such big files as String  Is there any way to improve nodejs memory management with huge datasets

User · Answer

The following environment variable can be used:

NODE_OPTIONS= --max-old-space-size=8192 .

Another note: console.log() also consumes memory in the terminal.

just tried to comment the console.log() on terminal. because that will also take memory.

User · Answer

i was struggling with this even after setting  --max-old-space-size    Then i realised need to put options  --max-old-space-size before the karma script   also best to specify both syntaxes --max-old-space-size and --max old space size my script for karma     node --max-old-space-size 8192 --optimize-for-size --max-executable-size 8192  --max old space size 8192 --optimize for size --max executable size 8192 node modules karma bin karma start --single-run --max new space size 8192   --prod --aot   reference https   github com angular angular-cli issues 1652

User · Answer

You can also change Window s environment variables with    env NODE OPTIONS  quot --max-old-space-size 8192 quot

User · Answer

In my case  I upgraded node js version to latest  version 12 8 0  and it worked like a charm

User · Answer

Here are some flag values to add some additional info on how to allow more memory when you start up your node server    1GB - 8GB   increase to 1gb node --max-old-space-size 1024 index js   increase to 2gb node --max-old-space-size 2048 index js    increase to 3gb node --max-old-space-size 3072 index js   increase to 4gb node --max-old-space-size 4096 index js   increase to 5gb node --max-old-space-size 5120 index js   increase to 6gb node --max-old-space-size 6144 index js   increase to 7gb node --max-old-space-size 7168 index js   increase to 8gb  node --max-old-space-size 8192 index js

User · Answer

Just in case anyone runs into this in an environment where they cannot set node properties directly  in my case a build tool    NODE OPTIONS  --max-old-space-size 4096  node       You can set the node options using an environment variable if you cannot pass them on the command line

User · Answer

Just in case it may help people having this issue while using nodejs apps that produce heavy logging  a colleague solved this issue by piping the standard output s  to a file

User · Answer

If you want to increase the memory usage of the node globally - not only single script  you can export environment variable  like this  export NODE OPTIONS --max old space size 4096  Then you do not need to play with files when running builds like  npm run build

User · Answer

Steps to fix this issue  In Windows  -    Open command prompt and type  appdata  press enter   Navigate to  appdata    npm folder Open or Edit ng cmd in your favorite editor Add --max old space size 8192 to the IF and ELSE block   Your node cmd file looks like this after the change    IF EXIST    dp0 node exe         dp0 node exe   --max old space size 8192     dp0 node modules  angular cli bin ng       ELSE      SETLOCAL    SET PATHEXT  PATHEXT   JS       node  --max old space size 8192     dp0 node modules  angular cli bin ng

User · Answer

fwiw  finding and fixing a memory hog with something like memwatch might help

User · Answer

Upgrade node to the latest version  I was on node 6 6 with this error and upgraded to 8 9 4 and the problem went away

User · Answer

For angular project bundling  I ve added the below line to my pakage json file in the scripts section    build-prod    node --max old space size 5120   node modules  angular cli bin ng build --prod --base-href      Now  to bundle my code  I use npm run build-prod instead of ng build --requiredFlagsHere  hope this helps

User · Answer

If you are trying to launch not node itself  but some other soft  for example webpack you can use the environment variable and cross-env package     cross-env NODE OPTIONS  --max-old-space-size 4096      webpack --progress --config build webpack config dev js

User · Answer

I encountered this issue when trying to debug with VSCode  so just wanted to add this is how you can add the argument to your debug setup    You can add it to the runtimeArgs property of your config in launch json    See example below       version    0 2 0    configurations               type    node            request    launch            name    Launch Program            program      workspaceRoot   server js                        type    node            request    launch            name    Launch Training Script            program      workspaceRoot   training-script js            runtimeArgs                  --max-old-space-size 4096

User · Answer

if you want to change the memory globally for node  windows  go to advanced system settings -  environment variables -  new user variable   variable name   NODE OPTIONS variable value   --max-old-space-size 4096

User · Answer

In my case I had ran npm install on previous version of node  after some day I upgraded node version and ram npm install for few modules  After this I was getting this error  To fix this problem I deleted node module folder from each project and ran npm install again   Hope this might fix the problem   Note   This was happening on my local machine and it got fixed on local machine only

User · Answer

This command works perfectly  I have 8GB ram in my laptop  So I set size 8192  It is all about ram and also you need set file name  I run npm run build command that s why I used build js    node --expose-gc --max-old-space-size 8192 node modules react-scripts scripts build js

User · Answer

I just faced same problem with my EC2 instance t2 micro which has 1 GB memory   I resolved the problem by creating swap file using this url and set following environment variable   export NODE OPTIONS --max old space size 4096  Finally the problem has gone   I hope that would be helpful for future

User · Answer

I had a similar issue while doing AOT angular build  Following commands helped me   npm install -g increase-memory-limit increase-memory-limit   Source  https   geeklearning io angular-aot-webpack-memory-trick

User · Answer

If I remember correctly  there is a strict standard limit for the memory usage in V8 of around 1 7 GB  if you do not increase it manually    In one of our products we followed this solution in our deploy script    node --max-old-space-size 4096 yourFile js   There would also be a new space command but as I read here  a-tour-of-v8-garbage-collection the new space only collects the newly created short-term data and the old space contains all referenced data structures which should be in your case the best option

User · Answer

I just want to add that in some systems  even increasing the node memory limit with --max-old-space-size   it s not enough and there is an OS error like this   terminate called after throwing an instance of  std  bad alloc    what     std  bad alloc Aborted  core dumped    In this case  probably is because you reached the max mmap per process   You can check the max map count by running  sysctl vm max map count   and increas it by running  sysctl -w vm max map count 655300   and fix it to not be reset after a reboot by adding this line  vm max map count 655300   in  etc sysctl conf file   Check here for more info   A good method to analyse the error is by run the process with strace  strace node --max-old-space-size 128000 my memory consuming process js

User · Answer

I ve faced this same problem recently and came across to this thread but my problem was with React App  Below changes in the node start command solved my issues   Syntax  node --max-old-space-size  lt size gt  path-to fileName js   Example  node --max-old-space-size 16000 scripts build js   Why size is 16000 in max-old-space-size   Basically  it varies depends on the allocated memory to that thread and your node settings   How to verify and give right size   This is basically stay in our engine v8  below code helps you to understand the Heap Size of your local node v8 engine   const v8   require  v8    const totalHeapSize   v8 getHeapStatistics   total available size  const totalHeapSizeGb    totalHeapSize   1024   1024   1024  toFixed 2   console log  totalHeapSizeGb     totalHeapSizeGb

User · Answer

For other beginners like me  who didn t find any suitable solution for this error  check the node version installed  x32  x64  x86   I have a 64-bit CPU and I ve installed x86 node version  which caused the CALL AND RETRY LAST Allocation failed - JavaScript heap out of memory error

User · Answer

I have tried  the below code and its working fine     open terminal from project root dir  execute the cmd to set new size   set NODE OPTIONS --max old space size 8172    Or you can check the link for more info https   github com nodejs node issues 10137 issuecomment-487255987

[javascript] Node.js heap out of memory

Examples related to javascript

Examples related to node.js

Examples related to crash

Examples related to out-of-memory

Examples related to heap-memory