[javascript] Node.js heap out of memory

Today I ran my script for filesystem indexing to refresh RAID files index and after 4h it crashed with following error:

[md5:]  241613/241627 97.5%  
[md5:]  241614/241627 97.5%  
[md5:]  241625/241627 98.1%
Creating missing list... (79570 files missing)
Creating new files list... (241627 new files)

<--- Last few GCs --->

11629672 ms: Mark-sweep 1174.6 (1426.5) -> 1172.4 (1418.3) MB, 659.9 / 0 ms [allocation failure] [GC in old space requested].
11630371 ms: Mark-sweep 1172.4 (1418.3) -> 1172.4 (1411.3) MB, 698.9 / 0 ms [allocation failure] [GC in old space requested].
11631105 ms: Mark-sweep 1172.4 (1411.3) -> 1172.4 (1389.3) MB, 733.5 / 0 ms [last resort gc].
11631778 ms: Mark-sweep 1172.4 (1389.3) -> 1172.4 (1368.3) MB, 673.6 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3d1d329c9e59 <JS Object>
1: SparseJoinWithSeparatorJS(aka SparseJoinWithSeparatorJS) [native array.js:~84] [pc=0x3629ef689ad0] (this=0x3d1d32904189 <undefined>,w=0x2b690ce91071 <JS Array[241627]>,L=241627,M=0x3d1d329b4a11 <JS Function ConvertToString (SharedFunctionInfo 0x3d1d3294ef79)>,N=0x7c953bf4d49 <String[4]\: ,\n  >)
2: Join(aka Join) [native array.js:143] [pc=0x3629ef616696] (this=0x3d1d32904189 <undefin...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/bin/node]
 2: 0xe2c5fc [/usr/bin/node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [/usr/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/bin/node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/bin/node]
 6: v8::internal::Runtime_SparseJoinWithSeparator(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/bin/node]
 7: 0x3629ef50961b

Server is equipped with 16gb RAM and 24gb SSD swap. I highly doubt my script exceeded 36gb of memory. At least it shouldn't

Script creates index of files stored as Array of Objects with files metadata (modification dates, permissions, etc, no big data)

Here's full script code: http://pastebin.com/mjaD76c3

I've already experiend weird node issues in the past with this script what forced me eg. split index into multiple files as node was glitching when working on such big files as String. Is there any way to improve nodejs memory management with huge datasets?

Steps to fix this issue (In Windows) -

1. Open command prompt and type %appdata% press enter
2. Navigate to %appdata% > npm folder
3. Open or Edit ng.cmd in your favorite editor
4. Add --max_old_space_size=8192 to the IF and ELSE block

Your node.cmd file looks like this after the change:

@IF EXIST "%~dp0\node.exe" (
  "%~dp0\node.exe" "--max_old_space_size=8192" "%~dp0\node_modules\@angular\cli\bin\ng" %*
) ELSE (
  @SETLOCAL
  @SET PATHEXT=%PATHEXT:;.JS;=;%
  node "--max_old_space_size=8192" "%~dp0\node_modules\@angular\cli\bin\ng" %*
)

For other beginners like me, who didn't find any suitable solution for this error, check the node version installed (x32, x64, x86). I have a 64-bit CPU and I've installed x86 node version, which caused the CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory error.

Just in case anyone runs into this in an environment where they cannot set node properties directly (in my case a build tool):

NODE_OPTIONS="--max-old-space-size=4096" node ...

You can set the node options using an environment variable if you cannot pass them on the command line.

In my case I had ran npm install on previous version of node, after some day I upgraded node version and ram npm install for few modules. After this I was getting this error. To fix this problem I deleted node_module folder from each project and ran npm install again.

Hope this might fix the problem.

Note : This was happening on my local machine and it got fixed on local machine only.

I have tried the below code and its working fine?.

• open terminal from project root dir

• execute the cmd to set new size.

  set NODE_OPTIONS=--max_old_space_size=8172
  

Or you can check the link for more info https://github.com/nodejs/node/issues/10137#issuecomment-487255987

In my case, I upgraded node.js version to latest (version 12.8.0) and it worked like a charm.

If you are trying to launch not node itself, but some other soft, for example webpack you can use the environment variable and cross-env package:

$ cross-env NODE_OPTIONS='--max-old-space-size=4096' \
  webpack --progress --config build/webpack.config.dev.js

Here are some flag values to add some additional info on how to allow more memory when you start up your node server.

1GB - 8GB

#increase to 1gb
node --max-old-space-size=1024 index.js

#increase to 2gb
node --max-old-space-size=2048 index.js 

#increase to 3gb
node --max-old-space-size=3072 index.js

#increase to 4gb
node --max-old-space-size=4096 index.js

#increase to 5gb
node --max-old-space-size=5120 index.js

#increase to 6gb
node --max-old-space-size=6144 index.js

#increase to 7gb
node --max-old-space-size=7168 index.js

#increase to 8gb 
node --max-old-space-size=8192 index.js 

The following environment variable can be used:

NODE_OPTIONS= --max-old-space-size=8192 .

Another note: console.log() also consumes memory in the terminal.

just tried to comment the console.log() on terminal. because that will also take memory.

If you want to increase the memory usage of the node globally - not only single script, you can export environment variable, like this:
export NODE_OPTIONS=--max_old_space_size=4096

Then you do not need to play with files when running builds like npm run build.

Just in case it may help people having this issue while using nodejs apps that produce heavy logging, a colleague solved this issue by piping the standard output(s) to a file.

This command works perfectly. I have 8GB ram in my laptop, So I set size=8192. It is all about ram and also you need set file name. I run npm run build command that's why I used build.js.

node --expose-gc --max-old-space-size=8192 node_modules/react-scripts/scripts/build.js

I encountered this issue when trying to debug with VSCode, so just wanted to add this is how you can add the argument to your debug setup.

You can add it to the runtimeArgs property of your config in launch.json.

See example below.

{
"version": "0.2.0",
"configurations": [{
        "type": "node",
        "request": "launch",
        "name": "Launch Program",
        "program": "${workspaceRoot}\\server.js"
    },
    {
        "type": "node",
        "request": "launch",
        "name": "Launch Training Script",
        "program": "${workspaceRoot}\\training-script.js",
        "runtimeArgs": [
            "--max-old-space-size=4096"
        ]
    }
]}

You can also change Window's environment variables with:

 $env:NODE_OPTIONS="--max-old-space-size=8192"

Upgrade node to the latest version. I was on node 6.6 with this error and upgraded to 8.9.4 and the problem went away.

if you want to change the memory globally for node (windows) go to advanced system settings -> environment variables -> new user variable

variable name = NODE_OPTIONS
variable value = --max-old-space-size=4096

I've faced this same problem recently and came across to this thread but my problem was with React App. Below changes in the node start command solved my issues.

Syntax

node --max-old-space-size=<size> path-to/fileName.js

Example

node --max-old-space-size=16000 scripts/build.js

Why size is 16000 in max-old-space-size?

Basically, it varies depends on the allocated memory to that thread and your node settings.

How to verify and give right size?

This is basically stay in our engine v8. below code helps you to understand the Heap Size of your local node v8 engine.

const v8 = require('v8');
const totalHeapSize = v8.getHeapStatistics().total_available_size;
const totalHeapSizeGb = (totalHeapSize / 1024 / 1024 / 1024).toFixed(2);
console.log('totalHeapSizeGb: ', totalHeapSizeGb);

I just faced same problem with my EC2 instance t2.micro which has 1 GB memory.

I resolved the problem by creating swap file using this url and set following environment variable.

export NODE_OPTIONS=--max_old_space_size=4096

Finally the problem has gone.

I hope that would be helpful for future.

fwiw, finding and fixing a memory hog with something like memwatch might help.

I just want to add that in some systems, even increasing the node memory limit with --max-old-space-size, it's not enough and there is an OS error like this:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

In this case, probably is because you reached the max mmap per process.

You can check the max_map_count by running

sysctl vm.max_map_count

and increas it by running

sysctl -w vm.max_map_count=655300

and fix it to not be reset after a reboot by adding this line

vm.max_map_count=655300

in /etc/sysctl.conf file.

Check here for more info.

A good method to analyse the error is by run the process with strace

strace node --max-old-space-size=128000 my_memory_consuming_process.js

i was struggling with this even after setting --max-old-space-size.

Then i realised need to put options --max-old-space-size before the karma script.

also best to specify both syntaxes --max-old-space-size and --max_old_space_size my script for karma :

node --max-old-space-size=8192 --optimize-for-size --max-executable-size=8192  --max_old_space_size=8192 --optimize_for_size --max_executable_size=8192 node_modules/karma/bin/karma start --single-run --max_new_space_size=8192   --prod --aot

reference https://github.com/angular/angular-cli/issues/1652

For angular project bundling, I've added the below line to my pakage.json file in the scripts section.

"build-prod": "node --max_old_space_size=5120 ./node_modules/@angular/cli/bin/ng build --prod --base-href /"

Now, to bundle my code, I use npm run build-prod instead of ng build --requiredFlagsHere

hope this helps!

I had a similar issue while doing AOT angular build. Following commands helped me.

npm install -g increase-memory-limit
increase-memory-limit

Source: https://geeklearning.io/angular-aot-webpack-memory-trick/