[git] Should I check in folder "node_modules" to Git when creating a Node.js app on Heroku?

I followed the basic getting started instructions for Node.js on Heroku here:

https://devcenter.heroku.com/categories/nodejs

These instruction don't tell you to create a .gitignore node_modules, and therefore imply that folder node_modules should be checked in to Git. When I included node_modules in Git repository, my getting started application ran correctly.

When I followed the more advanced example at:

• Building a Real-time, Polyglot Application with Node.js, Ruby, MongoDB and Socket.IO
• https://github.com/mongolab/tractorpush-server (source)

It instructed me to add folder node_modules to file .gitignore. So I removed folder node_modules from Git, added it to file .gitignore, and then redeployed. This time the deployed failed like so:

-----> Heroku receiving push
-----> Node.js app detected
-----> Resolving engine versions
       Using Node.js version: 0.8.2
       Using npm version: 1.0.106
-----> Fetching Node.js binaries
-----> Vendoring node into slug
-----> Installing dependencies with npm
       Error: npm doesn't work with node v0.8.2
       Required: [email protected] || 0.5 || 0.6
           at /tmp/node-npm-5iGk/bin/npm-cli.js:57:23
           at Object.<anonymous> (/tmp/node-npm-5iGk/bin/npm-cli.js:77:3)
           at Module._compile (module.js:449:26)
           at Object.Module._extensions..js (module.js:467:10)
           at Module.load (module.js:356:32)
           at Function.Module._load (module.js:312:12)
           at Module.require (module.js:362:17)
           at require (module.js:378:17)
           at Object.<anonymous> (/tmp/node-npm-5iGk/cli.js:2:1)
           at Module._compile (module.js:449:26)
       Error: npm doesn't work with node v0.8.2
       Required: [email protected] || 0.5 || 0.6
           at /tmp/node-npm-5iGk/bin/npm-cli.js:57:23
           at Object.<anonymous> (/tmp/node-npm-5iGk/bin/npm-cli.js:77:3)
           at Module._compile (module.js:449:26)
           at Object.Module._extensions..js (module.js:467:10)
           at Module.load (module.js:356:32)
           at Function.Module._load (module.js:312:12)
           at Module.require (module.js:362:17)
           at require (module.js:378:17)
           at Object.<anonymous> (/tmp/node-npm-5iGk/cli.js:2:1)
           at Module._compile (module.js:449:26)
       Dependencies installed
-----> Discovering process types
       Procfile declares types -> mongod, redis, web
-----> Compiled slug size is 5.0MB
-----> Launching... done, v9

Running "heroku ps" confirms the crash. OK, no problem, so I rolled back the change, added folder node_module back to the Git repository and removed it from file .gitignore. However, even after reverting, I still get the same error message on deploy, but now the application is running correctly again. Running "heroku ps" tells me the application is running.

What's the right way to do this? Include folder node_modules or not? And why would I still be getting the error message when I rollback? My guess is the Git repository is in a bad state on the Heroku side.

I believe that npm install should not run in a production environment. There are several things that can go wrong - npm outage, download of newer dependencies (shrinkwrap seems to have solved this) are two of them.

On the other hand, folder node_modules should not be committed to Git. Apart from their big size, commits including them can become distracting.

The best solutions would be this: npm install should run in a CI environment that is similar to the production environment. All tests will run and a zipped release file will be created that will include all dependencies.

My biggest concern with not checking folder node_modules into Git is that 10 years down the road, when your production application is still in use, npm may not be around. Or npm might become corrupted; or the maintainers might decide to remove the library that you rely on from their repository; or the version you use might be trimmed out.

This can be mitigated with repository managers like Maven, because you can always use your own local Nexus (Sonatype) or Artifactory to maintain a mirror with the packages that you use. As far as I understand, such a system doesn't exist for npm. The same goes for client-side library managers like Bower and Jam.js.

If you've committed the files to your own Git repository, then you can update them when you like, and you have the comfort of repeatable builds and the knowledge that your application won't break because of some third-party action.

Instead of checking in folder node_modules, make a package.json file for your application.

The package.json file specifies the dependencies of your application. Heroku can then tell npm to install all of those dependencies. The tutorial you linked to contains a section on package.json files.

I am using this solution:

1. Create a separate repository that holds folder node_modules. If you have native modules that should be build for specific platform then create a separate repository for each platform.

2. Attach these repositories to your project repository with git submodule:

   git submodule add .../your_project_node_modules_windows.git node_modules_windows

   git submodule add .../your_project_node_modules_linux_x86_64 node_modules_linux_x86_64

3. Create a link from platform-specific node_modules to node_modules directory and add node_modules to .gitignore.

4. Run npm install.

5. Commit submodule repository changes.

6. Commit your project repository changes.

So you can easily switch between node_modules on different platforms (for example, if you are developing on OS X and deploying to Linux).

Scenario 1:

One scenario:

You use a package that gets removed from npm. If you have all the modules in the folder node_modules, then it won't be a problem for you. If you do only have the package name in the package.json, you can't get it anymore.

If a package is less than 24 hours old, you can easily remove it from npm. If it's older than 24 hours old, then you need to contact them.

But:

If you contact support, they will check to see if removing that version of your package would break any other installs. If so, we will not remove it.

read more

So the chances for this are low, but there is scenario 2...

Scenario 2:

An other scenario where this is the case:

You develop an enterprise version of your software or a very important software and write in your package.json:

"dependencies": {
    "studpid-package": "~1.0.1"
}

You use the method function1(x)of that package.

Now the developers of studpid-package rename the method function1(x)to function2(x) and they make a fault... They change the version of their package from 1.0.1 to 1.1.0. That's a problem because when you call npm install the next time, you will accept version 1.1.0 because you used the tilde ("studpid-package": "~1.0.1").

Calling function1(x) can cause errors and problems now.

Pushing the whole node_modules folder (often more than 100 MB) to your repository, will cost you memory space. A few kb (package.json only) compared with hundreds of MB (package.json & node_modules)... Think about it.

You could do it / should think about it if:

• the software is very important.

• it costs you money when something fails.

• you don't trust the npm registry. npm is centralized and could theoretically be shut down.

You don't need to publish the node_modules folder in 99.9% of the cases if:

• you develop a software just for yourself.

• you've programmed something and just want to publish the result on GitHub because someone else could maybe be interested in it.

If you don't want the node_modules to be in your repository, just create a .gitignore file and add the line node_modules.

I have been using both committing the node_modules folder and shrink-wrapping. Both solutions did not make me happy.

In short: a committed node_modules folder adds too much noise to the repository.And shrinkwrap.json is not easy to manage and there isn't any guarantee that some shrink-wrapped project will build in a few years.

I found that Mozilla was using a separate repository for one of their projects: https://github.com/mozilla-b2g/gaia-node-modules

So it did not take me long to implement this idea in a Node.js CLI tool: https://github.com/bestander/npm-git-lock

Just before every build, add:

npm-git-lock --repo [[email protected]:your/dedicated/node_modules/git/repository.git]

It will calculate the hash of your package.json file and will either check out folder node_modules content from a remote repository, or, if it is a first build for this package.json file, will do a clean npm install and push the results to the remote repository.

I was going to leave this after this comment: Should I check in folder "node_modules" to Git when creating a Node.js app on Heroku?

But Stack Overflow was formatting it weirdly.

If you don't have identical machines and are checking in node_modules, do a .gitignore on the native extensions. Our .gitignore looks like:

# Ignore native extensions in the node_modules folder (things changed by npm rebuild)
node_modules/**/*.node
node_modules/**/*.o
node_modules/**/*.a
node_modules/**/*.mk
node_modules/**/*.gypi
node_modules/**/*.target
node_modules/**/.deps/
node_modules/**/build/Makefile
node_modules/**/**/build/Makefile

Test this by first checking everything in, and then have another developer do the following:

rm -rf node_modules
git checkout -- node_modules
npm rebuild
git status

Ensure that no files changed.

From "node_modules" in Git:

To recap.

• Only checkin node_modules for applications you deploy, not reusable packages you maintain.
• Any compiled dependencies should have their source checked in, not the compile targets, and should $ npm rebuild on deploy.

My favorite part:

All you people who added node_modules to your gitignore, remove that shit, today, it’s an artifact of an era we’re all too happy to leave behind. The era of global modules is dead.

_{(The original link was this one, but it is now dead. Thanks @Flavio for pointing it out.)*}

If you're rolling your own modules specific to your application, you can either:

• Keep those (and only those) in your application's /node_modules folder and move out all the other dependencies to parent ../node_modules folder. This will work because of how NodeJS CommonJS modules system works by moving up to the parent directory, and so on, until the root of the tree is reached. See: https://nodejs.org/api/modules.html

• Or gitignore all /node_modules/* except your /node_modules/your-modules. See: Make .gitignore ignore everything except a few files

This use case is pretty awesome. It lets you keep modules you created specifically for your application nicely with it and doesn't clutter with dependencies which can be installed later.

You should not include folder node_modules in your .gitignore file (or rather you should include folder node_modules in your source deployed to Heroku).

If folder node_modules:

• exists then npm install will use those vendored libraries and will rebuild any binary dependencies with npm rebuild.
• doesn't exist then npm install will have to fetch all dependencies itself which adds time to the slug compile step.

See the Node.js buildpack source for these exact steps.

However, the original error looks to be an incompatibility between the versions of npm and Node.js. It is a good idea to always explicitly set the engines section of your packages.json file according to this guide to avoid these types of situations:

{
  "name": "myapp",
  "version": "0.0.1",
  "engines": {
    "node": "0.8.x",
    "npm":  "1.1.x"
  }
}

This will ensure development/production parity and reduce the likelihood of such situations in the future.

Explicitly adding a npm version to file package.json ("npm": "1.1.x") and not checking in folder node_modules to Git worked for me.

It may be slower to deploy (since it downloads the packages each time), but I couldn't get the packages to compile when they were checked in. Heroku was looking for files that only existed on my local box.