[javascript] Node.js on multi-core machines

Node.js looks interesting, BUT I must miss something - isn't Node.js tuned only to run on a single process and thread?

Then how does it scale for multi-core CPUs and multi-CPU servers? After all, it is all great to make fast as possible single-thread server, but for high loads I would want to use several CPUs. And the same goes for making applications faster - seems today the way is use multiple CPUs and parallelize the tasks.

How does Node.js fit into this picture? Is its idea to somehow distribute multiple instances or what?

It's also possible to design the web-service as several stand alone servers that listen to unix sockets, so that you can push functions like data processing into seperate processes.

This is similar to most scrpting/database web server architectures where a cgi process handles business logic and then pushes and pulls the data via a unix socket to a database.

the difference being that the data processing is written as a node webserver listening on a port.

it's more complex but ultimately its where multi-core development has to go. a multiprocess architecture using multiple components for each web request.

[This post is up-to-date as of 2012-09-02 (newer than above).]

Node.js absolutely does scale on multi-core machines.

Yes, Node.js is one-thread-per-process. This is a very deliberate design decision and eliminates the need to deal with locking semantics. If you don't agree with this, you probably don't yet realize just how insanely hard it is to debug multi-threaded code. For a deeper explanation of the Node.js process model and why it works this way (and why it will NEVER support multiple threads), read my other post.

So how do I take advantage of my 16 core box?

Two ways:

• For big heavy compute tasks like image encoding, Node.js can fire up child processes or send messages to additional worker processes. In this design, you'd have one thread managing the flow of events and N processes doing heavy compute tasks and chewing up the other 15 CPUs.
• For scaling throughput on a webservice, you should run multiple Node.js servers on one box, one per core and split request traffic between them. This provides excellent CPU-affinity and will scale throughput nearly linearly with core count.

Scaling throughput on a webservice

Since v6.0.X Node.js has included the cluster module straight out of the box, which makes it easy to set up multiple node workers that can listen on a single port. Note that this is NOT the same as the older learnboost "cluster" module available through npm.

if (cluster.isMaster) {
  // Fork workers.
  for (var i = 0; i < numCPUs; i++) {
    cluster.fork();
  }
} else {
  http.Server(function(req, res) { ... }).listen(8000);
}

Workers will compete to accept new connections, and the least loaded process is most likely to win. It works pretty well and can scale up throughput quite well on a multi-core box.

If you have enough load to care about multiple cores, then you are going to want to do a few more things too:

1. Run your Node.js service behind a web-proxy like Nginx or Apache - something that can do connection throttling (unless you want overload conditions to bring the box down completely), rewrite URLs, serve static content, and proxy other sub-services.

2. Periodically recycle your worker processes. For a long-running process, even a small memory leak will eventually add up.

3. Setup log collection / monitoring

PS: There's a discussion between Aaron and Christopher in the comments of another post (as of this writing, its the top post). A few comments on that:

• A shared socket model is very convenient for allowing multiple processes to listen on a single port and compete to accept new connections. Conceptually, you could think of preforked Apache doing this with the significant caveat that each process will only accept a single connection and then die. The efficiency loss for Apache is in the overhead of forking new processes and has nothing to do with the socket operations.
• For Node.js, having N workers compete on a single socket is an extremely reasonable solution. The alternative is to set up an on-box front-end like Nginx and have that proxy traffic to the individual workers, alternating between workers for assigning new connections. The two solutions have very similar performance characteristics. And since, as I mentioned above, you will likely want to have Nginx (or an alternative) fronting your node service anyways, the choice here is really between:

Shared Ports: nginx (port 80) --> Node_workers x N (sharing port 3000 w/ Cluster)

vs

Individual Ports: nginx (port 80) --> {Node_worker (port 3000), Node_worker (port 3001), Node_worker (port 3002), Node_worker (port 3003) ...}

There are arguably some benefits to the individual ports setup (potential to have less coupling between processes, have more sophisticated load-balancing decisions, etc.), but it is definitely more work to set up and the built-in cluster module is a low-complexity alternative that works for most people.

It's possible to scale NodeJS out to multiple boxes using a pure TCP load balancer (HAProxy) in front of multiple boxes running one NodeJS process each.

If you then have some common knowledge to share between all instances you could use a central Redis store or similar which can then be accessed from all process instances (e.g. from all boxes)

Node Js is supporting clustering to take full advantages of your cpu. If you are not not running it with cluster, then probably you are wasting your hardware capabilities.

Clustering in Node.js allows you to create separate processes which can share same server port. For example, if we run one HTTP server on Port 3000, it is one Server running on Single thread on single core of processor.

Code shown below allow you to cluster your application. This code is official code represented by Node.js.

var cluster = require('cluster');
var numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
    // Fork workers.
    for (var i = 0; i < numCPUs; i++) {
        cluster.fork();
    }

    Object.keys(cluster.workers).forEach(function(id) {
        console.log("I am running with ID : " + cluster.workers[id].process.pid);
    });

    cluster.on('exit', function(worker, code, signal) {
        console.log('worker ' + worker.process.pid + ' died');
    });
} else {

    //Do further processing.
}

check this article for the full tutorial

The cluster module allows you to utilise all cores of your machine. In fact you can take advantage of this in just 2 commands and without touching your code using a very popular process manager pm2.

npm i -g pm2
pm2 start app.js -i max

You may run your node.js application on multiple cores by using the cluster module on combination with os module which may be used to detect how many CPUs you have.

For example let's imagine that you have a server module that runs simple http server on the backend and you want to run it for several CPUs:

// Dependencies._x000D_
const server = require('./lib/server'); // This is our custom server module._x000D_
const cluster = require('cluster');_x000D_
const os = require('os');_x000D_
_x000D_
 // If we're on the master thread start the forks._x000D_
if (cluster.isMaster) {_x000D_
  // Fork the process._x000D_
  for (let i = 0; i < os.cpus().length; i++) {_x000D_
    cluster.fork();_x000D_
  }_x000D_
} else {_x000D_
  // If we're not on the master thread start the server._x000D_
  server.init();_x000D_
}

Ryan Dahl answers this question in the tech talk he gave at Google last summer. To paraphrase, "just run multiple node processes and use something sensible to allow them to communicate. e.g. sendmsg()-style IPC or traditional RPC".

If you want to get your hands dirty right away, check out the spark2 Forever module. It makes spawning multiple node processes trivially easy. It handles setting up port sharing, so they can each accept connections to the same port, and also auto-respawning if you want to make sure a process is restarted if/when it dies.

UPDATE - 10/11/11: Consensus in the node community seems to be that Cluster is now the preferred module for managing multiple node instances per machine. Forever is also worth a look.

Multi-node harnesses all the cores that you may have.
Have a look at http://github.com/kriszyp/multi-node.

For simpler needs, you can start up multiple copies of node on different port numbers and put a load balancer in front of them.

Future version of node will allow you to fork a process and pass messages to it and Ryan has stated he wants to find some way to also share file handlers, so it won't be a straight forward Web Worker implementation.

At this time there is not an easy solution for this but it's still very early and node is one of the fastest moving open source projects I've ever seen so expect something awesome in the near future.

Spark2 is based on Spark which is now no longer maintained. Cluster is its successor, and it has some cool features, like spawning one worker process per CPU core and respawning dead workers.

As mentioned above, Cluster will scale and load-balance your app across all cores.

adding something like

cluster.on('exit', function () {
  cluster.fork();
});

Will restart any failing workers.

These days, a lot of people also prefer PM2, which handles the clustering for you and also provides some cool monitoring features.

Then, add Nginx or HAProxy in front of several machines running with clustering and you have multiple levels of failover and a much higher load capacity.

I'm using Node worker to run processes in a simple way from my main process. Seems to be working great while we wait for the official way to come around.

The new kid on the block here is LearnBoost's "Up".

It provides "Zero-downtime reloads" and additionally creates multiple workers (by default the number of CPUs, but it is configurable) to provide the best of all Worlds.

It is new, but seems to be pretty stable, and I'm using it happily in one of my current projects.

One method would be to run multiple instances of node.js on the server and then put a load balancer (preferably a non-blocking one like nginx) in front of them.

I have to add an important difference between using node's build in cluster mode VS a process manager like PM2's cluster mode.

PM2 allows zero down time reloads when you are running.

pm2 start app.js -i 2 --wait-ready

In your codes add the following

process.send('ready');

When you call pm2 reload app after code updates, PM2 will reload the first instance of the app, wait for the 'ready' call, then it move on to reloads the next instance, ensuring you always have an app active to respond to requests.

While if you use nodejs' cluster, there will be down time when you restart and waiting for server to be ready.

You can use cluster module. Check this.

var cluster = require('cluster');
var http = require('http');
var numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
    // Fork workers.
    for (var i = 0; i < numCPUs; i++) {
        cluster.fork();
    }

    cluster.on('exit', function(worker, code, signal) {
        console.log('worker ' + worker.process.pid + ' died');
    });
} else {
    // Workers can share any TCP connection
    // In this case its a HTTP server
    http.createServer(function(req, res) {
        res.writeHead(200);
        res.end("hello world\n");
    }).listen(8000);
}