[javascript] JavaScript, Node.js: is Array.forEach asynchronous?

I have a question regarding the native Array.forEach implementation of JavaScript: Does it behave asynchronously? For example, if I call:

[many many elements].forEach(function () {lots of work to do})

Will this be non-blocking?

Here is a small example you can run to test it:

[1,2,3,4,5,6,7,8,9].forEach(function(n){
    var sum = 0;
    console.log('Start for:' + n);
    for (var i = 0; i < ( 10 - n) * 100000000; i++)
        sum++;

    console.log('Ended for:' + n, sum);
});

It will produce something like this(if it takes too less/much time, increase/decrease the number of iterations):

(index):48 Start for:1
(index):52 Ended for:1 900000000
(index):48 Start for:2
(index):52 Ended for:2 800000000
(index):48 Start for:3
(index):52 Ended for:3 700000000
(index):48 Start for:4
(index):52 Ended for:4 600000000
(index):48 Start for:5
(index):52 Ended for:5 500000000
(index):48 Start for:6
(index):52 Ended for:6 400000000
(index):48 Start for:7
(index):52 Ended for:7 300000000
(index):48 Start for:8
(index):52 Ended for:8 200000000
(index):48 Start for:9
(index):52 Ended for:9 100000000
(index):45 [Violation] 'load' handler took 7285ms

If you need an asynchronous-friendly version of Array.forEach and similar, they're available in the Node.js 'async' module: http://github.com/caolan/async ...as a bonus this module also works in the browser.

async.each(openFiles, saveFile, function(err){
    // if any of the saves produced an error, err would equal that error
});

It is possible to code even the solution like this for example :

 var loop = function(i, data, callback) {
    if (i < data.length) {
        //TODO("SELECT * FROM stackoverflowUsers;", function(res) {
            //data[i].meta = res;
            console.log(i, data[i].title);
            return loop(i+1, data, errors, callback);
        //});
    } else {
       return callback(data);
    }
};

loop(0, [{"title": "hello"}, {"title": "world"}], function(data) {
    console.log("DONE\n"+data);
});

On the other hand, it is much slower than a "for".

Otherwise, the excellent Async library can do this: https://caolan.github.io/async/docs.html#each

There is a package on npm for easy asynchronous for each loops.

var forEachAsync = require('futures').forEachAsync;

// waits for one request to finish before beginning the next 
forEachAsync(['dogs', 'cats', 'octocats'], function (next, element, index, array) {
  getPics(element, next);
  // then after all of the elements have been handled 
  // the final callback fires to let you know it's all done 
  }).then(function () {
    console.log('All requests have finished');
});

Also another variation forAllAsync

This is a short asynchronous function to use without requiring third party libs

Array.prototype.each = function (iterator, callback) {
    var iterate = function () {
            pointer++;
            if (pointer >= this.length) {
                callback();
                return;
            }
            iterator.call(iterator, this[pointer], iterate, pointer);
    }.bind(this),
        pointer = -1;
    iterate(this);
};

Use Promise.each of bluebird library.

Promise.each(
Iterable<any>|Promise<Iterable<any>> input,
function(any item, int index, int length) iterator
) -> Promise

This method iterates over an array, or a promise of an array, which contains promises (or a mix of promises and values) with the given iterator function with the signature (value, index, length) where the value is the resolved value of a respective promise in the input array. Iteration happens serially. If the iterator function returns a promise or a thenable, then the result of the promise is awaited before continuing with next iteration. If any promise in the input array is rejected, then the returned promise is rejected as well.

If all of the iterations resolve successfully, Promise.each resolves to the original array unmodified. However, if one iteration rejects or errors, Promise.each ceases execution immediately and does not process any further iterations. The error or rejected value is returned in this case instead of the original array.

This method is meant to be used for side effects.

var fileNames = ["1.txt", "2.txt", "3.txt"];

Promise.each(fileNames, function(fileName) {
    return fs.readFileAsync(fileName).then(function(val){
        // do stuff with 'val' here.  
    });
}).then(function() {
console.log("done");
});

There is a common pattern for doing a really heavy computation in Node that may be applicable to you...

Node is single-threaded (as a deliberate design choice, see What is Node.js?); this means that it can only utilize a single core. Modern boxes have 8, 16, or even more cores, so this could leave 90+% of the machine idle. The common pattern for a REST service is to fire up one node process per core, and put these behind a local load balancer like http://nginx.org/.

Forking a child - For what you are trying to do, there is another common pattern, forking off a child process to do the heavy lifting. The upside is that the child process can do heavy computation in the background while your parent process is responsive to other events. The catch is that you can't / shouldn't share memory with this child process (not without a LOT of contortions and some native code); you have to pass messages. This will work beautifully if the size of your input and output data is small compared to the computation that must be performed. You can even fire up a child node.js process and use the same code you were using previously.

For example:

var child_process = require('child_process');
function run_in_child(array, cb) {
    var process = child_process.exec('node libfn.js', function(err, stdout, stderr) {
        var output = JSON.parse(stdout);
        cb(err, output);
    });
    process.stdin.write(JSON.stringify(array), 'utf8');
    process.stdin.end();
}

Array.forEach is meant for computing stuff not waiting, and there is nothing to be gained making computations asynchronous in an event loop (webworkers add multiprocessing, if you need multi-core computation). If you want to wait for multiple tasks to end, use a counter, which you can wrap in a semaphore class.

Although Array.forEach is not asynchronous, you can get asynchronous "end result". Example below:

function delayFunction(x) {
    return new Promise(
        (resolve) => setTimeout(() => resolve(x), 1000)
    );
}

[1, 2, 3].forEach(async(x) => {
    console.log(x);
    console.log(await delayFunction(x));
});

Edit 2018-10-11: It looks like there is a good chance the standard described below may not go through, consider pipelineing as an alternative (does not behave exactly the same but methods could be implemented in a similar manor).

This is exactly why I am excited about es7, in future you will be able to do something like the code below (some of the specs are not complete so use with caution, I will try to keep this up to date). But basically using the new :: bind operator, you will be able to run a method on an object as if the object's prototype contains the method. eg [Object]::[Method] where normally you would call [Object].[ObjectsMethod]

Note to do this today (24-July-16) and have it work in all browsers you will need to transpile your code for the following functionality:Import / Export, Arrow functions, Promises, Async / Await and most importantly function bind. The code below could be modfied to use only function bind if nessesary, all this functionality is neatly available today by using babel.

YourCode.js (where 'lots of work to do' must simply return a promise, resolving it when the asynchronous work is done.)

import { asyncForEach } from './ArrayExtensions.js';

await [many many elements]::asyncForEach(() => lots of work to do);

ArrayExtensions.js

export function asyncForEach(callback)
{
    return Promise.resolve(this).then(async (ar) =>
    {
        for(let i=0;i<ar.length;i++)
        {
            await callback.call(ar, ar[i], i, ar);
        }
    });
};

export function asyncMap(callback)
{
    return Promise.resolve(this).then(async (ar) =>
    {
        const out = [];
        for(let i=0;i<ar.length;i++)
        {
            out[i] = await callback.call(ar, ar[i], i, ar);
        }
        return out;
    });
};