[node.js] Mongoose (mongodb) batch insert?

Does Mongoose v3.6+ support batch inserts now? I've searched for a few minutes but anything matching this query is a couple of years old and the answer was an unequivocal no.

Edit:

For future reference, the answer is to use Model.create(). create() accepts an array as its first argument, so you can pass your documents to be inserted as an array.

See Model.create() documentation

This question is related to node.js mongodb mongoose

The answer is


It seems that using mongoose there is a limit of more than 1000 documents, when using

Potato.collection.insert(potatoBag, onInsert);

You can use:

var bulk = Model.collection.initializeOrderedBulkOp();

async.each(users, function (user, callback) {
    bulk.insert(hash);
}, function (err) {
    var bulkStart = Date.now();
    bulk.execute(function(err, res){
        if (err) console.log (" gameResult.js > err " , err);
        console.log (" gameResult.js > BULK TIME  " , Date.now() - bulkStart );
        console.log (" gameResult.js > BULK INSERT " , res.nInserted)
      });
});

But this is almost twice as fast when testing with 10000 documents:

function fastInsert(arrOfResults) {
var startTime = Date.now();
    var count = 0;
    var c = Math.round( arrOfResults.length / 990);

    var fakeArr = [];
    fakeArr.length = c;
    var docsSaved = 0

    async.each(fakeArr, function (item, callback) {

            var sliced = arrOfResults.slice(count, count+999);
            sliced.length)
            count = count +999;
            if(sliced.length != 0 ){
                    GameResultModel.collection.insert(sliced, function (err, docs) {
                            docsSaved += docs.ops.length
                            callback();
                    });
            }else {
                    callback()
            }
    }, function (err) {
            console.log (" gameResult.js > BULK INSERT AMOUNT: ", arrOfResults.length, "docsSaved  " , docsSaved, " DIFF TIME:",Date.now() - startTime);
    });
}

Here are both way of saving data with insertMany and save

1) Mongoose save array of documents with insertMany in bulk

/* write mongoose schema model and export this */
var Potato = mongoose.model('Potato', PotatoSchema);

/* write this api in routes directory  */
router.post('/addDocuments', function (req, res) {
    const data = [/* array of object which data need to save in db */];

    Potato.insertMany(data)  
    .then((result) => {
            console.log("result ", result);
            res.status(200).json({'success': 'new documents added!', 'data': result});
    })
    .catch(err => {
            console.error("error ", err);
            res.status(400).json({err});
    });
})

2) Mongoose save array of documents with .save()

These documents will save parallel.

/* write mongoose schema model and export this */
var Potato = mongoose.model('Potato', PotatoSchema);

/* write this api in routes directory  */
router.post('/addDocuments', function (req, res) {
    const saveData = []
    const data = [/* array of object which data need to save in db */];
    data.map((i) => {
        console.log(i)
        var potato = new Potato(data[i])
        potato.save()
        .then((result) => {
            console.log(result)
            saveData.push(result)
            if (saveData.length === data.length) {
                res.status(200).json({'success': 'new documents added!', 'data': saveData});
            }
        })
        .catch((err) => {
            console.error(err)
            res.status(500).json({err});
        })
    })
})

Sharing working and relevant code from our project:

//documentsArray is the list of sampleCollection objects
sampleCollection.insertMany(documentsArray)  
    .then((res) => {
        console.log("insert sampleCollection result ", res);
    })
    .catch(err => {
        console.log("bulk insert sampleCollection error ", err);
    });

You can perform bulk insert using mongoose, as the highest score answer. But the example cannot work, it should be:

/* a humongous amount of potatos */
var potatoBag = [{name:'potato1'}, {name:'potato2'}];

var Potato = mongoose.model('Potato', PotatoSchema);
Potato.collection.insert(potatoBag, onInsert);

function onInsert(err, docs) {
    if (err) {
        // TODO: handle error
    } else {
        console.info('%d potatoes were successfully stored.', docs.length);
    }
}

Don't use a schema instance for the bulk insert, you should use a plain map object.


You can perform bulk insert using mongoDB shell using inserting the values in an array.

db.collection.insert([{values},{values},{values},{values}]);

Indeed, you can use the "create" method of Mongoose, it can contain an array of documents, see this example:

Candy.create({ candy: 'jelly bean' }, { candy: 'snickers' }, function (err, jellybean, snickers) {
});

The callback function contains the inserted documents. You do not always know how many items has to be inserted (fixed argument length like above) so you can loop through them:

var insertedDocs = [];
for (var i=1; i<arguments.length; ++i) {
    insertedDocs.push(arguments[i]);
}

Update: A better solution

A better solution would to use Candy.collection.insert() instead of Candy.create() - used in the example above - because it's faster (create() is calling Model.save() on each item so it's slower).

See the Mongo documentation for more information: http://docs.mongodb.org/manual/reference/method/db.collection.insert/

(thanks to arcseldon for pointing this out)


Model.create() vs Model.collection.insert(): a faster approach

Model.create() is a bad way to do inserts if you are dealing with a very large bulk. It will be very slow. In that case you should use Model.collection.insert, which performs much better. Depending on the size of the bulk, Model.create() will even crash! Tried with a million documents, no luck. Using Model.collection.insert it took just a few seconds.

Model.collection.insert(docs, options, callback)
  • docs is the array of documents to be inserted;
  • options is an optional configuration object - see the docs
  • callback(err, docs) will be called after all documents get saved or an error occurs. On success, docs is the array of persisted documents.

As Mongoose's author points out here, this method will bypass any validation procedures and access the Mongo driver directly. It's a trade-off you have to make since you're handling a large amount of data, otherwise you wouldn't be able to insert it to your database at all (remember we're talking hundreds of thousands of documents here).

A simple example

var Potato = mongoose.model('Potato', PotatoSchema);

var potatoBag = [/* a humongous amount of potato objects */];

Potato.collection.insert(potatoBag, onInsert);

function onInsert(err, docs) {
    if (err) {
        // TODO: handle error
    } else {
        console.info('%d potatoes were successfully stored.', docs.length);
    }
}

Update 2019-06-22: although insert() can still be used just fine, it's been deprecated in favor of insertMany(). The parameters are exactly the same, so you can just use it as a drop-in replacement and everything should work just fine (well, the return value is a bit different, but you're probably not using it anyway).

Reference


Mongoose 4.4.0 now supports bulk insert

Mongoose 4.4.0 introduces --true-- bulk insert with the model method .insertMany(). It is way faster than looping on .create() or providing it with an array.

Usage:

var rawDocuments = [/* ... */];

Book.insertMany(rawDocuments)
    .then(function(mongooseDocuments) {
         /* ... */
    })
    .catch(function(err) {
        /* Error handling */
    });

Or

Book.insertMany(rawDocuments, function (err, mongooseDocuments) { /* Your callback function... */ });

You can track it on:


Examples related to node.js

Hide Signs that Meteor.js was Used Querying date field in MongoDB with Mongoose SyntaxError: Cannot use import statement outside a module Server Discovery And Monitoring engine is deprecated How to fix ReferenceError: primordials is not defined in node UnhandledPromiseRejectionWarning: This error originated either by throwing inside of an async function without a catch block dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.62.dylib error running php after installing node with brew on Mac internal/modules/cjs/loader.js:582 throw err DeprecationWarning: Buffer() is deprecated due to security and usability issues when I move my script to another server Please run `npm cache clean`

Examples related to mongodb

Server Discovery And Monitoring engine is deprecated Avoid "current URL string parser is deprecated" warning by setting useNewUrlParser to true MongoNetworkError: failed to connect to server [localhost:27017] on first connect [MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017] Failed to auto-configure a DataSource: 'spring.datasource.url' is not specified Failed to start mongod.service: Unit mongod.service not found db.collection is not a function when using MongoClient v3.0 MongoError: connect ECONNREFUSED 127.0.0.1:27017 MongoDB: How To Delete All Records Of A Collection in MongoDB Shell? How to resolve Nodejs: Error: ENOENT: no such file or directory How to create a DB for MongoDB container on start up?

Examples related to mongoose

Querying date field in MongoDB with Mongoose Server Discovery And Monitoring engine is deprecated Avoid "current URL string parser is deprecated" warning by setting useNewUrlParser to true Mongodb: failed to connect to server on first connect Push items into mongo array via mongoose Mongoose: findOneAndUpdate doesn't return updated document mongoError: Topology was destroyed Difference between MongoDB and Mongoose How can you remove all documents from a collection with Mongoose? E11000 duplicate key error index in mongodb mongoose