How can I save multiple documents concurrently in Mongoose Node js

Question

At the moment I use save to add a single document   Suppose I have an array of documents that I wish to store as single objects   Is there a way of adding them all with a single function call and then getting a single callback when it is done   I could add all the documents individually but managing the callbacks to work out when everything is done would be problematic

User · Answer

Use insertMany function to insert many documents  This sends only one operation to the server and Mongoose validates all the documents before hitting the mongo server  By default Mongoose inserts item in the order they exist in the array  If you are ok with not maintaining any order then set ordered false   Important - Error handling   When ordered true validation and error handling happens in a group means if one fails everything will fail   When ordered false validation and error handling happens individually and operation will be continued  Error will be reported back in an array of errors

User · Answer

Bulk inserts in Mongoose can be done with  insert   unless you need to access middleware   Model collection insert docs  options  callback   https   github com christkv node-mongodb-native blob master lib mongodb collection js L71-91

User · Answer

You can use the promise returned by mongoose save  Promise in mongoose does not have all  but you can add the feature with this module   Create a module that enhance mongoose promise with all   var Promise   require  mongoose   Promise   Promise all   function promises      var mainPromise   new Promise      if  promises length    0        mainPromise resolve null  promises          var pending   0    promises forEach function p  i        pending        p then function val          promises i    val        if  --pending     0            mainPromise resolve null  promises                  function err          mainPromise reject err                    return mainPromise     module exports   Promise    Then use it with mongoose   var Promise   require    promise         var tasks        for  var i 0  i  lt  docs length  i        tasks push docs i  save        Promise all tasks     then function results        console log results        function  err        console log err

User · Answer

Newer versions of MongoDB support bulk operations   var col   db collection  people    var batch   col initializeUnorderedBulkOp     batch insert  name   John     batch insert  name   Jane     batch insert  name   Jason     batch insert  name   Joanne      batch execute function err  result        if  err  console error err       console log  Inserted     result nInserted     row s

User · Answer

Use async parallel and your code will look like this     async parallel  obj1 save  obj2 save  obj3 save   callback     Since the convention is the same in Mongoose as in async  err  callback  you don t need to wrap them in your own callbacks  just add your save calls in an array and you will get a callback when all is finished   If you use mapLimit you can control how many documents you want to save in parallel  In this example we save 10 documents in parallell until all items are successfully saved   async mapLimit myArray  10  function document  next     document save next      done

User · Answer

Here is another way without using additional libraries  no error checking included   function saveAll  callback      var count   0    docs forEach function doc         doc save function err             count              if  count    docs length                 callback

User · Answer

Here is an example of using MongoDB s Model collection insert   directly in Mongoose  Please note that if you don t have so many documents  say less than 100 documents  you don t need to use MongoDB s bulk operation  see this        MongoDB also supports bulk insert through passing an array of   documents to the db collection insert   method    var mongoose   require  mongoose     var userSchema   mongoose Schema     email     type  String  index    unique  true        name    String         var User   mongoose model  User   userSchema     function saveUsers users      User collection insert users  function callback error  insertedDocs           Here I use KrisKowal s Q  https   github com kriskowal q  to return a promise          so that the caller of this function can act upon its success or failure     if   error        return Q resolve insertedDocs       else       return Q reject   error  error              var users     email   foo bar com   name   foo     email   baz bar com   name   baz     saveUsers users  then function          handle success case here     fail function error         handle error case here

User · Answer

I know this is an old question  but it worries me that there are no properly correct answers here  Most answers just talk about iterating through all the documents and saving each of them individually  which is a BAD idea if you have more than a few documents  and the process gets repeated for even one in many requests      MongoDB specifically has a batchInsert   call for inserting multiple documents  and this should be used from the native mongodb driver  Mongoose is built on this driver  and it doesn t have support for batch inserts  It probably makes sense as it is supposed to be a Object document modelling tool for MongoDB    Solution  Mongoose comes with the native MongoDB driver  You can use that driver by requiring it require  mongoose node modules mongodb    not too sure about this  but you can always install the mongodb npm again if it doesn t work  but I think it should  and then do a proper batchInsert

User · Answer

Mongoose does now support passing multiple document structures to Model create  To quote their API example  it supports being passed either an array or a varargs list of objects with a callback at the end   Candy create   type   jelly bean       type   snickers     function  err  jellybean  snickers        if  err               Or  var array      type   jelly bean       type   snickers      Candy create array  function  err  jellybean  snickers        if  err               Edit  As many have noted  this does not perform a true bulk insert - it simply hides the complexity of calling save multiple times yourself  There are answers and comments below explaining how to use the actual Mongo driver to achieve a bulk insert in the interest of performance

User · Answer

This is an old question  but it came up first for me in google results when searching  mongoose insert array of documents    There are two options model create    mongoose  and model collection insert    mongodb  which you can use  View a more thorough discussion here of the pros cons of each option   Mongoose  mongodb  batch insert

User · Answer

Add a file called mongoHelper js  var MongoClient   require  mongodb   MongoClient   MongoClient saveAny   function data  collection  callback        if data instanceof Array                saveRecords data collection  callback             else               saveRecord data collection  callback            function saveRecord data  collection  callback        collection save               data           w 1           function err  result                        if err                  throw new Error err               callback result                      function saveRecords data  collection  callback        save               data           collection          callback          function save data  collection  callback        collection save               data pop             w 1           function err  result                        if err                                               throw new Error err                             if data length  gt  0                  save data  collection  callback               else                 callback result                       module exports   MongoClient    Then in your code change you requires to  var MongoClient   require    mongoHelper js      Then when it is time to save call  after you have connected and retrieved the collection   MongoClient saveAny data  collection  function   db close         You can change the error handling to suit your needs  pass back the error in the callback etc

User · Answer

Mongoose 4 4 added a method called insertMany     Shortcut for validating an array of documents and inserting them into   MongoDB if they re all valid  This function is faster than  create     because it only sends one operation to the server  rather than one for each   document    Quoting vkarpov15 from issue  723      The tradeoffs are that insertMany   doesn t trigger pre-save hooks  but it should have better performance because it only makes 1 round-trip to the database rather than 1 for each document    The method s signature is identical to create   Model insertMany           err  docs    gt               Or  with promises   Model insertMany          then  docs    gt             catch  err    gt

User · Answer

Mongoose doesn t have bulk inserts implemented yet  see issue  723    Since you know the number of documents you re saving  you could write something like this   var total   docArray length     result         function saveAll      var doc   docArray pop       doc save function err  saved       if  err  throw err   handle error      result push saved 0         if  --total  saveAll        else    all saved here         saveAll      This  of course  is a stop-gap solution and I would recommend using some kind of flow-control library  I use q and it s awesome

[node.js] How can I save multiple documents concurrently in Mongoose/Node.js?

Examples related to node.js

Examples related to mongodb

Examples related to mongoose