Mongoose mongodb batch insert

Question

Does Mongoose v3 6  support batch inserts now  I ve searched for a few minutes but anything matching this query is a couple of years old and the answer was an unequivocal no   Edit   For future reference  the answer is to use Model create    create   accepts an array as its first argument  so you can pass your documents to be inserted as an array   See Model create   documentation

User · Answer

Model create   vs Model collection insert    a faster approach  Model create   is a bad way to do inserts if you are dealing with a very large bulk  It will be very slow  In that case you should use Model collection insert  which performs much better  Depending on the size of the bulk  Model create   will even crash  Tried with a million documents  no luck  Using Model collection insert it took just a few seconds   Model collection insert docs  options  callback     docs is the array of documents to be inserted  options is an optional configuration object - see the docs callback err  docs  will be called after all documents get saved or an error occurs  On success  docs is the array of persisted documents    As Mongoose s author points out here  this method will bypass any validation procedures and access the Mongo driver directly  It s a trade-off you have to make since you re handling a large amount of data  otherwise you wouldn t be able to insert it to your database at all  remember we re talking hundreds of thousands of documents here    A simple example  var Potato   mongoose model  Potato   PotatoSchema    var potatoBag       a humongous amount of potato objects       Potato collection insert potatoBag  onInsert    function onInsert err  docs        if  err               TODO  handle error       else           console info   d potatoes were successfully stored    docs length             Update 2019-06-22  although insert   can still be used just fine  it s been deprecated in favor of insertMany    The parameters are exactly the same  so you can just use it as a drop-in replacement and everything should work just fine  well  the return value is a bit different  but you re probably not using it anyway    Reference   Mongo documentation Aaron Heckman on Google Groups discussing bulk inserts

User · Answer

You can perform bulk insert using mongoose  as the highest score answer  But the example cannot work  it should be      a humongous amount of potatos    var potatoBag     name  potato1     name  potato2      var Potato   mongoose model  Potato   PotatoSchema   Potato collection insert potatoBag  onInsert    function onInsert err  docs        if  err               TODO  handle error       else           console info   d potatoes were successfully stored    docs length             Don t use a schema instance for the bulk insert  you should use a plain map object

User · Answer

Here are both way of saving data with insertMany and save  1  Mongoose save array of documents with insertMany in bulk     write mongoose schema model and export this    var Potato   mongoose model  Potato   PotatoSchema       write this api in routes directory     router post   addDocuments   function  req  res        const data       array of object which data need to save in db           Potato insertMany data         then  result    gt                console log  result    result               res status 200  json   success    new documents added     data   result                catch err   gt                console error  error    err               res status 400  json  err                 2  Mongoose save array of documents with  save    These documents will save parallel      write mongoose schema model and export this    var Potato   mongoose model  Potato   PotatoSchema       write this api in routes directory     router post   addDocuments   function  req  res        const saveData          const data       array of object which data need to save in db          data map  i    gt            console log i          var potato   new Potato data i           potato save            then  result    gt                console log result              saveData push result              if  saveData length     data length                    res status 200  json   success    new documents added     data   saveData                                      catch  err    gt                console error err              res status 500  json  err

User · Answer

Sharing working and relevant code from our project     documentsArray is the list of sampleCollection objects sampleCollection insertMany documentsArray         then  res    gt            console log  insert sampleCollection result    res               catch err   gt            console log  bulk insert sampleCollection error    err

User · Answer

You can perform bulk insert using mongoDB shell using inserting the values in an array   db collection insert   values   values   values   values

User · Answer

Mongoose 4 4 0 now supports bulk insert  Mongoose 4 4 0 introduces --true-- bulk insert with the model method  insertMany    It is way faster than looping on  create   or providing it with an array   Usage   var rawDocuments                 Book insertMany rawDocuments       then function mongooseDocuments                                   catch function err               Error handling              Or  Book insertMany rawDocuments  function  err  mongooseDocuments       Your callback function             You can track it on     https   github com Automattic mongoose issues 723 https   github com Automattic mongoose blob 1887e72694829b62f4e3547283783cebbe66b46b lib model js L1774

User · Answer

It seems that using mongoose there is a limit of more than 1000 documents  when using   Potato collection insert potatoBag  onInsert     You can use   var bulk   Model collection initializeOrderedBulkOp     async each users  function  user  callback        bulk insert hash      function  err        var bulkStart   Date now        bulk execute function err  res           if  err  console log    gameResult js  gt  err     err           console log    gameResult js  gt  BULK TIME      Date now   - bulkStart            console log    gameResult js  gt  BULK INSERT     res nInserted                  But this is almost twice as fast when testing with 10000 documents   function fastInsert arrOfResults    var startTime   Date now        var count   0      var c   Math round  arrOfResults length   990        var fakeArr           fakeArr length   c      var docsSaved   0      async each fakeArr  function  item  callback                 var sliced   arrOfResults slice count  count 999               sliced length              count   count  999              if sliced length    0                        GameResultModel collection insert sliced  function  err  docs                                docsSaved    docs ops length                             callback                                         else                       callback                        function  err                console log    gameResult js  gt  BULK INSERT AMOUNT     arrOfResults length   docsSaved      docsSaved    DIFF TIME   Date now   - startTime

User · Answer

Indeed  you can use the  create  method of Mongoose  it can contain an array of documents  see this example   Candy create   candy   jelly bean       candy   snickers     function  err  jellybean  snickers          The callback function contains the inserted documents  You do not always know how many items has to be inserted  fixed argument length like above  so you can loop through them   var insertedDocs       for  var i 1  i lt arguments length    i        insertedDocs push arguments i          Update  A better solution  A better solution would to use Candy collection insert   instead of Candy create   - used in the example above - because it s faster  create   is calling Model save   on each item so it s slower    See the Mongo documentation for more information  http   docs mongodb org manual reference method db collection insert    thanks to arcseldon for pointing this out

[node.js] Mongoose (mongodb) batch insert?

Model.create() vs Model.collection.insert(): a faster approach

A simple example

Reference

Examples related to node.js

Examples related to mongodb

Examples related to mongoose