Using async await with a forEach loop

Question

Are there any issues with using async await in a forEach loop  I m trying to loop through an array of files and await on the contents of each file    import fs from  fs-promise   async function printFiles        const files   await getFilePaths      Assume this works fine    files forEach async  file    gt        const contents   await fs readFile file   utf8       console log contents          printFiles     This code does work  but could something go wrong with this  I had someone tell me that you re not supposed to use async await in a higher order function like this  so I just wanted to ask if there was any issue with this

User · Answer

Here is a great example for using async in forEach loop.

Write your own asyncForEach

async function asyncForEach(array, callback) {  
    for (let index = 0; index < array.length; index++) {
        await callback(array[index], index, array)
    }
}

You can use it like this

await asyncForEach(array, async function(item,index,array){
     //await here
   }
)

User · Answer

In addition to  Bergi   s answer  I   d like to offer a third alternative  It s very similar to  Bergi   s 2nd example  but instead of awaiting each readFile individually  you create an array of promises  each which you await at the end   import fs from  fs-promise   async function printFiles        const files   await getFilePaths       const promises   files map  file    gt  fs readFile file   utf8       const contents   await Promise all promises     contents forEach console log       Note that the function passed to  map   does not need to be async  since fs readFile returns a Promise object anyway  Therefore promises is an array of Promise objects  which can be sent to Promise all     In  Bergi   s answer  the console may log file contents in the order they   re read  For example if a really small file finishes reading before a really large file  it will be logged first  even if the small file comes after the large file in the files array  However  in my method above  you are guaranteed the console will log the files in the same order as the provided array

User · Answer

it s pretty painless to pop a couple methods in a file that will handle asynchronous data in a serialized order and give a more conventional flavour to your code   For example   module exports   function        var self   this     this each   async  items  fn    gt        if  items  amp  amp  items length          await Promise all          items map async  item    gt              await fn item                              this reduce   async  items  fn  initialValue    gt        await self each        items  async  item    gt            initialValue   await fn initialValue  item                 return initialValue            now  assuming that s saved at    myAsync js  you can do something similar to the below in an adjacent file          your server setup here        var MyAsync   require    myAsync    var Cat   require    models Cat    var Doje   require    models Doje    var example   async      gt      var myAsync   new MyAsync      var doje   await Doje findOne   name   Doje   noises        save      var cleanParams             FOR EACH EXAMPLE   await myAsync each   bork    concern    heck         async  elem    gt          if  elem      heck             await doje update    push     noises   elem                         var cat   await Cat findOne   name   Nyan            REDUCE EXAMPLE   var friendsOfNyanCat   await myAsync reduce cat friends      async  catArray  friendId    gt          var friend   await Friend findById friendId         if  friend name      Long cat             catArray push friend name                            Assuming Long Cat was a friend of Nyan Cat      assert friendsOfNyanCat length      cat friends length - 1

User · Answer

Using Task  futurize  and a traversable List  you can simply do  async function printFiles       const files   await getFiles       List files  traverse  Task of  f   gt  readFile  f   utf-8         fork  console error  console log      Here is how you d set this up  import fs from  fs   import   futurize   from  futurize   import Task from  data task   import   List   from  immutable-ext    const future   futurizeP Task  const readFile   future fs readFile    Another way to have structured the desired code would be  const printFiles   files   gt     List files  traverse  Task of  fn   gt  readFile  fn   utf-8         fork  console error  console log    Or perhaps even more functionally oriented     90  of encodings are utf-8  making that use case super easy is prudent     handy-library js export const readFile   f   gt    future fs readFile   f   utf-8     export const arrayToTaskList   list   gt  taskFn   gt     List files  traverse  Task of  taskFn     export const readFiles   files   gt    arrayToTaskList  files  readFile    export const printFiles   files   gt     readFiles files  fork  console error  console log    Then from the parent function  async function main          awesome code with side-effects before      printFiles  await getFiles           awesome code with side-effects after        If you really wanted more flexibility in encoding  you could just do this  for fun  I m using the proposed Pipe Forward operator    import   curry  flip   from  ramda   export const readFile   fs readFile      gt  future      gt  curry      gt  flip  export const readFileUtf8   readFile  utf-8     PS - I didn t try this code on the console  might have some typos     straight freestyle  off the top of the dome   as the 90s kids would say   -p

User · Answer

Sure the code does work  but I m pretty sure it doesn t do what you expect it to do  It just fires off multiple asynchronous calls  but the printFiles function does immediately return after that   Reading in sequence  If you want to read the files in sequence  you cannot use forEach indeed  Just use a modern for     of loop instead  in which await will work as expected   async function printFiles        const files   await getFilePaths       for  const file of files        const contents   await fs readFile file   utf8        console log contents           Reading in parallel  If you want to read the files in parallel  you cannot use forEach indeed  Each of the async callback function calls does return a promise  but you re throwing them away instead of awaiting them  Just use map instead  and you can await the array of promises that you ll get with Promise all   async function printFiles        const files   await getFilePaths       await Promise all files map async  file    gt        const contents   await fs readFile file   utf8       console log contents

User · Answer

The p-iteration module on npm implements the Array iteration methods so they can be used in a very straightforward way with async await   An example with your case   const   forEach     require  p-iteration    const fs   require  fs-promise      async function printFiles        const files   await getFilePaths       await forEach files  async  file    gt        const contents   await fs readFile file   utf8        console log contents

User · Answer

Bergi s solution works nicely when fs is promise based  You can use bluebird  fs-extra or fs-promise for this   However  solution for node s native fs libary is as follows   const result   await Promise all filePaths      map  async filePath   gt          const fileContents   await getAssetFromCache filePath  async function                 1  Wrap with Promise                2  Return the result of the Promise         return await new Promise  res  rej    gt              fs readFile filePath   utf8   function err  data                if  data                  res data                                                            return fileContents             Note  require  fs   compulsorily takes function as 3rd arguments  otherwise throws error   TypeError  ERR INVALID CALLBACK   Callback must be a function

User · Answer

As other answers have mentioned  you re probably wanting it to be executed in sequence rather in parallel  Ie  run for first file  wait until it s done  then once it s done run for second file  That s not what will happen  I think it s important to address why this doesn t happen  Think about how forEach works  I can t find the source  but I presume it works something like this  const forEach    arr  cb    gt      for  let i   0  i  lt  arr length  i          cb arr i            Now think about what happens when you do something like this  forEach files  async logFile file      const contents   await fs readFile file   utf8      console log contents        Inside forEach s for loop we re calling cb arr i    which ends up being logFile file   The logFile function has an await inside it  so maybe the for loop will wait for this await before proceeding to i    No  it won t  Confusingly  that s not how await works  From the docs   An await splits execution flow  allowing the caller of the async function to resume execution  After the await defers the continuation of the async function  execution of subsequent statements ensues  If this await is the last expression executed by its function execution continues by returning to the function s caller a pending Promise for completion of the await s function and resuming execution of that caller   So if you have the following  the numbers won t be logged before  quot b quot   const delay    ms    gt      return new Promise  resolve    gt        setTimeout resolve  ms             const logNumbers   async      gt      console log 1     await delay 2000     console log 2     await delay 2000     console log 3       const main        gt      console log  quot a quot      logNumbers      console log  quot b quot        main     Circling back to forEach  forEach is like main and logFile is like logNumbers  main won t stop just because logNumbers does some awaiting  and forEach won t stop just because logFile does some awaiting

User · Answer

Today I came across multiple solutions for this  Running the async await functions in the forEach Loop  By building the wrapper around we can make this happen   More detailed explanation on how it works internally  for the native forEach and why it is not able to make a async function call and other details on the various methods are provided in link here  The multiple ways through which it can be done and they are as follows   Method 1   Using the wrapper   await      gt        return new Promise  resolve reject   gt          items forEach async  item index   gt              try                 await someAPICall                 catch e                  console log e                          count               if index     items length-1                resolve  Done                                                  Method 2  Using the same as a generic function of Array prototype  Array prototype forEachAsync js  if  Array prototype forEachAsync        Array prototype forEachAsync   function  fn         return new Promise  resolve reject   gt           this forEach async item index array   gt               await fn item index array               if index     array length-1                   resolve  done                                                    Usage     require    Array prototype forEachAsync     let count   0   let hello   async  items    gt        Method 1 - Using the Array prototype forEach       await items forEachAsync async      gt             try                 await someAPICall                 catch e                  console log e                       count                 console log  count       count      someAPICall        gt        return new Promise  resolve  reject    gt            setTimeout      gt                resolve  done      or reject  error              100             hello                       hello     empty array is also be handled by default   Method 3    Using Promise all    await Promise all items map async  item    gt            await someAPICall            count                  console log  count       count     Method 4   Traditional for loop or modern for loop     Method 4 - using for loop directly     1  Using the modern for    in    loop    for item in items            await someAPICall            count             2  Using the traditional for loop       for let i 0 i lt items length i              await someAPICall            count                console log  count       count

User · Answer

Currently the Array forEach prototype property doesn t support async operations  but we can create our own poly-fill to meet our needs      Example of asyncForEach Array poly-fill for NodeJs    file  asyncForEach js    Define asynForEach function  async function asyncForEach iteratorFunction     let indexer   0   for let data of this       await iteratorFunction data  indexer      indexer            Append it as an Array prototype property Array prototype asyncForEach   asyncForEach module exports    Array    And that s it  You now have an async forEach method available on any arrays that are defined after these to operations   Let s test it        Nodejs style    file  someOtherFile js  const readline   require  readline   Array   require    asyncForEach   Array const log   console log     Create a stream interface function createReader options  prompt    gt        return readline createInterface       input  process stdin      output  process stdout      prompt  options prompt     undefined   options prompt     gt             Create a cli stream reader async function getUserIn question  options  prompt   gt        log question    let reader   createReader options    return new Promise  res   gt       reader on  line    answer   gt         process stdout cursorTo 0  0        process stdout clearScreenDown         reader close         res answer                 let questions        What s your name      What s your favorite programming language      What s your favorite async function    let responses       async function getResponses       Notice we have to prepend await before calling the async Array function    in order for it to function as expected   await questions asyncForEach async function question  index       let answer   await getUserIn question      responses question    answer         async function main      await getResponses     log responses    main      Should prompt user for an answer to each question and then     log each question and answer as an object to the terminal   We could do the same for some of the other array functions like map     async function asyncMap iteratorFunction     let newMap        let indexer   0   for let data of this       newMap indexer    await iteratorFunction data  indexer  this      indexer         return newMap    Array prototype asyncMap   asyncMap       and so on     Some things to note    Your iteratorFunction must be an async function or promise Any arrays created before Array prototype  lt yourAsyncFunc gt     lt yourAsyncFunc gt  will not have this feature available

User · Answer

You can use Array prototype forEach  but async await is not so compatible  This is because the promise returned from an async callback expects to be resolved  but Array prototype forEach does not resolve any promises from the execution of its callback  So then  you can use forEach  but you ll have to handle the promise resolution yourself   Here is a way to read and print each file in series using Array prototype forEach  async function printFilesInSeries        const files   await getFilePaths      let promiseChain   Promise resolve     files forEach  file    gt        promiseChain   promiseChain then      gt          fs readFile file   utf8   then  contents    gt            console log contents                         await promiseChain     Here is a way  still using Array prototype forEach  to print the contents of files in parallel  async function printFilesInParallel        const files   await getFilePaths      const promises        files forEach  file    gt        promises push        fs readFile file   utf8   then  contents    gt            console log contents                        await Promise all promises

User · Answer

With ES2018  you are able to greatly simplify all of the above answers to  async function printFiles        const files   await getFilePaths      for await  const contents of files map file   gt  fs readFile file   utf8           console log contents         See spec  proposal-async-iteration  2018-09-10  This answer has been getting a lot attention recently  please see Axel Rauschmayer s blog post for further information about asynchronous iteration  ES2018  asynchronous iteration

User · Answer

A simple drop-in solution for replacing a forEach   await loop that is not working is replacing forEach with map and adding Promise all  to the beginning  For example  await y forEach async  x    gt    to await Promise all y map async  x    gt    An extra   is needed at the end

User · Answer

Instead of Promise all in conjunction with Array prototype map  which does not guarantee the order in which the Promises are resolved   I use Array prototype reduce  starting with a resolved Promise   async function printFiles        const files   await getFilePaths       await files reduce async  promise  file    gt           This line will wait for the last async function to finish         The first iteration uses an already resolved Promise        so  it will immediately continue      await promise      const contents   await fs readFile file   utf8        console log contents        Promise resolve

User · Answer

Just adding to the original answer   The parallel reading syntax in the original answer is sometimes confusing and difficult to read  maybe we can write it in a different approach   async function printFiles       const files   await getFilePaths      const fileReadPromises          const readAndLogFile   async filePath   gt        const contents   await fs readFile file   utf8        console log contents       return contents          files forEach file   gt        fileReadPromises push readAndLogFile file             await Promise all fileReadPromises         For sequential operation  not just for   of  normal for loop will also work   async function printFiles       const files   await getFilePaths       for  let i   0  i  lt  files length  i          const file   files i       const contents   await fs readFile file   utf8        console log contents

User · Answer

This solution is also memory-optimized so you can run it on 10 000 s of data items and requests  Some of the other solutions here will crash the server on large data sets   In TypeScript   export async function asyncForEach lt T gt  array  Array lt T gt   callback   item  T  index  number    gt  void            for  let index   0  index  lt  array length  index                  await callback array index   index                     How to use   await asyncForEach receipts  async  eachItem    gt        await

User · Answer

One important caveat is  The await   for    of method and the forEach   async way actually have different effect    Having await inside a real for loop will make sure all async calls are executed one by one  And the forEach   async way will fire off all promises at the same time  which is faster but sometimes overwhelmed if you do some DB query or visit some web services with volume restrictions and do not want to fire 100 000 calls at a time     You can also use reduce   promise less elegant  if you do not use async await and want to make sure files are read one after another    files reduce  lastPromise  file    gt    lastPromise then      gt      fs readFile file   utf8       Promise resolve       Or you can create a forEachAsync to help but basically use the same for loop underlying   Array prototype forEachAsync   async function cb       for let x of this           await cb x

User · Answer

To see how that can go wrong  print console log at the end of the method   Things that can go wrong in general    Arbitrary order  printFiles can finish running before printing files  Poor performance    These are not always wrong but frequently are in standard use cases   Generally  using forEach will result in all but the last  It ll call each function without awaiting for the function meaning it tells all of the functions to start then finishes without waiting for the functions to finish   import fs from  fs-promise   async function printFiles        const files    await getFilePaths    map file   gt  fs readFile file   utf8       for const file of files      console log await file     printFiles     This is an example in native JS that will preserve order  prevent the function from returning prematurely and in theory retain optimal performance   This will    Initiate all of the file reads to happen in parallel  Preserve the order via the use of map to map file names to promises to wait for  Wait for each promise in the order defined by the array    With this solution the first file will be shown as soon as it is available without having to wait for the others to be available first   It will also be loading all files at the same time rather than having to wait for the first to finish before the second file read can be started   The only draw back of this and the original version is that if multiple reads are started at once then it s more difficult to handle errors on account of having more errors that can happen at a time   With versions that read a file at a time then then will stop on a failure without wasting time trying to read any more files  Even with an elaborate cancellation system it can be hard to avoid it failing on the first file but reading most of the other files already as well   Performance is not always predictable  While many systems will be faster with parallel file reads some will prefer sequential  Some are dynamic and may shift under load  optimisations that offer latency do not always yield good throughput under heavy contention   There is also no error handling in that example  If something requires them to either all be successfully shown or not at all it won t do that   In depth experimentation is recommended with console log at each stage and fake file read solutions  random delay instead   Although many solutions appear to do the same in simple cases all have subtle differences that take some extra scrutiny to squeeze out   Use this mock to help tell the difference between solutions    async      gt      const start    new Date      const mock        gt        return         fs   readFile  file   gt  new Promise  resolve  reject    gt               Instead of this just make three files and try each timing arrangement             IE  all same   100  200  300    300  200  100    100  300  200   etc          const time   Math round 100   Math random     4900           console log  Read of   file  started at   new Date   - start  and will take   time ms            setTimeout      gt                 Bonus material here if random reject instead            console log  Read of   file  finished  resolving promise at   new Date   - start                resolve file              time                    console   log  file   gt  console log  Console Log of   file  finished at   new Date   - start             getFilePaths       gt    A    B    C    D    E                  const printFiles      fs  console  getFilePaths     gt        return async function           const files    await getFilePaths    map file   gt  fs readFile file   utf8            for const file of files          console log await file               mock        console log  Running at   new Date   - start       await printFiles      console log  Finished running at   new Date   - start

User · Answer

Here are some forEachAsync prototypes  Note you ll need to await them   Array prototype forEachAsync   async function  fn        for  let t of this    await fn t       Array prototype forEachAsyncParallel   async function  fn        await Promise all this map fn        Note while you may include this in your own code  you should not include this in libraries you distribute to others  to avoid polluting their globals

User · Answer

Similar to Antonio Val s p-iteration  an alternative npm module is async-af   const AsyncAF   require  async-af    const fs   require  fs-promise     function printFiles          since AsyncAF accepts promises or non-promises  there s no need to await here   const files   getFilePaths       AsyncAF files  forEach async file   gt        const contents   await fs readFile file   utf8        console log contents            printFiles      Alternatively  async-af has a static method  log logAF  that logs the results of promises   const AsyncAF   require  async-af    const fs   require  fs-promise     function printFiles       const files   getFilePaths       AsyncAF files  forEach file   gt        AsyncAF log fs readFile file   utf8              printFiles      However  the main advantage of the library is that you can chain asynchronous methods to do something like   const aaf   require  async-af    const fs   require  fs-promise     const printFiles        gt  aaf getFilePaths       map file   gt  fs readFile file   utf8       forEach file   gt  aaf log file     printFiles      async-af

User · Answer

Like  Bergi s response  but with one difference   Promise all rejects all promises if one gets rejected   So  use a recursion   const readFilesQueue   async  files  index   0        const contents   await fs readFile files index    utf8       console log contents       return files length  lt   index           readFilesQueue files    index            files     const printFiles async        gt        const files   await getFilePaths        const printContents   await readFilesQueue files       return printContents    printFiles     PS  readFilesQueue is outside of printFiles cause the side effect  introduced by console log  it s better to mock  test  and or spy so  it s not cool to have a function that returns the content sidenote    Therefore  the code can simply be designed by that  three separated functions that are  pure    and introduce no side effects  process the entire list and can easily be modified to handle failed cases   const files   await getFilesPath    const printFile   async  file    gt        const content   await fs readFile file   utf8       console log content     const readFiles   async    files  index   0    gt        await printFile files index        return files lengh  lt   index           readFiles files    index            files    readFiles files    Future edit current state  Node supports top-level await  this doesn t have a plugin yet  won t have and can be enabled via harmony flags   it s cool but doesn t solve one problem  strategically I work only on LTS versions   How to get the files   Using composition  Given the code  causes to me a sensation that this is inside a module  so  should have a function to do it  If not  you should use an IIFE to wrap the role code into an async function creating simple module that s do all for you  or you can go with the right way  there is  composition      more complex version with IIFE to a single module  async  files    gt  readFiles await files    getFilesPath    Note that the name of variable changes due to semantics  You pass a functor  a function that can be invoked by another function  and recieves a pointer on memory that contains the initial block of logic of the application   But  if s not a module and you need to export the logic   Wrap the functions in a async function   export const readFilesQueue   async      gt               to code goes here     Or change the names of variables  whatever         by side effect menans any colacteral effect of application  that can change the statate behaviour or introuce bugs in the application  like IO      by  pure   it s in apostrophe since the functions it s not pure and the code can be converged to a pure version  when there s no console output  only data manipulations   Aside this  to be pure  you ll need to work with monads that handles the side effect  that are error prone  and treats that error separately of the application

User · Answer

If you d like to iterate over all elements concurrently  async function asyncForEach arr  fn      await Promise all arr map fn       If you d like to iterate over all elements non-concurrently  e g  when your mapping function has side effects or running mapper over all array elements at once would be too resource costly   Option A  Promises function asyncForEachStrict arr  fn      return new Promise  resolve    gt        arr reduce         promise  cur  idx    gt  promise          then      gt  fn cur  idx  arr          Promise resolve          then      gt  resolve              Option B  async await async function asyncForEachStrict arr  fn      for  let idx   0  idx  lt  arr length  idx    1        const cur   arr idx        await fn cur  idx  arr

User · Answer

Both the solutions above work  however  Antonio s does the job with less code  here is how it helped me resolve data from my database  from several different child refs and then pushing them all into an array and resolving it in a promise after all is done   Promise all PacksList map  pack   gt       return fireBaseRef child pack folderPath  once  value   snap   gt           snap forEach  childSnap   gt                const file   childSnap val               file id   childSnap key              allItems push  file                         then     gt store dispatch  actions allMockupItems allItems

User · Answer

I would use the well-tested  millions of downloads per week  pify and async modules   If you are unfamiliar with the async module  I highly recommend you check out its docs   I ve seen multiple devs waste time recreating its methods  or worse  making difficult-to-maintain async code when higher-order async methods would simplify code    x000D   x000D  const async   require  async   x000D  const fs   require  fs-promise   x000D  const pify   require  pify   x000D   x000D  async function getFilePaths     x000D      return Promise resolve   x000D             package json   x000D             package-lock json   x000D          x000D    x000D   x000D  async function printFiles      x000D    const files   await getFilePaths   x000D   x000D    await pify async eachSeries  files  async  file    gt         lt -- run in series x000D       await pify async each  files  async  file    gt         lt -- run in parallel x000D      const contents   await fs readFile file   utf8   x000D      console log contents  x000D       x000D    console log  HAMBONE   x000D    x000D   x000D  printFiles   then      gt    x000D      console log  HAMBUNNY   x000D     x000D     ORDER OF LOGS  x000D     package json contents x000D     package-lock json contents x000D     HAMBONE x000D     HAMBUNNY x000D      x000D   x000D   x000D

[javascript] Using async/await with a forEach loop

Here is a great example for using async in forEach loop.

Examples related to javascript

Examples related to node.js

Examples related to promise

Examples related to async-await

Examples related to ecmascript-2017