Node js How to read a stream into a buffer

Question

I wrote a pretty simple function that downloads an image from a given URL  resize it and upload to S3  using  gm  and  knox    I have no idea if I m doing the reading of a stream to a buffer correctly   everything is working  but is it the correct way    also  I want to understand something about the event loop  how do I know that one invocation of the function won t leak anything or change the  buf  variable to another already running invocation  or this scenario is impossible because the callbacks are anonymous functions    var http   require  http    var https   require  https    var s3   require    s3    var gm   require  gm     module exports processImageUrl   function imageUrl  filename  callback    var client   http  if  imageUrl substr 0  5      https     client   https     client get imageUrl  function res        if  res statusCode    200            return callback new Error  HTTP Response code     res statusCode               gm res           geometry 1024  768    gt             stream  jpg   function err  stdout  stderr                if   err                    var buf   new Buffer 0                   stdout on  data   function d                        buf   Buffer concat  buf  d                                         stdout on  end   function                         var headers                              Content-Length   buf length                            Content-Type    Image jpeg                             x-amz-acl    public-read                                              s3 putBuffer buf    img d     filename     jpg   headers  function err  res                            if err                                return callback err                             else                               return callback null  res client  httpMessage url                                                                                       else                   callback err                                    on  error   function err            callback err

User · Accepted Answer

Overall I don t see anything that would break in your code   Two suggestions   The way you are combining Buffer objects is a suboptimal because it has to copy all the pre-existing data on every  data  event  It would be better to put the chunks in an array and concat them all at the end   var bufs       stdout on  data   function d   bufs push d       stdout on  end   function      var buf   Buffer concat bufs       For performance  I would look into if the S3 library you are using supports streams  Ideally you wouldn t need to create one large buffer at all  and instead just pass the stdout stream directly to the S3 library   As for the second part of your question  that isn t possible  When a function is called  it is allocated its own private context  and everything defined inside of that will only be accessible from other items defined inside that function   Update  Dumping the file to the filesystem would probably mean less memory usage per request  but file IO can be pretty slow so it might not be worth it  I d say that you shouldn t optimize too much until you can profile and stress-test this function  If the garbage collector is doing its job you may be overoptimizing   With all that said  there are better ways anyway  so don t use files  Since all you want is the length  you can calculate that without needing to append all of the buffers together  so then you don t need to allocate a new Buffer at all   var pause stream   require  pause-stream        Your other code   var bufs       stdout on  data   function d   bufs push d       stdout on  end   function      var contentLength   bufs reduce function sum  buf       return sum   buf length       0         Create a stream that will emit your chunks when resumed    var stream   pause stream      stream pause      while  bufs length  stream write bufs shift       stream end       var headers            Content-Length   contentLength                       s3 putStream stream

User · Answer

I suggest  loganfsmyths  method  using an array to hold the data   var bufs       stdout on  data   function d   bufs push d       stdout on  end   function      var buf   Buffer concat bufs       IN my current working example  i am working with GRIDfs and npm s Jimp      var bucket   new GridFSBucket getDBReference      bucketName   images           var dwnldStream   bucket openDownloadStream info 0   id     original size   dwnldStream on  data   function chunk           data push chunk             dwnldStream on  end   function         var buff  Buffer concat data       console log  buffer     buff          jimp read buff   then image   gt             console log  read the image              IMAGE SIZES forEach   size   gt            resize image size                       I did some other research   with a string method but that did not work  per haps because i was reading from an image file  but the array method did work   const DISCLAIMER    DONT DO THIS   var data       stdout on  data   function d               bufs  d                stdout on  end   function              var buf   Buffer from bufs                  do work with the buffer here                  When i did the string method i got this error from npm jimp  buffer    lt Buffer 00 00 00 00 00 gt    Error  Could not find MIME for Buffer  lt null gt    basically i think the type coersion from binary to string didnt work so well

User · Answer

I suggest to have array of buffers and concat to resulting buffer only once at the end  Its easy to do manually  or one could use node-buffers

User · Answer

You can easily do this using node-fetch if you are pulling from http s  URIs   From the readme    fetch  https   assets-cdn github com images modules logos page Octocat png        then res   gt  res buffer         then buffer   gt  console log

User · Answer

in ts     push bufferPart  is not compatible   so    getBufferFromStream stream  Part   null   Promise lt Buffer gt        if   stream            throw  FILE STREAM EMPTY             return new Promise           r  j    gt                let buffer   Buffer from                  stream on  data   buf   gt                   buffer   Buffer concat  buffer  buf                                stream on  end        gt  r buffer                stream on  error   j

User · Answer

I just want to post my solution  Previous answers was pretty helpful for my research  I use length-stream to get the size of the stream  but the problem here is that the callback is fired near the end of the stream  so i also use stream-cache to cache the stream and pipe it to res object once i know the content-length  In case on an error    var StreamCache   require  stream-cache    var lengthStream   require  length-stream     var  streamFile   function res   stream   cb       var cache   new StreamCache         var lstream   lengthStream function length            res header  Content-Length   length           cache pipe res                stream on  error   function err           return cb err                stream on  end   function            return cb null   true                return stream pipe lstream  pipe cache

User · Answer

You can convert your readable stream to a buffer and integrate it in your code in an asynchronous way like this  async streamToBuffer  stream        return new Promise  resolve  reject    gt          const data              stream on  data    chunk    gt            data push chunk                    stream on  end        gt            resolve Buffer concat data                   stream on  error    err    gt            reject err                           the usage would be as simple as      usage   const myStream    your stream   const buffer   await streamToBuffer myStream     this is a buffer

[node.js] Node.js: How to read a stream into a buffer?

Update

Examples related to node.js