How to append binary data to a buffer in node js

Question

I have a buffer with some binary data   var b   new Buffer   0x00  0x01  0x02      and I want to append 0x03   How can I append more binary data  I m searching in the documentation but for appending data it must be a string  if not  an error occurs  TypeError  Argument must be a string    var b   new Buffer  256   b write   hola    console log  b toString   utf8   0  4      hola b write     adios   4   console log  b toString   utf8   0  11      hola  adios   Then  the only solution I can see here is to create a new buffer for every appended binary data and copy it to the major buffer with the correct offset   var b   new Buffer  4     4 for having a nice printed buffer  but the size will be 16KB new Buffer   0x00  0x01  0x02   copy  b   console log  b      lt Buffer 00 01 02 00 gt  new Buffer   0x03   copy  b  3   console log  b      lt Buffer 00 01 02 03 gt    But this seems a bit inefficient because I have to instantiate a new buffer for every append   Do you know a better way for appending binary data   EDIT  I ve written a BufferedWriter that writes bytes to a file using internal buffers  Same as BufferedReader but for writing   A quick example     The BufferedWriter truncates the file because append    false new BufferedWriter   file        on   error   function  error           console log  error                 From the beginning of the file       write   0x00  0x01  0x02   0  3    Writes 0x00  0x01  0x02      write  new Buffer   0x03  0x04    1  1    Writes 0x04      write  0x05    Writes 0x05      close       Closes the writer  A flush is implicitly done     The BufferedWriter appends content to the end of the file because append    true new BufferedWriter   file   true       on   error   function  error           console log  error                 From the end of the file       write  0xFF    Writes 0xFF      close       Closes the writer  A flush is implicitly done     The file contains  0x00  0x01  0x02  0x04  0x05  0xFF   LAST UPDATE  Use concat

User · Accepted Answer

Updated Answer for Node js   0 8  Node is able to concatenate buffers on its own now   var newBuffer   Buffer concat  buffer1  buffer2      Old Answer for Node js  0 6  I use a module to add a  concat function  among others   https   github com coolaj86 node-bufferjs  I know it isn t a  pure  solution  but it works very well for my purposes

User · Answer

This is to help anyone who comes here looking for a solution that wants a pure approach  I would recommend understanding this problem because it can happen in lots of different places not just with a JS Buffer object  By understanding why the problem exists and how to solve it you will improve your ability to solve other problems in the future since this one is so fundamental   For those of us that have to deal with these problems in other languages it is quite natural to devise a solution  but there are people who may not realize how to abstract away the complexities and implement a generally efficient dynamic buffer  The code below may have potential to be optimized further   I have left the read method unimplemented to keep the example small in size   The realloc function in C  or any language dealing with intrinsic allocations  does not guarantee that the allocation will be expanded in size with out moving the existing data - although sometimes it is possible  Therefore most applications when needing to store a unknown amount of data will use a method like below and not constantly reallocate  unless the reallocation is very infrequent  This is essentially how most file systems handle writing data to a file  The file system simply allocates another node and keeps all the nodes linked together  and when you read from it the complexity is abstracted away so that the file buffer appears to be a single contiguous buffer   For those of you who wish to understand the difficulty in just simply providing a high performance dynamic buffer you only need to view the code below  and also do some research on memory heap algorithms and how the memory heap works for programs   Most languages will provide a fixed size buffer for performance reasons  and then provide another version that is dynamic in size  Some language systems opt for a third-party system where they keep the core functionality minimal  core distribution  and encourage developers to create libraries to solve additional or higher level problems  This is why you may question why a language does not provide some functionality  This small core functionality allows costs to be reduced in maintaining and enhancing the language  however you end up having to write your own implementations or depending on a third-party   var Buffer A1   function  chunk size        this buffer list           this total size   0      this cur size   0      this cur buffer           this chunk size   chunk size    4096       this buffer list push new Buffer this chunk size        Buffer A1 prototype writeByteArrayLimited   function  data  offset  length        var can write   length  gt   this chunk size - this cur size     this chunk size - this cur size    length       var lastbuf   this buffer list length - 1       for  var x   0  x  lt  can write    x            this buffer list lastbuf  this cur size   x    data x   offset              this cur size    can write      this total size    can write       if  this cur size    this chunk size            this buffer list push new Buffer this chunk size            this cur size   0             return can write             The  data  parameter can be anything that is array like  It just must     support indexing and a length and produce an acceptable value to be     used with Buffer     Buffer A1 prototype writeByteArray   function  data  offset  length        offset   offset    undefined   0   offset      length   length    undefined   data length   length       var rem   length      while  rem  gt  0            rem -  this writeByteArrayLimited data  length - rem  rem             Buffer A1 prototype readByteArray   function  data  offset  length                   If you really wanted to implement some read functionality         then you would have to deal with unaligned reads which could         span two buffers             Buffer A1 prototype getSingleBuffer   function          var obuf   new Buffer this total size       var cur off   0      var x       for  x   0  x  lt  this buffer list length - 1    x            this buffer list x  copy obuf  cur off           cur off    this buffer list x  length             this buffer list x  copy obuf  cur off  0  this cur size        return obuf

User · Answer

insert byte to specific place   insertToArray arr index item       return Buffer concat  arr slice 0 index  Buffer from item  utf-8   arr slice index

User · Answer

Buffers are always of fixed size  there is no built in way to resize them dynamically  so your approach of copying it to a larger Buffer is the only way   However  to be more efficient  you could make the Buffer larger than the original contents  so it contains some  free  space where you can add data without reallocating the Buffer  That way you don t need to create a new Buffer and copy the contents on each append operation

[node.js] How to append binary data to a buffer in node.js

Updated Answer for Node.js ~>0.8

Old Answer for Node.js ~0.6

Examples related to node.js

Examples related to binary

Examples related to append

Examples related to buffer