[node.js] How to append binary data to a buffer in node.js

I have a buffer with some binary data:

var b = new Buffer ([0x00, 0x01, 0x02]);

and I want to append 0x03.

How can I append more binary data? I'm searching in the documentation but for appending data it must be a string, if not, an error occurs (TypeError: Argument must be a string):

var b = new Buffer (256);
b.write ("hola");
console.log (b.toString ("utf8", 0, 4)); //hola
b.write (", adios", 4);
console.log (b.toString ("utf8", 0, 11)); //hola, adios

Then, the only solution I can see here is to create a new buffer for every appended binary data and copy it to the major buffer with the correct offset:

var b = new Buffer (4); //4 for having a nice printed buffer, but the size will be 16KB
new Buffer ([0x00, 0x01, 0x02]).copy (b);
console.log (b); //<Buffer 00 01 02 00>
new Buffer ([0x03]).copy (b, 3);
console.log (b); //<Buffer 00 01 02 03>

But this seems a bit inefficient because I have to instantiate a new buffer for every append.

Do you know a better way for appending binary data?

EDIT

I've written a BufferedWriter that writes bytes to a file using internal buffers. Same as BufferedReader but for writing.

A quick example:

//The BufferedWriter truncates the file because append == false
new BufferedWriter ("file")
    .on ("error", function (error){
        console.log (error);
    })

    //From the beginning of the file:
    .write ([0x00, 0x01, 0x02], 0, 3) //Writes 0x00, 0x01, 0x02
    .write (new Buffer ([0x03, 0x04]), 1, 1) //Writes 0x04
    .write (0x05) //Writes 0x05
    .close (); //Closes the writer. A flush is implicitly done.

//The BufferedWriter appends content to the end of the file because append == true
new BufferedWriter ("file", true)
    .on ("error", function (error){
        console.log (error);
    })

    //From the end of the file:
    .write (0xFF) //Writes 0xFF
    .close (); //Closes the writer. A flush is implicitly done.

//The file contains: 0x00, 0x01, 0x02, 0x04, 0x05, 0xFF

LAST UPDATE

Use concat.

This question is related to node.js binary append buffer

The answer is


Updated Answer for Node.js ~>0.8

Node is able to concatenate buffers on its own now.

var newBuffer = Buffer.concat([buffer1, buffer2]);

Old Answer for Node.js ~0.6

I use a module to add a .concat function, among others:

https://github.com/coolaj86/node-bufferjs

I know it isn't a "pure" solution, but it works very well for my purposes.


This is to help anyone who comes here looking for a solution that wants a pure approach. I would recommend understanding this problem because it can happen in lots of different places not just with a JS Buffer object. By understanding why the problem exists and how to solve it you will improve your ability to solve other problems in the future since this one is so fundamental.

For those of us that have to deal with these problems in other languages it is quite natural to devise a solution, but there are people who may not realize how to abstract away the complexities and implement a generally efficient dynamic buffer. The code below may have potential to be optimized further.

I have left the read method unimplemented to keep the example small in size.

The realloc function in C (or any language dealing with intrinsic allocations) does not guarantee that the allocation will be expanded in size with out moving the existing data - although sometimes it is possible. Therefore most applications when needing to store a unknown amount of data will use a method like below and not constantly reallocate, unless the reallocation is very infrequent. This is essentially how most file systems handle writing data to a file. The file system simply allocates another node and keeps all the nodes linked together, and when you read from it the complexity is abstracted away so that the file/buffer appears to be a single contiguous buffer.

For those of you who wish to understand the difficulty in just simply providing a high performance dynamic buffer you only need to view the code below, and also do some research on memory heap algorithms and how the memory heap works for programs.

Most languages will provide a fixed size buffer for performance reasons, and then provide another version that is dynamic in size. Some language systems opt for a third-party system where they keep the core functionality minimal (core distribution) and encourage developers to create libraries to solve additional or higher level problems. This is why you may question why a language does not provide some functionality. This small core functionality allows costs to be reduced in maintaining and enhancing the language, however you end up having to write your own implementations or depending on a third-party.

var Buffer_A1 = function (chunk_size) {
    this.buffer_list = [];
    this.total_size = 0;
    this.cur_size = 0;
    this.cur_buffer = [];
    this.chunk_size = chunk_size || 4096;

    this.buffer_list.push(new Buffer(this.chunk_size));
};

Buffer_A1.prototype.writeByteArrayLimited = function (data, offset, length) {
    var can_write = length > (this.chunk_size - this.cur_size) ? (this.chunk_size - this.cur_size) : length;

    var lastbuf = this.buffer_list.length - 1;

    for (var x = 0; x < can_write; ++x) {
        this.buffer_list[lastbuf][this.cur_size + x] = data[x + offset];
    }

    this.cur_size += can_write;
    this.total_size += can_write;

    if (this.cur_size == this.chunk_size) {
        this.buffer_list.push(new Buffer(this.chunk_size));
        this.cur_size = 0;
    }

    return can_write;
};

/*
    The `data` parameter can be anything that is array like. It just must
    support indexing and a length and produce an acceptable value to be
    used with Buffer.
*/
Buffer_A1.prototype.writeByteArray = function (data, offset, length) {
    offset = offset == undefined ? 0 : offset;
    length = length == undefined ? data.length : length;

    var rem = length;
    while (rem > 0) {
        rem -= this.writeByteArrayLimited(data, length - rem, rem);
    }
};

Buffer_A1.prototype.readByteArray = function (data, offset, length) {
    /*
        If you really wanted to implement some read functionality
        then you would have to deal with unaligned reads which could
        span two buffers.
    */
};

Buffer_A1.prototype.getSingleBuffer = function () {
    var obuf = new Buffer(this.total_size);
    var cur_off = 0;
    var x;

    for (x = 0; x < this.buffer_list.length - 1; ++x) {
        this.buffer_list[x].copy(obuf, cur_off);
        cur_off += this.buffer_list[x].length;
    }

    this.buffer_list[x].copy(obuf, cur_off, 0, this.cur_size);

    return obuf;
};

insert byte to specific place.

insertToArray(arr,index,item) {
   return Buffer.concat([arr.slice(0,index),Buffer.from(item,"utf-8"),arr.slice(index)]);
}

Buffers are always of fixed size, there is no built in way to resize them dynamically, so your approach of copying it to a larger Buffer is the only way.

However, to be more efficient, you could make the Buffer larger than the original contents, so it contains some "free" space where you can add data without reallocating the Buffer. That way you don't need to create a new Buffer and copy the contents on each append operation.


Examples related to node.js

Hide Signs that Meteor.js was Used Querying date field in MongoDB with Mongoose SyntaxError: Cannot use import statement outside a module Server Discovery And Monitoring engine is deprecated How to fix ReferenceError: primordials is not defined in node UnhandledPromiseRejectionWarning: This error originated either by throwing inside of an async function without a catch block dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.62.dylib error running php after installing node with brew on Mac internal/modules/cjs/loader.js:582 throw err DeprecationWarning: Buffer() is deprecated due to security and usability issues when I move my script to another server Please run `npm cache clean`

Examples related to binary

Difference between opening a file in binary vs text Remove 'b' character do in front of a string literal in Python 3 Save and retrieve image (binary) from SQL Server using Entity Framework 6 bad operand types for binary operator "&" java C++ - Decimal to binary converting Converting binary to decimal integer output How to convert string to binary? How to convert 'binary string' to normal string in Python3? Read and write to binary files in C? Convert to binary and keep leading zeros in Python

Examples related to append

List append() in for loop ValueError: all the input arrays must have same number of dimensions Append a tuple to a list - what's the difference between two ways? How merge two objects array in angularjs? How to add an element at the end of an array? Appending a list or series to a pandas DataFrame as a row? Can someone explain how to append an element to an array in C programming? How to append elements at the end of ArrayList in Java? Append value to empty vector in R? How to append new data onto a new line

Examples related to buffer

Convert a JSON Object to Buffer and Buffer to JSON Object back C char array initialization Save Screen (program) output to a file Flushing buffers in C Char array to hex string C++ How to append binary data to a buffer in node.js Convert a binary NodeJS Buffer to JavaScript ArrayBuffer How to clear input buffer in C? How to find the socket buffer size of linux Ring Buffer in Java