Convert a binary NodeJS Buffer to JavaScript ArrayBuffer

Question

How can I convert a NodeJS binary buffer into a JavaScript ArrayBuffer

User · Answer

A quicker way to write it  var arrayBuffer   new Uint8Array nodeBuffer  buffer    However  this appears to run roughly 4 times slower than the suggested toArrayBuffer function on a buffer with 1024 elements

User · Answer

1  A Buffer is just a view for looking into an ArrayBuffer  A Buffer  in fact  is a FastBuffer  which extends  inherits from  Uint8Array  which is an octet-unit view     partial accessor     of the actual memory  an ArrayBuffer       lib buffer js L65-L73 Node js 9 4 0 class FastBuffer extends Uint8Array     constructor arg1  arg2  arg3        super arg1  arg2  arg3         FastBuffer prototype constructor   Buffer  internalBuffer FastBuffer   FastBuffer   Buffer prototype   FastBuffer prototype    2  The size of an ArrayBuffer and the size of its view may vary  Reason  1  Buffer from arrayBuffer   byteOffset   length     With Buffer from arrayBuffer   byteOffset   length     you can create a Buffer with specifying its underlying ArrayBuffer and the view s position and size  const test buffer   Buffer from new ArrayBuffer 50   40  10   console info test buffer buffer byteLength      50  the size of the memory  console info test buffer length      10  the size of the view   Reason  2  FastBuffer s memory allocation  It allocates the memory in two different ways depending on the size   If the size is less than the half of the size of a memory pool and is not 0     small      it makes use of a memory pool to prepare the required memory  Else  it creates a dedicated ArrayBuffer that exactly fits the required memory        lib buffer js L306-L320 Node js 9 4 0 function allocate size      if  size  lt   0        return new FastBuffer          if  size  lt   Buffer poolSize  gt  gt  gt  1         if  size  gt   poolSize - poolOffset         createPool        var b   new FastBuffer allocPool  poolOffset  size       poolOffset    size      alignPool        return b      else       return createUnsafeBuffer size               lib buffer js L98-L100 Node js 9 4 0 function createUnsafeBuffer size      return new FastBuffer createUnsafeArrayBuffer size        What do you mean by a    memory pool     A memory pool is a fixed-size pre-allocated memory block for keeping small-size memory chunks for Buffers  Using it keeps the small-size memory chunks tightly together  so prevents fragmentation caused by separate management  allocation and deallocation  of small-size memory chunks  In this case  the memory pools are ArrayBuffers whose size is 8 KiB by default  which is specified in Buffer poolSize  When it is to provide a small-size memory chunk for a Buffer  it checks if the last memory pool has enough available memory to handle this  if so  it creates a Buffer that    views    the given partial chunk of the memory pool  otherwise  it creates a new memory pool and so on   You can access the underlying ArrayBuffer of a Buffer  The Buffer s buffer property  that is  inherited from Uint8Array  holds it  A    small    Buffer s buffer property is an ArrayBuffer that represents the entire memory pool  So in this case  the ArrayBuffer and the Buffer varies in size  const zero sized buffer   Buffer allocUnsafe 0   const small buffer   Buffer from  0xC0  0xFF  0xEE    const big buffer   Buffer allocUnsafe Buffer poolSize  gt  gt  gt  1       A  Buffer  s  length  property holds the size  in octets  of the view     An  ArrayBuffer  s  byteLength  property holds the size  in octets  of its data   console info zero sized buffer length       0  the view s size  console info zero sized buffer buffer byteLength       0  the memory   s size  console info Buffer poolSize       8192  a memory pool s size   console info small buffer length       3  the view s size  console info small buffer buffer byteLength       8192  the memory pool s size  console info Buffer poolSize       8192  a memory pool s size   console info big buffer length       4096  the view s size  console info big buffer buffer byteLength       4096  the memory s size  console info Buffer poolSize       8192  a memory pool s size    3  So we need to extract the memory it    views     An ArrayBuffer is fixed in size  so we need to extract it out by making a copy of the part  To do this  we use Buffer s byteOffset property and length property  which are inherited from Uint8Array  and the ArrayBuffer prototype slice method  which makes a copy of a part of an ArrayBuffer  The slice  -ing method herein was inspired by  ZachB  const test buffer   Buffer from new ArrayBuffer 10    const zero sized buffer   Buffer allocUnsafe 0   const small buffer   Buffer from  0xC0  0xFF  0xEE    const big buffer   Buffer allocUnsafe Buffer poolSize  gt  gt  gt  1    function extract arraybuffer buf           You may use the  byteLength  property instead of the  length  one      return buf buffer slice buf byteOffset  buf byteOffset   buf length         A copy - const test arraybuffer   extract arraybuffer test buffer      of the memory  const zero sized arraybuffer   extract arraybuffer zero sized buffer      of the    void  const small arraybuffer   extract arraybuffer small buffer      of the part of the memory  const big arraybuffer   extract arraybuffer big buffer      of the memory   console info test arraybuffer byteLength      10 console info zero sized arraybuffer byteLength      0 console info small arraybuffer byteLength      3 console info big arraybuffer byteLength      4096   4  Performance improvement If you re to use the results as read-only  or it is okay to modify the input Buffers  contents  you can avoid unnecessary memory copying  const test buffer   Buffer from new ArrayBuffer 10    const zero sized buffer   Buffer allocUnsafe 0   const small buffer   Buffer from  0xC0  0xFF  0xEE    const big buffer   Buffer allocUnsafe Buffer poolSize  gt  gt  gt  1    function obtain arraybuffer buf        if buf length     buf buffer byteLength                return buf buffer           else         You may use the  byteLength  property instead of the  length  one      return buf subarray 0  buf length         Its underlying  ArrayBuffer   const test arraybuffer   obtain arraybuffer test buffer      Just a zero-sized  ArrayBuffer   const zero sized arraybuffer   obtain arraybuffer zero sized buffer      A copy of the part of the memory  const small arraybuffer   obtain arraybuffer small buffer      Its underlying  ArrayBuffer   const big arraybuffer   obtain arraybuffer big buffer    console info test arraybuffer byteLength      10 console info zero sized arraybuffer byteLength      0 console info small arraybuffer byteLength      3 console info big arraybuffer byteLength      4096

User · Answer

Now there is a very useful npm package for this  buffer  https   github com feross buffer  It tries to provide an API that is 100  identical to node s Buffer API and allow    convert typed array to buffer  https   github com feross buffer convert-typed-array-to-buffer convert buffer to typed array  https   github com feross buffer convert-buffer-to-typed-array   and few more

User · Answer

Use the following excellent npm package  to-arraybuffer   Or  you can implement it yourself  If your buffer is called buf  do this   buf buffer slice buf byteOffset  buf byteOffset   buf byteLength

User · Answer

NodeJS  at one point  I think it was v0 6 x  had ArrayBuffer support  I created a small library for base64 encoding and decoding here  but since updating to v0 7  the tests  on NodeJS  fail  I m thinking of creating something that normalizes this  but till then  I suppose Node s native Buffer should be used

User · Answer

This Proxy will expose the buffer as any of the TypedArrays  without any copy      https   www npmjs com package node-buffer-as-typedarray  It only works on LE  but can be easily ported to BE  Also  never got to actually test how efficient this is

User · Answer

You can think of an ArrayBuffer as a typed Buffer    An ArrayBuffer therefore always needs a type  the so-called  Array Buffer View    Typically  the Array Buffer View has a type of Uint8Array or Uint16Array   There is a good article from Renato Mangini on converting between an ArrayBuffer and a String   I have summarized the essential parts in a code example  for Node js   It also shows how to convert between the typed ArrayBuffer and the untyped Buffer   function stringToArrayBuffer string      const arrayBuffer   new ArrayBuffer string length     const arrayBufferView   new Uint8Array arrayBuffer     for  let i   0  i  lt  string length  i          arrayBufferView i    string charCodeAt i         return arrayBuffer     function arrayBufferToString buffer      return String fromCharCode apply null  new Uint8Array buffer       const helloWorld   stringToArrayBuffer  Hello  World         ArrayBuffer   Uint8Array  const encodedString   new Buffer helloWorld  toString  base64        string  const decodedBuffer   Buffer from encodedString   base64        Buffer  const decodedArrayBuffer   new Uint8Array decodedBuffer  buffer      ArrayBuffer   Uint8Array   console log arrayBufferToString decodedArrayBuffer       prints  Hello  World

User · Answer

Instances of Buffer are also instances of Uint8Array in node js 4 x and higher   Thus  the most efficient solution is to access the buf buffer property directly  as per https   stackoverflow com a 31394257 1375574   The Buffer constructor also takes an ArrayBufferView argument if you need to go the other direction   Note that this will not create a copy  which means that writes to any ArrayBufferView will write through to the original Buffer instance    In older versions  node js has both ArrayBuffer as part of v8  but the Buffer class provides a more flexible API   In order to read or write to an ArrayBuffer  you only need to create a view and copy across   From Buffer to ArrayBuffer   function toArrayBuffer buf        var ab   new ArrayBuffer buf length       var view   new Uint8Array ab       for  var i   0  i  lt  buf length    i            view i    buf i             return ab      From ArrayBuffer to Buffer   function toBuffer ab        var buf   Buffer alloc ab byteLength       var view   new Uint8Array ab       for  var i   0  i  lt  buf length    i            buf i    view i             return buf

User · Answer

From ArrayBuffer to Buffer  could be done this way    var buffer   Buffer from  new Uint8Array ab

User · Answer

I have already update my node to Version 5 0 0 And I work work with this   function toArrayBuffer buffer       var array           var json   buffer toJSON        var list   json data      for var key in list           array push fixcode list key  toString 16               function fixcode key           if key length  1               return  0  key toUpperCase            else              return key toUpperCase                        return array     I use it to check my vhd disk image

User · Answer

No dependencies  fastest  Node js 4 x and later  Buffers are Uint8Arrays  so you just need to slice  copy  its region of the backing ArrayBuffer      Original Buffer let b   Buffer alloc 512      Slice  copy  its segment of the underlying ArrayBuffer let ab   b buffer slice b byteOffset  b byteOffset   b byteLength     The slice and offset stuff is required because small Buffers  less than 4 kB by default  half the pool size  can be views on a shared ArrayBuffer  Without slicing  you can end up with an ArrayBuffer containing data from another Buffer  See explanation in the docs   If you ultimately need a TypedArray  you can create one without copying the data      Create a new view of the ArrayBuffer without copying let ui32   new Uint32Array b buffer  b byteOffset  b byteLength   Uint32Array BYTES PER ELEMENT       No dependencies  moderate speed  any version of Node js  Use Martin Thomson s answer  which runs in O n  time   See also my replies to comments on his answer about non-optimizations  Using a DataView is slow  Even if you need to flip bytes  there are faster ways to do so    Dependency  fast  Node js   0 12 or iojs 3 x  You can use https   www npmjs com package memcpy to go in either direction  Buffer to ArrayBuffer and back   It s faster than the other answers posted here and is a well-written library  Node 0 12 through iojs 3 x require ngossen s fork  see this

User · Answer

I tried the above for a Float64Array and it just did not work   I ended up realising that really the data needed to be read  INTO  the view in correct chunks  This means reading 8 bytes at a time from the source Buffer   Anyway this is what I ended up with     var buff   new Buffer  40100000000000004014000000000000    hex    var ab   new ArrayBuffer buff length   var view   new Float64Array ab    var viewIndex   0  for  var bufferIndex 0 bufferIndex lt buff length bufferIndex bufferIndex 8                    view viewIndex    buff readDoubleLE bufferIndex       viewIndex

[javascript] Convert a binary NodeJS Buffer to JavaScript ArrayBuffer

Examples related to javascript

Examples related to node.js

Examples related to binary

Examples related to buffer

Examples related to arraybuffer