When dealing with large streams, like a file sized over 4GB - you don't want to load the file into memory (as a Byte[]
) because not only is it very slow, but also may cause a crash as even in 64-bit processes a Byte[]
cannot exceed 2GB (or 4GB with gcAllowVeryLargeObjects
).
Fortunately there's a neat helper in .NET called ToBase64Transform
which processes a stream in chunks. For some reason Microsoft put it in System.Security.Cryptography
and it implements ICryptoTransform
(for use with CryptoStream
), but disregard that ("a rose by any other name...") just because you aren't performing any cryprographic tasks.
You use it with CryptoStream
like so:
using System.Security.Cryptography;
using System.IO;
//
using( FileStream inputFile = new FileStream( @"C:\VeryLargeFile.bin", FileMode.Open, FileAccess.Read, FileShare.None, bufferSize: 1024 * 1024, useAsync: true ) ) // When using `useAsync: true` you get better performance with buffers much larger than the default 4096 bytes.
using( CryptoStream base64Stream = new CryptoStream( inputFile, new ToBase64Transform(), CryptoStreamMode.Read ) )
using( FileStream outputFile = new FileStream( @"C:\VeryLargeBase64File.txt", FileMode.CreateNew, FileAccess.Write, FileShare.None, bufferSize: 1024 * 1024, useAsync: true ) )
{
await base64Stream.CopyToAsync( outputFile ).ConfigureAwait(false);
}