Repository navigation
Add method to read file into user given Uint8Array #83
Description
Reading file contents into user supplied array avoids unnecessary copies. For example you can just get a pointer to an area in a WASM memory and directly read the file instead of first reading it into an useless ArrayBuffer and then doing a slow copy:
// file is a File or Blob
// wasm is a WebAssembly module instance wrapper with some additional methods like malloc added
const reader = new FileReader();
const ptr = wasm.malloc(file.size);
const array = new Uint8Array(wasm.memory.buffer, ptr, file.size);
try {
let bytesRead = await reader.readIntoUint8Array(file, array);
// Done
} catch (e) {
// Error
}It also allows reading files while using minimal memory and allocations:
const reader = new FileReader();
const bufferPtr = wasm.malloc(8192);
const buffer = new Uint8Array(wasm.memory.buffer, bufferPtr, 8192);
let totalBytesRead = 0;
while (totalBytesRead < file.size) {
const start = totalBytesRead;
const end = Math.min(start + 8192, file.size);
const bytesRead = await readIntoUint8Array(file.slice(start, end), buffer);
wasm.process(bufferPtr, bytesRead);
totalBytesRead += bytesRead;
}Doing the same with current APIs would allocate a ton of small ArrayBuffers...
Activity
mkruisselbrink commentedon Jun 19, 2017
This sounds very similar to what you would be able to do with a https://streams.spec.whatwg.org/#byob-reader-class right? I'm not necessarily against having similar functionality in FileReader as well, but I think eventually the streams API should be the one way to stream data from whatever source, so I'd be hesitant to add more functionality to FileReader. A somewhat sensible API to get a ReadableStream out of a Blob would seem like a much better solution to improving the ways you can read from blobs/files (note that you can already get a ReadableStream for a blob by wrapping the blob in a Response and calling its body method).
domenic commentedon Jun 19, 2017
See also #40
petkaantonov commentedon Jun 20, 2017
Readable stream with byob sounds kinda like what I'm looking for but since it's not a specific API I fear it will still internally perform copy.
thiccar commentedon Oct 12, 2017
Until BYOB reader is supported by more browsers, this would be nice to have for use with SharedArrayBuffer.
Fuzzyma commentedon Apr 6, 2019
For me it is not clear how byob-streams will solve this problem.
It is not possible to get a stream from a file isnt it? As far as I understand the streaming api (https://streams.spec.whatwg.org/) there is no plan on creating a stream from a file-input.
The FileReader still is the only option to create a buffer from the file-input. Even if you slice the file into chunks and stream it via a byob-stream, you still need to copy the contents of the FileReader-Buffer over to the buffer from the byob-stream.
It would be far easier, to just pass the byob-buffer to
readAsArrayBuffer(file, buf).The required changes for something like that should be very small!
Please enlighten me on this issue!
annevk commentedon Apr 8, 2019
@Fuzzyma #117 adds
blobOrFile.stream()(among some nice-to-haves).saschanaz commentedon May 14, 2023
Fixed by #188 I'd say.