Repository navigation
fs.openAsBlob() does not work properly for files > 2GB #52585
Description
Activity
- changed the title
[-]`fs.openAsBlob` does not work properly for files > 2GB[/-][+]`fs.openAsBlob()` does not work properly for files > 2GB[/+]on Apr 18, 2024 FWIW, this also reproduces on a
linux/arm64container image running in Docker for Mac.root@f7a53c9fe57c:/# uname -a Linux f7a53c9fe57c 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux root@f7a53c9fe57c:/# node -v v20.12.2 root@f7a53c9fe57c:/# dd if=/dev/random of=random.bin bs=1048576 count=3072 3072+0 records in 3072+0 records out 3221225472 bytes (3.2 GB, 3.0 GiB) copied, 8.45959 s, 381 MB/s root@f7a53c9fe57c:/# node Welcome to Node.js v20.12.2. Type ".help" for more information. > const b = await fs.openAsBlob("random.bin") undefined > b Blob { size: 3221225472, type: '' } > b.slice(2**31-1, 2**31) Blob { size: 0, type: '' } > b.slice(2**31, 2**31+1) # node[15]: static void node::Blob::ToSlice(const v8::FunctionCallbackInfo<v8::Value>&) at ../src/node_blob.cc:247 # Assertion failed: args[0]->IsUint32() ----- Native stack trace ----- 1: 0xcab09c node::Assert(node::AssertionInfo const&) [node] 2: 0xc65b38 node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) [node] 3: 0xf1f228 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [node] 4: 0xf1f9e8 [node] 5: 0xf1fe00 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node] 6: 0x189c964 [node] ----- JavaScript stack trace ----- 1: slice (node:internal/blob:266:21) 2: REPL4:1:3 3: runInThisContext (node:vm:136:12) 4: defaultEval (node:repl:598:22) 5: bound (node:domain:432:15) 6: runBound (node:domain:443:12) 7: onLine (node:repl:927:10) 8: emit (node:events:530:35) 9: emit (node:domain:488:12) 10: [_onLine] (node:internal/readline/interface:416:12) Aborted
Reacted by Johan WHi @joelrbrandt , In the implementation of
Blob.slice, it's hinted that thestartandendshould be within theuint32range,
Since it is never documented and actually there allowsBlobhave index out of rangeuint32, should it be documented to make it clear or validate the range of index instead of using such asstart | 0andend | 0? /cc @nodejs/bufferReacted by Ali Hassan@kylo5aby thanks for your quick response (and sorry for my slow follow-up).
it's hinted that the start and end should be within the uint32 range
I'm a bit confused by this part of your response. Where is this hinted? In the node implementation?
I don't see any mention of a 32-bit limitation in the File API spec. So, it doesn't seem like the
Blob.prototype.slice()API should be limited to the uint32 range. Also, if it were limited to the uint32 range, it should go up to2**32, not2*31. But, the.slice()API supports negative values (distance from end), so maybe you meant int32?Independent of the issue with
.slice(), node reports the wrong size for Blobs that are larger than 4GB.Interestingly, if I get a stream of the Blob (with
Blob.prototype.stream()), I can read all the bytes in Blobs larger than 4GB.See below where I do the following:
- Generate 5GB of random data
- Hash that data with openssl
- Launch node v20.12.2
- Open that random data in node with
fs.openAsBlob - Observe the incorrectly reported size of 1GB (instead of 5GB)
- Construct a stream, and send all the bytes to a hash using async iteration of the stream
- Digest the hash to exactly the same value returned by openssl
$ dd if=/dev/random of=random.bin bs=1048576 count=5120 5120+0 records in 5120+0 records out 5368709120 bytes transferred in 6.613665 secs (811760063 bytes/sec) $ openssl dgst random.bin SHA256(random.bin)= f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c $ node Welcome to Node.js v20.12.2. Type ".help" for more information. > b = await fs.openAsBlob("random.bin") Blob { size: 1073741824, type: '' } > b.size 1073741824 > const { createHash } = require("node:crypto") undefined > const h = createHash("sha256") undefined > const s = b.stream() undefined > for await (const a of s) { ... h.update(a); ... } undefined > h.digest("hex") 'f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c'
Finally, here's a codepen that can be used to check that files > 4GB work fine in all major modern browsers (open console to see output): https://codepen.io/joelrbrandt/pen/wvZRXvY
Finally, if
fs.openAsBlob()is going to have a 4GB limit (andBlob.prototype.slice()is going to have a 2GB limit), it would certainly be more ideal if they threwRangeErrorexceptions rather than crashing node on out-of-range values.Buffer.alloc()does this:> b = Buffer.alloc(2**33) Uncaught: RangeError [ERR_OUT_OF_RANGE]: The value of "size" is out of range. It must be >= 0 && <= 4294967296. Received 8_589_934_592 at Function.alloc (node:buffer:389:3) { code: 'ERR_OUT_OF_RANGE' }
Hi @joelrbrandt, Thank you for the information.
- For
fs.openAsBlob(), if it read a file with size larger than 4GB, the generated blob has a wrongsizeas you mentioned:
Observe the incorrectly reported size of 1GB (instead of 5GB)
There maybe a bug in
fs.openAsBlob(), it wrongly set thesize. do you think it works as expected? @jasnell . AFAIK，new Blob()can correctly set thesize.
2. the uint32 range is forBlob.sliceinstead ofBlob, it allows negative index as you mentioned, and will converted to positive under the hood and check whether it under the uin32 range. I have a PR #52588 to try to solve if there is a largestartorend(larger than 2^31)b.slice(231-1, 231)
Blob { size: 0, type: '' }
b.slice(231, 231+1)
node[15]: static void node::Blob::ToSlice(const v8::FunctionCallbackInfov8::Value&) at ../src/node_blob.cc:247
Assertion failed: args[0]->IsUint32()- For
AFAIK,
new Blob()can correctly set thesizeI'm not sure if this is accurate, either. Trying to construct a new Blob from a set of ArrayBuffers that total a size greater than 4GB also fails:
$ node Welcome to Node.js v18.13.0. Type ".help" for more information. > const arrs = [] undefined > for (let i = 0; i < 5 ; i++) { ... arrs.push(new Uint8Array(2**30)) ... } 5 > arrs.reduce((acc, cur) => acc + cur.byteLength, 0) 5368709120 > const b = new Blob(arrs) Uncaught: RangeError [ERR_BUFFER_TOO_LARGE]: Cannot create a Buffer larger than 4294967295 bytes at __node_internal_captureLargerStackTrace (node:internal/errors:491:5) at new NodeError (node:internal/errors:400:5) at new Blob (node:internal/blob:165:13) { code: 'ERR_BUFFER_TOO_LARGE' }
However, this code also works fine in a browser, resulting in a 5GB
Blob.You are right. In versions up to v21, on 64-bit systems, the maximum size of a Buffer was 4GB. However, in the version I last used, compiled from the main branch, this limit has been exceeded, that's why I said
new Blobis ok to create a blob larger than 4GB.
Conversely,fs.openAsBlobcan successfully create blobs larger than 4GB, but there might be issues with the size.- added a commit that references this issue
on May 6, 2024 - added a commit that references this issue
on May 8, 2024 - added a commit that references this issue
on Jun 17, 2024 - added a commit that references this issue
on Jun 20, 2024 It appears to persist this issue for files >4GB in latest Node.js v23.3.0 and v20.18.1:
$ dd if=/dev/random of=random.bin bs=1048576 count=5120 $ ls -l random.bin -rw-r--r-- 1 user user 5368709120 Dec 10 23:39 random.bin $ node Welcome to Node.js v23.3.0. Type ".help" for more information. > b = await fs.openAsBlob("random.bin") Blob { size: 1073741824, type: '' } > b.slice(2**31-1, 2**31) Blob { size: 0, type: '' }
Also:
$ node Welcome to Node.js v23.3.0. Type ".help" for more information. > arrs = []; for (let i = 0; i < 5; i++) { arrs.push(new Uint8Array(2**30)); } 5 > b = new Blob(arrs) Blob { size: 5368709120, type: '' } > b.slice(2**32-2, 2**32-1) // 4GB-2, 4GB-1 Blob { size: 1, type: '' } > b.slice(2**32-1, 2**32-0) // 4GB-1, 4GB-0 # node[91790]: static void node::Blob::ToSlice(const FunctionCallbackInfo<v8::Value> &) at ../src/node_blob.cc:264 # Assertion failed: args[1]->IsUint32() ----- Native stack trace ----- 1: 0x1026a2ff0 node::Assert(node::AssertionInfo const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 2: 0x10449da7c node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) (.cold.1) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 3: 0x10266c86c node::Blob::HasInstance(node::Environment*, v8::Local<v8::Value>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 4: 0x103452978 Builtins_CallApiCallbackGeneric [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 5: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 6: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 7: 0x10344e50c Builtins_JSEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 8: 0x10344e1b0 Builtins_JSEntry [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 9: 0x102a28760 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 10: 0x102a28d48 v8::internal::Execution::CallScript(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 11: 0x1028d3268 v8::Script::Run(v8::Local<v8::Context>, v8::Local<v8::Data>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 12: 0x102693098 node::contextify::ContextifyScript::EvalMachine(v8::Local<v8::Context>, node::Environment*, long long, bool, bool, bool, v8::MicrotaskQueue*, v8::FunctionCallbackInfo<v8::Value> const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 13: 0x1026927ac node::contextify::ContextifyScript::RunInContext(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 14: 0x103452978 Builtins_CallApiCallbackGeneric [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 15: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 16: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 17: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 18: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 19: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 20: 0x10a006cf4 21: 0x10a04082c 22: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 23: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 24: 0x10a033c4c 25: 0x10a03db94 26: 0x10a038ffc 27: 0x10a006ba0 28: 0x10a04082c 29: 0x10a038310 30: 0x1034902c4 Builtins_GeneratorPrototypeNext [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 31: 0x10a01d0a0 32: 0x10a006ba0 33: 0x10a04082c 34: 0x10a03a574 35: 0x10a039df8 36: 0x10a03be0c 37: 0x10a0417e4 38: 0x10a00adf0 39: 0x10344e50c Builtins_JSEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 40: 0x10344e1b0 Builtins_JSEntry [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 41: 0x102a28760 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 42: 0x102a280a8 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 43: 0x1028e4d64 v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 44: 0x1025c1c88 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 45: 0x1025d6b34 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 46: 0x1027aedbc node::StreamBase::CallJSOnreadMethod(long, v8::Local<v8::ArrayBuffer>, unsigned long, node::StreamBase::StreamBaseJSChecks) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 47: 0x1027b0fc0 node::EmitToJSStreamListener::OnStreamRead(long, uv_buf_t const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 48: 0x1027b55e0 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 49: 0x1027b5ce8 node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_stream_s*, long, uv_buf_t const*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 50: 0x10343a180 uv__stream_io [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 51: 0x103442384 uv__io_poll [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 52: 0x10342ef20 uv_run [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 53: 0x1025c24e4 node::SpinEventLoopInternal(node::Environment*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 54: 0x1026eb3b0 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 55: 0x1026eb14c node::NodeMainInstance::Run() [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 56: 0x1026615ec node::Start(int, char**) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node] 57: 0x18d294274 start [/usr/lib/dyld] ----- JavaScript stack trace ----- 1: slice (node:internal/blob:271:21) 2: REPL8:1:3 3: runInThisContext (node:vm:137:12) 4: defaultEval (node:repl:597:22) 5: bound (node:domain:433:15) 6: runBound (node:domain:444:12) 7: onLine (node:repl:926:10) 8: emit (node:events:525:35) 9: emit (node:domain:489:12) 10: [_onLine] (node:internal/readline/interface:416:12) fish: Job 1, 'node' terminated by signal SIGABRT (Abort)
Tested on
linux/amd64anddarwin/arm64.Reacted by Johan W and WU HUWhy hasn't this issue been resolved for several years? Is it really that difficult? Both deno and bunjs are correct!
Bun handles this by using a custom
u52type (Number.MAX_SAFE_INTEGER). Deno implementsBlobs in a WebIDL extension (deno_webidlconverts betweenunsigned long longandNumber).The
Blob.sizeissue is because of the usage ofV8::Uint32inBlobFromFilePathLines 123 to 126 in 9c4ca0a
Blob::ToSliceis also restricted toV8::Uint32Lines 272 to 283 in 9c4ca0a
The existing
CheckNumberToSizeshould be able to convert aV8::Numberto asize_t:Lines 1640 to 1654 in 6a3d80f
This is my best guess on how to convert a
size_tto aV8::Number:size_t val = 1; Number v8num = Number::New(env->isolate(), static_cast<double>(val));
This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.Still persist in Node.js v26.5.0. Same result as #52585 (comment)
- added a commit that references this issue
on Aug 17, 2026
Version
v21.7.3 (and v20.12.2)
Platform
Darwin bender.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64
Subsystem
fs
What steps will reproduce the bug?
Steps to reproduce bug:
dd if=/dev/random of=random.bin bs=1048576 count=3072const b = await fs.openAsBlob("random.bin")b.slice(2**31-2, 2**31-1)<-- worksb.slice(2**31-1, 2**31)<-- returns a slice of zero lengthb.slice(2**31, 2**31+1)<-- crashesFull textual output of doing this given below.
Also, for files greater than 4GB, the size of the blob is wrong. It appears to be exactly
4*(2**30)less than the actual size. For example, for a5*(2**30)byte file, the blob size is reported as1073741824, which is(2**30). And, for a file that is exactly4*(2**30), it reports a size of zero.How often does it reproduce? Is there a required condition?
Always reproduces.
What is the expected behavior? Why is that the expected behavior?
Can operate on the Blob normally.
What do you see instead?
Described above. See output below.
Additional information