Repository navigation
Accept ArrayBuffer (and typed array/data view?) anywhere Buffer is allowed in the API #1826
Description
Activity
I'm guessing that this would be within the confines of core and not 3rd party addons (at least such addons would need to explicitly support ArrayBuffers also)?
@domenic if you pass
new Buffer(new ArrayBuffer(5))would you expect the data to be copied or pointed to by the new buffer instance?Either way it can be done. This was actually on my list of things to implement, but I wanted to keep the initial PR small.
TBH it wouldn't be hard to support
ArrayBufferin most of the API. e.g.var buf = new Buffer(16); var ab = new ArrayBuffer(16); buf.copy(ab);
Third party support shouldn't be a problem. The native Buffer API still hands back the
void*that the ArrayBuffer points to. So anyone can operate on it as they usually have.@trevnorris to clarify I wasn't just talking about the buffer API; I was talking about the whole io.js API surface area. FS, streams, etc. Not sure if that came across in the OP... editing now.
if you pass new Buffer(new ArrayBuffer(5)) would you expect the data to be copied or pointed to by the new buffer instance?
Given how
new Uint8Array(new ArrayBuffer(5))behaves I'd expected pointed to, but I am more used to typed arrays than I am to buffers to be honest...This was actually on my list of things to implement, but I wanted to keep the initial PR small.
Yeah definitely, can leave this for a minor version probably.
A pointer is fine with me.
If we say that an ArrayBuffer is treated as binary data then this is feasible. Typed Arrays can be a little tricky because of user's expectations (e.g. is the "length" to be written the byte length or the index length). But it can definitely be investigated.
+1 for even talking ArrayBuffer / node.JS Buffer..
The native Buffer API still hands back the void* that the ArrayBuffer points to.
A pointer is fine with me.Let's say variable X = ArrayBuffer while the Y = Buffer (points to X's memory). As long as Y is alive, there is a need to keep X from GC.
A naive hope / question / discussion, (although it's a clear break in the API) why not just use ArrayBuffer and node.JS buffer as it's view (optional) ?
if you pass new Buffer(new ArrayBuffer(5)) would you expect the data to be copied or pointed to by the new buffer instance?
I would expect pointed to. All other argument types to the Buffer constructor get copied, but they're all array-like (have
lengthproperty and are indexable). ArrayBuffer is not array-like, so it's not a huge deal if we add a special case here. It's more useful to have it share memory, I think.This has been inactive for a while. I'm going to throw a
help wantedlabel on it, but feel free to remove it if, you know, help isn't actually wanted. (Why do I type these things?)This would be more easily done now that Buffer is a Uint8Array. In fact the native
Buffer::HasInstance()only checks if the object is a Uint8Array instance. Change would be tedious and require a lot of documentation updates. Especially since it may be confusing out of context reading an API takes a Uint8Array, which Buffer is, but not know that you can actually pass a Buffer.ArrayBuffer would require a bit more work, but also doable now.
It seems to me that the "right" way to fix this is to have everything internally operate on ArrayBuffers, then you have the equivalent of the following in all public binary-data accepting APIs:
function f(..., buffer) { if (ArrayBuffer.isView(buffer)) { buffer = buffer.buffer; // this will detect Buffers, Uint8Arrays, DataViews, etc. } // now you can process the ArrayBuffer buffer, maybe // after validating that it's a true ArrayBuffer. }
25 remaining items
- added a commit that references this issue
on Oct 17, 2018 - added a commit that references this issue
on Nov 2, 2018 - added a commit that references this issue
on Nov 2, 2018 - added a commit that references this issue
on Nov 2, 2018 I'm eager to help @TimothyGu. If possible, I would be incredibly thankful for any guidance/mentoring. I looked at the examples you provided and think I understand what needs to be done (maybe some initial guidance?). I apologize if I come off as a complete "noob", but I really enjoy Node, the community seems great, and I want to actually CONTRIBUTE!
- added 2 commits that reference this issue
on Apr 8, 2019 Not sure if it's exactly the same problem but in my case what I want is something that is exactly the same as
Buffer/Uint8Arrayin terms of API (either would work for me) but just allows an arbitrarily large size likeArrayBufferinstead of being capped likeBufferandUint8Array. Not sure if there is any plan to remove theUint8Arraycap but otherwise a good solution for a wrapper aroundArrayBufferwould be to allocate a series of internalUint8Arrayviews on construction and use those to set and grab values.E.g.
const UINT8_ARRAY_VIEW_SIZE = 1000 * 1000 * 500 // 500MB ? class UncappedUint8Array extends Uint8Array { static from(arr) { return arr.slice(); } constructor(arrayBuffer) { this._arrayBuffer = arrayBuffer; const { length } = this; this._views = Array(Math.ceil(length / UINT8_ARRAY_VIEW_SIZE)); let offset = 0; while (offset < length) { const size = Math.min(length - offset, UINT8_ARRAY_VIEW_SIZE); this._views.push(new Uint8Array(arrayBuffer, offset, size)); offset += size; } } get length() { return this._arrayBuffer.byteLength; } // could be used for index get _byteAt(index) { const viewSize = UINT8_ARRAY_VIEW_SIZE; return this._views[Math.floor(length / viewSize)][length % viewSize]; } // could be used for index set _setByteAt(index, value) { const viewSize = UINT8_ARRAY_VIEW_SIZE; this._views[Math.floor(length / viewSize)][length % viewSize] = value; } indexOf(value) { const { length } = this; for (let i = 0; i < length; i++) { if (this._byteAt(i) === value) { return i; } } return -1; } slice(startIndex, endIndex) { return new UncappedUint8Array(this._arrayBuffer.slice(startIndex, endIndex)); } // ... etc }
Not sure what amount of work would be required for me to implement the whole
Uint8ArrayorBufferinterfaces this way, if too complicated could be something to consider for core? I guess I would need a Proxy to handle indexes? But not sure if that would be sufficient for all the other methods.. would they defer to the index Proxy?I’ll close this issue because I think it’s (mostly?) done.
@benwiley4000 The fact that
ArrayBuffers are fixed-size is prescribed by the JS spec, it’s outside of Node.js’s control. This is something that probably would have to be taken up with TC39 first, if you want to see changes in the language or in Node.js.
That is, update fs, http, streams, etc.
This is a proposal, but I would think with @trevnorris's work on making Buffer a subclass of Uint8Array, it seems likely to be not too hard...
On the web the best practice has emerged that when accepting arguments, allow any of ArrayBuffer or typed array or DataView, which is why I think more than just Buffer | ArrayBuffer would make sense.
What do people think? @trevnorris, am I understanding correctly that after your work lands this would not be very hard?