Duplex seems to create unnecessary copies of Buffer data

The assertion below fails in iojs >= 3.0.0, and Node >= 4.0.0. It passes in every earlier version I checked (0.8.28 to 2.5.0).

var stream = require('readable-stream');

var duplex = new stream.Duplex({
    write: function(chunk, enc, cb) {
        this.push(chunk);
        cb();
    },
    read: function() {}
});

var bufA = new Buffer('a');

var writable = new stream.Writable({
    write: function (chunk, enc, cb) {
        assert.strictEqual(chunk, bufA, 'should be the same instance');
    }
});

duplex.pipe(writable);

duplex.write(bufA);

It can be made to pass by setting readableObjectMode: true, but it is unclear to me why that would be necessary (or the implications it would have).

@jamestalmage

@jamestalmage did you check if this is also the case for readable streams?

there is no guarantee of buffer equality for streams not in object mode, the buffers can be combined and they can be split. If it's just the possible perf issues then we should look to see where the copy is happening.

This does not happen with readable streams.

My main concern here is the performance impact.

objectMode:true seems undesirable since it seems it will have implications on buffering/backpressure.

there is no guarantee of buffer equality for streams not in object mode, the buffers can be combined and they can be split

Just for my understanding: Why is that ever necessary in the base streams implementation? It seems to me that the goal would be to never copy data unnecessarily, and I am having a hard time thinking of a reason why a base stream implementation would ever need to.

figured out the reason for this nodejs/node#3300

This does not happen with readable streams.

Oops! I obviously recalled incorrectly (probably got confused with my testing on node 12.X). Sorry. And thanks for the fix!

fixed and is in both node and readable-streams

Navigation Menu

Repository navigation

Duplex seems to create unnecessary copies of Buffer data #162

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Duplex seems to create unnecessary copies of Buffer data #162

Description

Activity

yoshuawuyts commented on Oct 9, 2015

calvinmetcalf commented on Oct 9, 2015

jamestalmage commented on Oct 9, 2015

calvinmetcalf commented on Oct 9, 2015

jamestalmage commented on Oct 9, 2015

calvinmetcalf commented on Nov 1, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions