StringDecoder#end doesn't flush state

v8.6.0
Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64
StringDecoder

StringDecoder#end doesn't flush the state, allowing it to accept the completing byte afterwards:

const { StringDecoder } = require('string_decoder');
const decoder = new StringDecoder('utf8');

decoder.write(Buffer.from([0xE2, 0x82])); // => ''
// We're waiting for the final byte (currently at byte 2 of a 3 byte sequence)
// Now, end the buffer because, eg, the client disconnected.
decoder.end(); // => '�' (UTF-8 Replacement Character)

Now, there's our setup. If we've ended the buffer and it has outputted a replacement character, it kind of implies it's back at its initial state. But:

decoder.write(Buffer.of(0x61)); // => '�a'

Or, we could "finish" that 3 byte sequence from above (if we haven't already written the 'a'):

decoder.write(Buffer.of(0xAC)); // => '€'

@jridgewell

@jridgewell Do you have any suggestions on what to do here? Should .write() after .end() throw, like it does for streams?

TextDecoder resets its state after the equivalent of an end call, and functions as a new TextDecoder instance basically. That sounds fair to me.

Make sure to check benchmarks on this one because of how widely string_decoder is used.

I think throwing is better than the current behavior, but I’d prefer it just reset and behave like a brand new instance.

Proposed a fix in #16594. I chose to reset silently instead of throwing.

string_decoder: reset decoder on end

string_decoder: make write after end to reset

string_decoder: reset decoder on end

Navigation Menu

Repository navigation

StringDecoder#end doesn't flush state #16564

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

StringDecoder#end doesn't flush state #16564

Description

Activity

addaleax commented on Oct 27, 2017

TimothyGu commented on Oct 27, 2017

mscdex commented on Oct 27, 2017

jridgewell commented on Oct 28, 2017

thefourtheye commented on Oct 29, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions