Repository navigation
new Buffer doesn't handle unicode well #2344
Description
var arr = [255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1]
//length 12;
var buf1 = new Buffer(arr)
// length 12
var hex1 = buf1.toString('hex');
// length 24
var str = buf1.toString('utf8')
// length 12
var buf2 = new Buffer(str);
//length 20
var hex2 = buf2.toString('hex');
// length 40;results for hex2
browserify - efbfbd00104a4649460001
node - efbfbdefbfbdefbfbdefbfbd00104a4649460001
the value of str in node is ����\u0000\u0010JFIF\u0000\u0001 which would suggest that the issue has to do with how new Buffer handles characters in text.
Activity
bnoordhuis commentedon Aug 10, 2015
255 is never valid in a UTF-8 byte sequence. It probably worked by accident but invalid UTF-8 is replaced with U+FFFD (the replacement character) as of node.js v0.10.29. The V8 UTF-8 decoder became stricter in v3.x so that might be a factor as well.
I don't think there is anything to do here except make a note of it somewhere in the documentation..
bnoordhuis commentedon Aug 10, 2015
That's
0xEF 0xBF 0xBDas UTF-8, by the way; the repeated pattern you see in the test's output.[-]new Buffer doens't handle unicode well[/-][+]new Buffer doesn't handle unicode well[/+]calvinmetcalf commentedon Aug 11, 2015
fixed on the browserify end