if Buffer#toString not consume all bytes? #6

Activity

1. mcollina commented on Jan 19, 2016

   

   mcollina

   on Jan 19, 2016

   Owner

   More actions

   It works fine for me.

   I've also checked:

   var split = require("split2");
   var buf = new Buffer("aa\r\nbbb", "utf8");
   var s = split();
   s.on("data", function (data) {
       console.log(data);
   });
   
   for (var i = 0; i < buf.length; ++i)
   {
       s.write(buf.slice(i, i + 1));
   }
   s.end();

   Which version of node are you using?
2. heroboy commented on Jan 19, 2016

   

   heroboy

   on Jan 19, 2016

   Author

   More actions

   no, you should not use string like "aaabbb".
   You should use strings that string.length not equals to buffer.length .
   For example, "烫" buffer in utf8 is [0xe7,0x83,0xab], in above code it will get three chunks in transform function,none of the three chunks can be converted to "烫".

   (new Buffer([0xe7,0x83,0xab])).toString("utf8") == "烫";
   (new Buffer([0xe7])).toString("utf8") == "";
   (new Buffer([0x83])).toString("utf8") == "";
   (new Buffer([0xab])).toString("utf8") == "";
   
3. mcollina commented on Jan 19, 2016

   

   mcollina

   on Jan 19, 2016

   Owner

   More actions

   I'm lost. I think the best way is for you to send a PR with a valid unit test and possibly a fix.

   It is working for me as far as I know: https://github.com/mcollina/split2/blob/master/test.js#L211-L221
4. heroboy commented on Jan 19, 2016

   

   heroboy

   on Jan 19, 2016

   Author

   More actions

   change

   input.end("烫烫烫\r\n锟斤拷")
   

   to

   var buf = new Buffer("烫烫烫\r\n锟斤拷", "utf8");
   for (var i = 0; i < buf.length; ++i)
   {
       input.write(buf.slice(i, i + 1));
   }
   input.end();
   
5. mcollina commented on Jan 19, 2016

   

   mcollina

   on Jan 19, 2016

   Owner

   More actions

   Confirmed, I'll see how it is fixable.
6. heroboy commented on Jan 19, 2016

   

   heroboy

   on Jan 19, 2016

   Author

   More actions

   try this

   through({encoding:"utf8"})                 //convert Buffer to String (buf => data)
       .pipe(through({decodeStrings:false},   //prevent convert String to Buffer (buf => chunk)
           function(chunk,enc,cb){}           //your transform function,now chunk is string.
   );
   

   And please expose the encoding in options.
7. mcollina

   closed this as completedin #7on Jan 20, 2016