fs.ReadFile throws asynchronously with encoding on large enough input

Given 'utf8' encoding is specified, and a large enough input file, fs.ReadFile will throw an asynchronous error.

Example program

var fs = require('fs');
fs.readFile('data.txt', 'utf8', function(err, data) {
  console.log(err, data && data.length)
})

Input: 268 megabytes of data

Success.

> dd if=/dev/zero of=data.txt  bs=1000000  count=268
> node -e "fs = require('fs'); fs.readFile('data.txt', 'utf8', (err, data) => console.log(err, data && data.length))"
null 268000000

Input: 269 megabytes of data

No Success.
Worse, the error isn't forwarded to the callback – the process is throwing asynchronously.

> dd if=/dev/zero of=data.txt  bs=1000000  count=269
> node -e "fs = require('fs'); fs.readFile('data.txt', 'utf8', (err, data) => console.log(err, data && data.length))"
buffer.js:359
    throw new Error('toString failed');
    ^

Error: toString failed
    at Buffer.toString (buffer.js:359:11)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:378:21)

The error is not something specific to fs.readFile though, the same error is produced if we toString on the buffer directly, without the 'utf8' parameter, unsurprisingly. The problem is the uncatchable throw.

> node -e "fs = require('fs'); fs.readFile('data.txt', (err, data) => console.log(err, String(data).length))"

While perhaps the throw in the Buffer.prototype.toString makes sense:

Buffer.prototype.toString = function() {
  if (arguments.length === 0) {
    var result = this.utf8Slice(0, this.length);
  } else {
    var result = slowToString.apply(this, arguments);
  }
  if (result === undefined)
    throw new Error('toString failed');
  return result;
};

node/lib/buffer.js

Lines 352 to 361 in f8152df

    
           Buffer.prototype.toString = function() { 
        
             if (arguments.length === 0) { 
        
               var result = this.utf8Slice(0, this.length); 
        
             } else { 
        
               var result = slowToString.apply(this, arguments); 
        
             } 
        
             if (result === undefined) 
        
               throw new Error('toString failed'); 
        
             return result; 
        
           };

It does seem like poor behaviour to have uncatchable errors being thrown in core APIs where the user has explicitly attached an errback.

Would a try…catch around the buffer = buffer.toString(context.encoding); in fs.ReadFile be appropriate?

function readFileAfterClose(err) {
  var context = this.context;
  var buffer = null;
  var callback = context.callback;

  if (context.err)
    return callback(context.err);

  if (context.size === 0)
    buffer = Buffer.concat(context.buffers, context.pos);
  else if (context.pos < context.size)
    buffer = context.buffer.slice(0, context.pos);
  else
    buffer = context.buffer;

  // maybe ?
  if (context.encoding) {
    try {
      buffer = buffer.toString(context.encoding); // currently this line is not wrapped in a try/catch
    } catch (e) {
      if (!err) err = e; // which error gets priority?
    }
  }

  callback(err, buffer);
}

node/lib/fs.js

Lines 377 to 378 in f8152df

    
           if (context.encoding) 
        
             buffer = buffer.toString(context.encoding);

fs.readFile is understandably hot code, I haven't done benchmarking on the impact of a try…catch here.

Given that there appears to be an upper limit to what can be stringified safely, perhaps the max length could be tested for before attempting the stringification, and a 'this is too big' Error would ideally pop out of the callback.

Suggest at a minimum a more helpful error message "our buffering is good but have you tried the streams?"

AFAIK V8 can optimize functions with try...catch blocks now. I'm just not sure if it is in 4.5

@targos

@targos looks like it should

p.s. Sorry it is just ‘for-of’, ‘class’, ‘with’ and computed property names., not try/catch

Sorry it is just ‘for-of’, ‘class’, ‘with’ and computed property names., not try/catch

And only with TF (--turbo) but that's disabled by default.

@bnoordhuis

@bnoordhuis are you sure?

from node --v8-options:

  --turbo_shipping (enable TurboFan compiler on subset)
        type: bool  default: true
  --turbo_try_catch (enable try-catch support in TurboFan)
        type: bool  default: true

depends what is the effect of the --turbo_shipping flag

Nothing until you specify --turbo. :-)

(Strictly speaking, --turbo_filter='*' accomplishes the same thing.)

You can check for yourself with --trace_opt --trace_deopt.

I have this problem too

Fixed in b620790

fs: pass err to callback if buffer is too big

@NikosEfthias Can you tell me the specific version and platform you are using?

Hi,

I am trying to read codiegniter cache file, In this file data stored in serialize, When i try to read it retrun proper data but some time break serialize string data and throw error. Please check below error :

I have try below code :

fs.watch('../application/cache', function (event, filename) {
    if (filename){
        fs.readFile('../application/cache/'+filename, 'utf8', function (err, data) {
            if (err) throw err;
                        if(typeof data != 'undefined'){
                            console.log("Final Data ============= ",data);
                            if(data && PHPUnserialize.unserialize(data)){
                                var result = PHPUnserialize.unserialize(data);
                                for(var k in result)
                                {
                                    var temp = result[k];
                                    console.log('Result', temp.match_games);
                                    if(typeof temp.match_games != 'undefined')
                                    {
                                        broadCastLiveScore(temp.match_games, temp);
                                    }
                                }
                            }
                        }
        });
    }
    // console.log('filename', filename);
    // Prints: <Buffer ...>
});

and i got below errors :

throw new that.window[type](msg, filename, line);
                           ^

TypeError: Cannot read property 'Error' of undefined

Can you provide me solution, Thanks in Advance

Error Data

@girishp15

@girishp15 That looks neither related nor an issue with node.js core. Please don't hijack issues.

Navigation Menu

Repository navigation

`fs.ReadFile` throws asynchronously with encoding on large enough input #2767

Example program

Input: 268 megabytes of data

Input: 269 megabytes of data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	Buffer.prototype.toString = function() {
	if (arguments.length === 0) {
	var result = this.utf8Slice(0, this.length);
	} else {
	var result = slowToString.apply(this, arguments);
	}
	if (result === undefined)
	throw new Error('toString failed');
	return result;
	};

	if (context.encoding)
	buffer = buffer.toString(context.encoding);

fs.ReadFile throws asynchronously with encoding on large enough input #2767

Description

Example program

Input: 268 megabytes of data

Input: 269 megabytes of data

Activity

targos commented on Sep 9, 2015

YurySolovyov commented on Sep 9, 2015

bnoordhuis commented on Sep 9, 2015

targos commented on Sep 9, 2015

targos commented on Sep 9, 2015

bnoordhuis commented on Sep 9, 2015

alecrajeev commented on Sep 15, 2015

evanlucas commented on Oct 23, 2015

evanlucas commented on Dec 3, 2015

girishp15 commented on May 19, 2016

bnoordhuis commented on May 19, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`fs.ReadFile` throws asynchronously with encoding on large enough input #2767