unable to deal with file names that have invalid encoding

There is a file in my user's file system and io.js cannot use any of the fs API on it, because the file name has an invalid encoding. File names are byte arrays; not strings. This is the root of the problem.

andrewrk/groovebasin#383

How to reproduce (use linux):

$ wget https://s3.amazonaws.com/superjoe/temp/encoding-test.tar.gz
$ tar xvf encoding-test.tar.gz 
encoding-test/
encoding-test/test.js
encoding-test/dir/
encoding-test/dir/\377
$ cd encoding-test/
$ cat test.js
var fs = require('fs');
var path = require('path');
var dir = 'dir/';
var list = fs.readdirSync(dir);
var filename = path.join(dir, list[0]);
console.log(filename);
console.log(fs.lstatSync(filename));
$ node test.js 
dir/�

fs.js:688
  return binding.lstat(pathModule._makeLong(path));
                 ^
Error: ENOENT, no such file or directory 'dir/�'
    at Object.fs.lstatSync (fs.js:688:18)
    at Object.<anonymous> (/home/andy/Downloads/encoding-test/test.js:7:16)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:906:3

It's a known issue, see e.g. nodejs/node-v0.x-archive#2387. (There is another, older issue but I can't find it just now.)

There is currently no workaround. Fixing it probably entails rewriting src/node_file.cc and lib/fs.js to accept both buffers and strings (and, perhaps, typed arrays.)

Most platforms treat file paths as byte arrays but Windows and OS X do not; I'm not sure if that would affect node.js but it's something to keep in mind. It's possible that the Windows port of libuv already takes care of all the details but the OS X port most assuredly does not.

At any rate, pull requests are welcome!

Libuv for windows assumes that filenames are valid UTF-16 and converts those to and from UTF-8.
However on windows are really uint16 arrays; they are not necessarily valid UTF-16, so a similar issue exists on windows.

If possible I'd like to avoid duplicating issues from the node tracker into the iojs tracker. Closing this in favor of the original issue.

I don't understand, isn't io.js a fork? Shouldn't it have independent issues? Let's leave the issue open in io.js until it's resolved. I'll see about making a pull request to make src/node_file.cc and lib/fs.js accept buffers in addition to strings.

Navigation Menu

Repository navigation

unable to deal with file names that have invalid encoding #128

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

unable to deal with file names that have invalid encoding #128

Description

Activity

bnoordhuis commented on Dec 9, 2014

piscisaureus commented on Dec 9, 2014

chrisdickinson commented on Dec 16, 2014

andrewrk commented on Jan 19, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions