Repository navigation
fs.Dir.read() is very slow #29941
Description
- Version: 12.12.0
- Platform: macOS catalina
- Subsystem: FS
I maintain the popular readdirp module. Decided to try fs.opendir from node 12.12. The results are suboptimal: it's 3x slower to walk file trees when compared to fs.readdir.
To reproduce:
- Check out the gist: https://github.com/proxy/gist.github.com/paulmillr/4e3a4da2ad8e98cf0fe7dc819242532a
- By switching logic from
if (opendir)toif (!opendir)I get 3x speedup, which means opendir is 3x slower than readdir.
Activity
addaleax commentedon Oct 12, 2019
Fwiw, #29893 isn’t released yet and is going to improve performance significantly.
That being said, the big advantage of
fs.opendir()is that it has a low upper limit on the memory necessary to read a directory, and not necessarily being faster thanfs.readdir().paulmillr commentedon Oct 12, 2019
I think it must have comparable performance (-+ 30%), because it's reasonable to assume the software doesn't know how big a directory is ahead of time. Would be especially useful for
readdirp.addaleax commentedon Oct 12, 2019
With that PR, I’m getting something closer to an 80 % perf difference.
One possibility that was suggested in it was making the number of entries read in one go configurable. I think that would bring this into acceptable parameters for you.
paulmillr commentedon Oct 12, 2019
Great!
[-]fs.opendir is very slow[/-][+]fs.Dir.read() is very slow[/+]Fishrock123 commentedon Oct 12, 2019
So, this will always be slower than
fs.readdir(). Always, because to use it reasonably, you're going to need an async iterator (Promises) or an object stream (ugh).You'd be able to get much closer by having a different but frankly weird/awful api: an async iteratable of sync iterables...
That being said that kind of subverts the point of the API. Sure, it should buffer to some extent because performance can reasonably be substantially improved without significant drawback. However, the large idea of such an API is to allow scheduled work on said directory entries to be started/return as iterator progresses, and to avoid such a large memory allocation on large directories.
piyukore06 commentedon Oct 12, 2019
Would you like a PR for that?
addaleax commentedon Oct 12, 2019
@Fishrock123 We have a
highWaterMarkoption for streams, so adding something similar here seems fine to me. In the end it just gives the user more control over the memory-vs-time tradeoff that’s involved here.@piyukore06 Unless there’s a strong objection, I’d say go for it. :)
Fishrock123 commentedon Oct 12, 2019
Oh, I wasn't clear but I certainly support being able to configure this. In fact, I was thinking about that as I was reviewing your PR. 😄
Fishrock123 commentedon Oct 15, 2019
@piyukore06 Please let us know if you need any help making that PR. It'l require a bit of C++, but only a little bit.
piyukore06 commentedon Oct 17, 2019
Yes @Fishrock123 I feel a bit stuck, I will push my changes as soon as possible. Maybe you people could take a look and point me in right direction..
Thanks in advance :)
piyukore06 commentedon Oct 19, 2019
@addaleax @Fishrock123 I don't think I completely get the whole building, compiling and testing process. It's taking more time than expected, so If anyone else wants to go ahead and finish it, please do. I will try and understand more about the ecosystem and try to find another issue to contribute to. Thank you for your patience.
fs: add `bufferSize` option to `fs.opendir()`
fs: add `bufferSize` option to `fs.opendir()`
fs: add `bufferSize` option to `fs.opendir()`
paulmillr commentedon Feb 5, 2020
@piyukore06 I don't see any performance improvements in the test from first post. Basically no changes with node 13-latest and bufferSize: 1024 / 4096 / whatevs. BufferSize: 1 makes this slower, so the arg works.
fs: add `bufferSize` option to `fs.opendir()`
fs: add `bufferSize` option to `fs.opendir()`