Repository navigation
zlib: memory leak with gunzipSync #1479
Description
I was running a script to read sequentially a lot of gzipped files, and print on stdout the result of a computation for each file.
After about 16'000 files, the process just stopped and Killed was printed on the terminal.
I guess that the memory used by the process increases until the kernel decides to kill it.
I could reduce the code to this testcase:
'use strict';
var zlib = require('zlib');
var data = 'abcdefghijklmnopqrstuvwxyz';
var gzipped = zlib.gzipSync(data);
while (true) {
var contents = zlib.gunzipSync(gzipped);
process.stdout.write(contents.toString() + '\n');
}> node test.js > /dev/null
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
[1] 15363 abort (core dumped) node test.js > /dev/nullThis is what I see with top command just before the process disappears:
> top
15363 mzasso 20 0 10,240g 3,716g 11776 R 111,0 24,2 0:53.59 nodeNB: a similar code, that would just write data in the while loop keeps a stable memory consumption
Activity
[-]memory leak with zlib.gunzipSync[/-][+]zlib: memory leak with gunzipSync[/+]targos commentedon Apr 25, 2015
It also happens on Windows :
Fishrock123 commentedon Apr 25, 2015
Can you get some deeper info on it? Maybe run it through valgrind?
dchusovitin commentedon Apr 25, 2015
Problem in your code. GC isn't called, because you use only blocking methods.
Run "node --expose_gc test.js"
Fishrock123 commentedon Apr 25, 2015
it's true. Using
while (true)without exit is very bad, and makes it easier to cause your own errors.targos commentedon Apr 25, 2015
Yeah of course,
while(true)was for the example. It is not what I was using in my real script.I had something like this:
Now I am using the async version without any issue. The code was just simpler using gunzipSync...
Thanks for your answers :)
ChALkeR commentedon May 10, 2015
https://github.com/iojs/io.js/blob/v2.0.1/lib/zlib.js#L458 — this is the line that causes it.
It delays emitting the
closeevent until the next tick, lockingthis. That's why that objects are not collected, because in sync code everything is running in a single tick.Looks like even the
*Syncmethods were created with async in mind.ChALkeR commentedon May 11, 2015
@dchusovitin You are incorrect. Both incremental and full GC runs are automatically called even in synchronous code. Moreover, your code sample doesn't even change anything (with and without fixing the mistype) except for slowing things down.
trevnorris commentedon May 11, 2015
I think it's more that everything
*Syncwas created after their async counterparts. If something async is happening in a sync call then it's a bug.bnoordhuis commentedon May 11, 2015
But the reason to defer the event is to prevent infinite recursion. I'm not sure if this is fixable without breaking something else.
al6x commentedon Jan 1, 2016
Same bug with
zlib.gzipSyncused in a loop - process out of memory.ChALkeR commentedon Mar 19, 2016
#5707 should fix this.