Memory leak introduced in v10.15.3

Version: v10.15.3
Platform: Linux
Subsystem: napi

Our module that has been migrated to N-API is experiencing a memory leak in version 10.15.3 but not in version 10.15.2. I verified that reverting the changes made in this pull request eliminates the memory leak. Clearly that isn't the right solution but hopefully it will point in the right direction!

Is there any chance you could share a reproduction?

I have a reproduction that works with our node-oracledb driver -- but if you don't have an Oracle Database handy that won't help much. If you do I can share the code that causes the issue; otherwise, it will take some time to develop a test case that demonstrates the issue!

@mhdawson

Depending on the complexity, looking at the code might help, because we know we’re looking for code that uses napi_create_reference() et al?

/cc @mhdawson as the PR author

The code for the driver can be found here. The code that is probably the most likely to be the source of the issue can be found here. Specifically, it acquires the value of a reference (for the constructor), creates an instance and then calls napi_wrap() to associate a structure with the newly created object.

I attach a zip file containing a simple module that simply creates an instance and wraps it:
issue_26667.zip. The pattern suggests that there is a race condition involved and that only occasionally it leaks -- but if you do it often enough it becomes obvious. Hopefully this is sufficient to track down the issue. After building the module, you can run the script as follows:

NODE_PATH=build/Release node --expose-gc demo.js

This will create a file called stats.txt which contains a dump of the number of iterations processed as well as the RSS and heap size.

@anthony-tuininga

The pattern suggests that there is a race condition involved and that only occasionally it leaks -- but if you do it often enough it becomes obvious. Hopefully this is sufficient to track down the issue.

@anthony-tuininga Are you saying that only occasionally there is a leak visible when running the program, or that you need many iterations within a single process in order to see the leak?

Because I’m having a hard time seeing an actual leak here – at the very least, the finalizer callback always seems to be called for exactly 25000 objects each iteration (so, the expected amount), at least locally…

What I meant was that the amount of memory lost is not sufficient to indicate that memory is being lost each iteration. Instead memory must be lost on some iterations only. Unfortunately, I ran both 10.15.2 and 10.15.3 with my simple module and discovered that both leak memory, albeit very slowly. I ran each of the loops 4 billion times and heap memory increased about 4 MB in both cases and RSS about 10 MB. So my example doesn't demonstrate the problem. :-(

But there is a marked difference between the two versions when using the node-oracledb driver test case and the difference goes away if I revert the pull request I mentioned earlier. I'll see if I can find a better test case next week.

Ok. I created a cut-down version of the node-oracledb module which has all of the Oracle bits ripped out. It demonstrates the problem. With Node.js 10.15.2 it stabilises around 350,000 iterations but with Node.js 10.15.3 it increases at around 100 bytes/iteration.

issue_26667.zip

Once unzipped and built, run with the following command:

NODE_PATH=. node --expose-gc demo.js

@addaleax

@addaleax, with the new test case are you able to replicate the issue?

This is what I see running your example on master:
``
Num Iters,RSS,Heap Total,Heap Used,Ext
25000,84611072,43147264,2840488,9146
50000,82440192,39477248,2716024,9146
75000,84344832,38952960,2722368,9146
100000,84377600,39477248,2724104,9146
125000,86728704,39477248,2724104,9146
150000,89051136,39477248,2724104,9146
175000,91439104,39477248,2724240,9146
200000,94646272,38428672,2725080,9146
225000,96698368,38952960,2725080,9146
250000,99041280,39477248,2713416,9146
275000,101924864,38428672,2725080,9146
300000,104325120,38428672,2725088,9146
325000,106901504,39477248,2725088,9146
350000,109899776,38428672,2725088,9146
375000,111890432,38428672,2725088,9146
400000,114360320,38952960,2725088,9146
425000,116314112,39477248,2725088,9146
450000,119160832,39477248,2725088,9146
475000,121499648,38952960,2713424,9146
500000,124059648,39477248,2725088,9146
525000,126201856,38952960,2725088,9146
550000,128806912,38952960,2713424,9146
575000,131485696,38952960,2725088,9146
600000,134086656,38952960,2726368,9146
625000,136589312,39477248,2714704,9146
650000,139042816,38428672,2726368,9146
675000,141357056,38952960,2726368,9146


It that similar to what you were seeing on 10.15.3?

Yes. If you run with Node.js 10.15.2 you will see that the RSS remains relatively stable.

Num Iters,RSS,Heap Total,Heap Used,Ext
25000,136855552,42713088,4252656,8272
50000,140029952,43237376,4256360,8272
75000,141082624,43761664,4264688,8272
100000,142774272,44285952,4251600,8272
125000,147193856,44810240,4263944,8272
150000,147931136,45334528,4252536,8272
175000,148590592,45334528,4264080,8272
200000,148500480,45334528,4264120,8272
225000,148443136,45334528,4252744,8272
250000,148815872,45858816,4264136,8272
275000,149032960,45858816,4264184,8272
300000,149032960,45858816,4252744,8272
325000,149032960,45858816,4264152,8272
350000,149037056,45858816,4264120,8272
375000,149037056,45858816,4252744,8272
400000,149037056,45858816,4252728,8272
425000,149045248,45858816,4264168,8272
450000,149045248,45858816,4252760,8272
475000,149045248,45858816,4264184,8272
500000,149045248,45858816,4252760,8272
525000,148520960,45334528,4264152,8272
550000,148656128,45334528,4264136,8272
575000,148709376,45334528,4264136,8272
600000,148873216,45334528,4264152,8272
625000,148852736,45858816,4252776,8272
650000,149315584,45858816,4264104,8272
675000,149315584,45858816,4264152,8272

I'll run with valgrind to see if anything obvious shows up.

@hashseed

@hashseed one thing I thought of is that I think the check in #24494 assumes that the Finalizer called by the gc will run on the main thread as opposed to some other thread. Can you confirm that is a valid assumption?

I have verified that a copy of 10.15.3, patched with the PR for optimization #27085 resolves the memory leak.

@anthony-tuininga

@anthony-tuininga, plan to land #27085 in master tomorrow.

Great. Thanks!

n-api: reduce gc finalization stress

Was this fixed in v10, or only in v12?

I believe it landed in master before 12.x was cut so it has #27085 as well.

But this is a regression in 10, right? What's the backporting policy?

I am running into this issue or something very closely related. I am using buffers with external memory. The fix does not seem to help – I am testing this on Node.js v12.10.0 on macOS.

I have modified the test case to demonstrate this behaviour with napi_create_external_buffer(). The finalizer is never run at all!

issue_26667_buffer.zip

These are my results:

2500000,630390784,37289984,1959768,786747
5000000,939368448,38866944,2062784,786211
7500000,1264635904,40177664,2052776,786211
10000000,1591042048,40701952,2052880,786211
12500000,1916497920,40964096,2053160,786211
15000000,2242945024,41488384,2053136,786211
17500000,2568470528,42274816,2119496,786211
20000000,2893910016,42799104,2053896,786211
22500000,2926735360,43323392,2054176,786211
25000000,2865909760,43585536,2054152,786211

However the finalizers are run if we occasionally allow the event loop to continue with setImmediate(). See the modified demo.js: demo_modified.js.zip. This is quite clear from the log, memory usage stabilises pretty quickly:

Num Iters,RSS,Heap Total,Heap Used,Ext
2500000,485388288,37027840,1963424,786266
5000000,717176832,39129088,2054568,785730
7500000,686436352,40177664,2055376,785730
10000000,704942080,40964096,2055880,785730
12500000,708456448,41750528,2056104,785730
15000000,713523200,42012672,2056104,785730
17500000,713523200,42012672,2056104,785730
20000000,715087872,42536960,2056104,785730
22500000,718864384,42274816,2056104,785730
25000000,708702208,42536960,2056104,785730

I hope this helps!

@rolftimmermans

@rolftimmermans That's working as expected. Your code has a finalizer and those are always deferred to the next event loop tick:

node/src/node_api.cc

Lines 38 to 59 in 064e111

    
           static void FinalizeBufferCallback(char* data, void* hint) { 
        
             std::unique_ptr<BufferFinalizer, Deleter> finalizer{ 
        
                 static_cast<BufferFinalizer*>(hint)}; 
        
             finalizer->_finalize_data = data; 
        
             node::Environment* node_env = 
        
                 static_cast<node_napi_env>(finalizer->_env)->node_env(); 
        
             node_env->SetImmediate( 
        
                 [finalizer = std::move(finalizer)](node::Environment* env) { 
        
               if (finalizer->_finalize_callback == nullptr) return; 
        
               v8::HandleScope handle_scope(finalizer->_env->isolate); 
        
               v8::Context::Scope context_scope(finalizer->_env->context()); 
        
               finalizer->_env->CallIntoModuleThrow([&](napi_env env) { 
        
                 finalizer->_finalize_callback( 
        
                     env, 
        
                     finalizer->_finalize_data, 
        
                     finalizer->_finalize_hint); 
        
               }); 
        
             }); 
        
           }

(Note that SetImmediate() call.)

@bnoordhuis

@bnoordhuis Thanks for the quick response. Is there any way to release memory earlier, or is this just the way things are expected to work with external buffers? Seems like it should be possible to release the memory sooner... :/

@rolftimmermans

@rolftimmermans It's by design. Finalizers can call into JS land. If n-api didn't defer the callback, it'd be pretty easy to create a cb→js→cb→js→etc. stack overflow

Seems the behaviour I saw only applies to napi_create_external_buffer() and not to napi_create_external_arraybuffer(). The latter works fine and leaks no memory in similar circumstances.

I'll open a new issue to address this.

That's because it doesn't defer the callback:

node/src/js_native_api_v8.cc

Lines 305 to 325 in 064e111

    
           static void SecondPassCallback(const v8::WeakCallbackInfo<Reference>& data) { 
        
             Reference* reference = data.GetParameter(); 
        
             if (reference->_finalize_callback != nullptr) { 
        
               reference->_env->CallIntoModuleThrow([&](napi_env env) { 
        
                 reference->_finalize_callback( 
        
                     env, 
        
                     reference->_finalize_data, 
        
                     reference->_finalize_hint); 
        
               }); 
        
             } 
        
             // this is safe because if a request to delete the reference 
        
             // is made in the finalize_callback it will defer deletion 
        
             // to this block and set _delete_self to true 
        
             if (reference->_delete_self) { 
        
               Delete(reference); 
        
             } else { 
        
               reference->_finalize_ran = true; 
        
             } 
        
           }

I'd say that's an oversight, possibly due to a misunderstanding of how the two GC passes work.

I think this can be closed out as the issue was addressed, and 10.x is also now out of service. Please let me know if that was not the right thing to do.

Navigation Menu

Repository navigation

Memory leak introduced in v10.15.3 #26667

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	static void FinalizeBufferCallback(char* data, void* hint) {
	std::unique_ptr<BufferFinalizer, Deleter> finalizer{
	static_cast<BufferFinalizer*>(hint)};
	finalizer->_finalize_data = data;

	node::Environment* node_env =
	static_cast<node_napi_env>(finalizer->_env)->node_env();
	node_env->SetImmediate(
	[finalizer = std::move(finalizer)](node::Environment* env) {
	if (finalizer->_finalize_callback == nullptr) return;

	v8::HandleScope handle_scope(finalizer->_env->isolate);
	v8::Context::Scope context_scope(finalizer->_env->context());

	finalizer->_env->CallIntoModuleThrow([&](napi_env env) {
	finalizer->_finalize_callback(
	env,
	finalizer->_finalize_data,
	finalizer->_finalize_hint);
	});
	});
	}

	static void SecondPassCallback(const v8::WeakCallbackInfo<Reference>& data) {
	Reference* reference = data.GetParameter();

	if (reference->_finalize_callback != nullptr) {
	reference->_env->CallIntoModuleThrow([&](napi_env env) {
	reference->_finalize_callback(
	env,
	reference->_finalize_data,
	reference->_finalize_hint);
	});
	}

	// this is safe because if a request to delete the reference
	// is made in the finalize_callback it will defer deletion
	// to this block and set _delete_self to true
	if (reference->_delete_self) {
	Delete(reference);
	} else {
	reference->_finalize_ran = true;
	}
	}

Memory leak introduced in v10.15.3 #26667

Description

Activity

addaleax commented on Mar 14, 2019

anthony-tuininga commented on Mar 14, 2019

addaleax commented on Mar 14, 2019

anthony-tuininga commented on Mar 14, 2019

anthony-tuininga commented on Mar 14, 2019

addaleax commented on Mar 16, 2019

anthony-tuininga commented on Mar 16, 2019

anthony-tuininga commented on Mar 18, 2019

anthony-tuininga commented on Mar 20, 2019

mhdawson commented on Mar 20, 2019

anthony-tuininga commented on Mar 20, 2019

mhdawson commented on Mar 20, 2019

mhdawson commented on Mar 20, 2019

24 remaining items

anthony-tuininga commented on Apr 5, 2019

mhdawson commented on Apr 8, 2019

anthony-tuininga commented on Apr 8, 2019

orgads commented on Aug 15, 2019

mhdawson commented on Aug 15, 2019

orgads commented on Aug 15, 2019

rolftimmermans commented on Oct 8, 2019

bnoordhuis commented on Oct 8, 2019

rolftimmermans commented on Oct 8, 2019

bnoordhuis commented on Oct 8, 2019

rolftimmermans commented on Oct 8, 2019

bnoordhuis commented on Oct 8, 2019

mhdawson commented on May 27, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions