Repository navigation
HTTP/2 POST loop fails with ENHANCE_YOUR_CALM after 40701 iterations #23116
Description
- Version: v10.10.0
- Platform: Linux david-Latitude-E6440 4.15.0-34-generic logo ideas #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
- Subsystem: http2
Here's a simple server app which reads HTTP/2 POST request bodies to their end and then responds:
const http2 = require('http2');
const fs = require('fs');
const path = require('path');
const { Writable } = require('stream');
const server = http2.createSecureServer({
key: fs.readFileSync(path.join(__dirname, 'server.key')),
cert: fs.readFileSync(path.join(__dirname, 'server.crt'))
});
server.on('session', function (session) {
session.on('stream', function (stream, headers) {
stream.on('end', function () {
this.respond({
':status': 200,
'Access-Control-Allow-Origin': 'http://localhost:8000'
}, {
endStream: true
});
});
stream.pipe(new Writable({
write: (chunk, encoding, cb) => cb()
}));
});
});
server.listen(7000, function () {
console.log('READY.');
});For this test, the cert and key are self-issued and my browser trusts them.
Here's a Web page which makes requests in series to the server:
<html>
<head>
<script>
async function test() {
while (true) {
response = await fetch('https://localhost:7000', { method: 'POST' });
await response.arrayBuffer();
}
}
</script>
</head>
<body onload='test()'>
</body>
</html>I expect this to continue indefinitely.
However, what happens is I get the following error after 40701 iterations:
memtest.html:6 POST https://localhost:7000/ net::ERR_SPDY_PROTOCOL_ERROR
I did some debugging using Wireshark and found the error was ENHANCE_YOUR_CALM.
After adding some tracing to node_http2.cc, I found the error being produced here: https://github.com/nodejs/node/blob/v10.10.0/src/node_http2.cc#L863
I've tracked this down to current_nghttp2_memory_ continually growing.
The cause for this is nghttp2 allocation of 232 bytes for each stream (https://github.com/nodejs/node/blob/v10.10.0/deps/nghttp2/lib/nghttp2_session.c#L1029) is never being released.
This is because nghttp2 keeps closed streams around (up to the concurrent connection limit).
If I add the following line to Http2Options::Http2Options in src/node_http2.cc:
nghttp2_option_set_no_closed_streams(options_, 1);then the test works as expected and doesn't fail at 40701 iterations.
Activity
davedoesdev commentedon Sep 27, 2018
The relevant code in nghttp2 is here: https://github.com/nodejs/node/blob/v10.10.0/deps/nghttp2/lib/nghttp2_session.c#L1211
I think this is going to cause a failure for any Node.js HTTP/2 app which processes many requests in the same session, due to tracking and limiting
current_nghttp2_memory_innode_http2.cc.lpinca commentedon Sep 27, 2018
cc: @nodejs/http2
davedoesdev commentedon Sep 27, 2018
Note in the conditional above,
!is_my_stream_idis checking that the stream is an incoming request andnghttp2_stream_in_dep_tree(stream)is checking the stream is in the dependency tree.I'm seeing both true in the logs. The requests are inserted as dependencies of the root, e.g.:
jasnell commentedon Sep 27, 2018
This is specifically there to prevent denial of service attacks. It is possible to change the default maximum session memory limit using the
maxSessionMemoryoption (https://nodejs.org/dist/latest-v10.x/docs/api/http2.html#http2_http2_createserver_options_onrequesthandler). Use of theno_closed_streamsoption is not ideal because it prevents nghttp2 from performing a number of security and state checks.Try setting a significantly higher
maxSessionMemorylimit and giving it another go.mcollina commentedon Sep 27, 2018
To clarify, a single session would keep around 232 bytes for every stream that was closed?
From the nghttp2 docs, it seems
nghttp2_option_set_no_closed_streamsis only doingwhat other checks it is doing?
jasnell commentedon Sep 27, 2018
Ah, right, given that we're not really make use of priority, this likely can be switched off. Still, in this particular use case, I would still set a significantly higher
maxSessionMemory.davedoesdev commentedon Sep 27, 2018
If I increase
maxSessionMemoryto 100Mb, the server starts failing requests after 400k of them on a session.The Web page has no way of resetting its session with the server - the
fetchAPI has no way to tell the browser to use a new connection when the failures start (unless I've missed something).So this means long running sessions to Node over HTTP/2 don't work reliably.
jasnell commentedon Sep 27, 2018
Will investigate further! :)
http2: set nghttp2_option_set_no_closed_streams
http2: set nghttp2_option_set_no_closed_streams
http2: set nghttp2_option_set_no_closed_streams