http2: How should I be avoiding "New streams cannot be created after receiving a GOAWAY" errors

Node.js Version: 10+
Scope (install, code, runtime, meta, other?): usage
Module (and version) (if relevant): http2

I maintain the module @grpc/grpc-js which uses the http2 module. I have gotten reports of occasional "New streams cannot be created after receiving a GOAWAY" errors. The code I have listens for the close, error, and goaway events, and if it sees any of them it stops using that session. Is that the intended usage pattern to avoid that kind of error? Is there anything else I should be checking to more consistently avoid that error?

@nodejs/http2

Can anyone help me with this?

After receiving a GOAWAY you cannot create new streams on the sessions. This is part of the http2 protocol. It is not clear what kind of help do you need on this.. would you mind to add some code to reproduce?

@mcollina

As @mcollina mentioned, here's relevant HTTP/2 spec which explains new streams can't be created after GOAWAY frame https://http2.github.io/http2-spec/#GOAWAY

GOAWAY allows an endpoint to gracefully stop accepting new streams while still finishing processing of previously established streams.

I understand that new streams cannot be created after receiving a GOAWAY. I am trying to understand how to use the Http2Session API to ensure that that requirement is met. As stated in my question, I am already listening for the goaway event, and not starting new streams on that session after that, but I am still occasionally getting the quoted error. To me this implies that there is something else I should be doing to avoid that and I am trying to understand what that something else is.

I think having an example that reproduce the error would help, as well as a stacktrace.

I don't know much about http2, but just wondering if this code serves any clues:

If you close the client too quick, then the error is emitted on the client request with goaway as the error type
sometimes a goaway event is emitted on the session object too
if you close the client after a while, the request object does not receive anything

is this indication of any racy behavior?

$ cat 2105.js

const h = require('http2')
const s = h.createServer()

s.on('session', (session) => {
  session.on('goaway', (e, l, o) => {
    console.log(`${e}, ${l}, ${o}`)
    s.close()
  })
})

s.listen(12000, () => {
  const c = h.connect('http://localhost:12000')
  const q = c.request()
  q.on('error', (e) => {
    console.log(e)
    s.close()
  })
  setTimeout(() => {
    c.close()
  }, +process.argv[2])
})

#node 2105.js 1
Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
    at ClientHttp2Stream.requestOnConnect (internal/http2/core.js:642:17)
    at internal/http2/core.js:1609:53
    at Array.forEach (<anonymous>)
    at ClientHttp2Session.<anonymous> (internal/http2/core.js:1609:38)
    at Object.onceWrapper (events.js:307:26)
    at ClientHttp2Session.emit (events.js:222:7)
    at emit (internal/http2/core.js:276:8)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_HTTP2_GOAWAY_SESSION'
}

#node 2105.js 1
Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
    at ClientHttp2Stream.requestOnConnect (internal/http2/core.js:642:17)
    at internal/http2/core.js:1609:53
    at Array.forEach (<anonymous>)
    at ClientHttp2Session.<anonymous> (internal/http2/core.js:1609:38)
    at Object.onceWrapper (events.js:307:26)
    at ClientHttp2Session.emit (events.js:222:7)
    at emit (internal/http2/core.js:276:8)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_HTTP2_GOAWAY_SESSION'
}
0, 0, undefined

#node 2105.js 100
0, 0, undefined
^C
#

Completely unrelated to gRPC, I just ran into this as well. I am handling the goaway event on my ClientHttp2Session, but it's never called:

const session = http2.connect(`https://${host}`);
    session
      .on('error', e => {
        console.error(`*ERROR*: ${e}`);
        delete sessions[host];
      })
      .on('goaway', (errorCode, lastStreamId) => {
        console.error(`*GOAWAY*: ${errorCode} : ${lastStreamId}`);
        delete sessions[host];
      });

But when making a request by running:

const req = session.request(headers);

I get the error:

Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
      at ClientHttp2Session.request (internal/http2/core.js:1621:13)
      at Object.request (build/src/http2.js:84:32)
          -> /Users/beckwith/Code/nodejs-googleapis-common/src/http2.ts:112:30
      at createAPIRequestAsync (build/src/apirequest.js:270:23)
          -> /Users/beckwith/Code/nodejs-googleapis-common/src/apirequest.ts:325:17
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async Context.<anonymous> (build/system-test/test.http2.js:119:9)
          -> /Users/beckwith/Code/nodejs-googleapis-common/system-test/test.http2.ts:128:5

I would expect to get a goaway event first 🤷

@murgatroid99

Sorry to answer an old issue. I just came to it because @murgatroid99 mentioned it in nodejs/TSC#1168.

is this indication of any racy behavior?

No, in fact, this is expected. When a c.request() is called, it doesn't mean that a request is sent at that moment.

From the source code:

// When a ClientHttp2Session is first created, the socket may not yet be
// connected. If request() is called during this time, the actual request
// will be deferred until the socket is ready to go.

The example provided by @gireeshpunathil is basically sending a request() when the server is already closed. That explains the GOWAY error.

I'll create a PR to improve the clienthttp2session.request explanation. Meanwhile, would be great to have a reproducible code in case if you think it is a bug.

@murgatroid99

@murgatroid99 if you think this has not been properly addressed.. could you please include some code to reproduce your problem?

@murgatroid99

ping @murgatroid99

It seems there has been no activity on this issue for a while, and it is being closed in 30 days. If you believe this issue should remain open, please leave a comment.
If you need further assistance or have questions, you can also search for similar issues on Stack Overflow.
Make sure to look at the README file for the most updated links.

It seems there has been no activity on this issue for a while, and it is being closed. If you believe this issue should remain open, please leave a comment.
If you need further assistance or have questions, you can also search for similar issues on Stack Overflow.
Make sure to look at the README file for the most updated links.

Navigation Menu

Repository navigation

http2: How should I be avoiding "New streams cannot be created after receiving a GOAWAY" errors #2105

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

http2: How should I be avoiding "New streams cannot be created after receiving a GOAWAY" errors #2105

Description

Activity

addaleax commented on Aug 7, 2019

murgatroid99 commented on Sep 6, 2019

mcollina commented on Sep 8, 2019

trivikr commented on Sep 9, 2019

murgatroid99 commented on Sep 16, 2019

mcollina commented on Sep 16, 2019

gireeshpunathil commented on Dec 23, 2019

JustinBeckwith commented on Jun 18, 2020

RafaelGSS commented on Mar 2, 2022

mcollina commented on Mar 7, 2022

gireeshpunathil commented on Jun 22, 2023

github-actions commented on May 14, 2024

github-actions commented on Jun 13, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions