AsyncLocalStorage does not work with http.Agent

Version: 14.5.0
Platform: macos
Subsystem: async_hooks

What steps will reproduce the bug?

'use strict';

const http = require('http');
const { AsyncLocalStorage } = require('async_hooks');

const session = new AsyncLocalStorage();

const agent = new http.Agent({
  maxSockets: 1
});

const get = (path, callback) => {
  http.request({
    agent,
    host: 'example.com',

    method: 'GET',
    path,
  }, callback).end();
};

let id = 0;
const server = http.createServer((req, res) => {
  session.run(id++, () => {
    console.error('new req', session.getStore());
    get('/', (remote) => {
      console.error('remote response', session.getStore());
      remote.pipe(res);
    });
  });
}).listen(0, () => {
  const { port } = server.address();
  http.request({ port }).end();
  http.request({ port }).end();
});

How often does it reproduce? Is there a required condition?

Always. No conditions.

What is the expected behavior?

I expect this code to print:

new req 0
new req 1
remote response 0
remote response 1

What do you see instead?

new req 0
new req 1
remote response 0
remote response 0

Additional information

This is very much expected behavior of CLS storage. Any sort of pooling would result in similar issues. What's concerning is that it can happen with use of an existing core primitive (http.Agent). In fact, use of any agent with maxSockets !== Infinity or keepAlive set to true would result in similar behavior.

This behavior can be documented, but I'd imagine that such documentation would look like an incompatibility within Node.js core. Given that whole async_hooks module experimental - should we consider removing AsyncLocalStorage instead?

cc @nodejs/collaborators

I think removing AsyncLocalStorage doesn't solve any problem. If there is no AsyncLocalStorage in core people implement it using async_hooks in userland and end up with the same problem.
We have several areas in core where resources are reused (e.g. HTTPParser) but create a new resource (and async id) everytime it's reused to avoid issues. Looks like there is a reuse where this wrapping is missing.

@Flarna

@Flarna even if we create a patch for this particular issue - similar problems would still be caused by other core or user-land modules. Whenever there is a pooling involved - there is a potential for a thread-local storage to get out of sync. From my experience, it is hard to impossible to debug and leads to security issues. I've personally seen developers using CLS as a storage for security data and application users performing activities for other random users as a result.

The issue with having it in the core is that core API comes with a promise both in the form of the documentation and provided by the fact that it is a part of the core and thus is can be trusted. Should we make a promise for something as frail and susceptible to issues as thread-local storage?

Perhaps, instead of shipping its version of CLS implementation - Node.js core should be more vocal about why CLS should not be used in user code at all?

I understand your point that we removing AsyncLocalStorage from core would remove the corresponding problems from core.
My feeling is that AsyncLocalStorage is as good/bad as async_hooks are. Do you think we should remove also async hooks?

The problem with not using CLS is that it disallows some usecases - concrete any APM tool I know uses CLS and I'm quite sure there is more then these.

I think removal of async_hooks should be discussed separately if discussed at all.

APM and any form of tracing/monitoring does not have an effect on the logic of the application so it is indeed safer. However, imagine a different scenario: thread-local storage is used for logging and the support team uses logs when the ground-truth is unavailable. What would happen if the logs of one request will get mixed with the logs of another? One can't argue that this is an incorrect use of the thread-local storage, because, well, it is what it was advertised for. However, evidently there are still possible ways in which this could go wrong even if this doesn't change the behavior of the server in general.

FWIW, I know plenty of tools that use CLS and contributed to removal of CLS at least in some of them. The fact that there exists software that uses faulty API/module doesn't mean that this functionality has to be present in broken form.

I have some exposure to CLS impl & integrations, and I'm coming at this from a high-level/black-box point of view. CLS is high value in my estimation.

It seems pretty clear that things that do pooling need to take steps to ensure reasonable CLS semantics for their callers. In the use-case mentioned, the Agent instance should have a CLS (if needed) branched from its point of creation; the http.request() should have CLS branched from its point of creation, with any Agent interactions passing the request's CLS through so they can be used in alignment with expected semantics.

Choosing one of these contexts accurately inside the Agent would depend on the situation, though. For instance, a) pool activity isn't request-specific, so logging cleanup of idle connections would use the Agent's CLS logger if that were a thing. Again with a hypothetical, b) logging "yay, this connection is still active, we'll hang onto it longer" might ALSO use the Agent's CLS logger even though an in-scope request having its own CLS context was the causation for it. ...but then c) it would use that request's CLS/async context explicitly, for a callback such as the one mentioned in this issue.

Maybe the issue title should be centered on http.request() and/or http.Agent not playing their expected cooperative role with CLS?

@rjharmon

@rjharmon what are you are saying is that we have a leaky abstraction that requires a full-stack support. Yet most Node.js apps are built around public npm modules that do not yet provide (and many won't for a long time) such support. Sure, it is possible to "fix" http.Agent to make it play better with this particular CLS and even introduce an API for doing so, but until whole ecosystem catches up - there are still going to be incompatibilities in the 3rd party modules and these modules may not even be in the direct dependencies of the application (as it was in my case some time ago).

@indutny

Chiming in on this as a former user of CLS for sensitive data: this is a tough issue to thread. There's a legitimate desire for CLS for operations like logging, APM, or associating resources with requests (like postgres connections, for example.) Doing this requires a lot of care, as @indutny notes -- if a dependency anywhere in your tree does any sort of connection pooling & doesn't use the AsyncResource API to correctly re-associate asyncwrap ids, context might be lost or swapped incorrectly. It's very hard to debug when this happens.

However.

Authors have been doing this with Node for a very long time – even before AsyncLocalStorage, through monkeypatching or via domains. There's a real desire for this feature; in my experience Node has not had much success in telling authors not to do something that they want to do. Providing a blessed API for doing this work gives Node an place to document the potential pitfalls, and maybe some amount of leverage over the situation (could Node provide an AsyncResourcePool API that issued AsyncWrap IDs in a CLS-compatible fashion? You could imagine this making the rounds to popular packages, like pg-pool, much in the same way domain support did.)

I'm in complete agreement that the feature, as it is now, is dangerous. However, it's a feature that authors continue to try to implement themselves. By acknowledging that authors want this functionality, we can better educate them on the pitfalls. Hopefully, eventually, we can work with the community to make the feature less dangerous. Backing out the feature would be doing node's users a disservice by continuing to let them fend for themselves.

@chrisdickinson

Fair points, for sure. I'd agree that the http module has a leaky abstraction. Possibly that's true more generally of AsyncLocalStorage too; that's not really how I'm seeing it, though I recognize I may be seeing wrongly. The facts of this issue's existence, and that both modules are included with nodejs, makes these distinctions pretty much boring, though.

With some clear guidelines, maybe we can help those 3rd-party module providers shorten "a long time" somewhat (perhaps in a way like @chrisdickinson elaborates) - the API seems straightforward enough, once one gets their head around it. Helping them to identify how/whether they're impacted, and to add not-v14 and not->v13.whatever clauses to their package.json might also be in order.

Shipping broken bundled modules in a v14-LTS in October would be a real bummer.

@chrisdickinson

@chrisdickinson beautiful take on it!

I can see the motivation behind having it in core. What I disagree with is that 3rd party modules will catch up and provide support for this. As you said single 3rd party module in indirect dependencies may mix up the async contexts for whole app and result in very unpredictable and dangerous behavior.

Having this API means that the ecosystem is going to fracture into modules that were modified to support this and the modules without the support. If we are to assume full cooperation from all module authors - there are still pure JavaScript code that is no longer (or lightly) maintained, but does not depend on Node.js core specifically and thus is perfectly usable.

It is okay to have a breaking an API change and we do it from time to time. What's unfortunate in the case of CLS is that the fact that the module does not support it is completely invisible to the module user up to the moment when it blows up in production.

@rjharmon

@rjharmon unfortunately I can't agree with you. http module implementation has its pitfalls, but in the context of this particular issue it suffers from the same symptoms that the rest of JavaScript modules might. There is nothing specifically incorrect about how it is written.

There's already many modules which properly use AsyncResource to bind the context and that's with async_hooks being marked as experimental. With AsyncLocalStorage we have a simpler API surface area than async_hooks and so it has a better chance of leaving experimental and thus a better chance of convincing module authors that they should use AsyncResource to support it properly.

Removing AsyncLocalStorage from core doesn't fix the issues, it just drops them all on the users to fix in userland where it can't be fixed safely or in a way that benefits everyone. The cost of fixing context propagation issues is large and so individuals tend to just leave them broken which has resulted in an ecosystem full of APM vendors that are likely to crash your app, which is obviously undesirable. By having it in core, we're committing to a goal of reaching greater stability and are thus making more fixes to do that, which is evidenced by the creation of executionAsyncResource, the thenables fix I made to V8, the promise hook fast path when not using the destroy hook in async_hooks--yes, it's not perfect, but in core we can work toward something more solid than the horribly broken and fragmented solutions the ecosystem has had for the last decade. We've made more progress on async_hooks performance, usability and stability in the last six months than in the entire lifetime of Node.js leading up to that, because we finally were able to get some of this stuff in core.

@Qard

@Qard I feel like this doesn't address the fact that one 3rd party module that does not use AsyncResource and does pooling will make whole system unstable. I'm all for stability and for providing a common space for people to work on ideas, but doesn't AsyncLocalStorage address only the latter one? From where I stand it looks like it is just conceptually unsound.

Note that it is not the same as being imperfect or having issues that can be fixed. In my opinion, the problem that was brought up is a serious design flaw. Similar issue (but in original CLS module and cls-hooked as well) have manifested in production (in fact, event enterprise) applications that I had chance to work on. I don't think that the Node.js core needs to encourage use of an API with undetectable caveats.

It depends how you think of "unstable" here. APM vendors are used to coding very defensively, and we DO communicate the importance of using AsyncResource to maintain context. It's not going to crash your app unless you use it fairly recklessly.

The feature is also considered experimental and arguably should maybe stay that way until a language spec can be decided and proven. As long as it remains experimental, I think it's not reasonable to expect the level of stability you seem to be expecting from it. Besides, it's not like there isn't plenty of other APIs in Node.js that can blow up in your face if you use them wrong.

Not sure if "unsecure" is the right word here as this somehow indicates that using AsyncLocal/async_hooks somewhere in an application automatically results in an security issue.

So is the use of home-grown cryptography. I don't see how marking it as "not secure for public use" is hurting APM tools. What such note means is that the person using this should very well be aware of the risks that it poses and understand that when used in an incorrect way ALS will leak sensitive information.

So is the use of home-grown cryptography. I don't see how marking it as "not secure for public use" is hurting APM tools. What such note means is that the person using this should very well be aware of the risks that it poses and understand that when used in an incorrect way ALS will leak sensitive information.

Using "not secure for public use" can cause some users to think APM tools that use ALS are insecure, while that is not exactly true. How about saying "Users should avoid using it directly. Improper utilization may lead to security issues."? Or something like it...

@mcollina

I know at least about one other edge case that breaks CLS-like APIS: queues. I have seen apps having a kind of rate-limiting feature by queueing requests, in that case, the current store can be missasociated.

My general opinion on the Async Hooks/AsyncLocalStorage situation is probably the same as expressed by many here:

we need something to provide such feature
having it in core has big advantages
packages maintainers should use AsyncResource <- I agree with @mcollina we are not good at letting them know

The use of AsyncLocalStorage is challenging for a lot of people. I was asked to review implementation a few times since it was released into node. But I don't think that's really a bad thing since in my mind, this API is mostly here for people building tools (APM, RASP, Framework maintainers, Tracing tool builders,...) rather than directly by the end user.

@mcollina

packages maintainers should use AsyncResource <- I agree with @mcollina we are not good at letting them know

We have a troubleshooting section in the doc: https://nodejs.org/api/async_hooks.html#async_hooks_troubleshooting, which is not much, but better than nothing.

To provide package maintainers with more information we could write a guide (https://nodejs.org/en/docs/guides/) dedicated to ALS. It could provide an overview of the API with necessary (not all of them) implementation details, mention performance cost and other quirks, like potential context loss due to various reasons (might be a good idea to list some of them), name some good use cases, as well as anti-patterns, and describe integration guidelines for package maintainers.

We just need better docs. The AsyncResource docs explain a bit about why you should be using that to fix gaps in the graph, but you wouldn't know to look for that if you're looking at the AsyncLocalStorage part of the docs. It's not particularly clear that the two APIs are linked.

As for the associated risks of context loss or adopting an incorrect context, that seems like a somewhat lesser risk to me than literally every production Node.js app on the internet deploying with a pile of ridiculously unsafe monkey patches and introducing thousands of subtle behavioural differences to the entire core API. I'd rather have the risk we can at least quantify and control.

The productive action here would be to identify where we could improve the docs and help to find solutions to the soundness problems of the current API and contribute solutions to Node.js core or the corresponding TC39 proposal.

Regardless of how you feel about this API, every Node.js app out there will be using it whether it's in core or not, and keeping it out of core just means it's going to be a lot less safe and a lot less maintained. I agree, it's not perfect, but there's no better alternative. If we had bytecode instrumentation I'd just use that, but we don't.

Better documentation for AsyncResource is certainly important and worth improving, but it won't give users a general understanding of best practices for ALS. That seems to be the main problem here.

Yep, I mean we should have some more prominent references to AsyncResource in the the docs for AsyncLocalStorage along with some explanation of the need for manual context binding in certain cases.

async_hooks: don't reuse resource in HttpAgent when queued

I'm seeing a similar issue as OP and it's not clear from this discussion where the breakage is or how one would go about dealing with it. My understanding of the docs is that session.run() should create a unique context for all down-stream async operations but that appears to not be the case when there is resource pooling? Can someone fill in the details here?

@lagnat

@lagnat the original issue was fixed by #34439. If you have a reproducer for the issue you've faced, could you describe it in a new GH issue?

@puzpuzpuz

@puzpuzpuz I don't have a reproducer. My concern is mainly about This is very much expected behavior of CLS storage. Any sort of pooling would result in similar issues. What Im tying to determine is if what I'm seeing has anything to do with the SQL connection pools that Im using or some implementation detail of nests/expressjs. I would like to understand where the pitfall is with ALS and pooling and I can go from there. Might end up that I make a new issue or perhaps it's something that I can handle myself in user code.

@puzpuzpuz

@puzpuzpuz I don't have a reproducer. My concern is mainly about This is very much expected behavior of CLS storage. Any sort of pooling would result in similar issues.

@lagnat I see.

Then I would say that whether you face a similar (context loss) issue really depends on how pooling (or queuing) is implemented and exposed in the public API. Say, if operations in a driver are queued and then sent over a single TCP socket, there might be ALS context loss because of that. But if those operations are exposed via a Promise-based API, there will be no issues.

If you have a concrete scenario where things go wrong, I can try to advice how to deal with it. We also have this short troubleshooting guide in the doc which might be helpful in case of context loss issues: https://nodejs.org/api/async_hooks.html#async_hooks_troubleshooting

async_hooks: don't reuse resource in HttpAgent when queued

Navigation Menu

Repository navigation

AsyncLocalStorage does not work with http.Agent #34401

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AsyncLocalStorage does not work with http.Agent #34401

Description

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

Activity

Flarna commented on Jul 16, 2020

indutny commented on Jul 16, 2020

indutny commented on Jul 16, 2020

Flarna commented on Jul 16, 2020

indutny commented on Jul 16, 2020

rjharmon commented on Jul 16, 2020

indutny commented on Jul 16, 2020

chrisdickinson commented on Jul 16, 2020

rjharmon commented on Jul 16, 2020

indutny commented on Jul 16, 2020

indutny commented on Jul 16, 2020

Qard commented on Jul 16, 2020

indutny commented on Jul 17, 2020

Qard commented on Jul 17, 2020

11 remaining items

indutny commented on Jul 17, 2020

ghermeto commented on Jul 17, 2020

vdeturckheim commented on Jul 18, 2020

puzpuzpuz commented on Jul 18, 2020

Qard commented on Jul 19, 2020

puzpuzpuz commented on Jul 19, 2020

Qard commented on Jul 19, 2020

lagnat commented on Nov 19, 2020

puzpuzpuz commented on Nov 19, 2020

lagnat commented on Nov 19, 2020

puzpuzpuz commented on Nov 19, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions