`.env` file support issue tracker

[anonrig]

This is a follow-up of dotenv support pr to track the development process, and answer some questions that require a general discussion for making the decision to implement them.

Todos:

• Precisely define the syntax of .env files, and consider a better format not based on ini
• Supporting absolute paths for .env file src: do not coerce dotenv paths #51425
• Should we support multiple --env-file through the CLI? (cc @GeoffreyBooth) src: support multiple --env-file declarations #49542
• Support multi-line values src: support multi-line values for .env file #51289
• Follow up after @tniessen's PR has landed to main test: improve UV_THREADPOOL_SIZE tests on .env #49213
• Should we add a programmatic API? If yes, what would be the use-case? (cc @cjihrig) src: add process.loadEnvFile and util.parseEnv #51476

Questions:

• Should we merge NODE_OPTIONS if both environment variable and .env version exist? Currently, we prefer environment variable over .env configuration. (cc @ljharb) src: merge env-file and env vars of NODE_OPTIONS #49217
• Should we throw if .env file does not exist? We currently follow the default behavior of dotenv package and do not throw. (cc @MoLow)

Please feel free to edit this issue in case there are more questions that needs to be answered.

cc @nodejs/tsc

[benjamingr]

I vote "throw on non-existing env file" and "throw on multiple --env-file parameters" since those are easy to change later and lets us land without waiting for a decision on both.

[gireeshpunathil]

IMO:

Should we merge NODE_OPTIONS if both environment variable and .env version exist?

The usual practice (of environment variable precedence ordering) is command line overriding stored setting in files. That way, the end user who runs the code has the last say on the settings.

Should we support multiple --env-file through the CLI?

Yes. This will help co-existence of application and its dependencies with their own env definitions.

Should we add a programmatic API?

Yes. This will help embedded use cases where node.js is not started through the regular launcher with regular command line parsing sequences.

Should we throw if .env file does not exist?

Yes. This will help the feature function to be deterministic.

[tniessen]

Should we add a programmatic API?

Yes. This will help embedded use cases where node.js is not started through the regular launcher with regular command line parsing sequences.

I assume that this refers to a C++ API for embedders. This might be useful if the API can be used before any of Node.js or any of its dependencies are initialized. Otherwise, embedders will run into the same problem as we did with NODE_OPTIONS.

[gireeshpunathil]

I assume that this refers to a C++ API for embedders.

Yes, I meant that.

This might be useful if the API can be used before any of Node.js or any of its dependencies are initialized. Otherwise, embedders will run into the same problem as we did with NODE_OPTIONS.

can you please elaborate on the problem we did with NODE_OPTIONS? sorry, I am not up-to-date with that.

[tniessen]

What I mean is that environment variables that affect Node.js, its dependenices, the embedding application, or the embedder's dependencies must be set before the respective component attempts to use it. In many cases, that means before initialization of the respective component.

(On the other hand, reading environment variables from a file is simple enough, so if an application doesn't use the Node.js CLI, it probably might as well implement this feature itself.)

[GeoffreyBooth]

What I mean is that environment variables that affect Node.js, its dependenices, the embedding application, or the embedder’s dependencies must be set before the respective component attempts to use it. In many cases, that means before initialization of the respective component.

Yes. As in, it doesn’t do much good to write JavaScript code like process.env.NODE_OPTIONS = '...' because by the time this JavaScript code runs, Node has already loaded and configured itself and so most (all?) of what you would set by this point is too late to have any effect.

Re merging versus overwriting, I feel somewhat strongly that we should overwrite, because this .env file support is general purpose for any environment variable and not just Node-specific ones, and we can’t know how to merge variables like DATABASE_PASSWORD or whatever. I think it would be confusing UX if Node environment variables got merged but non-Node ones didn’t. I was assuming that whatever variables were set in the environment would take precedence over ones set in a loaded file, but if others feel strongly that it should be the reverse I don’t have a strong opinion on this part.

[ljharb]

I would assume that individual variables would overwrite but that if the env has A and the file has B that the process would end up with both A and B.

[anonrig]

I would assume that individual variables would overwrite but that if the env has A and the file has B that the process would end up with both A and B.

@ljharb This is the default behavior right now, but do not merge NODE_OPTIONS.

➜  node git:(dotenv-support) cat .env
TESTING_KEY=THIS_IS_VALUE
➜  node git:(dotenv-support) TESTING_KEY=OVERRIDEN ./out/Release/node -e "console.log(process.env.TESTING_KEY)"
OVERRIDEN

[ljharb]

ah ok, gotcha. then yeah i think #49148 (comment) makes sense because indeed it would get confusing figuring out which node options are mutually exclusive, which are single values, which are multiple values, etc.

[RomainLanz]

I was wondering, do you plan to support also something like dotenv-vault that lets you encrypt your secret and decrypt it just in time.

[anonrig]

I was wondering, do you plan to support also something like dotenv-vault that lets you encrypt your secret and decrypt it just in time.

Not at the moment, but contributions are welcome.

[philnash]

I was just seeing how this feature worked and I think that either the documentation or the feature is wrong in the 20.6.0 release.

The docs say:

If the same variable is defined in the environment and
in the file, the value from the environment takes precedence.

But, I checked out the v20.6.0-proposal branch, built the latest executable and tried the following:

ϟ echo $PHIL_VAR
what
node (4e4bb9f) 
ϟ ./node --version
v20.6.0
node (4e4bb9f) 
ϟ ./node -e "console.log(process.env.PHIL_VAR)"                
what
node (4e4bb9f) 
ϟ ./node -e "console.log(process.env.PHIL_VAR)" --env-file .env
hello
node (4e4bb9f) 
ϟ cat .env
PHIL_VAR=hello

From the documentation, I would have expected the PHIL_VAR variable set in the environment to always be the result. Instead, it seems that the .env file version of the PHIL_VAR variable over-rides it when it is used.

Does this need a quick documentation fix ("the value from the .env file takes precedence") or does the feature need fixing?

[GeoffreyBooth]

does the feature need fixing?

The feature needs fixing. The environment should take precedence over the file.

[mcollina]

The environment should take precedence over the file.

I don't think so. The whole point of env files is to override the local environment.