Invalidate cache when using import

How do I invalidate the cache of import?

I have a function that installs missing modules when an import fails, but the import statement seems to preserve the failure while the script is still running.

import('some-module').catch(
  // this catch will only be reached the first time the script is run because resolveMissingModule will successfully install the module
  () => resolveMissingModule('some-module').then(
    // again, this will only be reached once, but it will fail, because the import seems to have cached the previous failure
    () => import('some-module')
  )
)

The only information I found in regards to import caching was this documentation, which does not tell me where the “separate cache” used by import can be found.

No require.cache

require.cache is not used by import. It has a separate cache.
— https://nodejs.org/api/esm.html#esm_no_require_cache

the import cache is purposely unexposed. adding a query has been the generally accepted ecosystem practice to re-import something.

however, a failure to import something will not fill the cache.

this trivial program works fine for me (assuming nope.mjs does not exist):

import fs from 'fs';

import('./nope.mjs')
  .catch(() => fs.writeFileSync('./nope.mjs'))
  .then(() => import('./nope.mjs'))
  .then(console.log);

@devsnek

@devsnek, hmm, might this be limited to imports that use node_modules? This similarly trivial program fails for me the first time, but not the second.

import child_process from 'child_process';

import('color-names')
  .catch(() => child_process.execSync('npm install --no-save color-names'))
  .then(() => import('color-names'))
  .then(console.log);

Note that the JS spec requires imports to be deterministic/idempotent on a source text. Exposure of a cache would not allow you to fix the code above.

…

On Fri, Apr 5, 2019, 12:01 PM Gus Caplan ***@***.***> wrote: the import cache is purposely unexposed. adding a query has been the generally accepted ecosystem practice to re-import something. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#49442>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOUo5N5tioTq2s_t431OrihH_wH8Qagks5vd4FSgaJpZM4cfV4g> .

if its just happening with node_modules it could be #26926

can this be closd?

I think a use case like this would hopefully be implemented as a loader. Do we already track this as a use case in that context?

@jkrems we have old documents with that as a feature, but no success criteria examples.

FYI, I'm implementing ESM support in Mocha (mochajs/mocha#4038), and cannot currently implement "watch mode", whereby Mocha watches the test files, and reruns them when they change. So "watch mode" in Mocha, in the first iteration, will probably not support ESM, which is a bummer.

While we could use cache busting query parameters, that would mean that we are always increasing memory usage, and old and never-to-be-used versions of the file will continue staying in memory due to the cache holding on to them.

And I'm not sure a loader would help here, as the loader also has no access to the cache.

An API for unloading modules certainly makes sense. Usually with a direct registry API there is the tracing issue. An API that handles dependency removal can be useful. A simple API might be something like - import { unload } from ‘module’; unload(import.meta.url); // returns true Where the unload function would remove that module including all its dependencies from the registry. If in a cycle the whole cycle would be removed. A subsequent module load would refresh all the loads anew. Other problems to ensure work out is what if modules in the tree are still in-progress. I’d be tempted to say it should fail for that case and only work when all modules have either errored or completed. We still have memory leak concerns as v8 doesn’t lend itself easily to module GC still. But Node.js can lead the way here as it should. It will be an ongoing process to get there, but the API can come first. The main questions then seem to be: * are we ok exposing this as a module or should it be tied to loaders? If tied to loaders how would userland code request this? Or don’t we want it to - as in Mocha should run a new context with a loader? * should it be a direct registry API (get/set) with tracing, or should it be a deep API like the example above * finally, ironing out the partially loaded tree edge cases

…

On Tue, Nov 26, 2019 at 00:09 Gil Tayar ***@***.***> wrote: FYI, I'm implementing ESM support in Mocha (mochajs/mocha#4038 <mochajs/mocha#4038>), and cannot currently implement "watch mode", whereby Mocha watches the test files, and reruns them when they change. So "watch mode" in Mocha, in the first iteration, will probably not support ESM, which is a bummer. While we *could* use cache busting query parameters, that would mean that we are always increasing memory usage, and old and never-to-be-used versions of the file will continue staying in memory due to the cache holding on to them. And I'm not sure a loader would help here, as the loader also has no access to the cache. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#49442?email_source=notifications&email_token=AAESFSTBSWVLXPTWDYZOHSDQVSVQRA5CNFSM4HD5LYQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFEXNDI#issuecomment-558462605>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAESFSWTJXGB4J6WWNMOLRDQVSVQRANCNFSM4HD5LYQA> .

I'm really not a fan of the idea of our module cache being anything except insert-only. CJS cache modification is bad already, and CJS modules don't even form graphs.

Additionally, other runtimes (like browsers) will never expose this functionality, so some alternative system will have to be used for them regardless of what node does, in which case it seems like that system could just be used for node.

@giltayar

@giltayar have you looked into using Workers or other solutions to have a module cache that you can destroy (such as by killing the Worker)?

@bmeck

@bmeck - interesting. That would mean that the tests themselves run in Workers. While I am theoretically familiar with workers, I haven't yet had any experience with them: is any code that runs in the main process compatible with worker inside a worker? In other words, compatibility-wise, would all test code that works today in the "main process" work inside workers?

I wouldn't want Mocha to have a version (even a semver-major breaking one) where developers will need to tweak their code because now it's running inside a worker. I'm guessing that there's a vast amount of that code running inside Mocha, and any incompatibility would be a deal breaker.

there are differences between workers and the main thread, mostly surrounding the functions on process, like process.exit() in a worker doesn't end the process, just the thread. There's a good list here: https://nodejs.org/api/worker_threads.html#worker_threads_class_worker

Looking at the list, I can see process.chdir() is not available, which is probably a deal breaker in many tests (unit tests probably don't use process.chdir(), but Mocha is used for all sorts of tests), as is breaking some native add-ons (although I'm not sure how big of a problem this is in the real world).

I would hesitate to say this, as my only contribution to Mocha currently is this pull request, but I would guess that the owners would veto this. Or maybe allow this only if we add a --run-in-workers option. In any case, without looking too much at the code, this is probably a significant investment to implement for supporting ES Modules, as this is not a simple refactor, but rather an architectural change in how Mocha works.

If it wasn't apparent from the above, I believe I would still prefer a "module unloading" API, unless the working group is adamant and official about not having one, of course. Which would probably mean going the "subprocess"/"worker" route.

Should probably close this as "wont do"?

My example

import fs from 'node:fs';
import { isBuiltin } from 'node:module';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import {
    createContext,
    type Module,
    type ModuleLinker,
    SourceTextModule,
    SyntheticModule
} from 'node:vm';

const ROOT_MODULE = '__root_module__';

const link: ModuleLinker = async (specifier: string, referrer: Module) => {
    // Node.js native module
    const isNative = isBuiltin(specifier);
    // node_modules
    const isNodeModules =
        !isNative && !specifier.startsWith('./') && !specifier.startsWith('/');
    if (isNative || isNodeModules) {
        const nodeModule = await import(specifier);
        const keys = Object.keys(nodeModule);
        const module = new SyntheticModule(
            keys,
            function () {
                keys.forEach((key) => {
                    this.setExport(key, nodeModule[key]);
                });
            },
            {
                identifier: specifier,
                context: referrer.context
            }
        );
        await module.link(link);
        await module.evaluate();
        return module;
    } else {
        const dir =
            referrer.identifier === ROOT_MODULE
                ? import.meta.dirname
                : path.dirname(referrer.identifier);
        const filename = path.resolve(dir, specifier);
        const text = fs.readFileSync(filename, 'utf-8');
        const module = new SourceTextModule(text, {
            initializeImportMeta,
            identifier: specifier,
            context: referrer.context,
            // @ts-expect-error
            importModuleDynamically: link
        });
        await module.link(link);
        await module.evaluate();

        return module;
    }
};

export async function importEsm(identifier: string): Promise<any> {
    const context = createContext({
        console,
        process,
        [ROOT_MODULE]: {}
    });
    const module = new SourceTextModule(
        `import * as root from '${identifier}';
        ${ROOT_MODULE} = root;`,
        {
            identifier: ROOT_MODULE,
            context
        }
    );
    await module.link(link);
    await module.evaluate();
    return context[ROOT_MODULE];
}

function initializeImportMeta(meta: ImportMeta, module: SourceTextModule) {
    meta.filename = import.meta.resolve(module.identifier, import.meta.url);
    meta.dirname = path.dirname(meta.filename);
    meta.resolve = import.meta.resolve;
    meta.url = fileURLToPath(meta.filename);
}

Use it

const module = await importEsm('filename');

If you are interested, we created Hot Hook to hot reload node imports during development.

https://adonisjs.com/blog/hmr-in-adonisjs
https://docs.adonisjs.com/guides/concepts/hot-module-replacement
https://github.com/julien-R44/hot-hook

Hi, the ability to invalidate cache when using import (or in my case even require) seems to be very important to be able to use the new experimental mock.module functionality from the native Node test runner.

https://nodejs.org/api/test.html#mockmodulespecifier-options

Here is a use-case, where you can run into a trouble.

fs-extended.mjs

import fs from 'node:fs';

function getFs() {
    return fs;
}

export { getFs };

my.test.mjs

import assert from 'node:assert';
import { mock, test } from 'node:test';

// This works well
test('Mock shallow module', async () => {
    for (let i = 0; i < 2; i++) {
        const fsMock = mock.module('node:fs', { namedExports: { writeFileSync: mock.fn(() => i) } });

        const fs = await import('node:fs');

        assert.strictEqual(fs.writeFileSync(), i);

        fsMock.restore();
    }
});

// This fails
test('Mock deep module', async () => {
    for (let i = 0; i < 2; i++) {
        const fsMock = mock.module('node:fs', { namedExports: { writeFileSync: mock.fn(() => i) } });

        const { getFs } = await import('./fs-extended.mjs');
        const fs = getFs();

        assert.strictEqual(fs.writeFileSync(), i);

        fsMock.restore();
    }
});

Result:

test at my.test.mjs:16:1
✖ Mock deep module (4.481709ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
  
  0 !== 1
  
      at TestContext.<anonymous> (file:///Users/filip.satek/git/cns-cli/my.test.mjs:23:16)
      at async Test.run (node:internal/test_runner/test:931:9)
      at async Test.processPendingSubtests (node:internal/test_runner/test:629:7) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: 0,
    expected: 1,
    operator: 'strictEqual'
  }

Even though I called fsMock.restore(), in the second iteration, the getFs call still gave me the node:fs module from the first iteration.

I hope this is related to this issue, otherwise please let me know and I will file a new issue for my use-case.

Edit: Tested on Node v22.9.0 using node --test --experimental-test-module-mocks my.test.js command.

I'm also running into issues with this with the native module mocking, where I've setup the mock only in one test & already imported the module under test at the top.

tested-module.js

import fs from "node:fs/promises";

export default MyClass {
	/* ... */
}

my-test.test.js

import { describe, it } from "node:test";
import MyClass from "./tested-module.js";

function setup() {
	// ...

	return new MyClass(...);
}

describe("myModule", () => {
	it("should run test1", () => { /* ... */ });
	it("should run test2", (t) => {
		t.mock.module("node:fs/promises", {
			defaultExport: {
				writeFile: t.mock.fn(),
			}
		});

		// This is mocked correctly because I haven't imported the original module
		const mockedFs = (await import("node:fs/promises")).default;		

		// This still calls the real fs methods because I'd already imported it at the top.
		const MyMockedClass = (await import("./tested-module.js")).default;
	});
})

I'm experiencing the same issue: Mocks from older tests seem to be reused in subsequent ones. Doing await import('./repro.mjs?a='+randomUUID()) doesn't help either.

Here's a simple reproduction:

repro.mjs

import {readFile} from 'node:fs/promises'

export const something = async () => readFile('something')

repro.test.mjs

import {
  describe, it,
} from 'node:test'
import assert from 'node:assert'

describe('repro', () => {
  it('test 1', async t => {
    const readFile = t.mock.fn(async () => t.name)
    t.mock.module('node:fs/promises', {
      namedExports: {
        readFile,
      },
    })
    const {something} = await import('./repro.mjs')
    assert.equal(await something(), 'test 1')
  })

  it('test 2', async t => {
    const readFile = t.mock.fn(async () => t.name)
    t.mock.module('node:fs/promises', {
      namedExports: {
        readFile,
      },
    })
    const {something} = await import('./repro.mjs')
    assert.equal(await something(), 'test 2')
  })
})

@Filipoliko

@Filipoliko it looks like your comment may be related to #59163 , it should be possible to reset module mocking but it doesn't seem to actually work.

I created a library using module.registerHooks: https://github.com/sxzz/import-without-cache. It allows you to load modules and their submodules without caching, which I hope it will be helpful. Please note that there are some known limitations.

Just a heads up there is a PR up for this now - #61767 ! 🙌

This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

Please do not make this issue as stale! Would love to see this issue resolved in node 🙏

Navigation Menu

Repository navigation

Invalidate cache when using import #49442

No require.cache

135 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Invalidate cache when using import #49442

Description

No require.cache

Activity

devsnek commented on Apr 5, 2019

jonathantneal commented on Apr 5, 2019

bmeck commented on Apr 5, 2019

devsnek commented on Apr 5, 2019

MylesBorins commented on Aug 29, 2019

hybrist commented on Aug 29, 2019

bmeck commented on Aug 30, 2019

giltayar commented on Nov 26, 2019

guybedford commented on Nov 26, 2019

devsnek commented on Nov 26, 2019

bmeck commented on Nov 26, 2019

giltayar commented on Nov 28, 2019

devsnek commented on Nov 28, 2019

giltayar commented on Nov 28, 2019

giltayar commented on Nov 28, 2019

135 remaining items

simlu commented on Jul 17, 2024

lzxb commented on Aug 14, 2024

RomainLanz commented on Aug 14, 2024

Filipoliko commented on Oct 4, 2024

ramblingenzyme commented on Jan 28, 2025

joshbalfour commented on Mar 18, 2025

thw0rted commented on Aug 1, 2025

sxzz commented on Nov 29, 2025

thescientist13 commented on Mar 1, 2026

github-actions commented on Aug 9, 2026

filmaj commented on Aug 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions