Node-API Conformance Test Suite (node-api-cts) ☔ #15

As Node-API is gaining popularity as a de-facto standard for ABI-stable access to JS engines and runtimes, I'd like to propose a project, implementing a pure ECMAScript Conformance Test Suite (CTS) for implementors of Node-API to exercise their implementations.

Why?

The Node.js project already maintains tests for Node-API in test/js-native-api and test/node-api. However, implementors of Node-API aren't necessarily implementing Node.js specific runtime APIs / built-ins modules (such as the process global, CommonJS require function or node:fs, node:path, etc.) and running these existing tests require polyfills for the Node.js runtime APIs or outright forking the tests - neither is ideal when when you want to downstream fixes and enhancements to these tests and the Node-API spec in general.

Goals

The Conformance Test Suite should:

1. Be maintained on the Node.js org (ex https://github.com/nodejs/node-api-cts), to affirm legitimacy and increase discoverability
2. Not use Node.js specific runtime APIs, as pure ECMAScript is needed for external implementors to use this effectively.
3. Target all of Node-API (both the engine and runtime specific functions).
4. Offer a way to gradually opt into / allow-list tests, to facilitate a gradual implementation for new implementors.
5. Drive the build of native addons while (optionally) delegating to the implementer to
   1. generate build system configuration files (such a CMakeList.txt for cmake-js)
   2. provide the command needed for building individual the Node-API addons
6. Implement each test as a separate file (like "Test262: ECMAScript Test Suite")
7. Rely on the implementor to provide a harness pre-loaded with helpers in globals, including:
   1. an assert implementation, provided to the harness by the CTS.
   2. a function to use to load an addon, since require and process.dlopen aren't documented as required (nor always possible to provide) for an implementor of Node-API.
8. Publish as a package on NPM with prebuilt addons, to make it trivial for implementors to run these tests as part of their own CI infrastructure.
9. Be have tests implemented
   1. in JavaScript and without use of a module system and instead rely on globals being provided by the harness
   2. using eslint to ensure limited globals are accessed and only known language features are used.
10. Use TypeScript to implement the any build scripts / CLI.

Non-goals

We don't want to:

1. Sync the contents of test/js-native-api in nodejs/node into the CTS tests, since these would require a transformations of Node.js runtime APIs and we expect this to be cumbersome / error-prone with little practical value (as Node-API isn't changed much and most changes to the tests will be capturing regressions or when adding new functions to Node-API).
2. Have the CTS export functions to inject TDD functions (like describe and it), to enable use of test tooling / reporting already known / maintained by the implementor nor do we want to export an object with "suites and tests" that a implementor-specific test harness can import and call as they see fit. Both would entail relying on a module system.

Open questions

1. Should we use TypeScript for the tests as well? This was a part of my initial proposal and I got some push-back on this and suggested using eslint instead, since it's simpler and doesn't force a "build step".
2. Would it be interesting to build out a public dashboard to track the conformance of implementors? Like https://wpt.fyi/ (maintained by The web-platform-tests Project).

Phases

1. Harness and a single test file (🟢 done)

I want to propose starting with a harness and a single test file to show off the approach before we get too deep into the implementation of tests.

Before progressing to the next phase, we want to ensure that it harness will be runnable by most (if not all) known implementors.

2. Start implementing the tests (🟡 in progress)

We'll be porting the tests from the upstream Node.js test suite into the CTS. I imagine many of these files could actually be written by an LLM with the prior art as context.

3. Upstream into Node.js main repository (optional side quest)

A stretch goal of this project would be to upstream the Node-API CTS into the main Node.js repository, by gradually introducing changes to the existing test/js-native-api and test/node-api test directories moving them closer to the ideal outlined in the phase 1.

Prior art:

• nodejs/js-native-api-test repo (see goals in nodejs/js-native-api-test/issues/7).
  Focussed only on the engine-specific part of Node-API and the repo is now archived.
• test/js-native-api in nodejs/node
• test/node-api in nodejs/node
• tests/napi in denoland/deno
• test/napi in oven-sh/bun
• unittests/NodeApi and specifically unittests/NodeApi/node_lite.cpp implemented by @vmoroz for the Hermes process, providing the minimal Node.js runtime polyfill needed for the existing Node.js tests against the Hermes engine.
• packages/node-tests in callstackincubator/react-native-node-api
• packages/test in toyobayashi/emnapi
• Test262: ECMAScript Test Suite could serve as inspiration.
• The WebGPU CTS could serve as inspiration: Tests and harness is written in TypeScript and it expects implementors to be able to interpret TS (using ts-node internally)
• The web-platform-tests Project could serve as inspiration.