Merging regression suites from engines #153

Activity

1. eqrion commented on Mar 20, 2026

   

   eqrion

   on Mar 20, 2026

   More actions

   I think this broadly makes sense and I support it.

   Regression tests are often times odd and have a different flavor than normal hand-written tests. I also could imagine we'll end up with a lot of them (sounds like you already have thousands!). So I'd lean towards a dedicated subfolder for them.

   We currently have automation that exports the spec tests her to WPT. We would need to be sure that WPT is okay with suddenly getting thousands of new tests. We could probably exclude them from the automation if that's an issue (SpiderMonkey directly imports the spec tests, not through WPT)

   Do you have a review policy in mind? Something lightweight would be nice.

   Also we don't have any regression tests to contribute because our regression tests are all written in JS using the wasm JS-API. We maybe could change that culturally, but that's what we do right now.
2. tlively commented on Mar 20, 2026

   

   tlively

   on Mar 20, 2026

   Member

   More actions

   V8 doesn't use .wast tests, either. Binaryen does use .wast tests, but they don't have assertions, so it's unclear how useful they would be to other projects. (They instead use FileCheck from LLVM to check the output of running the input modules through Binaryen tools against expectations that live in comments alongside the input.)
3. titzer commented on Mar 21, 2026

   

   titzer

   on Mar 21, 2026

   Author

   More actions

   @eqrion, yeah I think it makes sense to have subfolders. E.g. having a folder per engine contributor could make it easier to push/pull in tests from different engines periodically. From skimming the wasmtime code, it has a mix of internal tests written in Rust, API tests in .wat, a copy of the spec testsuite, and some of its own .wast regression tests.

   @tlively I'm wondering if the Binaryen .wast tests are runnable. Are the expectations of static transformations, or do they execute the programs?
4. alexcrichton commented on Mar 22, 2026

   

   alexcrichton

   on Mar 22, 2026

   Contributor

   More actions

   Personally I'd love to be able to merge regression test suites across engines. For me though a critical part of that would be automation, both automatically pulling in tests into this repo and then pushing them back down to other repos. While I agree manually pushing tests would be a good incremental step I'd fear things would get stale pretty quickly.

   I'd agree that there could be a worry that some tests that accidentally test engine-specific behavior that may or may not be spec-compliant, but I'd expect that to get loosely sorted out over time. Consumers of this test suite would likely need some sort of list of allowed and denied tests anyway so such tests could be lumped into there.

   Given the spec tests are already in *.wast I'd naively expect engine-specific tests to also be in that format, but I understand that not all engines use this format. Another possible alternative, along the lines of #119, would be to use a different format if the *.wast format is the sticking point. If the sticking point is just moving the tests to a declarative format at all, however, then that's likely going to be difficult to overcome.
5. redianthus commented on Mar 22, 2026

   

   redianthus

   on Mar 22, 2026

   More actions

   As a datapoint, we do use .wast files in Owi and we are using this repo as a git submodule for our tests.
6. rossberg commented on Mar 23, 2026

   

   rossberg

   on Mar 23, 2026

   Member

   More actions

   Is there any particular reason not to add .wast tests to the upstream spec repo directly, in subfolders or otherwise?

   That would also preclude any accidental merging of non-compliant tests, since spec CI runs all tests against the reference interpreter.

   I honestly do not understand what would be gained by introducing a different format. All engines already need to be able to execute .wast to run the existing test suite (be it directly, converted to .bin.wast, or converted to .js). Introducing a new format would just shift the burden to also support that one to either a new shared conversion tool, or to every individual engine/tool maker.
7. eqrion commented on Mar 23, 2026

   

   eqrion

   on Mar 23, 2026

   More actions

   Is there any particular reason not to add .wast tests to the upstream spec repo directly, in subfolders or otherwise?

   Speaking for myself, automation would lower the barrier to make this more worthwhile. I think there's a similar precedent with WPTs. It's much more convenient for engineers to just make one review/PR to their local codebase instead of having to duplicate it upstream later.

   That would also preclude any accidental merging of non-compliant tests, since spec CI runs all tests against the reference interpreter.

   Yeah that's tough. Another related issue is how to separate out regression tests for in-progress proposals. e.g. If we commit a 'stack-switching' regression test, it shouldn't get merged up here until stack-switching is merged.

   I honestly do not understand what would be gained by introducing a different format. All engines already need to be able to execute .wast to run the existing test suite (be it directly, converted to .bin.wast, or converted to .js). Introducing a new format would just shift the burden to also support that one to either a new shared conversion tool, or to every individual engine/tool maker.

   I don't have an opinion on a new format. At least for SpiderMonkey, we write our tests using the JS-API + 'wasmTextToBinary' because it lets us generate wasm modules programmatically to cover more cases (e.g. all valtypes) in a concise manner.

   Our simple regression tests though could be written in wast, but we don't have a wast runner in SM and need to use an offline tool to convert them to a JS file which can be checked in. We probably could add a wast runner into SM and start writing regression tests in wast. We just haven't had a reason to do that work yet.
8. titzer commented on Mar 23, 2026

   

   titzer

   on Mar 23, 2026

   Last edited by titzer

   Author

   More actions

   @rossberg I'd be OK pushing Wizard's tests upstream to spec in principle, but we generally prefer micro-tests rather than per-instruction combination files, unlike the current convention in the spec repo. E.g. there are thousands of Wizard tests that are individually very small. They're valuable to us to be split up this way as they are easier to debug.
9. tlively commented on Mar 23, 2026

   

   tlively

   on Mar 23, 2026

   Member

   More actions

   @tlively I'm wondering if the Binaryen .wast tests are runnable. Are the expectations of static transformations, or do they execute the programs?

   A few are meant to be executed (these live in https://github.com/WebAssembly/binaryen/tree/main/test/lit/exec), but that vast majority are just checking static transformations.
10. rossberg commented on Mar 24, 2026

    

    rossberg

    on Mar 24, 2026

    Member

    More actions

    @titzer, I think that's okay. We can organise secondary tests coming from engines in their own dedicated compartment of the suite, where they can follow their own conventions.
11. alexcrichton commented on Apr 6, 2026

    

    alexcrichton

    on Apr 6, 2026

    Contributor

    More actions

    @titzer I was curious so I tried a bit just now to get Wizard's *.wast tests running in Wasmtime, and I find it instructive insofar as it's an indicator of some pain points we'd in theory want to plan for in merging test suites from engines:

    • For 128mems.wast Wasmtime has a hardcoded limit of 100. Easy to change, but pushing the limits of things, especially before a proposal is finalized, can sometimes cause a good bit of churn.
    • For v128_extract_lane.wast Wasmtime has a slightly different error message. I believe it's expected that errors are possibly different across engines, and we e.g. don't validate any error messages from the upstream spec test suite, but it's someting downstream integrations will need to handle.
    • For memory_copy1.wast Wasmtime (and wasm-tools) currently reject an empty *.wast file. I'm not sure what's expected either way, but there's minor parsing differences between the two. For example currently wasm-tools/Wasmtime requires $foo in (module instance $foo $foo), but Wizard liberally uses (module instance). Easy to fix, however, and not an ongoing cost but just a one-time thing.
    • For cast_poly00.bin.wast this test fails in Wasmtime right now. It's difficult to debug a binary-format test (as opposed to a text-format test), and I think the difference here boils down to Wizard expecting an "unreachable" trap and Wasmtime instead raises "integer overflow" as a trap. My guess is that this is just trap error messages, but I'm not entirely sure myself.
    • For spec_bug.wast Wasmtime currently doesn't validate that module, and my understanding of the current Explainer.md is that the module should be invalid, but I'm also no stack-switching expert. This is a possible indicator of churn in proposals, however.
    • For add0def.bin.wast it currently has a (module definition binary ...) but no (module instance ...) anywhere. I think this may be a bug in Wizard, but I've misinterpreted the *.wast script format before. In general though I've found it's easy to get *.wast parsing wrong myself, and I tend to not know what the right way for something is and I just try to get tests passing.
    • I had to do some golfing to figure out which wasm proposals were enabled for which tests. This is a more general problem than just merging test suites, but for example tag0.wast asserts that tags can't have results, but that equates to asserting stack-switching is disabled (since that proposal relaxes vaildation). It's easy enough to hardcode things (which is what we've done for upstream spec tests), but this can also get out of hand relatively quickly. Wasmtime has an in-test format for specifying this, but it's not something I would say should be standardized. Basically I expect it to be a pain point, though.

    I mostly just wanted to write this all down as I think the friction can be illustrative of the kinds of problems that arise here, but I also want to be clear that I'm personally still very much in favor of somewhere there should be a giant soup of *.wast tests that all engines should strive to pass. I still think it'll be somewhat nontrivial to wrangle them all, but ultimately worth it in the long run.
12. rossberg commented on Apr 7, 2026

    

    rossberg

    on Apr 7, 2026

    Member

    More actions

    Interesting list! Wrt syntactic issues, the .wast format is specified here:

    https://github.com/WebAssembly/spec/tree/main/interpreter#scripts
    

    (Where it leaves doubts, running the reference interpreter can usually answer the rest.)

    With that:

    • For memory_copy1.wast Wasmtime (and wasm-tools) currently reject an empty *.wast file. I'm not sure what's expected either way, but there's minor parsing differences between the two.

    Empty script files are allowed.

    • For example currently wasm-tools/Wasmtime requires $foo in (module instance $foo $foo), but Wizard liberally uses (module instance). Easy to fix, however, and not an ongoing cost but just a one-time thing.

    Def sites and use sites of instance and module names can always be abbreviated to empty (cf. invoke, register), including here.

    • For add0def.bin.wast it currently has a (module definition binary ...) but no (module instance ...) anywhere.

    Agreed that this test looks bogus.
13. titzer commented on Apr 13, 2026

    

    titzer

    on Apr 13, 2026

    Author

    More actions

    @alexcrichton

    Thanks for taking a look. There are a couple Wizard tests that need to be adjusted; I noticed also some minor divergence with the error messages that the spec interpreter generates. Normally, when we check in a .wast, we check in the .bin.wast that's generated by the reference interpreter for that proposal at the time. We do that so that at least for internal tests, testing Wizard doesn't require cloning and building the spec interpreter. Once in a while the spec interpreter is temporarily broken or changes error messages slightly in normal refactoring. We should periodically make sure all of Wizard's checked-in tests pass on the reference interpreter too. That actually would be a good part of automating upstreaming.

    As for managing flags, Wizard's handling of proposals was in the past more-or-less "1.0 + flags enable extensions", and the script that runs the regression tests would automatically enable extensions based on the test/regress/ext:proposal directory name. Internally, Wizard tracks dependencies for proposal as implications on extension flags. Since Wasm 3.0 was merged though, Wizard defaults to "3.0 + flags enable proposals" and the extension flags for pre-3.0 flags don't do anything. With profiles down the line, we might rethink this. In the interim, it's sometimes useful to check Wizard isn't getting too fat by having slimmer builds that disable certain feature sets with build flags. In fact there's a bunch of build configurations now, so we'll have to sort this out more nicely in the future.

    For tests that are checked in that expect to fail on the current spec but pass (or fail a different way) on a proposal (e.g. reserved bits or a zero byte), we probably need to think about how to assert a proposal/extension is not enabled. It's good if such tests are split off in a different file from other tests that should always pass or fail regardless of future extensions (e.g. type errors).