镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

ci: fix linux-gnu glibc regression and enforce glibc ≤ 2.28 compatibility - #6467

Merged
lukastaegert merged 4 commits into
masterfrom
fix-6461-glibc-runner
Aug 1, 2026
Merged

lukastaegert merged 4 commits into
masterfrom
fix-6461-glibc-runner

Conversation

@lukastaegert

@lukastaegert lukastaegert commented Jul 31, 2026 •

Copy link
Copy Markdown
Member

This PR contains:

  • bugfix
  • feature
  • refactor
  • documentation
  • other

Are tests included?

  • yes (bugfixes and features will not be merged without tests)
  • no — this PR only changes CI infrastructure, and the change is the test: the new "Check glibc symbol compatibility" build step. It was verified against real artifacts and in CI: it failed on this branch's x86_64-gnu build while the toolchain was degraded (requires GLIBC_2.34, exceeding the supported floor of 2.28), and passes now that the fix landed (Highest required GLIBC: 2.14).

Breaking Changes?

  • yes (breaking changes will not be merged unless absolutely necessary)
  • no

List any relevant issue numbers:

Description

Problem

@rollup/rollup-linux-x64-gnu@4.62.3 requires GLIBC_2.34 (pthread_getattr_np@GLIBC_2.32, fstat64/stat64@GLIBC_2.33, pthread_*@GLIBC_2.34) whereas 4.62.2 required only GLIBC_2.14 — breaking builds on RHEL 8 / CentOS 8-based images (2.28), Debian 11 bullseye (2.31), Amazon Linux 2, and AWS build environments (#6461).

Root cause (confirmed via CI logs from this branch)

The linux-gnu x64 build intends to use napi build --use-napi-cross, which downloads a manylinux-derived gcc cross toolchain (glibc 2.17 sysroot) and drives both C compilation and linking through it:

  1. The build jobs restore node_modules from an actions/cache snapshot. A snapshot missing the optional platform package @napi-rs/lzma-linux-x64-gnu got cached on master (npm optional-dependency bug, npm/cli#4828).
  2. With that binding missing, require('@napi-rs/cross-toolchain') throws inside napi build, and the CLI silently swallows it (debug-only warning "Pick cross toolchain failed") and continues without CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER, TARGET_CC, or TARGET_SYSROOT.
  3. Cargo then falls back to the host cc and links against the ubuntu-24.04 runner's glibc 2.39 — producing the GLIBC 2.34 floor.
  4. The poisoned snapshot self-propagated through the cache: its hidden lockfile claims the binding is installed, so npm install is a no-op, and even an explicit npm install <pkg> is reconciled away against the lying hidden lockfile. Verified in CI on this branch — npm printed "added 4 packages, removed 2 packages" and the binding was still absent afterwards.
  5. Since the build was green, nothing surfaced this: the degradation only became visible in the published artifact's symbol table (PR chore(deps): lock file maintenance #6451's lockfile refresh forced a cold rebuild and exposed it). Thanks to @yoyo837 for the precise bisect in GLIBC Issue since 4.62.3 update #6461.

Changes (two commits)

1. ci: ensure napi cross-toolchain binding is present for glibc-compatible builds

  • Pin @napi-rs/lzma-linux-x64-gnu as an explicit optionalDependency so it's reliably installed on linux-x64-glibc hosts (not devDependencies: a regular dependency on a platform-restricted package breaks installs elsewhere with EBADPLATFORM). Mirrors the existing @emnapi/* workaround philosophy.
  • After the regular node_modules restore/install flow in install-and-cache-node-deps (flow otherwise unchanged), ensure the binding is present on Linux/X64 runners: if missing, repair it at tarball level by downloading the exact pinned URL from package-lock.json and verifying its recorded integrity hash — bypassing npm's trust in the stale hidden lockfile. Fail loudly if it's still absent, so a degraded toolchain can never ship an incompatible artifact again.

2. ci: add glibc symbol compatibility check for linux-gnu builds

  • New Check glibc symbol compatibility step in the build job: runs objdump -T on every native *.node artifact built via the napi cross toolchain (excluding android/ohos/musl/wasm), extracts the highest required GLIBC_x.y, and fails the build if it exceeds 2.28 — the floor for the oldest commonly deployed targets reported in GLIBC Issue since 4.62.3 update #6461 (RHEL 8 / Amazon Linux 2 era).
  • riscv64gc and loongarch64 gnu targets are excluded with a documented note: they use classic apt cross-gcc toolchains whose glibc floor tracks the runner image (currently 2.34/2.36) rather than the napi cross toolchain; pinning them lower is a separate toolchain change.
  • The check inspects the artifact, not the runner, so it keeps working as ubuntu-latest evolves and cannot be masked by cache behavior.

Verification

  • Guard proven to catch the bug: this branch's first CI run (guard only, no fix) failed the x86_64-gnu build with requires GLIBC_2.34, exceeding the supported floor of 2.28.
  • Root cause proven in CI: same run's logs contain Pick cross toolchain failed … Cannot find module '@napi-rs/lzma-linux-x64-gnu' right after a node_modules cache restore.
  • Fix proven in CI: subsequent run — the repair step detected the missing binding in the poisoned restored snapshot, restored it from the pinned tarball (integrity-verified), the napi cross toolchain engaged, and the guard now reports Highest required GLIBC: 2.14 on the x86_64-gnu artifact.
  • Full CI green on the final branch, including the gnu matrix builds, tests on Node 18.20.0/24, coverage, additional tests, and the s390x smoke test.

Open follow-ups (not blocking)

  • Upstream: @napi-rs/cli treats a missing --use-napi-cross toolchain as a debug warning instead of a hard failure; worth fixing in napi-rs. Our guard and binding pin make this non-blocking for rollup.
  • riscv64/loongarch64 gnu floors (2.34/2.36) could be brought down via a sysroot-pinned toolchain if anyone needs it, but they were unaffected by this regression.
  • Threshold 2.28 vs strict 2.17 (manylinux2014): the napi-cross sysroot legitimately references a few symbols above 2.17 (e.g. statx@2.28, getrandom@2.25, log2@2.29 from libm); 2.28 covers every distro family reported in GLIBC Issue since 4.62.3 update #6461 while accommodating those.

@vercel

vercel Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
rollup Ready Ready Preview Jul 31, 2026 6:57pm

Request Review

@github-actions

github-actions Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Thank you for your contribution! ❤️

You can try out this pull request locally by installing Rollup via

npm install rollup/rollup#fix-6461-glibc-runner

Notice: Ensure you have installed the latest nightly Rust toolchain. If you haven't installed it yet, please see https://www.rust-lang.org/tools/install to learn how to download Rustup and install Rust.

or load it into the REPL:
https://rollup-h0y7zr2m8-rollup-js.vercel.app/repl/?pr=6467

@github-actions

github-actions Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Performance report

  • BUILD: 7168ms, 810 MB (-3%)
    • initialize: 0ms, 23.9 MB
    • generate module graph: 2747ms, 626 MB
      • generate ast: 1399ms, 615 MB
    • sort and bind modules: 408ms, 687 MB
    • mark included statements: 3956ms, 810 MB (-3%)
      • treeshaking pass 1: 2310ms, 817 MB
      • treeshaking pass 2: 460ms, 818 MB
      • treeshaking pass 3: 394ms, 808 MB (-3%)
      • treeshaking pass 4: 383ms, 814 MB (-2%)
      • treeshaking pass 5: 380ms, 810 MB (-3%)
  • GENERATE: 624ms, 862 MB (-3%)
    • initialize render: 0ms, 810 MB (-3%)
    • generate chunks: 40ms, 818 MB (-3%)
      • optimize chunks: 0ms, 810 MB (-4%)
    • render chunks: 569ms, 841 MB (-2%)
    • transform chunks: 13ms, 862 MB (-3%)
    • generate bundle: 0ms, 862 MB (-3%)

@lukastaegert lukastaegert changed the title ci: enforce glibc ≤ 2.28 compatibility for linux-gnu native builds ci: fix linux-gnu glibc regression and enforce glibc ≤ 2.28 compatibility Jul 31, 2026
@lukastaegert
lukastaegert force-pushed the fix-6461-glibc-runner branch from 0fba799 to 0d0d4c8 Compare July 31, 2026 06:53
@lukastaegert
lukastaegert force-pushed the fix-6461-glibc-runner branch 2 times, most recently from 6704851 to d3a6fd8 Compare July 31, 2026 07:02
…le builds

Root cause of #6461: the linux-gnu x64 build intends to use
--use-napi-cross (manylinux-derived gcc, glibc 2.17 sysroot), but the
node_modules cache contained a snapshot missing the optional platform
package @napi-rs/lzma-linux-x64-gnu (npm/cli#4828). When it is absent,
@napi-rs/cli cannot load @napi-rs/cross-toolchain, silently skips
setting CARGO_TARGET_*_LINKER / TARGET_CC / TARGET_SYSROOT, and cargo
links against the ubuntu-24.04 host glibc 2.39 instead - raising the
published artifact's glibc floor to 2.34. The snapshot's hidden
lockfile claims the binding is installed, so npm never restores it: a
plain npm install is a no-op, and even an explicit single-package
install is reconciled away against the lying hidden lockfile.

- Pin @napi-rs/lzma-linux-x64-gnu as an explicit optionalDependency so
  it is reliably installed on linux-x64-glibc hosts (it cannot be a
  regular dependency due to EBADPLATFORM on other platforms).
- After installing node_modules, ensure the binding is present on
  Linux/X64 runners: if missing, repair it at tarball level by
  downloading the exact pinned tarball URL from package-lock.json and
  verifying its recorded integrity hash - bypassing npm's trust in the
  stale hidden lockfile. Fail loudly if the binding is still absent, so
  a degraded toolchain can never ship an incompatible artifact again.
  The repair logic was validated locally end-to-end (download,
  integrity verification, extraction).
Rollup 4.62.3 regressed @rollup/rollup-linux-x64-gnu to require
GLIBC_2.34 (breaking systems on glibc 2.28-2.31: RHEL8, Debian 11,
Amazon Linux 2, AWS build environments). The regression was invisible
in CI because nothing inspects the built artifacts: the build was green
while the toolchain had silently degraded (missing cross-toolchain
binding, see previous commit).

Add a step to the build job that runs objdump on every linux-gnu
artifact built via the napi cross toolchain (excluding
android/ohos/musl/wasm), extracts the highest required GLIBC_x.y, and
fails the build if it exceeds 2.28 - the floor of the oldest commonly
deployed targets reported in #6461. The check inspects the artifact
rather than the runner, so it keeps working as ubuntu-latest evolves
and cannot be masked by any cache behavior.

riscv64 and loongarch64 are excluded: they use classic apt cross-gcc
toolchains whose glibc floor tracks the runner image (currently
2.34/2.36) rather than the napi cross toolchain; pinning them lower
needs a separate toolchain change.

Verified on this branch: with only this check applied, the x86_64 gnu
build failed with 'requires GLIBC_2.34, exceeding the supported floor'
and climbed to GLIBC 2.14 once the toolchain fix landed; locally the
check passes against the 4.62.2 artifact (max GLIBC 2.14).
@codecov

codecov Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.78%. Comparing base (9c2c58d) to head (778c036).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #6467   +/-   ##
=======================================
  Coverage   98.78%   98.78%           
=======================================
  Files         275      275           
  Lines       10822    10822           
  Branches     2887     2887           
=======================================
  Hits        10690    10690           
  Misses         89       89           
  Partials       43       43           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lukastaegert
lukastaegert marked this pull request as ready for review July 31, 2026 08:34
Copilot AI review requested due to automatic review settings July 31, 2026 08:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The new glibc compatibility check can currently succeed without checking any artifacts, which undermines the CI guarantee it is meant to provide.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR hardens the Linux GNU native build pipeline to prevent publishing .node artifacts that require a newer glibc than the intended compatibility floor, addressing the regression reported in #6461.

Changes:

  • Pins @napi-rs/lzma-linux-x64-gnu as an optionalDependency to improve reliability of napi cross-toolchain installs.
  • Adds a composite-action step that verifies/repairs the cross-toolchain platform binding after restoring cached node_modules.
  • Adds a CI step that inspects built .node artifacts and fails the build if the highest required GLIBC_x.y exceeds 2.28.
File summaries
File Description
package.json Adds an explicit optional dependency and documents why it’s required for glibc-safe builds.
package-lock.json Locks the added optional dependency and updates lock metadata accordingly.
.github/workflows/build-and-tests.yml Introduces a post-build glibc symbol/version compatibility gate for linux-gnu targets.
.github/actions/install-and-cache-node-deps/action.yml Adds a cache-poisoning mitigation that ensures the napi cross-toolchain binding is actually present.
Review details
  • Files reviewed: 3/4 changed files
  • Comments generated: 2
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread .github/workflows/build-and-tests.yml Resolved
lukastaegert and others added 2 commits July 31, 2026 20:55
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The new glibc guard can currently mask real objdump/pipeline failures (and under-parse some GLIBC versions), reducing the reliability of the compatibility check.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

.github/workflows/build-and-tests.yml:306

  • The max_glibc extraction currently appends || true to the whole pipeline, which can mask real failures (e.g. objdump missing/errored) and let the compatibility guard pass with an empty result. Also, the regex only matches GLIBC_x.y and would ignore patch-level versions like GLIBC_2.2.5 if they appear.
            echo "Checking $file for glibc symbol versions..."
            # Extract highest required GLIBC version from the binary
            max_glibc=$(objdump -T "$file" | grep -oE 'GLIBC_[0-9]+\.[0-9]+' | sed 's/GLIBC_//' | sort -Vu | tail -1 || true)

.github/actions/install-and-cache-node-deps/action.yml:52

  • The integrity verification error message is too generic (integrity mismatch) and makes it hard to debug what went wrong (wrong algo vs wrong hash, truncated download, etc.). Including the computed vs expected hash (and algo) will make CI failures actionable.
            const [algo, expected] = '$integrity'.split('-', 2);
            const actual = crypto.createHash(algo).update(fs.readFileSync('/tmp/lzma-binding.tgz')).digest('base64');
            if (actual !== expected) { console.error('integrity mismatch'); process.exit(1); }
  • Files reviewed: 3/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@lukastaegert
lukastaegert merged commit 7beedfa into master Aug 1, 2026
49 checks passed
@lukastaegert
lukastaegert deleted the fix-6461-glibc-runner branch August 1, 2026 04:55
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

This PR has been released as part of rollup@4.62.4. You can test it via npm install rollup.

lieranderl added a commit to lieranderl/aestheticlab that referenced this pull request Oct 2, 2026
@builder.io/qwik 1.20.1 requires rollup ^4.61.0, but the override still
forced 4.59.0. Pin 4.62.5, the latest 4.62 patch, which includes the
upstream glibc compatibility fix. Rollup 4.62.4+ adds an optional
@napi-rs/lzma-linux-x64-gnu dependency, verified against the upstream
package.json at the v4.62.4 tag (rollup/rollup#6467). Rollup is a build
dependency and is not installed in the runtime image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lieranderl added a commit to lieranderl/aestheticlab that referenced this pull request Oct 2, 2026
* chore(deps): bump Qwik to 1.20.1

Supersedes #208, which raised @builder.io/qwik-city to 1.20.1 but left
the @builder.io/qwik override at 1.20.0, mismatching the two packages.
Raise both dev dependencies and the override together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(deps): raise rollup override to satisfy Qwik 1.20.1

@builder.io/qwik 1.20.1 requires rollup ^4.61.0, but the override still
forced 4.59.0. Pin 4.62.5, the latest 4.62 patch, which includes the
upstream glibc compatibility fix. Rollup 4.62.4+ adds an optional
@napi-rs/lzma-linux-x64-gnu dependency, verified against the upstream
package.json at the v4.62.4 tag (rollup/rollup#6467). Rollup is a build
dependency and is not installed in the runtime image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
stephendolan added a commit to stephendolan/omnifocus-cli that referenced this pull request Oct 7, 2026
`bun audit` reported 74 advisories (1 critical, 25 high). Every runtime
one came through @modelcontextprotocol/sdk 1.25.1 and its express, hono,
and ajv dependencies. Raising the SDK floor to ^1.32.0 and re-resolving
the lockfile leaves one low, dev-only advisory: esbuild's dev server on
Windows, held at 0.27 by tsup's range and never run by builds.

The lockfile was regenerated with --minimum-release-age=259200, so it
installs nothing published in the last three days. That held the SDK at
1.32.0 (1.32.1 is docs-only) and oxlint at 1.86.0. rollup now pulls in
@napi-rs/lzma-linux-x64-gnu on Linux x64; it ships with provenance and
was added deliberately in rollup/rollup#6467.

Dependabot's npm_and_yarn ecosystem cannot update bun.lock, which failed
the Oct 1 run; it now uses the bun ecosystem with a three-day cooldown.
CI and release installs use --frozen-lockfile, so they build from the
reviewed lockfile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 778c0367 Deployed Jul 31, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GLIBC Issue since 4.62.3 update

2 participants