Repository navigation
Issue with musl after upgrading to v1.1.10.8 #692
Description
After updating our project to use snappy-java v1.1.10.8, some of our users reported an issue when starting the process:
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.compress(SnappyBlockCompressor.java:54) ~[teku-networking-eth2-25.9.0.jar:25.9.0]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.encode(SszSnappyEncoding.java:38) ~[teku-networking-eth2-25.9.0.jar:25.9.0]
at tech.pegasys.teku.networking.eth2.gossip.subnets.AttestationSubnetSubscriptions.lambda$gossip$0(AttestationSubnetSubscriptions.java:90) ~[teku-networking-eth2-25.9.0.jar:25.9.0]
at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
... 46 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.10-5a7a1634-b5ef-475e-89cd-db2863cbec50-libsnappyjava.so: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
After looking through v1.1.10.8 changelog, I noticed the following change: #632
This change introduced a new code path that checks for the existence of /lib/ld-musl-x86_64.so.1 (https://github.com/xerial/snappy-java/blob/main/src/main/java/org/xerial/snappy/OSInfo.java#L238-L244). However, it looks like during runtime, libsnappyjava.so is trying to load libc.musl-x86_64.so.1, and it fails.
I am not sure of the reason for this mismatch. It feels unintentional but maybe I am just misinformed. Either way, we have found a few workarounds.
- We can create a symlink
libc.musl-x86_64.so.1 -> /lib/ld-musl-x86_64.so.1; or - We can just completely uninstall
musl.
Workaround 1 seems a bit hacky so we won't be asking our users to do so. Workaround 2 isn't great either, because musl is a dependency of grafana, and most of our users run a grafana instance on their nodes as well to monitor our process.
We are going to downgrade snappy-java to v1.1.10.7 in the mean time, until we have more information on this issue.
Activity
lucassaldanha commentedon Sep 3, 2025
Tagging @xerial and @samyuh (the author of the PR) in case they can help with some extra context.
samyuh commentedon Sep 3, 2025
Hey @lucassaldanha
Thanks for pointing this out! It looks like that the exception being caught here should be more general and not specific.
I tested snappy with different images, however the catch block was changed after my commits here:
c3723e5
I can open a MR in the next few days to catch all the exceptions so we don't have a problem with this anymore if @xerial agrees.
lucassaldanha commentedon Sep 3, 2025
Hi @samyuh, thanks for your reply.
I am not sure if the exception handling is the actual cause of the issue. Otherwise, we would've seen either a different stack trace or returned 'false' if there was a 'SecurityException', and we would be on the old code path.
I think the issue is related to the fact we look for
/lib/ld-musl-x86_64.so.1but then try to loadlibc.musl-x86_64.so.1.Unfortunately, this is a bit outside my expertise, so I'm not sure how it is supposed to behave.
agawron007 commentedon Sep 3, 2025
We are facing an identical issue also caused by musl libraries installed together with Grafana.
Steps to reproduce:
Snappy is detecting musl architecture incorrectly. It is looking for
/lib/ld-musl-x86_64.so.1, but then it tries to link to /lib/libc.musl-x86_64.so.1 which has been installed in /lib/usr on non musl systems.One fix could be checking if both libraries are present to determine if it is a musl architecture.
samyuh commentedon Sep 3, 2025
You're right, @lucassaldanha! I initially glanced over this and didn’t pay much attention.
I came across this thread in dmlc/xgboost#7915, which mentions that detecting musl from within the JVM can be a bit tricky. It's making me think that even checking for
/lib/libc.musl-x86_64.so.1might not be reliable across all environments :(They fixed the issue by adding a system property to manually specify the native library path when using libraries different than glibc. However, another option is to use an environment variable to indicate that the program should use the musl library.
oliverwi commentedon Jan 24, 2026
This bug is impacting us as users of the on-prem version of Confluence, because we have Grafana installed on the same VM, which is why we have /lib/ld-musl-x86_64.so.1 on a GNU Linux system.
Unfortunately, Atlassian refuses to downgrade snappy to the previous version. Therefore, on every new Confluence version I manually replace snappy 1.1.10.8 jar with 1.1.10.7.
For reference, here is the Atlassian bug: https://jira.atlassian.com/browse/CONFSERVER-100761
The issue is in
src/main/java/org/xerial/snappy/OSInfo.java, specifically theisX64Musl()method:This incorrectly assumes that the presence of
/lib/ld-musl-x86_64.so.1means the system is running musl. However, this file can exist on glibc systems when packages like Grafana install musl as a dependency, when the system itself runs on glibc.Proposed Solution
Check
/proc/self/map_filesfor dynamically loaded libraries containingld-musl. This detects what the JVM is actually using rather than what's installed on the system.I'm working on a PR with this fix and will have it ready soon.
XenoAmess commentedon Mar 14, 2026
be there any news?
Improve dependency update checks (#10842)
srstsavage commentedon Aug 10, 2026
@oliverwi Do you still have plans to submit a PR examining
/proc/self/map_filesto detect musl?I'm wondering if just removing the attempted musl autodetection and requiring specification of the library path on musl systems would be the best path forward, as was done here:
dmlc/xgboost#7921
In any case, this bug has been open for almost a year and a pragmatic immediate solution is probably better than a perfect future solution.
XenoAmess commentedon Aug 12, 2026
good point... why not implement it this way? shall I provide a pr? @oliverwi @xerial @samyuh
Detect musl from the C library loaded in the JVM process (#740)
xerial commentedon Oct 3, 2026
Fixed in #740 (merged). OSInfo now detects musl by checking whether musl's loader is mapped into the running JVM process (/proc/self/maps), instead of checking whether /lib/ld-musl-x86_64.so.1 exists. A glibc JVM on a system that also has musl installed (e.g. via Grafana or the musl package) now loads the regular glibc library again. CI now covers both Alpine and Ubuntu with musl installed. The fix will ship in 1.1.10.9.
Upgrade snappy-java to v1.1.10.10