Repository navigation
test-osuosl-aix72-ppc64_be-3 build failures #2718
Description
test-osuosl-aix72-ppc64_be-3 is failing repeatedly so I marked it offline.
Refs: nodejs/node#39604 (comment)
@nodejs/platform-aix
test-osuosl-aix72-ppc64_be-3 is failing repeatedly so I marked it offline.
Refs: nodejs/node#39604 (comment)
@nodejs/platform-aix
Activity
richardlau commentedon Jul 31, 2021
I had to clean up a bunch of leftover gmake/gcc processes on that machine on Thursday while preparing the releases. Given that https://ci.nodejs.org/job/node-test-commit-aix/37790/nodes=aix72-ppc64/ is encountering
EAGAINerrors spawning processes, the likely situation is that again there are lots of leftover processes running on the system. We'd previously seen that situation on the same machine in #2621 (comment).I won't be able to log into the machine until late Sunday night/Monday -- supposedly we have a playbook runnable from our AWX server that is able to clean up processes (https://github.com/nodejs/build/blob/master/ansible/playbooks/jenkins/worker/restart-agent.yml) but I don't actually see any templates under resources as I did before as documented in #2714 (cc @AshCripps).
richardlau commentedon Aug 2, 2021
Looks like
test-osuosl-aix72-ppc64_be-2may be exhibiting the same problem now 😞,e.g. https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/37812/console
richardlau commentedon Aug 2, 2021
hmm. I just logged into
test-osuosl-aix72-ppc64_be-3and there are no leftover gcc/gmake processes running -- but it also looks like the Jenkins agent crashed last night and restarted (which may have caused any leftover processes to have been cleaned up):richardlau commentedon Aug 2, 2021
I attempted to run the Jenkins worker playbook against
test-osuosl-aix72-ppc64_be-3to update Java -- currently we're on:while https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/tag/jdk8u292-b10_openj9-0.26.0 is the latest.
Unfortunately I got an error running the playbook:
which is because Ansible has failed to find
gtarand fallen back totar. Looking at thetarpackage on AIX it looks like that latest version has dropped the symlinks that used to be created in/usr/bin.Details
We saw a similar issue before for
curl(#2536) -- I'll open a PR to recreate thegtarsymlinks on AIX so we can run the playbook (and update to the most recent Java 8).richardlau commentedon Aug 2, 2021
#2720 to fix the
gtarissue. I've reansibledtest-osuosl-aix72-ppc64_be-3, which has updated Java torichardlau commentedon Aug 2, 2021
Current theory is that the instability is coming from
/varbeing full on all three hosts.e.g.
test-osuosl-aix72-ppc64_be-2:Where
/var/cacheis some adoptopenjdk downloaded tarballs (via our Ansible scripts) and theyumcache. I've cleaned out theyumcache viayum clean all.richardlau commentedon Aug 3, 2021
We've had a reasonable number of green builds on -3 but also some failures. For example on
test-osuosl-aix72-ppc64_be-3https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/37855/ and
https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/37856/
both got
java.io.IOException: Backing channel 'JNLP4-connect connection from p8-aix2-nodejs02.osuosl.org/140.211.9.131:56336' is disconnected.exceptions -- when this happens it looks like thegcc/gmakeprocesses are still left running on the machine.richardlau commentedon Aug 3, 2021
There was a suggestion to wipe out the agent jar and reinstall it -- I've done so (delete, rerun the Jenkins worker create playbook and the restart agent playbook) but we've already got the Java exceptions 😞.
richardlau commentedon Aug 4, 2021
And since those exceptions we've had ten straight green builds (with 3 green test 16.5.0 builds immediately before that) across all three AIX 7.2 CI hosts without me changing anything on them 🤷 .
richardlau commentedon Aug 4, 2021
nio disconnection in https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/37887/console on -2 😞 .
richardlau commentedon Aug 5, 2021
We appear to have had nio disconnects as far back as over a month ago, e.g. https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/37311/ (1 month 7 days ago on
-3). Not sure about earlier than that as we don't have the Jenkins logs for the earlier failed builds.It does feel like the disconnects have become more frequent recently.
aixtools commentedon Aug 5, 2021
richardlau commentedon Aug 25, 2021
Another nio disconnect: https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-ppc64/38061/ occurred two days ago on
test-osuosl-aix72-ppc64_be-3-- all subsequent builds failing with EAGAIN on child process spawns.Ran the
playbooks/jenkins/worker/restart-agent.ymlagainst it to recover.2 remaining items
richardlau commentedon Sep 9, 2021
We've had five disconnects in the last six days (since I last cleaned up). Captured logs before running restart-agent.
Details
richardlau commentedon Sep 17, 2021
Six more disconnects since the previous restart on
test-osuosl-aix72-ppc64_be-3. Agent restarted.Details
richardlau commentedon Sep 24, 2021
I'm trying out running the Jenkins agent with Java 11 on
test-osuosl-aix72-ppc64_be-3(#2772). Will keep an eye out to see if we still get disconnects and/or the zombie child processes with the agent running on the later Java version.aixtools commentedon Sep 24, 2021
The labels seem to have been renewed (be-3 bit is new to me, so let me figure out which system this really is, and I see it has had this since Aug. 1st. The system may be suffering from the crash still, maybe a refresh os the system is still needed.).
AshCripps commentedon Sep 24, 2021
@aixtools the ip should be:
140.211.9.131richardlau commentedon Oct 1, 2021
We had one disconnect and subsequent NoClassDefFoundError in the last week on
test-osuosl-aix72-ppc64_be-3with Java 11, similar stack trace as before.Details
Trying Java 17.
richardlau commentedon Oct 5, 2021
Unfortunately we still get the two problems with Java 17.
richardlau commentedon Oct 11, 2021
test-osuosl-aix72-ppc64_be-3was reinstalled/reansibled. Still has disconnects (back to Java 8):Details
Curiously the first disconnect on Oct 8 didn't leave any child processes behind but when I logged in just now there were child processes running but no (according to Jenkins) build in progress (note the 13:23 timestamps that correlate with the disconnect on Oct 10):
Details
aixtools commentedon Oct 18, 2021
Well, if the details above are compareable with the disconnects on the other systems - it might be safe to say that the disconnects are not related to side-effects left over from the POWER outage.
Is it possible to go to an old version of java (pre-August) and see if the agent is stable again (and/or check the ci server admin logs for changes made in the last week of July, 1st week of August).
Maybe what you see with AIX is just a symptom of something created elsewhere.
richardlau commentedon Oct 28, 2021
I've implemented a new Jenkins job (restricted currently to the Build WG) that runs
killall -. This will send a SIGTERM to all processes owned by theiojsuser that we use to run Jenkins except for the Jenkins agent process (i.e. the parent process) and then SIGKILL if the processes are still alive after 30s. The job is set up to run daily.https://ci.nodejs.org/job/aix-cleanup/
richardlau commentedon Feb 16, 2022
We're now seeing the disconnects on every build #2872 😞.
sxa commentedon Feb 16, 2022
I'm getting some slowness on interactive shells to OSUOSL AIX machines. For now I've added a job that's doing a ping every 20 seconds to see if that shows any connectivity issues which is writing to
/home/iojs/nohup.outrichardlau commentedon Feb 16, 2022
@sxa It would appear that the current issue we have is with Jenkins' "Ping thread" (https://www.jenkins.io/doc/book/system-administration/monitoring/#ping-thread) which is causing the Jenkins server to close down the agent on the OSUOSL AIX instances as it doesn't get a response within the ping timeout (both with 120s that we were using before and the default 240s). I've temporarily disabled the ping thread and restarted the AIX agents and so far the builds look like they're progressing much better (I got a Node.js 16 build through at least 🎉 https://ci.nodejs.org/job/node-test-commit-aix/40040/).
Possible issues with connectivity/network throughput could certainly explain the pings from the Jenkins server not being responded to.
sxa commentedon Feb 17, 2022
Looking a lot more green now which is good. Possibly one to alter @aixtools to about when he's back next week as it's looking like network connectivity outages. I've been having channel shutdowns on OSUOSL AIX boxes at Adoptium too: adoptium/infrastructure#2473 (Slightly different messages, although those are connecting in from the master over ssh instead of starting over jnlp)