You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thought it might be a good idea to have a tracker issue for ARM failures, I see a lot, but am not sure whether it's the same failures again and again or something new.
1 comment per new type of failure is probably a good starting point.
Thinking this might be a good use for the wiki, stuff that changes often, we don't really need to worry about source control for, basically a scratchpad for anyone to edit.
I'm starting to keep notes now on my maintenance of the cluster. I can drop them in here also if that's helpful. If there are repeating correlations with erroring machines even after replacing SD cards then we might be able to pinpoint ones that need to be retired.
Today I r/w tested the SD cards on test-requireio_bengl-debian7-arm_pi1p-1, test-requireio_mhdawson-debian7-arm_pi1p-1 and test-requireio_securogroup-debian7-arm_pi1p-1. Notice they are listed above in @refack's comment, they show up regularly and I'm pretty sure I've reprovisioned these before, test-requireio_ceejbot-debian7-arm_pi1p-1 is another one that shows up more often than I'd like. I don't have good enough records to make a solid assessment though so I'm just going with SD card testing for now.
I've thrown one of the cards out and inserted a new one and set these 3 up from scratch and they are back in the cluster.
Also, I'm hoping that by pulling back on the overclocking on these that we might have more stability. We'll see.
Note that the above is the aforementioned warning: failed to remove out/Release: Directory not empty but there is are other errors above it that may or may not be significant like:
11:56:09 + git fetch --no-tags file:///home/iojs/.ccache/node.shared.reference +refs/heads/jenkins-node-test-commit-arm-fanned-13496-binary-pi1p/cc-armv7:refs/remotes/jenkins_tmp11:56:09 fatal: The remote end hung up unexpectedly11:56:11
But it seems to be self-healing. https://ci.nodejs.org/job/node-test-binary-arm/12703/ is now 2/3 green and looking promising. All I've done is remove stale .git/index.lock files and re-run the one CI job a few times.
FWIW our (IBM) Jenkins farm used to have all the workspaces running on a shared NFS mount, but we moved to having everything local because it kept causing problems like this.
FWIW our (IBM) Jenkins farm used to have all the workspaces running on a shared NFS mount, but we moved to having everything local because it kept causing problems like this.
I imagine that's probably not an option with Raspberry Pi devices. :-( Still, good to know.
This is caused by the nfs server having some internal problems, twice now in a few days. I'm a bit embarrassed to admit that it's likely to do with the hot weather we've been having down here (no I don't have a cooled datacenter in my garage unfortunately). I've done some restarting and cleaning up and have a couple of jobs working at the moment that seem to indicate that it's all good now.
Activity
gibfahn commentedon Oct 27, 2017
Failure:
https://ci.nodejs.org/job/node-test-binary-arm/11211/RUN_SUBSET=3,label=pi1-raspbian-wheezy/console
Building remotely on test-requireio_chrislea-debian7-arm_pi1p-1 (pi1-raspbian-wheezy) in workspace /home/iojs/build/workspace/node-test-binary-arm
refack commentedon Oct 27, 2017
I had an idea, by way of cataloging toward a HOWTO, to add a "code" to such failures - then we can count them and add steps for troubleshooting:
failed-to-remove-.nfshas the following signature:[node] <defunct>zombies.If there are they need to be
sudo killed.Seen several times recently:
2017-10-25
test-requireio_chrislea-debian7-arm_pi1p-12017-10-17
test-requireio_securogroup-debian7-arm_pi1p-1&test-requireio_bengl-debian7-arm_pi1p-22017-10-29
test-requireio_davglass-debian7-arm_pi1p-1[node] <defunct>confirmed and killed by refack @ 2017-10-29-11:00EDTgibfahn commentedon Oct 27, 2017
@refack great idea, but I'd add something else to the
Resolution:section: who has the access (node/build, node/infra, just Rod etc.)rvagg commentedon Nov 4, 2017
@refack you offering to start a HOWTO somewhere?
gibfahn commentedon Nov 4, 2017
Thinking this might be a good use for the wiki, stuff that changes often, we don't really need to worry about source control for, basically a scratchpad for anyone to edit.
Looks like Johan had a similar idea a while ago.
rvagg commentedon Nov 6, 2017
I'm starting to keep notes now on my maintenance of the cluster. I can drop them in here also if that's helpful. If there are repeating correlations with erroring machines even after replacing SD cards then we might be able to pinpoint ones that need to be retired.
Today I r/w tested the SD cards on
test-requireio_bengl-debian7-arm_pi1p-1,test-requireio_mhdawson-debian7-arm_pi1p-1andtest-requireio_securogroup-debian7-arm_pi1p-1. Notice they are listed above in @refack's comment, they show up regularly and I'm pretty sure I've reprovisioned these before,test-requireio_ceejbot-debian7-arm_pi1p-1is another one that shows up more often than I'd like. I don't have good enough records to make a solid assessment though so I'm just going with SD card testing for now.I've thrown one of the cards out and inserted a new one and set these 3 up from scratch and they are back in the cluster.
Also, I'm hoping that by pulling back on the overclocking on these that we might have more stability. We'll see.
refack commentedon Nov 14, 2017
I've added the "kill defunct" line to the job config (after manual testing and one mistake):
https://ci.nodejs.org/job/node-test-binary-arm/jobConfigHistory/showDiffFiles?timestamp1=2017-11-05_03-13-42×tamp2=2017-11-14_06-49-15
BridgeAR commentedon Dec 19, 2017
There are multiple builds where pretty much all arm runs failed
See e.g.
https://ci.nodejs.org/job/node-test-binary-arm/12674/
https://ci.nodejs.org/job/node-test-binary-arm/12685/
Trott commentedon Dec 19, 2017
I'm removing stale
.git/index.lockon the Raspberry Pi devices as I find them, but I don't know the cause.Trott commentedon Dec 19, 2017
Also seeing multiple failures that look like this:
Trott commentedon Dec 19, 2017
Note that the above is the aforementioned
warning: failed to remove out/Release: Directory not emptybut there is are other errors above it that may or may not be significant like:Trott commentedon Dec 19, 2017
Certainly some NFS issues showing up now:
Trott commentedon Dec 19, 2017
But it seems to be self-healing. https://ci.nodejs.org/job/node-test-binary-arm/12703/ is now 2/3 green and looking promising. All I've done is remove stale
.git/index.lockfiles and re-run the one CI job a few times.gibfahn commentedon Dec 20, 2017
FWIW our (IBM) Jenkins farm used to have all the workspaces running on a shared NFS mount, but we moved to having everything local because it kept causing problems like this.
Trott commentedon Dec 20, 2017
I imagine that's probably not an option with Raspberry Pi devices. :-( Still, good to know.
rvagg commentedon Dec 21, 2017
This is caused by the nfs server having some internal problems, twice now in a few days. I'm a bit embarrassed to admit that it's likely to do with the hot weather we've been having down here (no I don't have a cooled datacenter in my garage unfortunately). I've done some restarting and cleaning up and have a couple of jobs working at the moment that seem to indicate that it's all good now.