[sctb512]

Try to address opencontainers/runc#3026

[lifubang]

Thank you for your PR. It would be greatly appreciated if you could provide some failure logs and help us analyze the root cause of this issue. For instance, could it be related to the Go stdlib’s file descriptor shuffling bug (issue opencontainers/runc#4452)?

I believe it's worth investing time to identify the underlying cause. Reopening the cgroup root file descriptor should be considered a last resort.

[kolyshkin]

@sctb512 can you reply to the comment above?

[sctb512]

Reopening the cgroup root file descriptor should be considered a last resort.

Yes, I agree with you. Unfortunately, it’s hard to reproduce this issue. We have restarted the kubelet service since the machine is in a production environment. I don’t think it’s related to https://github.com/opencontainers/runc/pull/4452, because the file descriptor pointing to /sys/fs/cgroup does not exist.

[kolyshkin]

Looking at it again, I'm still not sure this is needed, since we don't understand how this happens.

The only way I could reproduce it is by (lazily) unmounting and re-mounting the whole /sys/fs/cgroup tree on a cgroup v1 host, which is very unlikely to happen on a production machine.

Another possibility is some other code in the same process (kubelet) closing the file descriptor it does not own (and the fd number being reused), in which case reopening would only hide the real bug under the rug, not fix it.

@sctb512 can you please provide the exact error message from kubelet logs? It should contain cgroupFd N unexpectedly opened to X != /sys/fs/cgroup, and X (as well as the errno) will tell us what has happened.