Repository navigation
Feature request: os: os.availableProcessors() #28855
Description
Activity
I would first start investigating if Docker/Moby or their upstream projects can fix it.
Both
/proc/cpuinfoandgetconf _NPROCESSORS_ONLNare heavily used methods to obtain number of CPU cores but for some reason both do not reflect Docker's CPU limits.I think this issue happens also with memory, using
os.totalmem(), does not take container resources, It'll take the Docker engine resources, I guess@silverwind Okay, thank you, waiting for your good news!
@juanarbol I've did some investigation, it seems work with memory，please see below:
$ docker container run --rm --memory="3072m" -it node:10.16.0 bash root@e39346d06455:/# node > const os = require('os'); undefined > console.log(os.freemem()); 76783616 undefined > console.log(os.totalmem()); 2096164864 undefined > .exit root@e39346d06455:/# exit exit
@NicolasSchwarzer I got the same is total memory as you did, but I just gave
100mbas maximum memory usage, I think we're getting the Docker engine total memory also$ docker container run --rm --memory="100m" -it node:10.16.0 bash root@fe57c5bcdc46:/# node > const os = require('os'); undefined > console.log(os.freemem()) 119603200 undefined > console.log(os.totalmem()) 2096164864 undefined > .exit root@fe57c5bcdc46:/# exit
Also free memory is around
119.6mb, that's greater than my given100mbto that container, and total memory is more than1gbof ram@juanarbol You're right, I've tried and it shows exactly as what you described.
@silverwind Please also have a look at the same kind bug of
os. freemem()&os. totalmem().Guess you misunderstood me, I don't indent to chase this down to the correct project. I found this which might point someone in the right directions (kernel or runc). Out of curiosity, I did check in nginx's source and found it relies on
_NPROCESSORS_ONLNwhich is also affected by this issue.Anyways, if we want to do something about this in node, it should probably interface with
/sys/fs/cgroupbut the issue remains that this is something very Linux-specific and Node.js aims to be cross-platform, so it's a rather bad candidate for inclusion in core I think.Reacted by Juan JoséOkay, I get it. So is there a confirmed conclusion about whether to support in node? Indeed this is a platform-specific issue, I could raise an issue to docker if you confirmed not acceptable.
I'd say its pretty certain that we wouldn't accept something only relevant to one platform. There have been various proposals for Linux-specific features in the past and most of them have been rejected.
@NicolasSchwarzer most of the platform specific code is in https://github.com/libuv/libuv. You might look there to see if you can find the code Node.js uses in libuv and have a discussion in the libuv repo if that should reflect the limits withing a docker container. I personally think being able to find the actual number of processors available is useful.
@silverwind Thanks for your answering, I will close this issue.
@mhdawson Okay, I will investigate libuv later, thanks for your advice.
@NicolasSchwarzer @silverwind is it really a “Platform specific issue” if it’s the platform most production Node runs on, almost all build jobs and quite a lot of dev loops also use?
OS X and I believe Windows run Docker on a Linux VM. so you’re still dealing with Linux cgroup issues even when you are developing on a Mac.
And my Build agents pretty much only run on Windows or OS X if I need some UI testing and still have a small test matrix.
Update: It seems like , cputset related issue inside docker container can be solved using lxcfs ( even without patching docker project ) . See https://github.com/lxc/lxcfs#using-with-docker
With k8s becoming increasingly popular and nodejs running in k8s, this request is necessary.
Reacted by Deng RuoqiIt's coming but it's blocked on upgrading libuv, see #42340.
Reacted by liugangThe Q&A on the associated PR says that it does not fulfill the spirit of this request. If that's so I'm going to re-file this issue because Docker is where almost all Node applications run in production and still having to handle these problems out-of-process is a disservice to the community.
I commented on your comment in the linked PR but to recap/expound: as long as you use docker's cpuset functionality - not its exquisitely poorly named
--cpus=...flag - things should Just Work(TM).
Is your feature request related to a problem? Please describe.
It's unable to get actual available amount of processors in a docker container running on image
node:10.16.0, of which cpus is limited by docker container resource constraints. For detail information, please have a look at issue #28762.Describe the solution you'd like
Provide
os.availableProcessors()api to get the actual available amount of processors.Describe alternatives you've considered
If possible, it's better to fix the output of
os.cpus().length, e.g. by usingProxyto change the value oflengthproperty. Because most node modules useos.cpus().lengthto decide how many processes it should fork, which may exceed the limitation of cpus in the docker container.