Windows Container: fs.realpathSync() is broken on shared volumes

Version: 4.6.1, 6.9.1, 7.0.0
Platform: Windows 10 1607 or Windows Server 2016 with Docker + Windows Containers
Subsystem: fs

The function fs.realpathSync() and/or pathModule._makeLong() is broken using it on shared Docker volumes mapped from host into a Windows Container. It seems to interpret the junction point incorrectly which is an UNC path and interprets it as c:\ContainerMappedDirectories.

Steps to reproduce

You need Windows 10 Anniversary update with Containers feature and Docker installed. Or an Windows Server 2016 with Containers feature and Docker installed, eg. an Azure VM.

Create a test directory with a minimal Dockerfile to just put node.exe into a Windows container.

PS C:\> mkdir test
PS C:\> cd test
PS C:\test> notepad Dockerfile.

This is the content of the Dockerfile

FROM microsoft/nanoserver

ARG NODE_VERSION=4.6.1

RUN powershell -Command \
    Invoke-WebRequest https://nodejs.org/dist/v$env:NODE_VERSION/win-x64/node.exe \
      -OutFile node.exe -UseBasicParsing

CMD [ "node.exe" ]

Now build a Docker image eg. for Node.js 4.6.1 using a build argument:

PS C:\test> docker build -t nodeexe:4.6.1 --build-arg NODE_VERSION=4.6.1 .

We can check that the binary works inside the Windows container by showing the version

PS C:\test> docker run nodeexe:4.6.1 node --version
v4.6.1

Now check that fs.realpathSync() works with an existing directory inside the container.

PS C:\test> docker run nodeexe:4.6.1 node -p "const fs=require('fs'); fs.realpathSync('c:/windows')"
c:\windows

OK. Now we mount the current directory as shared volume into the container and have a look at it with the dir command:

PS C:\test> docker run -v "$(pwd):C:\test" nodeexe:4.6.1 cmd /c dir c:\
 Volume in drive C has no label.
 Volume Serial Number is B097-15C5

 Directory of c:\

10/09/2016  10:57 PM             1,894 License.txt
11/04/2016  12:50 PM        14,259,744 node.exe
07/16/2016  05:20 AM    <DIR>          Program Files
07/16/2016  05:09 AM    <DIR>          Program Files (x86)
11/04/2016  01:43 PM    <SYMLINKD>     test [\\?\ContainerMappedDirectories\6C4D3D41-7E32-4992-BC03-44E1FAC23C0B]
10/09/2016  10:58 PM    <DIR>          Users
11/04/2016  12:50 PM    <DIR>          Windows
               2 File(s)     14,261,638 bytes
               5 Dir(s)  21,210,304,512 bytes free

OK, the directory C:\test is there, but you can see that it is a symlink to an UNC path. Let's check that this directory is accessible.

PS C:\test> docker run -v "$(pwd):C:\test" nodeexe:4.6.1 cmd /c dir c:\test
 Volume in drive C has no label.
 Volume Serial Number is B097-15C5

 Directory of c:\test

11/04/2016  01:44 PM    <DIR>          .
11/04/2016  01:44 PM    <DIR>          ..
11/04/2016  01:46 PM               231 Dockerfile
               1 File(s)            231 bytes
               2 Dir(s)  21,837,713,408 bytes free

OK, we can see the Dockerfile from the host. So let's now check fs.realpathSync with that directory.

PS C:\test> docker run -v "$(pwd):C:\test" nodeexe:4.6.1 node -p "const fs=require('fs'); fs.realpathSync('c:/test')"
fs.js:839
  return binding.lstat(pathModule._makeLong(path));
                 ^

Error: ENOENT: no such file or directory, lstat 'c:\ContainerMappedDirectories'
    at Error (native)
    at Object.fs.lstatSync (fs.js:839:18)
    at Object.realpathSync (fs.js:1439:21)
    at [eval]:1:28
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:409:26)
    at node.js:579:27
    at nextTickCallbackWith0Args (node.js:420:9)
    at process._tickCallback (node.js:349:13)

Boom! As you can see the UNC path is misinterpreted as c:\ContainerMappedDirectories and then of course this directory does not exist inside the Windows container.

This error breaks eg. using npm install on such a shared volume or even running simple Node.js code from it.

Let's create a small hello.js, map the current folder into a container and try to run it.

PS C:\test> "console.log('hello');" | Out-File hello.js -Encoding Ascii
PS C:\test> docker run -v "$(pwd):C:\test" nodeexe:4.6.1 node c:\test\hello.js
fs.js:839
  return binding.lstat(pathModule._makeLong(path));
                 ^

Error: ENOENT: no such file or directory, lstat 'c:\ContainerMappedDirectories'
    at Error (native)
    at Object.fs.lstatSync (fs.js:839:18)
    at Object.realpathSync (fs.js:1439:21)
    at toRealPath (module.js:112:13)
    at Function.Module._findPath (module.js:151:20)
    at Function.Module._resolveFilename (module.js:323:25)
    at Function.Module._load (module.js:276:25)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:139:18)
    at node.js:974:3

When we copy the file from the shared volume into a local directory of the Windows container Node.exe is able to run it:

PS C:\test> docker run -v "$(pwd):C:\test" nodeexe:4.6.1 cmd "/c copy c:\test\hello.js . & node hello.js"
        1 file(s) copied.
hello

Another problem of this bug is that running npm install also does not work in a shared volume.

Does node --preserve-symlinks c:\nodejs\node_modules\npm\bin\npm-cli.js install work?

No, still the same. Can't see the --preserve-symlink in the argv output:

C:\code>node --preserve-symlinks c:\nodejs\node_modules\npm\bin\npm-cli.js install
npm info it worked if it ends with ok
npm info using npm@3.10.3
npm info using node@v6.7.0
npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\\nodejs\\node.exe" "c:\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v6.7.0
npm ERR! npm  v3.10.3
npm ERR! path C:\ContainerMappedDirectories
npm ERR! code ENOENT
npm ERR! errno -4058
npm ERR! syscall lstat

npm ERR! enoent ENOENT: no such file or directory, lstat 'C:\ContainerMappedDirectories'
npm ERR! enoent ENOENT: no such file or directory, lstat 'C:\ContainerMappedDirectories'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! Please include the following file with any support request:
npm ERR!     C:\code\npm-debug.log

C:\code>type npm-debug.log
0 info it worked if it ends with ok
1 verbose cli [ 'C:\\nodejs\\node.exe',
1 verbose cli   'c:\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   'install' ]
2 info using npm@3.10.3
3 info using node@v6.7.0
4 silly loadCurrentTree Starting
5 silly install loadCurrentTree
6 silly install readLocalPackageData
7 silly rollbackFailedOptional Starting
8 silly rollbackFailedOptional Finishing
9 silly runTopLevelLifecycles Starting
10 silly runTopLevelLifecycles Finishing
11 silly install printInstalled
12 verbose stack Error: ENOENT: no such file or directory, lstat 'C:\ContainerMappedDirectories'
12 verbose stack     at Error (native)
13 verbose cwd C:\code
14 error Windows_NT 10.0.14393
15 error argv "C:\\nodejs\\node.exe" "c:\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
16 error node v6.7.0
17 error npm  v3.10.3
18 error path C:\ContainerMappedDirectories
19 error code ENOENT
20 error errno -4058
21 error syscall lstat
22 error enoent ENOENT: no such file or directory, lstat 'C:\ContainerMappedDirectories'
23 error enoent ENOENT: no such file or directory, lstat 'C:\ContainerMappedDirectories'
23 error enoent This is most likely not a problem with npm itself
23 error enoent and is related to npm not being able to find a file.
24 verbose exit [ -4058, true ]

It seems that node can't find/load js files in the volume mount point.

C:\code>type test.js
console.log('hello');

C:\code>node
> process.cwd()
'C:\\code'
>
(To exit, press ^C again or type .exit)
>

C:\code>node test.js
fs.js:982
  return binding.lstat(pathModule._makeLong(path));
                 ^

Error: ENOENT: no such file or directory, lstat 'C:\ContainerMappedDirectories'
    at Error (native)
    at Object.fs.lstatSync (fs.js:982:18)
    at Object.realpathSync (fs.js:1647:19)
    at Function.Module._findPath (module.js:167:25)
    at Function.Module._resolveFilename (module.js:453:25)
    at Function.Module._load (module.js:403:25)
    at Module.runMain (module.js:590:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

C:\code>node c:\code\test.js
fs.js:982
  return binding.lstat(pathModule._makeLong(path));
                 ^

Error: ENOENT: no such file or directory, lstat 'c:\ContainerMappedDirectories'
    at Error (native)
    at Object.fs.lstatSync (fs.js:982:18)
    at Object.realpathSync (fs.js:1647:19)
    at Function.Module._findPath (module.js:167:25)
    at Function.Module._resolveFilename (module.js:453:25)
    at Function.Module._load (module.js:403:25)
    at Module.runMain (module.js:590:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

C:\code>node --preserve-symlinks test.js
fs.js:982
  return binding.lstat(pathModule._makeLong(path));
                 ^

Error: ENOENT: no such file or directory, lstat 'C:\ContainerMappedDirectories'
    at Error (native)
    at Object.fs.lstatSync (fs.js:982:18)
    at Object.realpathSync (fs.js:1647:19)
    at Function.Module._findPath (module.js:167:25)
    at Function.Module._resolveFilename (module.js:453:25)
    at Function.Module._load (module.js:403:25)
    at Module.runMain (module.js:590:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

Add node-example to investigate nodejs/node#8897

Just to clarify, if someone else tries this and wants a temporary solution. This only occurs with Node.js 6.x and works fine with Node.js 4.6.0

docker run -it --rm --name my-app -v "${pwd}:C:\App" -w "C:\App" -p 3000:3000 -p 9000:9000 stefanscherer/node-windows:4.6.0 powershell

This also affects opening files, eg. TLS certs mapped into a container through a volume. How can we proceed getting this fixed in LTS version of Node.js 6?

How sure are you it's a node issue and not a docker-on-windows issue? Does the same node binary work outside of the container?

@bnoordhuis

@bnoordhuis I'm sure this is a Node.js problem. I've stripped down the problem as in #7044 to fs.realpathSync()

An internal directory works:

PS C:\> docker run -v "$(pwd):c:\code" stefanscherer/node-windows:7 node -p "fs.realpathSync('c:\windows')"
c:\windows

But a volume directory throws an exception:

PS C:\> docker run -v "$(pwd):c:\code" stefanscherer/node-windows:7 node -p "fs.realpathSync('c:\code')"
fs.js:893
  return binding.lstat(pathModule._makeLong(path));
                 ^

Error: ENOENT: no such file or directory, lstat 'c:\ContainerMappedDirectories'
    at Object.fs.lstatSync (fs.js:893:18)
    at Object.realpathSync (fs.js:1527:21)
    at [eval]:1:4
    at ContextifyScript.Script.runInThisContext (vm.js:25:33)
    at Object.exports.runInThisContext (vm.js:77:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:573:32)
    at bootstrap_node.js:345:29
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Tested with both Node.js 6.9.1 and 7.0.0.

cc @nodejs/platform-windows @nodejs/docker

If this is really is a Node.js issue, can you edit the issue title and description with a purely Windows + Node.js test case? (without references to third-party tools)

So after reading this article is appears -- https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility

It appears I can't use the version that's based off windowservercore-1803, while I could try it, it really wouldn't solve my problem to begin with.

@Francommit

@Francommit if it‘s hard to have a Windows Server 1803 semi-annual release watch out for the next long-term supported Windows Server 2019 which will be available soon.

Yeah unfortunately I'm going to have to wait for WS2019.

Given I'm building node-packages in my container as a workaround I've just robocopied everything from a mount-point to a directory in the container then copied the artifacts back to the mount point.

Not the most elegant solution or quickest but it proves my code will build for now!|

EDIT: Why robocopy and not xcopy you might ask? Try and run xcopy in a Windows Server 2016 based windowsservercore image and see what happens 👎

@StefanScherer

@StefanScherer I still have this issue with your latest nanoserver nodejs image, so I assume this is still not resolved. Do you have a new workaround? The powershell-trick is not working, I still have the error from your first post.

EDIT:
Got a brainwave, I decided to copy the mount to a folder inside the container before attempting to run. It's not ideal, technically takes double the size, but it works...

Dockerfile

FROM stefanscherer/node-windows:latest

RUN mkdir C:\app
RUN mkdir C:\app\mount
RUN mkdir C:\app\user

COPY startup.bat C:/app/startup.bat

CMD ["C:\app\startup.bat"]

startup.bat

copy C:\app\mount\* C:\app\user
cd C:\app\user
node app.js

Would very much still like an actual fix or workaround :S

EDIT EDIT: I'm an idiot, I didn't switch to the new volume... The workaround still works fine.

@LennardF1989

@LennardF1989 The problem isn't fixed with the Windows 2016 base images. We have seen progress and fixes in the 1803 images, but people have to run Windows Server Core version 1803 which is quite rare.

Now Windows Server 2019 / Windows 10 1809 Update is around the corner with two new Windows base images

mcr.microsoft.com/windows/servercore:ltsc2019
mcr.microsoft.com/windows/nanoserver:1809

And with these images bind mounting a folder into the container works fine in combination with Node.js.

Edit: I also just tried to run a Windows 2016 based Node image on a Windows Server 2019 in HyperV isolation mode, but the problem still exists. This is because the Windows 2016 base image brings its old Windows kernel where the bind mount isn't fixed for Node.js. So the best experience will be only with the new 1809/ltsc2019 base images running on the latest Windows OS.

I can confirm this is fixed in 1803 Core. Got it working today!

Still having this issue with Windows 10!
I've tried tfrijsewijk's solution, but get this error

/bin/sh: Set-ItemProperty: not found

Dockerfile

FROM node:alpine
#Create app directory
WORKDIR /usr/src/app
#Install nodemon for hot reloading
RUN npm install nodemon -g
#Install app dependencies
#A wildcard is used to ensure both package.json AND package-lock.json are copied
COPY package*.json ./
RUN npm install
#Bundle app source
COPY . .
EXPOSE 8080
CMD [ "nodemon", "-L", "server/server.js" ]

I'm trying to setup hot reloading with nodemon for a dev environment.
However using a volume like so:

docker build -t node-api .
docker run --rm -it -p 8080:8080 -v "${PWD}:/usr/src/app" node-api

outputs

[nodemon] 1.18.6
[nodemon] to restart at any time, enter `rs`
[nodemon] watching: *.*
[nodemon] starting `node server/server.js`
internal/modules/cjs/loader.js:605
    throw err;
    ^

Error: Cannot find module 'express'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:603:15)
    at Function.Module._load (internal/modules/cjs/loader.js:529:25)
    at Module.require (internal/modules/cjs/loader.js:658:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/usr/src/app/server/server.js:1:79)
    at Module._compile (internal/modules/cjs/loader.js:722:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:733:10)
    at Module.load (internal/modules/cjs/loader.js:620:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
    at Function.Module._load (internal/modules/cjs/loader.js:552:3)
[nodemon] app crashed - waiting for file changes before starting...

If I exclude the -v flag and docker run then it starts with no errors, but doesn't detect changes or restart on file saves.

Github repo for code

@bcowell

Thanks @bcowell for sharing your code. But FROM node:alpine is a Linux image, there is no PowerShell inside. That's why Set-ItemPropery does not work.

This issue is about the problem with the "V1" of Windows containers beginning with Windows Server 2016 and Windows 10.

The problem is solved beginning with Windows Server, version 1803 and Windows 10 1803 when you run the 1803 Windows images and Node.js inside. The Server 1803 isn't available for all users, but the Windows Server 2019 ("V2" of Windows containers :-) ) now is here. Embracing the new 1809 / ltsc2019 Windows images also solves this problem.

The issue remains in the ltsc2016 images, but I think we can close this issue if people use the newer images soon.

@StefanScherer

Thanks @StefanScherer you're right.

@StefanScherer

Some of us would love for this to work in a ltc2016 container! @StefanScherer I'm yet to test this with the recent update to the ltc2016 base layers, will give it a try later in the week and report back.

On a seim-unrelated note.

I made a Windows Server 1803 version of my process using a shared volume mount that did part of a npm command, it was SLOWER overall than me performing a robocopy to a folder from the volume mount than performing the build on the volume mount directly.

It appears to be an open issue on Docker for Windows community edition, just interesting to see it in practice.

Is this problem still present? node v7 b488b19 (#3594) may have inadvently fixed it as a side effect, as uv_fs_realpath() on windows has always used the IO Manager magic since its introduction in libuv/libuv@e76b883.

Oh well, they decided the js readlink-and-stich-by-resolve version is superior. The native one is now fs.realpathSync.native().

path.resolve now recognizes UNC targets of the form \\foo\bar：

path.resolve('nothere', '\\\\ContainerMappedDirectories\\6C4D3D41-7E32-4992-BC 03-44E1FAC23C0B')

However, this breaks:

D:\>mklink /D smbex \\?\notRealDevice\foo
D:\> dir /aL
2019/12/13  20:13    <SYMLINKD>     smbex [\\?\notRealDevice\foo]

fs.readlinkSync('D:\\smbex\\')
// '\\??\\notRealDevice\\foo'
path.resolve('notreal', fs.readlinkSync('D:\\smbex\\'))
// 'C:\\??\\notRealDevice\\foo'

readlink somehow converted the \\?\ local dosDevices prefix into the NT \??\ prefix without removing it. path.resolve did not see two slashes, so it treated it as a relative path even though this is a thing in NT speak.

This is still an issue in v14.0.0, can't use it in my GitLab CI pipeline...

Some details: I need node.exe to compile the .ts files in .js files as part of a ASP.NET MVC project.

I use the GitLab CI + Docker functionality to compile my project, but I have noticed that the .js files were missing. By digging in the issue I have discovered that I get the C:\ContainerMappedDirectories bug.

I have tried some versions: 14.0.0, 10.19.0, 4.6.0 and 4.5.0, but no one will work with Docker with a mapped directory.

@schifazl

@schifazl I was also having trouble with NodeJS in my mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2016 based image, however, this article's solution solved my problem with subst: https://blog.sixeyed.com/docker-volumes-on-windows-the-case-of-the-g-drive/

I'm going to close this issue.
The problem was fixed with Windows Server 2019 and it's highly recommended to use Windows Containers with at least Windows Server 2019 or newer to have all benefits of the underlying improvements.

Navigation Menu

Repository navigation

Windows Container: fs.realpathSync() is broken on shared volumes #8897

Steps to reproduce

47 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Windows Container: fs.realpathSync() is broken on shared volumes #8897

Description

Steps to reproduce

Activity

bnoordhuis commented on Oct 3, 2016

StefanScherer commented on Oct 3, 2016

StefanScherer commented on Oct 3, 2016

StefanScherer commented on Oct 16, 2016

StefanScherer commented on Nov 2, 2016

bnoordhuis commented on Nov 3, 2016

StefanScherer commented on Nov 3, 2016

bnoordhuis commented on Nov 4, 2016

seishun commented on Nov 4, 2016

47 remaining items

Francommit commented on Sep 9, 2018

StefanScherer commented on Sep 10, 2018

Francommit commented on Sep 10, 2018

LennardF1989 commented on Oct 13, 2018

StefanScherer commented on Oct 14, 2018

Francommit commented on Oct 16, 2018

bcowell commented on Nov 21, 2018

StefanScherer commented on Nov 21, 2018

bcowell commented on Nov 21, 2018

Francommit commented on Nov 22, 2018

Artoria2e5 commented on Dec 13, 2019

schifazl commented on Apr 22, 2020

shichen85 commented on Jun 18, 2020

StefanScherer commented on Jun 20, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions