Docker-compose node modules not present in a volume after npm install succeeds

Question

I have an app with the following services    web  - holds and runs a python 3 flask web server on port 5000  Uses sqlite3  worker  - has an index js file which is a worker for a queue  the web server interacts with this queue using a json API over port 9730  The worker uses redis for storage  The worker also stores data locally in the folder worker images    Now this question only concerns the worker   worker Dockerfile  FROM node 0 12  WORKDIR  worker  COPY package json  worker  RUN npm install  COPY    worker    docker-compose yml  redis      image  redis worker      build    worker     command  npm start     ports          -  9730 9730      volumes          - worker   worker      links          - redis   When I run docker-compose build  everything works as expected and all npm modules are installed in  worker node modules as I d expect   npm WARN package json unfold 1 0 0 No README data   gt  phantomjs 1 9 2-6 install  worker node modules pageres node modules screenshot-stream node modules phantom-bridge node modules phantomjs  gt  node install js   lt snip gt    But when I do docker-compose up  I see this error   worker 1   Error  Cannot find module  async  worker 1       at Function Module  resolveFilename  module js 336 15  worker 1       at Function Module  load  module js 278 25  worker 1       at Module require  module js 365 17  worker 1       at require  module js 384 17  worker 1       at Object  lt anonymous gt    worker index js 1 75  worker 1       at Module  compile  module js 460 26  worker 1       at Object Module  extensions  js  module js 478 10  worker 1       at Module load  module js 355 32  worker 1       at Function Module  load  module js 310 12  worker 1       at Function Module runMain  module js 501 10    Turns out none of the modules are present in  worker node modules  on host or in the container     If on the host  I npm install  then everything works just fine  But I don t want to do that  I want the container to handle dependencies   What s going wrong here    Needless to say  all packages are in package json

User · Answer

If you want the node modules folder available to the host during development  you could install the dependencies when you start the container instead of during build-time  I do this to get syntax highlighting working in my editor  Dockerfile   We re using a multi-stage build so that we can install dependencies during build-time only for production     dev-stage FROM node 14-alpine AS dev-stage WORKDIR  usr src app COPY package json    COPY        yarn install  will run every time we start the container  We re using yarn because it s much faster than npm when there s nothing new to install CMD   quot sh quot    quot -c quot    quot yarn install  amp  amp  yarn run start quot      production-stage FROM node 14-alpine AS production-stage WORKDIR  usr src app COPY package json    RUN yarn install COPY       dockerignore Add node modules to  dockerignore to prevent it from being copied when the Dockerfile runs COPY      We use volumes to bring in node modules     node modules  docker-compose yml node app      container name  node app     build          context    node app         target  dev-stage    production-stage  for production     volumes            For development              If node modules already exists on the host  they will be copied             into the container here  Since  yarn install  runs after the             container starts  this volume won t override the node modules          -   node app  usr src app           For production                       -   node app  usr src app         -  usr src app node modules

User · Answer

I tried the most popular answers on this page but ran into an issue  the node modules directory in my Docker instance would get cached in the the named or unnamed mount point  and later would overwrite the node modules directory that was built as part of the Docker build process  Thus  new modules I added to package json would not show up in the Docker instance  Fortunately I found this excellent page which explains what was going on and gives at least 3 ways to work around it  https   burnedikt com dockerized-node-development-and-mounting-node-volumes

User · Answer

Installing node modules in container to different from project folder  and setting NODE PATH to your node modules folder helps me  u need to rebuild container    I m using docker-compose  My project file structure   - myproject --docker-compose yml --nodejs  ----Dockerfile   docker-compose yml   version   2  services    nodejs      image  myproject nodejs     build    nodejs       volumes        -   nodejs  workdir     ports        -  23005 3000      command  npm run server   Dockerfile in nodejs folder   FROM node argon RUN mkdir  workdir COPY   package json  workdir   RUN mkdir  data RUN ln -s  workdir package json  data   WORKDIR  data RUN npm install ENV NODE PATH  data node modules  WORKDIR  workdir

User · Answer

You can just move node modules into a   folder  How it works FROM node 0 12  WORKDIR  worker  COPY package json  worker  RUN npm install      amp  amp  mv node modules  node modules  COPY    worker

User · Answer

There is also some simple solution without mapping node module directory into another volume  It s about to move installing npm packages into final CMD command      Disadvantage of this approach          run npm install each time you run container  switching from npm to yarn might also speed up this process a bit        worker Dockerfile  FROM node 0 12 WORKDIR  worker COPY package json  worker  COPY    worker  CMD  bin bash -c  npm install  npm start    docker-compose yml  redis      image  redis worker      build    worker     ports          -  9730 9730      volumes          - worker   worker      links          - redis

User · Answer

The node modules folder is overwritten by the volume and no more accessible in the container  I m using the native module loading strategy to take out the folder from the volume    data node modules    dependencies installed here  data app    code base   Dockerfile   COPY package json  data  WORKDIR  data  RUN npm install ENV PATH  data node modules  bin  PATH  COPY    data app  WORKDIR  data app    The node modules directory is not accessible from outside the container because it is included in the image

User · Answer

You can try something like this in your Dockerfile   FROM node 0 12 WORKDIR  worker CMD bash   start sh   Then you should use the Volume like this   volumes    - worker   worker rw   The startscript should be a part of your worker repository and looks like this      bin sh npm install npm start   So the node modules are a part of your worker volume and gets synchronized and the npm scripts are executed when everything is up

User · Answer

The solution provided by  FrederikNS works  but I prefer to explicitly name my node modules volume  My project docker-compose yml file  docker-compose version 1 6     version   2  services    frontend               build    worker     volumes        -   worker  worker       - node modules  worker node modules          volumes    node modules   my file structure is   project       -- worker              - Dockerfile     -- docker-compose yml  It creates a volume named project node modules and re-use it every time I up my application  My docker volume ls looks like this   DRIVER              VOLUME NAME local               project mysql local               project node modules local               project2 postgresql local               project2 node modules

User · Answer

There s elegant solution   Just mount not whole directory  but only app directory  This way you ll you won t have troubles with npm modules    Example     frontend      build        context    ui frontend       dockerfile  Dockerfile dev     ports      - 3000 3000     volumes      -   ui frontend src  frontend src   Dockerfile dev   FROM node 7 2 0   Show colors in docker terminal ENV COMPOSE HTTP TIMEOUT 50000 ENV TERM  xterm-256color   COPY    frontend WORKDIR  frontend RUN npm install update RUN npm install --global typescript RUN npm install --global webpack RUN npm install --global webpack-dev-server RUN npm install --global karma protractor RUN npm install CMD npm run server dev

User · Answer

UPDATE  Use the solution provided by  FrederikNS  I encountered the same problem  When the folder  worker is mounted to the container - all of it s content will be syncronized  so the node modules folder will disappear if you don t have it locally   Due to incompatible npm packages based on OS  I could not just install the modules locally - then launch the container  so   My solution to this  was to wrap the source in a src folder  then link node modules into that folder  using this index js file  So  the index js file is now the starting point of my application  When I run the container  I mounted the  app src folder to my local src folder  So the container folder looks something like this   app    node modules    src      node modules - gt     node modules      app js    index js  It is ugly  but it works

User · Answer

If you don t use docker-compose you can do it like this  FROM node 10  WORKDIR  usr src app  RUN npm install -g  angular cli  COPY package json    RUN npm install  EXPOSE 5000  CMD ng serve --port 5000 --host 0 0 0 0  Then you build it  docker build -t myname   and you run it by adding two volumes  the second one without source  docker run --rm -it -p 5000 5000 -v  quot  PWD quot   usr src app  -v  usr src app node modules myname

User · Answer

This happens because you have added your worker directory as a volume to your docker-compose yml  as the volume is not mounted during the build   When docker builds the image  the node modules directory is created within the worker directory  and all the dependencies are installed there  Then on runtime the worker directory from outside docker is mounted into the docker instance  which does not have the installed node modules   hiding the node modules you just installed  You can verify this by removing the mounted volume from your docker-compose yml   A workaround is to use a data volume to store all the node modules  as data volumes copy in the data from the built docker image before the worker directory is mounted  This can be done in the docker-compose yml like this   redis      image  redis worker      build    worker     command  npm start     ports          -  9730 9730      volumes          -   worker   worker          -  worker node modules     links          - redis   I m not entirely certain whether this imposes any issues for the portability of the image  but as it seems you are primarily using docker to provide a runtime environment  this should not be an issue   If you want to read more about volumes  there is a nice user guide available here  https   docs docker com userguide dockervolumes   EDIT  Docker has since changed it s syntax to require a leading    for mounting in files relative to the docker-compose yml file

User · Answer

Due to the way Node js loads modules  node modules can be anywhere in the path to your source code  For example  put your source at  worker src and your package json in  worker  so  worker node modules is where they re installed

User · Answer

You can also ditch your Dockerfile  because of its simplicity  just use a basic image and specify the command in your compose file   version   3 2   services    frontend      image  node 12-alpine     volumes        -   frontend   app      command  sh -c  cd  app   amp  amp  yarn  amp  amp  yarn run start      expose   8080      ports        - 8080 4200   This is particularly useful for me  because I just need the environment of the image  but operate on my files outside the container and I think this is what you want to do too

User · Answer

In my opinion  we should not RUN npm install in the Dockerfile  Instead  we can start a container using bash to install the dependencies before runing the formal node service  docker run -it -v   app  usr src app  your node image name   bin bash root 247543a930d6  usr src app  npm install

User · Answer

There are two seperate requirements I see for node dev environments    mount your source code INTO the container  and mount the node modules FROM the container  for your IDE    To accomplish the first  you do the usual mount  but not everything    just the things you need  volumes      - worker src  worker src     - worker package json  worker package json     - etc        the reason to not do -  worker node modules is because docker-compose will persist that volume between runs  meaning you can diverge from what is actually in the image  defeating the purpose of not just bind mounting from your host     The second one is actually harder   My solution is a bit hackish  but it works   I have a script to install the node modules folder on my host machine  and I just have to remember to call it whenever I update package json  or  add it to the make target that runs docker-compose build locally    install node modules      docker build -t building       docker run -v  pwd  node modules  app node modules building npm install

User · Answer

I recently had a similar problem  You can install node modules elsewhere and set the NODE PATH environment variable   In the example below I installed node modules into  install  worker Dockerfile  FROM node 0 12  RUN   mkdir     install    ADD     package json     install   WORKDIR  install RUN npm install --verbose ENV NODE PATH  install node modules  WORKDIR  worker  COPY    worker    docker-compose yml  redis      image  redis worker      build    worker     command  npm start     ports          -  9730 9730      volumes          - worker   worker      links          - redis

[node.js] Docker-compose: node_modules not present in a volume after npm install succeeds

Dockerfile

.dockerignore

docker-compose.yml

Examples related to node.js

Examples related to docker

Examples related to ubuntu-14.04

Examples related to docker-compose