How to mount host volumes into docker containers in Dockerfile during build

Question

Original question  How to use the VOLUME instruction in Dockerfile  The actual question I want to solve is -- how to mount host volumes into docker containers in Dockerfile during build  i e   having the docker run -v  export  export capability during docker build  The reason behind it  for me  is when building things in Docker  I don t want those  apt-get install  caches locked in a single docker  but to share reuse them  That s the main reason I m asking about this question  Latest Update  Before docker v18 09  the correct answer should be the one that starts with   There is a way to mount a volume during a build  but it doesn t involve Dockerfiles   However  that was a poorly stated  organized and supported answer  When I was reinstalling my docker contains  I happened to stumble upon the following article  Dockerize an apt-cacher-ng service https   docs docker com engine examples apt-cacher-ng  That s the docker s solution to this my question  not directly but indirectly  It s the orthodox way docker suggests us to do  And I admit it is better than the one I was trying to ask here  Another way is  the newly accepted answer  e g   the Buildkit in v18 09  Pick whichever suits you   Was  There had been a solution -- rocker  which was not from Docker  but now that rocker is discontinued  I revert the answer back to  quot Not possible quot  again   Old Update  So the answer is  quot Not possible quot   I can accept it as an answer as I know the issue has been extensively discussed at https   github com docker docker issues 3156  I can understand that portability is a paramount issue for docker developer  but as a docker user  I have to say I m very disappointed about this missing feature  Let me close my argument with a quote from aforementioned discussion   quot I would like to use Gentoo as a base image but definitely don t want  gt  1GB of Portage tree data to be in any of the layers once the image has been built  You could have some nice a compact containers if it wasn t for the gigantic portage tree having to appear in the image during the install  quot  Yes  I can use wget or curl to download whatever I need  but the fact that merely a portability consideration is now forcing me to download  gt  1GB of Portage tree each time I build a Gentoo base image is neither efficient nor user friendly  Further more  the package repository WILL ALWAYS be under  usr portage  thus ALWAYS PORTABLE under Gentoo  Again  I respect the decision  but please allow me expressing my disappointment as well in the mean time  Thanks   Original question in details  From Share Directories via Volumes http   docker readthedocs org en v0 7 3 use working with volumes  it says that Data volumes feature  quot have been available since version 1 of the Docker Remote API quot   My docker is of version 1 2 0  but I found the example given in above article not working    BUILD-USING                docker build -t data     RUN-USING                    docker run -name DATA data FROM                   busybox VOLUME                 quot  var volume1 quot    quot  var volume2 quot   CMD                       quot  usr bin true quot    What s the proper way in Dockerfile to mount host-mounted volumes into docker containers   via the VOLUME command    apt-cache policy lxc-docker lxc-docker    Installed  1 2 0   Candidate  1 2 0   Version table       1 2 0 0         500 https   get docker io ubuntu  docker main amd64 Packages         100  var lib dpkg status    cat Dockerfile  FROM          debian sid  VOLUME          quot  export quot   RUN ls -l  export CMD ls -l  export    docker build -t data   Sending build context to Docker daemon  2 56 kB Sending build context to Docker daemon  Step 0   FROM          debian sid  --- gt  77e97a48ce6a Step 1   VOLUME          quot  export quot    --- gt  Using cache  --- gt  59b69b65a074 Step 2   RUN ls -l  export  --- gt  Running in df43c78d74be total 0  --- gt  9d29a6eb263f Removing intermediate container df43c78d74be Step 3   CMD ls -l  export  --- gt  Running in 8e4916d3e390  --- gt  d6e7e1c52551 Removing intermediate container 8e4916d3e390 Successfully built d6e7e1c52551    docker run data total 0    ls -l  export   wc       20     162    1131    docker -v Docker version 1 2 0  build fa7b24f

User · Accepted Answer

First  to answer  why doesn t VOLUME work   When you define a VOLUME in the Dockerfile  you can only define the target  not the source of the volume  During the build  you will only get an anonymous volume from this  That anonymous volume will be mounted at every RUN command  prepopulated with the contents of the image  and then discarded at the end of the RUN command  Only changes to the container are saved  not changes to the volume     Since this question has been asked  a few features have been released that may help  First is multistage builds allowing you to build a disk space inefficient first stage  and copy just the needed output to the final stage that you ship  And the second feature is Buildkit which is dramatically changing how images are built and new capabilities are being added to the build   For a multi-stage build  you would have multiple FROM lines  each one starting the creation of a separate image  Only the last image is tagged by default  but you can copy files from previous stages  The standard use is to have a compiler environment to build a binary or other application artifact  and a runtime environment as the second stage that copies over that artifact  You could have   FROM debian sid as builder COPY export  export RUN compile command here  gt  result bin  FROM debian sid COPY --from builder  result bin  result bin CMD    result bin     That would result in a build that only contains the resulting binary  and not the full  export directory     Buildkit is coming out of experimental in 18 09  It s a complete redesign of the build process  including the ability to change the frontend parser  One of those parser changes has has implemented the RUN --mount option which lets you mount a cache directory for your run commands  E g  here s one that mounts some of the debian directories  with a reconfigure of the debian image  this could speed up reinstalls of packages      syntax   docker dockerfile experimental FROM debian latest RUN --mount target  var lib apt lists type cache       --mount target  var cache apt type cache       apt-get update     amp  amp  DEBIAN FRONTEND noninteractive apt-get install -y --no-install-recommends         git   You would adjust the cache directory for whatever application cache you have  e g   HOME  m2 for maven  or  root  cache for golang     TL DR  Answer is here  With that RUN --mount syntax  you can also bind mount read-only directories from the build-context  The folder must exist in the build context  and it is not mapped back to the host or the build client     syntax   docker dockerfile experimental FROM debian latest RUN --mount target  export type bind source export       process export directory here      Note that because the directory is mounted from the context  it s also mounted read-only  and you cannot push changes back to the host or client  When you build  you ll want an 18 09 or newer install and enable buildkit with export DOCKER BUILDKIT 1   If you get an error that the mount flag isn t supported  that indicates that you either didn t enable buildkit with the above variable  or that you didn t enable the experimental syntax with the syntax line at the top of the Dockerfile before any other lines  including comments  Note that the variable to toggle buildkit will only work if your docker install has buildkit support built in  which requires version 18 09 or newer from Docker  both on the client and server

User · Answer

I think you can do what you want to do by running the build via a docker command which itself is run inside a docker container   See Docker can now run within Docker   Docker Blog   A technique like this  but which actually accessed the outer docker from with a container  was used  e g   while exploring how to Create the smallest possible Docker container   Xebia Blog   Another relevant article is Optimizing Docker Images   CenturyLink Labs  which explains that if you do end up downloading stuff during a build  you can avoid having space wasted by it in the final image by downloading  building and deleting the download all in one RUN step

User · Answer

Here is a simplified version of the 2-step approach using build and commit  without shell scripts   It involves    Building the image partially  without volumes Running a container with volumes  making changes  then committing the result  replacing the original image name     With relatively minor changes the additional step adds only a few seconds to the build time   Basically   docker build -t image-name     your normal docker build    Now run a command in a throwaway container that uses volumes and makes changes  docker run -v  some  volume --name temp-container image-name  some post-configure command    Replace the original image with the result     reverting CMD to whatever it was  otherwise it will be set to  some post-configure command     docker commit --change  CMD bash  temp-container image-name     Delete the temporary container  docker rm temp-container   In my use case I want to pre-generate a maven toolchains xml file  but my many JDK installations are on a volume that isn t available until runtime   Some of my images are not compatible with all the JDKS  so I need to test compatibility at build time and populate toolchains xml conditionally  Note that I don t need the image to be portable  I m not publishing it to Docker Hub

User · Answer

As you run the container  a directory on your host is created and mounted into the container  You can find out what directory this is with    docker inspect --format      Volumes      lt ID gt  map  export  var lib docker vfs dir  lt VOLUME ID    gt     If you want to mount a directory from your host inside your container  you have to use the -v parameter and specify the directory  In your case this would be   docker run -v  export  export data   SO you would use the hosts folder inside your container

User · Answer

It s ugly  but I achieved a semblance of this like so   Dockerfile   FROM foo COPY   m2   root  m2 RUN stuff   imageBuild sh   docker build   -t barImage container    docker run -d barImage   rm -rf   m2 docker cp   container  root  m2    m2 docker rm -f   container    I have a java build that downloads the universe into  root  m2  and did so every single time    imageBuild sh copies the contents of that folder onto the host after the build  and Dockerfile copies them back into the image for the next build   This is something like how a volume would work  i e  it persists between builds

User · Answer

UPDATE  Somebody just won t take no as the answer  and I like it  very much  especially to this particular question    GOOD NEWS  There is a way now --   The solution is Rocker  https   github com grammarly rocker  John Yani said    IMO  it solves all the weak points of Dockerfile  making it suitable for development    Rocker  https   github com grammarly rocker     By introducing new commands  Rocker aims to solve the following use cases  which are painful with plain Docker          Mount reusable volumes on build stage  so dependency management tools may use cache between builds    Share ssh keys with build  for pulling private repos  etc    while not leaving them in the resulting image    Build and run application in different images  be able to easily pass an artifact from one image to another  ideally have this logic in a single Dockerfile    Tag Push images right from Dockerfiles    Pass variables from shell build command so they can be substituted to a Dockerfile          And more  These are the most critical issues that were blocking our adoption of Docker at Grammarly    Update  Rocker has been discontinued  per the official project repo on Github     As of early 2018  the container ecosystem is much more mature than it was three years ago when this project was initiated  Now  some of the critical and outstanding features of rocker can be easily covered by docker build or other well-supported tools  though some features do remain unique to rocker  See https   github com grammarly rocker issues 199 for more details

User · Answer

There is a way to mount a volume during a build  but it doesn t involve Dockerfiles   The technique would be to create a container from whatever base you wanted to use  mounting your volume s  in the container with the -v option   run a shell script to do your image building work  then commit the container as an image when done   Not only will this leave out the excess files you don t want  this is good for secure files as well  like SSH files   it also creates a single image   It has downsides  the commit command doesn t support all of the Dockerfile instructions  and it doesn t let you pick up when you left off if you need to edit your build script   UPDATE   For example   CONTAINER ID   docker run -dit ubuntu 16 04  docker cp build sh  CONTAINER ID  build sh docker exec -t  CONTAINER ID  bin sh -c   bin sh  build sh  docker commit  CONTAINER ID  REPO  TAG docker stop  CONTAINER ID

User · Answer

If you are looking for a way to  quot mount quot  files  like -v for docker run  you can now use the --secret flag for docker build echo  WARMACHINEROX   gt  mysecret txt docker build --secret id mysecret src mysecret txt    And inside your Dockerfile you can now access this secret   syntax   docker dockerfile 1 0-experimental FROM alpine    shows secret from default secret location  RUN --mount type secret id mysecret cat  run secrets mysecret    shows secret from custom secret location  RUN --mount type secret id mysecret dst  foobar cat  foobar  More in-depth information about --secret available on Docker Docs

User · Answer

As many have already answered  mounting host volumes during the build is not possible  I just would like to add docker-compose way  I think it ll be nice to have  mostly for development testing usage  Dockerfile  FROM node 10 WORKDIR  app COPY     RUN npm ci CMD sleep 999999999   docker-compose yml  version   3  services    test-service      image  test image     build        context          dockerfile  Dockerfile     container name  test     volumes        -   export  app export       -   build  app build   And run your container by docker-compose up -d --build

User · Answer

It is not possible to use the VOLUME instruction to tell docker what to mount  That would seriously break portability  This instruction tells docker that content in those directories does not go in images and can be accessed from other containers using the --volumes-from command line parameter  You have to run the container using -v  path on host  path in container to access directories from the host   Mounting host volumes during build is not possible  There is no privileged build and mounting the host would also seriously degrade portability  You might want to try using wget or curl to download whatever you need for the build and put it in place

[share] How to mount host volumes into docker containers in Dockerfile during build

Examples related to docker

Examples related to host

Examples related to mount

[share] How to mount host volumes into docker containers in Dockerfile during build

Examples related to share

Examples related to docker

Examples related to host

Examples related to mount