How to include files outside of Docker s build context

Question

How can I include files from outside of Docker s build context using the  ADD  command in the Docker file   From the Docker documentation      The  path must be inside the context of the build  you cannot ADD      something something  because the first step of a docker build is to   send the context directory  and subdirectories  to the docker daemon    I do not want to restructure my whole project just to accommodate Docker in this matter  I want to keep all my Docker files in the same sub-directory   Also  it appears Docker does not yet  and may not ever  support symlinks  Dockerfile ADD command does not follow symlinks on host  1676    The only other thing I can think of is to include a pre-build step to copy the files into the Docker build context  and configure my version control to ignore those files   Is there a better workaround for than that

User · Accepted Answer

The best way to work around this is to specify the Dockerfile independently of the build context, using -f.

For instance, this command will give the ADD command access to anything in your current directory.

docker build -f docker-files/Dockerfile .

Update: Docker now allows having the Dockerfile outside the build context (fixed in 18.03.0-ce, https://github.com/docker/cli/pull/886). So you can also do something like

docker build -f ../Dockerfile .

User · Answer

Using docker-compose  I accomplished this by creating a service that mounts the volumes that I need and committing the image of the container  Then  in the subsequent service  I rely on the previously committed image  which has all of the data stored at mounted locations  You will then have have to copy these files to their ultimate destination  as host mounted directories do not get committed when running a docker commit command  You don t have to use docker-compose to accomplish this  but it makes life a bit easier    docker-compose yml  version   3    services      stage        image  alpine       volumes          -  host machine path  tmp container path       command  bash -c  quot cp -r  tmp container path  final container path quot      setup        image  stage    setup sh    Start  quot stage quot  service docker-compose up stage    Commit changes to an image named  quot stage quot  docker commit   docker-compose ps -q stage  stage    Start setup service off of stage image docker-compose up setup

User · Answer

An easy workaround might be to simply mount the volume  using the -v or --mount flag  to the container when you run it and access the files that way   example   docker run -v  path to file on host  desired path to file in container  image name   for more see  https   docs docker com storage volumes

User · Answer

I often find myself utilizing the --build-arg option for this purpose  For example after putting the following in the Dockerfile   ARG SSH KEY RUN echo   SSH KEY   gt   root  ssh id rsa   You can just do   docker build -t some-app --build-arg SSH KEY    cat   file outside build context id rsa       But note the following warning from the Docker documentation   Warning  It is not recommended to use build-time variables for passing secrets like github keys  user credentials etc  Build-time variable values are visible to any user of the image with the docker history command

User · Answer

In my case  my Dockerfile is written like a template containing placeholders which I m replacing with real value using my configuration file  So I couldn t specify this file directly but pipe it into the docker build like this  sed  quot s  email address   EMAIL ADDRESS   quot    Dockerfile   docker build -t katzda bookings latest   -f -   But because of the pipe  the COPY command didn t work  But the above way solves it by -f -  explicitly saying file not provided   Doing only - without the -f flag  the context AND the Dockerfile are not provided which is a caveat

User · Answer

If you read the discussion in the issue 2745 not only docker may never support symlinks they may never support adding files outside your context  Seems to be a design philosophy that files that go into docker build should explicitly be part of its context or be from a URL where it is presumably deployed too with a fixed version so that the build is repeatable with well known URLs or files shipped with the docker container       I prefer to build from a version controlled source - ie docker build   -t stuff http   my git org repo - otherwise I m building from some random place with random files       fundamentally  no      -- SvenDowideit  Docker Inc   Just my opinion but I think you should restructure to separate out the code and docker repositories  That way the containers can be generic and pull in any version of the code at run time rather than build time     Alternatively  use docker as your fundamental code deployment artifact and then you put the dockerfile in the root of the code repository  if you go this route probably makes sense to have a parent docker container for more general system level details and a child container for setup specific to your code

User · Answer

Workaround with links  ln path to file outside context file to copy   file to copy On Dockerfile  simply  COPY file to copy  path to file

User · Answer

I spent a good time trying to figure out  a good pattern and how to better explain what s going on with this feature support  I realized that the best way to explain it was as follows     Dockerfile  Will only see files under its own relative path Context  a place in  quot space quot  where the files you want to share and your Dockerfile will be copied to  So  with that said  here s an example of the Dockerfile that needs to reuse a file called start sh Dockerfile It will always load from its relative path  having the current directory of itself as the local reference to the paths you specify  COPY start sh  runtime start sh  Files Considering this idea  we can think of having multiple copies for the Dockerfiles building specific things  but they all need access to the start sh    all-services      start sh     service-X Dockerfile     service-Y Dockerfile     service-Z Dockerfile   docker-compose yaml  Considering this structure and the files above  here s a docker-compose yml docker-compose yaml  In this example  your shared context directory is the runtime directory   Same mental model here  think that all the files under this directory are moved over to the so-called context  Similarly  just specify the Dockerfile that you want to copy to that same directory  You can specify that using dockerfile    The directory where your main content is located is the actual context to be set   The docker-compose yml is as follows version   quot 3 3 quot  services       service-A     build        context    all-service       dockerfile    service-A Dockerfile    service-B     build        context    all-service       dockerfile    service-B Dockerfile    service-C     build        context    all-service       dockerfile    service-C Dockerfile   all-service is set as the context  the shared file start sh is copied there as well the Dockerfile specified by each dockerfile  Each gets to be built their own way  sharing the start file

User · Answer

I believe the simpler workaround would be to change the  context  itself    So  for example  instead of giving   docker build -t hello-demo-app     which sets the current directory as the context  let s say you wanted the parent directory as the context  just use   docker build -t hello-demo-app

User · Answer

As is described in this GitHub issue the build actually happens in  tmp docker-12345  so a relative path like    relative-add some-file is relative to  tmp docker-12345  It would thus search for  tmp relative-add some-file  which is also shown in the error message     It is not allowed to include files from outside the build directory  so this results in the  Forbidden path  message

User · Answer

You can also create a tarball of what the image needs first and use that as your context   https   docs docker com engine reference commandline build   tarball-contexts

User · Answer

I had this same issue with a project and some data files that I wasn t able to move inside the repo context for HIPAA reasons  I ended up using 2 Dockerfiles  One builds the main application without the stuff I needed outside the container and publishes that to internal repo  Then a second dockerfile pulls that image and adds the data and creates a new image which is then deployed and never stored anywhere   Not ideal  but it worked for my purposes of keeping sensitive information out of the repo

User · Answer

On Linux you can mount other directories instead of symlinking them  mount --bind olddir newdir   See https   superuser com questions 842642 for more details   I don t know if something similar is available for other OSes  I also tried using Samba to share a folder and remount it into the Docker context which worked as well

User · Answer

One quick and dirty way is to set the build context up as many levels as you need - but this can have consequences  If you re working in a microservices architecture that looks like this    Code Repo1   Code Repo2      You can set the build context to the parent Code directory and then access everything  but it turns out that  with a large number of repositories  this can result in the build taking a long time  An example situation could be that another team maintains a database schema in Repo1 and your team s code in Repo2 depends on this  You want to dockerise this dependency with some of your own seed data without worrying about schema changes or polluting the other team s repository  depending on what the changes are you may still have to change your seed data scripts of course  The second approach is hacky but gets around the issue of long builds  Create a sh  or ps1  script in   Code Repo2 to copy the files you need and invoke the docker commands you want  for example     bin bash rm -r   db schema mkdir   db schema  cp  -r    Repo1 db schema   db schema  docker-compose -f docker-compose yml down docker container prune -f docker-compose -f docker-compose yml up --build  In the docker-compose file  simply set the context as Repo2 root and use the content of the   db schema directory in your dockerfile without worrying about the path  Bear in mind that you will run the risk of accidentally committing this directory to source control  but scripting cleanup actions should be easy enough

[docker] How to include files outside of Docker's build context?

Examples related to docker