Using the RUN instruction in a Dockerfile with source does not work

Question

I have a Dockerfile that I am putting together to install a vanilla python environment  into which I will be installing an app  but at a later date    FROM ubuntu 12 04    required to build certain python libraries RUN apt-get install python-dev -y    install pip - canonical installation instructions from pip-installer org   http   www pip-installer org en latest installing html ADD https   bitbucket org pypa setuptools raw bootstrap ez setup py  tmp ez setup py ADD https   raw github com pypa pip master contrib get-pip py  tmp get-pip py RUN python  tmp ez setup py RUN python  tmp get-pip py RUN pip install --upgrade pip     install and configure virtualenv RUN pip install virtualenv  RUN pip install virtualenvwrapper ENV WORKON HOME    virtualenvs RUN mkdir -p  WORKON HOME RUN source  usr local bin virtualenvwrapper sh   The build runs ok until the last line  where I get the following exception    previous steps 1-9 removed for clarity      Successfully installed virtualenvwrapper virtualenv-clone stevedore Cleaning up     --- gt  1fc253a8f860 Step 10   ENV WORKON HOME    virtualenvs  --- gt  Running in 8b0145d2c80d  --- gt  0f91a5d96013 Step 11   RUN mkdir -p  WORKON HOME  --- gt  Running in 9d2552712ddf  --- gt  3a87364c7b45 Step 12   RUN source  usr local bin virtualenvwrapper sh  --- gt  Running in c13a187261ec  bin sh  1  source  not found   If I ls into that directory  just to test that the previous steps were committed  I can see that the files exist as expected     docker run 3a87 ls  usr local bin easy install easy install-2 7 pip pip-2 7 virtualenv virtualenv-2 7 virtualenv-clone virtualenvwrapper sh virtualenvwrapper lazy sh   If I try just running the source command I get the same  not found  error as above  If I RUN an interactive shell session however  source does work     docker run 3a87 bash source bash  line 1  source  filename argument required source  usage  source filename  arguments    I can run the script from here  and then happily access workon  mkvirtualenv etc   I ve done some digging  and initially it looked as if the problem might lie in the difference between bash as the Ubuntu login shell  and dash as the Ubuntu system shell  dash not supporting the source command   However  the answer to this appears to be to use     instead of source  but this just causes the Docker runtime to blow up with a go panic exception   What is the best way to run a shell script from a Dockerfile RUN instruction to get around this  am running off the default base image for Ubuntu 12 04 LTS

User · Answer

I ended up putting my env stuff in  profile and mutated SHELL something like  SHELL    bin bash    -c    -l      Install ruby version specified in  ruby-version RUN rvm install    lt  ruby-version     Install deps RUN rvm use    lt  ruby-version   amp  amp  gem install bundler  amp  amp  bundle install  CMD rvm use    lt  ruby-version   amp  amp    myscript rb

User · Answer

Original Answer  FROM ubuntu 14 04 RUN rm  bin sh  amp  amp  ln -s  bin bash  bin sh   This should work for every Ubuntu docker base image  I generally add this line for every Dockerfile I write   Edit by a concerned bystander  If you want to get the effect of  use bash instead of sh throughout this entire Dockerfile   without altering and possibly damaging  the OS inside the container  you can just tell Docker your intention  That is done like so   SHELL    bin bash    -c          The possible damage is that many scripts in Linux  on a fresh Ubuntu install grep -rHInE   bin sh    returns over 2700 results  expect a fully POSIX shell at  bin sh  The bash shell isn t just POSIX plus extra builtins  There are builtins  and more  that behave entirely different than those in POSIX  I FULLY support avoiding POSIX  and the fallacy that any script that you didn t test on another shell is going to work because you think you avoided basmisms  and just using bashism  But you do that with a proper shebang in your script  Not by pulling the POSIX shell out from under the entire OS   Unless you have time to verify all 2700 plus scripts that come with Linux plus all those in any packages you install     More detail in this answer below  https   stackoverflow com a 45087082 117471

User · Answer

If you have SHELL available you should go with this answer -- don t use the accepted one  which forces you to put the rest of the dockerfile in one command per this comment    If you are using an old Docker version and don t have access to SHELL  this will work so long as you don t need anything from  bashrc  which is a rare case in Dockerfiles    ENTRYPOINT   bash    --rcfile     usr local bin virtualenvwrapper sh    -ci     Note the -i is needed to make bash read the rcfile at all

User · Answer

Simplest way is to use the dot operator in place of source  which is the sh equivalent of the bash source command   Instead of   RUN source  usr local bin virtualenvwrapper sh   Use   RUN    usr local bin virtualenvwrapper sh

User · Answer

I also had issues in running source in a Dockerfile  This runs perfectly fine for building CentOS 6 6 Docker container  but gave issues in Debian containers  RUN cd ansible  amp  amp  source   hacking env-setup   This is how I tackled it  may not be an elegant way but this is what worked for me  RUN echo  source  ansible hacking env-setup   gt  gt   tmp setup RUN  bin bash -C   tmp setup  RUN rm -f  tmp setup

User · Answer

I ve dealing with a similar scenario for an application developed with Django web web framework and these are the steps that worked perfectly for me   content of my Dockerfile   mlazo srvjenkins project textile   cat docker Dockerfile debug  FROM malazo project textile ubuntu latest   ENV PROJECT DIR  proyectos project textile PROJECT NAME project textile WRAPPER PATH  usr share virtualenvwrapper virtualenvwrapper sh  COPY     PROJECT DIR   WORKDIR   PROJECT DIR   RUN echo  quot source   WRAPPER PATH  quot   gt     bashrc SHELL   quot  bin bash quot   quot -c quot   quot -l quot   RUN     mkvirtualenv -p   which python3    PROJECT NAME   amp  amp            workon   PROJECT NAME   amp  amp            pip3 install -r requirements txt   EXPOSE 8000  ENTRYPOINT   quot tests container entrypoint sh quot   CMD   quot public manage py quot   quot runserver quot   quot 0 8000 quot      content of the ENTRYPOINT file  quot tests container entrypoint sh quot     mlazo srvjenkins project textile   cat tests container entrypoint sh    bin bash    -  encoding   UTF-8  -  sh tests deliver env sh source    virtualenvs project textile bin activate  exec python  quot    quot     finally  the way I deploy the container was     mlazo srvjenkins project textile   cat   tests container deployment sh     bin bash  CONT NAME  quot cont app server quot  IMG NAME  quot malazo project textile app quot      docker ps -a  grep -i   CONT NAME   wc -l  -gt 0    amp  amp  docker rm -f   CONT NAME   docker run --name   CONT NAME  -p 8000 8000 -e DEBUG   DEBUG  -e MYSQL USER   MYSQL USER  -e MYSQL PASSWORD   MYSQL PASSWORD  -e MYSQL HOST   MYSQL HOST  -e MYSQL DATABASE   MYSQL DATABASE  -e MYSQL PORT   MYSQL PORT   -d   IMG NAME    I really hope this would be helpful for somebody else  Greetings

User · Answer

I had the same problem and in order to execute pip install inside virtualenv I had to use this command   RUN pip install virtualenv virtualenvwrapper RUN mkdir -p  opt virtualenvs ENV WORKON HOME  opt virtualenvs RUN  bin bash -c  source  usr local bin virtualenvwrapper sh        amp  amp  mkvirtualenv myapp        amp  amp  workon myapp        amp  amp  pip install -r  mycode myapp requirements txt    I hope it helps

User · Answer

You might want to run bash -v to see what s being sourced   I would do the following instead of playing with symlinks   RUN echo  source  usr local bin virtualenvwrapper sh   gt  gt   etc bash bashrc

User · Answer

RUN  bin bash -c  source  usr local bin virtualenvwrapper sh

User · Answer

According to https   docs docker com engine reference builder  run the default  Linux  shell for RUN is  bin sh -c  You appear to be expecting bashisms  so you should use the  exec form  of RUN to specify your shell   RUN    bin bash    -c    source  usr local bin virtualenvwrapper sh     Otherwise  using the  shell form  of RUN and specifying a different shell results in nested shells     don t do this    RUN  bin bash -c  source  usr local bin virtualenvwrapper sh    because it is the same as this    RUN    bin sh    -c     bin bash   -c   source  usr local bin virtualenvwrapper sh     If you have more than 1 command that needs a different shell  you should read https   docs docker com engine reference builder  shell and change your default shell by placing this before your RUN commands   SHELL    bin bash    -c     Finally  if you have placed anything in the root user s  bashrc file that you need  you can add the -l flag to the SHELL or RUN command to make it a login shell and ensure that it gets sourced   Note  I have intentionally ignored the fact that it is pointless to source a script as the only command in a RUN

User · Answer

Here is an example Dockerfile leveraging several clever techniques to all you to run a full conda environment for every RUN stanza  You can use a similar approach to execute any arbitrary prep in a script file  Note  there is a lot of nuance when it comes to login interactive vs nonlogin noninteractive shells  signals  exec  the way multiple args are handled  quoting  how CMD and ENTRYPOINT interact  and a million other things  so don t be discouraged if when hacking around with these things  stuff goes sideways  I ve spent many frustrating hours digging through all manner of literature and I still don t quite get how it all clicks     Conda with custom entrypoint from base ubuntu image    Build with e g   docker build -t monoconda       Run with  docker run --rm -it monoconda bash  to drop right into    the environment  foo    FROM ubuntu 18 04     Install things we need to install more things RUN apt-get update -qq  amp  amp       apt-get install -qq curl wget git  amp  amp       apt-get install -qq --no-install-recommends           libssl-dev           software-properties-common        amp  amp  rm -rf  var lib apt lists       Install miniconda RUN wget -nv https   repo anaconda com miniconda Miniconda3-4 7 12-Linux-x86 64 sh -O   miniconda sh  amp  amp         bin bash   miniconda sh -b -p  opt conda  amp  amp        rm   miniconda sh  amp  amp         opt conda bin conda clean -tipsy  amp  amp        ln -s  opt conda etc profile d conda sh  etc profile d conda sh     add conda to the path so we can execute it by name ENV PATH  opt conda bin  PATH     Create  entry sh which will be our new shell entry point  This performs actions to configure the environment    before starting a new shell  which inherits the env      The exec is important  This allows signals to pass RUN      echo     bin bash         amp  amp    echo    conda setup  quot    opt conda bin conda shell bash hook 2 gt   dev null  quot          amp  amp    echo  eval  quot    conda setup quot          amp  amp    echo  conda activate  quot   CONDA TARGET ENV -base  quot          amp  amp    echo   gt  amp 2 echo  quot ENTRYPOINT  CONDA DEFAULT ENV   CONDA DEFAULT ENV  quot          amp  amp    echo  exec  quot    quot               gt  gt   entry sh  amp  amp  chmod  x  entry sh     Tell the docker build process to use this for RUN     The default shell on Linux is   quot  bin sh quot    quot -c quot    and on Windows is   quot cmd quot    quot  S quot    quot  C quot   SHELL   quot  entry sh quot    quot  bin bash quot    quot -c quot      Now  every following invocation of RUN will start with the entry script RUN     conda update conda -y     Create a dummy env RUN     conda create --name foo     I added this variable such that I have the entry script activate a specific env ENV CONDA TARGET ENV foo     This will get installed in the env foo since it gets activated at the start of the RUN stanza RUN  conda install pip     Configure  bashrc to drop into a conda env and immediately activate our TARGET env RUN conda init  amp  amp  echo  conda activate  quot   CONDA TARGET ENV -base  quot    gt  gt      bashrc ENTRYPOINT   quot  entry sh quot

User · Answer

The default shell for the RUN instruction is    bin sh    -c     RUN  source file         translates to  RUN  bin sh -c  source file    Using SHELL instruction  you can change default shell for subsequent RUN instructions in Dockerfile   SHELL    bin bash    -c      Now  default shell has changed and you don t need to explicitly define it in every RUN instruction  RUN  source file       now translates to  RUN  bin bash -c  source file    Additional Note  You could also add --login option which would start a login shell  This means    bachrc for example would be read and you don t need to source it explicitly before your command

User · Answer

If you are using Docker 1 12 or newer  just use SHELL   Short Answer  general  SHELL   quot  bin bash quot    quot -c quot     for python vituralenv  SHELL   quot  bin bash quot    quot -c quot    quot source  usr local bin virtualenvwrapper sh quot    Long Answer  from https   docs docker com engine reference builder  shell  SHELL   quot executable quot    quot parameters quot    The SHELL instruction allows the default shell used for the shell form of commands to be overridden  The default shell on Linux is   quot  bin sh quot    quot -c quot    and on Windows is   quot cmd quot    quot  S quot    quot  C quot    The SHELL instruction must be written in JSON form in a Dockerfile  The SHELL instruction is particularly useful on Windows where there are two commonly used and quite different native shells  cmd and powershell  as well as alternate shells available including sh  The SHELL instruction can appear multiple times  Each SHELL instruction overrides all previous SHELL instructions  and affects all subsequent instructions  For example  FROM microsoft windowsservercore    Executed as cmd  S  C echo default RUN echo default    Executed as cmd  S  C powershell -command Write-Host default RUN powershell -command Write-Host default    Executed as powershell -command Write-Host hello SHELL   quot powershell quot    quot -command quot   RUN Write-Host hello    Executed as cmd  S  C echo hello SHELL   quot cmd quot    quot  S quot  quot    quot  C quot   RUN echo hello  The following instructions can be affected by the SHELL instruction when the shell form of them is used in a Dockerfile  RUN  CMD and ENTRYPOINT  The following example is a common pattern found on Windows which can be streamlined by using the SHELL instruction      RUN powershell -command Execute-MyCmdlet -param1  quot c  foo txt quot       The command invoked by docker will be  cmd  S  C powershell -command Execute-MyCmdlet -param1  quot c  foo txt quot   This is inefficient for two reasons  First  there is an un-necessary cmd exe command processor  aka shell  being invoked  Second  each RUN instruction in the shell form requires an extra powershell -command prefixing the command  To make this more efficient  one of two mechanisms can be employed  One is to use the JSON form of the RUN command such as      RUN   quot powershell quot    quot -command quot    quot Execute-MyCmdlet quot    quot -param1   quot c   foo txt  quot  quot        While the JSON form is unambiguous and does not use the un-necessary cmd exe  it does require more verbosity through double-quoting and escaping  The alternate mechanism is to use the SHELL instruction and the shell form  making a more natural syntax for Windows users  especially when combined with the escape parser directive    escape    FROM microsoft nanoserver SHELL   quot powershell quot   quot -command quot   RUN New-Item -ItemType Directory C  Example ADD Execute-MyCmdlet ps1 c  example  RUN c  example Execute-MyCmdlet -sample  hello world   Resulting in  PS E  docker build shell gt  docker build -t shell   Sending build context to Docker daemon 4 096 kB Step 1 5   FROM microsoft nanoserver  --- gt  22738ff49c6d Step 2 5   SHELL powershell -command  --- gt  Running in 6fcdb6855ae2  --- gt  6331462d4300 Removing intermediate container 6fcdb6855ae2 Step 3 5   RUN New-Item -ItemType Directory C  Example  --- gt  Running in d0eef8386e97       Directory  C     Mode                LastWriteTime         Length Name ----                -------------         ------ ---- d-----       10 28 2016  11 26 AM                Example    --- gt  3f2fbf1395d9 Removing intermediate container d0eef8386e97 Step 4 5   ADD Execute-MyCmdlet ps1 c  example   --- gt  a955b2621c31 Removing intermediate container b825593d39fc Step 5 5   RUN c  example Execute-MyCmdlet  hello world   --- gt  Running in be6d8e63fe75 hello world  --- gt  8e559e9bf424 Removing intermediate container be6d8e63fe75 Successfully built 8e559e9bf424 PS E  docker build shell gt   The SHELL instruction could also be used to modify the way in which a shell operates  For example  using SHELL cmd  S  C  V ON OFF on Windows  delayed environment variable expansion semantics could be modified  The SHELL instruction can also be used on Linux should an alternate shell be required such as zsh  csh  tcsh and others  The SHELL feature was added in Docker 1 12

User · Answer

According to Docker documentation     To use a different shell  other than     bin sh     use the exec form passing in the desired shell  For example   RUN    bin bash    -c    echo hello      See https   docs docker com engine reference builder  run

User · Answer

Building on the answers on this page I would add that you have to be aware that each RUN statement runs independently of the others with  bin sh -c and therefore won t get any environment vars that would normally be sourced in login shells   The best way I have found so far is to add the script to  etc bash bashrc and then invoke each command as bash login   RUN echo  source  usr local bin virtualenvwrapper sh   gt  gt   etc bash bashrc RUN  bin bash --login -c  your command    You could for instance install and setup virtualenvwrapper  create the virtual env  have it activate when you use a bash login  and then install your python modules into this env   RUN pip install virtualenv virtualenvwrapper RUN mkdir -p  opt virtualenvs ENV WORKON HOME  opt virtualenvs RUN echo  source  usr local bin virtualenvwrapper sh   gt  gt   etc bash bashrc RUN  bin bash --login -c  mkvirtualenv myapp  RUN echo  workon mpyapp   gt  gt   etc bash bashrc RUN  bin bash --login -c  pip install        Reading the manual on bash startup files helps understand what is sourced when

User · Answer

This might be happening because source is a built-in to bash rather than a binary somewhere on the filesystem   Is your intention for the script you re sourcing to alter the container afterward

User · Answer

If you re just trying to use pip to install something into the virtualenv  you can modify the PATH env to look in the virtualenv s bin folder first  ENV PATH   path to venv bin   PATH     Then any pip install commands that follow in the Dockerfile will find  path to venv bin pip first and use that  which will install into that virtualenv and not the system python

[bash] Using the RUN instruction in a Dockerfile with 'source' does not work

Examples related to bash

Examples related to shell

Examples related to docker