Clone private git repo with dockerfile

Question

I have copied this code from what seems to be various working dockerfiles around  here is mine   FROM ubuntu  MAINTAINER Luke Crooks  luke pumalo org     Update aptitude with new repo RUN apt-get update    Install software  RUN apt-get install -y git python-virtualenv    Make ssh dir RUN mkdir  root  ssh     Copy over private key  and set permissions ADD id rsa  root  ssh id rsa RUN chmod 700  root  ssh id rsa RUN chown -R root root  root  ssh    Create known hosts RUN touch  root  ssh known hosts    Remove host checking RUN echo  Host bitbucket org n tStrictHostKeyChecking no n   gt  gt   root  ssh config    Clone the conf files into the docker container RUN git clone git bitbucket org Pumalo docker-conf git  home docker-conf   This gives me the error  Step 10   RUN git clone git bitbucket org Pumalo docker-conf git  home docker-conf  --- gt  Running in 0d244d812a54 Cloning into   home docker-conf     Warning  Permanently added  bitbucket org 131 103 20 167   RSA  to the list of known hosts  Permission denied  publickey   fatal  Could not read from remote repository   Please make sure you have the correct access rights and the repository exists  2014 04 30 16 07 28 The command   bin sh -c git clone git bitbucket org Pumalo docker-conf git  home docker-conf  returned a non-zero code  128   This is my first time using dockerfiles  but from what I have read  and taken from working configs  I cannot see why this doesn t work   My id rsa is in the same folder as my dockerfile and is a copy of my local key which can clone this repo no problem   Edit    In my dockerfile I can add   RUN cat  root  ssh id rsa   And it prints out the correct key  so I know its being copied correctly   I have also tried to do as noah advised and ran   RUN echo  Host bitbucket org n tIdentityFile  root  ssh id rsa n tStrictHostKeyChecking no   gt  gt   etc ssh ssh config   This sadly also doesn t work

User · Accepted Answer

My key was password protected which was causing the problem  a working file is now listed below  for help of future googlers   FROM ubuntu  MAINTAINER Luke Crooks  luke pumalo org     Update aptitude with new repo RUN apt-get update    Install software  RUN apt-get install -y git   Make ssh dir RUN mkdir  root  ssh     Copy over private key  and set permissions   Warning  Anyone who gets their hands on this image will be able   to retrieve this private key file from the corresponding image layer ADD id rsa  root  ssh id rsa    Create known hosts RUN touch  root  ssh known hosts   Add bitbuckets key RUN ssh-keyscan bitbucket org  gt  gt   root  ssh known hosts    Clone the conf files into the docker container RUN git clone git bitbucket org User repo git

User · Answer

There s no need to fiddle around with ssh configurations  Use a configuration file  not a Dockerfile  that contains environment variables  and have a shell script update your docker file at runtime  You keep tokens out of your Dockerfiles and you can clone over https  no need to generate or pass around ssh keys    Go to Settings   Personal Access Tokens   Generate a personal access token with repo scope enabled  Clone like this  git clone https   MY TOKEN github com user-or-org repo   Some commenters have noted that if you use a shared Dockerfile  this could expose your access key to other people on your project  While this may or may not be a concern for your specific use case  here are some ways you can deal with that    Use a shell script to accept arguments which could contain your key as a variable  Replace a variable in your Dockerfile with sed or similar  i e  calling the script with sh rundocker sh MYTOKEN foo which would replace on https     MY TOKEN   github com user-or-org repo  Note that you could also use a configuration file  in  yml or whatever format you want  to do the same thing but with environment variables  Create a github user  and generate an access token for  for that project only

User · Answer

Another option is to use a multi-stage docker build to ensure that your SSH keys are not included in the final image   As described in my post you can prepare your intermediate image with the required dependencies to git clone and then COPY the required files into your final image   Additionally if we LABEL our intermediate layers  we can even delete them from the machine when finished     Choose and name our temporary image  FROM alpine as intermediate   Add metadata identifying these images as our build containers  this will be useful later   LABEL stage intermediate    Take an SSH key as a build argument  ARG SSH KEY    Install dependencies required to git clone  RUN apk update  amp  amp        apk add --update git  amp  amp        apk add --update openssh    1  Create the SSH directory    2  Populate the private key file    3  Set the required permissions    4  Add github to our list of known hosts for ssh  RUN mkdir -p  root  ssh   amp  amp        echo   SSH KEY   gt   root  ssh id rsa  amp  amp        chmod -R 600  root  ssh   amp  amp        ssh-keyscan -t rsa github com  gt  gt     ssh known hosts    Clone a repository  my website in this case  RUN git clone git github com janakerman janakerman git    Choose the base image for our final image FROM alpine    Copy across the files from our  intermediate  container RUN mkdir files COPY --from intermediate  janakerman README md  files README md   We can then build   MY KEY   cat    ssh id rsa  docker build --build-arg SSH KEY   MY KEY  --tag clone-example     Prove our SSH keys are gone   docker run -ti --rm clone-example cat  root  ssh id rsa   Clean intermediate images from the build machine   docker rmi -f   docker images -q --filter label stage intermediate

User · Answer

You often do not want to perform a git clone of a private repo from within the docker build  Doing the clone there involves placing the private ssh credentials inside the image where they can be later extracted by anyone with access to your image   Instead  the common practice is to clone the git repo from outside of docker in your CI tool of choice  and simply COPY the files into the image  This has a second benefit  docker caching  Docker caching looks at the command being run  environment variables it includes  input files  etc  and if they are identical to a previous build from the same parent step  it reuses that previous cache  With a git clone command  the command itself is identical  so docker will reuse the cache even if the external git repo is changed  However  a COPY command will look at the files in the build context and can see if they are identical or have been updated  and use the cache only when it s appropriate     If you are going to add credentials into your build  consider doing so with a multi-stage build  and only placing those credentials in an early stage that is never tagged and pushed outside of your build host  The result looks like   FROM ubuntu as clone    Update aptitude with new repo RUN apt-get update     amp  amp  apt-get install -y git   Make ssh dir   Create known hosts   Add bitbuckets key RUN mkdir  root  ssh      amp  amp  touch  root  ssh known hosts     amp  amp  ssh-keyscan bitbucket org  gt  gt   root  ssh known hosts    Copy over private key  and set permissions   Warning  Anyone who gets their hands on this image will be able   to retrieve this private key file from the corresponding image layer COPY id rsa  root  ssh id rsa    Clone the conf files into the docker container RUN git clone git bitbucket org User repo git  FROM ubuntu as release LABEL maintainer  Luke Crooks  lt luke pumalo org gt    COPY --from clone  repo  repo         More recently  BuildKit has been testing some experimental features that allow you to pass an ssh key in as a mount that never gets written to the image     syntax docker dockerfile experimental FROM ubuntu as clone LABEL maintainer  Luke Crooks  lt luke pumalo org gt      Update aptitude with new repo RUN apt-get update     amp  amp  apt-get install -y git    Make ssh dir   Create known hosts   Add bitbuckets key RUN mkdir  root  ssh      amp  amp  touch  root  ssh known hosts     amp  amp  ssh-keyscan bitbucket org  gt  gt   root  ssh known hosts    Clone the conf files into the docker container RUN --mount type secret id ssh id target  root  ssh id rsa       git clone git bitbucket org User repo git   And you can build that with     DOCKER BUILDKIT 1 docker build -t your image name     --secret id ssh id src   pwd  id rsa     Note that this still requires your ssh key to not be password protected  but you can at least run the build in a single stage  removing a COPY command  and avoiding the ssh credential from ever being part of an image     BuildKit also added a feature just for ssh which allows you to still have your password protected ssh keys  the result looks like     syntax docker dockerfile experimental FROM ubuntu as clone LABEL maintainer  Luke Crooks  lt luke pumalo org gt      Update aptitude with new repo RUN apt-get update     amp  amp  apt-get install -y git    Make ssh dir   Create known hosts   Add bitbuckets key RUN mkdir  root  ssh      amp  amp  touch  root  ssh known hosts     amp  amp  ssh-keyscan bitbucket org  gt  gt   root  ssh known hosts    Clone the conf files into the docker container RUN --mount type ssh       git clone git bitbucket org User repo git   And you can build that with     eval   ssh-agent    ssh-add    ssh id rsa  Input your passphrase here    DOCKER BUILDKIT 1 docker build -t your image name     --ssh default  SSH AUTH SOCK     Again  this is injected into the build without ever being written to an image layer  removing the risk that the credential could accidentally leak out     To force docker to run the git clone even when the lines before have been cached  you can inject a build ARG that changes with each build to break the cache  That looks like     inject a datestamp arg which is treated as an environment variable and   will break the cache for the next RUN command ARG DATE STAMP   Clone the conf files into the docker container RUN git clone git bitbucket org User repo git   Then you inject that changing arg in the docker build command   date stamp   date   Y m d- H M S  docker build --build-arg DATE STAMP  date stamp

User · Answer

You should create new SSH key set for that Docker image  as you probably don t want to embed there your own private key  To make it work  you ll have to add that key to deployment keys in your git repository  Here s complete recipe    Generate ssh keys with ssh-keygen -q -t rsa -N    -f repo-key which will give you repo-key and repo-key pub files  Add repo-key pub to your repository deployment keys  On GitHub  go to  your repository  -  Settings -  Deploy keys Add something like this to your Dockerfile     ADD repo-key   RUN     chmod 600  repo-key          echo  IdentityFile  repo-key      etc ssh ssh config          echo -e  StrictHostKeyChecking no      etc ssh ssh config             your git clone commands here       Note that above switches off StrictHostKeyChecking  so you don t need  ssh known hosts  Although I probably like more the solution with ssh-keyscan in one of the answers above

User · Answer

Above solutions did not work for bitbucket  I figured this does the trick   RUN ssh-keyscan bitbucket org  gt  gt   root  ssh known hosts        amp  amp  eval  ssh-agent         amp  amp  ssh-add    ssh  key         amp  amp  git clone git bitbucket org  team   repo  git

User · Answer

For bitbucket repository  generate App Password  Bitbucket settings -  Access Management -  App Password  see the image  with read access to the repo and project     Then the command that you should use is   git clone https   username generated password bitbucket org reponame projectname git

[git] Clone private git repo with dockerfile

Examples related to git

Examples related to bitbucket

Examples related to docker