How to automatically update your docker containers if base-images are updated

Question

Say I have a trivial container based on the ubuntu latest  Now there is a security update and ubuntu latest is updated in the docker repo     How would I know my local image and its containers are running behind   Is there some best practice for automatically updating local images and containers to follow the docker repo updates  which in practice would give you the same niceties of having unattended-upgrades running on a conventional ubuntu-machine

User · Answer

Dependency management for Docker images is a real problem  I m part of a team that built a tool  MicroBadger  to help with this by monitoring container images and inspecting metadata  One of its features is to let you set up a notification webhook that gets called when an image you re interested in  e g  a base image  changes

User · Answer

You can use Watchtower to watch for updates to the image a container is instantiated from and automatically pull the update and restart the container using the updated image  However  that doesn t solve the problem of rebuilding your own custom images when there s a change to the upstream image it s based on  You could view this as a two-part problem   1  knowing when an upstream image has been updated  and  2  doing the actual image rebuild   1  can be solved fairly easily  but  2  depends a lot on your local build environment practices  so it s probably much harder to create a generalized solution for that  If you re able to use Docker Hub s automated builds  the whole problem can be solved relatively cleanly using the repository links feature  which lets you trigger a rebuild automatically when a linked repository  probably an upstream one  is updated  You can also configure a webhook to notify you when an automated build occurs  If you want an email or SMS notification  you could connect the webhook to IFTTT Maker  I found the IFTTT user interface to be kind of confusing  but you would configure the Docker webhook to post to https   maker ifttt com trigger  docker xyz image built  with key  your key   If you need to build locally  you can at least solve the problem of getting notifications when an upstream image is updated by creating a dummy repo in Docker Hub linked to your repo s  of interest  The sole purpose of the dummy repo would be to trigger a webhook when it gets rebuilt  which implies one of its linked repos was updated   If you re able to receive this webhook  you could even use that to trigger a rebuild on your side

User · Answer

UPDATE  Use Dependabot - https   dependabot com docker   BLUF  finding the right insertion point for monitoring changes to a container is the challenge   It would be great if DockerHub would solve this    Repository Links have been mentioned but note when setting them up on DockerHub -  Trigger a build in this repository whenever the base image is updated on Docker Hub  Only works for non-official images     While trying to solve this myself I saw several recommendations for webhooks so I wanted to elaborate on a couple of solutions I have used    Use microbadger com to track changes in a container and use it s notification webhook feature to trigger an action   I set this up with zapier com  but you can use any customizable webhook service  to create a new issue in my github repository that uses Alpine as a base image    Pros  You can review the changes reported by microbadger in github before taking action  Cons  Microbadger doesn t let you track a specific tag   Looks like it only tracks  latest    Track the RSS feed for git commits to an upstream container   ex  https   github com gliderlabs docker-alpine commits rootfs library-3 8 x86 64   I used zapier com to monitor this feed and to trigger an automatic build of my container in Travis-CI anytime something is committed   This is a little extreme but you can change the trigger to do other things like open an issue in your git repository for manual intervention    Pros  Closer to an automated pipline   The Travis-CI build just checks to see if your container has issues with whatever was committed to the base image repository   It s up to you if your CI service takes any further action  Cons  Tracking the commit feed isn t perfect   Lots of things get committed to the repository that don t affect the build of the base image   Doesn t take in to account any issues with frequency number of commits and any API throttling

User · Answer

Another approach could be to assume that your base image gets behind quite quickly  and that s very likely to happen   and force another image build of your application periodically  e g  every week  and then re-deploy it if it has changed   As far as I can tell  popular base images like the official Debian or Java update their tags to cater for security fixes  so tags are not immutable  if you want a stronger guarantee of that you need to use the reference  image  digest   available in more recent Docker versions   Therefore  if you were to build your image with  docker build --pull  then your application should get the latest and greatest of the base image tag you re referencing   Since mutable tags can be confusing  it s best to increment the version number of your application every time you do this so that at least on your side things are cleaner   So I m not sure that the script suggested in one of the previous answers does the job  since it doesn t rebuild you application s image - it just updates the base image tag and then it restarts the container  but the new container still references the old base image hash   I wouldn t advocate for running cron-type jobs in containers  or any other processes  unless really necessary  as this goes against the mantra of running only one process per container  there are various arguments about why this is better  so I m not going to go into it here

User · Answer

A simple and great solution is shepherd

User · Answer

I had the same issue and thought it can be simply solved by a cron job calling unattended-upgrade daily   My intention is to have this as an automatic and quick solution to ensure that production container is secure and updated because it can take me sometime to update my images and deploy a new docker image with the latest security updates    It is also possible to automate the image build and deployment with Github hooks  I ve created a basic docker image with that automatically checks and installs security updates daily  can run directly by docker run itech docker-unattended-upgrade      I also came across another different approach to check if the container needs an update   My complete implementation   Dockerfile  FROM ubuntu 14 04     RUN apt-get update    amp  amp  apt-get install -y supervisor unattended-upgrades    amp  amp  rm -rf  var lib apt lists    COPY install  install RUN chmod 755 install RUN  install  COPY start  start RUN chmod 755  start   Helper scripts  install     bin bash set -e  cat  gt   etc supervisor conf d cron conf  lt  lt EOF  program cron  priority 20 directory  tmp command  usr sbin cron -f user root autostart true autorestart true stdout logfile  var log supervisor   program name s log stderr logfile  var log supervisor   program name s log EOF  rm -rf  var lib apt lists    ENTRYPOINT    start     start     bin bash  set -e  echo  Adding crontab for unattended-upgrade      echo  0 0       root  usr bin unattended-upgrade   gt  gt   etc crontab    can also use  daily syntax or use  etc cron daily  echo  Starting supervisord      exec  usr bin supervisord -n -c  etc supervisor supervisord conf   Edit  I developed a small tool docker-run that runs as docker container and can be used to update packages inside all or selected running containers  it can also be used to run any arbitrary commands   Can be easily tested with the following command   docker run --rm -v  var run docker sock  tmp docker sock itech docker-run exec  which by default will execute date command in all running containers and display the results  If you pass update instead of exec it will execute apt-get update followed by apt-get upgrade -y in all running containers

User · Answer

We use a script which checks if a running container is started with the latest image  We also use upstart init scripts for starting the docker image      usr bin env bash set -e BASE IMAGE  registry  REGISTRY  registry hub docker com  IMAGE   REGISTRY  BASE IMAGE  CID   docker ps   grep  IMAGE   awk   print  1    docker pull  IMAGE  for im in  CID do     LATEST  docker inspect --format     Id     IMAGE      RUNNING  docker inspect --format     Image     im      NAME  docker inspect --format     Name     im   sed  s     g       echo  Latest    LATEST     echo  Running    RUNNING     if     RUNNING       LATEST    then         echo  upgrading  NAME          stop docker- NAME         docker rm -f  NAME         start docker- NAME     else         echo   NAME up to date      fi done   And init looks like   docker run -t -i --name  NAME  im  bin bash

User · Answer

Premise to my answer    Containers are run with tags  The same tag can be pointed to different image UUID as we please  feel appropriate  Updates done to an image can be committed to a new image layer   Approach   Build all the containers in the first place with a security-patch update script Build an automated process for the following   Run an existing image to new container with security patch script as the command Commit changes to the image as   existing tag -  followed by restarting the containers one by one new version tag -  replace few containers with new tag -  validate -  move all containers to new tag     Additionally  the base image can be upgraded  the container with a complete new base image can be built at regular intervals  as the maintainer feels necessary  Advantages   We are preserving the old version of the image while creating the new security patched image  hence we can rollback to previous running image if necessary We are preserving the docker cache  hence less network transfer  only the changed layer gets on the wire  The upgrade process can be validated in staging before moving to prod This can be a controlled process  hence the security patches only when necessary  deemed important can be pushed

User · Answer

There are a lot of answers here  but none of them suited my needs  I wanted an actual answer to the asker s  1 question  How do I know when an image is updated on hub docker com   The below script can be run daily  On first run  it gets a baseline of the tags and update dates from the HUB registry and saves them locally  From then out  every time it is run it checks the registry for new tags and update dates  Since this changes every time a new image exists  it tells us if the base image has changed  Here is the script      bin bash  DATAPATH   data docker updater data   if     -d    DATAPATH      then         mkdir    DATAPATH    fi IMAGES   docker ps --format     Image     for IMAGE in  IMAGES  do         ORIGIMAGE   IMAGE          if      IMAGE              then                 IMAGE library   IMAGE          fi         IMAGE   IMAGE              echo  Checking   IMAGE           PARSED   IMAGE                if     -f    DATAPATH    PARSED      then                   File doesn t exist yet  make baseline                 echo  Setting baseline for   IMAGE                   curl -s  https   registry hub docker com v2 repositories   IMAGE  tags    gt     DATAPATH    PARSED           else                   File does exist  do a compare                 NEW   curl -s  https   registry hub docker com v2 repositories   IMAGE  tags                    OLD   cat    DATAPATH    PARSED                    if       VAR1         VAR2       then                         echo  Image   IMAGE  is up to date                   else                         echo   NEW   gt     DATAPATH    PARSED                           echo  Image   IMAGE  needs to be updated                           H  hostname                          ssh -i  data keys  lt KEYFILE gt   lt USER gt   lt REMOTEHOST gt  com    echo   MAIL FROM  root   H     echo   RCPT TO   lt USER gt   lt EMAILHOST gt  com    echo   DATA    echo   Subject    H  -   IMAGE  needs update    echo       echo -e    n  IMAGE  needs update  n ndocker pull   ORIGIMAGE     echo       echo        echo   quit    sleep 1      telnet  lt SMTPHOST gt  25                  fi          fi done    You will want to alter the DATAPATH variable at the top  and alter the email notification command at the end to suit your needs  For me  I have it SSH into a server on another network where my SMTP is located  But you could easily use the mail command  too   Now  you also want to check for updated packages inside the containers themselves  This is actually probably more effective than doing a  pull  once your containers are working  Here s the script to pull that off      bin bash   function needsUpdates             RESULT   docker exec   1  bash -c                     if    -f  etc apt sources list     then                   grep security  etc apt sources list  gt   tmp security list                    apt-get update  gt   dev null                    apt-get upgrade -oDir  Etc  Sourcelist  tmp security list -s                    fi                               RESULT   echo  RESULT          GOODRESULT  Reading package lists    Building dependency tree    Reading state information    Calculating upgrade    0 upgraded  0 newly installed  0 to remove and 0 not upgraded           if       RESULT             amp  amp        RESULT         GOODRESULT       then                 return 0         else                 return 1         fi    function sendEmail             echo  Container   1  needs security updates           H  hostname          ssh -i  data keys  lt KEYFILE gt   lt USRER gt   lt REMOTEHOST gt  com    echo   MAIL FROM  root   H     echo   RCPT TO   lt USER gt   lt EMAILHOST gt  com    echo   DATA    echo   Subject    H  -   1  container needs security update    echo       echo -e    n  1  container needs update  n n    echo -e   docker exec   1  bash -c  grep security  etc apt sources list  gt   tmp security list  apt-get update  gt   dev null  apt-get upgrade -oDir  Etc  Sourcelist  tmp security list -s  n n    echo   Remove the -s to run the update    echo       echo        echo   quit    sleep 1      telnet  lt SMTPHOST gt  25     CONTAINERS   docker ps --format     Names     for CONTAINER in  CONTAINERS  do         echo  Checking   CONTAINER           if needsUpdates  CONTAINER  then                 sendEmail  CONTAINER         fi done

User · Answer

I m not going into the whole question of whether or not you want unattended updates in production  I think not   I m just leaving this here for reference in case anybody finds it useful  Update all your docker images to the latest version with the following command in your terminal     docker images   awk   NR gt 1   amp  amp    2   none    print  1    2     xargs -L1 docker pull

User · Answer

Here is a simplest way to update docker container automatically   Put the job via   crontab -e   0         sh    docker cron sh   Create dir    docker with file cron sh      bin sh if grep -Fqe  Image is up to date   lt  lt  EOF  docker pull ubuntu latest  EOF then     echo  no update  just do cleaning      docker system prune --force else     echo  newest exist  recompose       cd  path to your compose file     docker-compose down --volumes     docker-compose up -d fi

User · Answer

Above Answers are also correct  There are two Approach         Use webhooks   Run script for every specific minute to get fresh pull of docker images      I am just sharing script may be it will helpful for you  You can use it with cronjob  I tried succesfully on OSX     bin bash   You can use below commented line for setting cron tab for running cron job and to store its O P in one  txt file               usr bin sudo -u admin -i bash -c  Users Swapnil Documents checkimg sh  gt   Users Swapnil Documents cron output log 2 gt  amp 1   Example for the Docker Hub V2 API   Returns all images and tags associated with a Docker Hub organization account    Requires  jq   https   stedolan github io jq     set username  password  and organization   Filepath where your docker-compose file is present FILEPATH   Users Swapnil Documents lamp-alpine    Your Docker hub user name UNAME  ur username    Your Docker hub user password UPASS  ur pwd    e g organisation name image name image tag ORG  ur org name  IMGNAME  ur img name  IMGTAG  ur img tag    Container name CONTNAME  ur container name    Expected built mins BUILDMINS  5   Generally cronjob frequency CHECKTIME  5  NETWORKNAME    IMGNAME  private-network   After Image pulling  need to bring up all docker services  DO DOCKER COMPOSE UP true   ------- echo  Eecuting Script   date and time in YmdHMS    date   Y m d H M S   set -e PIDFILE  Users Swapnil Documents  IMGNAME forever pid if   -f  PIDFILE   then   PID   cat  PIDFILE    ps -p  PID  gt   dev null 2 gt  amp 1   if      -eq 0     then     echo  Process already running      exit 1   else        Process not found assume not running     echo        echo     gt   PIDFILE     if      -ne 0       then       echo  Could not create PID file        exit 1     fi   fi else   echo     gt   PIDFILE   if      -ne 0     then     echo  Could not create PID file      exit 1   fi fi    Check Docker is running or not  If not runing then exit if docker info grep Containers   then     echo  Docker is running  else     echo  Docker is not running      rm  PIDFILE     exit 1 fi    Check Container is running or not  and set variable CONT INFO   docker ps -f  name  CONTNAME  --format     Names     if     CONT INFO      CONTNAME     then     echo  Container is running      IS CONTAINER RUNNING true else     echo  Container is not running      IS CONTAINER RUNNING false fi     get token echo  Retrieving token      TOKEN   curl -s -H  Content-Type  application json  -X POST -d    username       UNAME      password       UPASS      https   hub docker com v2 users login    jq -r  token     get list of repositories echo  Retrieving repository list      REPO LIST   curl -s -H  Authorization  JWT   TOKEN   https   hub docker com v2 repositories   ORG   page size 100   jq -r   results      name      output images  amp  tags echo  Images and tags for organization    ORG   echo for i in   REPO LIST  do   echo    i        tags   IMAGE TAGS   curl -s -H  Authorization  JWT   TOKEN   https   hub docker com v2 repositories   ORG    i  tags  page size 100   jq -r   results      name     for j in   IMAGE TAGS    do     echo    -   j     done    echo done    Check Perticular image is the latest or not  imm   curl -s -H  Authorization  JWT   TOKEN   https   hub docker com v2 repositories   ORG    IMGNAME  tags  page size 100  echo  -----------------  echo  Last built date details about Image   IMGNAME      IMGTAG  for organization    ORG   IMAGE UPDATED DATE   curl -s -H  Authorization  JWT   TOKEN   https   hub docker com v2 repositories   ORG    IMGNAME  tags  page size 100   jq -r   results     select  name   contains     IMGTAG      last updated   echo  On Docker Hub IMAGE UPDATED DATE--- IMAGE UPDATED DATE  echo  -----------------   IMAGE CREATED DATE   docker image inspect   ORG    IMGNAME    IMGTAG    jq -r       Created   echo  Locally IMAGE CREATED DATE--- IMAGE CREATED DATE   updatedDate   date -jf   Y- m- dT H  M     IMAGE UPDATED DATE 0 16     Y m d H M S   createdDate   date -jf   Y- m- dT H  M     IMAGE CREATED DATE 0 16     Y m d H M S  currentDate   date   Y m d H M S   start date   date -jf   Y m d H M S    currentDate     s   end date   date -jf   Y m d H M S    updatedDate     s   updiffMins       start date -  end date     60     if      updiffMins  -lt     CHECKTIME 1       then         if     -d    FILEPATH      then             mkdir    FILEPATH            fi         cd    FILEPATH           pwd         echo  updatedDate--- updatedDate   gt   ScriptOutput   currentDate  txt          echo  createdDate--- createdDate   gt  gt   ScriptOutput   currentDate  txt          echo  currentDate--- currentDate   gt  gt   ScriptOutput   currentDate  txt          echo  Found after regular checking time - gt  Docker hub s latest updated image is new  Diff   updiffMins  mins   gt  gt   ScriptOutput   currentDate  txt          echo  Script is checking for latest updates after every   CHECKTIME  mins   gt  gt   ScriptOutput   currentDate  txt          echo  Fetching all new          echo  ---------------------------          if  IS CONTAINER RUNNING   then             echo  Container is running                   else             docker-compose down             echo  Container stopped and removed  Network removed   gt  gt   ScriptOutput   currentDate  txt          fi         echo  Image Created Date  currentDate   gt    env          echo  ORG  ORG   gt  gt    env          echo  IMGNAME  IMGNAME   gt  gt    env          echo  IMGTAG  IMGTAG   gt  gt    env          echo  CONTNAME  CONTNAME   gt  gt    env          echo  NETWORKNAME  NETWORKNAME   gt  gt    env          docker-compose build --no-cache         echo  Docker Compose built   gt  gt   ScriptOutput   currentDate  txt          if  DO DOCKER COMPOSE UP   then             docker-compose up -d             echo  Docker services are up now  checked in   gt  gt   ScriptOutput   currentDate  txt            else             echo  Docker services are down  checked in   gt  gt   ScriptOutput   currentDate  txt          fi elif      updatedDate  -gt   createdDate      then      echo  Updated is latest      start date   date -jf   Y m d H M S    updatedDate     s       end date   date -jf   Y m d H M S    createdDate     s       diffMins       start date -  end date     60         if      BUILDMINS  -lt   diffMins      then         if     -d    FILEPATH      then             mkdir    FILEPATH            fi         cd    FILEPATH           pwd         echo  updatedDate--- updatedDate   gt   ScriptOutput   currentDate  txt          echo  createdDate--- createdDate   gt  gt   ScriptOutput   currentDate  txt          echo  currentDate--- currentDate   gt  gt   ScriptOutput   currentDate  txt          echo  Found after comparing times - gt  Docker hub s latest updated image is new  Diff   diffMins  mins   gt  gt   ScriptOutput   currentDate  txt          echo  Actual image built time is less i e    diffMins  mins than MAX expexted BUILD TIME i e    BUILDMINS  mins   gt  gt   ScriptOutput   currentDate  txt          echo  Fetching all new   gt  gt   ScriptOutput   currentDate  txt          echo  -----------------------------          if  IS CONTAINER RUNNING   then             echo  Container is running                   else             docker-compose down             echo  Container stopped and removed  Network removed   gt  gt   ScriptOutput   currentDate  txt          fi         echo  Image Created Date  currentDate   gt    env          echo  ORG  ORG   gt  gt    env          echo  IMGNAME  IMGNAME   gt  gt    env          echo  IMGTAG  IMGTAG   gt  gt    env          echo  CONTNAME  CONTNAME   gt  gt    env          echo  NETWORKNAME  NETWORKNAME   gt  gt    env          docker-compose build --no-cache         echo  Docker Compose built   gt  gt   ScriptOutput   currentDate  txt          if  DO DOCKER COMPOSE UP   then             docker-compose up -d             echo  Docker services are up now   gt  gt   ScriptOutput   currentDate  txt            else             echo  Docker services are down   gt  gt   ScriptOutput   currentDate  txt          fi     elif      BUILDMINS  -gt   diffMins      then         echo  Docker hub s latest updated image is NOT new  Diff   diffMins  mins          echo  Docker images not fetched      else         echo  Docker hub s latest updated image is NOT new  Diff   diffMins  mins          echo  Docker images not fetched      fi elif      createdDate  -gt   updatedDate      then      echo  Created is latest      start date   date -jf   Y m d H M S    createdDate     s       end date   date -jf   Y m d H M S    updatedDate     s       echo  Docker hub has older docker image than local  Older than       start date -  end date     60    mins  fi echo  echo  ------------end---------------  rm  PIDFILE   Here is my docker-compose file  version    3 2  services    lamp-alpine      build        context        container name     CONTNAME       image     ORG    IMGNAME    IMGTAG       ports        -  127 0 0 1 80 80      networks        - private-network   networks    private-network      driver  bridge

User · Answer

You would not know your container is behind without running docker pull   Then you d need to rebuild or recompose your image   docker pull image tag docker-compose -f docker-compose yml -f production yml up -d --build   The commands can be put in a script along with anything else necessary to complete the upgrade  although a proper container would not need anything additional

User · Answer

have you tried this  https   github com v2tec watchtower  it s a simple tool running in docker container watching other containers  if their base image changed  it will pull and redeploy

User · Answer

One of the ways to do it is to drive this through your CI CD systems  Once your parent image is built  have something that scans your git repos for images using that parent  If found  you d then send a pull request to bump to new versions of the image  The pull request  if all tests pass  would be merged and you d have a new child image based on updated parent  An example of a tool that takes this approach can be found here  https   engineering salesforce com open-sourcing-dockerfile-image-update-6400121c1a75    If you don t control your parent image  as would be the case if you are depending on the official ubuntu image  you can write some tooling that detects changes in the parent image tag or checksum not the same thing  tags are mutable  and invoke children image builds accordingly

User · Answer

A  docker way  would be to use docker hub automated builds  The Repository Links feature will rebuild your container when an upstream container is rebuilt  and the Webhooks feature will send you a notification     It looks like the webhooks are limited to HTTP POST calls  You d need to set up a service to catch them  or maybe use one of the POST to email services out there   I haven t looked into it  but the new Docker Universal Control Plane might have a feature for detecting updated containers and re-deploying

[docker] How to automatically update your docker containers, if base-images are updated

Examples related to docker

Examples related to automatic-updates