Docker Networking - nginx emerg host not found in upstream

Question

I have recently started migrating to Docker 1 9 and Docker-Compose 1 5 s networking features to replace using links   So far with links there were no problems with nginx connecting to my php5-fpm fastcgi server located in a different server in one group via docker-compose  Newly though when I run docker-compose --x-networking up my php-fpm  mongo and nginx containers boot up  however nginx quits straight away with  emerg  1 1  host not found in upstream  waapi php 1  in  etc nginx conf d default conf 16  However  if I run the docker-compose command again while the php and mongo containers are running  nginx exited   nginx starts and works fine from then on   This is my docker-compose yml file   nginx    image  nginx   ports      -  42080 80    volumes      -   config docker nginx default conf  etc nginx conf d default conf ro  php    build  config docker php   ports      -  42022 22    volumes      -    var www html   env file  config docker php  env development  mongo    image  mongo   ports      -  42017 27017    volumes      -  var mongodata wa-api  data db   command  --smallfiles   This is my default conf for nginx   server       listen  80       root  var www test       error log  dev stdout debug      access log  dev stdout       location               try to serve file directly  fallback to app php         try files  uri  index php is args args             location         php                  Referencing the php service host  Docker          fastcgi pass waapi php 1 9000           fastcgi split path info       php                 include fastcgi params             We must reference the document root of the external server ourselves here          fastcgi param SCRIPT FILENAME  var www html public fastcgi script name           fastcgi param HTTPS off            How can I get nginx to work with only a single docker-compose call

User · Answer

This can be solved with the mentioned depends on directive since it s implemented now  2016    version   2    services      nginx        image  nginx       ports          -  42080 80        volumes          -   config docker nginx default conf  etc nginx conf d default conf ro       depends on          - php      php        build  config docker php       ports          -  42022 22        volumes          -    var www html       env file  config docker php  env development       depends on          - mongo      mongo        image  mongo       ports          -  42017 27017        volumes          -  var mongodata wa-api  data db       command  --smallfiles   Successfully tested with     docker-compose version docker-compose version 1 8 0  build f3628c7   Find more details in the documentation   There is also a very interesting article dedicated to this topic  Controlling startup order in Compose

User · Answer

I believe Nginx dont take in account Docker resolver  127 0 0 11   so please  can you try adding   resolver 127 0 0 11   in your nginx configuration file

User · Answer

new to nginx  In my case it was wrong folder name  For config  upstream serv       server ex2 app 1 3000      make sure the app folder is in ex2 folder   ex2 app

User · Answer

With links there is an order of container startup being enforced  Without links the containers can start in any order  or really all at once    I think the old setup could have hit the same issue  if the waapi php 1 container was slow to startup   I think to get it working  you could create an nginx entrypoint script that polls and waits for the php container to be started and ready   I m not sure if nginx has any way to retry the connection to the upstream automatically  but if it does  that would be a better option

User · Answer

You can set the max fails and fail timeout directives of nginx to indicate that the nginx should retry the x number of connection requests to the container before failing on the upstream server unavailability    You can tune these two numbers as per your infrastructure and speed at which the whole setup is coming up  You can read more details about the health checks section of the below URL  http   nginx org en docs http load balancing html  Following is the excerpt from http   nginx org en docs http ngx http upstream module html server max fails number     sets the number of unsuccessful attempts to communicate with the   server that should happen in the duration set by the fail timeout   parameter to consider the server unavailable for a duration also set   by the fail timeout parameter  By default  the number of unsuccessful   attempts is set to 1  The zero value disables the accounting of   attempts  What is considered an unsuccessful attempt is defined by the   proxy next upstream  fastcgi next upstream  uwsgi next upstream    scgi next upstream  and memcached next upstream directives    fail timeout time     sets the time during which the specified number of unsuccessful   attempts to communicate with the server should happen to consider the   server unavailable  and the period of time the server will be   considered unavailable  By default  the parameter is set to 10   seconds    To be precise your modified nginx config file should be as follows  this script is assuming that all the containers are up by 25 seconds at least  if not  please change the fail timeout or max fails in below upstream section   Note  I didn t test the script myself  so you could give it a try   upstream phpupstream      server waapi php 1 9000 fail timeout 5s max fails 5    server       listen  80       root  var www test       error log  dev stdout debug      access log  dev stdout       location               try to serve file directly  fallback to app php         try files  uri  index php is args args             location         php                  Referencing the php service host  Docker          fastcgi pass phpupstream           fastcgi split path info       php                 include fastcgi params             We must reference the document root of the external server ourselves here          fastcgi param SCRIPT FILENAME  var www html public fastcgi script name           fastcgi param HTTPS off            Also  as per the following Note from docker  https   github com docker docker github io blob master compose networking md update-containers   it is evident that the retry logic for checking the health of the other containers is not docker s responsibility and rather the containers should do the health check themselves       Updating containers      If you make a configuration change to a service and run docker-compose   up to update it  the old container will be removed and the new one   will join the network under a different IP address but the same name    Running containers will be able to look up that name and connect to   the new address  but the old address will stop working       If any containers have connections open to the old container  they   will be closed  It is a container s responsibility to detect this   condition  look up the name again and reconnect

User · Answer

You have to use something like docker-gen to dynamically update nginx configuration when your backend is up    See    https   hub docker com r jwilder docker-gen  https   github com jwilder nginx-proxy   I believe Nginx   premium version  contains a resolve parameter too  http   nginx org en docs http ngx http upstream module html upstream

User · Answer

If you are so lost for read the last comment  I have reached another solution   The main problem is the way that you named the services names   In this case  if in your docker-compose yml  the service for php are called  api  or something like that  you must ensure that in the file nginx conf the line that begins with fastcgi pass have the same name as the php service  i e fastcgi pass api 9000

User · Answer

Had the same problem and solved it  Please add the following line to docker-compose yml nginx section   links    - php waapi php 1   Host in nginx config fastcgi pass section should be linked inside docker-compose yml nginx configuration

User · Answer

I had the same problem because there was two networks defined in my docker-compose yml  one backend and one frontend  When I changed that to run containers on the same default network everything started working fine

User · Answer

this error appeared to me because my php-fpm image enabled cron  and I have no idea why

User · Answer

Add the links section to your nginx container configuration   You have to make visible the php container to the nginx container   nginx    image  nginx   ports      -  42080 80    volumes      -   config docker nginx default conf  etc nginx conf d default conf ro   links      - php waapi php 1

User · Answer

My problem was that I forgot to specify network alias in docker-compose yml in php-fpm     networks        - u-online  It is works well  version   quot 3 quot  services     php-fpm      image  php 7 2-fpm     container name  php-fpm     volumes                   -   src  var www basic public html     ports        - 9000 9000     networks        - u-online          nginx       image  nginx 1 19 2     container name  nginx        depends on        - php-fpm            ports        -  quot 80 8080 quot        -  quot 443 443 quot      volumes        -   docker data etc nginx conf d default conf  etc nginx conf d default conf       -   docker data etc nginx nginx conf  etc nginx nginx conf       -   src  var www basic public html     networks        - u-online   Docker Networks networks    u-online      driver  bridge

User · Answer

Perhaps the best choice to avoid linking containers issues are the docker networking features  But to make this work  docker creates entries in the  etc hosts for each container from assigned names to each container       with docker-compose --x-networking -up is something like    docker compose folder - service - incremental number    To not depend on unexpected changes in these names you should use the parameter     container name   in your docker-compose yml as follows   php        container name  waapi php 1       build  config docker php       ports          -  42022 22        volumes          -    var www html       env file  config docker php  env development   Making sure that it is the same name assigned in your configuration file for this service  I m pretty sure there are better ways to do this  but it is a good approach to start

User · Answer

At the first glance  I missed  that my  web  service didn t actually start  so that s why nginx couldn t find any host  web 1      python3  can t open file   var www app app app py    Errno 2  No such file or directory web 1 exited with code 2 nginx 1     emerg  1 1  host not found in upstream  web 4044  in  etc nginx conf d nginx conf 2

User · Answer

Two things worth to mention     Using same network bridge  Using links to add hosts resol     My example   version   3  services    mysql      image  mysql 5 7     restart  always     container name  mysql     volumes        -   mysql-data  var lib mysql     environment        MYSQL ROOT PASSWORD  tima 123     network mode  bridge   ghost      image  ghost 2     restart  always     container name  ghost     depends on        - mysql     links        - mysql     environment        database  client  mysql       database  connection  host  mysql       database  connection  user  root       database  connection  password  xxxxxxxxx       database  connection  database  ghost       url  https   www itsfun tk     volumes        -   ghost-data  var lib ghost content     network mode  bridge   nginx      image  nginx     restart  always     container name  nginx     depends on        - ghost     links        - ghost     ports        -  80 80        -  443 443      volumes         -   nginx nginx conf  etc nginx nginx conf        -   nginx conf d  etc nginx conf d        -   nginx letsencrypt  etc letsencrypt     network mode  bridge   If you don t specify a special network bridge  all of them will use the same default one

User · Answer

My Workaround  after much trial and error     In order to get around this issue  I had to get the full name of the  upstream  Docker container  found by running docker network inspect my-special-docker-network and getting the full name property of the upstream container as such    Containers           39ad8199184f34585b556d7480dd47de965bc7b38ac03fc0746992f39afac338                Name    my upstream container name 1 2478f2b3aca0    Then used this in the NGINX my-network local conf file in the location block of the proxy pass property   Note the addition of the GUID to the container name    location         proxy pass http   my upsteam container name 1 2478f2b3aca0 3000     As opposed to the previously working  but now broken       location             proxy pass http   my upstream container name 1 3000   Most likely cause is a recent change to Docker Compose  in their default naming scheme for containers  as listed here   This seems to be happening for me and my team at work  with latest versions of the Docker nginx image    I ve opened issues with them on the docker compose GitHub here

User · Answer

There is a possibility to use  volumes from  as a workaround until depends on feature  discussed below  is introduced  All you have to do is change your docker-compose file as below   nginx    image  nginx   ports      -  42080 80    volumes      -   config docker nginx default conf  etc nginx conf d default conf ro   volumes from      - php  php    build  config docker php   ports      -  42022 22    volumes      -    var www html   env file  config docker php  env development  mongo    image  mongo   ports      -  42017 27017    volumes      -  var mongodata wa-api  data db   command  --smallfiles   One big caveat in the above approach is that the volumes of php are exposed to nginx  which is not desired  But at the moment this is one docker specific workaround that could be used    depends on feature This probably would be a futuristic answer  Because the functionality is not yet implemented in Docker  as of 1 9   There is a proposal to introduce  depends on  in the new networking feature introduced by Docker  But there is a long running debate about the same   https   github com docker compose issues 374 Hence  once it is implemented  the feature depends on could be used to order the container start-up  but at the moment  you would have to resort to one of the following    make nginx retry until the php server is up - I would prefer this one use volums from workaround as described above - I would avoid using this  because of the volume leakage into unnecessary containers

[nginx] Docker Networking - nginx: [emerg] host not found in upstream

Examples related to nginx

Examples related to docker

Examples related to docker-compose

Examples related to docker-networking