NGINX upstream timed out 110 Connection timed out while reading response header from upstream

Question

I have Puma running as the upstream app server and Riak as my background db cluster  When I send a request that map-reduces a chunk of data for about 25K users and returns it from Riak to the app  I get an error in the Nginx log      upstream timed out  110  Connection timed out  while reading   response header from upstream   If I query my upstream directly without nginx proxy  with the same request  I get the required data    The Nginx timeout occurs once the proxy is put in      nginx conf    http       keepalive timeout 10m      proxy connect timeout  600s      proxy send timeout  600s      proxy read timeout  600s      fastcgi send timeout 600s      fastcgi read timeout 600s      include  etc nginx sites-enabled   conf       virtual host conf    upstream ss api     server 127 0 0 1 3000 max fails 0  fail timeout 600     server     listen 81    server name xxxxx com    change to match your URL    location           match the name of upstream directive which is defined above     proxy pass http   ss api       proxy set header  Host  http host      proxy set header  X-Real-IP   remote addr      proxy set header  X-Forwarded-For  proxy add x forwarded for      proxy cache cloud      proxy cache valid  200 302  60m      proxy cache valid  404      1m      proxy cache bypass  http authorization      proxy cache bypass http   ss api account       add header X-Cache-Status  upstream cache status          Nginx has a bunch of timeout directives  I don t know if I m missing something important  Any help would be highly appreciated

User · Answer

From our side it was using spdy with proxy cache  When the cache expires we get this error till the cache has been updated

User · Answer

You should always refrain from increasing the timeouts  I doubt your backend server response time is the issue here in any case   I got around this issue by clearing the connection keep-alive flag and specifying http version as per the answer here  https   stackoverflow com a 36589120 479632  server       location             proxy set header   X-Real-IP  remote addr          proxy set header   Host       http host             these two lines here         proxy http version 1 1          proxy set header Connection              proxy pass http   localhost 5000            Unfortunately I can t explain why this works and didn t manage to decipher it from the docs mentioned in the answer linked either so if anyone has an explanation I d be very interested to hear it

User · Answer

I think this error can happen for various reasons  but it can be specific to the module you re using  For example I saw this using the uwsgi module  so had to set  uwsgi read timeout

User · Answer

I would recommend to look at the error logs  specifically at the upstream part where it shows specific upstream that is timing out   Then based on that you can adjust proxy read timeout  fastcgi read timeout or uwsgi read timeout   Also make sure your config is loaded   More details here Nginx upstream timed out  why and how to fix

User · Answer

As many others have pointed out here  increasing the timeout settings for NGINX can solve your issue    However  increasing your timeout settings might not be as straightforward as many of these answers suggest  I myself faced this issue and tried to change my timeout settings in the  etc nginx nginx conf file  as almost everyone in these threads suggest  This did not help me a single bit  there was no apparent change in NGINX  timeout settings  Now  many hours later  I finally managed to fix this problem   The solution lies in this forum thread  and what it says is that you should put your timeout settings in  etc nginx conf d timeout conf  and if this file doesn t exist  you should create it   I used the same settings as suggested in the thread   proxy connect timeout 600  proxy send timeout 600  proxy read timeout 600  send timeout 600

User · Answer

In your case it helps a little optimization in proxy  or you can use    time out settings   location                   time out settings   proxy connect timeout 159s    proxy send timeout   600    proxy read timeout   600    proxy buffer size    64k    proxy buffers     16 32k    proxy busy buffers size 64k    proxy temp file write size 64k    proxy pass header Set-Cookie    proxy redirect     off    proxy hide header  Vary    proxy set header   Accept-Encoding       proxy ignore headers Cache-Control Expires    proxy set header   Referer  http referer    proxy set header   Host    host    proxy set header   Cookie  http cookie    proxy set header   X-Real-IP   remote addr    proxy set header X-Forwarded-Host  host    proxy set header X-Forwarded-Server  host    proxy set header X-Forwarded-For  proxy add x forwarded for

User · Answer

For proxy upstream timeout  I tried the above setting but these didn t work   Setting resolver timeout worked for me  knowing it was taking 30s to produce the upstream timeout message  E g  me atwibble com could not be resolved  110  Operation timed out    http   nginx org en docs http ngx http core module html resolver timeout

User · Answer

This happens because your upstream takes too much to answer the request and NGINX thinks the upstream already failed in processing the request  so it responds with an error  Just include and increase proxy read timeout in location config block  Same thing happened to me and I used 1 hour timeout for an internal app at work   proxy read timeout 3600    With this  NGINX will wait for an hour  3600s  for its upstream to return something

User · Answer

Hopefully it helps someone  I ran into this error and the cause was wrong permission on the log folder for phpfpm  after changing it so phpfpm could write to it  everything was fine

User · Answer

I had the same problem and resulted that was an  every day  error in the rails controller  I don t know why  but on production  puma runs the error again and again causing the message   upstream timed out  110  Connection timed out  while reading response header from upstream  Probably because Nginx tries to get the data from puma again and again The funny thing is that the error caused the timeout message even if I m calling a different action in the controller  so  a single typo blocks all the app     Check your log puma stderr log file to see if that is the situation

User · Answer

First figure out which upstream is slowing by consulting the nginx error log  file and adjust the read time out accordingly  in my case it was fastCGI  2017 09 27 13 34 03  error  16559 16559   14381 upstream timed out  110  Connection timed out  while reading response header from upstream  client xxxxxxxxxxxxxxxxxxxxxxxxx   upstream   fastcgi   unix  var run php php5 6-fpm sock   host   xxxxxxxxxxxxxxx   referrer   xxxxxxxxxxxxxxxxxxxx    So i have to adjust the fastcgi read timeout in my server configuration   location     php         fastcgi read timeout 240                See  original post

[nginx] NGINX: upstream timed out (110: Connection timed out) while reading response header from upstream

Examples related to nginx

Examples related to timeout

Examples related to puma