Possible reason for NGINX 499 error codes

Question

I m getting a lot of 499 NGINX error codes   I see that this is a client side issue   It is not a problem with NGINX or my uWSGI stack   I note the correlation in uWSGI logs  when a get a 499   address space usage  383692800 bytes 365MB   rss usage  167038976 bytes 159MB   pid  16614 app  0 req  74184 222373  74 125 191 16     36 vars in 481 bytes   Fri Oct 19 10 07 07 2012  POST  bidder    gt  generated 0 bytes in 8 msecs  HTTP 1 1 200  1 headers in 59 bytes  1 switches on core 1760  SIGPIPE  writing to a closed pipe socket fd  probably the client disconnected  on request  bidder   ip 74 125 xxx xxx      Fri Oct 19 10 07 07 2012 - write    Broken pipe  proto uwsgi c line 143  during POST  bidder   74 125 xxx xxx  IOError  write error   I m looking for a more in depth explanation and hoping it is nothing wrong with my NGINX config for uwsgi  I m taking it on face value  It seems like a client issue

User · Answer

Turns out 499 s really does mean  quot client interrupted connection  quot  I had a client  quot read timeout quot  setting of 60s  and nginx also has a default proxy read timeout of 60s    So what was happening in my case is that nginx would error log an upstream timed out  110  Connection timed out  while reading upstream and then nginx retries  quot the next proxy server in the backend server group you configured  quot   That s if you have more than one  Then it tries the next and next till  by default  it has exhausted all of them  As each one times out  it removes them from the list of  quot live quot  backend servers  as well  After all are exhausted  it returns a 504 gateway timeout  So in my case nginx marked the server as  quot unavailable quot   re-tried it on the next server  then my client s 60s timeout  immediately  occurred  so I d see a upstream timed out  110  Connection timed out  while reading upstream log  immediately followed by a 499 log   But it was just timing coincidence  Related  If all servers in the group are marked as currently unavailable  then it returns a 502 Bad Gateway  for 10s as well  See here max fails and fail timeout   Inn the logs it will say no live upstreams while connecting to upstream  If you only have one proxy backend in your server group  it just try s the one server  and returns a 504 Gateway Time-out and doesn t remove the single server from the list of  quot live quot  servers  if proxy read timeout is surpassed   See here  quot If there is only a single server in a group  max fails  fail timeout and slow start parameters are ignored  and such a server will never be considered unavailable  quot  The really tricky part is that if you specify proxy pass to  quot localhost quot  and your box happens to also have ipv6 and ipv4  quot versions of location quot  on it at the same time  most boxes do by default   it will count as if you had a  quot list quot  of multiple servers in your server group  which means you can get into the situation above of having it return  quot 502 for 10s quot  even though you list only one server  See here  quot If a domain name resolves to several addresses  all of them will be used in a round-robin fashion  quot  One workaround is to declare it as proxy pass    http   127 0 0 1 5001   its ipv4 address  to avoid it being both ipv6 and ipv4    Then it counts as  quot only a single server quot  behavior  There s a few different settings you can tweak to make this  quot less quot  of a problem   Like increasing timeouts or making it so it doesn t mark servers as  quot disabled quot  when they timeout   or fixing the list so it s only size 1  see above    See also  https   serverfault com a 783624 27813

User · Answer

In my case  I have setup like   AWS ELB  gt  gt  ECS nginx   gt  gt  ECS php-fpm     I had configured the wrong AWS security group for ECS php-fpm  service  so Nginx wasn t able to reach out to php-fpm task container  That s why i was getting errors in nginx task log  499 0 - elb-healthchecker 2 0   Health check was configured as to check php-fpm service and confirm it s up and give back a response

User · Answer

In my case I got 499 when the client s API closed the connection before it gets any response  Literally sent a POST and immediately close the connection  This is resolved by option   proxy ignore client abort on  Nginx doc

User · Answer

One of the reasons for this behaviour could be you are using http for uwsgi instead of socket  Use the below command if you are using uwsgi directly    uwsgi --socket  8080 --module app-name wsgi   Same command in  ini file is  chdir    path to app folder socket    8080 module   app-name wsgi

User · Answer

In my case  I was impatient and ended up misinterpreting the log    In fact  the real problem was the communication between nginx and uwsgi  and not between the browser and nginx  If I had loaded the site in my browser and had waited long enough I would have gotten a  504 - Bad Gateway   But it took so long  that I kept trying stuff  and then refresh in the browser  So I never waited long enough to see the 504 error  When refreshing in the browser  that is when the previous request is closed  and Nginx writes that in the log as 499    Elaboration  Here I will assume that the reader knows as little as I did when I started playing around    My setup was a reverse proxy  the nginx server  and an application server  the uWSGI server behind it  All requests from the client would go to the nginx server  then forwarded to the uWSGI server  and then response was sent the same way back  I think this is how everyone uses nginx uwsgi and are supposed to use it    My nginx worked as it should  but something was wrong with the uwsgi server  There are two ways  maybe more  in which the uwsgi server can fail to respond to the nginx server    1   uWSGI says   I m processing  just wait and you will soon get a response   nginx has a certain period of time  that it is willing to wait  fx 20 seconds  After that  it will respond to the client  with a 504 error    2   uWSGI is dead  or uWSGi dies while nginx is waiting for it  nginx sees that right away and in that case  it returns a 499 error    I was testing my setup by making requests in the client  browser   In the browser nothing happened  it just kept hanging  After maybe 10 seconds  less than the timeout  I concluded that something was not right  which was true   and closed the uWSGI server from the command line  Then I would go to the uWSGI settings  try something new  and then restart the uWSGI server  The moment I closed the uWSGI server  the nginx server would return a 499 error    So I kept debugging with the 499 erroe  which means googling for the 499 error  But if I had waited long enough  I would have gotten the 504 error  If I had gotten the 504 error  I would have been able to understand the problem better  and then be able to debug    So the conclusion is  that the problem was with uWGSI  which kept hanging   Wait a little longer  just a little longer  then I will have an answer for you         How I fixed that problem  I don t remember  I guess it could be caused by a lot of things

User · Answer

Client closed the connection doesn t mean it s a browser issue   Not at all   You can find 499 errors in a log file if you have a LB  load balancer  in front of your webserver  nginx  either AWS or haproxy  custom   That said the LB will act as a client to nginx    If you run haproxy default values for       timeout client  60000     timeout server  60000   That would mean that LB will time out after 60000ms if there is no respond from nginx  Time outs might happen for busy websites or scripts that need more time for execution  You ll need to find timeout that will work for you  For example extend it to       timeout client  180s     timeout server  180s   And you will be probably set   Depending on your setup you might see a 504 gateway timeout error in your browser which indicates that something is wrong with php-fpm but that will not be the case with 499 errors in your log files

User · Answer

For my part I had enabled ufw but I forgot to expose my upstreams ports

User · Answer

I encountered this issue and the cause was due to Kaspersky Protection plugin on the browser  If you are encountering this  try to disable your plugins and see if that fixes your issue

User · Answer

I know this is an old thread  but it exactly matches what recently happened to me and I thought I d document it here   The setup  in Docker  is as follows   nginx proxy nginx php fpm running the actual app   The symptom was a  quot 502 Gateway Timeout quot  on the application login prompt   Examination of logs found   the button works via an HTTP POST to  login     and so     nginx-proxy got the  login request  and eventually reported a timeout  nginx returned a 499 response  which of course means  quot the host died  quot  the  login request did not appear at all    in the FPM server s logs  there were no tracebacks or error-messages in FPM     nada  zero  zippo  none   It turned out that the problem was a failure to connect to the database to verify the login   But how to figure that out turned out to be pure guesswork  The complete absence of application traceback logs     or even a record that the request had been received by FPM     was a complete  and  devastating      surprise to me   Yes  the application is supposed to log failures  but in this case it looks like the FPM worker process died with a runtime error  leading to the 499 response from nginx   Now  this obviously is a problem in our application     somewhere   But I wanted to record the particulars of what happened for the benefit of the next folks who face something like this

User · Answer

Once I got 499  Request has been forbidden by antivirus  as an AJAX http response  false positive by Kaspersky Internet Security with light heuristic analysis  deep heuristic analysis knew correctly there was nothing wrong

User · Answer

As you point 499 a connection abortion logged by the nginx  But usually this is produced when your backend server is being too slow  and another proxy timeouts first or the user software aborts the connection  So check if uWSGI is answering fast or not of if there is any load on uWSGI   Database server   In many cases there are some other proxies between the user and nginx  Some can be in your infrastructure like maybe a CDN  Load Balacer  a Varnish cache etc  Others can be in user side like a caching proxy etc   If there are proxies on your side like a LoadBalancer   CDN     you should set the timeouts to timeout first your backend and progressively the other proxies to the user   If you have   user  gt  gt  gt  CDN  gt  gt  gt  Load Balancer  gt  gt  gt  Nginx  gt  gt  gt  uWSGI   I ll recommend you to set    n seconds to uWSGI timeout n 1 seconds to nginx timeout n 2 senconds to timeout to Load Balancer n 3 seconds of timeout to the CDN    If you can t set some of the timeouts  like CDN  find whats is its timeout and adjust the others according to it  n  n-1       This provides a correct chain of timeouts  and you ll find really whose giving the timeout and return the right response code to the user

User · Answer

This doesn t answer the OPs question  but since I ended up here after searching furiously for an answer  I wanted to share what we discovered   In our case  it turns out these 499s are expected   When users use the type-ahead feature in some search boxes  for example  we see something like this in the logs     GET  api search q h  Status 499   GET  api search q he  Status 499  GET  api search q hel  Status 499  GET  api search q hell  Status 499  GET  api search q hello  Status 200    So in our case I think its safe to use proxy ignore client abort on which was suggested in a previous answer   Thanks for that

User · Answer

came here from a google search   I found the answer elsewhere here --  https   stackoverflow com a 15621223 1093174  which was to raise the connection idle timeout of my AWS elastic load balancer    I had setup a Django site with nginx apache reverse proxy   and a really really really log backend job view was timing out

User · Answer

This error is pretty easy to reproduce using standard nginx configuration with php-fpm    Keeping the F5 button down on a page will create dozens of refresh requests to the server   Each previous request is canceled by the browser at new refresh  In my case I found dozens of 499 s in my client s online shop log file  From an nginx point of view  If the response has not been delivered to the client before the next refresh request nginx logs the 499 error   mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 32  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 33  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 33  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 33  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 33  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 34  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 34  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 34  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 34  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 35  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string  mydomain com log 84 240 77 112 - -  19 Jun 2018 09 07 35  0200   GET   path  HTTP 2 0  499 0  -   user-agent-string    If the php-fpm processing takes longer  like a heavyish WP page  it may cause problems  of course  I have heard of php-fpm crashes  for instance  but I believe they can be prevented configuring services properly like handling calls to xmlrpc php

User · Answer

HTTP 499 in Nginx means that the client closed the connection before the server answered the request  In my experience is usually caused by client side timeout  As I know it s an Nginx specific error code

[nginx] Possible reason for NGINX 499 error codes

Examples related to nginx

Examples related to http-headers

Examples related to uwsgi