Nginx upstream prematurely closed connection while reading response header from upstream for large requests

Question

I am using nginx and node server to serve update requests  I get a gateway timeout when I request an update on large data  I saw this error from the nginx error logs    2016 04 07 00 46 04  error  28599 0   1 upstream prematurely closed connection while reading response header from upstream  client  10 0 2 77  server  gis oneconcern com  request   quot GET  update mbtiles atlas19891018000415 HTTP 1 1 quot   upstream   quot http   127 0 0 1 7777 update mbtiles atlas19891018000415 quot   host   quot gis oneconcern com quot   I googled for the error and tried everything I could  but I still get the error  My nginx conf has these proxy settings               Proxy settings             proxy connect timeout 1000      proxy send timeout 1000      proxy read timeout 1000      send timeout 1000   This is how my server is configured server   listen 80   server name gis oneconcern com  access log  home ubuntu Tilelive-Server logs nginx access log  error log  home ubuntu Tilelive-Server logs nginx error log   large client header buffers 8 32k  location         proxy pass http   127 0 0 1 7777      proxy redirect off       proxy http version 1 1      proxy set header Upgrade  http upgrade      proxy set header Connection  upgrade       proxy set header Host  http host      proxy cache bypass  http upgrade     location  faults       proxy pass http   127 0 0 1 8888      proxy http version 1 1      proxy buffers 8 64k      proxy buffer size 128k      proxy set header Upgrade  http upgrade      proxy set header Connection  upgrade       proxy set header Host  host      proxy cache bypass  http upgrade       I am using a nodejs backend to serve the requests on an aws server  The gateway error shows up only when the update takes a long time  about 3-4 minutes   I do not get any error for smaller updates  Any help will be highly appreciated  Node js code   app get  quot  update mbtiles  earthquake quot   function req  res   var earthquake   req params earthquake var command   spawn   dirname     update mbtiles sh     earthquake  pg details       var output         command stdout on  data   function chunk          logger info chunk toString            output push chunk toString          command stderr on  data   function chunk          logger error chunk toString           output push chunk toString          command on  close   function code        if  code     0            logger info  quot updating mbtiles successful for  quot    earthquake           tilelive reload and switch source earthquake           res send  quot Completed updating  quot              else           logger error  quot Error occured while updating  quot    earthquake           res status 500           res send  quot Error occured while updating  quot    earthquake                  function tilelive reload and switch source earthquake unique id    tilelive load  mbtiles        dirname   mbtiles tipp out    earthquake unique id     mbtiles   function err  source        if  err            logger error err message           throw err            sources set earthquake unique id  source        logger info  Updated source  New tiles            Thank you

User · Answer

I meet the same problem and no one of the solutions detailed here worked for me ... First of all I had an error 413 Entity too large so I updated my nginx.conf as following :

http {
        # Increase request size
        client_max_body_size 10m;

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

        ##
        # Proxy settings
        ##
        proxy_connect_timeout 1000;
        proxy_send_timeout 1000;
        proxy_read_timeout 1000;
        send_timeout 1000;
}

So I only updated the http part, and now I meet the error 502 Bad Gateway and when I display /var/log/nginx/error.log I got the famous "upstream prematurely closed connection while reading response header from upstream"

What is really mysterious for me is that the request works when I run it with virtualenv on my server and send the request to the : IP:8000/nameOfTheRequest

Thanks for reading

User · Answer

You can increase the timeout in node like so   app post   slow request   function req  res      req connection setTimeout 100000     100 seconds

User · Answer

I think that error from Nginx is indicating that the connection was closed by your nodejs server  i e    upstream    How is nodejs configured

User · Answer

I solved this by setting a higher timeout value for the proxy   location         proxy read timeout 300s      proxy connect timeout 75s      proxy pass http   localhost 3000      Documentation  https   nginx org en docs http ngx http proxy module html

User · Answer

Problem The upstream server is timing out and I don t what is happening  Where to Look first before increasing read or write timeout if your server is connecting to a database Server is connecting to a database and that connection is working just fine and within sane response time  and its not the one causing this delay in server response time  make sure that connection state is not causing a cascading failure on your upstream Then you can move to look at the read and write timeout configurations of the server and proxy

User · Answer

I don t think this is your case  but I ll post it if it helps anyone  I had the same issue and the problem was that Node didn t respond at all  I had a condition that when failed didn t do anything - so no response  - So if increasing all your timeouts didn t solve it  make sure all scenarios get a response

User · Answer

I ran into this issue as well and found this post  Ultimately none of these answers solved my problem  instead I had to put in a rewrite rule to strip out the location  rt as the backend my developers made was not expecting any additional paths   - william wkstn18 -- Thu  05 Nov 20 -   -   -- 16 13 - gt wscat -c ws   WebsocketServerHostname rt error  Unexpected server response  502  Testing with wscat repeatedly gave a 502 response  Nginx error logs provided the same upstream error as above  but notice the upstream string shows the GET Request is attempting to access localhost 12775 rt and not localhost 12775   2020 11 05 22 13 32  error  10175 10175   7 upstream prematurely closed  connection while reading response header from upstream  client  WANIP   server  WebsocketServerHostname  request   quot GET  rt socket io  transport websocket  HTTP 1 1 quot   upstream   quot http   127 0 0 1 12775 rt socket io  transport websocket quot    host   quot WebsocketServerHostname quot   Since the devs had not coded their websocket  listening on 12775  to expect  rt socket io but instead just  socket io   NOTE   socket io  appears to just be a way to specify websocket transport discussed here   Because of this  rather than ask them to rewrite their socket code I just put in a rewrite rule to translate WebsocketServerHostname rt to WebsocketServerHostname 12775 as below  upstream websocket-rt           ip hash           server 127 0 0 1 12775     server           listen 80          server name     WebsocketServerHostname           location  rt                   proxy http version 1 1                    rewrite  rt  out of all requests and proxy pass to 12775                 rewrite  rt        1  break                   proxy set header X-Forwarded-For  proxy add x forwarded for                  proxy set header Host  host                   proxy pass http   websocket-rt                  proxy set header Upgrade  http upgrade                  proxy set header Connection  connection upgrade

User · Answer

I had the same error for quite a while  and here what fixed it for me   I simply declared in service that i use what follows   Description  Your node service description After network target   Service  Type forking PIDFile  tmp node pid name pid Restart on-failure KillSignal SIGQUIT WorkingDirectory  path to node app root directory ExecStart  path to node  path to server js   Install  WantedBy multi-user target   What should catch your attention here is  After network target   I spent days and days looking for fixes on nginx side  while the problem was just that  To be sure  stop running the node service you have  launch the ExecStart command directly and try to reproduce the bug  If it doesn t pop  it just means that your service has a problem  At least this is how i found my answer   For everybody else  good luck

User · Answer

I got the same error  here is how I resolved it    Downloaded logs from AWS  Reviewed Nginx logs  no additional details as above  Reviewed node js logs  AccessDenied AWS SDK permissions error  Checked the S3 bucket that AWS was trying to read from  Added additional bucket with read permission to correct server role    Even though I was processing large files there were no other errors or settings I had to change once I corrected the missing S3 access

[node.js] Nginx upstream prematurely closed connection while reading response header from upstream, for large requests

Examples related to node.js

Examples related to nginx

Examples related to webserver