104 Connection reset by peer socket error or When does closing a socket result in a RST rather than FIN

Question

We re developing a Python web service and a client web site in parallel   When we make an HTTP request from the client to the service  one call consistently raises a socket error in socket py  in read    104   Connection reset by peer    When I listen in with wireshark  the  good  and  bad  responses look very similar    Because of the size of the OAuth header  the request is split into two packets   The service responds to both with ACK The service sends the response  one packet per header  HTTP 1 0 200 OK  then the Date header  etc     The client responds to each with ACK   Good request  the server sends a FIN  ACK   The client responds with a FIN  ACK   The server responds ACK   Bad request  the server sends a RST  ACK  the client doesn t send a TCP response  the socket error is raised on the client side    Both the web service and the client are running on a Gentoo Linux x86-64 box running glibc-2 6 1   We re using Python 2 5 2 inside the same virtual env   The client is a Django 1 0 2 app that is calling httplib2 0 4 0 to make requests   We re signing requests with the OAuth signing algorithm  with the OAuth token always set to an empty string   The service is running Werkzeug 0 3 1  which is using Python s wsgiref simple server   I ran the WSGI app through wsgiref validator with no issues   It seems like this should be easy to debug  but when I trace through a good request on the service side  it looks just like the bad request  in the socket  socketobject close   function  turning delegate methods into dummy methods   When the send or sendto  can t remember which  method is switched off  the FIN or RST is sent  and the client starts processing    Connection reset by peer  seems to place blame on the service  but I don t trust httplib2 either   Can the client be at fault      Further debugging - Looks like server on Linux     I have a MacBook  so I tried running the service on one and the client website on the other   The Linux client calls the OS X server without the bug  FIN ACK    The OS X client calls the Linux service with the bug  RST ACK  and a  54   Connection reset by peer      So  it looks like it s the service running on Linux   Is it x86 64  A bad glibc  wsgiref  Still looking        Further testing - wsgiref looks flaky     We ve gone to production with Apache and mod wsgi  and the connection resets have gone away   See my answer below  but my advice is to log the connection reset and retry   This will let your server run OK in development mode  and solidly in production

User · Accepted Answer

I ve had this problem   See The Python  Connection Reset By Peer  Problem   You have  most likely  run afoul of small timing issues based on the Python Global Interpreter Lock   You can  sometimes  correct this with a time sleep 0 01  placed strategically      Where   you ask   Beats me   The idea is to provide some better thread concurrency in and around the client requests   Try putting it just before you make the request so that the GIL is reset and the Python interpreter can clear out any pending threads

User · Answer

Don t use wsgiref for production   Use Apache and mod wsgi  or something else   We continue to see these connection resets  sometimes frequently  with wsgiref  the backend used by the werkzeug test server  and possibly others like the Django test server    Our solution was to log the error  retry the call in a loop  and give up after ten failures   httplib2 tries twice  but we needed a few more   They seem to come in bunches as well - adding a 1 second sleep might clear the issue   We ve never seen a connection reset when running through Apache and mod wsgi   I don t know what they do differently   maybe they just mask them   but they don t appear   When we asked the local dev community for help  someone confirmed that they see a lot of connection resets with wsgiref that go away on the production server   There s a bug there  but it is going to be hard to find it

User · Answer

Normally  you d get an RST if you do a close which doesn t linger  i e  in which data can be discarded by the stack if it hasn t been sent and ACK d  and a normal FIN if you allow the close to linger  i e  the close waits for the data in transit to be ACK d    Perhaps all you need to do is set your socket to linger so that you remove the race condition between a non lingering close done on the socket and the ACKs arriving

User · Answer

Normally  you d get an RST if you do a close which doesn t linger  i e  in which data can be discarded by the stack if it hasn t been sent and ACK d  and a normal FIN if you allow the close to linger  i e  the close waits for the data in transit to be ACK d    Perhaps all you need to do is set your socket to linger so that you remove the race condition between a non lingering close done on the socket and the ACKs arriving

User · Answer

Don t use wsgiref for production   Use Apache and mod wsgi  or something else   We continue to see these connection resets  sometimes frequently  with wsgiref  the backend used by the werkzeug test server  and possibly others like the Django test server    Our solution was to log the error  retry the call in a loop  and give up after ten failures   httplib2 tries twice  but we needed a few more   They seem to come in bunches as well - adding a 1 second sleep might clear the issue   We ve never seen a connection reset when running through Apache and mod wsgi   I don t know what they do differently   maybe they just mask them   but they don t appear   When we asked the local dev community for help  someone confirmed that they see a lot of connection resets with wsgiref that go away on the production server   There s a bug there  but it is going to be hard to find it

User · Answer

Normally  you d get an RST if you do a close which doesn t linger  i e  in which data can be discarded by the stack if it hasn t been sent and ACK d  and a normal FIN if you allow the close to linger  i e  the close waits for the data in transit to be ACK d    Perhaps all you need to do is set your socket to linger so that you remove the race condition between a non lingering close done on the socket and the ACKs arriving

User · Answer

I had the same issue however with doing an upload of a very large file using a python-requests client posting to a nginx uwsgi backend   What ended up being the cause was the the backend had a cap on the max file size for uploads lower than what the client was trying to send   The error never showed up in our uwsgi logs since this limit was actually one imposed by nginx    Upping the limit in nginx removed the error

User · Answer

Don t use wsgiref for production   Use Apache and mod wsgi  or something else   We continue to see these connection resets  sometimes frequently  with wsgiref  the backend used by the werkzeug test server  and possibly others like the Django test server    Our solution was to log the error  retry the call in a loop  and give up after ten failures   httplib2 tries twice  but we needed a few more   They seem to come in bunches as well - adding a 1 second sleep might clear the issue   We ve never seen a connection reset when running through Apache and mod wsgi   I don t know what they do differently   maybe they just mask them   but they don t appear   When we asked the local dev community for help  someone confirmed that they see a lot of connection resets with wsgiref that go away on the production server   There s a bug there  but it is going to be hard to find it

User · Answer

Normally  you d get an RST if you do a close which doesn t linger  i e  in which data can be discarded by the stack if it hasn t been sent and ACK d  and a normal FIN if you allow the close to linger  i e  the close waits for the data in transit to be ACK d    Perhaps all you need to do is set your socket to linger so that you remove the race condition between a non lingering close done on the socket and the ACKs arriving

User · Answer

I had the same issue however with doing an upload of a very large file using a python-requests client posting to a nginx uwsgi backend   What ended up being the cause was the the backend had a cap on the max file size for uploads lower than what the client was trying to send   The error never showed up in our uwsgi logs since this limit was actually one imposed by nginx    Upping the limit in nginx removed the error

User · Answer

Don t use wsgiref for production   Use Apache and mod wsgi  or something else   We continue to see these connection resets  sometimes frequently  with wsgiref  the backend used by the werkzeug test server  and possibly others like the Django test server    Our solution was to log the error  retry the call in a loop  and give up after ten failures   httplib2 tries twice  but we needed a few more   They seem to come in bunches as well - adding a 1 second sleep might clear the issue   We ve never seen a connection reset when running through Apache and mod wsgi   I don t know what they do differently   maybe they just mask them   but they don t appear   When we asked the local dev community for help  someone confirmed that they see a lot of connection resets with wsgiref that go away on the production server   There s a bug there  but it is going to be hard to find it

[python] 104, 'Connection reset by peer' socket error, or When does closing a socket result in a RST rather than FIN?

Examples related to python

Examples related to sockets

Examples related to wsgi

Examples related to httplib2

Examples related to werkzeug