Python requests - print entire http request raw

Question

While using the requests module  is there any way to print the raw HTTP request    I don t want just the headers  I want the request line  headers  and content printout  Is it possible to see what ultimately is constructed from HTTP request

User · Answer

An even better idea is to use the requests_toolbelt library, which can dump out both requests and responses as strings for you to print to the console. It handles all the tricky cases with files and encodings which the above solution does not handle well.

It's as easy as this:

import requests
from requests_toolbelt.utils import dump

resp = requests.get('https://httpbin.org/redirect/5')
data = dump.dump_all(resp)
print(data.decode('utf-8'))

Source: https://toolbelt.readthedocs.org/en/latest/dumputils.html

You can simply install it by typing:

pip install requests_toolbelt

User · Answer

Since v1 2 3 Requests added the PreparedRequest object  As per the documentation  it contains the exact bytes that will be sent to the server    One can use this to pretty print a request  like so   import requests  req   requests Request  POST   http   stackoverflow com  headers   X-Custom   Test   data  a 1 amp b 2   prepared   req prepare    def pretty print POST req               At this point it is completely built and ready     to be fired  it is  prepared        However pay attention at the formatting used in      this function because it is programmed to be pretty      printed and may differ from the actual request              print     n   r n   r n r n    format           -----------START-----------           req method         req url            r n  join          format k  v  for k  v in req headers items             req body          pretty print POST prepared    which produces   -----------START----------- POST http   stackoverflow com  Content-Length  7 X-Custom  Test  a 1 amp b 2   Then you can send the actual request with this   s   requests Session   s send prepared    These links are to the latest documentation available  so they might change in content  Advanced - Prepared requests and API - Lower level classes

User · Answer

Note  this answer is outdated  Newer versions of requests  support getting the request content directly  as AntonioHerraizS s answer documents   It s not possible to get the true raw content of the request out of requests  since it only deals with higher level objects  such as headers and method type  requests uses urllib3 to send requests  but urllib3 also doesn t deal with raw data - it uses httplib  Here s a representative stack trace of a request   - gt  r  requests get  http   google com      usr local lib python2 7 dist-packages requests api py 55 get   - gt  return request  get   url    kwargs     usr local lib python2 7 dist-packages requests api py 44 request   - gt  return session request method method  url url    kwargs     usr local lib python2 7 dist-packages requests sessions py 382 request   - gt  resp   self send prep    send kwargs     usr local lib python2 7 dist-packages requests sessions py 485 send   - gt  r   adapter send request    kwargs     usr local lib python2 7 dist-packages requests adapters py 324 send   - gt  timeout timeout    usr local lib python2 7 dist-packages requests packages urllib3 connectionpool py 478 urlopen   - gt  body body  headers headers     usr local lib python2 7 dist-packages requests packages urllib3 connectionpool py 285  make request   - gt  conn request method  url    httplib request kw     usr lib python2 7 httplib py 958 request   - gt  self  send request method  url  body  headers    Inside the httplib machinery  we can see HTTPConnection  send request indirectly uses HTTPConnection  send output  which finally creates the raw request and body  if it exists   and uses HTTPConnection send to send them separately  send finally reaches the socket   Since there s no hooks for doing what you want  as a last resort you can monkey patch httplib to get the content  It s a fragile solution  and you may need to adapt it if httplib is changed  If you intend to distribute software using this solution  you may want to consider packaging httplib instead of using the system s  which is easy  since it s a pure python module   Alas  without further ado  the solution   import requests import httplib  def patch send        old send  httplib HTTPConnection send     def new send  self  data            print data         return old send self  data   return is not necessary  but never hurts  in case the library is changed     httplib HTTPConnection send  new send  patch send   requests get  http   www python org     which yields the output   GET   HTTP 1 1 Host  www python org Accept-Encoding  gzip  deflate  compress Accept      User-Agent  python-requests 2 1 0 CPython 2 7 3 Linux 3 2 0-23-generic-pae

User · Answer

I use the following function to format requests  It s like  AntonioHerraizS except it will pretty-print JSON objects in the body as well  and it labels all parts of the request   format json   functools partial json dumps  indent 2  sort keys True  indent   functools partial textwrap indent  prefix        def format prepared request req          Pretty-format  requests PreparedRequest       Example          res   requests post              print format prepared request res request            req   requests Request              req   req prepare           print format prepared request res request               headers     n  join f  k    v   for k  v in req headers items        content type   req headers get  Content-Type           if  application json  in content type          try              body   format json json loads req body           except json JSONDecodeError              body   req body     else          body   req body     s   textwrap dedent         REQUEST                 endpoint   method   url      headers       headers      body       body                       strip       s   s format          method req method          url req url          headers indent headers           body indent body             return s   And I have a similar function to format the response   def format response resp          Pretty-format  requests Response         headers     n  join f  k    v   for k  v in resp headers items        content type   resp headers get  Content-Type           if  application json  in content type          try              body   format json resp json            except json JSONDecodeError              body   resp text     else          body   resp text     s   textwrap dedent         RESPONSE                  status code   status code      headers       headers      body       body                        strip        s   s format          status code resp status code          headers indent headers           body indent body             return s

User · Answer

test print py content  import logging import pytest import requests from requests toolbelt utils import dump   def print raw http response       data   dump dump all response  request prefix b    response prefix b        return   n    2   data decode  utf-8     pytest fixture def logger        log   logging getLogger       log addHandler logging StreamHandler        log setLevel logging DEBUG      return log  def test print response logger       session   requests Session       response   session get  http   127 0 0 1 5000        assert response status code    300  logger warning print raw http response    hello py content  from flask import Flask app   Flask   name      app route      def hello world        return  Hello  World    Run     python -m flask hello py    python -m pytest test print py  Stdout  ------------------------------ Captured log call ------------------------------ DEBUG    urllib3 connectionpool connectionpool py 225 Starting new HTTP connection  1   127 0 0 1 5000 DEBUG    urllib3 connectionpool connectionpool py 437 http   127 0 0 1 5000  quot GET   HTTP 1 1 quot  200 13 WARNING  root test print raw response py 25   GET   HTTP 1 1 Host  127 0 0 1 5000 User-Agent  python-requests 2 23 0 Accept-Encoding  gzip  deflate Accept      Connection  keep-alive   HTTP 1 0 200 OK Content-Type  text html  charset utf-8 Content-Length  13 Server  Werkzeug 1 0 1 Python 3 6 8 Date  Thu  24 Sep 2020 21 00 54 GMT  Hello  World

User · Answer

A fork of  AntonioHerraizS answer  HTTP version missing as stated in comments   Use this code to get a string representing the raw HTTP packet without sending it  import requests   def get raw request request       request   request prepare   if isinstance request  requests Request  else request     headers     r n  join f  k    v   for k  v in request headers items        body      if request body is None else request body decode   if isinstance request body  bytes  else request body     return f  request method   request path url  HTTP 1 1 r n headers  r n r n body     headers     User-Agent    Test   request   requests Request  POST    https   stackoverflow com   headers headers  json   quot hello quot    quot world quot    raw request   get raw request request  print raw request   Result  POST   HTTP 1 1 User-Agent  Test Content-Length  18 Content-Type  application json    quot hello quot    quot world quot      Can also print the request in the response object r   requests get  https   stackoverflow com   raw request   get raw request r request  print raw request

User · Answer

requests supports so called event hooks  as of 2 23 there s actually only response hook   The hook can be used on a request to print full request-response pair s data  including effective URL  headers and bodies  like   import textwrap import requests  def print roundtrip response   args    kwargs       format headers   lambda d    n  join f  k    v   for k  v in d items        print textwrap dedent             ---------------- request ----------------          req method   req url           reqhdrs            req body          ---------------- response ----------------          res status code   res reason   res url           reshdrs            res text           format          req response request           res response           reqhdrs format headers response request headers            reshdrs format headers response headers            requests get  https   httpbin org    hooks   response   print roundtrip     Running it prints   ---------------- request ---------------- GET https   httpbin org  User-Agent  python-requests 2 23 0 Accept-Encoding  gzip  deflate Accept      Connection  keep-alive  None ---------------- response ---------------- 200 OK https   httpbin org  Date  Thu  14 May 2020 17 16 13 GMT Content-Type  text html  charset utf-8 Content-Length  9593 Connection  keep-alive Server  gunicorn 19 9 0 Access-Control-Allow-Origin    Access-Control-Allow-Credentials  true   lt  DOCTYPE html gt   lt html lang  en  gt       lt  html gt    You may want to change res text to res content if the response is binary

User · Answer

Here is a code  which makes the same  but with response headers    import socket def patch requests        old readline   socket  fileobject readline     if not hasattr old readline   patched            def new readline self  size -1               res   old readline self  size              print res              return res         new readline patched   True         socket  fileobject readline   new readline patch requests     I spent a lot of time searching for this  so I m leaving it here  if someone needs

User · Answer

import requests  response   requests post  http   httpbin org post   data   key1   value1    print response request url  print response request body  print response request headers   Response objects have a  request property which is the original PreparedRequest object that was sent

[python] Python requests - print entire http request (raw)?

Examples related to python

Examples related to http

Examples related to python-requests