Python Requests and persistent sessions

Question

I am using the requests module  version 0 10 0 with Python 2 5   I have figured out how to submit data to a login form on a website and retrieve the session key  but I can t see an obvious way to use this session key in subsequent requests  Can someone fill in the ellipsis in the code below or suggest another approach    gt  gt  gt  import requests  gt  gt  gt  login data      formPosted   1    login email   me example com    password   pw    gt  gt  gt  r   requests post  https   localhost login py   login data   gt  gt  gt    gt  gt  gt  r text u You are being redirected  lt a href  profilePage  ck 1349394964  gt here lt  a gt    gt  gt  gt  r cookies   session id myapp    127-0-0-1-825ff22a-6ed1-453b-aebc-5d3cf2987065    gt  gt  gt    gt  gt  gt  r2   requests get  https   localhost profile data json

User · Answer

The documentation says that get takes in an optional cookies argument allowing you to specify cookies to use   from the docs    gt  gt  gt  url    http   httpbin org cookies   gt  gt  gt  cookies   dict cookies are  working     gt  gt  gt  r   requests get url  cookies cookies   gt  gt  gt  r text    cookies     cookies are    working       http   docs python-requests org en latest user quickstart  cookies

User · Answer

This will work for you in Python     Call JIRA API with HTTPBasicAuth import json import requests from requests auth import HTTPBasicAuth  JIRA EMAIL          JIRA TOKEN          BASE URL    https        atlassian net  API URL     rest api 3 serverInfo   API URL   BASE URL API URL  BASIC AUTH   HTTPBasicAuth JIRA EMAIL  JIRA TOKEN  HEADERS     Content-Type     application json charset iso-8859-1    response   requests get      API URL      headers HEADERS      auth BASIC AUTH    print json dumps json loads response text   sort keys True  indent 4  separators

User · Answer

You can easily create a persistent session using   s   requests Session     After that  continue with your requests as you would   s post  https   localhost login py   login data   logged in  cookies saved for future requests  r2   s get  https   localhost profile data json         cookies sent automatically   do whatever  s will keep your cookies intact      For more about sessions  https   requests kennethreitz org en master user advanced  session-objects

User · Answer

Save only required cookies and reuse them   import os import pickle from urllib parse import urljoin  urlparse  login    my email com  password    secret    Assuming two cookies are used for persistent login     Find it by tracing the login process  persistentCookieNames     sessionId    profileId   URL    http   example com  urlData   urlparse URL  cookieFile   urlData netloc     cookie  signinUrl   urljoin URL    signin   with requests Session   as session      try          with open cookieFile   rb   as f              print  Loading cookies                  session cookies update pickle load f       except Exception            If could not load cookies from file  get the new ones by login in         print  Login in              post   session post              signinUrl              data                    email   login                   password   password                                  try              with open cookieFile   wb   as f                  jar   requests cookies RequestsCookieJar                   for cookie in session cookies                      if cookie name in persistentCookieNames                          jar set cookie cookie                  pickle dump jar  f          except Exception as e              os remove cookieFile              raise e      MyPage   urljoin URL    mypage       page   session get MyPage

User · Answer

Check out my answer in this similar question   python  urllib2 how to send cookie with urlopen request  import urllib2 import urllib from cookielib import CookieJar  cj   CookieJar   opener   urllib2 build opener urllib2 HTTPCookieProcessor cj     input-type values from the html form formdata      username    username   password   password   form-id     1234    data encoded   urllib urlencode formdata  response   opener open  https   page com login php   data encoded  content   response read     EDIT   I see I ve gotten a few downvotes for my answer  but no explaining comments  I m guessing it s because I m referring to the urllib libraries instead of requests  I do that because the OP asks for help with requests or for someone to suggest another approach

User · Answer

the other answers help to understand how to maintain such a session  Additionally  I want to provide a class which keeps the session maintained over different runs of a script  with a cache file   This means a proper  login  is only performed when required  timout or no session exists in cache   Also it supports proxy settings over subsequent calls to  get  or  post     It is tested with Python3   Use it as a basis for your own code  The following snippets are release with GPL v3  import pickle import datetime import os from urllib parse import urlparse import requests      class MyLoginSession              a class which handles and saves login sessions  It also keeps track of proxy settings      It does also maintine a cache-file for restoring session data from earlier     script executions              def   init   self                   loginUrl                   loginData                   loginTestUrl                   loginTestString                   sessionFileAppendix     session dat                    maxSessionTimeSeconds   30   60                   proxies   None                   userAgent    Mozilla 5 0  Windows NT 6 1  WOW64  rv 40 0  Gecko 20100101 Firefox 40 1                    debug   True                   forceLogin   False                     kwargs                       save some information needed to login the session          you ll have to provide  loginTestString  which will be looked for in the         responses html to make sure  you ve properly been logged in           proxies  is of format    https     https   user pass server port    http                 loginData  will be sent as post data  dictionary of id   value            maxSessionTimeSeconds  will be used to determine when to re-login                      urlData   urlparse loginUrl           self proxies   proxies         self loginData   loginData         self loginUrl   loginUrl         self loginTestUrl   loginTestUrl         self maxSessionTime   maxSessionTimeSeconds         self sessionFile   urlData netloc   sessionFileAppendix         self userAgent   userAgent         self loginTestString   loginTestString         self debug   debug          self login forceLogin    kwargs       def modification date self  filename                       return last file modification date as datetime object                     t   os path getmtime filename          return datetime datetime fromtimestamp t       def login self  forceLogin   False    kwargs                       login to a session  Try to read last saved session from cache file  If this fails         do proper login  If the last cache access was too old  also perform a proper login          Always updates session cache file                      wasReadFromCache   False         if self debug              print  loading or generating session              if os path exists self sessionFile  and not forceLogin              time   self modification date self sessionFile                          only load if file less than 30 minutes old             lastModification    datetime datetime now   - time  seconds             if lastModification  lt  self maxSessionTime                  with open self sessionFile   rb   as f                      self session   pickle load f                      wasReadFromCache   True                     if self debug                          print  loaded session from cache  last access  ds ago                                    lastModification          if not wasReadFromCache              self session   requests Session               self session headers update   user-agent    self userAgent               res   self session post self loginUrl  data   self loginData                                       proxies   self proxies    kwargs               if self debug                  print  created new session with login                self saveSessionToCache              test login         res   self session get self loginTestUrl          if res text lower   find self loginTestString lower     lt  0              raise Exception  could not log into provided site   s                                  did not find successful login string                                 self loginUrl       def saveSessionToCache self                       save session to a cache file                       always save  to update timeout          with open self sessionFile   wb   as f              pickle dump self session  f              if self debug                  print  updated session cache-file  s    self sessionFile       def retrieveContent self  url  method    get   postData   None    kwargs                       return the content of the url with respect to the session           If  method  is not  get   the url will be called with  postData          as a post request                      if method     get               res   self session get url   proxies   self proxies    kwargs          else              res   self session post url   data   postData  proxies   self proxies    kwargs             the session has been updated on the server  so also update in cache         self saveSessionToCache                        return res   A code snippet for using the above class may look like this   if   name         main           proxies     https     https   user pass server port                    http     http   user pass server port        loginData     user     usr                     password      pwd        loginUrl    https            loginTestUrl    https            successStr    Hello Tom      s   MyLoginSession loginUrl  loginData  loginTestUrl  successStr                           proxies   proxies                               res   s retrieveContent  https              print res text         if  for instance  login via JSON values required try this      s   MyLoginSession loginUrl  None  loginTestUrl  successStr                           proxies   proxies                         json   loginData

User · Answer

Upon trying all the answers above  I found that using  RequestsCookieJar  instead of the regular CookieJar for subsequent requests fixed my problem      import requests import json    The Login URL authUrl    https   whatever com login     The subsequent URL testUrl    https   whatever com someEndpoint     Logout URL testlogoutUrl    https   whatever com logout     Whatever you are posting login data      formPosted   1                    login email   me example com                    password   pw                      The Authentication token or any other data that we will receive from the Authentication Request   token         Post the login Request loginRequest   requests post authUrl  login data  print      format loginRequest text      Save the request content to your variable  In this case I needed a field called token   token   str json loads loginRequest content   token       or   access token   print      format token      Verify Successful login print      format loginRequest status code      Create your Requests Cookie Jar for your subsequent requests and add the cookie jar   requests cookies RequestsCookieJar   jar set  LWSSO COOKIE KEY   token     Execute your next request s  with the Request Cookie Jar set r   requests get testUrl  cookies jar  print  R TEXT      format r text   print  R STCD      format r status code      Execute your logout request s  with the Request Cookie Jar set r   requests delete testlogoutUrl  cookies jar  print  R TEXT      format r text      should show  Request Not Authorized  print  R STCD      format r status code      should show 401

User · Answer

snippet to retrieve json data  password protected  import requests  username    my user name  password    my super secret  url    https   www my base url com  the page i want     my json data page   session   requests Session     retrieve cookie value resp   session get url   login   csrf token   resp cookies  csrftoken     login  add referer resp   session post url   login                     data                          username   username                         password   password                         csrfmiddlewaretoken   csrf token                         next   the page i want                                         headers dict Referer url   login    print resp json

[python] Python Requests and persistent sessions

Examples related to python

Examples related to python-requests