Use python requests to download CSV

Question

Here is my code    import csv import requests with requests Session   as s      s post url  data payload      download   s get  url that directly download a csv report     This gives me the access to the csv file  I tried different method to deal with the download   This will give the the csv file in one string   print download content   This print the first row and return error   csv Error  new-line character seen in unquoted field  cr   csv reader download  dialect csv excel tab  for row in cr      print row   This will print a letter in each row and it won t print the whole thing   cr   csv reader download content  dialect csv excel tab  for row in cr      print row   My question is  what s the most efficient way to read a csv file in this situation   And how to download it   thanks

User · Answer

You can also use the DictReader to iterate dictionaries of   columnname    value         import csv import requests  response   requests get  http   example test foo csv   reader   csv DictReader response iter lines    for record in reader      print record

User · Answer

I like the answers from The Aelfinn and aheld  I can improve them only by shortening a bit more  removing superfluous pieces  using a real data source  making it 2 x  amp  3 x-compatible  and maintaining the high-level of memory-efficiency seen elsewhere  import csv import requests  CSV URL    http   web cs wpi edu  cs1004 a16 Resources SacramentoRealEstateTransactions csv   with requests get CSV URL  stream True  as r      lines    line decode  utf-8   for line in r iter lines        for row in csv reader lines           print row   Too bad 3 x is less flexible CSV-wise because the iterator must emit Unicode strings  while requests does bytes  while the 2 x-only version   for row in csv reader r iter lines       is more Pythonic  shorter and easier-to-read   Anyhow  note the 2 x 3 x solution above won t handle the situation described by the OP where a NEWLINE is found unquoted in the data read  For the part of the OP s question regarding downloading  vs  processing  the actual CSV file  here s another script that does that  2 x  amp  3 x-compatible  minimal  readable  and memory-efficient  import os import requests  CSV URL    http   samplecsvs s3 amazonaws com Sacramentorealestatetransactions csv   with open os path split CSV URL  1    wb   as f            requests get CSV URL  stream True  as r      for line in r iter lines            f write line   n  encode

User · Answer

The following approach worked well for me  I also did not need to use csv reader   or csv writer   functions  which I feel makes the code cleaner  The code is compatible with Python2 and Python 3   from six moves import urllib  DOWNLOAD URL    https   raw githubusercontent com gjreda gregreda com master content notebooks data city-of-chicago-salaries csv  DOWNLOAD PATH   datasets city-of-chicago-salaries csv   urllib request urlretrieve URL DOWNLOAD PATH    Note - six is a package that helps in writing code that is compatible with both Python 2 and Python 3  For additional details regarding six see - What does from six moves import urllib do in Python

User · Answer

From a little search  that I understand the file should be opened in universal newline mode  which you cannot directly do with a response content  I guess    To finish the task  you can either save the downloaded content to a temporary file  or process it in memory   Save as file   import requests import csv import os  temp file name    temp csv csv  url    http   url to file csv  download   requests get url   with open temp file name   w   as temp file      temp file writelines download content   with open temp file name   rU   as temp file      csv reader   csv reader temp file  dialect csv excel tab      for line in csv reader          print line    delete the temp file after process os remove temp file name    In memory    To be updated

User · Answer

this worked nicely for me  from csv import DictReader  f   requests get  https   somedomain com file   content decode  utf-8   reader   DictReader f split   n    csv dict list   list reader

User · Answer

You can update the accepted answer with the iter lines method of requests if the file is very large  import csv import requests  CSV URL    http   samplecsvs s3 amazonaws com Sacramentorealestatetransactions csv   with requests Session   as s      download   s get CSV URL       line iterator    x decode  utf-8   for x in download iter lines decode unicode True        cr   csv reader line iterator  delimiter          my list   list cr      for row in my list          print row

User · Answer

Python3 Supported Code     with closing requests get PHISHTANK URL  stream True    as r          reader   csv reader codecs iterdecode r iter lines     utf-8    delimiter      quotechar   quot            for record in reader             print  record

User · Answer

This should help   import csv import requests  CSV URL    http   samplecsvs s3 amazonaws com Sacramentorealestatetransactions csv    with requests Session   as s      download   s get CSV URL       decoded content   download content decode  utf-8        cr   csv reader decoded content splitlines    delimiter          my list   list cr      for row in my list          print row    Ouput sample     street    city    zip    state    beds    baths    sq  ft    type    sale date    price    latitude    longitude     3526 HIGH ST    SACRAMENTO    95838    CA    2    1    836    Residential    Wed May 21 00 00 00 EDT 2008    59222    38 631913    -121 434879     51 OMAHA CT    SACRAMENTO    95823    CA    3    1    1167    Residential    Wed May 21 00 00 00 EDT 2008    68212    38 478902    -121 431028     2796 BRANCH ST    SACRAMENTO    95815    CA    2    1    796    Residential    Wed May 21 00 00 00 EDT 2008    68880    38 618305    -121 443839     2805 JANETTE WAY    SACRAMENTO    95815    CA    2    1    852    Residential    Wed May 21 00 00 00 EDT 2008    69307    38 616835    -121 439146           Related question with answer  https   stackoverflow com a 33079644 295246    Edit  Other answers are useful if you need to download large files  i e  stream True

User · Answer

To simplify these answers  and increase performance when downloading a large file  the below may work a bit more efficiently    import requests from contextlib import closing import csv  url    http   download-and-process-csv-efficiently python csv   with closing requests get url  stream True   as r      reader   csv reader r iter lines    delimiter      quotechar          for row in reader          print row      By setting stream True in the GET request  when we pass r iter lines    to csv reader    we are passing a generator to csv reader    By doing so  we enable csv reader   to lazily iterate over each line in the response with for row in reader    This avoids loading the entire file into memory before we start processing it  drastically reducing memory overhead for large files

User · Answer

I use this code  I use Python 3    import csv import io import requests  url    http   samplecsvs s3 amazonaws com Sacramentorealestatetransactions csv  r   requests get url  r encoding    utf-8     useful if encoding is not sent  or not sent properly  by the server csvio   io StringIO r text  newline     data      for row in csv DictReader csvio       data append row

[python] Use python requests to download CSV

Examples related to python

Examples related to csv

Examples related to python-requests