Given a URL to a text file what is the simplest way to read the contents of the text file

Question

In Python  when given the URL for a text file  what is the simplest way to access the contents off the text file and print the contents of the file out locally line-by-line without saving a local copy of the text file    TargetURL http   www myhost com SomeFile txt  read the file  print first line  print second line  etc

User · Answer

For me  none of the above responses worked straight ahead  Instead  I had to do the following  Python 3    from urllib request import urlopen  data   urlopen   your url goes here    read   decode  utf-8      Do what you need to do with the data

User · Answer

I do think requests is the best option  Also note the possibility of setting encoding manually  import requests response   requests get  quot http   www gutenberg org files 10 10-0 txt quot     response encoding    quot utf-8 quot  hehe   response text

User · Answer

There s really no need to read line-by-line  You can get the whole thing like this   import urllib txt   urllib urlopen target url  read

User · Answer

Edit 09 2016  In Python 3 and up use urllib request instead of urllib2   Actually the simplest way is   import urllib2    the lib that handles the url stuff  data   urllib2 urlopen target url    it s a file like object and works just like a file for line in data    files are iterable     print line   You don t even need  readlines   as Will suggested  You could even shorten it to      import urllib2  for line in urllib2 urlopen target url       print line   But remember in Python  readability matters   However  this is the simplest way but not the safe way because most of the time with network programming  you don t know if the amount of data to expect will be respected  So you d generally better read a fixed and reasonable amount of data  something you know to be enough for the data you expect but will prevent your script from been flooded    import urllib2  data   urllib2 urlopen  http   www google com   read 20000    read only 20 000 chars data   data split   n     then split it into lines  for line in data      print line       Second example in Python 3     import urllib request    the lib that handles the url stuff  for line in urllib request urlopen target url       print line decode  utf-8     utf-8 or iso8859-1 or whatever the page encoding scheme is

User · Answer

requests package works really well for simple ui as  Andrew Mao suggested import requests response   requests get  http   lib stat cmu edu datasets boston   data   response text for i  line in enumerate data split   n         print f  i     line     o p  0    The Boston house-price data of Harrison  D  and Rubinfeld  D L   Hedonic 1    prices and the demand for clean air   J  Environ  Economics  amp  Management  2    vol 5  81-102  1978    Used in Belsley  Kuh  amp  Welsch   Regression diagnostics 3          Wiley  1980    N B  Various transformations are used in the table on 4    pages 244-261 of the latter  5    6    Variables in order   Checkout kaggle notebook on how to extract dataset dataframe from URL

User · Answer

Just updating here solution suggested by  ken-kinder for Python 2 to work for Python 3   import urllib urllib request urlopen target url  read

User · Answer

import urllib2  f   urllib2 urlopen target url  for l in f readlines        print l

User · Answer

Another way in Python 3 is to use the urllib3 package    import urllib3  http   urllib3 PoolManager   response   http request  GET   target url  data   response data decode  utf-8     This can be a better option than urllib since urllib3 boasts having        Thread safety    Connection pooling    Client-side SSL TLS verification    File uploads with multipart encoding    Helpers for retrying requests and dealing with HTTP redirects    Support for gzip and deflate encoding    Proxy support for HTTP and SOCKS    100  test coverage

User · Answer

import urllib2 for line in urllib2 urlopen  http   www myhost com SomeFile txt        print line

User · Answer

I m a newbie to Python and the offhand comment about Python 3 in the accepted solution was confusing  For posterity  the code to do this in Python 3 is  import urllib request data   urllib request urlopen target url   for line in data            or alternatively  from urllib request import urlopen data   urlopen target url    Note that just import urllib does not work

User · Answer

The requests library has a simpler interface and works with both Python 2 and 3   import requests  response   requests get target url  data   response text

[python] Given a URL to a text file, what is the simplest way to read the contents of the text file?

Examples related to python