How can I read the contents of an URL with Python

Question

The following works when I paste it on the browser    http   www somesite com details pl urn 2344   But when I try reading the URL with Python nothing happens    link    http   www somesite com details pl urn 2344   f   urllib urlopen link              myfile   f readline      print myfile   Do I need to encode the URL  or is there something I m not seeing

User · Answer

usr bin python   - - coding  utf-8 - -   Works on python 3 and python 2    when server knows where the request is coming from   import sys  if sys version info 0     3      from urllib request import urlopen else      from urllib import urlopen with urlopen  https   www facebook com    as       url      data   url read    print data    When the server does not know where the request is coming from    Works on python 3   import urllib request  user agent          Mozilla 5 0  Windows  U  Windows NT 5 1  en-US  rv 1 9 0 7  Gecko 2009021910 Firefox 3 0 7   url    https   www facebook com   headers     User-Agent   user agent   request   urllib request Request url  None  headers  response   urllib request urlopen request  data   response read   print data

User · Answer

None of these answers are very good for Python 3  tested on latest version at the time of this post   This is how you do it    import urllib request  try     with urllib request urlopen  http   www python org    as f        print f read   decode  utf-8    except urllib error URLError as e     print e reason   The above is for contents that return  utf-8   Remove  decode  utf-8   if you want python to  quot guess the appropriate encoding  quot  Documentation  https   docs python org 3 library urllib request html module-urllib request

User · Answer

The URL should be a string   import urllib  link    http   www somesite com details pl urn 2344  f   urllib urlopen link             myfile   f readline     print myfile

User · Answer

I used the following code   import urllib  def read text          quotes   urllib urlopen  https   s3 amazonaws com udacity-hosted-downloads ud036 movie quotes txt         contents file   quotes read         print contents file  read text

User · Answer

We can read website html content as below    from urllib request import urlopen response   urlopen  http   google com    html   response read   print html

User · Answer

retrieving data from url   only for python 3  import urllib request  def main      url    http   docs python org     retrieving data from URL   webUrl   urllib request urlopen url    print  Result code      str webUrl getcode        print data from URL    print  Returned data  -----------------     data   webUrl read   decode  utf-8     print data   if   name         main       main

User · Answer

For python3 users  to save time  use the following code   from urllib request import urlopen  link    https   docs scipy org doc numpy user basics broadcasting html   f   urlopen link  myfile   f read   print myfile    I know there are different threads for error  Name Error  urlopen is not defined  but thought this might save time

User · Answer

from urllib request import urlopen    if has Chinese  apply decode   html   urlopen  https   blog csdn net qq 39591494 article details 83934260   read   decode  utf-8   print html

User · Answer

To answer your question   import urllib  link    http   www somesite com details pl urn 2344  f   urllib urlopen link  myfile   f read   print myfile    You need to read    not readline    EDIT  2018-06-25   Since Python 3  the legacy urllib urlopen   was replaced by urllib request urlopen    see notes from https   docs python org 3 library urllib request html urllib request urlopen for details    If you re using Python 3  see answers by Martin Thoma or i n n m within this question  https   stackoverflow com a 28040508 158111  Python 2 3 compat  https   stackoverflow com a 45886824 158111  Python 3   Or  just get this library here  http   docs python-requests org en latest  and seriously use it     import requests  link    http   www somesite com details pl urn 2344  f   requests get link  print f text

User · Answer

A solution with works with Python 2 X and Python 3 X makes use of the Python 2 and 3 compatibility library six   from six moves urllib request import urlopen link    http   www somesite com details pl urn 2344  response   urlopen link  content   response read   print content

User · Answer

You can use requests and beautifulsoup libraries to read data on a website  Just install these two libraries and type the following code  import requests import bs4 help requests  help bs4   You will get all the information you need about the library

[python] How can I read the contents of an URL with Python?

Examples related to python