TypeError can t use a string pattern on a bytes-like object in re findall

Question

I am trying to learn how to automatically fetch urls from a page  In the following code I am trying to get the title of the webpage   import urllib request import re  url    http   www google com  regex   r  lt title gt       lt  title gt   pattern    re compile regex   with urllib request urlopen url  as response     html   response read    title   re findall pattern  html  print title    And I get this unexpected error   Traceback  most recent call last     File  path to file Crawler py   line 11  in  lt module gt      title   re findall pattern  html    File  C  Python33 lib re py   line 201  in findall     return  compile pattern  flags  findall string  TypeError  can t use a string pattern on a bytes-like object   What am I doing wrong

User · Accepted Answer

You want to convert html  a byte-like object  into a string using  decode  e g   html   response read   decode  utf-8      See Convert bytes to a Python String

User · Answer

The problem is that your regex is a string  but html is bytes    gt  gt  gt  type html   lt class  bytes  gt    Since python doesn t know how those bytes are encoded  it throws an exception when you try to use a string regex on them   You can either decode the bytes to a string   html   html decode  ISO-8859-1      encoding may vary  title   re findall pattern  html     no more error   Or use a bytes regex   regex   rb  lt title gt       lt  title gt                  In this particular context  you can get the encoding from the response headers   with urllib request urlopen url  as response      encoding   response info   get param  charset    utf8       html   response read   decode encoding    See the urlopen documentation for more details

[python] TypeError: can't use a string pattern on a bytes-like object in re.findall()

Examples related to python

Examples related to python-3.x

Examples related to web-crawler