TypeError can t use a string pattern on a bytes-like object in re findall

Question

I am trying to learn how to automatically fetch urls from a page  In the following code I am trying to get the title of the webpage   import urllib request import re  url    http   www google com  regex   r  lt title gt       lt  title gt   pattern    re compile regex   with urllib request urlopen url  as response     html   response read    title   re findall pattern  html  print title    And I get this unexpected error   Traceback  most recent call last     File  path to file Crawler py   line 11  in  lt module gt      title   re findall pattern  html    File  C  Python33 lib re py   line 201  in findall     return  compile pattern  flags  findall string  TypeError  can t use a string pattern on a bytes-like object   What am I doing wrong

User · Answer

The problem is that your regex is a string, but html is bytes:

>>> type(html)
<class 'bytes'>

Since python doesn't know how those bytes are encoded, it throws an exception when you try to use a string regex on them.

You can either decode the bytes to a string:

html = html.decode('ISO-8859-1')  # encoding may vary!
title = re.findall(pattern, html)  # no more error

Or use a bytes regex:

regex = rb'<title>(,+?)</title>'
#        ^

In this particular context, you can get the encoding from the response headers:

with urllib.request.urlopen(url) as response:
    encoding = response.info().get_param('charset', 'utf8')
    html = response.read().decode(encoding)

See the urlopen documentation for more details.

User · Answer

You want to convert html  a byte-like object  into a string using  decode  e g   html   response read   decode  utf-8      See Convert bytes to a Python String

[python] TypeError: can't use a string pattern on a bytes-like object in re.findall()

Examples related to python

Examples related to python-3.x

Examples related to web-crawler