I'm having trouble parsing HTML elements with "class" attribute using Beautifulsoup. The code looks like this
soup = BeautifulSoup(sdata)
mydivs = soup.findAll('div')
for div in mydivs:
if (div["class"] == "stylelistrow"):
print div
I get an error on the same line "after" the script finishes.
File "./beautifulcoding.py", line 130, in getlanguage
if (div["class"] == "stylelistrow"):
File "/usr/local/lib/python2.6/dist-packages/BeautifulSoup.py", line 599, in __getitem__
return self._getAttrMap()[key]
KeyError: 'class'
How do I get rid of this error?
This question is related to
python
html
web-scraping
beautifulsoup
You can refine your search to only find those divs with a given class using BS3:
mydivs = soup.find_all("div", {"class": "stylelistrow"})
Concerning @Wernight's comment on the top answer about partial matching...
You can partially match:
<div class="stylelistrow">
and<div class="stylelistrow button">
with gazpacho:
from gazpacho import Soup
my_divs = soup.find("div", {"class": "stylelistrow"}, partial=True)
Both will be captured and returned as a list of Soup
objects.
Use class_=
If you want to find element(s) without stating the HTML tag.
For single element:
soup.find(class_='my-class-name')
For multiple elements:
soup.find_all(class_='my-class-name')
The following should work
soup.find('span', attrs={'class':'totalcount'})
replace 'totalcount' with your class name and 'span' with tag you are looking for. Also, if your class contains multiple names with space, just choose one and use.
P.S. This finds the first element with given criteria. If you want to find all elements then replace 'find' with 'find_all'.
CSS selectors
single class first match
soup.select_one('.stylelistrow')
list of matches
soup.select('.stylelistrow')
compound class (i.e. AND another class)
soup.select_one('.stylelistrow.otherclassname')
soup.select('.stylelistrow.otherclassname')
Spaces in compound class names e.g. class = stylelistrow otherclassname
are replaced with ".". You can continue to add classes.
list of classes (OR - match whichever present
soup.select_one('.stylelistrow, .otherclassname')
soup.select('.stylelistrow, .otherclassname')
bs4 4.7.1 +
Specific class whose innerText
contains a string
soup.select_one('.stylelistrow:contains("some string")')
soup.select('.stylelistrow:contains("some string")')
Specific class which has a certain child element e.g. a
tag
soup.select_one('.stylelistrow:has(a)')
soup.select('.stylelistrow:has(a)')
This worked for me:
for div in mydivs:
try:
clazz = div["class"]
except KeyError:
clazz = ""
if (clazz == "stylelistrow"):
print div
Update: 2016 In the latest version of beautifulsoup, the method 'findAll' has been renamed to 'find_all'. Link to official documentation
Hence the answer will be
soup.find_all("html_element", class_="your_class_name")
Try to check if the div has a class attribute first, like this:
soup = BeautifulSoup(sdata)
mydivs = soup.findAll('div')
for div in mydivs:
if "class" in div:
if (div["class"]=="stylelistrow"):
print div
A straight forward way would be :
soup = BeautifulSoup(sdata)
for each_div in soup.findAll('div',{'class':'stylelist'}):
print each_div
Make sure you take of the casing of findAll, its not findall
This works for me to access the class attribute (on beautifulsoup 4, contrary to what the documentation says). The KeyError comes a list being returned not a dictionary.
for hit in soup.findAll(name='span'):
print hit.contents[1]['class']
From the documentation:
As of Beautiful Soup 4.1.2, you can search by CSS class using the keyword argument class_
:
soup.find_all("a", class_="sister")
Which in this case would be:
soup.find_all("div", class_="stylelistrow")
It would also work for:
soup.find_all("div", class_="stylelistrowone stylelistrowtwo")
the following worked for me
a_tag = soup.find_all("div",class_='full tabpublist')
Alternatively we can use lxml, it support xpath and very fast!
from lxml import html, etree
attr = html.fromstring(html_text)#passing the raw html
handles = attr.xpath('//div[@class="stylelistrow"]')#xpath exresssion to find that specific class
for each in handles:
print(etree.tostring(each))#printing the html as string
How to find elements by class
I'm having trouble parsing html elements with "class" attribute using Beautifulsoup.
You can easily find by one class, but if you want to find by the intersection of two classes, it's a little more difficult,
From the documentation (emphasis added):
If you want to search for tags that match two or more CSS classes, you should use a CSS selector:
css_soup.select("p.strikeout.body") # [<p class="body strikeout"></p>]
To be clear, this selects only the p tags that are both strikeout and body class.
To find for the intersection of any in a set of classes (not the intersection, but the union), you can give a list to the class_
keyword argument (as of 4.1.2):
soup = BeautifulSoup(sdata)
class_list = ["stylelistrow"] # can add any other classes to this list.
# will find any divs with any names in class_list:
mydivs = soup.find_all('div', class_=class_list)
Also note that findAll has been renamed from the camelCase to the more Pythonic find_all
.
Other answers did not work for me.
In other answers the findAll
is being used on the soup object itself, but I needed a way to do a find by class name on objects inside a specific element extracted from the object I obtained after doing findAll
.
If you are trying to do a search inside nested HTML elements to get objects by class name, try below -
# parse html
page_soup = soup(web_page.read(), "html.parser")
# filter out items matching class name
all_songs = page_soup.findAll("li", "song_item")
# traverse through all_songs
for song in all_songs:
# get text out of span element matching class 'song_name'
# doing a 'find' by class name within a specific song element taken out of 'all_songs' collection
song.find("span", "song_name").text
Points to note:
I'm not explicitly defining the search to be on 'class' attribute findAll("li", {"class": "song_item"})
, since it's the only attribute I'm searching on and it will by default search for class attribute if you don't exclusively tell which attribute you want to find on.
When you do a findAll
or find
, the resulting object is of class bs4.element.ResultSet
which is a subclass of list
. You can utilize all methods of ResultSet
, inside any number of nested elements (as long as they are of type ResultSet
) to do a find or find all.
My BS4 version - 4.9.1, Python version - 3.8.1
Specific to BeautifulSoup 3:
soup.findAll('div',
{'class': lambda x: x
and 'stylelistrow' in x.split()
}
)
Will find all of these:
<div class="stylelistrow">
<div class="stylelistrow button">
<div class="button stylelistrow">
As of BeautifulSoup 4+ ,
If you have a single class name , you can just pass the class name as parameter like :
mydivs = soup.find_all('div', 'class_name')
Or if you have more than one class names , just pass the list of class names as parameter like :
mydivs = soup.find_all('div', ['class1', 'class2'])
This should work:
soup = BeautifulSoup(sdata)
mydivs = soup.findAll('div')
for div in mydivs:
if (div.find(class_ == "stylelistrow"):
print div
Source: Stackoverflow.com