Webscraping - iherb - ratings

Python Help
1

Hi all.
I’m trying to web scarp iherb but I’m having trouble extrating the data of the ratings (aka the number of stars of each product). Here’s my work:

import requests
from bs4 import BeautifulSoup as soup

url = ‘Whey Protein • Whey Protein Isolate and Concentrate | iHerb’
html = requests.get(url=url,headers=header_1)
html.status_code

bsobj = soup(html.content, ‘lxml’)

whey = bsobj.findAll(‘div’,{‘class’:‘product-inner product-inner-wide’})

for item in whey:
dict_1 = {
“title” : item.find(‘div’,{“class”:“product-title”}).text,
“link” : item.find(‘a’,{“class”:“absolute-link product-link”})[‘href’],
“reviews” : int(item.find(‘a’,{“class”:“rating-count”}).text),
“stars” : item.find(‘a’,{“class”:“stars”}).text
}

so the problem is I can’t get the number of stars out of 5 of each product. What should I do? Many thanks.

2

Is the stars in the HTML that you fetched?
I suggest that you save the page in a file after your requests.get()
so that you can check where the stars are.

Often these days a web page is made on the by javascript code.
If so then you will not find the stars in the HTML.

You may need to use a tool like selenium to load the web page then you can query for the stars after the javascript has run.