Hi, I’m looking to get some information from a website using python and pycharm. The website in some areas loads the whole page even though there are buttons to push that load more information. For this particular URL it loads 25 items and has a button to load more. The page actually has 100 pieces of information on it. Once it loads and I press the expand all button it can see all 100 items in the code but when I read the page with a request it has not expanded the list yet. Is there a way to get the code to expand before it is read? The page is using the accordion feature also.
Here is just a little of the code:
import requests from bs4 import BeautifulSoup html_text = requests.get('https://www.baseballamerica.com/rankings/2022-top-100-prospects/').text soup = BeautifulSoup(html_text, 'lxml') # print(soup.prettify()) list = soup.find_all('li', class_='clearfix list-individual-rank-item') # print(type(list)) print(len(list)) print(list)
Here is also some page source code:
<li class="clearfix list-individual-rank-item" data-rank="1"> <div class="list-ranking "> <div class="current-ranking"> <span class="rank-number">1</span> <span class="rank-movement rank-up"> <span class="fa-chevron-up fa"></span> </span> </div> <div class="previous-rank"> Last: <span class="previous-rank-number">6</span> </div> </div> <div class="player-details-container"> <div class="player-image"> <img class="lazyload mlb-player-image" data-aspectratio="90/135" height="135" width= "90" data-src="/media/22spm1bl/gunnar-henderson-2022.jpg?preset=playerList" data-src-2x="/media/22spm1bl/gunnar-henderson-2022.jpg?preset=playerList@2x" alt="Gunnar Henderson 2022" /> </div> <div class="player-details"> <h3> <a href="/players/102352/gunnar-henderson/" title="Gunnar Henderson">Gunnar Henderson</a> </h3> <div class="team-position-container"> <span class="team">Orioles</span> <span class="position">SS</span> </div> </div> <div class="player-notes"> <span class="label">Notes:</span> <p><strong>Tools: </strong>Hit: 60 | Power: 60 | Run: 55 | Fielding: 55 | Arm: 60 <br /><strong>Skinny: </strong> The former Alabama prep star has flown through the Orioles system to Triple-A and flashed enticing five-tool potential. His approach, strength and feel for the barrel make him a well-rounded force at the plate, while he continues to show he can play shortstop despite his larger frame. <br /><p><br /><a rel="noopener" href="/teams/1002/baltimore-orioles/organizational/?year=2022&type=M" target="_blank" data-anchor="?year=2022&type=P"><span class="rte-btn">Click here to read his full scouting report</span></a></p></p> </div> <div class="accordion-btn-container text-center"> <span class="btn btn-alt accordion-btn"> <span class="expand-text">More</span> <span class="collapsed-text">Less</span> <span class="fa fa-angle-down accordion-icon"></span> </span> </div> </div> <div class="desktop-only spacer"></div> </li>