So I am trying to scrape a website for a table and create a csv file (or really any type of file)…like I have done forever without issue. I am using python and beautifulsoup… The website has changed their front end and no longer paginate the URL. So there are like 30 pages but when you click to the next page, the URL remains the same. I cannot figure out how to extract the data for the pages other than the 1st one.
Normally I would say use the inspect functionality of the browser to look at the network traffic and find the API. And you can do that. But man is it really screwy. It looks like there’s a call for each individual player. The data’s all there but really weirdly. And I don’t see how it knows what to pull for the next page.
Yes… I tried looking at network tab in inspect but it seems I would have to call each player… That would be way too time consuming… For right now I am just manually copying and pasting all pages… Pain.