Web scraping using Python- dynamic websites

Python Help
1

Hello everyone,
I am a beginner level coder in Python and wanted to explore the field of Web Scraping. I am familiar with Beautiful soup however it has not been very helpful for my task.
I am working on a website, from which I need to count the number of samples which start with a specific code. I need to count the number of samples at a certain time, as it keeps changing as the day progresses. This website is dynamic and I believe I need to use advanced web scraping.
Can anyone please suggest a good source of information (website or blog or video) that I can use for understanding web scraping using python for complex web sites?
If you have worked on a similar task before, any suggestions are also most welcome.
Any help is appreciated. Thank you

Gajraj Rathore

1 Like
2

Most (all?) web scrapers in languages don’t support dynamic JavaScript-controlled web content. However, that’s good news: instead of web scraping, instead watch the requests in the ‘network’ tab of your browser’s developer tools (F12 in Firefox) and use the requests Python library to emulate that

2 Likes
3

All right, I’ll try that. Thank you

1 Like
4

I would normally recommend using selenium, but recently I’ve been looking into playwright and I’m really liking it.

I would suggest you take a look t it Fast and reliable end-to-end testing for modern web apps | Playwright Python