I want to scrape this big data is there a way to scrape them in 24 hours?

Python Help
1

So i need the data in this url but the data is keep in different pages so i need to get in the pages one by one but there is about 1000 or more pages and the beautiful soup and request always return an error. The only tool i can try is selenium but it so slow and return error if keep there too long is there a way to get the data in 24 hours??

Here is the link
https://egov.basgov.com/

Thanks

2

Can you show us your code with requests? There are usually ways to make requests work unless the website employs advanced client-side trickeries, and the code of that particular website looks fairly primitive.

1 Like
3

Honestly, I’d just go with Selenium for its simplicity even if it takes longer to run. It’s also highly likely that you’ll be interrupted with a captcha that prevents scraping.

Write your code such that it can resume from where it left off if an error stops it partway.