I’m looking for a web scraping tutorial on the web. It doesn’t have to be a video, it can just be web pages.
I’ve seen 5 tutorials so far on Youtube. When I do a search I only ask for videos from the past year. I’ve had lots of problems with older videos where the code does not work. 1 gave me the basics and the program worked. 3 could not be completed due to the code not working and I check my code 3 times. 1 was not helpful. So out of 5 videos one was helpful.
Another question: should I use selenium, playwright, requests, or something else to do web scraping?
I appreciate that there’s a lot of low-quality “tutorial” content on Youtube. However, if you’ve already found one that you found useful, it isn’t clear to me why you’re still looking for a tutorial. Is there perhaps something more specific you’re hoping to learn about the task?
As for tools, it really depends on the specifics of what you’re trying to do. If one tool were just clearly better than another in every common case, then you would never have heard of the “another” tool, because nobody would use it or talk about it. Aside from that, it’s very common with “web scraping” tasks that you don’t actually want to try to do web scraping, but instead use some already provided API. (Or even just get the data from another, friendlier source.)
The tutorials I’ve found and watched are VERY basic, from 10-20 minutes. I’m looking for tutorials that are general but more in-depth with a wider variety of examples, perhaps more complex examples. And more tutorials would help me practice my skills.
Because in the end the software I will be automating is very complex and it would not surprise me if the HTML elements in our web-based software at work have no ID attributes at all, or had very few. A unique ID HTML attribute is the best way to identify an HTML element, with a class attribute in second place, depending on the software.
Also the tutorials I’ve watched did not cover popout menus which our software uses extensively and which I will have to click on in the automated Selenium program. You hover the mouse over the menu, a side menu pops out to the right, over over a submenu item, another subsubmenu pops out to the right, etc.
I don’t think there’s really that much depth to web scraping in terms of standard technique. There are some common issues like obeying rate limits or deciding what to do with CAPTCHA; but in terms of actually interpreting the results, that depends too much on the individual case to make a tutorial about. It’s just something where you have to apply problem-solving skills.
For example:
Well, to my understanding, Selenium works on the level of simulating moving the mouse over various elements, waiting for things to appear, etc. So presumably you’d write code that simulates hovering over the menu, then simulates waiting until the side menu pops out (you would play with the website yourself first so that you know what pops out, and come up with an appropriate way to identify it - whether thats By.ID or whatever else), etc. And if you need to repeat some pattern of those steps with some underlying logic to them, maybe you’d… write a for loop or a while loop or some such, right?
From a high level that’s what it looks like. In reality the programmer has use the Chrome or Firefox Inspect feature to find the element to click by getting a unique HTML ID attribute from the element, or a unique combination of class names from the element to click. What that means is, if that element does not have a unique ID, and the classes are not unique for that element, I’m guessing it cannot be clicked very reliably.
But I haven’t done actually testing on our real app to make sure yet.
The same goes for Microsoft Power Automate Desktop which does the same thing as Selenium (basically). But PAD makes it easy to choose the element you want clicked via a GUI. And it uses a full, lengthy CSS selector to make it more likely you got the right element.
PAD has a free version but some features are limited. I may find out that even the number of steps in a “Flow” (what they call it) may be limited. Which is why I want to start with Selenium. We are a small business and like most small businesses, it is difficult to get funding to pay a yearly fee for yet more software. SaaS has bad points as well since the cost never ends.