yeah yeah used I used content instead, away from the norm lol. Now that’s whats up, I’ve now learnt current and previous element which is concise and neater way of handling the logic
Btw I stumbled into some weird error the other day while running the script but its network related… so the gist is that the response I was getting was a redirect to Pharmacy points in Russia instead of malls in Dubai lol… l understand the 2gis is a Russian entity but why that redirect
Beats me, but that’s a whole new topic/thread.
Yeah it was just weird… nywy lemme finish up the script… till next time
So it violates the rules of this forum if the person doesn’t have the agreement to do that?
I don’t know if there is an explicit policy (couldn’t find it). I just don’t feel comfortable helping someone if I’d be helping them do something that is either illegal or known to be burdensome for those websites. At the very least it’s the polite thing to do - verify that the website allows scraping. (Of course those big companies often also engage in dubious actions - and there are lots of gray areas.)
When newbies engage with this, I think it’s also best to point out to them that
- if they only half know what they are doing, they might inadvertently be running a DoS attack against the website
- there may be easier, unproblematic alternatives for getting the info they think they can only get by scraping
For an interesting gray area case: https://law.justia.com/cases/federal/appellate-courts/ca9/17-16783/17-16783-2022-04-18.html
TL;TR: LinkedIn tried to prevent another company from accessing (scraping) its publicly available member profiles. The court ruled that LinkedIn could not do so. (I have to say that much of the legalese in that court order makes my poor software developer’s head hurt…)
1 Like
Hey relax Hans, I totally understand your values and principles… like you said it’s a gray area and we tend to favor mostly what works for us…plus I also do it with moderation,… nywy check this
Thanks - It seems you’re all set then. So, it seems you did not notice that I hinted at the “logical” method in my initial reply. In general those websites are pretty hairy.
No your initial reply came out a little aggressive lol the fact that I didn’t have consent lol nywy it’s all good now. Yeah that’s what I thought sometimes they are pretty much tangled like I said I thought it was just easy peazy to locate(the next button). But what is trivial for me might not be trivial for all. The reason why I wanted the next button was to make it more dynamic in the case that there if there are no more pages then the script stops. But now I have to know how many products per page to get the number of pages to initialize the for loop… which still gets the job done.
Counterpoint: all your replies have been a little defensive.
Everyone here volunteers their time and effort to help people for free. It is 100% reasonable for anyone to drill down into things up front, if they want to be certain the products of their help won’t land outside their own ethical boundaries.
Hey Bryan, I totally acknowledged his principles and I think now we are on the same boat with Hans. And he replied: “Thanks - It seems you’re all set then”. And you forgot the lol part and the rest of the statement. I think this shouldn’t be back and forth thread and we are good.