https://pastebin.com/pXLHTxF1
Hello everyone am trying to scrape for the name and location of users on LinkedIn and am running into a problem that i cant find the element that I’m trying to locate on the webpage. It can login successfully but after logging and opening the target URLs it cant locate the names am trying to locate.Am using selenium.
Have you checked the terms and conditions of the website?
I suspect you are not allowed to do this.
Hey yeah I checked the robots.txt… but the script can login but after logging in it cant locate the element I’ve specified. I have instructed the script to wait for elements to be visible and present… but there is just something I feel am missing. The reason why am logging in first is because you cant view the user profile unlike on Twitter it was just easier to grab the specified texts.
LinkedIn has an explicit policy forbidding registered users from web scraping. See:
https://www.linkedin.com/legal/user-agreement#general
You agree that you will not:
…
2. Develop, support or use software, devices, scripts, robots or any other means or processes (including crawlers, browser plugins and add-ons or any other technology) to scrape the Services or otherwise copy profiles and other data from the Services;
Hey thanks a lot, I never thought of reading the user agreement. I have read rule 13 also… that’s why I was getting the error, now it makes sense.
so there’s no way about it seems
You could consider contacting LinkedIn Help. It may be a bit of a gray area - since many (though not all) users on LinkedIn presumably want their user names to be known (and easily discoverable) to recruiters.
funny enough I think I came across to something like that or am day dreaming… like I can reach out if I want to perform such like an operation. So I think ill try that its a long shot but better than not trying. Thanks alot for you guys coming through.