Crawling-python

Hi guys, let me introduce my self.It is my first post here, I hope somebody could guide me about the metasearch engine, I tried to search info about web crawling but all that I found was about scraping, all I got was info related with selenium, bs4,etc.
If somebody know any book about search engines, or videos or courses, or any stuff related with web index and browser envolving python let me know to try it, what I want to do it’s a “metasearch engine” just to learn nothing really profesional, just get some url, what I got for the moment it’s a script that gets some urls from google and writing them in a file.txt .In order to do this I used search() from googlesearch but I though that there must be some better and efficent way

Hi Carl, and welcome!

https://duckduckgo.com/?q=how+to+write+a+web+spider+python

https://www.bing.com/search?q=how+to+write+a+web+spider+python

https://search.lilo.org/?q=how%20to%20write%20a%20web%20spider%20python

https://www.google.com/search?q=how+to+write+a+web+spider+python

Some of the hits are to do with scraping pages, but many of them are
about spiders and crawling the web.

Note: Google will definitely filter your results according to whatever
you have searched for in the past. DuckDuckGo claims to not do that, but
it will filter your results according to your region.

With DDG set to search with the location “Australia”, the very first
result I get is this:

With no location set, I get these:

and more. If your search results are not very good, it is really worth:

  • trying different search terms, both more specific and less;

  • try another search engine, Google’s popularity is no longer a
    reflection of objective superiority over the competition;

  • don’t filter by location;

  • try searching in a private/anonymous tab, to see what Google doesn’t
    think you will be interested in

  • scroll down the page, don’t stop at the first one or two results;

  • and click through to the second, third page.

Thanks Steven, I’ll keep that for the next time, those results must be because of the filters for things that I had searched for. Thanks for the advices I won’t forget them