Is using requests post method the correct way to fill in a search box
on a website?
I have been reading the docs and they all seem to suggest using the
requests get method adding the search query to the url.
It is common to use GET to make search requests.
The problem I have is the website uses a search box which I need to put
text in (I think). So I need to
go to page
Locate search box
Enter text
Submit
Scrape results
Try doing a search but hand. Do you end up as a page with a URL like
this?
This is the result of a DuckDuckGo search for “search term”.
If your results page URL looks like that (base URL, question mark,
parameters) then you can do ths search using a GET.
There’s no need to go to the page and find the form (except yourself, by
hand, to find out the search form URL). Just do a GET request directly
to the search form URL, see what you get back.
The site uses json as well and I am not sure if that would make a difference to how the code would need done.
I can use selenium to carry out what I want to do but I want to do it with just python and requests as selenium is not compatible with my iPhone and Pythonista.
I could use mechanize, but I would still prefer to do it with just python.