Python request (post)

Is using requests post method the correct way to fill in a search box on a website?

I have been reading the docs and they all seem to suggest using the requests get method adding the search query to the url.

The problem I have is the website uses a search box which I need to put text in (I think). So I need to

  1. go to page
  2. Locate search box
  3. Enter text
  4. Submit
  5. Scrape results

The site uses json as well and I am not sure if that would make a difference to how the code would need done.

Any help with what I should be looking at would be greatly appreciated.

Is using requests post method the correct way to fill in a search box
on a website?
I have been reading the docs and they all seem to suggest using the
requests get method adding the search query to the url.

It is common to use GET to make search requests.

The problem I have is the website uses a search box which I need to put
text in (I think). So I need to

  1. go to page
  2. Locate search box
  3. Enter text
  4. Submit
  5. Scrape results

Try doing a search but hand. Do you end up as a page with a URL like
this?

 https://duckduckgo.com/?q=search+term&bext=mcl&atb=v223-1

This is the result of a DuckDuckGo search for “search term”.

If your results page URL looks like that (base URL, question mark,
parameters) then you can do ths search using a GET.

There’s no need to go to the page and find the form (except yourself, by
hand, to find out the search form URL). Just do a GET request directly
to the search form URL, see what you get back.

The site uses json as well and I am not sure if that would make a difference to how the code would need done.

You’ll need to elaborate a bit on this.

Cheers,
Cameron Simpson cs@cskk.id.au

Hi Cameron,

Thanks for commenting.

[Preformatted text](https://online.transport.wa.gov.au/webExternal/registration/;jsessionid=xme-A4_hQ3DKLrW5Vmijzn-5Dzn154tqqldXSg8aiaB_JNQRjyQg!1563753950!-1720409949?0)

This is the page that the search box is on.

https://online.transport.wa.gov.au/webExternal/registration/wicket/page?1

This is the results page after a search.

I can use selenium to carry out what I want to do but I want to do it with just python and requests as selenium is not compatible with my iPhone and Pythonista.

I could use mechanize, but I would still prefer to do it with just python.

https://online.transport.wa.gov.au/webExternal/registration/?0-1.IBehaviorListener.2-layout-layout_body-registrationRequestForm-plate&random=0.49404731684915326

This is from inspecting the header after I have pasted in the full Rego

https://online.transport.wa.gov.au/webExternal/registration/?0

This is what the URL looks like before any information is passed into the search box.

I assume i need to do something with this part of the URL

-1.IBehaviorListener.2-layout-layout_body-registrationRequestForm-plate&random=0.49404731684915326)

I dont understand how i can replace the plate&random.

The request content type is

application/x-www-form-urlencoded;charset=UTF-8

requests.post('https://httpbin.org/post', data={'key':'value'})

Can anyone explain this for me? I think i need to fill the dictionary with the rego something like

requests.post('https://online.transport.wa.gov.au/webExternal/registration/post', data={'plate' : 'rego'}

Getting confused cant rreally locate an example of what i am trying to do

import requests

URL = "https://online.transport.wa.gov.au/webExternal/registration/?0"

rego = "1hdv242"

PARAMS = {'plate':rego}

response = requests.get(url = URL, params = PARAMS)

results = requests.get('https://online.transport.wa.gov.au/webExternal/registration/wicket/page?2')


#print(response.headers)
print(results.json)
print(results.content)
print(results.text)
print(results.headers)
print(results.body)

Some code i have been trying.

Seems to pull some stuff back but not what i want…

And also says page has expired :confounded: