Reading text of an open url using python

Python Help
1

I want to read data from a device to an IP address using Python. I can open the web browser page using Python:

import webbrowser
webbrowser.open('IP address', new=2)

and data are shown in the browser. I want to write a Python program that converts this page to text or any other format that can be saved easily.
I tried:

import urllib.request as ul

url = 'IP address'
req = ul.Request(url, headers={'User-Agent': ' Mozilla/5.0'})
client = ul.urlopen(req)
htmldata = client.read()
print(htmldata)

but still can’t reach the data. The connection is closed because of the unknown user agent. However, I can see data when I manually open the Mozilla browser.

Any help is appreciated!

2

I don’t know for sure, because I’ve never done what you are doing, but it could be that you need to expand on the User-Agent info?

3

You can go here to see the full User-Agent string that your browser sends:

https://www.whatismybrowser.com/detect/what-is-my-user-agent/

and copy and paste that into the code.

4

As a side note be aware of modern sites that use js frameworks like React/Vue/(how many are there really?). When you open it in your browser, the page that a server sends you is mostly empty. It contains small HTML snippent and lots of js imports. Browser gets them, executes, so content of the page is filled via JS.

It might not be a problem for your current task, but IMO the majority of new sites follow this pattern.

5

If it’s done right, this actually makes it easier to scrape, because the relevant data is either stored in the page as inline JSON, or fetched in a separate API call (in which case you ignore the HTML and JS, and just do the same fetch).

1 Like
6

Thank you for your replies. I finally solved the problem by importing the socket library. Thus there is no need to open a browser.