and data are shown in the browser. I want to write a Python program that converts this page to text or any other format that can be saved easily.
I tried:
but still can’t reach the data. The connection is closed because of the unknown user agent. However, I can see data when I manually open the Mozilla browser.
As a side note be aware of modern sites that use js frameworks like React/Vue/(how many are there really?). When you open it in your browser, the page that a server sends you is mostly empty. It contains small HTML snippent and lots of js imports. Browser gets them, executes, so content of the page is filled via JS.
It might not be a problem for your current task, but IMO the majority of new sites follow this pattern.
If it’s done right, this actually makes it easier to scrape, because the relevant data is either stored in the page as inline JSON, or fetched in a separate API call (in which case you ignore the HTML and JS, and just do the same fetch).