How to Scrape all pages from Realtor website that has pagination using Python?

I am needing a property list generated for Multifamily Properties in the St Louis Missouri area. I would be looking for a complete list of all duplex, tri plex, fourplex and 5+ Multifamily Properties within the ST LOUIS MSA. Data I would be needing is Property Address, Square Footage, Unit Count, Owner Name, Owner Address, Owner Phone # and Owner E-mail. I decided to scrape through different website for fetching this information - Realtor.com & Zillow.com, etc. I am using Python in Pycharm IDE. Making use of Beautifulsoup. I am stuck while paginating. Getting 308 error when reading from the URL I am dynamically forming. Here is my code

from urllib.request import urlopen
from bs4 import BeautifulSoup
import urllib.request
import urllib.parse
import urllib.error
#from bs4 import BeautifulSoup
import ssl
import json
import ast
import os
from urllib.request import Request, urlopen

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter Url- ')

req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()

soup = BeautifulSoup(webpage, 'html.parser')

links = []
for link in soup.findAll('a'):
    try:
        if(str(link.get('href')).__contains__("realestateandhomes-detail")):
            url = str(url).replace('realestateandhomes-search','')
            newURL =  url[0:len(url) - 13] + link.get('href')
            print(newURL)
            print('1')
            req1 = Request(newURL, headers={'User-Agent': 'Mozilla/5.0'})
            print('2')
            webpage1 = urlopen(req1).read()
            print('3')
            soup1 = BeautifulSoup(webpage1, 'html.parser')
            for x in soup1.findAll('bed'):
                print(str(x))
    except Exception as e:
        print(str(e))

When we run this script it asks for the URL. I enter “St. Louis, MO Real Estate - St. Louis Homes for Sale | realtor.com®”. And then we can see this script failing every time it tries to run “urlopen”. Have included those print statements for debugging.

The for loop is used for forming the pagination URL dynamically. Requesting the URL formed works fine. But, it throws 308 error when I try doing this from Python. Is this because of security restrictions on the website? Or am I missing anything?

I tried suggestions made in these links. But, they were not of much help - Scraping realtor data with beautifulsoup

How to scrape page with pagination with python BeautifulSoup

It is often the case with modern websites to generate the page using JavaScript.
That being the case there is nothing to “scrape” from the URL.
You may well need to load the page into a browser and grab the results of the JavaScript running.

Look at using selenium to allow you to do this.

Thanks, Barry
I am not a Developer. Despite being in the industry for over 2 decades my engagement as Developer was for a very small time. I am losing a lot of time trying to get the right resource with actionable information. I will google the suggestion you have made. In the meantime, if you have something available readily, can you point me to this link\article\blog, etc that I can refer to, please?

Please be advised that scraping is explicitly against the Terms of Service for those sites :man_shrugging:

Among other things, you agree not to … engag[e] in the practices of “screen scraping”

Agree with you. Will stop the idea to scrape these sites immediately. Thanks