Need Help with Web Scraping

I want to write a program that extracts all the words from a web page and puts them into a file so that the program can scan through for particular words. I can do the latter but extracting the words seems to only be possible in small chunks (each class on the Inspect page). How can I extract all the words without having to write every class?

I’ll give a couple hints: I’d say use requests to load the web page and then bs4 to parse it.

Here’s some example code:

import collections
import requests # pip install requests
import string
from bs4 import BeautifulSoup # pip install beautifulsoup4 
from pprint import pprint

def remove_punctuation(s: str) -> str:
    Attempts to trivialy remove punctuations from a string
    return s.translate(str.maketrans("", "", string.punctuation))

def get_word_counter(url: str) -> collections.Counter:
    Returns a counter for text words on a given url

    Note: All words are converted to lowercase
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    all_text = soup.text
    all_text_no_punctuation = remove_punctuation(all_text)
    counter = collections.Counter()
    for word in all_text_no_punctuation.lower().split():
        counter[word] += 1
    return counter

def print_word_count(url: str):

if __name__ == "__main__":

Running that on Python 3.11 gives:

Counter({'domain': 4,
         'in': 3,
         'example': 2,
         'this': 2,
         'for': 2,
         'use': 2,
         'is': 1,
         'illustrative': 1,
         'examples': 1,
         'documents': 1,
         'you': 1,
         'may': 1,
         'literature': 1,
         'without': 1,
         'prior': 1,
         'coordination': 1,
         'or': 1,
         'asking': 1,
         'permission': 1,
         'more': 1,
         'information': 1})

collections.Counter is good for counting things like words, it’s a slightly fancy dict. so I’ve used it here. For more info on that, see: collections — Container datatypes — Python 3.11.4 documentation.