Mocking a webpage response for testing

Python Help
1

I’m using httpx for my wrapper that does a GET request to a page and then parses it’s HTML to convert it into a python object. I have also written tests for it using pytest.
I wanted to mock the response to get consistent results but I cannot seem to figure it out. I found respx for httpx mocking so that’s what I’m using.

How would I go about:

  1. Saving the page response
  2. loading it up with mock
  3. Run tests

My initia idea was to loop over all the links in my tests and generate pickle.dumped files and load them in later for testing

import pickle

import httpx
import respx

text = "..."
base_url = "https://example.com"
#  pickle.dump(httpx.get(f"{base_url}/2323453"), open("2323453.response", "wb"))

@respx.mock
def test_decorator() -> None:
    resp = pickle.load(open("2323453.response", "rb"))
    my_route = respx.get(f"{base_url}/2323453").mock(return_value=resp)
    response = httpx.get(f"{base_url}/2323453")
    assert my_route.called
    assert response.text == text

This raises an error:

Error 
test.py:12:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv\Lib\site-packages\httpx\_api.py:198: in get
    return request(
.venv\Lib\site-packages\httpx\_api.py:106: in request
    return client.request(
.venv\Lib\site-packages\httpx\_client.py:827: in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
.venv\Lib\site-packages\httpx\_client.py:928: in send
    raise exc
.venv\Lib\site-packages\httpx\_client.py:922: in send
    response.read()
.venv\Lib\site-packages\httpx\_models.py:813: in read
    self._content = b"".join(self.iter_bytes())
.venv\Lib\site-packages\httpx\_models.py:829: in iter_bytes
    for raw_bytes in self.iter_raw():
.venv\Lib\site-packages\httpx\_models.py:883: in iter_raw
    for raw_stream_bytes in self.stream:
.venv\Lib\site-packages\httpx\_client.py:126: in __iter__
    for chunk in self._stream:
.venv\Lib\site-packages\httpx\_transports\default.py:113: in __iter__
    for part in self._httpcore_stream:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <httpx._content.UnattachedStream object at 0x000001FC432ADF40>

    def __iter__(self) -> Iterator[bytes]:
>       raise StreamClosed()
E       httpx.StreamClosed: Attempted to read or stream content, but the stream has been closed.

.venv\Lib\site-packages\httpx\_content.py:98: StreamClosed
================================================================================= short test summary info =================================================================================
FAILED test.py::test_decorator - httpx.StreamClosed: Attempted to read or stream content, but the stream has been closed.
==================================================================================== 1 failed in 0.53s ====================================================================================

So it seems my idea to pickle them with their IDs and load them probably won’t work. So how should i go about it?

2

We need to mock HTTP requests in our tests all the time to keep them from actually reaching out to the internet. We use the responses library (specifically RequestsMock) to help us do this. From the code you wrote it looks like this has the functionality you need.

3

Hmm I don’t think that’ll help my case (or maybe I’m not understanding it). For some clarity here’s my usecase:

I’m using httpx (and respx) over requests (and responses) because I need the async interface.

I went with generating pickled responses once, saving them to file and loading them up with pickle.load going forward for mocking instead of passing it inline is because the response is giant and pasting the whole thing would be super messy (I need the .content since I parse the whole page)

4

RequestsMock just mocks the HTTP response. You provide it with the content to return. We use it to return text based (HTML/JSON/XML) as well as binary content all the time. AFAIK it should have no impact whether the code you are testing is async or not, it just captures the request and provides the canned response.