Provide the response body from the HTTP server when return the HTTPError

Ideas
1

Hi there, I would like to add a feature to urllib.error.HTTPError

When we use urllib.request.urlopen to read the data from a HTTP server and encounter an error, the error traceback only shows the HTTP response status code and the description of the status code, e.g.: HTTPError: HTTP Error 400: Bad Request. If the HTTP server returns the detailed error message in the response body, the error traceback won’t show it.

For example:
A simple HTTP server that returns 400 with a detailed message.

from fastapi import FastAPI, HTTPException


app = FastAPI()


@app.get("/")
async def main():
    raise HTTPException(
        status_code=400,
        detail="The format of the parameter is wrong!",
    )
    return None

When using curl to get the data from the HTTP server, it shows the status code with the detailed error message.

curl http://127.0.0.1:8000

>>> {"detail":"The format of the parameter is wrong!"}

When using urllib.request.urlopen to get the data, it only shows the status code.

from urllib.request import urlopen

urlopen("http://127.0.0.1:8000")

Traceback (most recent call last):
    ......
urllib.error.HTTPError: HTTP Error 400: Bad Request

I think the traceback shown as follows would be very useful for the end-users.

urllib.error.HTTPError: HTTP Error 400: Bad Request, {"detail":"The format of the parameter is wrong!"}

If this feature requests meaningful, I would like to open a PR for this.

1 Like
2

What if the web server returns a complete HTML page for errors, such as https://www.python.org/fish.html?

Although the docs aren’t super clear, you can access the response body from the HTTPError object with its read() method. You could catch the HTTPError and format the exception yourself with the body from that method.

3

Hm…, you are right. If the web server returns the HTML source code, it gonna mess up the traceback.

The workaround I’ve used now is:

from urllib.request import urlopen
from urllib.error import HTTPError

try:
    urlopen("http://127.0.0.1:8000")
except HTTPError as e:
    print(e.read().decode())

I just think is there a simple solution that can show the detailed message without using the try-except?

4

By Andrew Li via Discussions on Python.org at 09Jun2022 09:29:

Hm…, you are right. If the web server returns the HTML source code,
it gonna mess up the traceback.

The workaround I’ve used now is:

from urllib.request import urlopen
from urllib.error import HTTPError

try:
   urlopen("http://127.0.0.1:8000")
except HTTPError as e:
   print(e.read().decode())

I just think is there a simple solution that can show the detailed message without using the try-except?

From the looks of it, no. See:

It raises URLError on error:

which includes HTTPError.

So you need to catch that exception in order to extract the detailed
message.

Cheers,
Cameron Simpson cs@cskk.id.au