Choosing an HTTP Client Dependency for Sync and Async APIs

Python Help
1

Disclaimer: this is not a well-thought-out proposal. This is me thinking out loud after staring at a dependency problem that affects me personally as both a library maintainer and a consumer of similar libraries and feeling unhappy with every solution I’ve come up with so far.

Let me start with the use case:

  • I have an API wrapper.
  • It doesn’t rely on anything beyond the usual, universally common HTTP client features.
  • It offers both FooClient and AsyncFooClient.
  • They share everything except the actual HTTP client implementation.

Example:

@dataclass
class Movie:
    title: str
    director: str
    year: int
    genre: str
    rating: float

@dataclass
class TVShow:
    title: str
    creator: str
    seasons: int
    genre: str
    rating: float

class FooClient:
    def __init__(self, client: SomeHTTPClient | None = None) -> None:
        self.client = client or SomeHTTPClient()

    def get_movie(self, title: str) -> Movie:
        r = self.client.get(f"https://api.foo.com/movies/{title}")
        # Imagine there's code here
        return Movie(...)

    def get_tv_show(self, title: str) -> TVShow:
        r = self.client.get(f"https://api.foo.com/tvshows/{title}")
        # And here
        return TVShow(...)

class AsyncFooClient:
    def __init__(self, client: SomeAsyncHTTPClient | None = None) -> None:
        self.client = client or SomeAsyncHTTPClient()

    async def get_movie(self, title: str) -> Movie:
        r = await self.client.get(f"https://api.foo.com/movies/{title}")
        # Imagine there's code here, but this time it's async
        return Movie(...)

    async def get_tv_show(self, title: str) -> TVShow:
        r = await self.client.get(f"https://api.foo.com/tvshows/{title}")
        # And here
        return TVShow(...)

Now the problem:

Realistically, a user will only ever need one of these, either sync or async. That leaves me with these options:

  1. Unconditionally depend on both a sync and an async HTTP library.
    This means I’m forcing two arbitrary HTTP clients on users. If they already use something else in their app, congrats, they now have three HTTP libraries in their dependency tree for absolutely no good reason.

  2. Pick a favorite (sync or async), depend on it by default, and throw the other one in extras.
    If you use the “favored” one, you save a dependency. If you don’t, you’re right back to option 1.

  3. Move both to extras.[1]
    Technically valid, but this basically ships a library that’s broken by default and relies on docs and error messages to tell users how to make it work.

  4. Split into three packages: core + sync + async.
    This still forces at least one HTTP library of my choice on downstream users, just with more packaging and more maintenance overhead.

I honestly dislike every single one of these.

So far, I’ve been using httpx purely because it gives me both sync and async in one dependency, which is the least terrible version of option 1. But even then, if you’re using requests in your app and want to depend on my library, you now have two HTTP libraries in your environment for no real reason.

Now, what actually prompted this post is that httpx’s long-term future feels a bit uncertain to me[2][3][4], so I’m back to rethinking which flavor of bad decision I’m going to lock into if I have to switch HTTP clients again.

Sometimes, I really wish the stdlib just defined something like HTTPClientProtocol and AsyncHTTPClientProtocol so third-party HTTP libraries could interoperate properly. My wrapper could just accept “anything that satisfies HTTPClientProtocol” and be done with it. We have standardized similar boundaries before (e.g., WSGI in PEP 333 and the DB-API in PEP 249), so the idea itself isn’t totally unprecedented, but I honestly don’t know what combination of technical and social factors made those succeed, or whether that would translate cleanly to HTTP clients today.

After all, HTTP itself is standardized, and most Python HTTP clients already expose similar APIs. requests, httpx, and niquests are almost swappable in practice, with aiohttp being the biggest outlier.

Even in that world, though, I’d still want to ship one default HTTP library so the package works out of the box, ideally something the user can opt out of entirely if they’re passing their own client.

So what are people doing in such a scenario? Or is this just not a common enough problem?

  1. This can become a viable alternative if PEP 771 - Default Extras gets accepted. Personally, I’m very excited for that PEP, and I really hope it gets accepted. ↩︎

  2. https://github.com/encode/httpx/issues/3644 ↩︎

  3. https://github.com/encode/httpx/discussions/3344#discussioncomment-14170874 ↩︎

  4. I get that httpx is a FOSS project maintained by humans with limited time. This is not a complaint about the maintainers. ↩︎

3 Likes
2
1 Like
3

This sounds like you’re a considerate, thoughtful maintainer. I admire your restraint too - you didn’t considering rolling your own http client?.. xkcd incoming :wink:

If all the API really does, is make calls to client.get (it doesn’t also use post, etc.?), then rolling your own interface API layer and having multiple optional back-ends isn’t such a huge burden.

Have you thought about deprecating the sync code, either dropping FooClient, or merging it with AsyncFooClient)? As long as the library has one working implementation, I don’t think you need to feel bad about a breaking change. Especially if it’s more of a forced ‘upgrade’, requiring users on the sync code to add a dependency, and get improved responsiveness as a result of that.

1 Like
4

It is not just GET, but that level of detail is not really the point. What I meant is that a library like mine only uses standard HTTP things such as GET, POST, DELETE, etc, along with request data, headers, and cookies. There is nothing unusual beyond what a typical HTTP client already supports.

1 Like