New project: httpx-pycurl

Async-SIG
1

I’m finally learning asyncio with the release of httpx-pycurl, a httpx transport that runs through curl, driving curl’s event loop with asyncio’s. It’s not pretty but it’s fast, with most httpx tests passing when patched to use httpx-pycurl instead of the default transport. Performing 1000 http/2 requests with gather(),

# First run:
Fetch 1000x https://httpbingo.org/get
aiohttp: 1.029s
httpx: 1.369s
httpx_pycurl: 0.637s
niquests: 0.715s

# Second run:
Fetch 1000x https://httpbingo.org/get
aiohttp: 0.927s
httpx: 1.346s
httpx_pycurl: 0.677s
niquests: 0.655s

Let me know if it is useful to you.

Daniel Holth

2

Looks good. Is it much work to get it to work with httpxyz?

3

I’m aware of the difficulties with Python http libraries. Might be able to either try / except import httpxyz; allow the module to accept a similar-to-httpx module used as an initializer; implement very-similar requests transport adapters interface such that httpx and requests might share a single transport instance. The adapter between Request / Response objects is much less interesting than being able to drive curl’s event loop from asyncio.

4

Using curl’s event loop did sound interesting to me too - it’s incredibly powerful, and I’ve barely scratched the surface of it.

5

It’s a little confusing to get working. Once it is working there is not much to it. It gives you file handles and you call back into it when one of those handles is readable, or writable; and it just works, and it works efficiently.

1 Like