Does anyone have a hard dependency on `_pyio`? (Proposing PEP 399 exception)

The Steering Council asked for this post so the proposal gets broader visibility (steering-council#372). If you use _pyio in a project, I’d like to hear from you so we can make sure we have a good path forward while making the CPython I/O stack easier to develop and maintain.

I’m working on some significant changes to Buffered I/O in CPython. One challenge in making improvements to the I/O stack is that there are two implementations which currently differ significantly but need to be maintained to have:

  1. Identical behavior (PEP 399)
  2. Backwards compatibility (PEP 387)

Unfortunately _pyio and _io have distinct algorithms and behavior around major cases. _io for instance has a unified buffered C object which is used by the various Buffered* classes while _pyio has distinct buffering logic implementations in BufferedReader and BufferedWriter.

What I would like to do, at least in the short term, is to drop _pyio as it has no known users. That needs a PEP 399 exception: C _io would no longer have a pure-Python eqivalent in _pyio. This is effectively codifying the current maintenance state: It sometimes gets changed with other _io changes but not always.

With just one implementation it will be easier to land features such as free-threading safety as well as performance improvements. Security changes and long-standing bugs take fewer steps to fix.

This proposal does not imply there could never be another _pyio. I personally find it useful for prototyping changes. The goal is to reduce development and maintenance burden.

Known cases for keeping _pyio

  1. PyPy or some unknown third party might have a hard requirement on it. PyPy uses a native _io. I audited open-source projects and couldn’t find active users. If your project has a hard dependency on it, please let us know.
  2. Contribution and interest in _pyio is up (History for Lib/_pyio.py - python/cpython · GitHub).
  3. It is useful for teaching and understanding I/O in Python. Two core developers raised this during the sprint.

What I’d like to hear

  • Do you import _pyio anywhere? Production code, tests, tooling and teaching material all count. What do you use it for? If your project depends on it, I’m happy to work with you on a path to a maintainable state.
  • Do you contribute to _pyio? What draws you to it, and what would you want to see kept? _pyio could also be developed outside of CPython’s core, reducing barriers to entry.
  • Is there a reason for CPython to keep maintaining _pyio that isn’t listed above?
3 Likes

Searching the top 5k PyPI projects (downloaded 21 September):

Unguarded imports:

  • cheroot 11.1.2: cheroot/makefile.py#L4 does import _pyio as io at module level (“prefer slower Python-based io module”) and subclasses its BufferedWriter

  • eventlet 0.41.2: eventlet/greenio/py3.py#L1 does import _pyio as _original_pyio at module level, then at #L203-L210 rebuilds _pyio.open

Guarded imports:

Test only:

  • backports.zstd 1.7.0: a vendored copy of CPython’s test_zipfile does import _pyio
4 Likes

gut feeling: it’s an internal implementation detail, we should not worry about breaking people by removing _pyio. why? I’m convinced by those search results: those uses are not important.

1 Like