Encode `str` (utf-8 uncodeobject) without a copy

[cmaloney (Cody Maloney)]

@vstinner

It’s not easy to judge proposed options without a concrete implementation. Can you show that using a Py_buffer in C or memoryview in Python is faster than creating a bytes object at each call? The benchmark likely depends on the string length. For example, is it worth it for strings shorter than 100 characters?

Planning to gather a CPU profile + runtime traces / benchmarks soon; will update this thread when I have.

This is the only common copy in the Text I/O stack I don’t currently have a way to resolve (caveats apply / there are some non-copy fast paths today). With Reworking "Buffered I/O" in CPython the copy of the .write argument into the shared buffer in BufferedReader will go away unless it’s required (the buffer is mutable/non-frozen and need to return from write while not flushing immediately). Inside TextIOWrapper there’s the encoding copy (this) and a second copy to merge the encoded strings into one final buffer which is then passed to BufferedIO.write. I’m hoping with the rework I can eliminate most that / just defer to BufferedIO which internally I plan to have keep a list of parts like TextIOWrapper does internally for efficiency.

All of that hopefully will also mean eliminating locking in many cases; although that depends in part on if I have Rust for making it an experimental I/O stack (_rustio ). My hope is to have a full working _pyio prototype which demonstrates the ideas + efficiency before end of February. Starting this discussion so hopefully can make it a zero-copy I/O stack in the 3.15 ship window :).

@vstinner

Better API for encoding unicode objects with UTF-8 proposed:

@vstinner

You may have a look at withdrawn PEP 756 – Add PyUnicode_Export() and PyUnicode_Import() C functions which proposed adding a PyUnicode_Export() function. The PEP discuss tricky issues about surrogate characters and embedded null characters.

@methane

The best solution in the UTF-8 era is for Unicode objects to use UTF-8 as their internal representation and only cache PEP 393 data when requested for compatibility.

Really appreciate these references. Working on reading through them / ingesting what they say. I wasn’t aware that if we start with utf-8 (ex. parsed Python source code or .decode('utf-8')) that CPython always currently copies that out of UTF-8 form and needs to then copy back into it. Resolving that + this should make CPython file → text → file round trip dramatically lower overhead.

That parsing utf-8 doesn’t keep the original bytes if we know they are 1. immutable / owned by python internals already and 2. correctly normalized/encoded is something I’d like to incorporate into shipping this change set potentially. I Think it would get Python a really nice place for UTF-8 processing.