# PEP 782: Add PyBytesWriter C API **URL:** \ **Category:** PEPs\ **Created:** [March 31, 2025, 12:28pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617 "2025-03-31T12:28:43Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [March 31, 2025, 12:28pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/1 "2025-03-31T12:28:43Z")
Hi, After [multiple iterations on the API](https://discuss.python.org/t/81182), I decided to write down a PEP for the PyBytesWriter C API. It’s easier to understand the API with its documentation, examples, and discussions around it. A single document (the PEP) should help the discussion. → Read **[PEP 782: Add PyBytesWriter C API](https://peps.python.org/pep-0782/)** ← The API is now based on sizes rather than pointers, even if two functions using pointers are provided for convenience. ## Abstract Add a new `PyBytesWriter` C API to create `bytes` objects. Soft deprecate `PyBytes_FromStringAndSize(NULL, size)` and `_PyBytes_Resize()` APIs. These APIs treat an immutable `bytes` object as a mutable object. They remain available and maintained, don’t emit deprecation warning, but are no longer recommended when writing new code. ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [March 31, 2025, 1:27pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/2 "2025-03-31T13:27:13Z")
> [@vstinner](#): > > They remain available and maintained, don’t emit deprecation warning, but are no longer recommended when writing new code. Perhaps we should at least add a compile-time deprecation warning for `_PyBytes_Resize`? I understand that it’s not possible to do it for `PyBytes_FromStringAndSize(NULL, size)` while still allowing `PyBytes_FromStringAndSize(, size)`. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [March 31, 2025, 3:49pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/3 "2025-03-31T15:49:27Z")
From my reading and assumptions, I guess `PyBytesWriter_WriteBytes` and `PyBytesWriter_Format` will extend the buffer? It should say so explicitly in their description, including how it relates to a prior overallocation (e.g. if I `PyBytesWriter_Create(10)` and then `PyBytesWriter_WriteBytes(<10 bytes>)`, does it use the 10 I specified or does it allocate 10 more?). If I call `PyBytesWriter_WriteBytes` multiple times, does it append or overwrite? Does `PyBytesWriter_GetData` get the start of the data or the position where `WriteBytes` would next write to? (Obviously irrelevant if it overwrites from the start each time.) Why disallow shrinking with `PyBytesWriter_Grow`? It can handle negative growth just fine (potentially with data loss, but that’s likely intentional). I have plenty of cases where I would `PyBytesWriter_Grow(-1)` to trim a trailing null or character. What does “update the buf pointer” mean for `PyBytesWriter_GrowAndUpdatePointer`? Maybe it needs a realistic example, because it seems like `Grow` is going to allocate garbage/zeros and `UpdatePointer` is going to move my pointer past it so that I don’t write values into the new part of the allocation? Doesn’t seem useful. * * * Got up to the next example, and I see it would make sense to pass in the result of a `strlen` previously used to `strcpy` into the result of `GetData`? Is that the intent? Still not entirely clear how or when I’d use this API. * * * The “Overallocation” section should mention that (whether?) finishing the writer will trim overallocations. > The implementation allocates internally a `bytes` object … Maybe call out that this isn’t a required part of the design? Other implementations may do it differently, and CPython may do it differently in the future. > There is no impact on the backward compatibility I don’t think you can deprecate (even soft deprecate) functions and say there’s no impact 😉 A short example of how to change code using `PyBytes_FromStringAndSize(NULL, size)` into code that will compile with both the old API and the new API would be useful here - checking `PY_VERSION_HEX` is probably the best option here? ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [March 31, 2025, 4:13pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/4 "2025-03-31T16:13:13Z")
Hi Steve, > [@steve.dower](#): > > From my reading and assumptions, I guess `PyBytesWriter_WriteBytes` and `PyBytesWriter_Format` will extend the buffer? It should say so explicitly in their description, including how it relates to a prior overallocation (e.g. if I `PyBytesWriter_Create(10)` and then `PyBytesWriter_WriteBytes(<10 bytes>)`, does it use the 10 I specified or does it allocate 10 more?). The key here is `PyBytesWriter_GetSize` which returns the current size of the writer. `PyBytesWriter_WriteBytes` and `PyBytesWriter_Format` increase this size: they call `Resize()` internally to increase the size. `PyWriterWriter_Create(n)` sets the writer size to _n_. I will try to clarify that in the PEP. > [@steve.dower](#): > > If I call `PyBytesWriter_WriteBytes` multiple times, does it append or overwrite? It does append. The [high-level API example](https://peps.python.org/pep-0782/#examples) is based on that. > [@steve.dower](#): > > Does `PyBytesWriter_GetData` get the start of the data or the position where `WriteBytes` would next write to? (Obviously irrelevant if it overwrites from the start each time.) It’s always the start of the data. It’s similar to `PyBytes_AS_STRING()` and `PyByteArray_AS_STRING()`. > [@steve.dower](#): > > Why disallow shrinking with `PyBytesWriter_Grow`? It can handle negative growth just fine (potentially with data loss, but that’s likely intentional). I have plenty of cases where I would `PyBytesWriter_Grow(-1)` to trim a trailing null or character. Ah. It’s just an arbitrary limit to help detecting bugs in the code. I’m open to remove this limitation. `PyBytesWriter_Grow()` is just a convenient helper around `PyBytesWriter_Resize()` and `PyBytesWriter_GetSize()`. > [@steve.dower](#): > > What does “update the buf pointer” mean for `PyBytesWriter_GrowAndUpdatePointer`? Maybe it needs a realistic example, because it seems like `Grow` is going to allocate garbage/zeros and `UpdatePointer` is going to move my pointer past it so that I don’t write values into the new part of the allocation? Doesn’t seem useful. Ah, I added pseudo-code in the hope that it would be enough to explain the behavior. The `UpdatePointer` part only means that if the internal buffer is moved in memory, the pointer is updated to point to the new memory address. It’s only a helper function. The position inside the writer buffer is unchanged. > [@steve.dower](#): > > Got up to the next example, and I see it would make sense to pass in the result of a `strlen` previously used to `strcpy` into the result of `GetData`? Is that the intent? Still not entirely clear how or when I’d use this API. I’m sorry, I’m confused, which example are you referring to? > [@steve.dower](#): > > > The implementation allocates internally a `bytes` object … > > Maybe call out that this isn’t a required part of the design? Other implementations may do it differently, and CPython may do it differently in the future. I only described the “reference implementation”, it’s not part of the Specification. I will try to clarify that. > [@steve.dower](#): > > > There is no impact on the backward compatibility > > I don’t think you can deprecate (even soft deprecate) functions and say there’s no impact 😉 There are only soft deprecations. Existing code will continue to work as if nothing happened. No warnings is emitted. But I can repeat the soft deprecations there if you want. > [@steve.dower](#): > > A short example of how to change code using `PyBytes_FromStringAndSize(NULL, size)` into code that will compile with both the old API and the new API would be useful here - checking `PY_VERSION_HEX` is probably the best option here? Ok, I will add such example. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [March 31, 2025, 4:18pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/5 "2025-03-31T16:18:09Z")
Thanks, those sound like the changes I wanted to see in the PEP text. > [@vstinner](#): > > > [@steve.dower](#): > > > > Got up to the next example, and I see it would make sense to pass in the result of a `strlen` previously used to `strcpy` into the result of `GetData`? Is that the intent? Still not entirely clear how or when I’d use this API. > > I’m sorry, I’m confused, which example are you referring to? Should’ve linked, sorry. I meant the [GrowAndUpdatePointer example](https://peps.python.org/pep-0782/#growandupdatepointer-example). > [@vstinner](#): > > I only described the “reference implementation”, it’s not part of the Specification. Yeah, but I’m sure people will misquote it later on (I’ve just come from an argument where someone was doing it…). Better to be clear now and not have to have arguments later. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [March 31, 2025, 6:41pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/6 "2025-03-31T18:41:25Z")
> [@steve.dower](#): > > A short example of how to change code using `PyBytes_FromStringAndSize(NULL, size)` into code that will compile with both the old API and the new API would be useful here - checking `PY_VERSION_HEX` is probably the best option here? I plan to backport PEP 782 (PyBytesWriter) implementation to Python 3.13 in [pythoncapi-compat](https://pythoncapi-compat.readthedocs.io/). I [wrote a draft PR](https://github.com/python/pythoncapi-compat/pull/139) to show that it’s doable. The backport might be less efficient (e.g. cannot use a free list), but it allows to have a single code base working on all Python versions (my backport works on Python 2.7 and PyPy 2/3). ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [March 31, 2025, 7:05pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/7 "2025-03-31T19:05:50Z")
You need to reference `pythoncapi-compat` in the PEP then, and it wouldn’t surprise me if a PEP delegate preferred you only used official/core examples - things in the past that have tried to refer to 3rd party projects have been contentious. ---
**Author:** ![cmaloney](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cmaloney/32/15851_2.png) [@cmaloney](https://discuss.python.org/u/cmaloney)\ **Post date:** [March 31, 2025, 8:28pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/8 "2025-03-31T20:28:43Z")
Overall looks good to me, two nitpick API addition ideas from me that definitely could be deferred: 1. A way to get a “Buffer Protocol” view onto the `PyBytesWriter` without having to “finalize” it into a `bytes` first. The use case I’m thinking of is to build up a block of _machine bytes_, and then introspect/debug/validate the bytes state in Python code or present to a user (ex. a progress bar) the “partial” built bytes could then be further modified. Some tradeoff in added complexity as I suspect would want to “lock” the buffer from being resizable though when there are any exports. 2. `PyBytesWriter_FromBytes(PyObject *)` where a pre-existing bytes object is passed in. I’m thinking about cases where an API takes a `bytes` in its constructor (or a `str` + `encoding=`). It simplifies boiler plate in those cases, enables removing an otherwise unavoidable copy of the _machine bytes_ from the initial buffer value. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [March 31, 2025, 9:03pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/9 "2025-03-31T21:03:21Z")
> [@cmaloney](#): > > then introspect/debug/validate the bytes state in Python code or present to a user The whole point of this API is to have the data in a state that isn’t valid or suitable for exposing to Python code. Particularly once we start worrying about threading, it’s nice to have an API that deliberately doesn’t support access from multiple threads (so nothing goes through locks or mutexes, unless the caller does it, which we usually won’t have to). If you want to work with partially initialised buffers, just use `bytearray`. > [@cmaloney](#): > > `PyBytesWriter_FromBytes(PyObject *)` where a pre-existing bytes object is passed in. I’m thinking about cases where an API takes a `bytes` in its constructor Again, this API is intended for constructing a `bytes` object from other _native_ data. The copy is indeed unavoidable, because the original `bytes` object has to remain immutable, so it only saves a `PyBytes_AsString` call and then `PyBytesWriter_WriteBytes` of that string. The functions that would be involved in converting `str`+`encoding` into `bytes` is likely going to be using a bytes writer internally. So proposing to expose the partially initialised but un-finished writer from those falls under the first point. ---
**Author:** ![cmaloney](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cmaloney/32/15851_2.png) [@cmaloney](https://discuss.python.org/u/cmaloney)\ **Post date:** [March 31, 2025, 9:14pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/10 "2025-03-31T21:14:04Z")
I appreciate the thoughts and comments. I’m trying to build myself a picture of `BytesIO`, `bytes`, `bytearray` and now `PyBytesWriter` what they do the same, what they do differently, and when should use each. There is also `_BlocksOutputBuffer` but that is more specifically isolated. They have a lot of similarities and similarly structured code. I personally would like to reduce the amount of code and “when do I use which” / tradeoffs. Definitely understand this is a new and somewhat distinct use case. Hoping to find ways it can make the existing ones simpler as well (even just in implementation). Thanks for the thoughts. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [April 1, 2025, 9:35am UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/11 "2025-04-01T09:35:35Z")
> [@pitrou](#): > > Perhaps we should at least add a compile-time deprecation warning for `_PyBytes_Resize`? That would break projects treating compiler warnings as errors. In PyPI top 8,000 projects, there are 41 projects (C extensions) using `_PyBytes_Resize()`, it’s significant. I would prefer to have at least one Python version with PEP 782 API (ex: in Python 3.14) before considering to deprecate `_PyBytes_Resize()` (ex: in Python 3.15). Even if I really dislike `_PyBytes_Resize()` 😃 ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [April 1, 2025, 10:03am UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/12 "2025-04-01T10:03:53Z")
> [@cmaloney](#): > > 1. A way to get a “Buffer Protocol” view onto the `PyBytesWriter` without having to “finalize” it into a `bytes` first. It would make the implementation more complicated, I would prefer to not add it for now. Instead, functions should be specialized to use directly the `PyBytesWriter` C API (for now). > [@cmaloney](#): > > 1. `PyBytesWriter_FromBytes(PyObject *)` where a pre-existing bytes object is passed in. I’m thinking about cases where an API takes a `bytes` in its constructor (or a `str` + `encoding=`). It simplifies boiler plate in those cases, enables removing an otherwise unavoidable copy of the _machine bytes_ from the initial buffer value. The `PyUnicodeWriter` API has such optimization to store a read-only Unicode object which is copied on the next write. It allows some micro-optimizations on specific cases such as `"%s" % "abc"` or `"{}".format("abc")`. It makes the implementation more complicated. I don’t think that it’s worth it for `PyBytesWriter` API since I didn’t find any function in the Python code base which would benefit from such `PyBytesWriter_FromBytes(PyObject *)` function. ---
**Author:** ![cmaloney](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cmaloney/32/15851_2.png) [@cmaloney](https://discuss.python.org/u/cmaloney)\ **Post date:** [September 11, 2025, 5:37am UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/13 "2025-09-11T05:37:23Z")
Any updates on the PEP status? Changing the layout of `bytearray` to provide zero-copy `.take_bytes()` (or similar) to go from `bytearray` → `bytes` came up again today for me and this is currently a blocker for that. There’s also two pieces of code I’m working on in CPython which create a `PyBytes` and then export its contents as a memoryview (which then gets written into from inside the interpreter). Older code, but would be nice to have a clean / supported API to move to. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 11, 2025, 10:54am UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/14 "2025-09-11T10:54:26Z")
> [@cmaloney](#): > > Any updates on the PEP status? I submitted the PEP to the Steering Council in May: [PEP 782 -- Add PyBytesWriter C API · Issue #290 · python/steering-council · GitHub](https://github.com/python/steering-council/issues/290). The SC should take a decision soon. ---
**Author:** ![corona10](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/corona10/32/9943_2.png) [@corona10](https://discuss.python.org/u/corona10)\ **Post date:** [September 11, 2025, 4:50pm UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/15 "2025-09-11T16:50:16Z")
Hi Victor, On behalf of the Steering Council, I am pleased to inform you that we approve [PEP 782](https://peps.python.org/pep-0782/). Congratulations! We also have some minor requests we would like to see once this is implemented: - Thread-safety documentation: Please ensure that API documentation explicitly states whether each API is thread-safe. This follows [the consensus](https://discuss.python.org/t/pep-779-criteria-for-supported-status-for-free-threaded-python/84319/123) established after [PEP 779](https://peps.python.org/pep-0779/). - Discard/Finish API: As some APIs cannot be called after the Discard/Finish API is invoked, to avoid user confusion, please clearly document what APIs can be called after calling either the Discard or Finish APIs. We appreciate your patience and thank you again for your effort! Warm regards, Donghee (on behalf of the Python Steering Council) ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 12, 2025, 11:32am UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/16 "2025-09-12T11:32:14Z")
> [@corona10](#): > > On behalf of the Steering Council, I am pleased to inform you that we approve [PEP 782](https://peps.python.org/pep-0782/). Congratulations! Nice! Thank you. > [@corona10](#): > > We also have some minor requests we would like to see once this is implemented: Ok, I added some sentences to clarify these points, see [gh-129813, PEP 782: Add PyBytesWriter C API by vstinner · Pull Request #138822 · python/cpython · GitHub](https://github.com/python/cpython/pull/138822). ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 25, 2025, 10:27am UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/17 "2025-09-25T10:27:34Z")
I implemented PEP 782 `PyBytesWriter` API in Python 3.15 and I modified many functions to use it. I marked the PEP as Final. The documentation can be read at: [Bytes Objects — Python 3.15.0a0 documentation](https://docs.python.org/dev/c-api/bytes.html#pybyteswriter). ---
**Author:** ![rhpvorderman](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rhpvorderman/32/8438_2.png) [@rhpvorderman](https://discuss.python.org/u/rhpvorderman)\ **Post date:** [July 14, 2026, 8:01am UTC](https://discuss.python.org/t/pep-782-add-pybyteswriter-c-api/86617/18 "2026-07-14T08:01:30Z")
Hi, I maintain the python-isal and python-zlib-ng libraries which are basically reimplementations of Python’s zlib module using faster C libraries to back them up. Being compression APIs they do a lot of buffer growing and finalizing. The fastest way before the PyBytesWriter API was using `_PyBytes_resize` unfortunately that is not part of the stable ABI (understandably so). I was hoping that the `PyBytesWriter` API would be part of the stable ABI, but alas. Are ther plans to move the PyBytesWriter API into the stable ABI? Or are there simple technical limitations here given that it is so tied to the in-memory representation of the bytes object? (Apologies if this is the wrong place to ask, there are quite a lot of issues, PRs and discussion tackling this API)