# PEP 756 – \[C API\] Add PyUnicode\_Export() and PyUnicode\_Import() C functions **URL:** \ **Category:** PEPs\ **Created:** [September 14, 2024, 9:10am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891 "2024-09-14T09:10:22Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 14, 2024, 9:10am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/1 "2024-09-14T09:10:22Z")
**Read the PEP:** [PEP 756 – Add PyUnicode\_Export() and PyUnicode\_Import() C functions | peps.python.org](https://peps.python.org/pep-0756/) The [Add PyUnicode\_Export() and PyUnicode\_Import() to the limited C API](https://github.com/capi-workgroup/decisions/issues/33) decision issue of the C API Working Group has now more than 50 comments which makes it difficult to read/navigate. I wrote PEP 756 to summarize the discussion. It might be easier to make a decision on a PEP. Obviously, it’s an opinionated PEP 🙂 For this first C API Working Group PEP, I used: `PEP-Delegate: C API Working Group`. ## Abstract Add functions to the limited C API version 3.14: - `PyUnicode_Export()`: export a Python str object as a `Py_buffer` view. - `PyUnicode_Import()`: import a Python str object. In general, `PyUnicode_Export()` has an _O_(1) complexity: no memory copy is needed. See the specification for cases when a copy is needed. ## Open Questions - Should we guarantee that the exported buffer always ends with a NUL character? Is it possible to implement it in _O_(1) complexity in all Python implementations? - Is it ok to allow surrogate characters? - Should we add a flag to disallow embedded NUL characters? It would have an _O_(_n_) complexity. - Should we add a flag to disallow surrogate characters? It would have an _O_(_n_) complexity. ---
**Author:** ![da-woods](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/da-woods/32/7507_2.png) [@da-woods](https://discuss.python.org/u/da-woods)\ **Post date:** [September 15, 2024, 7:46am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/2 "2024-09-15T07:46:36Z")
Thinking about where Cython would use these interfaces, it’s mostly just micro-optimizations. - For export, we’d probably prefer something that fails rather than something that makes a copy in most cases. But I can see that other users might feel differently. I guess that’s probably possible by requesting everything, then rejecting results in the wrong format, but long-term more formats might be added. - I’m not convinced it needs to go into the limited API immediately - I’m personally of the opinion that the Limited API should be as small as possible while still being useful, and this feels like an optimization rather than a necessity. The `flags` argument exists but isn’t discussed in the PEP. I assume it’s mostly for future use and is ignored for now? ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [September 15, 2024, 10:32am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/3 "2024-09-15T10:32:03Z")
This PEP looks very well thought out. I have a minor comment: instead of saying the buffer format for `PyUnicode_FORMAT_UCS4` is either `"I"` or `"L"`, why not always `"I"`? Now to your questions: > [@vstinner](#): > > - Should we guarantee that the exported buffer always ends with a NUL character? No. Any decent text-handling library, in any language, should accept explicitly-sized strings. > [@](#): > > - Is it ok to allow surrogate characters? I think so. > [@](#): > > - Should we add a flag to disallow embedded NUL characters? It would have an _O_(_n_) complexity. No. It’s trivial for callers to implement this if they want. > [@](#): > > - Should we add a flag to disallow surrogate characters? It would have an _O_(_n_) complexity. No. It’s almost trivial for callers to implement this if they want. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 15, 2024, 12:25pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/4 "2024-09-15T12:25:41Z")
> [@pitrou](#): > > I have a minor comment: instead of saying the buffer format for `PyUnicode_FORMAT_UCS4` is either `"I"` or `"L"`, why not always `"I"`? It should be `"I"` in the common case, but this format is related to the `unsigned int` type which can be 16-bit on some platforms. In this case, `"L"` is used instead. It’s more reliable to rely on the export format (ex: `PyUnicode_FORMAT_UCS4`) than than `Py_buffer.format` (ex: `"I"`). ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 15, 2024, 12:39pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/5 "2024-09-15T12:39:26Z")
> [@da-woods](#): > > - For export, we’d probably prefer something that fails rather than something that makes a copy in most cases. I can add a `PyUnicode_EXPORT_NO_COPY` flag for that. But can’t you just pass enough formats as recommended by the PEP to get O(1) complexity (avoid copy)? > [@da-woods](#): > > The `flags` argument exists but isn’t discussed in the PEP. I assume it’s mostly for future use and is ignored for now? Oops, I added it by mistake to the PEP, it’s not part of the API: I just remove it. (I was working on a patch to add flags for surrogates and embedded null characters. See Open Questions.) ---
**Author:** ![da-woods](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/da-woods/32/7507_2.png) [@da-woods](https://discuss.python.org/u/da-woods)\ **Post date:** [September 15, 2024, 1:08pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/6 "2024-09-15T13:08:01Z")
> [@vstinner](#): > > But can’t you just pass enough formats as recommended by the PEP to get O(1) complexity (avoid copy)? Yes, but if you add more supported formats in future then our O(1) code suddenly becomes O(n) with no change on our part. ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [September 15, 2024, 5:19pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/7 "2024-09-15T17:19:53Z")
> [@vstinner](#): > > It should be `"I"` in the common case, but this format is related to the `unsigned int` type which can be 16-bit on some platforms. Is CPython even supported on such platforms? I doubt it. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [September 16, 2024, 9:44am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/8 "2024-09-16T09:44:23Z")
> [@da-woods](#): > > if you add more supported formats in future then our O(1) code suddenly becomes O(n) with no change on our part. Generally, I think that - feature-wise, we should design API to not break in the future, but - performance-wise, we should design API for what we (and e.g. PyPy) have now. This means that if we change things in the future, extensions should continue working, but perhaps they’ll be (much) slower than before – until they update to use some new features or APIs. IMO, this is better than trying to predict the future _right_ now. * * * > [@vstinner](#): > > Should we guarantee that the exported buffer always ends with a NUL character? Is it possible to implement it in _O_(1) complexity in all Python implementations? IMO, we need to do this, but discourage relying on it. In practice, most of the returned buffers _are_ NUL-terminated. No matter what we say or document, some users _will_ expect the trailing NUL, and they won’t be bothered by embedded NULs truncating their strings. Unfortunately, using C string functions on NUL terminated can easily become a security issue. And we’re still building a C API. So, to make the world safe, we need to always the terminating NUL. We can add a “no NUL please” flag in the future. Or alternate implementations where this is a bottleneck can add a `XPyUnicode_Export_NoNUL` function (which CPython can adopt later). But right now, let’s export the NUL and - document that we do it, for the benefit of alternate implementations - document that you shouldn’t rely on it, and use _size_ whenever possible, since strings can have embedded NULs We don’t need to add the flag now – if/when we do, all already-released CPython versions will simply ignore the flag and add an extra NUL. > [@vstinner](#): > > Should we add a flag to disallow surrogate characters? It would have an _O_(_n_) complexity. No. If a user needs to reject surrogate characters, they can scan the string themselves. However, we do need to document that our exports can contain these. Again, if some day there appears an (alternate) implementation where this check is cheap, we/they can add a flag/function for it. (Already-released CPython versions will ignore the flag if it’s requested, and they won’t set it in the output.) > [@vstinner](#): > > Should we add a flag to disallow embedded NUL characters? It would have an _O_(_n_) complexity. No. Embedded NULs are a normal feature of Python strings. Again, users that need to reject these can scan the result. In many cases, they might want to reject some other control characters as well, not just NUL – for example terminal escapes, lone surrogates, or BIDI overrides. Let’s not build the tool now if CPython doesn’t need it. It would be an ill-fitting tool. * * * > [@pitrou](#): > > instead of saying the buffer format for `PyUnicode_FORMAT_UCS4` is either `"I"` or `"L"`, why not always `"I"`? I guess the proper format would be [`"=I"`](https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment). Some consumers might not expect the full `struct` syntax here, but since this is new API, perhaps that’s OK? If so we should also use `"=H"`. But I’d keep `"B"` alone; that’s unambiguous, and more likely to be special-cased. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 16, 2024, 12:10pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/9 "2024-09-16T12:10:38Z")
I asked PyPy devs about PEP 756: a memory copy will be needed anyway, because Python str objects can be moved in memory (it’s not possible to pin a `str` object in memory, or at least, it’s not recommended for best performance). Also, obviously, PyPy doesn’t use UCS-1, UCS-2 or UCS-4 internally and so requesting this format would either fail or have to copy memory. PyPy uses UTF-8 internally. IMO it’s ok that PyPy has to copy memory. It’s still better than the current PyUnicode\_AsUTF8() API which has an undefined lifetime. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 16, 2024, 1:13pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/10 "2024-09-16T13:13:32Z")
I wrote the [PR: PEP 756: Remove Open Questions, add flags](https://github.com/python/peps/pull/3968) to update the PEP: - ~~Add `uint32 flags` to `PyUnicode_Export()` and `PyUnicode_Import()` for future usage. It will allow to extend the API without having the break the ABI or need to add a new API. For now, _flags_ must be set to `0`.~~ - Add a _soft_ requirement on ending the buffer with a trailing NUL character: “The buffer **should** end with a trailing NUL character” (and not **must** ). - Remove Open Questions: they have been answered: - Soft requirement on ending the buffer with a trailing NUL character. - It’s ok to allow surrogate characters. - Don’t add a flag to reject embedded NUL characters. - Don’t add a flag to reject surrogate characters. - Update information about PyPy: mention the moving GC and the need to copy the string anyway. > [@da-woods](#): > > I’m not convinced it needs to go into the limited API immediately - I’m personally of the opinion that the Limited API should be as small as possible while still being useful, and this feels like an optimization rather than a necessity. The lack of this API prevents some C extensions to use the limited C API. I would like to promote limited C API usage and allow to write efficient code. > [@pitrou](#): > > This PEP looks very well thought out. I have a minor comment: instead of saying the buffer format for `PyUnicode_FORMAT_UCS4` is either `"I"` or `"L"`, why not always `"I"`? I updated the PEP, it now always use `"=I"` buffer format for the `PyUnicode_FORMAT_UCS4` export format. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [September 16, 2024, 4:22pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/11 "2024-09-16T16:22:38Z")
Just to transfer my most current concerns (as Victor said, we’ve been working on this API for a while already): The reasons that `PyUnicode_AsAnythingElse` aren’t sufficient right now are purely due to performance - where having O(1) access to the entire string\[1\] is more important than duplicating your processing code to handle different internal representations. Certainly for some cases I can see the value - a regex library is likely more efficient handling a million characters stored as UCS-4 than converting to UTF-8 and then processing that. But the value is _entirely_ in that zero-copy, zero-process “export”. So as soon as we add any algorithm over the string contents, even a copy, the value is gone. My opinion is that if we’re going to add high-performance APIs where we already have interoperable APIs, they should keep that guarantee or else fail and the caller can fall back. One day, our internal representation may change in a way to make this API no longer fast, at which point it _should_ fail dynamically, but because all callers have a fallback already in place they’ll continue working (and if they’re running tests that ensure the fallback is not used, they’ll find out that we changed it early). We shouldn’t even be considering starting with “high performance” APIs that break that guarantee. O(1) or nothing. Given that constraint, the only thing I’d change in the PEP as it stands is that we should only ever return the buffer if we already have the requested format. Enabling any kind of conversion or filtering here inevitably breaks the O(1) guarantee, which is the entire point of the function. If that makes it uninteresting to others, then we just shouldn’t add it. (`PyUnicode_Import` in my mind is for round-tripping data. Because we have to guarantee internal consistency of our own data structures, it’s very unlikely that we can make it O(1). Export is the important operation as far as perf is concerned, but if we add one then we should add both. I wouldn’t take `PyUnicode_Import` on its own.) As a side note, I would _love_ if we could find some consistency between this API and the `PyLong` export and import functions also being proposed. A general pattern for “give me a raw view/untranslated access to this PyObject” would help save us from having to keep designing these from scratch each time. * * * 1. Or more likely, O(1N) rather than O(2N) with a copy. ---
**Author:** ![methane](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/methane/32/43_2.png) [@methane](https://discuss.python.org/u/methane)\ **Post date:** [September 17, 2024, 8:05am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/12 "2024-09-17T08:05:43Z")
I don’t like this API to limited APIs. This API design is optimized for current CPython implementation. So it should be CPython public API, not limited API. For limited API and stable ABI, we can add `PyUnicode_ExportUTF8()`. FWI, There is a discussion about deprecating PEP 393 based APIs in Python 3.14 and use PyPy-like representation in the future. So I am not sure even public (not limited) API is needed. > > > I'm sure this has been discussed elsewhere and I don't know when, or if, we'll h…ave time to implement it, but I think it's worth adding here. > > Currently Python \`str\`s (\`PyUnicodeObject\` in C) are implemented as arrays of 1, 2 or 4 byte unicode code points, plus a header > I propose that we implement them as a \[utf8\](https://en.wikipedia.org/wiki/UTF-8) encoded array of bytes, plus a header. > > The advantages are numerous: > \* Non-ASCII strings will generally be more compact, without making ASCII strings any bigger. > \* Strings can be joined easily by simple concatenation of the data > \* The internet is utf8, so the vast majority of encoding and decoding operations should be fast > \* There need only be one implmentation of each \`str\` method and each \`PyUnicode\_\` C function, saving considerable code size and simplifying the code > \* Algorithms for fast operations on utf8 are well known, so many operations will fast, despite the variable length encoding. > \* The C struct is clearer, as we don't need an awkward union of \`uint8\_t\`, \`uint16\_t\` and \`uint32\_t\` for character data. > > However there are two problems: > \* Indexing into strings is supposed to be O(1). > \* Some of the C API exposes the internal encoding > > \### Keeping indexing O(1) > > To maintain this properly we will have to lazily create an offset table for larger, non-ASCII strings. > This is won't be a problem in practice because: > \* Creating the index is no more expensive than creating the current UCS1/2/4 strings. > \* We only allocate indexes if we need them, which should be relatively rare. > > \### The C API > > We will have to deprecate, and then remove, the C API that exposes the implementation details. > We should probably deprecate for 3.14, so that we can implement utf8 strings in 3.16, allowing a proper deprecation period. > > \## Implementation > > We will probably want to embed the string data directly into the object, so the struct will look something like this: > \`\`\`C > typedef struct { > PyObject\_HEAD > uintptr\_t interned: 2; > uintptr\_t ascii: 1; > uintptr\_t valid\_utf8: 1; > uintptr\_t length: (WORD\_SIZE-4); /\* Number of code points in the string \*/ > Py\_hash\_t hash; /\* Hash value; -1 if not set \*/ > PyUnicodeIndex \*index; /\* NULL unless needed \*/ > uint8\_t data\[1\]; > } PyUnicodeObject; > \`\`\` > The \`valid\_utf8\` bit helps fast encoding. It is false if the string contains half surrogate pairs or any other code point not allowed in legal utf-8. > > \### Indexing operations > > \`setitem(self, index)\` would be implemented something like this: > > \`\`\`Py > def getitem(self, index): > if self.ascii: > return self.data\[index\] > if self.index is NULL: > self.index = make\_index(self) > offset = offset\_from\_index(self.index, index) > return read\_one\_char(self.data, offset) > \`\`\` > > The index table would be composed of len(s)/64 entries, each entry being: > \`\`\`C > struct index\_entry { > uintptr\_t base\_offset; > uint8\_t additional\_offset\[64\]; > }; > \`\`\` > With the offset being computed as \`base\_offset\[index/64\] + additional\_offset\[index%64\]\`. 7 years ago, I ported MarkupSafe speedup module from Python 2 to PEP 393. My motivation was making Python 3 as fast as Python 2 to motivate Flask users to go Python 3. > > > markupsafe doesn't support Python 3.2 anymore. > So just use new API at Python 3. Now I hate this code. I needed to use long C macro for UCS-1/2/4 template that is really maintenance burdon. Many HTML snippets are ASCII. Using UTF-8 API doesn’t slow down them. And if we change the str internal representation to UTF-8, escaping it will be much faster than escaping current UCS-4 too. So I hope future CPython has PyPy-like internal representation. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [September 17, 2024, 3:08pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/13 "2024-09-17T15:08:49Z")
> [@methane](#): > > I don’t like this API to limited APIs. This API design is optimized for current CPython implementation. It’s actually been designed specifically for the limited API, and to be stable even if we change internal layout. We’ve spent quite a while working on that on the earlier GitHub issues. A non-limited API would be much closer to the macros we currently have. Of course, you’re welcome to dislike the API still. I don’t particularly like it either, but with the “must be O(1)” and “must be limited API” constraints, it’s probably the best we can do. (You need to drop the O(1) constraint to always use UTF-8.) ---
**Author:** ![methane](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/methane/32/43_2.png) [@methane](https://discuss.python.org/u/methane)\ **Post date:** [September 17, 2024, 5:09pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/14 "2024-09-17T17:09:53Z")
Why O(1) limited API is must have? The PEP uses MarkupSafe for example why performance is needed. But MarkupSafe cannot be O(1) anyway. And I, the author of the MarkupSafe speedup think UTF-8 is acceptable performance and much easier to maintain. Additionally, O(1) is impossible with PyPy. Why the O(1) is required is not explained neither in the PEP nor this thread. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [September 17, 2024, 5:52pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/15 "2024-09-17T17:52:07Z")
> [@methane](#): > > Why the O(1) is required is not explained neither in the PEP nor this thread. It’s because we already have O(N) functions (which convert to UTF-8 or wchar\_t). Therefore, we don’t need another function that has the same characteristic. Ensuring it is O(1) is the only value we can add with a new function, otherwise, we shouldn’t add it at all. O(1) is possible with PyPy because it can refuse to return any kind but UTF-8. The PEP should change to say that the function returns failure if a suitable format is not requested, and should never do conversions. I already suggested this earlier - without it, the function provides no additional value. ---
**Author:** ![methane](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/methane/32/43_2.png) [@methane](https://discuss.python.org/u/methane)\ **Post date:** [September 18, 2024, 1:52am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/16 "2024-09-18T01:52:35Z")
Please read the @vstinner 's [this comment](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/9). O(1) is not possible with PyPy and new API is better than current API because of limited lifetime. If limited lifetime is not required, I agree that we don’t need new limited API. ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [September 18, 2024, 8:09am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/17 "2024-09-18T08:09:32Z")
> [@methane](#): > > O(1) is not possible with PyPy Is it not? AFAIU, PyPy supports `memoryview` zero-copy views, so it would at least be theoretically possible to also support zero-copy views of `str` objects? @cfbolz ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [September 18, 2024, 1:18pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/18 "2024-09-18T13:18:48Z")
> [@methane](#): > > Please read the @vstinner 's [this comment](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/9). > > O(1) is not possible with PyPy and new API is better than current API because of limited lifetime. Ah right, they can’t just pin the memory when the view is requested. (I never learned the intricacies of PyPy’s GC, so I typically just assume they’re similar to .NET’s GC.) In any case, it means O(1) is _never_ possible with PyPy, and that’s a tradeoff of the entire implementation. At least with this API it can be a straight memcpy, whereas if we implement the scanning and conversions being proposed up the thread then _every_ implementation has to become more complicated and more complex. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 23, 2024, 11:59am UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/19 "2024-09-23T11:59:46Z")
> [@steve.dower](#): > > It’s because we already have O(N) functions (which convert to UTF-8 or wchar\_t). Therefore, we don’t need another function that has the same characteristic. Ensuring it is O(1) is the only value we can add with a new function, otherwise, we shouldn’t add it at all. When you ask **only** for UCS-4 and data is stored as UCS-1 or UCS-2, I prefer to convert to UCS-4. IMO it makes the API more convenient to use. Anyway, we cannot guarantee O(1) in all code paths on all Python implementations. As said before, in the future, CPython [might change to store strings as UTF-8](https://github.com/faster-cpython/ideas/issues/684). Moreover, on PyPy, it seems like O(1) is not possible, only O(n). ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [September 23, 2024, 12:45pm UTC](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891/20 "2024-09-23T12:45:55Z")
> [@vstinner](#): > > When you ask **only** for UCS-4 and data is stored as UCS-1 or UCS-2, I prefer to convert to UCS-4. IMO it makes the API more convenient to use. Yes I know, and this is what I’m opposing. Clearly it will be up to others to decide one way or the other. I don’t particularly want to maintain a function that can do every permutation of conversion. And I don’t want to add a function where the _only_ benefit over the existing ones is that it uses arguments to decide the format rather than separate functions. The only thing people have asked for here is O(1) access to the contents of a string. If we can’t provide that, we should just apologise and move on. * * * I’ll also note that PyPy’s inability to pin memory is completely unrelated to Unicode. They (apparently) can’t provide O(1) access to _any_ internal structure through a pointer, because it’s how they work (as you’ve described it). Just because they don’t use a fixed location memory buffer for every string object doesn’t give us permission to convert formats and still say it’s fast enough. [Next page](https://discuss.python.org/t/pep-756-c-api-add-pyunicode-export-and-pyunicode-import-c-functions/63891.md?page=2)