# Deprecating the direct use of str internals, e.g. PyASCIIObject, PyCompactUnicodeObject, PyUnicodeObject structs
**URL:** https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543
**Category:** Ideas
**Tags:** c-api
**Created:** [February 28, 2025, 5:31pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543 "2025-02-28T17:31:07Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [February 28, 2025, 5:31pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/1 "2025-02-28T17:31:07Z")
Hello,
In the C API, I’d like to deprecate direct use of `str` internals – specifically:
- `PyASCIIObject`, `PyCompactUnicodeObject`, `PyUnicodeObject` structs
- the `PyUnicode_IS_COMPACT` macro
See a [preview of porting notes](https://cpython-previews--130697.org.readthedocs.build/en/130697/c-api/unicode.html#c.PyASCIIObject).
Allowing direct access to the structs is starting to block development of new features/optimizations:
- The free-threaded build needs to change the memory layout to allow atomic access to the `interned` field (which can be done in an API-compatible way), and it needs to _ensure_ that the access is atomic (which a `struct` field can’t do)
- There’s been some talk of string objects where UTF-8 would be the primary representation (and PEP 393 KIND/DATA filled on demand); such changes can’t really be done with the current structs.
Does that sound reasonable?
Is there any use case that would need a new function?
---
### Author:  [@kumaraditya303](https://discuss.python.org/u/kumaraditya303)
#### Post date: [February 28, 2025, 6:16pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/2 "2025-02-28T18:16:19Z")
> [@encukou](#):
>
> The free-threaded build needs to change the memory layout to allow atomic access to the `interned` field (which can be done in an API-compatible way), and it needs to _ensure_ that the access is atomic (which a `struct` field can’t do)
I think you meant bitfield, as struct fields can be accessed atomically and those accesses can be wrapped in function calls.
---
### Author:  [@ngoldbaum](https://discuss.python.org/u/ngoldbaum)
#### Post date: [February 28, 2025, 6:22pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/3 "2025-02-28T18:22:16Z")
NumPy uses `PyUnicodeObject` to define the scalar type for the `np.str_` dtype, in C:
```python
>>> isinstance(np.str_(1), str)
True
```
See [numpy/numpy/\_core/include/numpy/arrayscalars.h at da268d45aab791023c8d953db6f4597019f770cb · numpy/numpy · GitHub](https://github.com/numpy/numpy/blob/da268d45aab791023c8d953db6f4597019f770cb/numpy/_core/include/numpy/arrayscalars.h#L146)
It’s already excluded from the public C API for limited API builds, but NumPy uses `PyUnicodeScalarObject` internally. I don’t know offhand if we can avoid using `PyUnicodeObject` here, suggestions are very welcome.
It looks like we still have an old vendored version of `PyUnicode_FromUCS4` in a file named `ucsnarrow.c` (that tells you how old this is) that explicitly returns `PyUncidoeObject`, but I think we can replace that with `PyUnicode_FromKindAndData` these days.
---
### Author:  [@ngoldbaum](https://discuss.python.org/u/ngoldbaum)
#### Post date: [February 28, 2025, 6:46pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/4 "2025-02-28T18:46:59Z")
> [@ngoldbaum](#):
>
> It looks like we still have an old vendored version of `PyUnicode_FromUCS4` in a file named `ucsnarrow.c` (that tells you how old this is) that explicitly returns `PyUncidoeObject`, but I think we can replace that with `PyUnicode_FromKindAndData` these days.
See: [MAINT: remove legacy ucsnarrow module by ngoldbaum · Pull Request #28404 · numpy/numpy · GitHub](https://github.com/numpy/numpy/pull/28404)
---
### Author:  [@ronaldoussoren](https://discuss.python.org/u/ronaldoussoren)
#### Post date: [February 28, 2025, 7:12pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/5 "2025-02-28T19:12:47Z")
PyObjC uses `PyUnicodeObject` to create a subclass in C with an additional C field in the subclass. That’s not really possible without using the `PyUnicodeObject` and setting fields in it to initialize a value.
BTW. I don’t mind adjusting to changes to the internals of strings.
---
### Author:  [@da-woods](https://discuss.python.org/u/da-woods)
#### Post date: [March 1, 2025, 8:47am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/6 "2025-03-01T08:47:10Z")
Cython uses a few of these, but has fallback `#ifdef`s that avoid them in all cases (so it wouldn’t break anything in a way that’s hard to fix):
- `PyCompactUnicodeObject` for fast access to a length field (but only in versions \<3.12 anyway)
- `PyASCIIObject` to look up the `->hash` attribute either for interned names (where we know it’ll be set) or as an optimization.
The latter is something that might be nice to keep in some form - just fast access to the cached hash without forcing it to be calculated if not yet cached.
---
### Author:  [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [March 3, 2025, 2:58pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/7 "2025-03-03T14:58:04Z")
The NumPy and ObjC cases are the same: adding an extra field in a subclass.
Looks like the first breaking change we should do is to add a supported way of doing that.
> I don’t mind adjusting to changes to the internals of strings.
Looks like I should add aliases like `PyUnstable_ASCIIObject`, so you can avoid build warnings. That would be a formal promise that we won’t change the API in a bugfix/security release, but we _can_ change it in 3.x.0 and y’all should we prepared to adapt.
(My current draft implementation uses fully-private aliases like `_PyASCIIObject`, to allow the macros to reach into the structs; I’ll just rename those and document them.)
> fast access to a length field
`PyUnicode_GET_LENGTH()` should have you covered. (It’ll keep working. There’s no promise that it’ll remain the fastest way to get the length, but perf issues should be easier to adapt to.)
> `PyASCIIObject` to look up the `->hash` attribute either for interned names (where we know it’ll be set) or as an optimization.
So, do you want one or both of these?
- `PyUnicode_GET_HASH` with an inlined fast path, only doing a function call when it sees -1?
- `PyUnstable_Unicode_GET_CACHED_HASH`? (The unstable behaviour being not the function itself, but relying on which operations set the hash.)
---
### Author:  [@da-woods](https://discuss.python.org/u/da-woods)
#### Post date: [March 5, 2025, 8:32am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/8 "2025-03-05T08:32:09Z")
> [@encukou](#):
>
> `PyUnstable_Unicode_GET_CACHED_HASH`? (The unstable behaviour being not the function itself, but relying on which operations set the hash.)
This one matches what we currently do.
---
### Author:  [@malemburg](https://discuss.python.org/u/malemburg)
#### Post date: [March 20, 2025, 6:14pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/9 "2025-03-20T18:14:37Z")
How would people create `str` subtypes in C without access to `PyUnicodeObject` ?
---
### Author:  [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [March 21, 2025, 8:36am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/10 "2025-03-21T08:36:10Z")
(I assume you’re asking about suptypes with extra C data; creating a subclass with just ` __dict__ ` is straightforward.)
They can’t.
Today, they need `PyUnicodeObject` _and_ they also need to rewrite most of `PyUnicode_New`.
Users of `PyUnicodeObject` also don’t get any warning when CPython adds a new complication (like for example: in the free-threading build, access to `PyASCIIObject->interned` needs to be atomic).
That’s why I’m proposing `PyUnstable_UnicodeObject` for this use case. I hope to get people that currently need `PyUnicodeObject` to remove it if they can, and share more of their use cases if they can’t.
---
### Author:  [@malemburg](https://discuss.python.org/u/malemburg)
#### Post date: [March 21, 2025, 9:04am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/11 "2025-03-21T09:04:51Z")
> [@encukou](#):
>
> They can’t.
The current approach requires access to the type object you want to subclass.
And this is a documented feature of CPython: [2. Defining Extension Types: Tutorial — Python 3.13.2 documentation](https://docs.python.org/3/extending/newtypes_tutorial.html#subclassing-other-types)
We can’t just go about removing this feature for arbitrary types without first offering a stable working new method of doing the same.
What do you propose as an alternative method ?
---
### Author:  [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [March 21, 2025, 9:35am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/12 "2025-03-21T09:35:14Z")
Subclassing as documented in the tutorial currently _does not_ work with strings, since `PyUnicode_New` assumes that there is no extra state, and will allocate string data at the same location as your extension data. (It will also pick one of the three instance structs for `str`.)
To make this work, you need to write a modified version of `str` allocation, making sure you don’t miss anything (including new fields/invariants that CPython can add in new versions).
But, if you want a no-underscored-API version, keep your `PyUnicode_New` replacement, and add something like this (untested):
```c
typedef struct {
int state;
bool flag;
} MyExtraData;
static PyTypeObject SubUnicodeType = {
...
.tp_basicsize = sizeof(PyUnicodeType->tp_basicsize) + sizeof(MyExtraData),
...
};
static MyExtraData *get_extra_data(PyObject *mystring) {
MyExtraData *data = (MyExtraData *)(((char*)mystring) + PyUnicodeType->tp_basicsize);
}
```
I do recommend to use `PyUnstable_UnicodeObject` instead.
---
### Author:  [@ronaldoussoren](https://discuss.python.org/u/ronaldoussoren)
#### Post date: [March 21, 2025, 9:41am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/13 "2025-03-21T09:41:30Z")
> [@encukou](#):
>
> That’s why I’m proposing `PyUnstable_UnicodeObject` for this use case. I hope to get people that currently need `PyUnicodeObject` to remove it if they can, and share more of their use cases if they can’t.
I’ve shared my use case, and cannot move away from accessing `PyUnicodeObject` unless there’d be another way to create a subclass with additional C fields, or there’d be a different way to have string-like types (in a way that’s supported by both Python code and native extensions). The latter would be preferable for me due to a slight mismatch between the semantics of Objective-C and Python strings.
For completeness sake: There is a way for me to drop using `PyUnicodeObject`, but that would break code for users of PyObjC.
---
### Author:  [@malemburg](https://discuss.python.org/u/malemburg)
#### Post date: [March 21, 2025, 10:25am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/14 "2025-03-21T10:25:40Z")
> [@encukou](#):
>
> Subclassing as documented in the tutorial currently _does not_ work with strings
Yes, I know. But that’s not the point.
The point is that if we want go ahead hiding type structs from C extension writers, we need to provide an alternative way of adding more data to such objects at a C level, both for object types which do extend the size of the object to store variable sized data and for the more common ones which don’t.
> [@encukou](#):
>
> But, if you want a no-underscored-API version, keep your `PyUnicode_New` replacement, and add something like this (untested):
If I understand correctly, you want to put the new data between the end of the static entries in PyUnicodeObject and the variable sized part, right ?
I don’t think that’ll work, since the standard Unicode APIs will happily overwrite your added data, since they believe the variable sized part starts right where you just put your new data.
A better way is to not touch the initialization logic and add your data at the end, after `PyUnicode_New()` has done its work and the object has been finalized by adding data to it. This will require replacing the object type with the subtype (aka subclass) and then possibly reallocating the object to make room for the extra data after the variable sized part (this can be avoided by asking for some extra room in the object when allocating it).
But regardless, we need a generic non-hacky solution for these things.
---
### Author:  [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [March 21, 2025, 11:31am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/15 "2025-03-21T11:31:53Z")
> [@malemburg](#):
>
> we need to provide an alternative way of adding more data to such objects at a C level
I agree! We do! I’d love to do this for strings.
I believe there is a generic way to do this for types _whose instance structs are not public_. If we can’t hide the structs, we can add a solution specific for strings. I started on that; hopefully it’ll work.
An extra constraint is that the alternative way _can’t break the existing hacky ways_. That requires knowing what the existing ways are. This thread has been very helpful for that.
So far, I haven’t seen a use case that doesn’t rely on _other_ undocumented details.
> [@malemburg](#):
>
> A better way is to not touch the initialization logic and add your data at the end, after `PyUnicode_New()` has done its work and the object has been finalized by adding data to it.
And after reallocation, don’t forget to update all the internal pointers. But, we don’t need to redesign that: PyObjC/NumPy have it covered (but they don’t show all subtleties they needed to avoid).
AFAICS, we either need to keep their proven approach working (by not changing `PyUnicodeObject`), or design a generic non-hacky solution (which, I currently believe, would be far easier if we can change the layout).
* * *
BTW, it looks like: the free-threading build _will_ change the semantics of `PyASCIIObject.interned`. As far as I understand, before free-threading is made default, at least that field should be hidden behind the accessor function – or it should be documented as one of the things to check before setting `Py_MOD_GIL_NOT_USED`. (That’s largely unrelated to my goal here, I’m pointing it out to anyone interested in removing roadblocks for free-threading.)
---
### Author:  [@malemburg](https://discuss.python.org/u/malemburg)
#### Post date: [March 21, 2025, 1:48pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/16 "2025-03-21T13:48:12Z")
> [@encukou](#):
>
> I believe there is a generic way to do this for types _whose instance structs are not public_. If we can’t hide the structs, we can add a solution specific for strings. I started on that; hopefully it’ll work.
Why make a special case for strings rather than come up with a clean design for all Python objects ?
If this needs changes to the basic PyObject structs, then now is a good time for this, since AFAIK the free threading code does need such changes as well (but could be wrong - I only remember reading such comments occasionally).
There’s no need to rush any of this, so let’s take time.
> [@encukou](#):
>
> I haven’t seen a use case that doesn’t rely on _other_ undocumented details.
I think you should call this _implementation strategy_ rather than _use case_. It is pretty clear that there are lots of use cases for subclassing `str`, e.g. in order to hold references to objects in other applications (think interfaces to other programming languages, libraries or systems), to extend strings for particular purposes (think path and URL objects), to reference alternative content (think translated strings), etc. etc.
---
### Author:  [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [March 21, 2025, 5:52pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/17 "2025-03-21T17:52:48Z")
Here’s a Python 3.12+ extension module `mymod` that defines a type `MyStr` with settable attributes `number` (int) and `flag` (bool), and a `get_data` method, without using `PyUnicodeObject` et. al.:
```c
#define Py_LIMITED_API 0x030c0000 /* 3.12 */
#include
#include
typedef struct {
int number;
char flag;
} MyStr_Data;
static PyTypeObject *MyStr_Type = NULL;
static MyStr_Data *
mystr_get_data(PyObject *self) {
return PyObject_GetTypeData(self, MyStr_Type);
}
static PyObject *
mystr_get_data_meth(PyObject *self, PyObject *dummy)
{
MyStr_Data *data = mystr_get_data(self);
if (!data) {
return NULL;
}
return Py_BuildValue("nl", data->number, data->flag);
}
static PyType_Spec mystr_spec = {
.name = "mymod.MyStr",
.basicsize = (int)-sizeof(MyStr_Data),
.slots = (PyType_Slot[]) {
{Py_tp_members, (PyMemberDef[]) {
{"number", Py_T_INT, offsetof(MyStr_Data, number),
Py_RELATIVE_OFFSET},
{"flag", Py_T_BOOL, offsetof(MyStr_Data, flag),
Py_RELATIVE_OFFSET},
{0} /* sentinel */
}},
{Py_tp_methods, (PyMethodDef[]){
{"get_data", mystr_get_data_meth, METH_NOARGS,
"get the extra data as a tuple"},
{0} /* sentinel */
}},
{0} /* sentinel */
}
};
int
mymod_exec(PyObject *mod)
{
if (MyStr_Type) {
PyErr_SetString(PyExc_ImportError,
"cannot load module more than once per process");
return -1;
}
MyStr_Type = (PyTypeObject *)PyType_FromSpecWithBases(
&mystr_spec, (PyObject *)&PyUnicode_Type);
if (!MyStr_Type) {
return -1;
}
if (PyModule_AddType(mod, MyStr_Type) < 0) {
Py_DECREF(MyStr_Type);
return -1;
}
Py_DECREF(MyStr_Type);
return 0;
}
static PyModuleDef mymod_def = {
.m_name = "mymod",
.m_slots = (PyModuleDef_Slot[]) {
{Py_mod_exec, mymod_exec},
{0} /* sentinel */
},
};
PyObject *
PyInit_mymod(void) {
return PyModuleDef_Init(&mymod_def);
}
```
This of course ignores a lot of complications, but, most are orthogonal.
AFAICS, PyObjC would want a bit of new API: `PyUnicode_NewSubtype`, a function like `PyUnicode_New` that additionally takes a “type” argument, and returns an uninitialized string that’s “fillable” with the same caveats as `PyUnicode_New`.
I have a [_draft_ (!) branch](https://github.com/python/cpython/compare/main...encukou:cpython:pyunicode-new-sub) adding that, with [a test type that exercises some of the complications](https://github.com/python/cpython/compare/main...encukou:cpython:pyunicode-new-sub#diff-77645a51ae29e375591d4743da76099199a21c5348a5f8d1d9835bd86392c6a4).
I didn’t get PyObjC tests to run yet, I didn’t try porting NumPy, I didn’t audit the edge cases, but so far this seems viable.
(Subtype support in `PyUnicode_Writer` would also be nice, later.)
> [@malemburg](#):
>
> both for object types which do extend the size of the object to store variable sized data and for the more common ones which don’t.
Do you have an example of one that doesn’t?
---
### Author:  [@malemburg](https://discuss.python.org/u/malemburg)
#### Post date: [March 21, 2025, 7:11pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/18 "2025-03-21T19:11:07Z")
> [@encukou](#):
>
> > [@malemburg](#):
> >
> > both for object types which do extend the size of the object to store variable sized data and for the more common ones which don’t.
>
> Do you have an example of one that doesn’t?
I guess I wasn’t clear enough. I meant standard PyObject object types, which are not variable sized (i.e. not PyVarObjects); not cases where you want to subclass a type in C, but don’t add extra data fields, which seems to be what you’re asking for.
That said, I can imagine such a use case as well, e.g. if you just want to add additional methods to an existing type without changing the original type.
---
### Author:  [@malemburg](https://discuss.python.org/u/malemburg)
#### Post date: [March 21, 2025, 7:45pm UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/19 "2025-03-21T19:45:14Z")
> [@encukou](#):
>
> ```python
> .basicsize = (int)-sizeof(MyStr_Data),
>
> ```
Looks like you’re using [PEP 697 – Limited C API for Extending Opaque Types | peps.python.org](https://peps.python.org/pep-0697/)
> [@encukou](#):
>
> AFAICS, PyObjC would want a bit of new API: `PyUnicode_NewSubtype`, a function like `PyUnicode_New` that additionally takes a “type” argument, and returns an uninitialized string that’s “fillable” with the same caveats as `PyUnicode_New`.
To make this more generic, we’d need a new type slot for this purpose, e.g. `tp_newsubtype`.
---
### Author:  [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [March 24, 2025, 9:36am UTC](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543/20 "2025-03-24T09:36:51Z")
> [@malemburg](#):
>
> I guess I wasn’t clear enough. I meant standard PyObject object types, which are not variable sized (i.e. not PyVarObjects)
I still don’t think I know what you’re asking for, unfortunately.
> [@malemburg](#):
>
> Looks like you’re using [PEP 697 – Limited C API for Extending Opaque Types | peps.python.org](https://peps.python.org/pep-0697/)
As the title suggests, that’s API for the case where the struct is opaque (or not available at all). And it turns out that it works for `str`, better than I thought! :)
The issue for PyObjC’s use case is that you can’t construct such strings from C arrays (UCS2 in this case): the existing API (`PyUnicode_New` or the new `PyUnicodeWriter`) give you exact `str`.
> [@malemburg](#):
>
> To make this more generic, we’d need a new type slot for this purpose, e.g. `tp_newsubtype`.
What would be the signature of a `tp_newsubtype` function?
Obviously it can’t be the `str`-specific `PyObject *PyUnicode_New(Py_ssize_t size, Py_UCS4 maxchar)`.
It could require boxing, and take Python `args` tuple & `**kwargs` dict – that’s `tp_new`, we have it already.
It could take `void*` and require you to pass in a correct structure depending on the type – but that seems rather fragile.
[Next page](https://discuss.python.org/t/deprecating-the-direct-use-of-str-internals-e-g-pyasciiobject-pycompactunicodeobject-pyunicodeobject-structs/82543.md?page=2)