We have been looking in PyO3 at using the Python Thread-Specific-Storage API to provide thread-local storage for extensions which don’t want to depend on the implementation in the Rust standard library.
One feature which seems to be missing to me is the ability to register a destructor when the thread exits; it seems to me that there’s no option but to leak allocated thread-local values when their owning thread exits.
To support this pthread_key_create accepts a function-pointer second argument as a destructor function on thread exit. It looks like CPython currently sets this unconditionally to NULL within PyThread_tss_create.
It might be useful to have a new API function PyThread_tss_create_with_destructor which accepts a destructor function to be passed to pthread (or equivalents on other platforms, if possible).
This is obviously platform-specific behaviour and not all platforms will be able to guarantee the same behaviour as pthread. Rust, for example, calls out that thread-local value destruction is best-effort, and I think any effort by CPython here would be similarly limited.
I think for PyO3’s own internal use our current thread-local values are just integers so this is not a huge current concern, however I expect we might have more complicated thread-locals in future to support PEP 788 properly. I also think if PyO3 was to expose the TSS to downstream Rust programs it would be useful if destructors were best-effort supported.
Does this seem like a reasonable extension of the TSS API?
I’ve also missed this. (Admittedly, that was when thinking about how to implement projects that I haven’t yet got around to actually starting, so it’s a fairly abstract version of “missed this”).
It looks to me like it wouldn’t be easily supported on Windows though, which might be a fairly major platform limitation.
Out of curiosity, why don’t extensions want to do this? I’ve always thought of the PyThread_tss* API as obsolete since the introduction of C11’s _Thread_local.
As a general rule, we don’t want CPython to be treated as a library of non-CPython-related OS helper functions. If there use of CPython APIs isn’t specifically about interacting with a CPython runtime, then we’re inclined[1] to not guarantee stability at the same levels as our own APIs.
If it’s a case where there’s a gap in the ecosystem that would benefit from a shared/common library, I’m quite happy to see CPython depend on such a shared library for it. I’m less happy to see CPython become the shared library for it.
That may not matter to you here, if you’re happy enough for the TSS APIs to not be in the stable ABI (which is where this question has come up in the past), even though you’d like all the other APIs to be guaranteed that stable.
None of this may even apply to this particular case, I don’t understand it well enough from the original post. If the request is for a Python-level callback when a Python thread is tearing down, that might be viable. If it’s supposed to be independent of Python threads and solely dependent on OS thread features, then I’d be leaning against, based on the rationale above.
Thanks for the discussion and sorry for the slow follow-up.
I agree that in most circumstances the Rust stdlib would be the right thing to use. Some counterexamples might be:
Extensions might want to opt out of depending on it if they’re building for rare architectures where the Rust standard library is poorly supported (I think for now CPython still goes more places than Rust, although the gap is narrowing).
wasm32-unknown-unknown target has no platform APIs (though in that case I believe there is also no threading so this particular case is moot).
I am hoping to one day make PyO3 compatible with MicroPython as a backend; for embedded Rust it’s common to opt out of the standard library. (Though again this would not concern CPython APIs, and possibly not threading either.)
Rust for CPython is also considering whether to avoid the Rust stdlib to avoid having CPython take on a dependency on the stdlib (there are pros and cons of having that dependency which we can defer if we choose initially not to have it).
I think for now it seems like there’s not much justification to extend the TSS API; I would guess C11 _Thread_local is also not guaranteed to work in places that Rust’s stdlib doesn’t reach.