I would like to propose a project to offer a strong integration between CPython and OpenTelemetry API compatible observability.
Disclosure: I work for Pydantic where we are developing an OpenTelemetry compatible observability platform with an emphasis on Python. OpenTelemetry is an industry standard for observability developed as a CNCF project. It’s the second most popular CNCF project after Kubernetes.
I was very excited when I first saw PEP 669 and tried to integrate the OpenTelemetry SDK around it, but failed to do so. I do not remember the details around why (this was a while ago now) but it might have been a couple missing events, issues with exception handling or something else that might be easily solvable or just lack of skills on my end. My understanding is that while there were discussions about higher level observability in previous threads about the PEP none of this was in the context of a public industry standard, and the PEP itself was developed with the objective of lower level monitoring of the runtime itself.
I would like to propose that we develop a working integration between the APIs in PEP 669 and the OpenTelemetry SDK. I’m not sure exactly what that would look like but concepts I think are worth exploring:
- Runtime / continuous profiling.
- Emitting spans / non point events by e.g. hooking into every function call.
- Emitting logs / point events when a certain line is hit.
- Inspecting the state around every raised exception.
Concretely, I would like the output to be the right shim code or a library that exposes the power of PEP 669 to users or libraries trying to implement high level observability. In particular I think this shim code or library should be compatible with OpenTelemetry’s APIs, which means that it’s also compatible with many non-OTEL observability solutions since OTEL was explicitly designed to be API compatible with other existing solutions and libraries.
As far as I can see this still does not exist. Please correct me if I’m wrong about this. I’m also happy to be told that this is easy and I just lack the knowledge to do so, but regardless I think it would be a very fun project for PyCon 2025 as a collaboration between our team, some core devs familiar with PEP 669 and anyone else who wants to join. I imagine the outcome being a headline along the lines of “Python has best of class observability making it an excellent choice for building robust production grade systems”.