PEP 799 – A dedicated profilers package for organizing Python profiling tool

[pablogsal (Pablo Galindo Salgado)]

@steve.dower

why is it important for this functionality to be stdlib compared to a PyPI package? (bearing in mind that subinterpreters were suitable for PyPI, despite being entirely a core runtime feature)

• Sampling profilers like this require a extremelly tight coupling with CPython internals. They depend on accessing low-level execution state with minimal overhead, supporting multi-threaded programs, free-threading, async code, and even remote attachment. All of this requires syncing the profilers AND the features, even between patch versions. Because of this deep integration, changes in CPython internals can easily break or degrade sampling profilers (this happens constantly and is very painful for everyone). Keeping it in the stdlib ensures synchronization with those changes.

• Having the profiler in the stdlib ensures users have a high performance profiler with these features working on the release of CPython and they don’t need to wait months or years until the other tools reverse engineer all the ways CPython broke them or make their life harder.

• We can use the profiler to profile Python code during development of CPython. Normally not having into the sodlib means no profiler works until the release so we cannot benefit from it during development cycles.

• The stdlib now lacks a performant profiler that works for the different things that we offer (free threading, asyncio, …etc). Including a well-supported, zero-overhead profiler in the stdlib provides a known baseline and reference implementation that other tools can use as a baseline or emulate. It also avoids fragmentation, which is especially important in performance tooling. The PEP goes into some more detail.

• It also provides a stable, officially supported reference implementation that external tools can study or build on without needing to resort to undocumented or fragile techniques. This will help everyone know a baseline on how to adapt their things to new versions and don’t force them to have to find out contorted ways to keep the use case working and ensures that we don’t break these tools on every release.

• Including this in the stdlib helps steer users away from outdated tools like profile, and toward modern, efficient practices.

@steve.dower

we’ll have compression and concurrent namespaces - profilers language-wise stands out. Why not profiling for consistency? (Arguably reusing profile would be more consistent, but too ambiguous IMHO)

I don’t mind that If everyone else agrees. How strongly do you feel about this? One downside of this is that profiling exists on PyPI but profilers doesn’t.

(Arguably reusing profile would be more consistent, but too ambiguous IMHO)

That would be super confusing for everyone and make the deprecation much harder IMHO