# PEP 669: Low Impact Monitoring for CPython

**URL:** <https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018>\
**Category:** PEPs\
**Created:** [January 10, 2022, 3:31pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018 "2022-01-10T15:31:08Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [August 11, 2022, 2:36pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/21 "2022-08-11T14:36:17Z")

</div>

Thanks for the PEP!

> Tools may see events after returning `DISABLE`, in which case, they will not see those events until `sys.monitoring.restart_events()` is called. Note that `sys.monitoring.restart_events()` is not specific to one tool, so tools must be prepared to recieve events that they have chosen to DISABLE.

This looks like an editing mistake, should it be something like “Tools may return `DISABLE` from an event callback, in which case, they will not see that event…”?

> Thus, if [PEP 523](https://peps.python.org/pep-0523) is in use, then calling `sys.monitoring.set_events()` or `sys.monitoring.set_local_events()` will raise an exception.

I first read that as “after any function specified in PEP 523 has been called”, but I don’t see how `_PyCode_SetExtra` would interfere with monitoring. Did you mean “if a custom frame evaluation function is set”?

* * *

It seems that `JUMP` and `BRANCH` events are each generated by a specific set of bytecodes, which change as new optimizations come in. Would it make sense to expose the sets in `dis`, so coverage tools can more easily detect blocks? Or is `dis.hasjrel+dis.hasjabs` already guaranteed to contain all jump opcodes?

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [August 11, 2022, 3:16pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/22 "2022-08-11T15:16:07Z")

</div>

> I first read that as “after any function specified in PEP 523 has been called”, but I don’t see how `_PyCode_SetExtra` would interfere with monitoring. Did you mean “if a custom frame evaluation function is set”?

What use does `_PyCode_SetExtra` have without setting a frame evaluation function?  
It seems simpler to ban any combination of PEP 523 and PEP 699, than attempting to reason about what might be OK.

> Or is `dis.hasjrel+dis.hasjabs` already guaranteed to contain all jump opcodes?

`opcode.hasjrel` is what you want.

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [August 11, 2022, 4:07pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/23 "2022-08-11T16:07:25Z")

</div>

> [@markshannon](#):
>
> It seems simpler to ban any combination of PEP 523 and PEP 699, than attempting to reason about what might be OK.

Simpler to write, sure, but at least in the implementation/review you’ll need a precise definition to reason about.  
It’s unclear (=hard to reason about) what “using” a document means. Accessing `interp->eval_frame` probably doesn’t qualify as “use of PEP 523”. Setting it probably does, but can’t check if `sys.monitoring.set_events()` has been called (is the check delayed in that case?). Calling `_PyInterpreterState_SetEvalFrameFunc` probably does qualify, though the function isn’t even mentioned in PEP 523. Calling `_PyEval_EvalFrameDefault` obviously doesn’t qualify, even though it was introduced in PEP 523.  
Please be precise when writing a PEP, so as a reader I can be sure I know what you mean.

> [@markshannon](#):
>
> What use does `_PyCode_SetExtra` have without setting a frame evaluation function?

I would be surprised if no one found another use for marking code objects.  
Generally it seems weird to group API based on the proposal that introduced it. Does `co_extra` actually interfere with monitoring?

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [August 11, 2022, 5:00pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/24 "2022-08-11T17:00:05Z")

</div>

Ok, you’ve convinced me. [PEP 669: Clarify and restrict interaction with PEP 523. by markshannon · Pull Request #2760 · python/peps · GitHub](https://github.com/python/peps/pull/2760)

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [August 18, 2022, 11:32am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/25 "2022-08-18T11:32:31Z")

</div>

Thank you!

And thank you again for the PEP. It is a great feature, and I have no doubt in your ability to make a self-consistent feature.  
I can also see how these “edges” where the feature meets the rest of the system can be annoying – they’re not part of the actual improvement, and they can be under-documented and used in surprising ways. But that’s also why I think a PEP should specify them as carefully than the feature itself, if not more.  
So please bear with me. (Or maybe delegate this so you focus on the meat of the change?)

* * *

With the current wording, “using PEP 523” as it is written – that is, setting `PyInterpreterState.eval_frame` directly – will avoid the exception.  
I suggest adding the following:

> To avoid bypassing `_PyInterpreterState_SetEvalFrameFunc()` by  
> setting `PyInterpreterState.eval_frame` directly (as specified in  
> :pep:`523`), the field will be renamed to `_eval_frame` and documentation  
> will be updated to avoid references to :pep:`523`.

I’m happy to help with that documentation update, but since I’m not an expert in this area and don’t know who’ll be affected, I’d go through the PEP process on the decision.

* * *

Another place where I’m not sure how this PEP interacts with the rest of CPython is Quickening. I’m [still unhappy](https://discuss.python.org/t/14258/3) about PEP 659 being referred to while it’s still a draft.  
Was PEP 659 implemented as written, or are there any notable changes?

* * *

For better introspection in [Tool identifiers](https://peps.python.org/pep-0669/#tool-identifiers), may I suggest an API like:

```python
sys.monitoring.use_tool_id(id, name=None) -> None
sys.monitoring.free_tool_id(id) -> None
sys.monitoring.get_used_tools() -> dict[int, str|None] # unclaimed IDs are not included
# str(name) should help a human identify the tool, name has no requirements beyond that

```

so `use_tool_id` can raise `ValueError: monitoring ID 3 is already used by Cinder`, and tools can get the name for their own error messages, like warnings on the profiler+debugger case mentioned in [Events in callback functions](https://peps.python.org/pep-0669/#events-in-callback-functions).

* * *

I sent [PR 2767](https://github.com/python/peps/pull/2767) with some smaller suggestions.

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [August 18, 2022, 2:09pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/26 "2022-08-18T14:09:14Z")

</div>

Thanks for reminding me that we need to sort out the status of PEP 659.  
PEP 659 is up to date and accurate.

I like the idea of using names. Clashes should be very rare, but it will be a lot easily for a user to sort out if tools have names.  
If we are going to name tools, we might as well make the name compulsory.

```auto
sys.monitoring.use_tool_id(id, name:str) -> None
sys.monitoring.free_tool_id(id) -> None
sys.monitoring.get_tool(id) -> str | None

```

---

<div class="post-metadata">

**Author:** ![TeamSpen210](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/teamspen210/32/1004_2.png) [@TeamSpen210](https://discuss.python.org/u/TeamSpen210)\
**Post date:** [August 18, 2022, 10:26pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/27 "2022-08-18T22:26:02Z")

</div>

To reduce magic numbers (and seamlessly allow for increasing the number of tools in the future), perhaps it’d be a good idea to add a `sys.monitoring.VALID_IDS = range(6)` constant? Or just an integer `MAX_ID`, if using a `range` here would be problematic.

---

<div class="post-metadata">

**Author:** ![barry-scott](https://avatars.discourse-cdn.com/v4/letter/b/e9c0ed/32.png) [@barry-scott](https://discuss.python.org/u/barry-scott)\
**Post date:** [August 19, 2022, 8:22am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/28 "2022-08-19T08:22:31Z")

</div>

Why not add a call to allocate an Id for a tool?  
If that call fails then the caller can report too many tools are using the feature?

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [August 22, 2022, 1:55pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/29 "2022-08-22T13:55:02Z")

</div>

> [@markshannon](#):
>
> Thanks for reminding me that we need to sort out the status of PEP 659.  
> PEP 659 is up to date and accurate.

OK, I [submitted it to the SC](https://github.com/python/steering-council/issues/134).

> [@markshannon](#):
>
> Clashes should be very rare, but it will be a lot easily for a user to sort out if tools have names.

IMO Python & the tools should treat names purely as human-oriented “flavor text”, so there’s no need to worry about clashes.  
The `get_used_tools() -> dict` was meant to give you all registered tools at once, which might be useful for introspection. I’d like to see that dict in automated error reports, for example.

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [August 24, 2022, 10:09am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/30 "2022-08-24T10:09:43Z")

</div>

I didn’t mean clashes in names, but in IDs.

I would expect that if a tool’s preferred ID is in use, it would fail.  
No one wants two different debuggers running at the same time (unless they are debugging a debugger).

If you really need `get_used_tools()`, it can be implemented as

```auto
def get_used_tools():
     res = {}
     for id in range(6):
          name = sys.monitoring.get_tool(id)
          if name is not None:
              res[id] = name
    return res

```

---

<div class="post-metadata">

**Author:** ![fabioz](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/fabioz/32/2950_2.png) [@fabioz](https://discuss.python.org/u/fabioz)\
**Post date:** [September 1, 2022, 6:56pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/31 "2022-09-01T18:56:18Z")

</div>

Hi @markshannon one thing which I don’t see in the PEP is the interaction that it has with exceptions from the monitoring callbacks.

Is it possible to add some info on what’s expected in such cases?

In particular the following points:

- What happens when an exception originates from a monitoring callback?

i.e.: One of the hard things in the settrace is that if some exception originates in the tracing callback the tracing is disabled (so, if for some instance the user is paused in a breakpoint and he does a `Ctrl+C` the debugger will no longer work, which may not be what the user expects). Ideally this wouldn’t happen (or at least `KeyboardInterrupt` would have special treatment).

- What happens in a `RecursionError`?

A `RecursionError` also disables the tracing right now (when this happens it can be reasonably hard for users to know why it happened, especially if they silenced it accidentally) – the problem is that even in the new monitoring structure trying to hook into when a stack overflow error is raised there’d probably be no more available stack to handle it in the monitoring, so, it’d be interesting to have some alternative here (or at least make users aware of the issue somehow prior to disabling it).

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [September 2, 2022, 10:26am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/32 "2022-09-02T10:26:52Z")

</div>

> What happens when an exception originates from a monitoring callback?

It propagates like any other exception.

> What happens in a `RecursionError`?

It is treated like any other exception.

If the RecursionError is from hitting the recursion limit, then temporarily raising the recursion limit in the callback should allow it to operate normally.

If the RecursionError is from C stack exhaustion, you might find that the VM gives a fatal error if you consume much stack in the callback. Not much we can do there, we can’t magic up extra stack.

---

<div class="post-metadata">

**Author:** ![nedbat](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nedbat/32/8744_2.png) [@nedbat](https://discuss.python.org/u/nedbat)\
**Post date:** [September 4, 2022, 7:29pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/33 "2022-09-04T19:29:40Z")

</div>

This is a really interesting PEP. I apologize for taking so long to get to it. I have a number of comments/questions…

1. In the list of events, what is the logic for when they are named PY\_\* vs C\_\* vs no prefix? Why isn’t LINE called PY\_LINE? Why isn’t PY\_START called PY\_CALL? I’m assuming there’s a reason, but it seems asymmetric at a first reading.

2. It took me a while to understand `sys.monitoring.use_tool_id(id, name:str) -> None`. Perhaps we can have fleshed out docstrings for these functions. IIUC, `use_tool_id` means I want to claim an id, and I am associating `name` with it. I’m not sure what use will be made of `name` though?

3. The pre-defined ids make some presumptions about the composability of tools. For example, it assumes that I can’t coverage-measure a coverage tool. It is difficult, and coverage.py uses some tricks to accomplish it, but it’s valuable. Since there’s no enforcement to the idea that only one tool of each kind can be running at once, I suppose everything is fine, but I wonder if this idea will appear in other places with real consequences?

4. I should know what this means, but I don’t:

5. This sentence could use some clarification:

6. In the Coverage Tools section:

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [September 7, 2022, 9:07am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/34 "2022-09-07T09:07:36Z")

</div>

Thanks for the feedback.

1. Most of the design and discussion has focused on semantics, not syntax. So the names might not be the best. Suggestions for improvement are very welcome.  
`PY_START` occurs within the callee, so the call has already happened, whereas `C_CALL` happens before the call.  
Would you prefer it if `C_CALL` were changed to `CALL` and included Python functions?

2. The name is just a name. The VM doesn’t care what it is. It should help debug id clashes.

3. The pre-defined IDs are just suggestions. They are there to help the common case where you don’t to debug a debugger, or do coverage on a coverage tool. For those unusual cases, you are free to choose any ID you want, and it is then your problem to avoid clashes and provide sensible error messages.

4. No. Callbacks must be callable Python objects. You can implement those in C, or C++ or Rust, provided the resulting object is callable. Using the vectorcall protocol will give you near C function-pointer performance.  
For a coverage tool, Python will be plenty fast enough. The trick is to return `DISABLE` and only get called once per location.

5. a. Since the line number is fixed for any (code, instruction\_offset), the two are equivalent.  
b. All branches from that point. I can see the advantage of tracking each direction independently, but it would be a special case and would impact performance and memory consumption.

6. Using `JUMP` and `BRANCH` is more efficient than line based tracing. Using line numbers will continue to work, though.

---

<div class="post-metadata">

**Author:** ![nedbat](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nedbat/32/8744_2.png) [@nedbat](https://discuss.python.org/u/nedbat)\
**Post date:** [September 7, 2022, 11:29pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/35 "2022-09-07T23:29:03Z")

</div>

> Most of the design and discussion has focused on semantics, not syntax. So the names might not be the best. Suggestions for improvement are very welcome. `PY_START` occurs within the callee, so the call has already happened, whereas `C_CALL` happens before the call.

I was trying to infer a pattern from the names, but if there isn’t one, that’s OK too.

> Would you prefer it if C\_CALL were changed to CALL and included Python functions?

I wouldn’t want C and Python functions mixed together.

> No. Callbacks must be callable Python objects. You can implement those in C, or C++ or Rust, provided the resulting object is callable. Using the vectorcall protocol will give you near C function-pointer performance.

I don’t know what “vectorcall protocol” means, but I will figure it out when the time comes.

> For a coverage tool, Python will be plenty fast enough. The trick is to return DISABLE and only get called once per location.

I have a sense overall that you have a specific idea about how a coverage tool will work, and that coverage.py doesn’t work quite that way. In particular, there are a number of reasons why getting called just once per location wouldn’t always be sufficient. There are options in coverage.py that require collecting more data than that: branch coverage and contexts are two options that would mean I can’t disable an event after it is fired.

> All branches from that point. I can see the advantage of tracking each direction independently, but it would be a special case and would impact performance and memory consumption.

You’ve placed great emphasis on the idea of disabling a callback after it has fired. In order to measure branch coverage, I need to know all of the branches that have been taken. If I disable the branch event once I’ve received it, and that disables all branches from that point, then I don’t have the information I need. That’s why I said it would be useless to disable the branch event if it didn’t take the destination into account.

> Using JUMP and BRANCH is more efficient than line based tracing. Using line numbers will continue to work, though.

I haven’t worked this all through, so I might not be understanding your idea completely. `JUMP` and `BRANCH` would give me data about bytecode offsets. If I track that information, then I need to map that back to line numbers to produce a report for the user. Is that right? Can you say more about why `JUMP` and `BRANCH` are more efficient than line-based?

I definitely don’t want to have to understand the specifics of individual bytecode operations, but I don’t think you are saying that.

---

<div class="post-metadata">

**Author:** ![pablogsal](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pablogsal/32/32246_2.png) [@pablogsal](https://discuss.python.org/u/pablogsal)\
**Post date:** [September 15, 2022, 6:56pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/36 "2022-09-15T18:56:56Z")

</div>

Hi,

**I am writing this message on behalf of the Python Steering Council.**

We are quite happy with PEP 669 and we think this will be a great addition to Python. Before we are ready to accept the PEP we would like to discuss some aspects of it:

- The PEP does not include anything regarding threads. The following questions are pertinent:

- The pep mentions the following:

- The PEP puts a lot of emphasis on debuggers ([PEP 669 – Low Impact Monitoring for CPython | peps.python.org](https://peps.python.org/pep-0669/#debuggers)) but there are some questions regarding the APIs provided.

- The PEP mentions the following:

- Could you also add a section outlining how new events can be added in the future if necessary?

- Although we can more or less understand it from the PEP, is unclear how a profile function can request granular results. For example, let’s say a profile function doesn’t want the line number and uses `PY_START` events, how can the API ensures that this information is not calculated if the callback doesn’t need it?

- The questions that @nedbat asks in [PEP 669: Low Impact Monitoring for CPython - #35 by nedbat](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/35) should also be answered to ensure that the API makes sense and that it can be leveraged as much as it can by coverage tools.

- In general the PEP lacks time benchmarks for some common usages like simple coverage, profile or tracing functions. Having time benchmark information is important so we can make an informed decision

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [September 20, 2022, 11:32am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/37 "2022-09-20T11:32:01Z")

</div>

> I don’t know what “vectorcall protocol” means

[PEP 590](https://peps.python.org/pep-0590/)

> If I track that information, then I need to map that back to line numbers to produce a report for the user. Is that right?

Yes, you’ll need to do that. `code.co_lines()` has the offset to line information.

> Can you say more about why `JUMP` and `BRANCH` are more efficient than line-based?

Two reasons.

1. There are fewer `JUMP` and `BRANCH` events than line events
2. `JUMP` and `BRANCH` events map to specific VM instructions, so can be instrumented more efficiently.

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [September 20, 2022, 1:00pm UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/38 "2022-09-20T13:00:14Z")

</div>

> The PEP does not include anything regarding threads.

The PEP makes no mention of threads, because they are not relevant.  
Instrumentation is per-interpreter not per-thread. I’ve added a line to the PEP to make this a bit clearer.

> “sys.setprofile() can be made a lot faster by using the API provided by this PEP”

The full sentence has a typo in it, which doesn’t help. I fixed it in the PEP. The full sentence _should_ have read:

> However, tools relying on `sys.settrace()` and `sys.setprofile()` can be made a lot faster by using the API provided by this PEP.

How is this true? Not because the proposed approach is amazingly fast, but because `sys.settrace()` and `sys.setprofile()` are really slow.

> How are debuggers supposed to translate that into the provided APIs that receive code objects in a performant way?

I don’t know what “receive code objects in a performant way” means, but if you are asking how one should implement a breakpoint in a way that minimizes performance impact, here is one way:

- When debugger is attached, create an empty map of filenames to code objects and an empty map of filenames to uninstrumented breakpoints.
- When receiving a `PY_CALL` event:
  - For all breakpoints in the uninstrumented map, if they lie within the code object, insert them. Finding the breakpoint is `O(log n)` where `n` is the number of uninstrumented breapoints per file.
  - Add the code object to the code object map, then return `DISABLE`.

- To add a breakpoint:
  - If the code object containing the breakpoint is in the map, use `insert_marker()` to set the breakpoint.
  - If not in the map, then add the breakpoint to the set of uninstrumented breakpoints
  - Finding the code object is `O(log m)` where `m` is the number of code objects per filename.

Feel free to design your own scheme, but the above scheme is fast enough to implement in Python without noticeable overhead.

> We don’t believe that is True [that `sys.settrace` is incompatible with [PEP 523](https://peps.python.org/pep-0523)]

Are you claiming that all tools using [PEP 523](https://peps.python.org/pep-0523) support `sys.settrace` and `sys.setprofile` perfectly? Cinder doesn’t. I doubt that any of the debuggers using [PEP 523](https://peps.python.org/pep-0523) work flawlessly with `pdb`. It isn’t even clear what is debugging what.

Rather than hoping for the best, I think it better to just say: “This doesn’t work”.

> Could you also add a section outlining how new events can be added in the future if necessary?

I don’t think that makes sense in the PEP.  
Future events are likely to come from future language changes, and I have no way to predict how those would be implemented.

Also I’m not sure whether you are referring to the social or technical process.  
The social process; a new PEP or just an issue?  
Or do you mean how would the CPython source be changed to support additional events?  
If the latter, then no different from any other code change, I guess. Make a PR with the changes.

> Although we can more or less understand it from the PEP, is unclear how a profile function can request granular results

I don’t understand what you mean by “granular results”

> a profile function doesn’t want the line number and uses `PY_START` events

The callback for `PY_START` events is `func(code: CodeType, instruction_offset: int)`. No line number.

> how can the API ensures that this information is not calculated if the callback doesn’t need it?

You can’t. Although I am puzzled why any user of the API would worry about the VM doing pointless calculations.

> In general the PEP lacks time benchmarks for some common usages like simple coverage, profile or tracing functions. Having time benchmark information is important so we can make an informed decision.

I’m afraid there will be no benchmarks until it is approved, as I’m not willing to implement it until at least conditionally approved.

You could make approval conditional on the performance being good enough. That way I’m not wasting my time implementing this for you to reject it, and you are not accepting it without performance being satisfactory.

Regarding coverage, take a look at [Slipcover](https://github.com/plasma-umass/slipcover) which uses instrumentation  
and is [faster with coverage on 3.11 than no coverage at all on 3.10](https://github.com/plasma-umass/slipcover/issues/17#issuecomment-1183622971). The instrumentation is a bit fragile as there is no VM support. With VM support performance would be even better.

For debuggers, the scheme I described above costs one call into the debugger for each code object (not per call) plus the overhead of the actual breakpoints, and _no_ other overhead.

For profilers, instrumentation will be quicker than `sys.setprofile()`, but if you care about performance use a statistical profiler 🙂

* * *

I hope that clarifies things.

---

<div class="post-metadata">

**Author:** ![carljm](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/carljm/32/959_2.png) [@carljm](https://discuss.python.org/u/carljm)\
**Post date:** [September 21, 2022, 1:32am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/39 "2022-09-21T01:32:02Z")

</div>

> [@markshannon](#):
>
> Are you claiming that all tools using [PEP 523](https://peps.python.org/pep-0523) support `sys.settrace` and `sys.setprofile` perfectly? Cinder doesn’t.

Cinder doesn’t use PEP 523 either, so probably not a relevant example here. (It is true that the Cinder JiT doesn’t support `sys.settrace` or `sys.setprofile` at all.)

---

<div class="post-metadata">

**Author:** ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)\
**Post date:** [September 21, 2022, 10:51am UTC](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/40 "2022-09-21T10:51:36Z")

</div>

IIUC, Cinder replaces the entirety of `_PyEval_EvalFrameDefault()`. So while it may not use PEP 523, it does the equivalent.  
I think the same argument also applies to Pyston.

My point is that replacing the `_PyEval_EvalFrameDefault()` with anything but the most trivial wrapper and correctly supporting `sys.settrace()` is sufficiently difficult that we might as well just declare it impossible.

[Previous page](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018.md?page=1)

[Next page](https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018.md?page=3)
