Add a new `ApiMisuseException`

Core Development
1

We currently raise SystemError for a variety of API misuses. This is misleading to users and results in spurious bug reports to CPython.

I think we should add an ApiMisuseException for this, and keep SystemError for actual system errors.

Some examples where ApiMisuseException would replace SystemError:

  • A builtin function returns non-NULL and sets the exception or returns NULL and does not set the exception.
  • PyBytes_FromStringAndSize, PyUnicode_FromStringAndSize, etc. is called with a negative size
  • PyFile_WriteString is called with a NULL file
  • PyUnicode_New is called with an invalid character
  • One of the PyArg_Parse variants is passed an invalid formats or arguments

There are, no doubt, other examples.

7 Likes
2

This would be nice for downstream projects like PyO3, where we could likely assume that any such ApiMisuseException in packages implemented in Rust is indicative of a bug in PyO3 :slight_smile:

1 Like
3

I dare say that’s what normally qualifies as an “assertion error”, but I suppose it makes sense to keep it separate from explicit user assertions.

4

+1, but to bikeshed…: which API was misused?
Perhaps name it BadInternalCall (after the function). Perhaps slightly incorrect (it’s not always a call), but more obviously related to the (interpreter) internals.

5 Likes
5

What about 3rd party extension modules (arguably a more likely source of
bad API calls)?

1 Like
6

I think that’s the point - it would mostly be third-party extensions that trigger this exception. And Mark’s (probably optimistic) hope its that people would report the bugs to the authors of those extensions rather than to the authors of Python.

7

What are examples of SystemError which are not API misuses? Would not be easier and more compatibly to add InternalSystemError for such cases if there are any?

3 Likes
8

Re: The `SystemError` description is misleading · Issue #129407 · python/cpython · GitHub

I think it makes sense to add a separate exception, but I’m a bit concerned about what weird things we might break by changing exception types (e.g., will tests start failing because they were expecting SystemError?)

Another interesting case to think about is extensions that themselves expose ABIs; in the past, I’ve raised SystemError from pyawaitable to denote misuse, but in practice, that caused some confusion. It would be quite helpful if a new exception could be used for extension ABIs instead of SystemError.

Adding on the bikeshedding: I like something along the lines of NativeError more than BadInternalCall–IMO, we should preferably keep the phrase “internal” out of the exception if it wasn’t caused by the core.

9

My initial thinking is that this is the more compatible way to do it. Any SystemError that’s entirely internal can be a subclass of SystemError, which can be distinguished by handlers that care but still be handled by those that don’t.

But I suspect we don’t actually have any internal ones, as we tend to rely on plain old assertions for cases where we are sure we control the inputs. So perhaps we just need to document it more clearly?

10

If we had a way to often detect if the likely cause of the error (just the C caller?) is “internals” vs “non-internals” code we could even differentiate by subtypes… being correctish even 90% of the time here is probably useful?

11

FYI this doesn’t follow PEP 8 as it says to capitalize acronyms.

3 Likes
12

Actually, I encountered one case of an internal SystemError 2 years ago. SystemError in re.match with a "*+" pattern · Issue #101955 · python/cpython · GitHub

            PyErr_SetString(PyExc_SystemError,
                            "The span of capturing group is wrong,"
                            " please report a bug for the re module.");

It clearly states that it is is an internal error. But I don’t think that there are many such cases. The internals are mostly bug-free, and consistency checks are usually implemented as asserts.

Even in case of genuine internal errors, it is not always possible to differentiate them from API misuse, because the internal code uses the API, so an internal bug can lead to API misuse.

2 Likes