Remove semi-public C-API for doing arithmetic with Py_complex type

[skirpichev (Sergey B Kirpichev)]

(Opened per a suggestion in the C-API WG issue.)

Historically, in CPython we have functions like _Py_c_sum and _Py_c_prod to do arithmetic with Py_complex (a custom type for complex numbers). Recent pr python/cpython#124829 added support for mixed-mode arithmetic and this API was extended with functions like _Py_cr_prod (i.e. when one argument is a real number). (JFR: this new C-API is hidden for now.)

It was suggested by @vstinner in python/cpython#124829 (comment) either to remove this from the public API or rename properly (e.g. like Py_complex_add() and Py_complex_add_real(), in the GNU GSL style).

Of course, people could use PyNumber_*() API. Main motivation to provide a different C-API for doing correct complex arithmetic — is speed. But with python/cpython#124829 — complex arithmetic in CPython follows to most implementations of the C99+ Annex G. So, users can use native double _Complex type instead of the Py_complex. Removing low-level arithmetic API & Py_complex struct possibly open a door to eventually switch CPython to use native complex type (double _Complex) internally (like for floats).

Proposal:

1. deprecate _Py_c_*() functions
2. add PyComplex_From/As*() methods to import/export from/to native complex type. (conditionally)
3. deprecate Py_complex struct, PyComplex_FromCComplex() and PyComplex_AsCComplex().
4. (in even more distant future) switch to using double _Complex internally

I see several arguments to keep these functions (from high to low priority):

1. special arithmetic functions will make sense if we go beyond most Annex G implementations, i.e. add support for pure-imaginary numbers. This was recently suggested here. Probably, we can conclude that this proposal (which I’m fan of) has no enough support.
2. not all Tier 1 PEP 11’s platforms have Annex G support: it’s Windows with MSVC. I don’t know if here are some plans to add such support. On another hand, this platform has a different native complex type, _Dcomplex, with some arithmetic primitives to do math with. People could use this type or some external library (like GSL) to do complex arithmetic.
3. current Annex G has no special case for real/complex division (which we have in our implementation since python/cpython#124829). So, current CPython complex arithmetic has a small incompatibility with the C standard. This is something, going to be fixed by N3460 in upcoming standard.

CPython issue: Make _Py_c_sum(), _Py_c_diff(), etc (elementary operations on Py_complex) - part of the public API · Issue #128813 · python/cpython · GitHub
C-API WG issue: Remove public low-level API functions for Py_complex? · Issue #56 · capi-workgroup/decisions · GitHub

[CAM-Gerlach]

@skirpichev Might this be better off moved to Core Development ? At least historically Ideas has typically been reserved for more speculative, PEP-sized ideas to improve the Python language and stdlib, while Core Development is for discussion of more practical immediate-term issues. Given the issue history you highlight, this is quite mature and ready to execute whatever decision is made, and smaller in scope and impact than a PEP. It will also reach a broader yet more focused audience of core developers and technically-inclined contributors than Ideas , which many/most core devs shy away from due to the prevalence of half-baked, impractical and repeated proposals.

[skirpichev (Sergey B Kirpichev)]

@CAM-Gerlach

Might this be better off moved to Core Development ?

Maybe, I was thinking about it first.
Though, last time I put post to that category it was moved to the ideas. So, I don’t trust my own judgment.

[encukou (Petr Viktorin)]

deprecate _Py_c_*() functions
deprecate Py_complex struct, PyComplex_FromCComplex() and PyComplex_AsCComplex().

I’d recommend for soft-deprecation. There’s no security issues and not much maintenance burden beyond the initial implementation.
(IMO, we should keep our implementations at least until MSVC supports Annex G. There’s not much trouble in exposing the implementations; and if/when we do switch, the functions become one-liners with existing tests.)

add PyComplex_From/As*() methods to import/export from/to native complex type. (conditionally)

By “native complex type” you mean double _Complex, right?
If MSVC ever adds double _Complex, we’ll want these functions to use that.

Removing low-level arithmetic API & Py_complex struct possibly open a door to eventually switch CPython to use native complex type (double _Complex) internally (like for floats).

We don’t need to remove the existing struct & functions for that.
What we do need is make the PyComplexObject struct opaque (or just deprecate/remove it from public headers), and require existing users to go through the existing PyComplex_FromCComplex/PyComplex_AsCComplex.
(Your proposal to deprecate Py_complex doesn’t say what to do with PyComplexObject…)

[skirpichev (Sergey B Kirpichev)]

@encukou

By “native complex type” you mean double _Complex, right?

Yes.

@encukou

If MSVC ever adds double _Complex, we’ll want these functions to use that.

Sure, but for a while — these functions will be available only if platform supports Annex G.

Proposed API:

PyObject * PyComplex_FromNativeComplex(double _Complex v);
double _Complex PyComplex_AsNativeComplex(PyObject *op)

@encukou

We don’t need to remove the existing struct & functions for that.

Ok, soft deprecation does make sense for me.

If so, we should add _Py_c_*() functions as public, together with new mixed-mode API (_Py_cr_*()) as public and soft-deprecate all that stuff.

Then the remaining thing to decide will be naming scheme. I suggest GSL-like style (as in the post). There was suggestion to use PyComplex_-prefix, but this looks wrong for me (as this API works with more low-level objects).

@encukou

Your proposal to deprecate Py_complex doesn’t say what to do with PyComplexObject…

Well, in current docs we have here same situation as with the PyFloatObject: the PyComplexObject structure declared in docs, but no fields are documented. Probably, this should stay that way. The Py_complex struct will be soft-deprecated.

[encukou (Petr Viktorin)]

@skirpichev

PyObject * PyComplex_FromNativeComplex(double _Complex v);
double _Complex PyComplex_AsNativeComplex(PyObject *op)

Sounds good! (Maybe DoubleComplex rather than NativeComplex, if it’s already time for naming details?)

@skirpichev

together with new mixed-mode API (_Py_cr_*()) as public

I don’t think we need to do that. Soft-deprecation is there to avoid breaking existing code; if we want people to use such functions we shouldn’t (soft-)deprecate them.

@skirpichev

Well, in current docs we have here same situation as with the PyFloatObject: the PyComplexObject structure declared in docs, but no fields are documented.

PyComplexObject.cval is public API that needs deprecation to remove. PEP 387 says:

Note that if something is not documented at all, it is not automatically considered private.

[skirpichev (Sergey B Kirpichev)]

@encukou

if we want people to use such functions we shouldn’t (soft-)deprecate them.

I’m not sure that we want. This API seems redundant, unless we aren’t going beyond complex arithmetic, as specified in the Annex G of the upcoming C standard (without imaginary type).

Soft-deprecation does make sense, as it’s not hard to support that API and it might help poor Windows people.

@encukou

PyComplexObject.cval is public API that needs deprecation to remove.

Ok, then this field should be renamed to underscore-prefixed and the cval alias — documented and deprecated. This will disentangle Py_complex struct from the PyComplexObject.

[h-vetinari]

@skirpichev

not all Tier 1 PEP 11’s platforms have Annex G support: it’s Windows with MSVC.

As hopeless as it may seem, people with a Microsoft account should upvote this issue to help tip the scales for MSVC to finally implement this.

[skirpichev (Sergey B Kirpichev)]

I’m not sure which issue link is canonical. Here is another one.

Using a different compiler seems to be not a solution, as MSVC is a build requirement on Windows.

[h-vetinari]

@skirpichev

I’m not sure which issue link is canonical. Here is another one.

Funnily enough, I have comments on both of these^[1]

@skirpichev

Using a different compiler seems to be not a solution, as MSVC is a build requirement on Windows.

This topic came up recently: Introducing clang-cl as an alternative MSBuild backend

---

1. I won’t comment on the UX of devcommunity. ↩︎

[skirpichev (Sergey B Kirpichev)]

@h-vetinari

This topic came up recently: Introducing clang-cl as an alternative MSBuild backend

That looks interesting, but as @chris-eibl said: “it is a drop-in compiler for MSVC and brings also its own linker. But it uses the MSVC headers and links to the MSVC (static/dynamic) libraries.” The MSVC ships own “complex.h”, which is not compatible with the Annex G. So, I doubt this open door for support native complex types on Windows.

Though, using a different compiler on Windows seems possible, e.g. in an MSYS2 environment.

[steve.dower]

@h-vetinari

As hopeless as it may seem, people with a Microsoft account should upvote this issue to help tip the scales for MSVC to finally implement this

Scales are indeed tipped by votes on issues like this.

@skirpichev

I’m not sure which issue link is canonical. Here is another one.

I’m sure they’ll get duped eventually, which should combine the votes (it used to, at least), but voting for both won’t hurt!

[skirpichev (Sergey B Kirpichev)]

Per @encukou suggestion in the C-API WG issue, I’ll update this thread with alternative proposal, i.e. what if we will expose current _Py_c_* functions as public? This is a more conservative solution than proposed in my original post.

Historically, we had above functions as semi-public (quoted from Include/cpython/complexobject.h):

// Operations on complex numbers.
PyAPI_FUNC(Py_complex) _Py_c_sum(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) _Py_c_diff(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) _Py_c_neg(Py_complex);
PyAPI_FUNC(Py_complex) _Py_c_prod(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) _Py_c_quot(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) _Py_c_pow(Py_complex, Py_complex);
PyAPI_FUNC(double) _Py_c_abs(Py_complex);

Note that the _Py_c_abs() is not shown in docs.

I propose following primitives (GSL-like) as a public API:

PyAPI_FUNC(Py_complex) Py_c_add(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) Py_c_add_real(Py_complex, double);
PyAPI_FUNC(Py_complex) Py_c_sub(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) Py_c_sub_real(Py_complex, double);
PyAPI_FUNC(Py_complex) Py_c_neg(Py_complex z);  // -z
PyAPI_FUNC(Py_complex) Py_c_mul(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) Py_c_mul_real(Py_complex, double);
PyAPI_FUNC(Py_complex) Py_c_div(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) Py_c_div_real(Py_complex, double);
PyAPI_FUNC(Py_complex) Py_c_inv(Py_complex); // 1/z = 1/(x+iy)=(x-iy)/(x^2+y^2)

This set should be appropriate to implement mathematical functions on top of that. Later it could be extended with primitives for pure-imaginary numbers, e.g.:

PyAPI_FUNC(Py_complex) Py_c_add_imag(Py_complex z, double a); // z + ia

Possible additions:

PyAPI_FUNC(Py_complex) Py_c_abs(Py_complex z);
PyAPI_FUNC(Py_complex) Py_c_arg(Py_complex z); // cmath.phase
PyAPI_FUNC(Py_complex) Py_c_pow(Py_complex, Py_complex);
PyAPI_FUNC(Py_complex) Py_c_pow_real(Py_complex, double);

Naming variants: 1) different prefix, e.g. Py_complex_, 2) more verbose names, e.g. subtract instead of sub or negative instead of neg.

[vstinner (Victor Stinner)]

Does anyone need these functions outside Python? I only found the Numba project which uses a single of existing documented functions.

[skirpichev (Sergey B Kirpichev)]

That’s a good point. GH search shows mostly clones of the CPython tree. I found two other projects:

• GitHub - WarrenWeckesser/ufunclab: NumPy ufuncs and utilities. (see e.g. log1p)
• GitHub - tchaumeny/PyPoly: A Python extension written in C, implementing the Polynomial type.

[encukou (Petr Viktorin)]

@skirpichev

Possible additions:

I am worried about unbounded scope here. If people start using this API for general complex math in C, it will be useful to add C versions for all of cmath.

If we go this way, we’ll either continue using PyComplex as CPython’s internal implementation, or we’ll be stuck with many public functions that convert PyComplex to a new representation, do the operation, and convert back. Of course, the problem is worse the more functions we add. What’s the right boundary between Py_c_add and Py_c_isclose?

If we treat PyComplex as an import/export format, the boundary would include PyComplex_FromCComplex & PyComplex_AsCComplex only.
I can’t see any other well-defined boundaries.

(Of course, if the existing _Py_c_sum etc. fall outside whatever scope we choose, they should be soft-deprected – even though we wouldn’t add them today, existing code should be able to continue using them. But if it needs any new features – or a better naming convention – it needs to look elsewhere.)

[pitrou (Antoine Pitrou)]

@skirpichev

Per @encukou suggestion in the C-API WG issue, I’ll update this thread with alternative proposal, i.e. what if we will expose current _Py_c_* functions as public?

It’s not CPython’s business to expose and maintain a general-purpose library of C helper functions.

[malemburg (Marc-André Lemburg)]

That’s not necessarily true. Python often provides helpers for cases where the compilers do not (yet) provide an agreed upon basis for standard C functions.

But perhaps in the case of complex numbers that has changed.

Does anyone know what the current status of C compilers supporting complex types looks like ?

Complex number arithmetic - cppreference.com suggests that they are part of C99, but I could not find a reference on which compilers actually implement these.

On POSIX systems, they are usually provided via libm: C mathematical functions - Wikipedia

If those already cover the _Py_c_* functions we have, then I guess we can simply remove them after a deprecation phase.

[h-vetinari]

@malemburg

Complex number arithmetic - cppreference.com suggests that they are part of C99, but I could not find a reference on which compilers actually implement these.

Complex numbers were made optional in C11, and MSVC does not support them (at least not in the way the C standard specifies); more context is further up the thread.

The one thing that wasn’t discussed yet AFAICT is that MSVC does have some support for complex using non-standard, microsoft-specific types^[1], which in theory should provide everything needed (modulo a few typedefs). I haven’t seen it in use much^[2], e.g. cffi released support for it less than a year ago.

PS. cppreference also has a page for C compiler support, but it is lamentably incomplete and outdated. The status page for clang is the best IMO, GCC just has some prose (though generally the most complete implementation) and the MSVC docs are pretty thin too.

---

1. which obviously raises the question why this wasn’t elevated to full support over the last 20 years. Sigh. ↩︎

2. standard caveat about small sample size. ↩︎

[skirpichev (Sergey B Kirpichev)]

@encukou

If people start using this API for general complex math in C, it will be useful to add C versions for all of cmath.

In general, probably yes.

On another hand, arithmetic primitives are bare minimum, required for implementation of other mathematical functions, including elementary.

@malemburg

Does anyone know what the current status of C compilers supporting complex types looks like ?

Complex number arithmetic - cppreference.com suggests that they are part of C99, but I could not find a reference on which compilers actually implement these.

Almost all (at least among Tier 1 platforms), except MSVC.

But with some remarks:

1. neither major compiler has support for pure-imaginary complex numper, per C99-C23 standards. (though, imaginary types going to be removed by upcoming C2y)
2. C99-C23 standards have no special case for real/complex division (CPython complex arithmetic has). This is going to be fixed by N3460.

@h-vetinari

The one thing that wasn’t discussed yet AFAICT is that MSVC does have some support for complex using non-standard, microsoft-specific types[1], which in theory should provide everything needed (modulo a few typedefs).

This was mentioned in the C-API WG issue.