PEP 791: imath --- module for integer-specific mathematics functions

Note that gmpy2’s factorial function requires integer arguments and returns a float result. Instead gmpy2 has a fac function that is analogous to math.factorial (or imath.factorial) and that computes and returns exact integer values. The interface of gmpy2 is like having imath, fractions, math and cmath all merged together in a single module so it is probably not a great comparison for what you would want the imath module’s interface to be.

Also I think aiming for compatibility with gmpy2 specifically is misguided in the sense that really what you want is ecosystem wide compatibility and ideally even compatibility beyond Python with comparable functions in other languages as well.

Let’s take math.comb as an example. There is gmpy2.comb which has the same name and is almost the same function but has a slightly different domain. I assume that the matching name here is an afterthought though because there is also an identical gmpy2.bincoef function.

The domain for math.comb(n, k) is nonnegative integer n and k but gmpy2.comb allows n to be negative. In mpmath and SymPy this function is called binomial and is defined for arbitrary complex n and k although specifically for negative integers n and k SymPy and mpmath disagree about the value of the function because there are two different conventions (I think mpmath has the right one).

SciPy calls the function comb and matches SymPy’s definition for negative integer values. The SciPy docs say that n and k have to be integers but it accepts and always returns floats unless exact=True is passed in which case it requires and returns ints. It doesn’t look like SciPy provides a complex extension of this function.

Looking beyond Python comb is called nchoosek in Matlab and requires nonnegative integers. In Mathematica and Maple the function is called binomial and is defined for complex arguments I think in a way that is consistent with mpmath. In Julia the function is called binomial. I might be using an outdated version of Julia here because the online docs say it is for Integer but I can pass float/complex for n but k needs to be an integer. For negative integer arguments Julia’s binomial matches the SymPy/SciPy convention rather than mpmath/Mathematica.

Ideally the authors of all Python libraries including the stdlib would consider this kind of compatibility when choosing a name, domain and definition of a mathematical function. You want the names to be consistent so that everyone agrees whether it is called e.g. atan or arctan. You also want the domains and definitions to be consistent as well though.

In the case of comb I think that means it should be called binomial although there is still the question of the negative integer convention. The math and gmpy2 decision to require nonnegative k might seem like it dodges that question but really it does not because it just creates a different kind of incompatibility that the different implementations of seemingly the same functions don’t have the same domains.

The case of atan vs arctan has to some extent been resolved as part of the array API since I think NumPy (and possibly imitators) was the only part of the Python ecosystem that used arctan and they have now all agreed to provide atan. It would be good to have some more standardisation/coordination around these things for functions that go beyond the current array API but I don’t think that choosing any one particular library like gmpy2 (or the stdlib or NumPy or …) is a good basis for defining any conventions.

5 Likes

That’s why I said about interfaces.

Yes, it wraps a lot of libraries, not just the GMP (but also MPFR, MPC). The python-gmp package might be more relevant just as the GMP wrapper.

This seems to be an arbitrary restriction, motivated by particular application domain (combinatorics).

I think this is less important than (1) common interfaces and (2) domains.

1 Like

Just a kindly reminder for voters, I’ll close poll in 1day.

The Array API that @NeilGirdhar mentioned covers many of the popular mathematically-oriented Python libraries such as numpy, pytorch, tensorflow, jax, etc. As far as I’m concerned, that sounds like relevant and useful information here, and might even be worth mentioning in the PEP.

2 Likes

Although I think it’s worthwhile to mention its history: __index__ exists because of numpy. numpy’s Travis Oliphant wrote the PEP (357) introducing it.

13 Likes

PEP updated to address review comments, some changes:

  • submodule variant added
  • new Rationale section
  • Motivation section expanded
  • module renamed to intmath
  • disbanded “Open Issues“ section

Sorry for a delay.

4 Likes

As a variant, the new module can be added as a submodule of the
:external+py3.14:mod:math: integer (most preferred), discrete
or ntheory.

I talked about this PEP at the sprint with @vstinner , and I think it would be fine to adopt the new conventions proposed for profilers (PEP 799), concurrent (PEP 734), and compression (PEP 784) since new modules are following this way.

2 Likes

I would like to separate idea of math’s splitting and new module naming.

Later seems much less important and I think that SC should just do final decision here. How to make this more clear, before submitting PEP? Restore “Open Issues” section with variants of naming? Mention submodule alternative in the Specification?

3 Likes

FYI, I’m not clear on what’s being proposed here. It the SC wants a new submodule of math, that’s fine by me. Getting some new coherent namespace is 95+% of what’s wanted. The new submodule’s name then is of at most minor importance, just so long as we don’t end up with gibberish like math.concurrent :wink:.

1 Like

This seems to be the case, looking on mentioned PEP’s (profiling.tracing, profiling.sampling, concurrent.interpreters and compression.zstd).

Sure. I’m going with reverting back “Open Issues” section for module naming.

Edit: PEP 791: final edits by skirpichev · Pull Request #4586 · python/peps · GitHub

I think that SC should take into account just proposed variants. (Though, may ignore people preferences.)

1 Like

The SC won’t show its colors until forced to by the submission of a specific proposal in the form of a PEP that has been discussed and revised etc. and submitted to the SC. If you don’t know what they want namespace-wise, just propose what you want and see if the SC has feedback. They will ask to revise the PEP if they have an issue like this, not reject it.

7 Likes

This topic was brought back to life by @corona10’s post, which appeared to suggest a new submodule of math. Since he’s on the SC now, his opinion matters more than mine :wink:, although he didn’t claim to be speaking for the SC. But it’s hardly a speculative leap to imagine that he posted because his belief is that a new top-level name wouldn’t fly with the SC. Which wouldn’t be speaking for the SC, but his is the most informed guess on the table.

1 Like

Regardless of whether this module is needed or not (IMO, it’s not and we can simply add/keep the functions to the math module), I don’t think we should add more nested namespaces to the stdlib. Flat is better than nested, after all.

The solution to addressing the top-level namespace collision issues is putting the stdlib into a new std package. I’ve started to work on a PEP for this.

12 Likes

There was some feedback at the Language Summit this year that SC members don’t seem very involved in discussions at DPO. As for this thread, @guido pointed out that my comment was just personal feedback, not an official SC opinion. I’m only one of five, and I’m a bit concerned about whether leaving comments is appropriate, since it may naturally be perceived as influencing someone’s draft PEP.

4 Likes

Please do!

Understood.

Which, for us, supplies infinitely more info than hearing from 0 of 5 :wink:.

All opinions will influence a draft PEP.. People are elected to the SC largely on the basis that their technical perspectives have earned respect, and IMO it’s a loss to the community for any of you to stay out of tech discussions due to overblown concern over “undue influence”.

Will your opinions carry more weight? Sure, probably so. And they should. It was certainly part of Guido’s old BDFL job (and of mine, as his oracle) not just to give final pronouncements, but to actively shape and guide discussions. Proactive collaboration and cooperation. Also to kill off ideas that were dead on arrival, to spare everyone’s time, effort, and attention from pursuing futile crusades.

Say what you like! Unless you say you’re speaking for the SC, you’re speaking only for yourself. Wthholding your perspectives certainly won’t improve a proposal.

32 Likes

100% agreed. As PEP delegate for packaging, I consider it a crucial part of my role to get involved in discussions, and guide the debate towards a conclusion that I would be willing to accept. A lot of work goes into a PEP, and I think it’s important to ensure that work has the best possible chance of success.

I do the same for non-packaging discussions, although obviously my authority is substantially less in those cases.

24 Likes

JFR, PEP has been submitted to SC.

11 Likes

Thank you Sergey, for proposing PEP 791 (intmath – module for integer-specific mathematics functions).

The Steering Council recently discussed the PEP, and while we’re mildly favorable, we want to provide some feedback and ask some questions first.

Let’s take the bike out of the shed first: The SC prefers a submodule rather than a new top-level module, and prefers math.integer, which under the Open Issues in the PEP is also listed as the “most preferred”. The Zen of Python enlightens us with “flat is better than nested”, but we recognize that there are times where some nesting is useful. We have a growing precedent of recommending some strategic, useful, and obvious (to us!) nesting such as putting interpreters in the concurrent package, and adding a compression package which contains the new zstd module. Even though the stdlib has top-level modules math and cmath we think adding intmath (or any of the other top-level names suggested in the Open Issues section) muddies the top-level namespace, thus our preference for a submodule.

We question whether the rationale against “fixing the math module documentation instead” is really that compelling, and we wonder whether most of the problems described by this PEP actually could be fixed with documentation. Fixing the problem with documentation also eliminates the need for a 3rd party package for backward compatibility. The PEP specifically defers consideration of new integer-only functions, but if future-proofing is a rationale for this PEP, it should explicitly say so, and share the plans for adding new functions in the future, even if it doesn’t commit to doing so.

We do suggest a more comprehensive, broader look at the organization of the math-related modules and functions, but such can be addressed in a future PEP if appropriate.

-Barry (on behalf of the Steering Council)

11 Likes

Thanks for the feedback. I’ll try to answer, corrections welcome.

First, an easy part:

This option was left on discretion of the SC. The real issue is a separate namespace (lack of it currently), not the module naming.

So, I’m fine if SC will pick some module name from presented or just decide whether it should be a submodule and leave concrete naming decision on the math module maintainers (if any).

Now the rest:

First, PEP mentioned issues, that can’t be fixed by documentation at all: i.e. we can’t make IDE completions more useful in this way — the math namespace just too polluted by unrelated functions.

The rest, well, probably could be solved given no limits in verbosity for sphinx docs and docstrings. As there is no common interface across the module (accepted types, returned values), no common expectations on functions behavior (whether it’s exact or not) — all that should be mentioned in documentation, including docstrings.

I should emphasize again, that there is no compatibility break per se. New package will help people to use new naming scheme in old Python versions. But nothing will be broken, if people will keep using math.gcd() and so on.

The Motivation section has some proposals for new functions, I can add more.

There is a lot of number-theoretic functions, that clearly fit to the Specification. Initially I though we can provide some boundary, a rough guide on which functions can be eventually included and which clearly can’t. But I don’t think it’s easy. Rather, each addition should be discussed on case-by-case basis (without a dedicated PEP, just like before).

Some rough boundary listed in the Possible Extensions section: probably we don’t want to turn Python into a number-theoretical library. So, if an efficient (to be practically useful) implementation for function is too complex — the stdlib is not a right place. But e.g. this depends on implementation details of the CPython. If it’s eventually will use some external library for arbitrary precision arithmetic, like the GNU GMP — more functions may be provided as free lunch.

Clearly, new functions could be added to the current math namespace, as it happened before. So, I don’t think that future-proofing is a rationale for this PEP. (The math module already has enough integer-related functions to warrant a separate namespace.) Though, new module might simplify this a bit: new functions will share same application domain. Just one reason less to reject it (i.e. “not present in the math.h”, “not related to floating-point arithmetic”, etc).

Edit: In PEP 791: edits (address SC feedback) by skirpichev · Pull Request #4639 · python/peps · GitHub I’ll try to address SC feedback.

3 Likes

A reminder that this seemingly eternal discussion was reignited by a simple and frequent request, to supply a uniform way to spell “ceiling division” for integers. Trivial for CPython to implement, but frequently implemented incorrectly by users. There are subtleties, which are shallow to “math people” but not to end users. Anyone, e.g, who has ever needed to account for alignment padding in structs faces the basic problem. CPython itself uses more than one approach internally, noted for obscurity. Implementations that are both correct and efficient are “clever”.

math.ceil is absolutely not a solution. It “sounds like it should be”, which makes it an attractive nuisance. It’s a float function, and appears to handle integer arguments correctly only by frequent accident.

Much the same could be said for math.floor, which somehow became an incoherent mess of not only not performing int floor division, but neither implementing what the libm function of the same name does (although all Pythons < Python 3 did, and Python no longer supplies an efficient way to get at libm’s floor’s useful functionality).

The people who have contributed to mathmodule.c in significant ways appear all to be behind creating a new namespace for the “number theoretic” functions. And have been for years already. The name is the least of our concerns.

@malemburg would like to put everything under a new catch-all reserved-name “std” module so that we’d never have to argue about stomping on user-space names again. I’m +0.5 on that idea myself, but it’s “perfect is the enemy of the good” here & now.

There are many dozens of number theoretic functions that could be added to a new module. But nobody wants to open a floodgate here. Trust that we (“math people”) have some common sense, and aren’t looking to make busy work for ourselves. We want to address user concerns, which we hear about more than anyone else. Start with ceildiv(i, j).

9 Likes