A previous draft had the user pass in a PyLongLayout to better ensure the format matches. That would be too expensive to check, so we switched to a single global layout. But in doing that, it’s now easy to skip checking whether the user “understands” the format.
I suggest bringing that check back, in a minimal form – a single “version” integer:
typedef struct PyLongLayout {
uint32_t version;
// Bits per digit
uint8_t bits_per_digit;
// Digit size in bytes
uint8_t digit_size;
} PyLongLayout;
// we drop the endiannes fields; we can bring them back in the future.
const PyLongLayout* PyLong_GetNativeLayout(uint32_t version);
And these functions would take a version:
int PyLong_AsDigitArray(
PyObject *obj,
PyLong_DigitArray *array,
uint32_t version);
PyLongWriter* PyLongWriter_Create(
uint32_t version,
int negative,
Py_ssize_t ndigits,
void **digits);
The versions would be:
- version 0, single hardcoded value: bits_per_digit is 30, and digit_size is 4.
- version 1, generalized: bits_per_digit and digit_size can have any value.
- future versions reserved for extensions – for example,
PyLong_GetNativeLayout could return a bigger struct PyLongLayout_v2 with something like endiannes, or two different formats and a cut-off point.
By passing 0, the user says they can only handle 30-bit digits. They don’t need any check beforehand, and they also don’t need a comprehensive test matrix. As in all cases, they should be prepared to fall back to PyLong_AsNativeBytes. (This option allows users to take shortcuts and still use the API correctly.)
By passing 1, the user says they’ve checked the PyLong_GetNativeLayout(1) result, and are prepared for Java’s ints, non-default CPython builds with 15-bit digits, etc.
By passing 2 or more, the user says they checked the PyLong_GetNativeLayout(ver) result, and it is compatible. (Note that the version in that result can still be 0 or 1!)
30-bit-digit CPython would allow any version; GraalPy would allow only 1+.
I’d rather make it part of runtime state, i.e. the pointer might not stay valid after finalization and reinitialization.
Proposed API are efficient for large integers. Compared to accessing directly Python internals, the proposed API can have a significant performance overhead on small integers.
I’d add:
The cut-off point for “small integers” is not exposed by this API, and it might change in future versions.
We suggest that extension authors either benchmark with target versons of Python implementations and choose suitable values themselves, or use version-specific API like PyUnstable_Long_IsCompact.