Proposal: int.from_bytes and int.to_bytes, but for buffers

Right now you have to take a slice before passing any data to int.from_bytes if you want to parse int at a certain offset. Which is extra allocations, even if it is something as cheap as memoryview. The same goes for int.to_bytes, it allocates bytes every time.

By having something like e.g. int.from_buffer(buffer, size, offset=0, byteorder='big', signed=False) and int.to_buffer(self, buffer, size, offset=0, byteorder='big', signed=False) it will make it possible to squish extra performance in parsers/serializers. int.from_bytes(data[offset:offset+size], ...) is rather common pattern, so there should be quite a lot of beneficiaries.

Altering existing methods does not look like a good idea to me, since current API for from_bytes assumes (essentially) slices (passing size argument would be awkward), and handling offset in both of them likely will harm the readability and/or performance.

Only concern is that this functionality highly overlaps with struct’s unpack_from/pack_into, but I think proposal has its own merit, since we do not have any built-in zero-allocation (except for int instance itself, ofc) alternatives to parse/serialize integers.

The zero-allocation goal is strongest if the new API takes a readonly buffer plus offset and length, rather than requiring a sliced view. That keeps int.from_bytes unchanged while making the parser use case explicit.

First thing to mind was that the JIT (and maybe T2 OPs? Idk) should be capable of easily and transparently optimize this out, but while that is still not the case, I see its utility and also the zen says explicit is always better than implicit.

I guess the metric will be if this feature outweighs its maintenance burden

Implementation-wise from_buffer/to_buffer will look much like from_bytes/to_bytes (minus bytes-specific stuff, plus offset validation). So in terms of maintenance, having two suspiciously-similarly looking pieces of code might not be so great.

Idk about JIT stuff, but I think something like Cython might benefit from methods you can pass e.g. unsigned char[:] directly.

With Python 3.15 int.from_bytes will get cheaper for these cases as I added Buffer protocol support (Add Buffer Protocol support to int.from_bytes · Issue #132108 · python/cpython · GitHub). In 3.14 and below everything other than an exact bytes was always copied to a bytes before being converted to an integer. Now if the object is a memoryview / supports buffer protocol it should go faster, so moving from direct byte slice to memoryview will likely improve this sort of case with 3.15.

If it is in a critical path for a big codebase I could see adding a new function here, but overall it’s not the direction I prefer for this sort of performance improvement. A new method means codebases need to be updated to use it. It needs unit tests. It needs to be maintained overtime to Python’s API stability guarantees. This feels like a case where if we iteratively improve Python’s JIT / smartness we should be able to make the existing usages run faster.

In particular, we know a lot about the code in the line, that the function called isn’t doing special introspection, and should be able to eliminate allocations with enough iteration. There’s a lot of features it would take to do that generally, but it makes every python program faster if we can do it.

4 Likes