Int.to_bytes returning as many bytes as necessary

Python Help
1

I have an int. I want it represented as bytes — as many bytes as are needed to represent it. (Yes, the integer is nonnegative and I want an unsigned bytes representation.)

So far as I can see the only sensible way to achieve that right now is:

b = i.to_bytes( (i.bit_length()+7)//8 )

This feels surprisingly clunky for what I’d expect to be a relatively common operation. Have I overlooked an easier alternative?

3 
>>> import struct
>>> struct.pack('<q', 3)
b'\x03\x00\x00\x00\x00\x00\x00\x00'

This struct can be used to represent integers as bytes. You can specify endianness and the type.

4

@facelessuser What if the number is let’s say 3**40?

5

You’re going to need a bigger int :upside_down_face: . It’s possible I read things too fast and didn’t realize it was a request for an int representation of any size.

6

And this example explicitly included in docs. Maybe it would be more obvious with something like div(i.bit_length(), 8, rounding='up') as length.

Anyway, “workaround” seems to be clear. Though, it will not work for twos complement:

>>> n = -129
>>> l = (n.bit_length()+7)//8; l
1
>>> n.to_bytes(l, signed=True)
Traceback (most recent call last):
  File "<python-input-7>", line 1, in <module>
    n.to_bytes(l, signed=True)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^
OverflowError: int too big to convert

You have to decide: should length default be same when signed=True or not. So far your proposal is not complete.

Can you provide some justification for your expectations? Examples from popular projects, etc.
I would rather say, that more common are exports/imports from fixed-size integer formats.

Until v3.11 (see issue 89318), no defaults were provided for length/byteorder. Now we have default value for length. Changing that will introduce a backward incompatibility.

7

Just to check: you’re envisaging some new function, there, not talking about an existing one?

It feels like there are three points of friction here, and addressing any of them would improve things:

  • int.to_bytes() can’t give you as many bytes as needed
  • There is int.bit_length() but no int.byte_length()
  • There is no ceiling integer division function (i.e. your div(), above)

One possibility might be a special length value. Off the top of my head, -1 and None are plausible choices. Another would be a new function on int.

I did say, in my case, that “the integer is nonnegative and I want an unsigned bytes representation”. And I was asking a question rather than proposing a new feature. Notwithstanding that, I see two natural options:

  • Don’t support variable length for signed representations
  • For signed encodings, output sufficient bytes that there is at least one sign bit (if people want this, it would be good to provide an implementation as it looks like a major headache to get right unassisted)

(If some kind of int.to_var_bytes() function was provided it would be trivial to omit the signed argument.)

My use case happens to be encoding integers larger than 2**63 in an sqlite database as BLOBs. It could equally be useful for putting them in a websocket binary message, chunked data formats, etc.

8

Do you truly need “as many as are needed”, or can you put some sort of plausible limit on it?

>>> i.to_bytes(50).lstrip(b"\0")
b'\x07[\xcd\x15'

(Use rstrip if little-endian.)

9

Yes, we have no currently some methods for integer divisions with a different rounding (e.g. to nearest, round up, etc). But it’s a recurring proposal, see e.g. this thread.

My point is that such function could make obvious that ceiling rounding is happened here in integer division. That makes length calculation obvious too, isn’t?

One possibility might be a special length value.

And why this is better than an explicit length?

Don’t support variable length for signed representations

That will just complicate API for no good reasons.

My use case happens to be encoding integers larger than 2**63 in an sqlite database as BLOBs.

Methods on Python builtins are designed not just to fit your use case.

10

An interesting idea, though I worry about its efficiency. (I just tested and, though it’s hard to be precise, it seems to be about 10% slower for small numbers in small buffers, running up to 30-40% slower for large numbers in large buffers.

I guess you could employ some kind of doubling retry strategy to cope with arbitrarily large integers, but…

11

I’ve just noticed that PyLong_AsNativeBytes() was added to the C API in 3.13 , and seems to be a pretty thorough implementation that covers all the bases in this area.

It would form a lovely basis for both an API and implementation for a Python equivalent…

12

Fair enough, but if it comes to performance, my baseline would be this:

>>> i = 123456789
>>> while i:
...     i, r = divmod(i, 256)
...     base256.append(r)
...     
>>> bytes(reversed(base256))
b'\x07[\xcd\x15'

That way, ALL your options look fast!! :slight_smile: More seriously though, you could try using hex() and bytes.fromhex(), which may be easier.

13

…and by “more seriously”, you mean only three times slower than int.to_bytes() rather than 40 like the divmod scheme. :zany_face:

I did have the divmod idea and tried (then instantly abandoned) it, though for what it’s worth my take on it was a tiny bit quicker:

def b(i):
    while i:
        i,j = divmod(i, 256)
        yield j

def f3(i):
    return bytes(b(i))
14