Reading data from a binary buffer

I have a bytes buffer called data and need to read its content. It was created by a C++ program and sent over the network. Its layout is known, but a has a variable length. Here is an example:

print(data)
id = int.from_bytes(data[:4], byteorder='little')
w = int.from_bytes(data[4:8], byteorder='little')
h = int.from_bytes(data[8:12], byteorder='little')
a = np.frombuffer(data[12:], dtype='f4')
print(id, w, h, a)

It produces:

b'\x13\x00\x00\x00{\x00\x00\x00\xc8\x01\x00\x00ff\x86@ff\x06@\xcd\xcc\xfc@'
19 123 456 [4.2 2.1 7.9]

Is there a more elegant way of extracting its content without specifying exact positions of each variable, e.g. data[8:12], but specifying their order and individual size instead? Any other way this kind of task is usually done?

Have you tried the struct module from Python’s standard library?

You could do:

i, w, h = np.frombuffer(data[:12], dtype='i4')
a = np.frombuffer(data[12:], dtype='f4')
1 Like

I’ve seen it, but not sure how to handle array a with it.

Nice. Thank you.

>>> import struct
>>> data = b'\x13\x00\x00\x00{\x00\x00\x00\xc8\x01\x00\x00ff\x86@ff\x06@\xcd\xcc\xfc@'
>>> # The leading '<' means little-endian.
>>> struct.unpack('<iiifff', data)
(19, 123, 456, 4.199999809265137, 2.0999999046325684, 7.900000095367432)
>>> # Alternatively...
>>> struct.unpack('<3i3f', data)
(19, 123, 456, 4.199999809265137, 2.0999999046325684, 7.900000095367432)
>>>
1 Like

struct.unpack(‘<3i3f’, data)

Thank you, but how to handle a variable length array a?

The struct syntax only allows for fixed amounts of data. But you can mix and match:

a, b, c = struct.unpack('<3i', data)
arr = np.frombuffer(data[12:], dtype='f4')

If you need a NumPy array as the result for the variable-length data, then you’ll have to use a NumPy-provided unpacker (or else an array constructor) anyway.

1 Like