Reading data from a binary buffer

Python Help
1

I have a bytes buffer called data and need to read its content. It was created by a C++ program and sent over the network. Its layout is known, but a has a variable length. Here is an example:

print(data)
id = int.from_bytes(data[:4], byteorder='little')
w = int.from_bytes(data[4:8], byteorder='little')
h = int.from_bytes(data[8:12], byteorder='little')
a = np.frombuffer(data[12:], dtype='f4')
print(id, w, h, a)

It produces:

b'\x13\x00\x00\x00{\x00\x00\x00\xc8\x01\x00\x00ff\x86@ff\x06@\xcd\xcc\xfc@'
19 123 456 [4.2 2.1 7.9]

Is there a more elegant way of extracting its content without specifying exact positions of each variable, e.g. data[8:12], but specifying their order and individual size instead? Any other way this kind of task is usually done?

2

Have you tried the struct module from Python’s standard library?

3

You could do:

i, w, h = np.frombuffer(data[:12], dtype='i4')
a = np.frombuffer(data[12:], dtype='f4')
4

I’ve seen it, but not sure how to handle array a with it.

5

Nice. Thank you.

6 
>>> import struct
>>> data = b'\x13\x00\x00\x00{\x00\x00\x00\xc8\x01\x00\x00ff\x86@ff\x06@\xcd\xcc\xfc@'
>>> # The leading '<' means little-endian.
>>> struct.unpack('<iiifff', data)
(19, 123, 456, 4.199999809265137, 2.0999999046325684, 7.900000095367432)
>>> # Alternatively...
>>> struct.unpack('<3i3f', data)
(19, 123, 456, 4.199999809265137, 2.0999999046325684, 7.900000095367432)
>>>
7

struct.unpack(‘<3i3f’, data)

Thank you, but how to handle a variable length array a?

8

The struct syntax only allows for fixed amounts of data. But you can mix and match:

a, b, c = struct.unpack('<3i', data)
arr = np.frombuffer(data[12:], dtype='f4')

If you need a NumPy array as the result for the variable-length data, then you’ll have to use a NumPy-provided unpacker (or else an array constructor) anyway.