Coverting a serial stream

Python Help
1

Hi,
I am trying to read a UART serial stream, and the data comes in as hex (I think). I’m not actually sure.

Below is a short snippet of what I am getting.

b’\t\x9c\x1d\xc8LX\t\xfdXh\t\xbf\x1e\x8a^X{|\xbe\x85’

I’m pretty sure the “” is a delimiter. And I believe the \n means new line, \t means tab. So those are ascii. I am also guessing that anything starting with \x is a hex value. But that’s where I get lost. Hex \x9c is convertible. But what is \xfdXh?

If anyone can help, or know if there is a python command to parse this, thanks in advance.

Andy

1 Like
2

You can view the bytes individually by looping through the byte-string. That should help you decipher what the byte-string contains.

b = b'\t\x9c\x1d\xc8LX\t\xfdXh\t\xbf\x1e\x8a^X{|\xbe\x85'
for c in b:
    print(c)
1 Like
3

Thank you.

4

I am trying to read a UART serial stream, and the data comes in as hex
(I think). I’m not actually sure.

Doesn’t look like it. But the bytes you’re receiving: some of them are
_rendered in hex when you print the bytes object.

Below is a short snippet of what I am getting.

b’\t\x9c\x1d\xc8LX\t\xfdXh\t\xbf\x1e\x8a^X{|\xbe\x85’

This is repr(some_bytes_object). That is printed as a valid Python bytes
literal, just as you’d compose one in a programme, in a string-like
format. The leading b’ says that it is a bytes literal “string”.

See this documentation:

https://docs.python.org/3/reference/lexical_analysis.html#index-18

including the escape sequence table.

I’m pretty sure the "" is a delimiter.

I don’t see a " above. The leading b’ says “bytes string” and the final
’ is the end of the string.

But remember that your bytes are only being printed in a string-like
way. They may well not be text of any kind.

And I believe the \n means new line, \t means tab. So those are ascii.

No.

Those are byte values which happen to correspond to some ASCII values. A
bytes literal is usually used by programmers when there’s composing
bytes that contain some text. But an arbittrary bytes object need not,
and nothing in your literal above looks very texty to me.

I am also guessing that anything starting with \x is a hex value. But
that’s where I get lost. Hex \x9c is convertible. But what is \xfdXh?

“\xfdXh” is “\xfd” then “X” then “h”. Or rather, byte values which can
be written that way.

There’s no such thing as a “hex value”, except that it is a byte value
written in hex because there’s no ASCII equivalent character.

However, these do not look like text to me. So you should consider their
values directly:

>>> list(b'\t\x9c\x1d\xc8LX\t\xfdXh\t\xbf\x1e\x8a^X{|\xbe\x85')
[9, 156, 29, 200, 76, 88, 9, 253, 88, 104, 9, 191, 30, 138, 94, 88, 123, 124, 190, 133]

If anyone can help, or know if there is a python command to parse this,
thanks in advance.

Parsing this data stream requires knowing the data format it is using,
which could be anyway. And since you’re reading from a UART, that
includes stuff like: reading the wrong serial protocol. If your bit rate
is wrong, or the characters per byte are wrong, or the stop or parity
bits are wrong, you could be reading complete garbage, let alone
decoding what should be coming in.

So:

  • are you sure you have the correct bit rate (eg 19200, or whatever),
    bits per byte, presence of start/stop/parity bits? These need to be
    correct to even get the correct binary data.
  • what else do you know about the source of the data and what should be
    in it?

Given those, you can start decoding the stream. But you need to answer
those questions first. Then start with the “struct” builtin module.
There are other things available if that is not enough.

Cheers,
Cameron Simpson cs@cskk.id.au

5

Thank you very much. You bring up a couple really good points. Unfortunately I know very little about the data I am getting, other than that it is coming from a solar charge controller.

6

Ok. Time to reverse engineer things. Make sure your serial speed/parity
etc is correct (matches whatever is documented or what is in use by some
proprietry thing if you’ve got one); that is the critical component -
without it you are trying to decode garbage.

After that I guess do things and see if the data start to become
recognisable. If nothing else, if you’re getting things in bursts,
particularly in response to something, you might infer that easy burst
is a complete “packet”, or perhaps a few. Which might aid recognising
the start of a regular piece of data, and so forth.

If you have a “packet”, you could start trying to use the struct module
to decode “standard” binary things, like floats and ints of various
sizes (try both big endian and little endian flavours) - what you know
about the solar system might let you hope to recognise likely things
because their values fall in expected ranges, eg an expected voltage, or
AC frequency etc.

Cheers,
Cameron Simpson cs@cskk.id.au

7

Cameron,
Just to fill you in a little better, I have a charge controller and a display. I am taking the signal that should go to the display and running it into python to figure out what it is. When I hook up the display, I get an idea of what values are present, and what they should be. So I’m not completely blind.

I have no idea of the baud rate, and I have no way of figuring that out. There is zero documentation for this thing. But I don’t think this is impossible. There’s not many rates to choose from. I would guess that it’s either 9600 or 19200. It’s possible that it’s a different rate, but I’ve seen few devices that don’t work at one of those two rates.

I think I can guess at a few configurations until I get a data stream that appears to be meaningful. I doubt that there are any floats. My guess, on something this inexpensive, is that anything with a decimal is just the base value x 10. So if the voltage is 13.7 volts, it will appear as 137. The decimal conversion probably occurs in the display.

Anyway, I very much appreciate your comments. They help me consider ways to solve this riddle.

Andy

8

Just to fill you in a little better, I have a charge controller and a
display. I am taking the signal that should go to the display and
running it into python to figure out what it is. When I hook up the
display, I get an idea of what values are present, and what they should
be. So I’m not completely blind.

Excellent.

I have no idea of the baud rate, and I have no way of figuring that out. There is zero documentation for this thing. But I don’t think this is impossible. There’s not many rates to choose from. I would guess that it’s either 9600 or 19200. It’s possible that it’s a different rate, but I’ve seen few devices that don’t work at one of those two rates.

Glad to hear it. In the terminal world 8 data bits, 1 or 0 stop bits and
no parity is pretty common. I have no idea about work practices in the
device world.

I think I can guess at a few configurations until I get a data stream that appears to be meaningful. I doubt that there are any floats. My guess, on something this inexpensive, is that anything with a decimal is just the base value x 10. So if the voltage is 13.7 volts, it will appear as 137. The decimal conversion probably occurs in the display.

Sounds more tractable than I’d thought. Also good. Who knows, maybe BCD
is possible too: 2 decimal digits packaged into a byte, each digit in
one 4 bit chunk. Eg 32 might be 0011 0010. Just to complicate your life.

Cheers,
Cameron Simpson cs@cskk.id.au

9

I figured I’d ask you this question because you seem like you know what you are talking about. I decoded the uart stream from the charge controller. I had the wrong baud rate, and once I got the right one, it was quite easy to decode. The character string is only 160 bytes.

Now the question is how do I write to this device? The charge controller has settings, and those settings are in a specific addressed place in memory. How do I write to these guys? Any ideas?

Andy

10

I figured I’d ask you this question because you seem like you know what
you are talking about.

Ha. Hahahaha.

I decoded the uart stream from the charge controller. I had the wrong
baud rate, and once I got the right one, it was quite easy to decode.

Glad to hear it. Nitpick: baud and bps are pretty much synonyms with
serial lines, but not with modems. I know we’re only dealing with a
serial line here, it just nags at me. bps is your data rate - bits per
second - band is state changes per second - on a modem you get multiple
bits per state. Not relevant really. Just propping up my fraudlent know
what I’m talking aboutness.

[…] Looks like wikipedia, again, has a good write up:

https://en.wikipedia.org/wiki/Symbol_rate

The character string is only 160 bytes.

And it was textual? Just curious.

Now the question is how do I write to this device? The charge
controller has settings, and those settings are in a specific addressed
place in memory. How do I write to these guys? Any ideas?

I have no idea. It isn’t a memory bus, so you’re not necessarily talking
with addresses.

Do you have to issue any kind of request to get your 160 byte packet? Or
does it arrive spontaneously?

But I genuinely have no idea. If there’s stuff you can meaningfully
change I imagine there is some protocol for saying “change this”. If you
make a request to get your data, are there variants on that to get
specific data (eg the setting you want to change)? If so, maybe there’s
a similar bigger request to write that setting?

This is all just speculation I’m afraid.

Happy to help play guesswork, but I expect that I don’t have any more
knowledge here about devices than you.

Cheers,
Cameron Simpson cs@cskk.id.au

11

So I overestimated you? I don’t think so.

Yes, the stream was textual. There were a few characters above decimal 127, and those were just checksum. The stream is sent once per second, without prompting. Once every minute, the stream is increased to 240 characters. Those additional characters occur at the end of the stream, and they are operating history.

What I have now is a charge controller and a display. The display reads out from the charge controller. This is the stream I fed into python. But the display also “writes” to the controller, because you can enter settings. I put “writes” in quotes because I am not certain this is how it works. But data must get transferred from the display to the controller. Somehow.

I suspect the display has a small amount of memory, and that’s where all user input gets stored. So maybe it doesn’t have to write anything to the controller. When a user enters something into the display, that value goes to a local memory register, and the entire register is scanned by the controller. I don’t know.

I don’t actually need the display. What I’m trying to do is to make a custom touch screen display. But in order to do so I have to figure out how the settings are getting to the controller. An address map would be ideal.

Any ideas now?

12

Well, serial lines are birectional. You’ve listened to the controller.
Can you listen to the display instead? Then operate it. You’ll not see
the effects on the controller but you can see what the display send, if
it sends.

Also, if that works: Do you have 2 serial ports by any chance?

If so, in principle you could put your machine between them. Read from
one and write to the other. You would also want to do the same in the
other direction.

You can do that in a few ways. You can poll each with nonblocking reads
or you can monitor them both with select/epoll and then read when data
are reported available. Or… you can do what I would do, which is run a
thread for each direction. It makes for much clearer code: each thread
is just a simple read, write to the other device (copy the data), print
(and decode, when you know the code).

Start with just reading from the controller on its own. If that’s
productive, if you’ve got 2 serial ports you could try being the go
between between the display and the controller.

Cheers,
Cameron Simpson cs@cskk.id.au