Serial read with python

Hi,

Firstly I’m new with python. I’m trying to read values on the serial line to plot a graph. In my serial monitor putty everything is in the right way printed on the serial line.

But when I’m trying to read it with python it adds an character and because of this the graph can not be plotted.
image

Can someone help me out where this character comes from and how I can get rid of it?

Kind regards,

JP

It add an \x but don’t filters it with data.decode()

Welcome, JP!

We will be able to help you better if you post your code and data as characters (no screen images).

Post code and data between backtics like this:

```
<code here>
```


Yes, it appears that Python is not decoding the \x00 byte. However, the problem is probably not with Python.

Serial communication is simple, but also complex and not easy.

What is your programming experience before Python?

Have you worked with serial communication before?

It looks like your port settings might be wrong. Do you know what settings your serial connection uses? It appears that your STOPBITS are too small or BYTESIZE is too large and Python is receiving a control byte along with the data bytes.

import serial

ser = serial.Serial(
    port='/COM4',
    baudrate=57600,
    parity=serial.PARITY_ODD,
    stopbits=serial.STOPBITS_TWO,
    bytesize=serial.SEVENBITS
)

while(True):
    print(ser.readline())

ser.close()

Hi Leland,

I’m not familiar with python, but I can do things with C. I use an nrf52840 to send adc values over ble. This is the client side. I do receive the values with an ble dongle and try to read it to plot it in a graph. This is the python code I found online and give it a try:

import serial
import matplotlib.pyplot as plt

plt.ion()
fig=plt.figure()


i=0
x=list()
y=list()
i=0
ser = serial.Serial('COM7',115200)
ser.close()
ser.open()
while True:

    data = ser.readline()
    print(data.decode())
    #x.append(i)
    #y.append(data.decode())

    #plt.scatter(i, float(data.decode()))
    #i += 1
    #plt.show()
    #plt.pause(0.0001)  # Note this correction

Is this enough to understand the problem? Sorry for the missing code.

Kind regard,

JP

This is what I get:

b'0.03\r\n'
b'\x000.03\r\n'
b'\x000.03\r\n'
b'\x000.03\r\n'

The settings need to be specifically for your serial connection. It looks like you’re picking up an extra byte in your data that is actually part of the previous data packet. This is why the \x00 is not in the first data value.

I can’t help you find the correct settings. Sometimes you just have to try different settings until it works.

I usually write nested loops to auromatically try all possible settings. Even it it takes 40 minutes to write and run the code, it will still be faster than performing and tracking hundreds of manual tests.

Where does this go wrong now? Do I have to fix it on python or on my dongle software?

float(data.replace(b'\x00', b'').decode())

This worked for now, but I still don’t know where it exactly goes wrong?

I think it is practically impossible that wrong parameter settings of the serial port would cause it to read extra NUL characters (b'\x00', also called null character) and otherwise read correct data. It looks like your device sends null-terminated records. PuTTY and other terminals do not show NUL characters.

The following code should resolve the problem. Just add the serial port parameters (baudrate etc.) as needed, adjust the timeout and you will probably want to collect the received values into a list (your_list.append(value)).

import serial

with serial.Serial(
        port='COM4', timeout=5                     # set your parameters as needed, reading timeout 5 seconds
        ) as ser:
    while True:
        record = ser.read_until(expected=b'\x00')  # read null-terminated record
        if not record or record[-1] != 0:          # incomplete record read means timeout
            break
        value = float(record.rstrip(b'\r\n\x00'))  # remove CR, LF and NUL at the end
        print(value)                               # do whatever with the value

I tested that float() can convert directly from bytes, it treats the characters as ASCII and calling .decode() is not needed.[1]


  1. There is this open issue Explicitly mention bytes and other buffers in the documentation for float() · Issue #77748 · python/cpython · GitHub because documentation does not describe this possibility. ↩︎

You can adjust the settings at your dongle OR in your python program. They just need to be the same.

With serial connections, the main ‘best practice’ is to start with low speeds (like 9600) and get everything else working before increasing the speed.

html test

Error checking active only on the transmitting side will add a checksum to the data packet that is then mishandled on the receiving side. The checksum can be any number of bytes.

(I tried to warn everyone. :stuck_out_tongue_winking_eye: )

image

The '\x00' appears to be a two-byte checksum added by the sending device. @juniorprogrammer, what is your dongle talking to? That is the device that might be adding an error detection checksum. If the receiving device isn’t expecting a checksum in the data, it will treat the checksum as data. If the receiving device is expecting a checksum and it knows how many bytes are in the checksum, then the receiving device will strip out the checksum.

PuTTY is stripping off the extra two bytes after the \r\n. The PuTTY manual refers to detecting CRC errors but doesn’t say anything about enabling Error Checking, so it might be ‘always enabled’ internally. If your dongle can be set to use CRC (or whatever error checking the sending device is using), then your dongle will strip the extra byte (assuming it’s a checksum). There is also one more setting to check: Flow Control.

image

The serial.Serial() class has a flow control argument, so check the PuTTY ‘Flow Control’ setting above and put one of these in your ser = serial.Serial() statement to use the same flow control that PuTTY is using.

xonxoff=True,
rtscts=True,

THEN…  Get the serial settings of the transmitting (sending) device and look for error checking enabled. And make sure both sides of the serial link have the same settings–including the error checking.

Future Step: Once it’s talking and you’re receiving data, you might need to specify the encoding. It looks like you’re only sending basic numerals and a decimal point, though, so the encoding probably won’t be a factor.

#for example-
msg = ser.readline().decode('utf-8').rstrip()  #rstrip() drops trailing spaces.

Anyone reading this topic later on who IS sending data with an extended character set will definitely need to specify the encoding.

I do not agree with this hypothesis (which complicate things) :slight_smile:

  • b'\x00' is a single byte (it is a single character in bytes as well as in str), not two
  • '\x00' or we can write it as '\0'[1] is ASCII NUL character which is often used to terminate records. In C-language it is used to terminate strings. In Unix tools it is used to terminate records (for example find ... -print0). I am almost sure that it is this role of b'\x00' we see here.
  • We have always seen the byte b'\x00' when the characters before it were b'0.07\r\n' or b'0.03\r\n'. This is highly improbable for a checksum.
  • In a binary protocol[2] I would expect to see multiple non-ASCII characters like: field separators/headers, field lengths, padding bytes etc. This communication looks like a simple ASCII stream.

Yes, but this is premature. No flow-control problems have been seen in the examples of the communication. There we saw no lost characters or misinterpreted start-bits (which could mangle characters). I think the computer has no problems reading the characters at the speed the device sends them.


  1. Which is a little ambiguous! It must not be followed by a digit, for example '\052' == '*' ↩︎

  2. You expect a binary checksum - so binary protocol. ↩︎

1 Like

The extra data is coming from somewhere. Let’s focus on finding where it’s coming from.

As a foundation to that, both devices need to be set up the same–including error checking, whether it s checksum, CRC, or another method, and regardless of how many bytes the error checking might add (the number of bytes depends on the error checking method used).

To start fixing things without confirming settings is asking for something fundamental to be overlooked. It’s like skipping the first step in any troubleshooting procedure: “Is the device plugged in, does the receptacle it’s plugged into have power, etc.”