PySerial not sending string to serial port of Arduino

Python Help
1

Hello. I’m kind of new to Python and Arduino but I started to work recently with these two, so I need some help.

This is the scenario:
At work:
We are working with an Arduino Uno and a Rpi 4. They are conencted via USB cable and we connect to the Rpi 4 via work Wireless network.
We use Windows computers, so we are connecting to Rpi 4 via Putty.
The Arduino Uno has a code that is prepared to send a JSON object to its serial port when it catches a string “getData” on its serial port! So, my colleague asked me to write a script to automatically send this string and fetch the JSON object into a variable.
At work I was able to get the JSON object in Putty even if completely out of format, but I got it, using 2 separate scripts (goal is to use only one). I don’t have a screenshot of the result from work, though!

At home:
On my free time, I’m also trying to do the same at home, but I use Debian instead of Windows and therefore, I use minicom instead of Putty. Other than that, I just wrote a small program to try to mimic the program my colleague uploaded on Arudino at work. Meaning that if I send the string “getData” to Arduino serial port via minicom, I get the JSON object back at minicom.
Something like this:
image
We don’t see the string “getData” here because I deactivated local Echo in minicom.
If I type something else I just get a “Not detected” message.

So, the goal is, with a single script, to mimic this behavior. Send “getData” to Arduino serial, wait for the JSON object, fetch it into a variable and print it out to the terminal window.

So far, I was not able to reproduce this behaviour with a single script, so I tried to split the script in 2 so that I could run both at the same time. I don’t want to detail all the things I tried because the post will become huge and nobody will read it.

So, I have this code for now, in a single script:

import serial

ser = serial.Serial(
        port = '/dev/ttyUSB0',
        baudrate = 115200,
        timeout = 10,
        xonxoff = False
)
ser.reset_output_buffer()
ser.reset_input_buffer()
val = 0
command = ''
command = input("Enter command: ")
val = ser.write(command.encode(encoding = 'ascii', errors = 'strict'))
print("Waiting bytes: ", ser.in_waiting)
print("Bytes written: ", val)
in_data = ''
in_data = ser.read_until(b'}')
print(in_data)

This is the result, after the timeout of 10 secs:

$ python3 serialWriteRead.py 
Enter command: getData
Waiting bytes:  0
Bytes written:  7
b''

The problem is that I can’t even catch the “getData” string reaching the Rx line of Arduino on my scope (at home), using the script! I can’t also catch it if I use minicom where I can see the Tx (??) led blinking per each charater of the string “getData” that I type in minicom window. I can only catch the JSON object reply when using minicom, on my scope!
When I added the (??) above, I meant that I was expecting to see the string “getData” on the Rx line of the Arduino and the JSON object on the Tx line of the Arduino. What I see is, while using minicom, is the opposite: “getData” on Tx line and the JSON object on the Rx line. But while using the script above, I only see Tx line blinking. I assume is on the ser.write() command! No Rx line at all!

So, what can I do/change to try to make this work?
Thanks
Psy

2

This is the scenario:
At work:
We are working with an Arduino Uno and a Rpi 4. They are conencted via USB cable and we connect to the Rpi 4 via work Wireless network.
We use Windows computers, so we are connecting to Rpi 4 via Putty.

I’m assuming this is Windows-(wifi/ssh)->Rpi-(serial)->Arduino.

The Arduino Uno has a code that is prepared to send a JSON object to its serial port when it catches a string “getData” on its serial port! So, my colleague asked me to write a script to automatically send this string and fetch the JSON object into a variable.
At work I was able to get the JSON object in Putty even if completely out of format, but I got it, using 2 separate scripts (goal is to use only one). I don’t have a screenshot of the result from work, though!

We much prefer plain text inline to screenshots. You’re dealing with
text, so you can copy/paste from the putty window’s text. Just place it
here between triple backticks as you’ve doing with your Python code
(without the python annotication), eg:

 ```
 pasted output
 goes here
 ```

With that we can copy/paste the text ourselves, and people with visual
impairment can read it readily in their preferred form.

At home:
On my free time, I’m also trying to do the same at home, but I use Debian instead of Windows and therefore, I use minicom instead of Putty.

Wouldn’t it be neater to just ssh from your debian box to the Rpi? Or
are you using the Rpi’s serial console? Not very important though.

Other than that, I just wrote a small program to try to mimic the
program my colleague uploaded on Arudino at work. Meaning that if I
send the string “getData” to Arduino serial port via minicom, I get the
JSON object back at minicom.
Something like this:
image

Also, those of us on email don’t see these. I had to duck out to the
forum to see your screenshot, and the important bits are just text.

We don’t see the string “getData” here because I deactivated local Echo in minicom.
If I type something else I just get a “Not detected” message.

So, the goal is, with a single script, to mimic this behavior. Send “getData” to Arduino serial, wait for the JSON object, fetch it into a variable and print it out to the terminal window.

So far, I was not able to reproduce this behaviour with a single script, so I tried to split the script in 2 so that I could run both at the same time. I don’t want to detail all the things I tried because the post will become huge and nobody will read it.

So, I have this code for now, in a single script:

import serial

ser = serial.Serial(
       port = '/dev/ttyUSB0',
       baudrate = 115200,
       timeout = 10,
       xonxoff = False
)
ser.reset_output_buffer()
ser.reset_input_buffer()
val = 0
command = ''
command = input("Enter command: ")
val = ser.write(command.encode(encoding = 'ascii', errors = 'strict'))
print("Waiting bytes: ", ser.in_waiting)
print("Bytes written: ", val)
in_data = ''
in_data = ser.read_until(b'}')
print(in_data)

This is the result, after the timeout of 10 secs:

$ python3 serialWriteRead.py
Enter command: getData
Waiting bytes:  0
Bytes written:  7
b''

The problem is that I can’t even catch the “getData” string reaching the Rx line of Arduino on my scope (at home), using the script! I can’t also catch it if I use minicom where I can see the Tx (??) led blinking per each charater of the string “getData” that I type in minicom window. I can only catch the JSON object reply when using minicom, on my scope!
When I added the (??) above, I meant that I was expecting to see the
string “getData” on the Rx line of the Arduino and the JSON object on
the Tx line of the Arduino. What I see is, while using minicom, is
the opposite: “getData” on Tx line and the JSON object on the
Rx line. But while using the script above, I only see Tx line
blinking. I assume is on the ser.write() command! No Rx line at
all!

I’m assuming your setup is Debian-(serial)->Rpi-(serial)->Arduino.

You’re using minicom to talk to the Rpi, yes? I see in your screenshot
that minicom itself is using ttyUSB0. I assume that is your Debian
box’s ttyUSB0. But the implication is that at the Rpi end it is
connected to a serial port as well. Which one?

And the Arduino is presumably also connected to an Rpi serial port.

Your Rpi programme is opening ttyUSB0 expecting to talk to the Arduino
device. Is the Arduino connected to tyUSB0 on the Rpi? Can you check
that?

Might ttyUSB0 on the Rpi be the same device you’re using for the Deian
host?

The fact that you do not see the LEDs flickering on your scope suggests
that the Rpi programme is sending data to the wrong serial port,
something other than the port talking to the Arduino.

Can you check that? Can you experiment with other port devices on the
Rpi to see if they activate your scope’s LEDs?

Finally, I don’t see any login prompts etc in your screenshot. Are you
going Debian-(serial)->Arduino with no Rpi?

Cheers,
Cameron Simpson cs@cskk.id.au

3

When you’re using minicom or putty, you’re typing the command followed by end-of-line (LF or CRLF) and all that is being sent and it flushes the output.
When you write using ser.write, there’s no end-of-line and no explicit ser.flush() either.
I suspect that it’s all still in the output buffer.

4

Exacty that!

I sometimes use screenshots more because of the importance of showing the position of the mouse cursor after some action due to the differences that there are betwen Windows an Linux handling \r and \n characters. But ok, when I’m at work, I’ll use copy/paste. It’s just that sometimes is important to see where the cursor mouse is to understand if we have an extra \r or \n where it shouldn’t be.

You mean to be able to work on the Rpi 4 and the (work) Arduino instead of trying to mimic the situation at home? If it is this what you mean, I could do it but I would need to get the ssh keys to be able to login from home (we setup RSA key pairs to access the Rpi 4 at work because I think we had already 2 ransomware attempts on MongoDB we have in Rpi 4 before).

At home no, I’m using my laptop if it was the Rpi 4. So, setup is: laptop-Debian<->(USB|serial)<->Arduino.

From the previous reply, no. I use minicom to talk directly to my (home) Arduino. The only difference from home to work is the Wireless and Rpi 4. I’ll try to picture the 2 situations one next to the other:

Work:
Laptop → ssh to Rpi (via Wireless) → Arduino (via minicom) – I use this option
or
Laptop → ssh to Rpi (via Wireless) → Arduino (via Aruino Serial Monitor) – my colleague uses this option

Home
Laptop → Arduino (via minicom)

So, at home, my laptop is as if it was the Rpi 4 at work.
I’m not sure but I think Rpi 4 somehow sees Arduino on it’s ttyACM0 port and at home, my laptop sees my Arduino at ttyUSB0. I’m not sure this is a physical thing or a software level thing, because I use Debian at home and the Rpi 4 at work is using Ubuntu.

At this moment I can only test things at home. No Rpi 4 at home, unfortunately.

Yes, at home I’m not using Rpi 4. I’m really sorry, it’s clear I wasn’t clear enough in my opening post about the setups. I hope I was better now clearing out the scenarios!

Hum, so I need to explicitely send the \n along with the string? Like getData\n ? Is that what you mean?

Also, from the docs, the Serial.flushOutput() seems to have been renamed to serial.reset_output_buffer(). Am I wrong??? They say there Changed in version 3.0: renamed from flushOutput().

5

Well, peeps, I finally overcome all obstacles with the help of a friend and the guys in IRC channel! I was able to pick up all stuff in Rx and Tx lines, and get the entire JSON response as I wanted.

I had 2 problems. One was that at some point in time, I saved one of the scripts with the wrong name (serialWrte.py instead of serialWrite.py) and I was running a different file from what I was editing. The other problem was that I needed to explicitely send the \n char along with the getData string and I wasn’t doing so.
At this point I’m getting what I want!

Next Thursday I’ll test the script at work and I hope it works flawlessly despite the differences of the scenario, otherwise I’ll be back here.

Here is the single script working:

import serial

ser = serial.Serial(
        port = '/dev/ttyUSB0',
        baudrate = 115200,
        timeout = 10,
        xonxoff = False
)

val = 0
command = ''
command = input("Enter command: ")
command += str('\n')
val = ser.write(command.encode(encoding = 'ascii', errors = 'strict'))
print("Bytes written: ", val)
in_data = ''
in_data = ser.read_until(b'}')
print(in_data)

I know I was sugegsted to change the terminating char of the JSON object but at this point I’m not sure I can do that because I’m not the “owner” of the object. I don’t know if adding any other char at the end of the JSON object isn’t going to break somehting else. I’ll see when I have the chance to talk to my work colleague!

Thanks all for now!

1 Like
6

The .write method writes only the bytes that you give it. If you were writing a block of binary data, you wouldn’t want an additional 0x0A to be added at the end, would you? :slight_smile:
Only the print function adds a line ending.
The .flush() method that I mentioned will send the contents of the output buffer immediately; that’s different from the .reset_output_buffer() method that will discard the contents of the output buffer.

7

I’m not sure I understand why would I need the flush() command. What does flush() do that Serial.write() don’t?

Also, I thought you were referring to flushOutput(). This one was renamed recently, iiuc.

8

I’m not sure I understand why would I need the flush() command. What
does flush() do that Serial.write() don’t?

Nothing, if I’m reading the docs correctly:
https://pythonhosted.org/pyserial/pyserial_api.html#serial.Serial.write

It exists for compatibility with normal Python “filelike” objects, in
that you can pass a Serial object to functions expecting to write to
files. But a Serial object is effectively an unbuffered file in this
use.

But this interface lets you go, for exqmple:

 print("blah", file=serial)

Now, print() will just write what you tell it to a file (default
sys.stdout), and not flush. Flushing is at the discretion of the
file object itself. Data going to a regular file is typically flushed
(written) when the buffer fills, and a tty file such as your terminal
is typically set to flush when a newline is seen (or the buffer files).

The print() function s not asking to flush the buffer - the data might
land in the file’s buffer and go nowhere (yet). Calling .flush() on
the file forces the file to write the data now. print() has an
optional flush parameter, default False, to call flush() on the
file.

The Serial.flush() method exists so that people using it as a file can
call flush. But because the Serial object is unbuffered (the write
method really writes to the serial port) it has no effect. Other than to
no break code expecting to be able to call flush.

Also, I thought you were referring to flushOutput(). This one was
renamed recently, iiuc.

Yes, to the less confusingly named (and more conventionally snake_cased)
reset_output_buffer. reset_input_buffer and reset_output_buffer
exist to discard the contents of the input and output buffers, and the
latter aborts any pending write. I expect they’re to aid resetting your
state because you’ve lost track, been interrupted (eg by the user) or
are otherwise trying to start afresh with something.

Cheers,
Cameron Simpson cs@cskk.id.au

2 Likes
9

Yesterday after I get this working I still add another feature, which was to beautify the JSON object sent by Arduino, using json library, by adding the following line at the end of the code:

print(json.dumps(json.loads(in_data.decode()), indent = 4))

At home I was able to get the JSON object properly formatted.

Today, I brought my own Arduino clone (DCcEle DCcduino UNO) to work to make sure the code I wrote at home and worked, would also work here… Guess what? It doesn’t.

From a Windows machine, directly connected to my Arduino clone, the code that yesterday worked at home, doesn’t work at work, with my own Arduino.
This is the code:

import serial, json
 
ser = serial.Serial(
        port = '/dev/ttyUSB0',
        baudrate = 115200,
        timeout = 10,
        xonxoff = False
)
val = 0
command = ''
command = input("Enter command: ")
command += str('\n')
val = ser.write(command.encode(encoding = 'ascii', errors = 'strict'))
print("Bytes written: ", val)
in_data = ''
in_data = ser.read_until(b'}')
print(in_data)
print(json.dumps(json.loads(in_data.decode()), indent = 4))

I get this error:

>python serialWriteRead.py
Enter command: getData
Bytes written:  8
b'Detected{"Temperature": 25,"Pressure": 11,"Light": 1200}'
Traceback (most recent call last):
  File "C:\Users\cecol\Documents\Arduino\Arduino-Home\serialWriteRead.py", line 18, in <module>
    print(json.dumps(json.loads(in_data.decode()), indent = 4))
  File "C:\Users\cecol\AppData\Local\Programs\Python\Python310\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\cecol\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\cecol\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

If you guys can help me figure out what is wrong, I would appreciate!
Is this the common problem about the different way of Windows and Linux handling CR and LF chars?

10

Ok, nevermind peeps, it’s all working.
At work I didn’t know the baudrate on the Arduino was different from what I was using at home.
The script works and pretty prints the JSON object just perfectly!

Thanks every one
Psy

11

You might also want to try:

 from pprint import pprint
 pprint(json.loads(in_data.decode()))

which is also nicer to read. The output is Python syntax rather than
JSON, but sometimes that’s nice.

Cheers,
Cameron Simpson cs@cskk.id.au

2 Likes
12

Well, after all I need to make a small change and this change is breaking the script again.
Instead of requesting the user the command to send to serial port, I need to send it hardcoded!

So, I tried several changes but they all hang up the script:

import serial, json

ser = serial.Serial(
        port = '/dev/ttyACM0',
        baudrate = 9600,
        timeout = 10,
        xonxoff = False
)
val = 0
#command = ''
#command = input("Enter command: ")
#command = "getData"
#command += str('\n')
val = ser.write("getData".encode(encoding = 'ascii', errors = 'strict'))
print("Bytes written: ", val)
in_data = ''
in_data = ser.read_until(b'}')
print(json.dumps(json.loads(in_data.decode()), indent = 4))

I almost can bet that this is the missing or extra \n char or something related.

I also tried:

val = ser.write("getData\n".encode(encoding = 'ascii', errors = 'strict'))

val = ser.write('getData\n'.encode(encoding = 'ascii', errors = 'strict'))

val = ser.write('getData'.encode(encoding = 'ascii', errors = 'strict'))

and also tried to do something like:

command = ''
#command = input("Enter command: ")
command = "getData"
command += str('\n')
val = ser.write(command.encode(encoding = 'ascii', errors = 'strict'))
print("Bytes written: ", val)
in_data = ''
in_data = ser.read_until(b'}')
print(json.dumps(json.loads(in_data.decode()), indent = 4))

or,

command = ''
#command = input("Enter command: ")
command = 'getData'
command += str('\n')
val = ser.write(command.encode(encoding = 'ascii', errors = 'strict'))
print("Bytes written: ", val)
in_data = ''
in_data = ser.read_until(b'}')
print(json.dumps(json.loads(in_data.decode()), indent = 4))

But they all hang, and eventually throw out something like:

d$ python3 serialReadWrite.py
Bytes written:  8
Traceback (most recent call last):
  File "serialReadWrite.py", line 18, in <module>
    print(json.dumps(json.loads(in_data.decode()), indent = 4))
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

What am I missing’

13

These will both produce identical output as your previous user-entered command:

>>> command1 = "getData\n".encode(encoding = 'ascii', errors = 'strict')
>>> command1
b'getData\n'
>>> command2 = input("Enter command: ")
Enter command: getData
>>> command2 += str('\n')
>>> command2 = command2.encode(encoding = 'ascii', errors = 'strict')
b'getData\n'
>>> command1 == command2
True

Are you certain the script works if you take the command as user input instead of hardcoding it? I suspect something else is wrong.

14

Yes, I’m sure. I will paste this screenshot to show both the code running and the output:

1652×522 35.3 KB

15

Mysterious. Perhaps your terminal is adding extra characters to your input string? Could you add print(repr(command)) after the call to serial.write and run the user input version of the script again? What does it output?

16

The only difference I see is the b' thing indicating bytw, iirc.

1655×526 40.1 KB

The contents of command before sending it to the serial port is:

b'getData\n'

After I see:

'getData\n'
17

The contents of command do not change when you write it to the port. It’s always "getData\n". Encoding it turns it into bytes, which is represented as b'getData\n', but you do not store the encoded command.

Either way, that does not seem to have been the issue.

I’m stumped, to be honest. Some weird race condition maybe, where waiting for user input allows the serial bus time to set up properly but writing the hardcoded command immediately makes it hang? A long shot, but try importing time and add a short sleep before the write call.

18

Yeah, I already tried that with a 1s delay in place of the input() line of code.
I think I’ll try smaller delay!

19

Yeah, looks like that might be it: serial port - pySerial write() works fine in Python interpreter, but not Python script - Stack Overflow

Try 5 seconds or so.

2 Likes
20

The minimum was 1.8 seconds. And it worked!
Thanks @abessman

2 Likes