time setup for realtime plotting of serial datas at high sampling rate

[alice93]

Hello,

I connect to the computer an Arduino 33 Nano BLE Sense with a sensor.
I would like to plot the datas with a sampling frequency of 256 Hz (eg 1 point every 3 ms) in real time and I am not sure how I should set up the 3 time parameters I have:
1) the delay in the Arduino program -> I guess it should be 3 ms or no delay
2) the timeout of the serial lfunction in python -> I thought it should be set at 0.003 s but I can't go below 0.1 without this error to occur :
File "c:\Users\lemarqua\Documents\python_arduino\plot IR\mesureIR_v1bis.py", line 62, in <module>
plt.scatter(i, float(data.decode()))
ValueError: could not convert string to float: ''
3) the time indicated in plt.pause() -> I don't really understand why this is needed

Here is my Arduino program:

#include <Wire.h>
#include "MAX30105.h"

MAX30105 particleSensor;

float ir;

void setup() {
  Serial.begin(115200);
  while (!Serial);

  // Initialize sensor
  if (!particleSensor.begin(Wire, I2C_SPEED_FAST)) //Use default I2C port, 400kHz speed
  {
    Serial.println("MAX30105 was not found. Please check wiring/power. ");
    while (1);
  }

    //Setup to sense a nice looking saw tooth on the plotter
  byte ledBrightness = 0x1F; //Options: 0=Off to 255=50mA
  byte sampleAverage = 8; //Options: 1, 2, 4, 8, 16, 32
  byte ledMode = 3; //Options: 1 = Red only, 2 = Red + IR, 3 = Red + IR + Green
  int sampleRate = 100; //Options: 50, 100, 200, 400, 800, 1000, 1600, 3200
  int pulseWidth = 411; //Options: 69, 118, 215, 411
  int adcRange = 4096; //Options: 2048, 4096, 8192, 16384

  particleSensor.setup(ledBrightness, sampleAverage, ledMode, sampleRate, pulseWidth, adcRange); //Configure sensor with these settings

}

void loop() {
  ir=particleSensor.getIR();
    Serial.println(ir); //Send raw data to plotter

  delay(3); // time in ms
}

And here the python one:

import serial
import matplotlib.pyplot as plt
import numpy as np
plt.ion()
fig=plt.figure()

nbPulse=3
tPause=0.1
tTimeout=0.003
xNumber=nbPulse/0.003/60

i=0
x=list()
y=list()
ser = serial.Serial('COM16',115200)
ser.timeout = tTimeout #specify timeout when using readline()
  
while True:

    print(i)
    data = ser.readline()
    print(data.decode())
    x.append(i)
    y.append(data.decode())

    if i>xNumber:
        plt.xlim(i-xNumber,i)
        x.pop(0)
        y.pop(0)
    plt.scatter(i, float(data.decode()))
    plt.show()
    plt.pause(tPause)  # time in second
    i += 1

Any help would be welcome !

[alice93]

Also if I include a line to update the y scale, the plotting is not carried out. Would anyone has an idea of why?

plt.ylim(min(y),max(y))

[Jeff_t]

I'm a complete noob at python and enjoying it, your project has the elements I enjoy most, being a noob I don't know how useful my help will be but I will give it a try.

The serial time out is to prevent the ser.readline from blocking and causing your program to hang. If readline does hang it will time out after tTimeout seconds and program execution will continue, something that should not occur too often. In this program I think a value of 20-40 mS would be adequate, therefore tTimeout = 40 should do.

plt.pause(tPause) is new to me but reading the docs appears to be a way of giving time to the GUI so that it can update etc. , a little like Visual Basics DoEvents() if you have ever come across that. A value of 100 mS does not seem bad and if anything I would probably drop it a little lower, perhaps something to tweek and observe.

The microcontroller transmitting at intervals of 3 mS is extremely fast, the serial can transmit/read and buffer the data ok but I am unsure whether the graphics can keep up with that pace for any length of time I would be interested to learn more on that. So while you are in the debugging stage perhaps you can drop the rate down to 500 mS, delay(500).

Lastly I have a modified version of your python code for you to look at, to help with speed a little I removed the print instructions but added another one, the print(ser.in_waiting). This will display the number of bytes accumulating in the serial buffer which if it gets too high will present a problem with buffer over run.

import serial
import matplotlib.pyplot as plt
import numpy as np

plt.ion()
fig=plt.figure()
 
nbPulse=3
tPause=0.1
tTimeout=0.40
xNumber=nbPulse/0.003/60
 
i=0
x=list()
y=list()

ser = serial.Serial('COM16',115200)
ser.timeout = tTimeout

   
while True:
 
  
    data = ser.readline().strip().decode("utf-8")

    print(ser.in_waiting)

    x.append(i)
    y.append(data)
 
    if i>xNumber:
        plt.xlim(i-xNumber,i)
        x.pop(0)
        y.pop(0)
   

    plt.scatter(i, data)
    plt.show()
   
    plt.pause(tPause)  # time in second
    i += 1

[alice93]

I'm a complete noob at python and enjoying it, your project has the elements I enjoy most, being a noob I don't know how useful my help will be but I will give it a try.

The serial time out is to prevent the ser.readline from blocking and causing your program to hang. If readline does hang it will time out after tTimeout seconds and program execution will continue, something that should not occur too often. In this program I think a value of 20-40 mS would be adequate, therefore tTimeout = 40 should do.

plt.pause(tPause) is new to me but reading the docs appears to be a way of giving time to the GUI so that it can update etc. , a little like Visual Basics DoEvents() if you have ever come across that. A value of 100 mS does not seem bad and if anything I would probably drop it a little lower, perhaps something to tweek and observe.

The microcontroller transmitting at intervals of 3 mS is extremely fast, the serial can transmit/read and buffer the data ok but I am unsure whether the graphics can keep up with that pace for any length of time I would be interested to learn more on that. So while you are in the debugging stage perhaps you can drop the rate down to 500 mS, delay(500).

Lastly I have a modified version of your python code for you to look at, to help with speed a little I removed the print instructions but added another one, the print(ser.in_waiting). This will display the number of bytes accumulating in the serial buffer which if it gets too high will present a problem with buffer over run.

import serial
import matplotlib.pyplot as plt
import numpy as np

plt.ion()
fig=plt.figure()
 
nbPulse=3
tPause=0.1
tTimeout=0.40
xNumber=nbPulse/0.003/60
 
i=0
x=list()
y=list()

ser = serial.Serial('COM16',115200)
ser.timeout = tTimeout

   
while True:
 
  
    data = ser.readline().strip().decode("utf-8")

    print(ser.in_waiting)

    x.append(i)
    y.append(data)
 
    if i>xNumber:
        plt.xlim(i-xNumber,i)
        x.pop(0)
        y.pop(0)
   

    plt.scatter(i, data)
    plt.show()
   
    plt.pause(tPause)  # time in second
    i += 1

Hello,

Thank you for your reply and sorry for the delay.

I think it is still not enough for real time, but it helps because now I am having the error message on my previous code and I don't get where it is coming from

Error:
File "c:\Users\lemarqua\Documents\python_arduino\serial\plot IR\mesureIR_v1_posted_forum.py", line 30, in <module>      
    plt.scatter(i, float(data.decode()))
ValueError: could not convert string to float: ''

I will check more in details your proposition and also think about an animation alternative.

Thanks again !

[bowlofred]

Looks like sometimes data is empty. That it's reading a blank line. When you feed an empty line to float(), it generates an exception.

You should either figure out what is causing the blank lines or skip out of the loop and retry if that's what is read.

[alice93]

Thank you, I added for now a

if len(data)!=0:

[deanhystad]

Your particle sensor sets the rate of data transfer. Your program should read data whenever it is made available. The only pause in your program should be the readline() function waiting for a newline.

The serial timeout is how long readline() will wait for the newline character. I think 1 second is fine for something like this. A long timeout is not going to slow your program down. All it does is reduce the number of nuisance timeouts.

The matplotlib.pause() function is used to create an animation. Let's say that you have 100 data points that you want to plot out over the period of 1 second. You would call plt.plot(x, y) and then plt.pause(0.01). This would plot an x, y point every 0.01 seconds until it plots all 100 x, y points.

You do not need to use animation. Your data will be arriving in real time and you want to plot the most recent data instead of collecting a bunch of data points and then running an animation to plot the results. You might want to use canvas.draw() and canvas_flush_events() as mentioned in this article.

https://www.delftstack.com/howto/matplot...atplotlib/

Note: The example contains a call to time.sleep(0.1). This is to simulate data arriving at 0.1 second intervals. Your serial readline() function will control how often the plot gets updated, so you don't need any kind of sleep/pause.

As mentioned by bowlofred, a serial timeout may/will result in something that cannot be converted to a float (probably an empty bytes). I would write the code to check the readline() result and only do the conversion if you get data. An easy way to do this is wrap an exception handler around the code in the update loop.

while True:
    try:
        y.append(float(ser.readline().decode()))  # This might raise an exception
    finally:
        # Only plot point if the above doesn't raise an exception
        while len(y) > xNumber:  # Throw away old points
            y.pop(0)
        # Update the plot