Metasploit Graph and Append Array Difficulties

[Tuurbo46]

Hello,

So I an creating a graph to display ‘temp’ against ‘time’ reading data from a live temp module. I have created a brief example in notepad++ to confirm it works. I would like to display the temp curve over 60 minutes and append ‘channel_1_temp[]’ and ‘time_minutes_temp[]’ from the start of the test - so essentially the graph grows over time. Below are my notepad++ and the actually example from the live temp module.

The problem I am having with the ‘temp module example’, is the arrays don’t grow in size. They just stay at two elements. This is odd because the notepad++ example grew in size? Also looking at the below output, the ‘channel_1_temp [ 1 22]’ has a leading space at the start that seems to grow when left running?

I'm sure I am missing something simple but I cannot see what I am doing wrong. Or I have completely the wrong concept for what I am trying to do :-)

Many thanks,

Tuurbo46

Notepad++ example:

import numpy as np
import matplotlib.pyplot as plt

time_minutes = np.array([10, 20, 30, 40, 50]) # Minutes

temp_c = np.array([18, 20, 25, 25, 26])       # Celsius 

time_minutes = np.append(time_minutes, "60")  # add a time to the end of array
temp_c = np.append(temp_c, "30")              # add a temp to the end of array
  
plt.ylabel("C")
plt.xlabel("Minutes")

plt.plot(temp_c, time_minutes)
plt.show()

Temp module example called every 1 minute:

def temperature_graph():
    
    global time_minutes_temp
    global channel_1_temp
    global temp_chl_1
    global count
    
    count =+ 1 
        
    time_minutes_temp = np.array([1]) # Minutes
    
    channel_1_temp = np.array([1])    # Centigrade
    
    #---- Append time array by 1 minute ----
    time_minutes_temp = np.append(time_minutes_temp, [count]) 
 
    #---- Convert global temp float value to an int ----
    temp_chl_1_int = temp_chl_1   
    temp_chl_1_int = int(temp_chl_1_int)
    
    #---- Append temp array with latest temp value
    channel_1_temp = np.append(channel_1_temp, [temp_chl_1_int]) 
    
    print('time_minutes_temp', time_minutes_temp)
    
    print('channel_1_temp', channel_1_temp)
    
     
    plt.ylabel("C")
    plt.xlabel("Minutes")

    plt.plot(channel_1_temp, time_minutes_temp)
    plt.show()

Temp module output:

time_minutes_temp [1 1]

channel_1_temp [ 1 22]

time_minutes_temp [1 1]

channel_1_temp [ 1 22]

[deanhystad]

temperature_graph() should not create the array. It should only append values to an existing array. The arrays never grow longer than two values because each time you call temperature_graph() you throw away the old data.

I don't think this does what you think.
'

    plt.ylabel("C")
    plt.xlabel("Minutes")
 
    plt.plot(channel_1_temp, time_minutes_temp)
    plt.show()

This will draw a new plot each time it is called instead of updating an existing plot. show() blocks your program from running as long as the plot window is visable. Every minute you would have to close the plot window so a new plot could be drawn.

For plotting live data you might want to use FuncAnimation(). There are other ways to plot live data, but FuncAnimation is the easiest. The example below continuously updates the plot of a half Hz sine wave using FuncAnimation.

import math
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
from collections import deque


frequency = 1.0
UPDATE_INTERVAL = 50  # in milliseconds
DELTA_T = UPDATE_INTERVAL / 1000.0  # update interval in seconds
FREQUENCY = 0.5  # sinewave frequency in Hz
PLOT_PERIOD = 10.0  # graph width in seconds
PLOT_POINTS = int(PLOT_PERIOD / DELTA_T)  # Number of points displayed in plot window.
TO_RADIANS = FREQUENCY * 2 * math.pi  # Convert seconds to radians for sine function.

# Data storage (deque automatically discards old points)
x_data = deque(maxlen=PLOT_POINTS)
y_data = deque(maxlen=PLOT_POINTS)

# Create figure and axis
fig, ax = plt.subplots()
(line,) = ax.plot([], [])
ax.set_ylim(-1.0, 1.0)
ax.set_title("Continuous Animation with Limited Points")
ax.set_xlabel("Time")
ax.set_ylabel("Sine")


# Initialization function
def init():
    line.set_data([], [])
    return (line,)


# Update function for animation
def update(frame):
    # Append new data
    x = frame * DELTA_T
    x_data.append(x)
    y_data.append(math.sin(x * TO_RADIANS))

    # Shift x-axis to follow the latest points
    ax.set_xlim(x_data[0], x_data[0] + PLOT_PERIOD)

    # Update line data
    line.set_data(x_data, y_data)
    return (line, ax)


# Create animation
ani = FuncAnimation(fig, update, init_func=init, interval=UPDATE_INTERVAL, blit=False)

plt.show()

To display temperature data updated once a minute, you would set the update interval to 60000 (60 seconds). Instead of calling the sine function you would call a routine that gets the current temperature from your temp module. Set the plot period to how much data (how long a period in seconds) that you want displayed in the plot window. The routine will fill the window and then start scrolling old data off the left edge of the plot. You can set a frames count that will stop the animation when reached.

Read about FuncAnimation here:

https://matplotlib.org/stable/api/_as_ge...ation.html

It is easy to find examples online.

[Tuurbo46]

Thanks for your help.

Cheers,
Tuurbo46

[deanhystad]

If you want to include your plot in a GUI, some GUI toolkits have a "back end" that lets you embed a matplotlib plot in your program. The example below plots a function in a tkinter window. It has a button to start and stop the plot.

import tkinter as tk
import matplotlib.pyplot as plt
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
import math
from collections import deque


class Graph(tk.Frame):
    """Tkinter frame that displays matplotlib plot."""

    def __init__(self, parent, function, sample_rate=10, duration=5.0):
        super().__init__(parent)
        figure, self.axis = plt.subplots()
        self.line = self.axis.plot([], [])[0]
        self.axis.set_ylim(-1, 1)
        self.canvas = FigureCanvasTkAgg(figure, self)
        self.canvas.get_tk_widget().pack()
        self.function = function
        self.sample_period = 1.0 / sample_rate
        self.duration = duration
        count = int(duration * sample_rate)
        self.x = deque([0.0], maxlen=count)
        self.y = deque([0.0], maxlen=count)
        self.running = False

    def start(self):
        """Start the animation."""
        self.running = True
        self.animate()

    def stop(self):
        """Stop the animation."""
        self.running = False

    def animate(self):
        """Update the graph with new data."""
        if not self.running:
            return
        x = self.x[-1] + self.sample_period
        self.x.append(x)
        self.y.append(self.function(x))
        self.line.set_data(self.x, self.y)
        self.axis.set_xlim(self.x[0], self.x[0] + self.duration)
        self.canvas.draw()
        self.after(int(self.sample_period * 1000), self.animate)


class SinePlotter(tk.Tk):
    """Demonstrate using Graph frame to plot a sinewave."""

    def __init__(self, sample_rate=100, duration=5.0):
        super().__init__()
        self.graph = Graph(self, self.func, sample_rate, duration)
        self.graph.grid(row=0, column=0, padx=10, pady=(10, 0), sticky="nsew")
        self.button = tk.Button(self, text="Run", command=self.toggle)
        self.button.grid(row=1, column=0, padx=10, pady=(0, 10), sticky="ew")

    def func(self, x):
        """1/2 Hz sine wave."""
        return math.sin(x * math.pi)

    def toggle(self):
        """Start/stop the graph animation."""
        if self.graph.running:
            self.button["text"] = "Run"
            self.graph.stop()
        else:
            self.button["text"] = "Stop"
            self.graph.start()


SinePlotter().mainloop()

[deanhystad]

I just noticed this in your code.

time_minutes_temp = np.append(time_minutes_temp, [count]) 

It probably doesn't matter much when you add 1 temperature reading a minute, but numpy.append is horribly inefficient. I wrote a short program to time building a collection of 100,000 values. Using numpy.append took 3.4 seconds to run. Using a python list took 0.023 seconds. numpy takes longer because it creates a new array and copies the values from the existing array. As the existing array gets longer, so does the copy time, increasing at an exponential rate.

Even if you need numpy arrays for some data processing, I suggest storing the data in a list, creating a numpy array from the list when needed.

[Tuurbo46]

deanhystad

Thank you for all your previous comments.

I have a GUI that has 2 tabs (tab2 is the graphs tab that includes 4 graphs similar to below). However after running for 5 hours, tab2 becomes very laggy. If I click between tab1 and tab2, tab2 takes a few seconds to populate. I think this is due to my graphs using up too much memory.

So the below graph (4 in total on tab2) is called every 1 minute, displays 60 readings (60 minutes of data), then deletes all data, and starts from zero again (I dont know how to do a constantly shifting graph so I reset to zero at 60 minutes).

I know previously you mentioned to possibly stop using np.array as it is slow. Do you think this could be my laggy problem or I am not appending correctly?

Many thanks,

Tuurbo46

def temperature_graph():
    
    global graph_time_minutes
    global channel_1_temp, channel_2_temp, channel_3_temp
    global temp_chl_1, temp_chl_2, temp_chl_3
    global temp_graph_minute_count
    
    temp_graph_minute_count = temp_graph_minute_count + 1 

    #---- display first graph at 1 minute ----
    if(temp_graph_minute_count == 1):
        
        graph_time_minutes = 0
             
        channel_1_temp  = 0
        channel_2_temp  = 0
        channel_3_temp  = 0
            
        ax1.clear()
        
        ax1.set_ylabel("Celsius")
        ax1.set_xlabel("Minutes")
        
        ax1.plot(graph_time_minutes, channel_1_temp, label="T1", color='red')
        ax1.plot(graph_time_minutes, channel_2_temp, label="T2", color='blue')
        ax1.plot(graph_time_minutes, channel_3_temp, label="T3", color='green')
        
        
        ax1.set_xlim(0, 65) # minutes
        ax1.set_ylim(0, 65) # deg C
        
        ax1.legend(['T1', 'T2', 'T3'])
        
        fig_1.tight_layout()
        canvas_1.draw()
        
    
    else:
         
        graph_time_minutes = np.append(graph_time_minutes, [temp_graph_minute_count]) # append array with latest minute value
        
        temp_chl_1_int = temp_chl_1   
        temp_chl_1_int = int(temp_chl_1_int) # convert ch1 temp from a float to an int
        
        temp_chl_2_int = temp_chl_2   
        temp_chl_2_int = int(temp_chl_2_int) # convert ch2 temp from a float to an int
        
        temp_chl_3_int = temp_chl_3   
        temp_chl_3_int = int(temp_chl_3_int) # convert ch3 temp from a float to an int
        
        channel_1_temp  = np.append(channel_1_temp, [temp_chl_1_int]) # append temp array with latest temperature value
        channel_2_temp  = np.append(channel_2_temp, [temp_chl_2_int]) # append temp array with latest temperature value
        channel_3_temp  = np.append(channel_3_temp, [temp_chl_3_int]) # append temp array with latest temperature value
            
        ax1.clear()
        
        ax1.set_ylabel("Celsius")
        ax1.set_xlabel("Minutes")
                
        ax1.plot(graph_time_minutes, channel_1_temp, label="T1", color='red')
        ax1.plot(graph_time_minutes, channel_2_temp, label="T2", color='blue')
        ax1.plot(graph_time_minutes, channel_3_temp, label="T3", color='green')
        
        ax1.set_xlim(0, 65) # minutes
        ax1.set_ylim(0, 65) # deg C
        
        ax1.legend(['T1', 'T2', 'T3'])
        
        fig_1.tight_layout()
        canvas_1.draw()

    #---- after 60 minutes reset graph to zero ----
    if(temp_graph_minute_count == 60):

        temp_graph_minute_count = 0
        graph_time_minutes = 0
        channel_1_temp = 0
        channel_2_temp = 0
        channel_3_temp = 0     
        print(" ")
        print("temp_graph_minute_count --> ", temp_graph_minute_count)
        print("graph_time_minutes --> ", graph_time_minutes)
        print("channel_1_temp --> ", channel_1_temp)
        print("channel_2_temp --> ", channel_2_temp)
        print("channel_3_temp --> ", channel_3_temp)

[deanhystad]

5 hours is only 300 data points. Building up a numpy array didn't get slow until the array contains thousands of values. I still wouldn’t use numpy arrays. You see numpy arrays in online examples, but those examples are not plotting real-time data that grows over time, and some people mistakenly think numpy is always faster than regular python.

Look elsewhere for your problem, like where you make a new plot each minute instead of adding data to your existing plot. After 5 hours you have 900 plots with a total of 135,450 plotted points. I can see why that might cause things to slow down a bit.

This is my previous example modified to look more like your code. Notice plot is only called once per signal trace (3 times total) not once for each temperature reading.

import tkinter as tk
import matplotlib.pyplot as plt
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
import math
import threading
from collections import deque


class EventTask(threading.Thread):
    """Use threading.Event to make a task run periodically."""

    def __init__(self, period, function, *args, **kwargs):
        super().__init__()
        self.period = period
        self.function = function
        self.args = args
        self.kwargs = kwargs
        self.event = threading.Event()
        self.start()

    def stop(self):
        """Stop running the task."""
        self.event.set()

    def run(self):
        while self.event.wait(self.period) is False:
            self.function(*self.args, **self.kwargs)


def update_plot():
    """Add new point to the plot"""
    x = 0 if len(xdata) == 0 else xdata[-1] + 1
    t1data.append(math.sin(x / 12))
    t2data.append(math.sin(x / 6))
    t3data.append(math.sin(x / 4))
    xdata.append(x)
    lines[0].set_data(xdata, t1data)
    lines[1].set_data(xdata, t2data)
    lines[2].set_data(xdata, t3data)
    axis.set_xlim(xdata[0], xdata[0] + 60)
    canvas.draw()


def on_close():
    """Stop periodic tasks when the window is closed."""
    update_task.stop()
    root.destroy()


root = tk.Tk()
root.protocol("WM_DELETE_WINDOW", on_close)
xdata = deque(maxlen=60)
t1data = deque(maxlen=60)
t2data = deque(maxlen=60)
t3data = deque(maxlen=60)
figure, axis = plt.subplots()
axis.set_ylim(-1, 1)
lines = (
    axis.plot(xdata, t1data, color="red", label="T1")[0],
    axis.plot(xdata, t2data, color="blue", label="T2")[0],
    axis.plot(xdata, t3data, color="green", label="T3")[0],
)
axis.set_ylim(-1, 1)
axis.set_xlim(0, 60)
axis.legend()
axis.set_xlabel("Minutes")
axis.set_ylabel("Celcius")
canvas = FigureCanvasTkAgg(figure, root)
canvas.get_tk_widget().pack()
update_plot()
update_task = EventTask(0.1, update_plot)

root.mainloop()

I dont know how to do a constantly shifting graph

See use of deque above. Once you fill a deque to its maximum size, appending new values throws away the oldest value. In the example above the plot displays the 60 most recent values.

[Tuurbo46]

Hello,

Just so I understand what I am doing wrong. I have read you above comments, but I want to incrementally update my code so I understand. I have removed numpy arrays - prevously I was appending and creating a new array every 1 minute, and replaced with lists - that I append and update every 1 minute and then delete all data after 60 minutes?. After 5 hours I should not have this problem ---> 900 plots with a total of 135,450 plotted points, but an updated 180 plots with a total of 27,090 plotted points - that is deleted every 60 minutes and resets to 0 plots and 0 plotted points?. If my lists update is correct, I will move onto adding deque.

Thanks for your help.
Tuurbo46

def temperature_graph():
    
    global tp_graph_time_minutes
    global channel_1_temp, channel_2_temp, channel_3_temp
    global temp_chl_1, temp_chl_2, temp_chl_3
    global temp_graph_minute_count
        

    #---- display first graph at 1 minute ----
    if(temp_graph_minute_count == 0):
        
        tp_graph_time_minutes = [0]
        channel_1_temp  = [0]
        channel_2_temp  = [0]
        channel_3_temp  = [0]
            
        ax1.clear()
        
        ax1.set_ylabel("Celsius")
        ax1.set_xlabel("Minutes")
        
        ax1.plot(0, 0, label="T1", color='red')
        ax1.plot(0, 0, label="T2", color='blue')
        ax1.plot(0, 0, label="T3", color='green')
        
        
        ax1.set_xlim(0, 65) # minutes
        ax1.set_ylim(0, 65) # deg C
        
        ax1.legend(['T1', 'T2', 'T3'])
        
        fig_1.tight_layout()
        canvas_1.draw()
        
    
    else:
           
        tp_graph_time_minutes.append(temp_graph_minute_count) # append time array
        
        temp_chl_1_int = temp_chl_1   
        temp_chl_1_int = int(temp_chl_1_int) # convert ch1 temp from a float to an int
        
        temp_chl_2_int = temp_chl_2   
        temp_chl_2_int = int(temp_chl_2_int) # convert ch2 temp from a float to an int
        
        temp_chl_3_int = temp_chl_3   
        temp_chl_3_int = int(temp_chl_3_int) # convert ch3 temp from a float to an int
                
        channel_1_temp.append(temp_chl_1_int) # append temp list with latest temperature value
        channel_2_temp.append(temp_chl_2_int) # append temp list with latest temperature value
        channel_3_temp.append(temp_chl_3_int) # append temp list with latest temperature value
            
        ax1.clear()
        
        ax1.set_ylabel("Celsius")
        ax1.set_xlabel("Minutes")
                
        ax1.plot(tp_graph_time_minutes, channel_1_temp, label="T1", color='red')
        ax1.plot(tp_graph_time_minutes, channel_2_temp, label="T2", color='blue')
        ax1.plot(tp_graph_time_minutes, channel_3_temp, label="T3", color='green')
        
        ax1.set_xlim(0, 65) # minutes
        ax1.set_ylim(0, 65) # deg C
        
        ax1.legend(['T1', 'T2', 'T3'])
        
        fig_1.tight_layout()
        canvas_1.draw()

    #---- delete all graph data after 60 minutes ----
    if(temp_graph_minute_count == 60):
        
        del temp_graph_minute_count[:]
        del tp_graph_time_minutes[:]
        del channel_1_temp[:] 
        del channel_2_temp[:] 
        del channel_3_temp[:]     
    
    temp_graph_minute_count = temp_graph_minute_count + 1

[snippsat]

So the below graph (4 in total on tab2) is called every 1 minute, displays 60 readings (60 minutes of data), then deletes all data, and starts from zero again (I dont know how to do a constantly shifting graph so I reset to zero at 60 minutes).

I know previously you mentioned to possibly stop using np.array as it is slow. Do you think this could be my laggy problem or I am not appending correctly?

np.append() is inefficient here,but not the biggest problem.
ax1.clear() Big one clearing and redrawing the whole axes every update is expensive.
tight_layout() It should usually be called once when setting up the figure, not every minute.
canvas_1.draw() This forces an immediate full redraw.
So a lot of problems,and the use of global is not good,

Here’s a runnable example that run 4 graph,that removes all global usage and uses a class instead.
No to all of this, np.append(), No ax.clear() on every update, No legend() on every update, No tight_layout() on every update.

import tkinter as tk
from tkinter import ttk
from collections import deque
import random
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
from matplotlib.figure import Figure


class GraphPanel:
    def __init__(self, parent, title, y_label="Celsius", y_min=0, y_max=65):
        self.frame = ttk.LabelFrame(parent, text=title, padding=5)
        # Store only the last 60 readings
        self.times = deque(maxlen=60)
        self.ch1 = deque(maxlen=60)
        self.ch2 = deque(maxlen=60)
        self.ch3 = deque(maxlen=60)

        self.minute_count = 0
        self.fig = Figure(figsize=(5, 3), dpi=100)
        self.ax = self.fig.add_subplot(111)
        self.ax.set_xlabel("Minutes")
        self.ax.set_ylabel(y_label)
        self.ax.set_ylim(y_min, y_max)
        self.ax.set_xlim(0, 60)
        self.line1, = self.ax.plot([], [], color="red", label="T1")
        self.line2, = self.ax.plot([], [], color="blue", label="T2")
        self.line3, = self.ax.plot([], [], color="green", label="T3")
        self.ax.legend(loc="upper left")
        self.fig.tight_layout()

        self.canvas = FigureCanvasTkAgg(self.fig, master=self.frame)
        self.canvas.get_tk_widget().pack(fill="both", expand=True)

    def pack_grid(self, row, column):
        self.frame.grid(row=row, column=column, padx=5, pady=5, sticky="nsew")

    def update_graph(self, value1, value2, value3):
        self.minute_count += 1
        self.times.append(self.minute_count)
        self.ch1.append(int(value1))
        self.ch2.append(int(value2))
        self.ch3.append(int(value3))
        x = list(self.times)
        self.line1.set_data(x, list(self.ch1))
        self.line2.set_data(x, list(self.ch2))
        self.line3.set_data(x, list(self.ch3))

        # Moving 60-minute window
        if x:
            x_max = x[-1]
            x_min = max(0, x_max - 59)
            self.ax.set_xlim(x_min, x_min + 60)
        self.canvas.draw_idle()


class App(tk.Tk):
    UPDATE_MS = 1000   # demo speed; change to 60000 for real 1-minute updates
    def __init__(self):
        super().__init__()
        self.title("2 Tabs with 4 Live Graphs")
        self.geometry("1200x800")

        notebook = ttk.Notebook(self)
        notebook.pack(fill="both", expand=True)
        self.tab1 = ttk.Frame(notebook)
        self.tab2 = ttk.Frame(notebook)
        notebook.add(self.tab1, text="Tab 1")
        notebook.add(self.tab2, text="Tab 2 - Graphs")
        self._build_tab1()
        self._build_tab2()
        self.after(self.UPDATE_MS, self.refresh_graphs)

    def _build_tab1(self):
        ttk.Label(
            self.tab1,
            text="This is tab 1.\nSwitch to tab 2 to see the 4 live graphs.",
            font=("Arial", 14)
        ).pack(padx=20, pady=20)

    def _build_tab2(self):
        self.tab2.rowconfigure(0, weight=1)
        self.tab2.rowconfigure(1, weight=1)
        self.tab2.columnconfigure(0, weight=1)
        self.tab2.columnconfigure(1, weight=1)
        self.graph1 = GraphPanel(self.tab2, "Temperature Graph 1")
        self.graph2 = GraphPanel(self.tab2, "Temperature Graph 2")
        self.graph3 = GraphPanel(self.tab2, "Temperature Graph 3")
        self.graph4 = GraphPanel(self.tab2, "Temperature Graph 4")
        self.graph1.pack_grid(0, 0)
        self.graph2.pack_grid(0, 1)
        self.graph3.pack_grid(1, 0)
        self.graph4.pack_grid(1, 1)

    def read_temperatures(self, base):
        # Replace this with your real sensor readings
        t1 = base + random.randint(-3, 3)
        t2 = base + random.randint(-3, 3)
        t3 = base + random.randint(-3, 3)
        return t1, t2, t3

    def refresh_graphs(self):
        values = [
            self.read_temperatures(20),
            self.read_temperatures(30),
            self.read_temperatures(40),
            self.read_temperatures(50),
        ]
        self.graph1.update_graph(*values[0])
        self.graph2.update_graph(*values[1])
        self.graph3.update_graph(*values[2])
        self.graph4.update_graph(*values[3])
        self.after(self.UPDATE_MS, self.refresh_graphs)

if __name__ == "__main__":
    app = App()
    app.mainloop()

[deanhystad]

The main problem is you call plot each time you want to update the plot. Plot creates a new plot, it does not update an existing plot. Call plot once to create the plot, then use set_data to update the plotted data. Fix that first