File handling issue

[GiggsB]

Hi,
I have this python script where I am getting data from microcontroller (MCU) and calculating the CRC value of the received data. As the data is received, I am storing it into a new file with the name as DATE.T.HOUR.MINUTE.SECOND. After all data is received (~1 MB data), I am calculating the CRC value and comparing this with what MCU has sent. If the value matches, then good....if it does not matches, I plan to overwrite the data on the same file.
Can someone help me with how can I overwrite the data on the same file? Thanks.

import datetime 
import os 
import struct 
import time 
import pigpio 
import spidev 
import zlib

bus = 0
device = 0
spi = spidev.SpiDev()
spi.open(bus, device)
spi.max_speed_hz = 4000000
spi.mode = 0
pi.set_mode(25, pigpio.INPUT)
rpi_crc=0
 
def output_file_path():
    return os.path.join(os.path.dirname(__file__),
               datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
 
def spi_process(gpio,level,tick):
    #print("Detected")
    data = bytes([0]*2048)
    crc_data = bytes([0]*4)
    spi.xfer2([0x02])
    rpi_crc=0
    with open(output_file_path(), 'w') as f:
        for x in range(392):
            recv = spi.xfer2(data)
            values = struct.unpack("<" +"I"*512, bytes(recv))
            rpi_crc = zlib.crc32(bytes(recv),rpi_crc)
            f.write("\n")
            f.write("\n".join([str(x) for x in values]))
        mcu_crc_bytes = spi.xfer2(crc_data)
        mcu_crc = struct.unpack("<"+"I"*1,bytes(mcu_crc_bytes))
        mcu_crc_int = int(''.join(map(str,mcu_crc)))
        if (rpi_crc != mcu_crc_int):
            spi.xfer([0x03])
            print("CRC did not match!!")
            
            #Some function to open the same file and overwrite!
            for x in range(392):
                recv = spi.xfer2(data)
                values = struct.unpack("<" +"I"*512, bytes(recv))
                f.write("\n")
                f.write("\n".join([str(x) for x in values]))

        else:
            print("CRC matched!!")
            spi.xfer([0x04])

input("Press Enter to start the process ")
spi.xfer2([0x01])

cb1=pi.callback(INTERRUPT_GPIO, pigpio.RISING_EDGE, spi_process)

while True:
    time.sleep(1)

[deanhystad]

You really can't overwrite data in a file. You can append to a file, but any other kind of modification is performed by rewriting the file.

To rewrite a file you just open it for writing ("w"). This resets the file to empty. If the file is already open for writing you can use truncate(). I've never seen truncate() used, but you can read about it here.

https://docs.python.org/3/library/io.html

[GiggsB]

.................
To rewrite a file you just open it for writing ("w"). This resets the file to empty. If the file is already open for writing you can use truncate(). I've never seen truncate() used, but you can read about it here.

Thanks,
I changed my code to something like this (change line 38-48) in original posted code:

if (rpi_crc != mcu_crc_int):
    spi.xfer([0x03])
    print("NOOOOOOOOOOOOOOOO")
    f.seek(0)
    #f.truncate()
    for x in range(392):
       recv = spi.xfer2(data)
       values = struct.unpack("<" +"I"*512, bytes(recv))
       f.write("\n")
       f.write("\n".join([str(x) for x in values]))

I have a doubt here, if I don't add the truncate and/or only seek(0), the size of the file doubles, basically the data is getting written after the last line where previously the data was stored. In both cases, if I only use seek or both truncate and seek, the size of the file remains same as previously.
I can't check the data for some reason if both produce same result but if anyone knows, would using either the seek only or both seek and truncate produce same results/do the same work?

Thanks.

[deanhystad]

How are you opening your file? If you open a file for writing (mode = "w") it automatically truncates the file to zero. If you open a file with the mode set to append ("a") or read+write ("r+") this does not happen. Nor does it happen if you don't close and reopen the file. That is what's happening in your case when you call this:

            #Some function to open the same file and overwrite!
            for x in range(392):
                recv = spi.xfer2(data)
                values = struct.unpack("<" +"I"*512, bytes(recv))
                f.write("\n")
                f.write("\n".join([str(x) for x in values]))

Thinking more about your problem, I would not open the file at all unless the CRC checks out. This should make it easier to write your program.

Oh, and instead of this:

            f.write("\n")
            f.write("\n".join([str(x) for x in values]))

Do this

f.writelines(values)

[GiggsB]

How are you opening your file?

Hi,
The file is already opened in "w" mode, I don't close it again or open it again. I have added my full code here:

import datetime 
import os 
import struct 
import time 
import pigpio 
import spidev 
import zlib

bus = 0
device = 0
spi = spidev.SpiDev()
spi.open(bus, device)
spi.max_speed_hz = 4000000
spi.mode = 0

pi.set_mode(25, pigpio.INPUT)

rpi_crc=0
 
def output_file_path():
    return os.path.join(os.path.dirname(__file__),
               datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
 
def spi_process(gpio,level,tick):
    #print("Detected")
    data = bytes([0]*2048)
    crc_data = bytes([0]*4)
    spi.xfer2([0x02])
    rpi_crc=0
    with open(output_file_path(), 'w') as f: #OPENED FILE HERE
        t1=datetime.datetime.now()
        for x in range(392):
            recv = spi.xfer2(data)
            values = struct.unpack("<" +"I"*512, bytes(recv))
            rpi_crc = zlib.crc32(bytes(recv),rpi_crc)
            f.write("\n")
            f.write("\n".join([str(x) for x in values]))
        mcu_crc_bytes = spi.xfer2(crc_data)
        mcu_crc = struct.unpack("<"+"I"*1,bytes(mcu_crc_bytes))
        mcu_crc_int = int(''.join(map(str,mcu_crc)))
        if (rpi_crc != mcu_crc_int):
            spi.xfer([0x03])
            f.seek(0) #FILE REMAINS OPENED
            #f.truncate()
            for x in range(392):
                recv = spi.xfer2(data)
                values = struct.unpack("<" +"I"*512, bytes(recv))
                f.write("\n")
                f.write("\n".join([str(x) for x in values]))
        else:
            spi.xfer([0x04])
        t2=datetime.datetime.now()
        print(t2-t1)

input("Press Enter to start the process ")
spi.xfer2([0x01])

cb1=pi.callback(INTERRUPT_GPIO, pigpio.RISING_EDGE, spi_process)

while True:
    time.sleep(1)

So based on what you mentioned, I don't need to write the f.truncate().Right?
And thanks for the suggestions, I will implement in in my code and check it.