Binary File Read

[Aussie]

Hi, I have tried a number of examples to try and get the code below to step through correctly.
The header needs to be stepped through once and the sample code at the end to run through until there are no 4 bytes left.
At the moment the while does the whole lot over and over.
The header is 20 bytes long and consists of values made up of different byte sizes.
byte, int16 and int32.

Hope you can help.
Thanks,

import time
import struct

    # Header = 20 words
    # 32bit integer - Size
    # 1 byte - Variation
    # 1 byte - Object
    # Object  Variation
    #	30	1	32-bit integer samples
    #	30	2	16-bit integer samples
    #	30	3	32-bit integer samples
    #	30	4	16-bit integer samples
    #	30	5	32-bit floating point samples
    # 16bit unsigned integer - Point index, analog point index being sampled
    # 32bit integer - Start Time EPOCH
    # 32bit integer - Sample Rate, the interval (in seconds) between samples of the analog point
    # 32bit integer - Number of Samples, the number of sample records in this file
    # Size = 20 + (Number of Samples) * 4 - 16bit integer
    # Size = 20 + (Number of Samples) * 8 - 32bit integer

f = open("C:\Python\XXXXXXXXXXX", "rb")
byte = f.read(4)
  
while byte:
    
    #Step through once only
    #Header Section

    #Size 32 bit int, should be 484
    byte = f.read(4)
    Size = struct.unpack('>i', byte)[0] # As integere
    print('The size is ')
    print(Size)
    
    #Variation 1 byte, should be 1 to 5
    byte = f.read(1)
    Variation = int.from_bytes(byte, byteorder='big', signed=False)
    print('The variation is ')
    print(Variation)
     
    #Object 1 byte, should be 30
    byte = f.read(1)
    Object = int.from_bytes(byte, byteorder='big', signed=False)
    print('The object is ')
    print(Object)
    
    #Point Index 16 bit int
    byte = f.read(2)
    PointIndex = int.from_bytes(byte, byteorder='big', signed=False)
    print('The point index is ')
    print(PointIndex)
     
    #StartTime 32 bit int
    byte = f.read(4)
    temp = struct.unpack('>i', byte)[0] # As integer
    bin(struct.unpack('>i', byte)[0]) # As binary
    StartTime = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(struct.unpack('>i', byte)[0])) # as time
    print('The start time is ')
    print(StartTime)
     
    #Sample Rate 32 bit int
    byte = f.read(4)
    SampleRate = int.from_bytes(byte, byteorder='big', signed=False)
    print('The sample rate is ')
    print(SampleRate)
 
    #Number of Samples
    byte = f.read(4)
    NoOfSamples = int.from_bytes(byte, byteorder='big', signed=False)
    print('The number of samples is ')
    print(NoOfSamples)
    #End of header section


    #Do until end of file, until only 4 more bytes left
    #Sample(s)
    byte = f.read(4)
    Sample = struct.unpack('>i', byte)[0] # As integer
    print(Sample)    
    
f.close()

[bowlofred]

Please put your code in python tags. We can't tell the program flow without them.

Can you describe more what is happening or show the output? Right now I don't know what you mean by "the while does the whole lot over and over".

[Aussie]

Hi, I have put the code in Python tags, edited previous post.
The output is below.
The file has a header of 20 bytes. Broke down in various byte sizes.
One byte
Two bytes
and four bytes
The header provides setup data. The rest of the file is then sample values.

Thanks,


66
The object is
220
The point index is
2220
The start time is
1970-01-17 08:17:36
The sample rate is
1121716396
The number of samples is
1441792
1121749822
The size is
1507328
The variation is
66
The object is
219
The point index is
45472
...........
The start time is
2004-07-15 17:11:17
The sample rate is
7012352
The number of samples is
1090932632
7208960
The size is
1090642939
The variation is
0
The object is
111
The point index is
0
The start time is
2004-07-12 14:54:27
The sample rate is
7471104
The number of samples is
1089652637
7536640
The size is
1090687508
The variation is
0
The object is
0
The point index is
0
Traceback (most recent call last):
File "C:\Python\DNP3FileRead.py", line 55, in <module>
temp = struct.unpack('>i', byte)[0] # As integer
struct.error: unpack requires a buffer of 4 bytes
>>>

[deanhystad]

I think this is your header (in C).

unsigned int size;           // 4 bytes
unsigned char verification;  // 1 byte
unsigned char object;        // 1 byte
unsigned short pointindex;   // 2 bytes
unsigned int starttime;      // 4 bytes
unsigned int samplerate;     // 4 bytes
unsigned int samplecount;    // 4 bytes
                             // 20 bytes total

If this is correct, and if the header starts at the very top of the file, then your program has already messed up reading the file by the end of this:

f = open("C:\Python\XXXXXXXXXXX", "rb")
byte = f.read(4)

Why did you read 4 bytes? Isn't that supposed to be Size?

I did my best to mimic what I think you are trying to do. writedata is used to make a file that I can read. readdata is what you are trying to do. Maybe there should be a readheader function and a readsamples function. But I just wanted to show you that this kind of code can be really compact and it should't be confusing at all.

import struct
import time

def writedata(filename, data):
    '''Write data to a file with a header'''
    count = len(data)
    stime = int(time.time())
    with open(filename, "wb") as file:
        buffer = struct.pack('=IBBHiII', 484, 1, 30, 0, stime, 10, count)
        file.write(buffer)
        buffer = struct.pack(f'={count}i', *data)
        file.write(buffer)

def readdata(filename):
    '''Read data from a file.  Data count is in the header'''
    with open(filename, "rb") as file:
        buffer = file.read(20)
        size, vtype, x, index, stime, srate, count = struct.unpack('=IBBHIII', buffer)
        print(size, x, index, stime, srate, count)

        buffer = file.read(4 * count)
        values = struct.unpack(f'={count}I', buffer)
        return values

writedata('junk.bin', [1, 1, 2, 3, 5, 8, 13])
print(readdata('junk.bin'))

Output:
484 30 0 1598932280 10 7
(1, 1, 2, 3, 5, 8, 13)

I am assuming that the data in the file is in native format. If the file is a different format you can use '<' or '>' to fix any endian issues. If you are generating the file on your computer you should use '=' or '@'.

I did not use the vtype or variation argument. It would be easy to check the type and unpack 16 bit (h or H) or 32 bit (i or I or maybe f?) sample values.

[Aussie]

Hi, I changed it a little bit

def readdata(filename):
    '''Read data from a file.  Data count is in the header'''
    with open(filename, "rb") as file:
        buffer = file.read(20)
        size, variation, objectType, pointIndex, startTime, sampleRate, count = struct.unpack('>IBBHiII', buffer)
        print(size, variation, objectType, startTime, sampleRate, count)
         
        buffer = file.read(4 * count)
        values = struct.unpack(f'={count}I', buffer)
        return values

print(readdata('XXXXXXXXXXXXXXXX'))

[Aussie]

Hi, further to this I want to print to a csv file.
The code below does create a file and prints to it.

Though I want the "value" part of the writerow to an actual sample value.
And each sample value to have its own row. Kind of step through.
Would I use an array or list.
Thanks,

import struct
import time
import csv
    
# Get Data out of Binary File
def readdata(filename):
    '''Read data from a file.  Data count is in the header'''
    with open(filename, "rb") as file:
        buffer = file.read(20)
        size, variation, objectType, pointIndex, startTime, sampleRate, count = struct.unpack('>IBBHiII', buffer)
        print(size, variation, objectType, startTime, sampleRate, count)
         
        buffer = file.read(4 * count)
        values = struct.unpack(f'={count}I', buffer)
        return values

print(readdata('XXXXXXXXXXX'))

# Create and Write to CSV File
with open('XXXXXXXXXXX.csv', 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(["Blank1", "DateTime", "Blank2", "Value"])

[deanhystad]

New question, start a new topic.