Posts: 7
Threads: 1
Joined: Jan 2023
Hi, I am very new to Python so I apologise if this is an obvious fix!
I have a script that was written a long time ago (not by me) so it is in Python 2. I have been working through the errors that are being caused by the update to Python 3, but there is one I am stuck on. The following line seems to be causing the issue:
t = struct.unpack(fmt, str) I have done some research and I understand this would be fine in Python 2, but it will not work in Python 3; thus it generates the error:
Error: TypeError: a bytes-like object is required, not 'str'
I did a bit more research and found that bytes() should be used in Python 3.
My question is how should it be used with the line in question? I have tried a few different ways and none of them seem to work. Here is the whole script if it helps (line 82 is the one causing the error):
#!/usr/bin/python -t
#
# vim:set ts=4 sw=4:
"Pickup file (.PIC) editor"
from pprint import pprint
import struct
name_table = {
0: 'Trojax'
,1: 'Pyrolite'
,2: 'Transpulse'
,3: 'SussGun'
,4: 'Laser'
,5: 'Mug'
,6: 'Mugs'
,7: 'Heatseaker'
,8: 'HeatseakerPickup'
,9: 'Thief'
,10: 'Scatter'
,11: 'Gravgon'
,12: 'Launcher'
,13: 'TitanStar'
,14: 'PurgePickup'
,15: 'PinePickup'
,16: 'QuantumPickup'
,17: 'SpiderPod'
,18: 'Parasite'
,19: 'Flare'
,20: 'GeneralAmmo'
,21: 'PyroliteAmmo'
,22: 'SussGunAmmo'
,23: 'PowerPod'
,24: 'Shield'
,25: 'Inv'
,26: 'ExtraLife'
,27: 'Computer'
,28: 'Smoke'
,29: 'Nitro'
,30: 'Goggles'
,31: 'Gold'
,32: 'Mantle'
,33: 'Crystal'
,34: 'Orb'
,35: 'GoldenPowerPod'
,36: 'DNA'
,37: 'SkeletonKey'
,38: 'Bomb'
,39: 'GoldFigure'
,40: 'Flag'
,41: 'Bounty'
,42: 'Flag1'
,43: 'Flag2'
,44: 'Flag3'
,45: 'Flag4'
}
class BinFile:
"File which allows for easier reading from binary files."
def __init__(self, source, mode='r'):
if type(source) == str:
self.source = open(source, mode, encoding='ANSI')
elif hasattr(source, 'read'):
self.source = source
else:
raise ValueError('Source must be filename or something that can read()')
self.close = self.source.close
self.read = self.source.read
self.seek = self.source.seek
self.tell = self.source.tell
self.write = self.source.write
def binread(self, fmt):
"Read little-endian binary data specified by fmt."
fmt = '<' + fmt
size = struct.calcsize(fmt)
str = self.read(size)
if len(str) < size:
raise EOFError('Unexpected end of file (%d bytes read instead of %d)' % (len(str), size))
t = struct.unpack(fmt, str)
if len(t) == 1:
return t[0]
else:
return t
def read(self, n):
return self.source.read(n)
def seek(self, offset, whence=None):
if whence is None:
return self.source.seek(offset)
else:
return self.source.seek(offset, whence)
def tell(self):
return self.source.tell()
def write(self, str):
return self.source.write(str)
class PickupList:
def __init__(self, source):
self.read_from(source)
def pprint(self):
print ('Version: %d') % self.version
print ('Number of pickups: %d') % self.num_pickups
print ('Pickups:')
pprint(self.pickups)
def read_from(self, source):
if type(source) == str:
source = BinFile(source, 'r')
elif not hasattr(source, 'binread'):
raise ValueError('Source must be filename or something that can binread()')
magic = source.binread('4s')
if magic != 'PRJX':
raise InvalidPICFileError('Invalid magic number ' + magic)
self.version, self.num_pickups = source.binread('Ih')
self.version = int(self.version)
self.pickups = []
for i in xrange(self.num_pickups):
pickup = {}
pickup['gen_type'], pickup['regen_type'], pickup['gen_delay'], \
pickup['life_span'] = source.binread('HHff')
pickup['pos'] = {}
pickup['pos']['x'], pickup['pos']['y'], pickup['pos']['z'] \
= source.binread('fff')
pickup['group'], pickup['type'], pickup['trigger_mod'] \
= source.binread('HHH')
self.pickups.append(pickup)
source.close()
def write_to(self, target):
if type(target) == str:
target = BinFile(target, 'w')
elif not hasattr(target, 'write'):
raise ValueError('Target must be a filename or something we can write() to')
target.write('PRJX')
target.write(struct.pack('Ih', self.version, self.num_pickups))
assert self.num_pickups == len(self.pickups)
for pickup in self.pickups:
target.write(struct.pack('HHff', pickup['gen_type'], \
pickup['regen_type'], pickup['gen_delay'], \
pickup['life_span']))
target.write(struct.pack('fff', pickup['pos']['x'], \
pickup['pos']['y'], pickup['pos']['z']))
target.write(struct.pack('HHH', pickup['group'], pickup['type'], \
pickup['trigger_mod']))
target.close()
class InvalidPICFileError(Exception):
def __init__(self, msg):
self.msg = msg
def __str__(self):
return self.msg
if __name__ == '__main__':
import sys
import yaml
if len(sys.argv) < 2:
print ('Convert PIC files to and from YAML')
print ('Usage: %s <-pictoyaml|-yamltopic> [source] [> destination]') % sys.argv[0]
sys.exit(0)
elif len(sys.argv) < 3:
source = BinFile(sys.stdin)
else:
source = BinFile(sys.argv[2])
if sys.argv[1] == '-pictoyaml':
pl = PickupList(source)
sys.stdout.write(yaml.dump(pl))
elif sys.argv[1] == '-yamltopic':
pl = yaml.load(source)
pl.write_to(sys.stdout)
else:
print ('first argument must be -pictoyaml or -yamltopic')
sys.exit(1)Thanks!
Posts: 1,602
Threads: 3
Joined: Mar 2020
str is the name of a class in python (and you use it that way in some places like line 63. But line 79 reassigns it to your string. Better would be to use a different name (but it's not causing your problems).
If you're truly reading binary data, why bother opening the file in non-binary mode with an ANSI encoding? Seems this would be much simpler to open in binary mode. Then you're reading bytes and they go straight into the struct.
You're having to deal with the fact that the str type changed between python2 and python3 and is now different than the bytes class.
When appropriate, you can encode a str to bytes, and you can decode bytes to str. Your struct call takes bytes only. So if you need the bytes that correspond to "PRJX" in UTF-8, you can do:
data = "PRJX" # might be from an input() or read() from file opened without binary mode.
buffer = data.encode() # I can't do an ANSI encoding as that is only available on windows.
print(type(data), type(buffer)) But reading the binary file and decoding it via ANSI to a str, then re-encoding it later to pass to struct is messy. I think it would be better to read and write the file in binary mode and just deal with the binary characters/bytes.
Posts: 7
Threads: 1
Joined: Jan 2023
Thank you for your reply! The input file is using ANSI encoding, so I added that part in, otherwise I was getting the UnicodeDecodeError (I presume if no encoding is specified it will default to UTF-8?)
I am a bit confused about "PRJX", as I understand it this is an 8 byte ID at the beginning of the file but I am not sure what this actually does. Initially I mistook this for the actual encoding of the file, then realised that "PRJX" doesn't exist as any documented kind of encoding. It seems it is just being used as a flag to check whether the input file is valid?
I think the below is the binary format, I don't know if this helps or if it is of no use to helping with this specific issue!
Pickup Locations (*.PIC):
Magic Number: uint32
Pic Version Number: uint32
NumRegenPoints: int16
{
gen_type: int16
regen_type: int16
gen_delay: float
life_span: float
pos: VECTOR
{
x: float
y: float
z: float
}
group: int16
Type: int16
TriggerMod: int16
}Sorry if this sounds like a stupid question, but if the file is read and written as Binary will it not be readable?
Posts: 7,431
Threads: 125
Joined: Sep 2016
Jan-04-2023, 07:38 PM
(This post was last modified: Jan-04-2023, 07:39 PM by snippsat.)
Change to.
t = struct.unpack(fmt, str.encode())
# or
t = struct.unpack(fmt, bytes(str, 'utf-8')) Don't use str as variable as Python use this word.
>>> str
<class 'str'>
Posts: 6,981
Threads: 22
Joined: Feb 2020
Jan-04-2023, 08:38 PM
(This post was last modified: Jan-04-2023, 08:38 PM by deanhystad.)
By "readable" do you mean "Can I look at it in an editor?" Editors try to display bytes as ASCII characters. If every byte in your binary file matches an ASCII character, the file will look like a text file when opened in an editor. if you are writing floats or ints to the file, they will look like garbage. As long as all you write to the file is text, a binary file is the same as a text file using 8 bit encoding.
Posts: 7
Threads: 1
Joined: Jan 2023
Yes, I believe the intended purpose of the script is to output in the YAML format so it is readable, but from your post I am guessing that even when it is working there will still be some bytes in the binary that will not be readable in an editor, even if the binary file structure is known?
The other posts did help get around my original problem, but now I get the following error:
Error: InvalidPICFileError: Invalid magic number PRJX
I thought this error would only show if the magic number was not equal to "PRJX":
if magic != 'PRJX':
raise InvalidPICFileError('Invalid magic number ' + magic.decode('utf-8'))I think this might be a little bit above me, essentially what I was hoping to do was to get this working, then work on other files with known binary structures and create scripts to convert them into YAML too.
Posts: 6,981
Threads: 22
Joined: Feb 2020
Jan-04-2023, 09:23 PM
(This post was last modified: Jan-04-2023, 09:23 PM by deanhystad.)
YAML is human readable. Any ints or floats will be converted to ascii characters when serialized by YAML. If you write YAML to a binary file it should be readable in an editor.
Magic must not equal 'PRJX'. Probably because magic is bytes and 'PRJX' is a str?
Posts: 7
Threads: 1
Joined: Jan 2023
Ah ok, I think I have misunderstood the purpose of the magic number. I thought the magic number was a way of identifying the file so a program knows the structure and how to read it, but this doesn't appear to be the case. There are many files I have which actually all start with the same magic number ("PRJX") but the structures are completely different.
In this case, is it actually the file extension that is important, not the magic number? The files all have different extensions as they are for different things.
Posts: 7
Threads: 1
Joined: Jan 2023
I have the below to read the file and output the hex/byte data:
import binascii
import hexdump
my_file = "vol2.pic"
with open(my_file, 'rb') as file_t:
# read the file as xxd do
blob_data = hexdump.hexdump(file_t, result='return')
print(blob_data)
with open(my_file, 'rb') as file_t:
blob_data = binascii.hexlify(file_t.read())
print(blob_data)
with open(my_file, 'rb') as file_t:
blob_data = bytearray(file_t.read())
print(blob_data)Which produces the below:
00000000: 50 52 4A 58 01 00 00 00 12 00 02 00 02 00 00 00 PRJX............
00000010: 00 00 00 00 80 BF 40 17 AC C5 19 F4 E7 44 E0 77 [email protected]
00000020: 8B C5 0F 00 1F 00 FF FF 02 00 02 00 00 00 00 00 ................
00000030: 00 00 80 BF 35 E9 AB C5 90 52 E8 44 75 08 B4 C5 ....5....R.Du...
00000040: 0E 00 1F 00 FF FF 02 00 02 00 00 00 00 00 00 00 ................
00000050: 80 BF 69 66 37 C5 0A 02 C0 44 CB FF 2F C5 24 00 ..if7....D../.$.
00000060: 08 00 FF FF 02 00 02 00 00 00 00 00 00 00 80 BF ................
00000070: F9 E9 BB C4 42 2B 12 45 64 0D 80 43 14 00 1F 00 ....B+.Ed..C....
00000080: FF FF 00 00 00 00 00 00 00 00 00 00 80 BF 2C 90 ..............,.
00000090: 89 C5 5B 8C F4 44 24 09 80 43 2A 00 03 00 FF FF ..[..D$..C*.....
000000A0: 00 00 02 00 00 00 00 00 00 00 80 BF 83 4A 1A C5 .............J..
000000B0: 83 F6 D2 44 A7 FC DF C4 25 00 17 00 0B 00 02 00 ...D....%.......
000000C0: 02 00 00 00 00 00 00 00 80 BF D2 D2 29 C5 A6 13 ............)...
000000D0: DA 44 D6 AC 8B 43 2B 00 1F 00 0F 00 02 00 02 00 .D...C+.........
000000E0: 00 00 00 00 00 00 80 BF 11 5B 01 C5 F1 A7 02 45 .........[.....E
000000F0: 62 0D 80 43 2C 00 1F 00 11 00 00 00 02 00 00 00 b..C,...........
00000100: 00 00 00 00 80 BF A3 BF 6E C4 96 69 0E 45 E6 FD ........n..i.E..
00000110: 9F C4 0B 00 14 00 FF FF 00 00 02 00 00 00 00 00 ................
00000120: 00 00 80 BF 00 00 00 44 A5 59 C6 44 95 4E 66 C5 .......D.Y.D.Nf.
00000130: 0C 00 08 00 FF FF 02 00 02 00 00 00 00 00 00 00 ................
00000140: 80 BF 55 08 53 C5 D5 AE D4 44 F4 FF 87 C5 22 00 ..U.S....D....".
00000150: 17 00 FF FF 00 00 02 00 00 00 00 00 00 00 80 BF ................
00000160: B7 0E C9 C4 DA B4 91 C5 51 D4 F9 45 2F 00 1F 00 ........Q..E/...
00000170: FF FF 01 00 02 00 00 00 00 00 00 00 80 BF 64 CF ..............d.
00000180: C5 C4 C2 5D AE C5 E0 3D 04 45 06 00 1F 00 FF FF ...]...=.E......
00000190: 00 00 02 00 00 00 00 00 00 00 80 BF 28 88 37 C5 ............(.7.
000001A0: 4D 00 BC C5 6D FD 3F C4 11 00 1F 00 FF FF 00 00 M...m.?.........
000001B0: 02 00 00 00 00 00 00 00 80 BF 32 80 DC 43 3A 02 ..........2..C:.
000001C0: AE C5 FF 01 00 44 07 00 1F 00 FF FF 00 00 02 00 .....D..........
000001D0: 00 00 00 00 00 00 80 BF 0A B3 40 C4 31 BA 90 C5 [email protected]...
000001E0: 57 00 E4 45 0A 00 1F 00 FF FF 00 00 00 00 00 00 W..E............
000001F0: 00 00 00 00 80 BF 75 AB 0B C4 35 FB BC C5 FD 68 ......u...5....h
00000200: E0 C4 13 00 21 00 3C 00 00 00 02 00 00 00 00 00 ....!.<.........
00000210: 00 00 80 BF A6 80 9E 44 68 80 05 43 5E 92 77 44 .......Dh..C^.wD
00000220: 21 00 0D 00 40 00 !...@.
b'50524a580100000012000200020000000000000080bf4017acc519f4e744e0778bc50f001f00ffff0200020000000000000080bf35e9abc59052e8447508b4c50e001f00ffff0200020000000000000080bf696637c50a02c044cbff2fc524000800ffff0200020000000000000080bff9e9bbc4422b1245640d804314001f00ffff0000000000000000000080bf2c9089c55b8cf444240980432a000300ffff0000020000000000000080bf834a1ac583f6d244a7fcdfc4250017000b000200020000000000000080bfd2d229c5a613da44d6ac8b432b001f000f000200020000000000000080bf115b01c5f1a70245620d80432c001f0011000000020000000000000080bfa3bf6ec496690e45e6fd9fc40b001400ffff0000020000000000000080bf00000044a559c644954e66c50c000800ffff0200020000000000000080bf550853c5d5aed444f4ff87c522001700ffff0000020000000000000080bfb70ec9c4dab491c551d4f9452f001f00ffff0100020000000000000080bf64cfc5c4c25daec5e03d044506001f00ffff0000020000000000000080bf288837c54d00bcc56dfd3fc411001f00ffff0000020000000000000080bf3280dc433a02aec5ff01004407001f00ffff0000020000000000000080bf0ab340c431ba90c55700e4450a001f00ffff0000000000000000000080bf75ab0bc435fbbcc5fd68e0c4130021003c000000020000000000000080bfa6809e44688005435e92774421000d004000'
bytearray(b'PRJX\x01\x00\x00\x00\x12\x00\x02\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf@\x17\xac\xc5\x19\xf4\xe7D\xe0w\x8b\xc5\x0f\x00\x1f\x00\xff\xff\x02\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf5\xe9\xab\xc5\x90R\xe8Du\x08\xb4\xc5\x0e\x00\x1f\x00\xff\xff\x02\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbfif7\xc5\n\x02\xc0D\xcb\xff/\xc5$\x00\x08\x00\xff\xff\x02\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\xf9\xe9\xbb\xc4B+\x12Ed\r\x80C\x14\x00\x1f\x00\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\xbf,\x90\x89\xc5[\x8c\xf4D$\t\x80C*\x00\x03\x00\xff\xff\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\x83J\x1a\xc5\x83\xf6\xd2D\xa7\xfc\xdf\xc4%\x00\x17\x00\x0b\x00\x02\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\xd2\xd2)\xc5\xa6\x13\xdaD\xd6\xac\x8bC+\x00\x1f\x00\x0f\x00\x02\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\x11[\x01\xc5\xf1\xa7\x02Eb\r\x80C,\x00\x1f\x00\x11\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\xa3\xbfn\xc4\x96i\x0eE\xe6\xfd\x9f\xc4\x0b\x00\x14\x00\xff\xff\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\x00\x00\x00D\xa5Y\xc6D\x95Nf\xc5\x0c\x00\x08\x00\xff\xff\x02\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbfU\x08S\xc5\xd5\xae\xd4D\xf4\xff\x87\xc5"\x00\x17\x00\xff\xff\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\xb7\x0e\xc9\xc4\xda\xb4\x91\xc5Q\xd4\xf9E/\x00\x1f\x00\xff\xff\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbfd\xcf\xc5\xc4\xc2]\xae\xc5\xe0=\x04E\x06\x00\x1f\x00\xff\xff\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf(\x887\xc5M\x00\xbc\xc5m\xfd?\xc4\x11\x00\x1f\x00\xff\xff\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf2\x80\xdcC:\x02\xae\xc5\xff\x01\x00D\x07\x00\x1f\x00\xff\xff\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\n\xb3@\xc41\xba\x90\xc5W\x00\xe4E\n\x00\x1f\x00\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\xbfu\xab\x0b\xc45\xfb\xbc\xc5\xfdh\xe0\xc4\x13\x00!\x00<\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x80\xbf\xa6\x80\x9eDh\x80\x05C^\x92wD!\x00\r\x00@\x00') Can this be used for anything in terms of being human readable?
Posts: 6,981
Threads: 22
Joined: Feb 2020
Jan-07-2023, 05:33 AM
(This post was last modified: Jan-07-2023, 05:33 AM by deanhystad.)
I was confused about your post asking if you could read the binary file. I didn't look closely enough at your program. I thought it read a binary format file and produced a YAML format file. That is only one of the possibilities.
I should have realized that you were asking if you could read pic binary file, not the YAML file. As you now know, the answer is "not really". About the only text in your pic file is the magic number (PRJX). This must be some kind of sentinel value that lets your program know you have opened a file of the correct type.
If you are translating a pic file->YAML open the source file as binary ("rb"). This will return bytes when you read the file instead of strings. If you are translating a YAML file to pic open the source file as text ("r") with an encoding like utf-8. It looks like the output is always directed to stdout. Does this get piped to a file?
|
