Memory Error While generating cheksum no

[mg24]

Hi Team,

I am trying to create checksum no , getting error memory error.
for syntax====> lines = f.read()

file size of CSV is 60gb , how to avoid memory error while creating checksum.

code I am using.
import hashlib
import os

def Create_chk_sum(fpath,tbl):
	fnameCSV = f'{tbl}.csv'
	fn = os.path.join(fpath,fnameCSV)
	fn_dn = fn
	m = hashlib.md5()
	with open(fn,'rb') as f:
		lines = f.read()
		m.update(lines)
		chk_sum = m.hexdigest()
		fn = os.path.join(fpath,f'ChkSum_No_{tbl}.csv')
		with open(fn,'w') as data:
			data.write(chk_sum)

if __name__ == "__main__":   
    folderpath = "C:\\Users\\mg\\OneDrive\\Desktop\\C\\test_data"
    tbl = "SQL_Table1"
    Create_chk_sum(folderpath, tbl)

[ibreeden]

f.read() will try to read the entire file. You tell us the file is 60GB. If your computer does not have 60GB of memory you will get a memory error.
Solution: read the file in chunks of let's say 10240 bytes (f.read(10240)) and repeat until you read a chunk of size 0.

[mg24]

Hi ibreeden,

superb ! Thanks for your help. it worked.



Thanks
mg