Segmentation fault with large files

[kusal1]

Hi,

Here I'm trying to remove double quotation character from all text files. I'm getting an error "Segmentation fault" while I reading more than 8gb files in folder '/data/DWH/29SEP/'. Any comments?

import os

def Main():
    path = '/data/DWH/29SEP/'
    files = []
    for r, d, f in os.walk(path):
                for file in f:
                        if '.TXT' in file:
                                files.append(os.path.join(r, file))
    for f in files:
        print(f)
        with open(f + 'e', 'w') as outfile:
            with open(f, 'r') as infile:
                temp = infile.read().replace("\"", "")
                outfile.write(temp)

if __name__ == "__main__":
    Main()

[Gribouillis]

infile.read() loads the whole content of infile at once in memory. Read the file by chunks instead.

[kusal1]

infile.read() loads the whole content of infile at once in memory. Read the file by chunks instead.

Hi Gribouillis,

Have you sample code?

Regards
Kusal

[Gribouillis]

Hi Gribouillis,

Have you sample code?

Yes