Read and save file in chucksize

[zinho]

I need read a txt file using chuncksize, after save into xlsx.

import os, sys
import pandas as pd
import xlwings as xw


caminho = os.path.abspath(os.path.dirname(sys.argv[0]))
txt = caminho + '\\Fat2.txt'
xls = caminho + '\\cobfat.xlsx'
existe_txt = os.path.exists(txt)
existe_xls = os.path.exists(xls)


def check_path(x):
    # Função para verificar se há arquivo no caminho e com o nome correto. 
    if x != True:
        st = 'O arquivo não foi encontrado.\nVerifique se o nome está no local e com o nome correto.'
    else:
        st = 'ok'
    return st


big_file = 5000

if check_path(existe_txt):
    # Lista para guardados os dados do arquivo
    lst_xl = []

    # Leitura do arquivo usando chuck size
    for df in pd.read_csv(txt, sep="|", header=None, encoding='ISO-8859-1', chunksize=big_file):
        lst_xl.append(df)
        print('Gravando ...', df.shape)

    # Gravar dados no Excel
    df_xl = pd.concat(lst_xl, axis=0)
    writer = pd.ExcelWriter('output.xlsx')
    df_xl.to_excel(writer,'Plan1')
    writer.save()
    # I get error "error memory limit exceeded" after try save this in file.

[zinho]

Hi, any help

Cross-post
I don't know how to write after read this 80k lines from file into excel via pandas
Thank you

[zinho]

Hi guys.
I solved using this line

xw.Range('A2').options(index=False, header=False).value = df_x

[DeaD_EyE]

This part is useless:

existe_txt = os.path.exists(txt)
existe_xls = os.path.exists(xls)

The cause is, even if the files both are existing,
they could disappear after the check. This is not
a protection against Exception like FileNotFoundError
or PermissionError.

Python has for everything Exceptions.

def main():
    try:
        data = pd.read_csv(txt, sep="|", header=None, encoding='ISO-8859-1', chunksize=big_file)
        writer = pd.ExcelWriter('output.xlsx')
    except (FileNotFoundError, PermissionError) as err:
        print(err)
        return
    for df in data:
        ...

    ...


    writer.save()

You could catch more general Exception.
If you get for example a PermissionError and want to look up the inheritance:

PermissionError.mro()

Output:
[PermissionError, OSError, Exception, BaseException, object]

This Exception could be catched by: PermissionError, OSError, Exception