UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 34: character

[Melcu54]

hello. I donit know what to to with this error:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 34: character maps to <undefined>

This is the Python Code:

import fileinput
import glob
import os
import re
with open('c:\\Folder6\\merged.txt', 'w', encoding='UTF-8') as f:

        for line in fileinput.input(sorted(glob.glob('c:\\Folder6\\*.txt'))):
            f.write(line)
        fileinput.close()
        print(f)

And this is the ERROR:

Traceback (most recent call last):
  File "E:\Carte\BB\17 - Site Leadership\alte\Ionel Balauta\Aryeht\Task 1 - Traduce tot site-ul\Doar Google Web\Andreea\Meditatii\Sedinta 31 august 2022\merge txt - versiune 3 .py", line 8, in <module>
    for line in fileinput.input(sorted(glob.glob('c:\\Folder6\\*.txt'))):
  File "C:\Program Files\Python39\lib\fileinput.py", line 256, in __next__
    line = self._readline()
  File "C:\Program Files\Python39\lib\fileinput.py", line 389, in _readline
    return self._readline()
  File "C:\Program Files\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 34: character maps to <undefined>

This is a print screen:



What can I do, as not to apear that error again? Can anyone help me?

[deanhystad]

You should set the encoding when you read the file (fileinput). Windows must thing it is something other than utf-8.

[Melcu54]

hello, sir. Thank you for answer.

Can you modify my script as to work with your solution? I don't know Python very good...I am a beginner.

I don't know if this is good, as I make it now. Doesn't do nothing...but have no error..

import fileinput
import glob
import os
import re
 
def read_text_from_file(file_path):
 
    with open(file_path, encoding='utf8') as f:
        text = f.read()
        return text
 
def write_to_file(text, file_path):
 
    with open(file_path, 'wb') as f:
        f.write(text.encode('utf8', 'ignore'))
     
    with open('c:\\Folder6\\translated\\merged.txt', 'w', encoding='UTF-8') as f:
  
        for file_name in sorted(glob.glob('c:\\Folder6\\translated\\*.txt')):
            contents = read_text_from_file(file_name)
            f.write(line)
        fileinput.close()
        print(f)

OR, SECOND VERSION:

import fileinput
import glob
import os
import re

with open('c:\\Folder6\\translated\\merged.txt', 'w', encoding='UTF-8') as f:
         current_content = f.read()
         modified = new_content != current_content
if modified and args.diff:
 
        for line in fileinput.input(sorted(glob.glob('c:\\Folder6\\translated\\*.txt'))) :
            f.write(line)
        fileinput.close()
        print(f)

OR, 3' SOLUTION

import fileinput
import glob
import os
import re
 

     
read_files = sorted(glob.glob("c:\\Folder6\\translated\\merged.txt\\*.txt"))

with open("c:\\Folder6\\translated\\merged.txt", "wb") as outfile:
    for f in read_files:
        with open(f, "rb") as infile:
            outfile.write(infile.read())
        fileinput.close()
        print(f)

None of them works. It creates the file, but does not write it

[Melcu54]

I found a solution:

import fileinput
import glob
import os
import re
 
def read_files(file_path):
 
    with open(file_path, encoding='utf8') as f:
        text = f.read()
        return text
 
def read_files(text, file_path):
 
    with open(file_path, 'rb') as f:
        f.write(text.encode('utf8', 'ignore'))
        
     
read_files = sorted(glob.glob("c:\\Folder6\\translated\\*.txt"))

with open("c:\\Folder6\\translated\\merged.txt", "wb") as outfile:
    for f in read_files:
        with open(f, "rb") as infile:
            outfile.write(infile.read())
            outfile.write(b"\n\n")

            
        fileinput.close()
        print(f)

[snippsat]

In your first code it would be like this.

import fileinput
import glob
import os

with open('c:\\Folder6\\merged.txt', 'w', encoding='UTF-8') as f:
    for line in fileinput.input(sorted(glob.glob('c:\\Folder6\\*.txt')), encoding="utf-8"):
        print(line)
        f.write(line)
    fileinput.close()

This need Python 3.10 to work as in fileinput doc

Changed in version 3.10: The keyword-only parameter encoding and errors are added.

[Melcu54]

In your first code it would be like this.

import fileinput
import glob
import os

with open('c:\\Folder6\\merged.txt', 'w', encoding='UTF-8') as f:
    for line in fileinput.input(sorted(glob.glob('c:\\Folder6\\*.txt')), encoding="utf-8"):
        print(line)
        f.write(line)
    fileinput.close()

This need Python 3.10 to work as in fileinput doc

Changed in version 3.10: The keyword-only parameter encoding and errors are added.

ok, thanks. But if I want to put an [b]f.write("\n\n") in order to have a dividing line between the files, where should I put it?[/b]

[snippsat]

But if I want to put an [b]f.write("\n\n") in order to have a dividing line between the files

Change line 8:

f.write(f'{line}\n\n')

[Melcu54]

Change line 8:

f.write(f'{line}\n\n')

I try also this. But, in this case, will double all my lines from all text files, into one file.

See the duplicate lines after using your code (is better with f.write('\n')) , except this will put a new empty lines between all paragraphs)