Python: reading a text file - character

Share

Tweet

Situation:

   Reading a text file in python3 (csv or txt) there is a character that can be appreciated using "more" in terminal but in python3 the situation is more complicated.

Example:

  $ more epa.csv

<U+FEFF>the text

Problem:

   Python3 reads the file well, it doesn't throw an error, but that invisible "character" remains in the variables, the texts, etc. and can cause some inconvenience.

Solution:

   The solution is to read the file and specify the encoding, something as simple as:

FILENAME="epa.csv"

with open(FILENAME, encoding='utf-8-sig') as file:

     for line in file:

         print(line)

Explanation (taken from: https://stackoverflow.com/questions/17912307/u-ufeff-in-python-string):

The Unicode character U+FEFF is the byte order mark, or BOM, and is used to tell the difference between big- and little-endian UTF-16 encoding.

Good luck,