We Keep Coding

How to write a iadx or text file with BOM in python

I want to use BOM with UTF-8. But it only saves files in UTF-8. What can I do ?I'm rather new, could you please write an answer as an addition to the sample code I shared directly?
import os
import codecs
a=1
filelist=os.listdir("name")
for file in filelist:
filelen=len(os.listdir("name/"+file))
if filelen==10:
with open(file + ".iadx", "w", encoding="UTF-8") as f:
f.write("<name>")
f.write("\n")
f.write('something')

From Python documentation on codecs (search for "-sig") :
On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file.
So just doing :
with open(file + ".iadx", "w", encoding="utf-8-sig") as f:
# ^^^^
will do the trick.

Avoiding UnicodeEncodeError in python

I tried to parse an html table into csv using python with a following script:
from bs4 import BeautifulSoup
import requests
import csv
csvFile = open('log.csv', 'w', newline='')
writer = csv.writer(csvFile)
def parse():
html = requests.get('https://en.wikipedia.org/wiki/Comparison_of_text_editors')
bs = BeautifulSoup(html.text, 'lxml')
table = bs.select_one('table.wikitable')
rows = table.select('tr')
for row in rows:
csvRow = []
for cell in row.findAll(['th', 'td']):
csvRow.append(cell.getText())
writer.writerow(csvRow)
print(csvRow)
parse()
csvFile.close()
This code outputed a clear formated CSV file with no encoding issues.
All was just fine before Enrico Tröger's Geany. My script was unable to write ö
into a csv file, so i tried this:
csvRow.append(cell.text.encode('ascii', 'replace')) instead of that: csvRow.append(cell.getText())
All was fine, despite the fact that each table cell was nested in b''. So, how can i get a clear formated csv file withous encoding issues(like in the first screenshot) and replaced or ignored all
non-unicode symbols(like in the second screenshot) using my scipt?

Change this one:
csvFile = open('log.csv', 'w', newline='')
To this one:
csvFile = open('log.csv', 'w', newline='', encoding='utf8')
csv module documentation:
Since open() is used to open a CSV file for reading, the file will by default be decoded into unicode using the system default encoding (see locale.getpreferredencoding()). To decode a file using a different encoding, use the encoding argument of open:
import csv
with open('some.csv', newline='', encoding='utf-8') as f:
reader = csv.reader(f)
for row in reader:
print(row)
The same applies to writing in something other than the system default encoding: specify the encoding argument when opening the output file.
I suppose your system default encoding is not utf8.
You can check it like this:
import locale
locale.getpreferredencoding()
Hope it helps!

Looks like the csv module expects strings, not bytes. So you could de-encode your bytes before passing them:
cell.text.encode('ascii', 'replace').decode('ascii')

Change file encoding scheme in Python

I'm trying to open a file using latin-1 encoding in order to produce a file with a different encoding. I get a NameError stating that unicode is not defined. Here the piece of code I use to this:
sourceEncoding = "latin-1"
targetEncoding = "utf-8"
source = open(r'C:\Users\chsafouane\Desktop\saf.txt')
target = open(r'C:\Users\chsafouane\Desktop\saf2.txt', "w")
target.write(unicode(source.read(), sourceEncoding).encode(targetEncoding))
I'm not used at all to handling files so I don't know if there is a module I should import to use "unicode"

The fact that you see unicode not defined suggests that you're in Python3. Here's a code snippet that'll generate a latin1-encoded file, then does what you want to do, slurp the latin1-encoded file and spit out a UTF8-encoded file:
# Generate a latin1-encoded file
txt = u'U+00AxNBSP¡¢£¤¥¦§¨©ª«¬SHY­®¯U+00Bx°±²³´µ¶·¸¹º»¼½¾¿U+00CxÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏU+00DxÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßU+00ExàáâãäåæçèéêëìíîïU+00Fxðñòóôõö÷øùúûüýþÿ'
latin1 = txt.encode('latin1')
with open('example-latin1.txt', 'wb') as fid:
fid.write(latin1)
# Read in the latin1 file
with open('example-latin1.txt', 'r', encoding='latin1') as fid:
contents = fid.read()
assert contents == latin1.decode('latin1') # sanity check
# Spit out a UTF8-encoded file
with open('converted-utf8.txt', 'w') as fid:
fid.write(contents)
If you want the output to be something other than UTF8, add an encoding argument to open, e.g.,
with open('converted-utf_32.txt', 'w', encoding='utf_32') as fid:
fid.write(contents)
The docs have a list of all supported codecs.

python - writing hex digits to csv

I am having a the following string:
>>> line = '\x00\t\x007\x00\t\x00C\x00a\x00r\x00d\x00i\x00o\x00 \x00M\x00e\x00t\x00a\x00b\x00o\x00l\x00i\x00c\x00 \x00C\x00a\x00r\x00e\x00\t\x00\t\x00\t\x00\t\x00 \x001\x002\x00,\x007\x008\x008\x00,\x005\x002\x008\x00.\x000\x004\x00\r\x00\n'
When I type the variable line in the python terminal it showing the following:
>>> line
'\x00\t\x007\x00\t\x00C\x00a\x00r\x00d\x00i\x00o\x00 \x00M\x00e\x00t\x00a\x00b\x00o\x00l\x00i\x00c\x00 \x00C\x00a\x00r\x00e\x00\t\x00\t\x00\t\x00\t\x00 \x001\x002\x00,\x007\x008\x008\x00,\x005\x002\x008\x00.\x000\x004\x00\r\x00\n'
When I am printing it, its showing the following:
>>> print line
7 Cardio Metabolic Care 12,788,528.04
In the variable line each word is separated using \t and I wanted to save it to a csv file. So I tried using the following code:
import csv
with open('test.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',')
spamwriter.writerow(line.split('\t'))
When I look into the test.csv file, I am getting only the following
,,,,,,
Is there any to get the words into the csv file. Kindly help.

Your input text is not corrupted, it's encoded - as UTF-16 (Big Endian in this case). And it's CSV itself, just with tab as the delimiter.
You must decode it into a string, after that you can use it normally.
Ideally you declare the proper byte encoding when you read it from a source. For example, when you open a file you can state the encoding the file uses so that the file reader will decode the contents for you.
If you have that byte string from a source where you can't declare an encoding while reading it, you can decode manually:
line = '\x00\t\x007\x00\t\x00C\x00a\x00r\x00d\x00i\x00o\x00 \x00M\x00e\x00t\x00a\x00b\x00o\x00l\x00i\x00c\x00 \x00C\x00a\x00r\x00e\x00\t\x00\t\x00\t\x00\t\x00 \x001\x002\x00,\x007\x008\x008\x00,\x005\x002\x008\x00.\x000\x004\x00\r\x00\n'
decoded = line.decode('utf_16_be')
print decoded
# 7 Cardio Metabolic Care 12,788,528.04
But since I suppose that you are actually reading it from a file:
import csv
import codecs
with codecs.open('input.txt', 'r', encoding='utf16') as in_file, codecs.open('output.csv', 'w', encoding='utf8') as out_file:
reader = csv.reader(in_file, delimiter='\t')
writer = csv.writer(out_file, delimiter=',', quotechar='"')
writer.writerows(reader)

Output CSV file encoding is incorrect UTF-8

I have UTF-8 (no BOM) encoded CSV file:
aaa;bbb;ccc
fff;äää;ööö
Following snippet reads the the file and then saves it again using different encoding:
import csv
rows = []
with open('test_in.csv', 'r', newline='') as file:
csvReader = csv.reader(file, delimiter=';')
for row in csvReader:
rows.append(row)
with open('test_out.csv', 'w', newline='', encoding='iso-8859-1') as file:
csvWriter = csv.writer(file, delimiter=';', quoting=csv.QUOTE_MINIMAL)
for row in rows:
csvWriter.writerow(row)
Problem: Saved file is not iso-8859-1, but utf-8 encoded.
If I replace the file read with following list in my UTF-8 encoded source code file, it works correctly:
rows = [
['aaa','bbb','ccc'],
['fff','äää','ööö']
]
Is this a bug in Python? Or do I have to use additional encoding options?
Tested with Python 3.4.

I tried with python3.5.1 and it worked fine for me:
sharad#ss:~$ rm test_out.csv
sharad#ss:~$ ls test_in.csv
test_in.csv
sharad#ss:~$ cat my.py
import csv
rows = []
#with open('my.csv', 'r', newline='', encoding='utf-8') as file:
with open('test_in.csv', 'r', newline='') as file:
csvReader = csv.reader(file, delimiter=';')
for row in csvReader:
rows.append(row)
with open('test_out.csv', 'w', newline='', encoding='iso-8859-1') as file:
csvWriter = csv.writer(file, delimiter=';', quoting=csv.QUOTE_MINIMAL)
for row in rows:
csvWriter.writerow(row)
sharad#ss:~$
sharad#ss:~$ python3.5 my.py
sharad#ss:~$ ls test_out.csv
test_out.csv
sharad#ss:~$ file test_*.csv
test_in.csv: UTF-8 Unicode text
test_out.csv: ISO-8859 text, with CRLF line terminators
sharad#ss:~$

It seems encoding option for open() doesn't work as I thought (I assumed it defaults to UTF-8). Docs for open say:
encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any text encoding supported by Python can be used. ...
And on my system it seems to default to cp1252. Thus repr(rows) returns
[['aaa', 'bbb', 'ccc'], ['fff', 'äää', 'ööö']]
Fix is to explicitly specify encoding also for the input file:
with open('test_in.csv', 'r', newline='', encoding='utf-8') as file: