I am having a the following string:
>>> line = '\x00\t\x007\x00\t\x00C\x00a\x00r\x00d\x00i\x00o\x00 \x00M\x00e\x00t\x00a\x00b\x00o\x00l\x00i\x00c\x00 \x00C\x00a\x00r\x00e\x00\t\x00\t\x00\t\x00\t\x00 \x001\x002\x00,\x007\x008\x008\x00,\x005\x002\x008\x00.\x000\x004\x00\r\x00\n'
When I type the variable line in the python terminal it showing the following:
>>> line
'\x00\t\x007\x00\t\x00C\x00a\x00r\x00d\x00i\x00o\x00 \x00M\x00e\x00t\x00a\x00b\x00o\x00l\x00i\x00c\x00 \x00C\x00a\x00r\x00e\x00\t\x00\t\x00\t\x00\t\x00 \x001\x002\x00,\x007\x008\x008\x00,\x005\x002\x008\x00.\x000\x004\x00\r\x00\n'
When I am printing it, its showing the following:
>>> print line
7 Cardio Metabolic Care 12,788,528.04
In the variable line each word is separated using \t and I wanted to save it to a csv file. So I tried using the following code:
import csv
with open('test.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',')
spamwriter.writerow(line.split('\t'))
When I look into the test.csv file, I am getting only the following
,,,,,,
Is there any to get the words into the csv file. Kindly help.
Your input text is not corrupted, it's encoded - as UTF-16 (Big Endian in this case). And it's CSV itself, just with tab as the delimiter.
You must decode it into a string, after that you can use it normally.
Ideally you declare the proper byte encoding when you read it from a source. For example, when you open a file you can state the encoding the file uses so that the file reader will decode the contents for you.
If you have that byte string from a source where you can't declare an encoding while reading it, you can decode manually:
line = '\x00\t\x007\x00\t\x00C\x00a\x00r\x00d\x00i\x00o\x00 \x00M\x00e\x00t\x00a\x00b\x00o\x00l\x00i\x00c\x00 \x00C\x00a\x00r\x00e\x00\t\x00\t\x00\t\x00\t\x00 \x001\x002\x00,\x007\x008\x008\x00,\x005\x002\x008\x00.\x000\x004\x00\r\x00\n'
decoded = line.decode('utf_16_be')
print decoded
# 7 Cardio Metabolic Care 12,788,528.04
But since I suppose that you are actually reading it from a file:
import csv
import codecs
with codecs.open('input.txt', 'r', encoding='utf16') as in_file, codecs.open('output.csv', 'w', encoding='utf8') as out_file:
reader = csv.reader(in_file, delimiter='\t')
writer = csv.writer(out_file, delimiter=',', quotechar='"')
writer.writerows(reader)
Related
I tried to parse an html table into csv using python with a following script:
from bs4 import BeautifulSoup
import requests
import csv
csvFile = open('log.csv', 'w', newline='')
writer = csv.writer(csvFile)
def parse():
html = requests.get('https://en.wikipedia.org/wiki/Comparison_of_text_editors')
bs = BeautifulSoup(html.text, 'lxml')
table = bs.select_one('table.wikitable')
rows = table.select('tr')
for row in rows:
csvRow = []
for cell in row.findAll(['th', 'td']):
csvRow.append(cell.getText())
writer.writerow(csvRow)
print(csvRow)
parse()
csvFile.close()
This code outputed a clear formated CSV file with no encoding issues.
All was just fine before Enrico Tröger's Geany. My script was unable to write ö
into a csv file, so i tried this:
csvRow.append(cell.text.encode('ascii', 'replace')) instead of that: csvRow.append(cell.getText())
All was fine, despite the fact that each table cell was nested in b''. So, how can i get a clear formated csv file withous encoding issues(like in the first screenshot) and replaced or ignored all
non-unicode symbols(like in the second screenshot) using my scipt?
Change this one:
csvFile = open('log.csv', 'w', newline='')
To this one:
csvFile = open('log.csv', 'w', newline='', encoding='utf8')
csv module documentation:
Since open() is used to open a CSV file for reading, the file will by default be decoded into unicode using the system default encoding (see locale.getpreferredencoding()). To decode a file using a different encoding, use the encoding argument of open:
import csv
with open('some.csv', newline='', encoding='utf-8') as f:
reader = csv.reader(f)
for row in reader:
print(row)
The same applies to writing in something other than the system default encoding: specify the encoding argument when opening the output file.
I suppose your system default encoding is not utf8.
You can check it like this:
import locale
locale.getpreferredencoding()
Hope it helps!
Looks like the csv module expects strings, not bytes. So you could de-encode your bytes before passing them:
cell.text.encode('ascii', 'replace').decode('ascii')
I have a csv file and I want to transfer the raw data without the headers to a new csv file and have the rows and columns the same as the original.
IRIS_data = "IRIS_data.csv"
with open(IRIS_data, 'wb') as data:
wr = csv.writer(data, quoting=csv.QUOTE_ALL)
with open(IRIS) as f:
next(f)
for line in f:
wr.writerow(line)
The code above is my most recent attempt, when I try run it I get the following error:
a bytes-like object is required, not 'str'
It's because you opened the input file with with open(IRIS_data, 'wb'), which opens it in binary mode, and the output file with just with open(IRIS) which opens it in text mode.
In Python 3, you should open both files in text mode and specify newline='' option)—see the examples in the csv module's documentation)
To fix it, change them as follows:
with open(IRIS_data, 'w', newline='') as data:
and
with open(IRIS, newline='') as f:
However there are other issues with you code. Here's how to use those statements to get what I think you want:
import csv
IRIS = "IRIS.csv"
IRIS_data = "IRIS_data.csv"
with open(IRIS, 'r', newline='') as f, open(IRIS_data, 'w', newline='') as data:
next(f) # Skip over header in input file.
writer = csv.writer(data, quoting=csv.QUOTE_ALL)
writer.writerows(line.split() for line in f)
Contents of IRIS_data.csv file after running the script with your sample input data:
"6.4","2.8","5.6","2.2","2"
"5","2.3","3.3","1","1"
"4.9","2.5","4.5","1.7","2"
"4.9","3.1","1.5","0.1","0"
"5.7","3.8","1.7","0.3","0"
"4.4","3.2","1.3","0.2","0"
"5.4","3.4","1.5","0.4","0"
"6.9","3.1","5.1","2.3","2"
"6.7","3.1","4.4","1.4","1"
"5.1","3.7","1.5","0.4","0"
You have to encode the line you are writing like this:
wr.writerow( line.encode(”utf8”))
Also open your file using open(..., ‘wb’). This will open the file in binary mode. So you are certain the file is actually open in binary mode. Indeed it is better to now explicitly the encoding than assuming it. Enforcing encoding for both reading and writing will save you lots of trouble.
This is my code:
filepath = sys.argv[1]
csvdata = list(csv.reader(open(filepath)))
How can I fix it?
I saved my excel file as a csv and receieved this error:
_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
An Excel file is not a csv file. First export / save the file as csv.
There are differences between python versions about whether to open the file as binary or text. This has relevance to how newlines are handled.
In Python 2.x, open as binary: open(filepath, 'rb')
In Python 3.x, don't : open('file.csv', 'r')
The second part I learned from this link about reading in csv files
For some operating systems (Mac OS for sure) you need to open with the mode 'rU' See: this link with same problem specifically on Mac OS
try this (put actual location of csv file)...
with open('c:\pytest.csv', 'rb') as csvfile:
data = csv.reader(csvfile)
mylist = list (data)
print mylist
from tkFileDialog import askopenfilename
import csv
filename = askopenfilename()
with open(filename, 'rb') as csvfile:
data = csv.reader(csvfile)
mylist = list (data)
print mylist
I have UTF-8 (no BOM) encoded CSV file:
aaa;bbb;ccc
fff;äää;ööö
Following snippet reads the the file and then saves it again using different encoding:
import csv
rows = []
with open('test_in.csv', 'r', newline='') as file:
csvReader = csv.reader(file, delimiter=';')
for row in csvReader:
rows.append(row)
with open('test_out.csv', 'w', newline='', encoding='iso-8859-1') as file:
csvWriter = csv.writer(file, delimiter=';', quoting=csv.QUOTE_MINIMAL)
for row in rows:
csvWriter.writerow(row)
Problem: Saved file is not iso-8859-1, but utf-8 encoded.
If I replace the file read with following list in my UTF-8 encoded source code file, it works correctly:
rows = [
['aaa','bbb','ccc'],
['fff','äää','ööö']
]
Is this a bug in Python? Or do I have to use additional encoding options?
Tested with Python 3.4.
I tried with python3.5.1 and it worked fine for me:
sharad#ss:~$ rm test_out.csv
sharad#ss:~$ ls test_in.csv
test_in.csv
sharad#ss:~$ cat my.py
import csv
rows = []
#with open('my.csv', 'r', newline='', encoding='utf-8') as file:
with open('test_in.csv', 'r', newline='') as file:
csvReader = csv.reader(file, delimiter=';')
for row in csvReader:
rows.append(row)
with open('test_out.csv', 'w', newline='', encoding='iso-8859-1') as file:
csvWriter = csv.writer(file, delimiter=';', quoting=csv.QUOTE_MINIMAL)
for row in rows:
csvWriter.writerow(row)
sharad#ss:~$
sharad#ss:~$ python3.5 my.py
sharad#ss:~$ ls test_out.csv
test_out.csv
sharad#ss:~$ file test_*.csv
test_in.csv: UTF-8 Unicode text
test_out.csv: ISO-8859 text, with CRLF line terminators
sharad#ss:~$
It seems encoding option for open() doesn't work as I thought (I assumed it defaults to UTF-8). Docs for open say:
encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any text encoding supported by Python can be used. ...
And on my system it seems to default to cp1252. Thus repr(rows) returns
[['aaa', 'bbb', 'ccc'], ['fff', 'äää', 'ööö']]
Fix is to explicitly specify encoding also for the input file:
with open('test_in.csv', 'r', newline='', encoding='utf-8') as file:
I am trying to read an Excel file using xlrd to write into txt files. Everything is being written fine except for some rows which has some spanish characters like 'Téd'. I can encode those using latin-1 encoding. However the code then fails for other rows which have a 'â' with unicode u'\u2013'. u'\2013' can't be encoded using latin-1. When using UTF-8 'â' are written out fine but 'Téd' is written as 'Téd' which is not acceptable. How do I correct this.
Code below :
#!/usr/bin/python
import xlrd
import csv
import sys
filePath = sys.argv[1]
with xlrd.open_workbook(filePath) as wb:
shNames = wb.sheet_names()
for shName in shNames:
sh = wb.sheet_by_name(shName)
csvFile = shName + ".csv"
with open(csvFile, 'wb') as f:
c = csv.writer(f)
for row in range(sh.nrows):
sh_row = []
cell = ''
for item in sh.row_values(row):
if isinstance(item, float):
cell=item
else:
cell=item.encode('utf-8')
sh_row.append(cell)
cell=''
c.writerow(sh_row)
print shName + ".csv File Created"
Python's csv module
doesn’t support Unicode input.
You are correctly encoding your input before writing it -- so you don't need codecs. Just open(csvFile, "wb") (the b is important) and pass that object to the writer:
with open(csvFile, "wb") as f:
writer = csv.writer(f)
writer.writerow([entry.encode("utf-8") for entry in row])
Alternatively, unicodecsv is a drop-in replacement for csv that handles encoding.
You are getting é instead of é because you are mistaking UTF-8 encoded text for latin-1. This is probably because you're encoding twice, once as .encode("utf-8") and once as codecs.open.
By the way, the right way to check the type of an xlrd cell is to do cell.ctype == xlrd.ONE_OF_THE_TYPES.