I have a piece of code that has been working for a while that uses Python’s DictReader.
The code initializes the csv reader, csv_reader = csv.DictReader(my_csv) and then I access csv_reader.fieldnames. Historically this has been working fine.
However today it started throwing this error iterator should return strings, not bytes (did you open the file in text mode?) when I try to access csv_reader.fieldnames.
csv_reader.__dict__ shows an object with an attribute _fieldnames, and it is empty. I’m not sure why this changed or what I can do to resolve it, any suggestions are welcome.
You might need to specify your file's encoding explicitly:
with (open('my.csv', 'rt', encoding='utf-8')) as file:
Related
I am trying to process a large JSON file using the follow code:
dctx = zst.ZstdDecompressor(max_window_size=2147483648)
with open(filename+".zst", 'rb') as infile, open(outpath, 'wb') as outfile:
dctx.copy_stream(infile, outfile)
with pd.read_json(filename+".json", lines=True, chunksize=5000) as reader:
reader
# Making list of column headers
df_titles = []
for chunk in reader:
chunk_titles = list(chunk.keys())
df_titles.extend(chunk_titles)
df_titles = list(set(df_titles))
However, when I attempt to run the code, I get an error message: ValueError: Expected object or value. The file is formatted with one JSON object per line, and looking at the JSON file itself, it seems the issue is that one of the JSON objects has a bunch of empty space in front of it.
If I manually delete the 'nul' line, the file processes with no issues. However, for the sake of reproducibility, I would like to be able to address the issue from within my code itself. I'm pretty new to working in Python, and I have tried googling the issue, but solutions seem to focus on removing white space from the beginning of JSON values, rather than the start of a line in this kind of file. Is there any easy way to deal with this issue either when decompressing the initial file, or reading the decompressed file in?
I am getting what seems to be an odd error when trying to open a file in python. I am simply trying to open a csv:
with open(filename, 'a') as history:
filename is simply a string pointing the file:
filename = file_path + "\\dashboards\\" + csv_file
it is identified as a string in python, but whenever I get to the open statement, it returns:
TypeError}'int' object not callable
That seems odd as it is just a csv file, with a header. Has anyone run across this before?
``
It looks like you may have defined open as a variable somewhere else with an int value. This is causing the error message.
Simply, I need to work with files uploaded without saving it on server
working in cli script using open() every thing is fine,
using flask with file sent from data by ajax request
neither open() function nor stream.read() method helped to work with the csv
open throws an exception itself
csv_f = open(request.files['csvfile'].stream, 'rb')
TypeError: expected str, bytes or os.PathLike object, not SpooledTemporaryFile
using .read() I can print it
csv_f = request.files['csvfile'].stream.read()
data = csv.reader(csv_f, delimiter = ',')
print(csv_f)
b'...'
but Iterating also throws exception
for row in data:
_csv.Error: iterator should return strings, not int (did you open the file in text mode?)
I need only a way to work with csv files using csv module on the fly
I found out the problem
the file is going throw request as a binary stream not a normal text
that's why it has a read method but unuseful when itrating
I had to use .decode()
like this
request.files['csvfile'].stream.read().decode("utf-8")
instead of this
request.files['csvfile'].stream.read()
I was looking at the very helpful answer to a previous SO question which can be found here when attempting to write a list of dicts to a CSV file. The code I used was:
with open((filename), 'wb') as outfile:
write = csv.DictWriter(outfile, keyList)
write.writer.writerow(keyList)
write.writerows(data)
where keyList is a list of headers for the csv file.
The code worked great, which is nice, but I don't understand why I had to explictly call the underlying writer instance to write the keyList (the headers). I tried that line as write.writerow(keyList) and it didn't work. I'm curious why that is so I can better understand how Python's DictWriter works.
Is there a cleaner/nicer way of writing this?
You appear to be relying on undocumented behavior. A DictWriter object doesn't have an "official" writer method.
The correct way to output the CSV headers is to call
write.writeheader()
I am trying to save output from a module to CSV file and I got an error when I ran the following code, which is a part of a module:
base_keys = ['path', 'rDATE', 'cDate', 'cik', 'risk', 'word_count']
outFile = open('c:\\Users\\ahn_133\\Desktop\\Python Project\\MinkAhn_completed2.csv','wb')
dWriter = csv.DictWriter(outFile, fieldnames=base_keys)
dWriter.writerow(headerDict)
Here is the error message (base_keys are the headings.)
return self.writer.writerow(self._dict_to_list(rowdict))
TypeError: 'str' does not support the buffer interface
I dont' even understand what the error message is about. I use Python 3.3 and Windows 7.
Thanks for your time.
Opening a file in binary mode to write csv data to doesn't work in Python 3, simply put. What you want is to open in text mode and either use the default encoding or specify one yourself, i.e., your code should be written like:
import csv
k = ['hi']
out = open('bleh.csv', 'w', newline='', encoding='utf8') # mode could be 'wt' for extra-clarity
writer = csv.DictWriter(out, k)
writer.writerow({'hi': 'hey'})
Now, due to a bug, you also need to specify newline='' when opening this file for writing the CSV output.