I was looking at the very helpful answer to a previous SO question which can be found here when attempting to write a list of dicts to a CSV file. The code I used was:
with open((filename), 'wb') as outfile:
write = csv.DictWriter(outfile, keyList)
write.writer.writerow(keyList)
write.writerows(data)
where keyList is a list of headers for the csv file.
The code worked great, which is nice, but I don't understand why I had to explictly call the underlying writer instance to write the keyList (the headers). I tried that line as write.writerow(keyList) and it didn't work. I'm curious why that is so I can better understand how Python's DictWriter works.
Is there a cleaner/nicer way of writing this?
You appear to be relying on undocumented behavior. A DictWriter object doesn't have an "official" writer method.
The correct way to output the CSV headers is to call
write.writeheader()
Related
I want to read a CSV file without using the open() function.
file.txt
'xxr'|'wer'|'xxr'|'xxr'
'xxt'|'dse'|'xxt'|'xxt'
'xxv'|'cad'|'xxv'|'xxv'
'xxe'|'sdf'|'xxe'|'xxe'
'xxw'|'sder'|'xxw'|'xxw'
'xxz'|'csd'| 'xxz'| 'xxz'
I've tried this, but this doesn't open a file. Just use 'file.txt' as a string.
file = ('file.txt')
reader = csv.reader(file,delimiter="|")
mylist = list(reader)
I cannot use the regular with open('file.txt', 'r') ....
Reason: The customer is sending this data pipeline to a platform that doesn't support the open() function, due to directory function restrictions (not a permissions issue).
I also cannot read as Dataframe, because they are unstructured lists, and this template is much simpler.
This is a conversion from a python script to Data Dream, with spark. Kind of odd... but they can reproduce pandas and numpy. They can't use Function open() and with.
Any ideas?
You could use fileinput although I'm unsure of how the module deals with opening the files and if it is any different than the open function but it does allow for multiple files to be opened in order using one stream and it seems to allow for more flexibility in how the file is read:
import fileinput
with fileinput.input('file.txt') as f:
reader = csv.reader(file,delimiter="|")
mylist = list(reader)
There is nothing wrong with:
reader = csv.reader(open(file),delimiter="|")
Or with pandas:
import pandas as pd
mylist = pd.read_csv(file, sep="|").to_numpy().tolist()
Is there a way, in the code below, to access the variable utterances_dict outside of the with-block? The code below obviously returns the error: ValueError: I/O operation on closed file.
from csv import DictReader
utterances_dict = {}
utterance_file = 'toy_utterances.csv'
with open(utterance_file, 'r') as utt_f:
utterances_dict = DictReader(utt_f)
for line in utterances_dict:
print(line)
I am not an expert on DictReader implementation, however their documentation leaves the implementation open to the reader itself parsing the file after construction. Meaning it may be possible that the underlying file has to remain open until you are done using it. In this case, it would be problematic to attempt to use the utterances_dict outside of the with block because the underlying file will be closed by then.
Even if the current implementation of DictReader does in fact parse the whole csv on construction, it doesn't mean their implementation won't change in the future.
DictReader returns a view of the csv file.
Convert the result to a list of dictionaries.
from csv import DictReader
utterances = []
utterance_file = 'toy_utterances.csv'
with open(utterance_file, 'r') as utt_f:
utterances = [dict(row) for row in DictReader(utt_f) ]
for line in utterances:
print(line)
I'm been learning python and playing around with dictionaries and .csv files and the csv module. It seems like the csv.DictReader() function can help turn .csv files into dictionary objects, but there's a bit of a quirk with the Reader objects that I'm confused about.
I've read a little bit into the documentation (and then tried to find answers looking up at the csv.Reader() function), but I'm still a little unsure.
Why does this code run as expected:
with open("cool_csv.csv") as cool_csv_file:
cool_csv_text = cool_csv_file.read()
print(cool_csv_text)
and yet the following code returns a ValueError: I/O operation on closed file.
with open("cool_csv.csv") as cool_csv_file:
cool_csv_dict = csv.DictReader(cool_csv_file)
for row in cool_csv_dict:
print(row["Cool Fact"])
Since we saved the DictReader object to a python variable, shouldn’t we be able to call the variable after we close the file, like if I were assigned cool cool_csv.read()?
I know the proper way to code this would be:
with open("cool_csv.csv") as cool_csv_file:
cool_csv_dict = csv.DictReader(cool_csv_file)
for row in cool_csv_dict:
print(row["Cool Fact"])
But why does the for row in cool_csv_dict: section have to be nested in the open() section?
My only guess would be that because the csv.DictReader() object is not quite an actual dictionary (or something like that), there’s some shenanigans because it still needs to point somewhere (because maybe thats the "reader" part?).
Can anyone shed any light?
csv.DictReader doesn't read the entire file into memory when you create the cool_csv_dict object. Each time you call it to get the next record from the CSV, it reads the next line from cool_csv_file. Therefore, it needs this to be kept open so it can read from it as needed.
The argument to csv.DictReader can be any iterator that returns lines. So if you don't want to keep the file open, you can call readlines() to get all the lines into a list, then pass that to csv.DictReader
with open("cool_csv.csv") as cool_csv_file:
cool_csv_lines = cool_csv_file.readlines()
cool_csv_dict = csv.DictReader(cool_csv_lines)
for row in cool_csv_dict:
print(row("Cool Fact")
I have a piece of code that has been working for a while that uses Python’s DictReader.
The code initializes the csv reader, csv_reader = csv.DictReader(my_csv) and then I access csv_reader.fieldnames. Historically this has been working fine.
However today it started throwing this error iterator should return strings, not bytes (did you open the file in text mode?) when I try to access csv_reader.fieldnames.
csv_reader.__dict__ shows an object with an attribute _fieldnames, and it is empty. I’m not sure why this changed or what I can do to resolve it, any suggestions are welcome.
You might need to specify your file's encoding explicitly:
with (open('my.csv', 'rt', encoding='utf-8')) as file:
I looked through the questions already posted and my problems isn't quite solved. I think this should be fairly straight forward but I am getting tripped up with the variations. Hoping that after getting walked through this one file, then I can upload and parse the rest.
What I am trying to do:
File is open states data (and other files in dropbox): ca_bills.csv
Convert .csv to python: I think it should be converted to a python list of dicts
Use the headers in the file as keys within dicts
I tried this but it didn't do what I wanted + I wonder if there is way to pull the fieldname from the headers of each file
def csv_dict_writer(fp, fieldnames, data):
with open(fp, "wb") as out_file:
writer = csv.DictWriter(out_file, deliminter=',', fieldnames=fieldnames)
writer.writeheader()
for row in data:
writer.writerow(row)
I also did this but this only prints and doesn't write to a file:
with open('ca_bills.csv') as output_file:
reader = csv.reader(output_file)
for row in reader:
print row
Thanks so much! This may be similar to other question but really couldn't extract what I needed. Appreciate your insights.
result=list(csv.DictReader(fp))