I looked through the questions already posted and my problems isn't quite solved. I think this should be fairly straight forward but I am getting tripped up with the variations. Hoping that after getting walked through this one file, then I can upload and parse the rest.
What I am trying to do:
File is open states data (and other files in dropbox): ca_bills.csv
Convert .csv to python: I think it should be converted to a python list of dicts
Use the headers in the file as keys within dicts
I tried this but it didn't do what I wanted + I wonder if there is way to pull the fieldname from the headers of each file
def csv_dict_writer(fp, fieldnames, data):
with open(fp, "wb") as out_file:
writer = csv.DictWriter(out_file, deliminter=',', fieldnames=fieldnames)
writer.writeheader()
for row in data:
writer.writerow(row)
I also did this but this only prints and doesn't write to a file:
with open('ca_bills.csv') as output_file:
reader = csv.reader(output_file)
for row in reader:
print row
Thanks so much! This may be similar to other question but really couldn't extract what I needed. Appreciate your insights.
result=list(csv.DictReader(fp))
Related
I have seen similar posts to this but they all seem to be print statements (viewing the cleaned data) rather than overwriting the original csv with the cleaned data so I am stuck. When I tried to write back to the csv myself, it just deleted everything in the file. Here is the format of the csv:
30;"unemployed";"married";"primary";"no";1787;"no";"no";"cellular";19;"oct";79;1;-1;0;"unknown";"no"
33;"services";"married";"secondary";"no";4747;"yes";"cellular";11;"may";110;1;339;2;"failure";"no"
35;"management";"single";"tertiary";"no";1470;"yes";"no";"cellular";12;"apr"185;1;330;1;"failure";"no"
It is delimited by semicolons, which is fine, but all text is wrapped in quotes and I only want to remove the quotes and write back to the file. Here is the code I reverted back to that successfully reads the file, removes all quotes, and then prints the results:
import csv
f = open("bank.csv", 'r')
try:
for row in csv.reader(f, delimiter=';', skipinitialspace=True):
print(' '.join(row))
finally:
f.close()
Any help on properly writing back to the csv would be appreciated, thanks!
See here: Python CSV: Remove quotes from value
I've done this basically two different ways, depending on the size of the csv.
You can read the entire csv into a python object (list), do some things and then
overwrite the other existing file with the cleaned version
As in the link above, you can use one reader and one writer, Create a new file, and write line by-line as you clean the input from the csv reader, delete the original csv and rename the new one to replace the old file.
In my opinion option #2 is vastly preferable as it avoids the possibility of data loss if your script has an error part way through writing. It also will have lower memory usage.
Finally: It may be possible to open a file as read/write, and iterate line-by-line overwriting as you go: But that will leave you open to half of your file having quotes, and half not if your script crashes part way through.
You could do something like this. Read it in, and write using quoting=csv.QUOTE_NONE
import csv
f = open("bank.csv", 'r')
inputCSV = []
try:
for row in csv.reader(f, delimiter=';', skipinitialspace=True):
inputCSV.append(row)
finally:
f.close()
with open('bank.csv', 'w', newline='') as csvfile:
csvwriter = csv.writer(csvfile, delimiter=';')
for row in inputCSV:
csvwriter.writerow(row)
I am trying to write 2 values as a row into a excel file I have saved as a .csv.
I tried looking around to see if I could find a solution but I couldn't find exactly what I wanted or the answer was too complicated.
I am fairly new to coding so most of the stuff you say will go over my head, but if someone could give a simple(ish) solution I would be very grateful.
By the way, it needs to add it onto the bottom of the file and not onto a specific line, just in case that info was needed.
All I know how to do is open, close, and read the file.
Thanks in advance
Use module csv to work with CSV file and open it in "append" mode to append row at the end of file
import csv
with open('data.csv', 'a') as fp:
csvwriter = csv.writer(fp)
csvwriter.writerow( [1, 2] )
The same little different
import csv
fp = open('data.csv', 'a')
csvwriter = csv.writer(fp)
csvwriter.writerow( [1, 2] )
fp.close()
you can either write one row, or many rows at the same time using default python's csv module.
docs with examples: csv module
I have looked at previous answers to this question, but in each of those scenarios the questioners were asking about something specific they were doing with the file, but the problem occurs for me even when I am not.
I have a .csv file of 27,204 rows. When I open the python interpreter:
python
import csv
o = open('btc_usd1hour.csv','r')
p = csv.reader(o)
for row in p:
print(row)
I then only see roughly the last third of the document displayed to me.
Try so, at me works:
with open(name) as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
print(row)
reference:
https://docs.python.org/3.6/library/csv.html#csv.DictReader
Try the following code
import csv
fname = 'btc_usd1hour.csv'
with open(fname, newline='') as f:
reader = csv.reader(f)
for row in reader:
print(row)
It is difficult to tell what is the problem without having the sample. I guess the problem would be removed if you add that newline='' for opening the file.
Use the with construct to close the file automatically. Use the f name for a file object when no further explanation is needed. Store the file name to fname to make future modifications easier (and also for easy copying the code fragment for your later programs).
olisch may be right that the console just scrolled so fast you could not see the result. You can write the result to another text file like this:
with open(fname, newline='') as fin,\
open('output.txt', 'w') as fout:
reader = csv.reader(fin)
for row in reader:
fout.write(repr(row) + '\n')
The repr function converts the row list into its string representation. The print calls that function internally, so you will have the same result that you otherwise observe on screen.
maybe your scrollback buffer is just to short to see the whole list?
In general your csv.reader call should be working fine, except your 27k rows aren't extremly long so that you might be able to hit any 64bit boundaries, which would be quite uncommon.
len(o) might be interesting to see.
I have a large csv file that I want to work with and the column heading may change over time. The first thing I want to do is see the field names. I can see the csvreader.fieldnames object in the documentation [link], but I can't find any examples.
What's the simplest way to get started?
You can use a csv.DictReader when you already know the columns that the CSV file will have.
If you don't know that then you could consider reading the first line with csv.reader to obtain the names of the columns and then read the file again with a csv.DictReader (which can then be instantiated with the right fieldnames).
Here is an example,
with open('yourFile.csv', "rt", newline='') as csvfile:
csvreader = csv.DictReader(csvfile,
delimiter=',',
quotechar='|')
return csvreader.fieldnames
I've tried many solutions to add a header to my csv file, but nothing's working properly. Here they are :
I used the writerow method, but my data are overwriting the first row.
I used the DictWriter method, but I don't know how to fill it correctly. Here is my code:
csv = csv.DictWriter(open(directory +'/csv.csv', 'wt'), fieldnames = ["stuff1", "stuff2", "stuff3"], delimiter = ';')
csv.writeheader(["stuff1", "stuff2", "stuff3"])
I got a "2 arguments instead of one" error and I really don't know why.
Any advice?
All you need to do is call DictWriter.writeheader() without arguments:
with open(os.path.join(directory, 'csv.csv'), 'wb') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames = ["stuff1", "stuff2", "stuff3"], delimiter = ';')
writer.writeheader()
You already told DictWriter() what your headers are.
I encountered a similar problem when writing the CSV file.
I had to read the csv file and modify some of the fields in it.
To write the header in the CSV file, i used the following code:
reader = csv.DictReader(open(infile))
headers = reader.fieldnames
with open('ChartData.csv', 'wb') as outcsv:
writer1 = csv.writer(outcsv)
writer1.writerow(headers)
and when you write the data rows, you can use a DictWriter in the following way
writer = csv.DictWriter(open("ChartData.csv", 'a' ), headers)
In the above code "a" stands for appending.
In conclusion - > use a to append data to csv after you have written your header to the same file