I have two programs in Python. One writes a customer's information to a CSV. The other accesses it. When the first has written it, I can open the CSV file (in Excel) and see that it has been written correctly. However for the other program to access the new data in the CSV file I have to manually open it and save it (in Excel) otherwise it doesn't work. Does anyone know why this may be?
Edit:
This writes to it (from first program):
f = open('details.csv', 'at', newline=''); csv_f = csv.reader(f)
csv_w.writerow(clientList)
f.close()
And this reads it (second program):
f = open('details.csv', 'rt', newline=''); csv_f = csv.reader(f)
for row in csv_f:
name.append(row[0])
I get this error when trying to append row[0] to a list.
Traceback (most recent call last):
File "C:\Users\Dan\Desktop\Garden Centre\work.py", line 8, in <module>
name.append(row[0])
IndexError: list index out of range
I have seen a number of such problems stemming from differnt platform line endings. Under Python 2, you might try the "universal line endings" file reading mode:
with open('data.csv', 'rU') as f:
for row in csv.reader(f):
print row
Because Excel does often use the old Mac (\r) and Windows (\r\n) standards, which can get in the csv module's way, esp. on a Unix or Mac platform where Python expects the Unix standard line ending (\n). Python 3 is generally smarter about this (and other file/string encoding issues), so generally doesn't need a special mode.
I found an answer after hours of trying. In Excel, each item in an 'empty' row contains '' for the largest number of items in any row. Python doesn't write it like that to the CSV and instead only one item on an empty row contains None. As it iterated through each row, there was no first item to add to the list on the empty rows.
I had to manually add an extra '' to the list that is wrote by the first program.
Related
I am unable to handle case where user forgot to put newline.
I am trying to append new line to a csv file in python using below code -
with open(local_file_location, 'a') as file:
writer = csv.writer(file)
writer.writerow(row)
But in case if file do not have a new line it simply appends to same last line for the first time. And I am not sure how to handle that.
Ex-
Input-
1,2,3,4 <no-newline>
add row - {a,b,c,d}
Output-
1,2,3,4,a,b,c,d
but I want handle output in this event case to be as below -
1,2,3,4
a,b,c,d
Note: it should just append as well in case user file already have a new line.-> which current program does perfectly.
Let me know what can I do.
You can check if a file ends with a newline.
with open(local_file_location, 'r') as file:
data = file.read()
if data.endswith('\n'):
# some logic
I've noticed a really weird bug and didn't know if anyone else had seen this / knows how to stop it.
I'm writing to a CSV file using this:
def write_to_csv_file(self, object, string):
with open('data_model_1.csv', 'a') as f:
writer = csv.writer(f)
writer.writerow([object, string])
and then write to the file:
self.write_to_csv_file(self.result['outputLabel'], string)
If I open the CSV file to look at the results, the next time I write to the file, it will start in column 3 of the last line (column 1 is object, column 2 is string).
If I run self.write_to_csv_file(self.result['outputLabel'], string) multiple times without manually opening the file (obviously I open the file in the Python script), everything is fine.
It's only when I open the file so I get the issue of starting on Column 3.
Any thoughts on how to fix this?
You're opening the file in append mode, so the data is appended to the end of the file. If the file doesn't end in a newline, rows may get concatenated. Try writing a newline to the file before appending new rows:
with open("data_model_1.csv", "a") as f:
f.write("\n")
wrote a python script in windows 8.1 using Sublime Text editor and I just tried to run it from terminal in OSX Yosemite but I get an error.
My error occurs when parsing the first line of a .CSV file. This is the slice of the code
lines is an array where each element is the line in the file it is read from as a string
we split the string by the desired delimiter
we skip the first line because that is the header information (else condition)
For the last index in the for loop i = numlines -1 = the number of lines in the file - 2
We only add one to the value of i because the last line is blank in the file
for i in range(numlines):
if i == numlines-1:
dataF = lines[i+1].split(',')
else:
dataF = lines[i+1].split(',')
dataF1 = list(dataF[3])
del(dataF1[len(dataF1)-1])
del(dataF1[len(dataF1)-1])
del(dataF1[0])
f[i] = ''.join(dataF1)
return f
All the lines in the csv file looks like this (with the exception of the header line):
"08/06/2015","19:00:00","1","410"
So it saves the single line into an array where each element corresponds to one of the 4 values separated by commas in a line of the CSV file. Then we take the 3 element in the array, "410" ,and create a list that should look like
['"','4','1','0','"','\n']
(and it does when run from windows)
but it instead looks like
['"','4','1','0','"','\r','\n']
and so when I concatenate this string based off the above code I get 410 instead of 410.
My question is: Where did the '\r' term come from? It is non-existent in the original files when ran by a windows machine. At first I thought it was the text format so I saved the CSV file to a UTF-8, that didn’t work. I tried changing the tab size from 4 to 8 spaces, that didn’t work. Running out of ideas now. Any help would be greatly appreciated.
Thanks
The "\r" is the line separator. The "\r\n" is also a line separator. Different platforms have different line separators.
A simple fix: if you read a line from a file yourself, then line.rstrip() will remove the whitespace from the line end.
A proper fix: use Python's standard CSV reader. It will skip the blank lines and comments, will properly handle quoted strings, etc.
Also, when working with long lists, it helps to stop thinking about them as index-addressed 'arrays' and use the 'stream' or 'sequential reading' metaphor.
So the typical way of handling a CSV file is something like:
import csv
with open('myfile.csv') as f:
reader = csv.reader(f)
# We assume that the file has 3 columns; adjust to taste
for (first_field, second_field, third_field) in reader:
# do something with field values of the current lines here
import csv, Tkinter
with open('most_common_words.csv') as csv_file: # Opens the file in a 'closure' so that when it's finished it's automatically closed"
csv_reader = csv.reader(csv_file) # Create a csv reader instance
for row in csv_reader: # Read each line in the csv file into 'row' as a list
print row[0] # Print the first item in the list
I'm trying to import this list of most common words using csv. It continues to give me the same error
for row in csv_reader: # Read each line in the csv file into 'row' as a list
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
I've tried a couple different ways to do it as well, but they didn't work either. Any suggestions?
Also, where does this file need to be saved? Is it okay just being in the same folder as the program?
You should always open a CSV file in binary mode (Python 2) or universal newline mode (Python 3). Also, make sure that the delimiters and quote characters are , and ", or you'll need to specify otherwise:
with open('most_common_words.csv', 'rb') as csv_file:
csv_reader = csv.reader(csv_file, delimiter=';', quotechar='"') # for EU CSV
You can save the file in the same folder as your program. If you don't, you can provide the correct path to open() as well. Be sure to use raw strings if you're on Windows, otherwise the backslashes may trick you: open(r"C:\Python27\data\table.csv")
It seems you have a file with one column as you say here:
It is a simple list of words. When I open it up, it opens into Excel
with one column and 500 rows of 500 different words.
If so, you don't need the csv module at all:
with open('most_common_words.csv') as f:
rows = list(f)
Note in this case, each item of the list will have the newline appended to it, so if your file is:
apple
dog
cat
rows will be ['apple\n', 'dog\n', 'cat\n']
If you want to strip the end of line, then you can do this:
with open('most_common_words.csv') as f:
rows = list(i.rstrip() for i in f)
[Please note that this is a different question from the already answered How to replace a column using Python’s built-in .csv writer module?]
I need to do a find and replace (specific to one column of URLs) in a huge Excel .csv file. Since I'm in the beginning stages of trying to teach myself a scripting language, I figured I'd try to implement the solution in python.
I'm having trouble when I try to write back to a .csv file after making a change to the contents of an entry. I've read the official csv module documentation about how to use the writer, but there isn't an example that covers this case. Specifically, I am trying to get the read, replace, and write operations accomplished in one loop. However, one cannot use the same 'row' reference in both the for loop's argument and as the parameter for writer.writerow(). So, once I've made the change in the for loop, how should I write back to the file?
edit: I implemented the suggestions from S. Lott and Jimmy, still the same result
edit #2: I added the "rb" and "wb" to the open() functions, per S. Lott's suggestion
import csv
#filename = 'C:/Documents and Settings/username/My Documents/PALTemplateData.xls'
csvfile = open("PALTemplateData.csv","rb")
csvout = open("PALTemplateDataOUT.csv","wb")
reader = csv.reader(csvfile)
writer = csv.writer(csvout)
changed = 0;
for row in reader:
row[-1] = row[-1].replace('/?', '?')
writer.writerow(row) #this is the line that's causing issues
changed=changed+1
print('Total URLs changed:', changed)
edit: For your reference, this is the new full traceback from the interpreter:
Traceback (most recent call last):
File "C:\Documents and Settings\g41092\My Documents\palScript.py", line 13, in <module>
for row in reader:
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
You cannot read and write the same file.
source = open("PALTemplateData.csv","rb")
reader = csv.reader(source , dialect)
target = open("AnotherFile.csv","wb")
writer = csv.writer(target , dialect)
The normal approach to ALL file manipulation is to create a modified COPY of the original file. Don't try to update files in place. It's just a bad plan.
Edit
In the lines
source = open("PALTemplateData.csv","rb")
target = open("AnotherFile.csv","wb")
The "rb" and "wb" are absolutely required. Every time you ignore those, you open the file for reading in the wrong format.
You must use "rb" to read a .CSV file. There is no choice with Python 2.x. With Python 3.x, you can omit this, but use "r" explicitly to make it clear.
You must use "wb" to write a .CSV file. There is no choice with Python 2.x. With Python 3.x, you must use "w".
Edit
It appears you are using Python3. You'll need to drop the "b" from "rb" and "wb".
Read this: http://docs.python.org/3.0/library/functions.html#open
Opening csv files as binary is just wrong. CSV are normal text files so You need to open them with
source = open("PALTemplateData.csv","r")
target = open("AnotherFile.csv","w")
The error
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
comes because You are opening them in binary mode.
When I was opening excel csv's with python, I used something like:
try: # checking if file exists
f = csv.reader(open(filepath, "r", encoding="cp1250"), delimiter=";", quotechar='"')
except IOError:
f = []
for record in f:
# do something with record
and it worked rather fast (I was opening two about 10MB each csv files, though I did this with python 2.6, not the 3.0 version).
There are few working modules for working with excel csv files from within python - pyExcelerator is one of them.
the problem is you're trying to write to the same file you're reading from. write to a different file and then rename it after deleting the original.