So I've got about 5008 rows in a CSV file, a total of 5009 with the headers. I'm creating and writing this file all within the same script. But when i read it at the end, with either pandas pd.read_csv, or python3's csv module, and print the len, it outputs 4967. I checked the file for any weird characters that may be confusing python but don't see any. All the data is delimited by commas.
I also opened it in sublime and it shows 5009 rows not 4967.
I could try other methods from pandas like merge or concat, but if python wont read the csv correct, that's no use.
This is one method i tried.
df1=pd.read_csv('out.csv',quoting=csv.QUOTE_NONE, error_bad_lines=False)
df2=pd.read_excel(xlsfile)
print (len(df1))#4967
print (len(df2))#5008
df2['Location']=df1['Location']
df2['Sublocation']=df1['Sublocation']
df2['Zone']=df1['Zone']
df2['Subnet Type']=df1['Subnet Type']
df2['Description']=df1['Description']
newfile = input("Enter a name for the combined csv file: ")
print('Saving to new csv file...')
df2.to_csv(newfile, index=False)
print('Done.')
target.close()
Another way I tried is
dfcsv = pd.read_csv('out.csv')
wb = xlrd.open_workbook(xlsfile)
ws = wb.sheet_by_index(0)
xlsdata = []
for rx in range(ws.nrows):
xlsdata.append(ws.row_values(rx))
print (len(dfcsv))#4967
print (len(xlsdata))#5009
df1 = pd.DataFrame(data=dfcsv)
df2 = pd.DataFrame(data=xlsdata)
df3 = pd.concat([df2,df1], axis=1)
newfile = input("Enter a name for the combined csv file: ")
print('Saving to new csv file...')
df3.to_csv(newfile, index=False)
print('Done.')
target.close()
But not matter what way I try the CSV file is the actual issue, python is writing it correctly but not reading it correctly.
Edit: Weirdest part is that i'm getting absolutely no encoding errors or any errors when running the code...
Edit2: Tried testing it with nrows param in first code example, works up to 4000 rows. Soon as i specify 5000 rows, it reads only 4967.
Edit3: manually saved csv file with my data instead of using the one written by the program, and it read 5008 rows. Why is python not writing the csv file correctly?
I ran into this issue also. I realized that some of my lines had open-ended quotes, which was for some reason interfering with the reader.
So for example, some rows were written as:
GO:0000026 molecular_function "alpha-1
GO:0000027 biological_process ribosomal large subunit assembly
GO:0000033 molecular_function "alpha-1
and this led to rows being read incorrectly. (Unfortunately I don't know enough about how csvreader works to tell you why. Hopefully someone can clarify the quote behavior!)
I just removed the quotes and it worked out.
Edited: This option works too, if you want to maintain the quotes:
quotechar=None
My best guess without seeing the file is that you have some lines with too many or not enough commas, maybe due to values like foo,bar.
Please try setting error_bad_lines=True. From Pandas documentation: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html to see if it catches lines with errors in them, and my guess is that there will be 41 such lines.
error_bad_lines : boolean, default True
Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will dropped from the DataFrame that is returned. (Only valid with C parser)
The csv.QUOTE_NONE option seems to not quote fields and replace the current delimiter with escape_char + delimiter when writing, but you didn't paste your writing code, but on read it's unclear what this option does. https://docs.python.org/3/library/csv.html#csv.Dialect
Related
I have just started a data quality class in which I got zero instruction on Python but am expected to create a script. There are three instructions for my Python script:
Create a script that loads an entire CSV file and replace all the blank values to NAN
Use genfromtxt function
Write the results set into a different file
I have been working on this for a few hours, but with no previous experience with Python, I am completely stuck! This is what I have so far:
import csv
file = open(quality.csv, 'r')
csvreader = csv.reader(file)
header = next(csvreader)
print(header)
rows = []
for row in csvreader:
rows.append(row)
print(rows)
My first problem is that when I tried using genfromtxt, it would not print out the headers or the entire csv file, it would only print out a few lines. If it matters, all of the values of the csv file are ints/floats, but the headers are strings.
See here
The next problem is I have tried several different ways to replace blank values, but I was not successful. All of the blank fields in this file are in the last column. When I print out the csv in full, this is what the line looks like (I've highlighted the empty value):
See here
Finally, I have no idea what instruction #3 means. I am completely new at this with zero Python knowledge! I think I am unsure of the Python syntax and rules - which I will look into more and learn, however I only had two days to complete this assignment and I do not know anything yet! Thank you in advance.
What you did with genfromtxt seems correct already. With big data like this, terminal only shows some data from the beginning and at the end, and those 3 dots in the middle also indicates the other records you're not seeing there!
question regarding pandas:
Say I created a dataframe and generated output under separate variables, rather than printing them, how would I go about combining them back into another dataframe correctly to either send as a CSV and then upload to a DB or directly upload to a DB?
Everything works fine code wise, I just haven't really seen or know of the best practice to do this. I know we can store things in list, dict, etc
What I did was:
#imported all modules
object = df.iloc[0,0]
#For loop magic goes here
#nested for loop
#if conditions are met, do this
result = df.iloc[i, k+1]
print(object, result)
I've also stored them into a separate DataFrame trying:
df2 = pd.DataFrame({'object': object, 'result' : result}, index=[0])
df2.to_csv('output.csv', index=False, mode='a')
The only problem with that is that it appends everything to each row, most likely do to the append and perhaps not including it in the for loop. Which is odd because the raw output is EXACTLY how I'm trying to get it into a csv or into a DB
As saying though, looking to combine both values back into a dataframe for speed. I tried concat etc, but no luck, so I was wondering what the correct format would be? Thanks
So it turns out that after more research and revising, I solved my issue
Referenced this and personal revisions, this is a basis of what I did:
Empty space in between rows after using writer in python
import csv
/* Had to wrap in a for loop that is not listed and append to file while clearing it first to remove spaces in each row*/
with open('csvexample.csv', wb+, newline='') as file:
reader = csv.reader(file)
for row in reader:
print(row)
Additional supporting material:
Confused by python file mode "w+"
Ok, I get this error...:
"pandas.errors.ParserError: Error tokenizing data. C error: Expected 6 fields in line 12, saw 7"
...when trying to import a csv into a python script with pandas.read_csv():
path,Drawing_but_no_F5,Paralell_F5,Fixed,Needs_Attention,Errors
R:\13xx Original Ranch Buildings\1301 Stonehouse\1301-015\F - Bid Documents and Contract Award,Yes,No,No,No,No
R:\13xx Original Ranch Buildings\1302 Carriage House\1302-026A Carriage House, Redo North Side Landscape\F - Bid Document and Contract Award,Yes,No,No,No,No
R:\13xx Original Ranch Buildings\1302 Carriage House\1302-028\F - Bid Documents and Contract Award,Yes,No,No,No,No
R:\13xx Original Ranch Buildings\1302 Carriage House\1302-029\F - Bid Documents and Contract Award,Yes,No,No,No,No
Obviously, in the above entries, it is the third line that throws the error. Caveats include that I have to use that column as a path to process files there so changing the entry is not allowed. CSV is created elsewhere; I get it as-is.
I do want to preserve the column header.
This filepath column is used later as an index, so I would like to preserve that.
Many, many similar issues, but solutions seem very specific and I cannot get them to cooperate for my use case:
Pandas, read CSV ignoring extra commas
Solutions seem to change entry values or rely on the cells being in the last column
Commas within CSV Data
Solution involves sql tools methinks. I don't want to read the csv into sql tables...
csv file is already delimited by commas so I don't think I changing the sep value will work.. (I cannot get it to work -- yet)
Problems reading CSV file with commas and characters in pandas
Solution throws error: "for line in reader:_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)"
Not too optimistic since op had the cell value in quotes whereas I do not.
Here is a solution which is a minor modification of the accepted answer by #DSM in the last thread to which you linked (Problems reading CSV file with commas and characters in pandas).
import csv
with open('original.csv', 'r') as infile, open('fixed.csv', 'w') as outfile:
reader = csv.reader(infile)
writer = csv.writer(outfile)
for line in reader:
newline = [','.join(line[:-5])] + line[-5:]
writer.writerow(newline)
After running the above preprocessing code, you should be able to read fixed.csv using pd.read_csv().
This solution depends on knowing how many of the rightmost columns are always formatted correctly. In your example data, the rightmost five columns are always good, so we treat everything to the left of these columns as a single field, which csv.writer() wraps in double quotes.
Interesting problem, I'm using python's CSVreader to read comma delimited data from a UTF-8 formatted CSV file. It appears the reader is truncating column names when it encounters a period.
For example, here is a sample of my column names.
time,b12.76org2101.xz,b12.75org2001.xz,b11.72ogg8090.xy
Here's how I'm reading this data
def parseCSV(inputData):
file_to_open = inputData
with open(file_to_open) as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
headerLine = True
line = []
for row in csv_reader:
//column manipulation code here
And here's how CSVReader interprets those column names
time,76org2101,75org2001,72ogg8090
Here's the important bit, the code I shared is the first thing in the program that touches that CSV file. After the code has finished execution I can also verify that the CSV file itself is unchanged. The problem must lie with how CSVReader interprets periods but I'm not sure what the fix is
Here's another interesting find. Later in the program I use Pandas to read a list of identical names from a column in another file.
The data is formatted as follows
COLUMN_NAMES
b12.76org2101.xz,
b12.75org2001.xz,
b11.72ogg8090.xy,
Where COLUMN_NAMES is the CSV's header and the items below are rows.
You can see the code I use to read these values here.
data = pandas.read_csv(file_to_open)
Headers = data['COLUMN_NAMES'].tolist()
And this is how Pandas interprets those rows
76org2101
75org2001
72ogg8090
The Data is exactly the same, and we see exactly the same behavior! The column names with periods are truncated in exactly the same way.
So what's up? Because both Pandas and CSVReader have identical issues I'm tempted to think this is a python problem, but I'm not sure how to resolve it. Any ideas are appreciated!
EDIT: The issue was with my code, I was reading the wrong files which incidentally happened to have the same column names as my expected files, just without anything before or after the periods. What're the odds!
Using pd.__version__ '0.23.0' and python version 3.6.5, I get the expected results:
print(pd.read_csv('test.csv'))
COLUMN_NAMES
0 b12.76org2101.xz
1 b12.75org2001.xz
2 b11.72ogg8090.xy
headers = pd.read_csv('test.csv')['COLUMN_NAMES'].tolist()
print(headers)
['b12.76org2101.xz', 'b12.75org2001.xz', 'b11.72ogg8090.xy']
It also works if those values are columns:
pd.DataFrame(columns=headers).to_csv('test1.csv', index=None)
print(pd.read_csv('test1.csv'))
Empty DataFrame
Columns: [b12.76org2101.xz, b12.75org2001.xz, b11.72ogg8090.xy]
Index: []
Maybe try updating your version of python?
There is one answer to this question:
Getting "newline inside string" while reading the csv file in Python?
But this didn't work when I used the accepted answer.
If the answer in the above link doesn't work and you have opened multiple files during the execution of your code, go back and make sure you have closed all your previous files when you were done with them.
I had a script that opened and processed multiple files. Then at the very end, it kept throwing a _csv.Error in the same manner that Amit Pal saw.
My code runs about 500 lines and has three stages where it processes multiple files in succession. Here's the section of code that gave the error. As you can see, the code is plain vanilla:
f = open('file.csv')
fread = csv.reader(f)
for row in fread:
do something
And the error was:
for row in fread:
_csv.Error: newline inside string
So I told the script to print what the row....OK, that's not clear, here's what I did:
print row
f = open('file.csv')
fread = csv.reader(f)
for row in fread:
do something
Interestingly, what printed was the LAST LINE from one of the previous files I had opened and processed.
What made this really weird was that I used different variable names, but apparently the data was stuck in a buffer or memory somewhere.
So I went back and made sure I closed all previously opened files and that solved my problem.
Hope this helps someone.