I am trying to import a CSV file of IP addresses into Postgres via python script. This is what I am at
Python script
since this is for testing. this is how test csv file is. CSV FILE
Also this is the error I am getting
Error
I ran same python script with text file, same error.
Also, I tried manually uploading the same file via pgadmin. No issue. so its probably something I am missing in my code.
Also, i am able to connect to DB as in the screenshot above so not connection issue for issue.
Thanks in advance.
You do not open anywhere the actual file. You are trying to iterate over the file name.
You need to read the file lines and pass those lines into execute/execute_many.
Sample code:
import csv
with open("test.csv", "r") as my_file:
readers = csv.reader(my_file)
for line in reader:
cur.execute("INSERT INTO x(y) VALUES (%s)", line)
cur.commit()
Related
I've just managed to run my python code on ubuntu, all seems to be going well. My python script writes out a .csv file every hour and I can't seem to find the .csv file.
Having the .csv file is important as I need to do research on the data. I am also using Filezilla, I would have thought the .csv would have run into there.
import csv
import time
collectionTime= datetime.now().strftime('%Y-%m-%d %H:%M:%S')
mylist= [d['Spaces'] for d in data]
mylist.append(collectionTime)
print(mylist)
with open("CarparkData.csv","a",newline="") as f:
writer = csv.writer(f)
writer.writerow(mylist)
In short, your code is outputting to wherever the file you're opening is in this line:
with open("CarparkData.csv","a",newline="") as f:
You can change this filename to the location of wherever you'd like the file to be read/written from/to. For example, data/CarparkData.csv if you had a folder named data/ within your application dedicated to holding data files.
As written in your code, writer.writerow will write the lines to both python's in-memory object of the file (instantiated with open("filename.csv"...), and the file itself (in this case, CarparkData.csv).
The way your code is structured, it won't be creating a new .csv every hour because it is using a static filename. If a file with this name did not exist at time of opening, it will create one, and if it did, it will continue to append new lines to the existing file.
I am working on a project that requires me to read a file with a .dif extension. Dif stands for data information exchange. The file opens nicely in Open Office Calc. Then you can easily save as a csv file, however when I open in Python all I get are random characters that don't make sense. Here is the last code that I tried just to see if I could read.
txt = open('C:\myfile.dif', 'rb').read()
print txt
I would even be open to programatically converting the file to csv first. before opening if someone knows how to do that. As always, any help is much appreciated. Below is a partial screenshot of what I get when I run the code.
Hadn't heard of this file format. Went and got a sample here.
I tested your method and it works fine:
>>> content = open(r"E:\sample.dif", 'rb').read()
>>> print (content)
b'TABLE\r\n0,1\r\n"EXCEL"\r\nVECTORS\r\n0,8\r\n""\r\nTUPLES\r\n0,3\r\n""\r\nDATA\r\n0,0\r\n""\r\n-1,0\r\nBOT\r\n1,0\r\n"Welcome to File Extension FYI Center!"\r\n1,0\r\n""\r\n1,0\r\n""\r\n-1,0\r\nBOT\r\n1,0\r\n""\r\n1,0\r\n""\r\n1,0\r\n""\r\n-1,0\r\nBOT\r\n1,0\r\n"ID"\r\n1,0\r\n"Type"\r\n1,0\r\n"Description"\r\n-1,0\r\nBOT\r\n0,1\r\nV\r\n1,0\r\n"ASP"\r\n1,0\r\n"Active Server Pages"\r\n-1,0\r\nBOT\r\n0,2\r\nV\r\n1,0\r\n"JSP"\r\n1,0\r\n"JavaServer Pages"\r\n-1,0\r\nBOT\r\n0,3\r\nV\r\n1,0\r\n"PNG"\r\n1,0\r\n"Portable Network Graphics"\r\n-1,0\r\nBOT\r\n0,4\r\nV\r\n1,0\r\n"GIF"\r\n1,0\r\n"Graphics Interchange Format"\r\n-1,0\r\nBOT\r\n0,5\r\nV\r\n1,0\r\n"WMV"\r\n1,0\r\n"Windows Media Video"\r\n-1,0\r\nEOD\r\n'
>>>
The question is what is in the file and how do you want to handle it. Personally I liked:
with open(r"E:\sample.dif", 'rb') as f:
for line in f:
print (line)
In the first code block, that long line that has a b'' (for bytes!) in front of it can be iterated on \r\n:
b'TABLE\r\n'
b'0,1\r\n'
b'"EXCEL"\r\n'
b'VECTORS\r\n'
b'0,8\r\n'
b'""\r\n'
b'TUPLES\r\n'
b'0,3\r\n'
b'""\r\n'
b'DATA\r\n'
b'0,0\r\n'
.
.
.
b'"Windows Media Video"\r\n'
b'-1,0\r\n'
b'EOD\r\n'
I'm supposed to upload a database for an assignment, but I'm having a problem. These are the instructions:
This application will read the mailbox data (mbox.txt) count up the
number email messages per organization (i.e. domain name of the email
address) using a database with the following schema to maintain the
counts.
CREATE TABLE Counts (org TEXT, count INTEGER) When you have run the
program on mbox.txt upload the resulting database file above for
grading. If you run the program multiple times in testing or with
different files, make sure to empty out the data before each run.
The data file for this application is the same as in previous
assignments: http://www.pythonlearn.com/code/mbox.txt.
Because the sample code is using an UPDATE statement and committing
the results to the database as each record is read in the loop, it
might take as long as a few minutes to process all the data. The
commit insists on completely writing all the data to disk every time
it is called.
The error message that it keeps sending me is:
Traceback (most recent call last):
File "<pyshell#11>", line 1, in <module>
fh = open(fname)
IOError: [Errno 2] No such file or directory: 'mbox.txt'
I saved them both in the same folder.
Can anybody help with this?
The code that I entered is here
Your code is having problem to find the file mbox.txt. It has nothing to do
with anything in the database as you did not run so far yet.
Good practice (at least during development) is to make sure, the things you
hope are true are really true. For this purpose I would use following code
which makes sure, the file really exists.
import os.path
fname = "mbox.txt"
assert os.path.exists(fname), "The file shall exist"
If you happen to run the code in situation, the file does not exist, it will throw an
AssertionError telling you what went wrong.
This exception is very practical as it will tell you quickly what assumption does not holds true and
you know, what to fix.
Your code is looking for a file called mbox.txt and not finding it. My guess is that open(fname) is looking for mbox.txt in the current directory, but the code is being run from a different directory.
Something like this might help resolve your issue:
import os
# figure out directory of the Python file
mdir = os.path.dirname(os.path.abspath(__file__))
# assuming that mbox.txt is in the same folder as the Python file,
# get the path to that file
mpath = os.path.join(mdir, 'mbox.txt')
# open the file
with open(mpath, 'r') as fh:
# ...
Another approach is using command line arguments. Perhaps there are other files like mbox.txt that you want to work with. In these cases, you could accept the path to mbox.txt as a command line option:
import argparse
argp = argparse.ArgumentParser(description='foo the mbox')
argp.add_argument('mbox_path', help='Path to mbox file')
opts = argp.parse_args()
with open(opts.mbox_path, 'r') as fh:
# ...
Or get fancier and use argparse.FileType for the type argument to argparse.add_argument.
I am using urllib.urlopen with Python 2.7 to read csv files located on an external webserver:
# Try & Except statements removed for clarity
import urllib
import csv
url = ...
csv_file = urllib.urlopen(url)
for row in csv.reader(csv_file):
do_something()
All 100+ files can be read fine, except one that has been updated recently and that returns:
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
The file is accessible here. According to my text editor, its mode is Mac (CR), as opposed to Windows (CRLF) for the other files.
I found that based on this thread, python urlopen will handle correctly all formats of newlines. Therefore, the problem is likely to come from somewhere else. I have no clue though. The file opens fine with all my text editors and my speadsheet editors.
Does any one have any idea how to diagnose the problem ?
* EDIT *
The creator of the file informed me by email that I was not the only one to experience such issues. Therefore, he decided to make it again. The code above now works fine again. Unfortunately, using a new file also means that the issue can no longer be reproduced, and the solutions tested properly.
Before closing the question, I want to thank all the stackers who dedicated some of their time to figure out a solution and post it here.
It might be a corrupt .csv file? Otherwise, this code runs perfectly.
#!/usr/bin/python
import urllib
import csv
url = "http://www.football-data.co.uk/mmz4281/1213/I1.csv"
csv_file = urllib.urlopen(url)
for row in csv.reader(csv_file):
print row
Credits to J.F. Sebastian for the .csv file.
Altough, you might want to consider sharing the specific .csv file with us? So we can try to re-create the error.
The following code runs without any error:
#!/usr/bin/env python
import csv
import urllib2
r = urllib2.urlopen('http://www.football-data.co.uk/mmz4281/1213/I1.csv')
for row in csv.reader(r):
print row
I was having the same problem with a downloaded csv.
I know the fix would be to use open with 'rU'. But I would rather not have to save the file to disk, just to open back up into a variable. That seems unnecessary.
file = open(filepath,'rU')
mydata = csv.reader(file)
So if someone has a better solution that would be nice. Stackoverflow links that got me this far:
CSV new-line character seen in unquoted field error
Open the file in universal-newline mode using the CSV Django module
I found what I actually wanted with stringIO, or cStringIO, or io:
Using Python, how do I to read/write data in memory like I would with a file?
I ended up getting io working,
import csv
import urllib2
import io
# warning its a 20MB csv
url = 'http://poweredgec.com/latest_poweredge-11g.csv'
urlRead = urllib2.urlopen(url).read()
ramFile = io.open(urlRead, mode='w')
openRamFile = open(ramFile, 'rU')
csvCurrent = csv.reader(openRamFile)
csvTuple = map(tuple, csvCurrent)
print csvTuple
I am writing this small code which is part of my Django application. It is supposed to pick up data from a DB table(MySql) and make a csv file. May be its a very simple error I am getting, but I am not able to resolve it.
Name of the file: write_to_csv.py
import csv
def createCSV():
from django.db import connection, transaction
cursor = connection.cursor()
cursor.execute("select * from avg_max_min;")
csv_writer = csv.writer(open("out.csv", "wt"), delimiter=',')
csv_writer.writerow([i[0] for i in cursor.description]) # write headers
csv_writer.writerows(cursor)
del csv_writer # this will close the CSV file
Error
Exception Value:
'module' object has no attribute 'writer'
Exception Location: C:\Python26\Lib\site-packages\django\bin\report\src\report ..\report\report_view\write_to_csv.py in createCSV, line 6
open's second argument should be wb not wt. Other than that, it looks like you are doing everything right.
If it's still not working, can you update your question with the results of doing dir(csv)? (It's most likely that you have some other module installed in your Python distribution or in the same directory as write_to_csv.py with the same name.)
I'm guessing it's a setup problem (something like this). Make sure you don't have a file named csv.py or some other weirdness that is "hiding" python's csv module.