I'm using the following code to query a SQL Server DB, and storing the returned results in a CSV file.
import pypyodbc
import csv
connection = pypyodbc.connect('Driver={SQL Server};'
'Server=localhost;'
'Database=testdb;')
cursor = connection.cursor()
SQLCommand = (""" SELECT A as First,
SELECT B as Second,
FROM AB """)
cursor.execute(SQLCommand)
results = cursor.fetchall()
myfile = open('test.csv', 'w')
wr = csv.writer(myfile,dialect='excel')
wr.writerow(results)
connection.close()
The SQL command is just a sample, my query contains a lot more columns, this is just for example sake.
With this code, my CSV looks like this:
But I want my CSV to look like so, and plus I want the headers to show as well, like this:
I'm guessing the formatting needs to be done within the 'csv.writer' part of the code but I cant seem to figure it out. Can someone please guide me?
You are seeing that strange output because fetchall returns multiple rows of output but you are using writerow instead of writerows to dump them out. You need to use writerow to output a single line of column headings, followed by writerows to output the actual results:
with open(r'C:\Users\Gord\Desktop\test.csv', 'w', newline='') as myfile:
wr = csv.writer(myfile)
wr.writerow([x[0] for x in cursor.description]) # column headings
wr.writerows(cursor.fetchall())
cursor.close()
connection.close()
Related
import csv
from cs50 import SQL
db = SQL("sqlite:///roster.db")
with open ("students.csv" , "r") as file :
reader = csv.DictReader(file)
record = {}
same = []
for row in reader :
n = db.execute("INSERT INTO houses(house_id , house) VALUES (?, ?)", row['id'] , row['house'])
a = db.execute("SELECT * from houses")
print(a)
the program above keeps telling me some error messages that I do not really understand
I do not know how to fix that. I did try to put the variable row['id'] directly to the value parenthesis, but I got a empty table with nothing in it.
That is the part when I ran ".schema" to get the table.
The table "name" is created in the command line argument with sqlite3 instead of running python code, is that why the error above mentioned about the "name" table?
enter image description here
Assuming the second image is the schema (better to post as text not image!), there is a typo in the REFERENCES clauses of the house and head CREATE statements. Read them carefully and critically. It will not fail on the CREATE, only when trying to insert into either of the tables.
I'm new with psycopg2 and I do have a question (which I cannot really find a respond in the Internet): Do we have any difference (for exemaple in the aspect of performance) between using copy_xxx() method and combo execute() + fetchxxx() method when we try to write the result of query into a CSV file?
...
query_str = "SELECT * FROM mytable"
cursor.execute(query_str)
with open("my_file.csv", "w+") as file:
writer = csv.writer(file)
while True:
rows = cursor.fetchmany()
if not rows:
break
writer.writerows(rows)
vs
...
query_str = "SELECT * FROM mytable"
output_query = f"COPY {query_str} TO STDOUT WITH CSV HEADER"
with open("my_file.csv", "w+") as file:
cursor.copy_expert(output_query, file)
And if I try to do a very complex query (my assumption is that we cannot simplify this query anymore for ex) with psycopg2, which method should I use? Or do you guys have any advice, please?
Many thanks!!!
COPY is faster, but if query execution time is dominant or the file is small, it won't matter much.
You don't show us how the cursor was declared. If it is an anonymous cursor, then execute/fetch will read all query data into memory upfront, leading to out of memory conditions for very large queries. If it is a named cursor, then you will individually request every row from the server, leading to horrible performance (which can be overcome by specifying a count argument to fetchmany, as the default is bizarrely set to 1)
In my use case, I have a csv stored as a string and I want to load it into a MySQL table. Is there a better way than saving the string as a file, use LOAD DATA INFILE, and then deleting the file? I find this answer but it's for JDBC and I haven't find a Python equivalent to it.
Yes what you describe is very possible! Say, for example, that your csv file has three columns:
import MySQLdb
conn = MySQLdb.connect('your_connection_string')
cur = conn.cursor()
with open('yourfile.csv','rb') as fin:
for row in fin:
cur.execute('insert into yourtable (col1,col2,col3) values (%s,%s,%s)',row)
cur.close(); conn.close()
I would like to execute this query:
select datetime(date/1000,'unixepoch','localtime') as DATE, address as RECEIVED, body as BODY from sms;
And save it's output to a .csv file in a specified directory. Usually in Ubuntu terminal it is far more easy to manually give commands to save the output of the above query to a file. But i am not familiar with Python-sqlite3. I would like to know how do i execute this query and save it's output to custom directory in a .csv file. Please help me out !
Quick and dirty:
import sqlite
db = sqlite.connect('database_file')
cursor = db.cursor()
cursor.execute("SELECT ...")
rows = cursor.fetchall()
# Itereate rows and write your CSV
cursor.close()
db.close()
Rows will be a list with all matching records, which you can then iterate and manipulate into your csv file.
If you just want to make a csv file, look at the csv module. The following page should get you going https://docs.python.org/2/library/csv.html
You can also look at the pandas module to help create the file.
I have some problems with parsing huge csv file into mysql databse.
Csv file looks like this:
ref1 data1 data2 data3...
ref1 data4 data5 data6...
ref2 data1 data2 data3 data4 data5..
ref2 data12 data13 data14
ref2 data21 data22...
.
.
.
Csv file has about 1 milion lines or about 7MB in zip file or about 150MB unzip.
My job is to parse the data from csv into mysql, but only the data/lines when references matches. Another problem is, that from multiple lines in csv i must parse it in only one line in mysql for one reference.
I tryed to do this with csv.reader and for loops on each references, but is ultra slow.
with con:
cur.execute("SELECT ref FROM users")
user=cur.fetchall()
for i in range(len(user)):
with open('hugecsv.csv', mode='rb') as f:
reader = csv.reader(f, delimiter=';')
for row in reader:
if(str(user[i][0])==row[0]):
writer.writerow(row)
So i have all references which i would like to parse, in my list user. Which is the fastes way to parse?
Please help!
The first obvious bottleneck is that you are reopening and scanning the whole CSV file for each user in your database. Doing a single pass on the csv would be faster :
# faster lookup on users
cur.execute ("select ref from users")
users = set(row[0] for row in cur.fetchall())
with open("your/file.CSV") as f:
r = reader(f)
for row in r:
if row[0] in users:
do_something_with(row)
Use:
LOAD DATA INFILE 'EF_PerechenSkollekciyami.csv' TO `TABLE_NAME` FIELDS TERMINATED BY ';'
This is an internal query command in mysql.
I don't recommend you to use tabs to separate columns, and recommend you to change this by sed to ; or something another character. But you can try with tabs too.
You haven't included all your logic. If you just want to import everything into a single table,
cur.execute("LOAD DATA INFILE 'path_to_file.csv' INTO TABLE my_table;")
MySQL does it directly. You can't get any faster than that.
Documentation