In my use case, I have a csv stored as a string and I want to load it into a MySQL table. Is there a better way than saving the string as a file, use LOAD DATA INFILE, and then deleting the file? I find this answer but it's for JDBC and I haven't find a Python equivalent to it.
Yes what you describe is very possible! Say, for example, that your csv file has three columns:
import MySQLdb
conn = MySQLdb.connect('your_connection_string')
cur = conn.cursor()
with open('yourfile.csv','rb') as fin:
for row in fin:
cur.execute('insert into yourtable (col1,col2,col3) values (%s,%s,%s)',row)
cur.close(); conn.close()
Related
I started created a database with postgresql and I am currently facing a problem when I want to copy the data from my csv file to my database
Here is my code:
connexion = psycopg2.connect(dbname= "db_test" , user = "postgres", password ="passepasse" )
connexion.autocommit = True
cursor = connexion.cursor()
cursor.execute("""CREATE TABLE vocabulary(
fname integer PRIMARY KEY,
label text,
mids text
)""")
with open (r'C:\mypathtocsvfile.csv', 'r') as f:
next(f) # skip the header row
cursor.copy_from(f, 'vocabulary', sep=',')
connexion.commit()
I asked to allocate 4 column to store my csv data, the problem is that datas in my csv are stored like this:
fname,labels,mids,split
64760,"Electric_guitar,Guitar,Plucked_string_instrument,Musical_instrument,Music","/m/02sgy,/m/0342h,/m/0fx80y,/m/04szw,/m/04rlf",train
16399,"Electric_guitar,Guitar,Plucked_string_instrument,Musical_instrument,Music","/m/02sgy,/m/0342h,/m/0fx80y,/m/04szw,/m/04rlf",train
...
There is comas inside my columns label and mids, thats why i get the following error:
BadCopyFileFormat: ERROR: additional data after the last expected column
Which alternativ should I use to copy data from this csv file?
ty
if the file is small, then the easiest way is to open the file in LibreOffice and save the file with a new separetor.
I usually use ^.
If the file is large, write a script to replace ," and "," on ^" and "^", respectively.
COPY supports csv as a format, which already does what you want. But to access it via psycopg2, I think you will need to use copy_expert rather than copy_from.
cursor.copy_expert('copy vocabulary from stdin with csv', f)
I'm attempting to import an sq file that already has tables into python. However, it doesn't seem to import what I had hoped. The only things I've seen so far are how to creata a new sq file with a table, but I'm looking to just have an already completed sq file imported into python. So far, I've written this.
# Python code to demonstrate SQL to fetch data.
# importing the module
import sqlite3
# connect withe the myTable database
connection = sqlite3.connect("CEM3_Slice_20180622.sql")
# cursor object
crsr = connection.cursor()
# execute the command to fetch all the data from the table emp
crsr.execute("SELECT * FROM 'Trade Details'")
# store all the fetched data in the ans variable
ans= crsr.fetchall()
# loop to print all the data
for i in ans:
print(i)
However, it keeps claiming that the Trade Details table, which is a table inside the file I've connected it to, does not exist. Nowhere I've looked shows me how to do this with an already created file and table, so please don't just redirect me to an answer about that
As suggested by Rakesh above, you create a connection to the DB, not to the .sql file. The .sql file contains SQL scripts to rebuild the DB from which it was generated.
After creating the connection, you can implement the following:
cursor = connection.cursor() #cursor object
with open('CEM3_Slice_20180622.sql', 'r') as f: #Not sure if the 'r' is necessary, but recommended.
cursor.executescript(f.read())
Documentation on executescript found here
To read the file into pandas DataFrame:
import pandas as pd
df = pd.read_sql('SELECT * FROM table LIMIT 10', connection)
There are two possibilities:
Your file is not in the correct format and therefore cannot be opened.
The SQLite file can exist anywhere on the disk e.g. /Users/Username/Desktop/my_db.sqlite , this means that you have to tell python exactly where your file is otherwise it will look inside the scripts directory, see that there is no file with the same name and therefore create a new file with the provided filename.
sqlite3.connect expects the full path to your database file or '::memory::' to create a database that exists in RAM. You don't pass it a SQL file. Eg.
connection = sqlite3.connect('example.db')
You can then read the contents of CEM3_Slice_20180622.sql as you would a normal file and execute the SQL commands against the database.
I'm using the following code to query a SQL Server DB, and storing the returned results in a CSV file.
import pypyodbc
import csv
connection = pypyodbc.connect('Driver={SQL Server};'
'Server=localhost;'
'Database=testdb;')
cursor = connection.cursor()
SQLCommand = (""" SELECT A as First,
SELECT B as Second,
FROM AB """)
cursor.execute(SQLCommand)
results = cursor.fetchall()
myfile = open('test.csv', 'w')
wr = csv.writer(myfile,dialect='excel')
wr.writerow(results)
connection.close()
The SQL command is just a sample, my query contains a lot more columns, this is just for example sake.
With this code, my CSV looks like this:
But I want my CSV to look like so, and plus I want the headers to show as well, like this:
I'm guessing the formatting needs to be done within the 'csv.writer' part of the code but I cant seem to figure it out. Can someone please guide me?
You are seeing that strange output because fetchall returns multiple rows of output but you are using writerow instead of writerows to dump them out. You need to use writerow to output a single line of column headings, followed by writerows to output the actual results:
with open(r'C:\Users\Gord\Desktop\test.csv', 'w', newline='') as myfile:
wr = csv.writer(myfile)
wr.writerow([x[0] for x in cursor.description]) # column headings
wr.writerows(cursor.fetchall())
cursor.close()
connection.close()
I would like to execute this query:
select datetime(date/1000,'unixepoch','localtime') as DATE, address as RECEIVED, body as BODY from sms;
And save it's output to a .csv file in a specified directory. Usually in Ubuntu terminal it is far more easy to manually give commands to save the output of the above query to a file. But i am not familiar with Python-sqlite3. I would like to know how do i execute this query and save it's output to custom directory in a .csv file. Please help me out !
Quick and dirty:
import sqlite
db = sqlite.connect('database_file')
cursor = db.cursor()
cursor.execute("SELECT ...")
rows = cursor.fetchall()
# Itereate rows and write your CSV
cursor.close()
db.close()
Rows will be a list with all matching records, which you can then iterate and manipulate into your csv file.
If you just want to make a csv file, look at the csv module. The following page should get you going https://docs.python.org/2/library/csv.html
You can also look at the pandas module to help create the file.
I need some help with my weather station. I would like to save all results into mysql database, but at the moment i've got all results in txt files.
Can you help me to write a script in python, to read from txt file and save into mysql?
My txt file (temperature.txt) contains data and temperature. It looks like:
2013-09-29 13:24 22.60
I'm using python script to get temperature and current time from big "result.txt" file:
#!/usr/bin/python
import time
buffer = bytes()
fh = open("/home/style/pomiar/result.txt")
for line in fh:
pass
last = line
items = last.strip().split()
fh.close();
print time.strftime("%Y-%m-%d %H:%M"), items[1]
But I would like to "print" that into mysql table. I know how to connect, but I dont know how to save data into table.
I know I need to use:
#!/usr/bin/python
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","user","password","weather" )
And I've got my database "weather" with table "temperature". Dont know if I made good table (first - datatime, second varchar (5)). And now I need python script to read from this file and save into mysql.
Thanks a lot for ur support.
Next step is simple:
from contextlib import closing
with closing(self.db.cursor()) as cur:
cur.execute("INSERT INTO table1(`measured_at`,`temp`) VALUES(%s, %s)", (measured_at, temp))
self.db.commit()
P.S. It looks like you ask this question because you didn't make your homework and didn't read via ANY python tutorial how to work with MySQL.