We Keep Coding

Python mysql parameters and load data command (still not working)

Based on Python MySQLdb execute table variable and MySQL LOAD DATA LOCAL INFILE example in python? this should work:
import pymysql, os
directory = os.path.join('path', 'to', 'directory')
filename = 'my_filename.csv'
filepath = os.path.join(directory, filename)
to_table_name = "my_table"
connection = pymysql.connect(..., local_infile=True)
with connection.cursor() as cursor:
load_statement = """
load data local infile %s
into table %s
fields terminated by ','
optionally enclosed by '"'
lines terminated by '\\n'
ignore 1 lines
"""
cursor.execute(load_statement % (filepath, to_table_name, ))
connection.commit()
connection.close
But I'm still seeing this error:
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '/path/to/directory/my_filename.csv\n into ' at line 1")
When I run this without the parameters i.e. writing the actual filepath and table name it works.
Any help would be much appreciated.

You should use the built in ability of Execute to do your string formatting also (this avoids MYSQL Injection attacks and errors) ... Rather than passing the parameters to the load_statement using % (String Interpolation) , pass that as parameters to execute
cursor.execute(load_statement , (filepath, to_table_name ))
Notice the comma instead of a %

LOAD DATA LOCAL INFILE with incremental field

I have multiple unstructured txt files in a directory and I want to insert all of them into mysql; basically, the entire content of each text file should be placed into a row . In MySQL, I have 2 columns: ID (auto increment), and LastName(nvarchar(45)). I used Python to connect to MySql; used LOAD DATA LOCAL INFILE to insert the whole content. But when I run the code I see the following messages in Python console:
.
Also, when I check MySql, I see nothing but a bunch of empty rows with Ids being automatically generated.
Here is the code:
import MySQLdb
import sys
import os
result = os.listdir("C:\\Users\\msalimi\\Google Drive\\s\\Discharge_Summary")
for x in result:
db = MySQLdb.connect("localhost", "root", "Pass", "myblog")
cursor = db.cursor()
file1 = os.path.join(r'C:\\Discharge_Summary\\'+x)
cursor.execute("LOAD DATA LOCAL INFILE '%s' INTO TABLE clamp_test" %(file1,));
db.commit()
db.close()
Can someone please tell me what is wrong with the code? What is the right way to achieve my goal?

I edited my code with:
.....cursor.execute("LOAD DATA LOCAL INFILE '%s' INTO TABLE clamp_test LINES TERMINATED BY '\r' (Lastname) SET id = NULL" %(file1,))
and it worked :)

Load CSV MySQLdb - Python

I have a CSV that I'm attempting to load into a MySQL database. I'm running the following code:
import MySQLdb
con = MySQLdb.connect(host="myhost",
user="me",
passwd="mypw",
db="mydb")
cur = con.cursor()
sqlscript =r"""
DROP TABLE IF EXISTS MyTable;
CREATE TABLE MyTable
(Col1 VARCHAR(255),
Col2 VARCHAR(255),
CONSTRAINT PK_MyTable PRIMARY KEY (Col1));
LOAD DATA LOCAL INFILE 'C:\\Users\\me\\Documents\\Rec\\New Files\\mycsv.csv'
INTO TABLE MyTable
CHARACTER SET UTF8
FIELDS TERMINATED BY ','
ESCAPED BY '!'
ENCLOSED BY '"'
OPTIONALLY ENCLOSED BY '\''
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;"""
cur.execute(sqlscript)
cur.close()
This runs without error, but does not load data from my CSV file. It correctly drops the table and creates it using the script. When I then query the table, it has zero rows. What am I missing?

empty set after load data in file mysql

I'm using pymysql to load a large csv file into a database, because of memory limitations im using load infile rather than insert. however after the code completes when i query the server it for the data in the table it returns an empty set.
import pymysql
conn = pymysql.connect(host = 'localhost', port = 3306, user = 'root', passwd = '', local_infile = True)
cur = conn.cursor()
cur.execute("CREATE SCHEMA IF NOT EXISTS `test`DEFAULT "
"CHARACTER SET utf8 COLLATE utf8_unicode_ci ;")
cur.execute("CREATE TABLE IF NOT EXISTS "
"`test`.`scores` ( `date` DATE NOT NULL, "
"`name` VARCHAR(15) NOT NULL,"
"`score` DECIMAL(10,3) NOT NULL);")
conn.commit()
def push(fileName = '/home/pi/test.csv', tableName = '`test`.`scores`'):
push = """LOAD DATA LOCAL INFILE "%s" INTO TABLE %s
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(date, name, score);""" % (fileName, tableName)
cur.execute(push)
conn.commit()
push()
I get some truncation warnings but no other errors or warnings to work off of. any ideas on how to fix this?

I did a few things to fix this, First I changed the config files for my sql server to allow load infile, following this MySQL: Enable LOAD DATA LOCAL INFILE. Then the problem was with the line,
LINES TERMINATED BY '\r\n'
the fix was to change it to
LINES TERMINATED BY '\n'
after that the script runs fine and is significantly faster than inserting row by row

import csv file into Mysql Database using python

This is my code:
#!/usr/bin/python
import MySQLdb
import csv
db = MySQLdb.connect(host="host", # The Host
user="username", # username
passwd="pwd", # password
db="databasename") # name of the data base
sqlLoadData = 'LOAD DATA LOCAL INFILE "csv?_file_name.csv" INTO TABLE tablename '
sqlLoadData += 'FIELDS TERMINATED BY "," LINES TERMINATED BY "\n"'
sqlLoadData += 'IGNORE 1 LINES'
sqlLoadData += 'ENCLOSED BY '"' ESCAPED BY "\\" '
try:
curs = db.cursor()
curs.execute(sqlLoadData)
resultSet = curs.fetchall()
except StandardError, e:
print e
db.rollback()
db.close()
I recieve the error Message : You have an error in your SQL Syntax; chekc the manual that correcpond to your Mysql Server.
When I remove the part sqlLoadData += 'ENCLOSED BY '"' ESCAPED BY "\\" ' everything work perfect. I used the last part just to remove the quote from the values.
I also tried:
cursor = mydb.cursor()
reader = csv.reader(open('Cumulative.csv', 'rb'))
reader.next() for row in reader[1:]:
cursor.execute('INSERT INTO Cumulative (C1, C2, C3, C4, C5, C6) VALUES(%s, %s, %s, %s, %s, %s)', row)
cursor.commit()
close the connection to the database.
cursor.close()
I want just to remove the quote so the integer field will support the data. so with quote "1" will be considered as a String instead of integer
Can Anyone please help me to understand this?
Thanks!

looks like you forgot to terminate the preceding line with a space or newline character. Thi sis causing a syntax error when the parser tries to understand LINESENCLOSED which obviously isn't a keyword.
sqlLoadData += 'IGNORE 1 LINES \n'
sqlLoadData += ''ENCLOSED BY '"' ESCAPED BY "\" ''
As a rule of thumb: when you're debugging, and you're able to fix you're code by removing a line, don't rule out the line immediately above
EDIT: Modified the quotes around the second line. I think it was breaking in the "enclosed by" statement.

After 2 days worth of research I found the answer:
!/usr/bin/python
import MySQLdb
import csv
db = MySQLdb.connect(host="host", # The Host
user="username", # username
passwd="pwd", # password
db="databasename") # name of the data base
cursor = connection.cursor()
Query = """ LOAD DATA LOCAL INFILE 'usrl to csv file' INTO TABLE
table_nameFIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED
BY '"' Lines terminated by '\n' IGNORE 1 LINES """
cursor.execute(Query)
connection.commit()
cursor.close()
hope it will help somebody out there.

After days and hours of searching the internet and running into all sort of errors and warnings, this worked perfectly. I hope this saves someone some time
import MySQLdb
import os
import string
db = MySQLdb.connect (host="host",
user="user",
passwd="pwd",
db="database_name",
local_infile = 1) #Grants permission to write to db from an input file. Without this you get sql Error: (1148, 'The used command is not allowed with this MySQL version')
print "\nConnection to DB established\n"
#The statement 'IGNORE 1 LINES' below makes the Python script ignore first line on csv file
#You can execute the sql below on the mysql bash to test if it works
sqlLoadData = """load data local infile 'file.csv' into table table_name FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"""
try:
curs = db.cursor()
curs.execute(sqlLoadData)
db.commit()
print "SQL execution complete"
resultSet = curs.fetchall()
except StandardError, e:
print "Error incurred: ", e
db.rollback()
db.close()
print "Data loading complete.\n"
Thanks, I hope this helps :)