This is my code:
#!/usr/bin/python
import MySQLdb
import csv
db = MySQLdb.connect(host="host", # The Host
user="username", # username
passwd="pwd", # password
db="databasename") # name of the data base
sqlLoadData = 'LOAD DATA LOCAL INFILE "csv?_file_name.csv" INTO TABLE tablename '
sqlLoadData += 'FIELDS TERMINATED BY "," LINES TERMINATED BY "\n"'
sqlLoadData += 'IGNORE 1 LINES'
sqlLoadData += 'ENCLOSED BY '"' ESCAPED BY "\\" '
try:
curs = db.cursor()
curs.execute(sqlLoadData)
resultSet = curs.fetchall()
except StandardError, e:
print e
db.rollback()
db.close()
I recieve the error Message : You have an error in your SQL Syntax; chekc the manual that correcpond to your Mysql Server.
When I remove the part sqlLoadData += 'ENCLOSED BY '"' ESCAPED BY "\\" ' everything work perfect. I used the last part just to remove the quote from the values.
I also tried:
cursor = mydb.cursor()
reader = csv.reader(open('Cumulative.csv', 'rb'))
reader.next() for row in reader[1:]:
cursor.execute('INSERT INTO Cumulative (C1, C2, C3, C4, C5, C6) VALUES(%s, %s, %s, %s, %s, %s)', row)
cursor.commit()
close the connection to the database.
cursor.close()
I want just to remove the quote so the integer field will support the data. so with quote "1" will be considered as a String instead of integer
Can Anyone please help me to understand this?
Thanks!
looks like you forgot to terminate the preceding line with a space or newline character. Thi sis causing a syntax error when the parser tries to understand LINESENCLOSED which obviously isn't a keyword.
sqlLoadData += 'IGNORE 1 LINES \n'
sqlLoadData += ''ENCLOSED BY '"' ESCAPED BY "\" ''
As a rule of thumb: when you're debugging, and you're able to fix you're code by removing a line, don't rule out the line immediately above
EDIT: Modified the quotes around the second line. I think it was breaking in the "enclosed by" statement.
After 2 days worth of research I found the answer:
!/usr/bin/python
import MySQLdb
import csv
db = MySQLdb.connect(host="host", # The Host
user="username", # username
passwd="pwd", # password
db="databasename") # name of the data base
cursor = connection.cursor()
Query = """ LOAD DATA LOCAL INFILE 'usrl to csv file' INTO TABLE
table_nameFIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED
BY '"' Lines terminated by '\n' IGNORE 1 LINES """
cursor.execute(Query)
connection.commit()
cursor.close()
hope it will help somebody out there.
After days and hours of searching the internet and running into all sort of errors and warnings, this worked perfectly. I hope this saves someone some time
import MySQLdb
import os
import string
db = MySQLdb.connect (host="host",
user="user",
passwd="pwd",
db="database_name",
local_infile = 1) #Grants permission to write to db from an input file. Without this you get sql Error: (1148, 'The used command is not allowed with this MySQL version')
print "\nConnection to DB established\n"
#The statement 'IGNORE 1 LINES' below makes the Python script ignore first line on csv file
#You can execute the sql below on the mysql bash to test if it works
sqlLoadData = """load data local infile 'file.csv' into table table_name FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"""
try:
curs = db.cursor()
curs.execute(sqlLoadData)
db.commit()
print "SQL execution complete"
resultSet = curs.fetchall()
except StandardError, e:
print "Error incurred: ", e
db.rollback()
db.close()
print "Data loading complete.\n"
Thanks, I hope this helps :)
Related
I need to store a pdf file in MySql. Whether I use escape_string or not, I always get the same error
b_blob = open(dir + fname_only, "rb")
myblob = b_blob.read() ####<- b'%PDF-1.4\n%\xaa\xab\xac\xad\n4 0 obj\n<<\n/Producer (Apache FOP Version 0.94)\
try:
conn = mysql.connector.connect( usual stuff )
cursor =conn.cursor(buffered=True, dictionary=True)
newblob = conn._cmysql.escape_string(myblob)
query = """INSERT INTO `mytable` (`storing`) VALUES('%s')""" %(newblob)
cursor.execute(query)
except Exception as exc:
Functions.error_handler(exc);
return
b_blob.close()
...MySQL server version for the right syntax to use near '\n%\xaa\xab\xac\xad\n4 0 obj\n<<\n/Producer (Apache FOP Version 0.94)\n/Creation' at line 1
So it looks like your problem is arriving from the quotes at the start of your string. I would consider putting double quotes around the newblob variable. Should look like this.
query = """INSERT INTO `mytable` (`storing`) VALUES("%s")""" %(newblob)
Based on Python MySQLdb execute table variable and MySQL LOAD DATA LOCAL INFILE example in python? this should work:
import pymysql, os
directory = os.path.join('path', 'to', 'directory')
filename = 'my_filename.csv'
filepath = os.path.join(directory, filename)
to_table_name = "my_table"
connection = pymysql.connect(..., local_infile=True)
with connection.cursor() as cursor:
load_statement = """
load data local infile %s
into table %s
fields terminated by ','
optionally enclosed by '"'
lines terminated by '\\n'
ignore 1 lines
"""
cursor.execute(load_statement % (filepath, to_table_name, ))
connection.commit()
connection.close
But I'm still seeing this error:
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '/path/to/directory/my_filename.csv\n into ' at line 1")
When I run this without the parameters i.e. writing the actual filepath and table name it works.
Any help would be much appreciated.
You should use the built in ability of Execute to do your string formatting also (this avoids MYSQL Injection attacks and errors) ... Rather than passing the parameters to the load_statement using % (String Interpolation) , pass that as parameters to execute
cursor.execute(load_statement , (filepath, to_table_name ))
Notice the comma instead of a %
I'm using pymysql to load a large csv file into a database, because of memory limitations im using load infile rather than insert. however after the code completes when i query the server it for the data in the table it returns an empty set.
import pymysql
conn = pymysql.connect(host = 'localhost', port = 3306, user = 'root', passwd = '', local_infile = True)
cur = conn.cursor()
cur.execute("CREATE SCHEMA IF NOT EXISTS `test`DEFAULT "
"CHARACTER SET utf8 COLLATE utf8_unicode_ci ;")
cur.execute("CREATE TABLE IF NOT EXISTS "
"`test`.`scores` ( `date` DATE NOT NULL, "
"`name` VARCHAR(15) NOT NULL,"
"`score` DECIMAL(10,3) NOT NULL);")
conn.commit()
def push(fileName = '/home/pi/test.csv', tableName = '`test`.`scores`'):
push = """LOAD DATA LOCAL INFILE "%s" INTO TABLE %s
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(date, name, score);""" % (fileName, tableName)
cur.execute(push)
conn.commit()
push()
I get some truncation warnings but no other errors or warnings to work off of. any ideas on how to fix this?
I did a few things to fix this, First I changed the config files for my sql server to allow load infile, following this MySQL: Enable LOAD DATA LOCAL INFILE. Then the problem was with the line,
LINES TERMINATED BY '\r\n'
the fix was to change it to
LINES TERMINATED BY '\n'
after that the script runs fine and is significantly faster than inserting row by row
I was wondering if anyone could help me figure out why the data that my python script has gathered is not being input into the mysql database properly
Here is my code
/*** index is a dictionary that has the struture:
index = {links:[words], links: [words],.......} ***/
#There are multiple items in words
for k, v in index.iteritems():
links = str(k)
words = str(v) [1:-1]
import MySQLdb
db = MySQLdb.connect("localhost","root","","my_db" )
cursor = db.cursor()
try:
cursor.execute(('insert into SITE(url, keywords) values("%s", "%s")' % \
(links,words)))
db.commit()
except:
db.rollback()
db.close()
Each row in my table should have two feilds: url and keywords
Instead of insterting 6 rows it only inserts two.
Please help?!
Perhaps there is a problem, because you open a new connection for every item. Then you shouldn't format values into a SQL-statement, use parameters instead. Third, you shouldn't ignore exceptions, because then, you cannot figure out, what's going wrong. The representation of a list is nothing, to work with in production code, use join instead
import logging
import MySQLdb
db = MySQLdb.connect("localhost","root","","my_db" )
for links, words in index.iteritems():
cursor = db.cursor()
try:
cursor.execute('insert into SITE(url, keywords) values(%s, %s)', (links, ','.join(words)))
db.commit()
except Exception:
logging.exception('insert went wrong')
db.rollback()
db.close()
I have a .sql file with multiple insert statements ( 1000 + ) and I want to run the statements in this file into my Oracle database.
For now, im using a python with odbc to connect to my database with the following:
import pyodbc
from ConfigParser import SafeConfigParser
def db_call(self, cfgFile, sql):
parser = SafeConfigParser()
parser.read(cfgFile)
dsn = parser.get('odbc', 'dsn')
uid = parser.get('odbc', 'user')
pwd = parser.get('odbc', 'pass')
try:
con = pyodbc.connect('DSN=' + dsn + ';PWD=' + pwd + ';UID=' + pwd)
cur = con.cursor()
cur.execute(sql)
con.commit()
except pyodbc.DatabaseError, e:
print 'Error %s' % e
sys.exit(1)
finally:
if con and cur:
cur.close()
con.close()
with open('theFile.sql','r') as f:
cfgFile = 'c:\\dbinfo\\connectionInfo.cfg'
#here goes the code to insert the contents into the database using db_call_many
statements = f.read()
db_call(cfgFile,statements)
But when i run it i receive the following error:
pyodbc.Error: ('HY000', '[HY000] [Oracle][ODBC][Ora]ORA-00911: invalid character\n (911) (SQLExecDirectW)')
But all the content of the file are only:
INSERT INTO table (movie,genre) VALUES ('moviename','horror');
Edit
Adding print '<{}>'.format(statements) before the db_db_call(cfgFile,statements) i get the results(100+):
<INSERT INTO table (movie,genre) VALUES ('moviename','horror');INSERT INTO table (movie,genre) VALUES ('moviename_b','horror');INSERT INTO table (movie,genre) VALUES ('moviename_c','horror');>
Thanks for your time on reading this.
Now it's somewhat clarified - you have a lot of separate SQL statements such as INSERT INTO table (movie,genre) VALUES ('moviename','horror');
Then, you're effectively after cur.executescript() than the current state (I have no idea if pyodbc supports that part of the DB API, but any reason, you can't just execute an execute to the database itself?
When you read a file using read() function, the end line (\n) at the end of file is read too. I think you should use db_call(cfgFile,statements[:-1]) to eliminate the end line.