I am using Python and a MySQL database and am attempting to itterate through rows in a CSV file and insert them in my database. I have the following:
import mysql.connector
import pandas as pd
mydb = mysql.connector.connect(
host="localhost",
user="root",
passwd="root",
database="mydb")
cursor = mydb.cursor()
cursor.execute("SET FOREIGN_KEY_CHECKS=0")
csv_data = pd.read_csv("file path")
sql = "INSERT INTO table (ID, SecondID, StartDate, EndDate) VALUES (%s, %s, %s, %s)"
for index, row in csv_data.iterrows():
cursor.execute(sql, row)
cursor.execute("SET FOREIGN_KEY_CHECKS=1")
mydb.commit()
cursor.close()
mydb.close()
I can't see what's wrong with the SQL.
Getting the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near '%s, %s, %s, %s)'
NOTE - The rest of the code seems to work okay and the SQL works fine if I insert specific values but when I try to use the %s construct it fails yet other responses I have seen appear to recommend this as the correct syntax.
Please help- what am I doing wrong?
I think you better use pandas to_sql function.
I'm not sure whether mysql.connector works so i'll use sqlalchemy.
It looks like that:
ENGINE = sqlalchemy.create_engine('mysql+pymysql://root:root#localhost:3306/mydb')
with ENGINE.connect() as connection:
ENGINE.execute("SET FOREIGN_KEY_CHECKS=0")
csv_data.to_sql('table_name', connection, if_exists='append', index=False)
ENGINE.execute("SET FOREIGN_KEY_CHECKS=1")
Look like that the problem is that you are invoking the query without scape the value.
The execute function its getting a class and not an array
for index, row in csv_data.iterrows():
cursor.execute(sql, row)
You should generate an array with all the values and then invoke the query.
Something like:
for index, row in csv_data.iterrows():
params = map(lambda x : x.value, row)
cursor.execute(sql,params)
Be carefull, the size of the array has to be the same size as the values params.
In this case 4
Thank you that was very helpful, I made one minor change and it works perfectly now. Here is the final solution I used:
import pandas as pd
import sqlalchemy
engine = sqlalchemy.create_engine('mysql+pymysql://root:root#localhost:3306/mydb')
csv_data = pd.read_csv("file path")
engine.execute("SET FOREIGN_KEY_CHECKS=0")
with engine.connect() as connection:
csv_data.to_sql('table', connection, if_exists='append', index=False)
engine.execute("SET FOREIGN_KEY_CHECKS=1")
Related
I want to use prepared statements to remove a row from a table, but it results in an error: mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
Relevant code:
db = mysql.connector.connect(
host='localhost',
user='user',
database='web_board',
password='password',
auth_plugin='mysql_native_password'
)
crs = db.cursor()
# construct the query and remove post from the database
query = 'DELETE FROM posts WHERE postid=%s'
crs.execute(query, tuple(request.data))
db.commit()
crs.close()
db.close()
request.data looks like this: b'9b23f24e-ff4d-4113-85ae-ff8a4a5de3be'
As the documentation states, to use prepared statements, you should instantiate your cursor with following config:
crs = db.cursor(prepared=True)
Prepared statements executed with MySQLCursorPrepared can use the format (%s) or qmark (?) parameterization style.
I'm trying to create a database with the name a user will provide. As far as I know the correct way is to use the second argument of execute().
So I did as follows:
import psycopg2
conn = psycopg2.connect(host="...", dbname="...",
user="...", password="...", port='...')
cursor = conn.cursor()
query = ''' CREATE DATABASE %s ;'''
name = 'stackoverflow_example_db'
conn.autocommit = True
cursor.execute(query, (name,))
cursor.close()
conn.close()
And I got this error:
psycopg2.errors.SyntaxError: syntax error at or near "'stackoverflow_example_db'"
LINE 1: CREATE DATABASE 'stackoverflow_example_db' ;
I need to do this statement avoiding SQL injection, so using the second argument is a must.
You can't pass values as second argument of execute(), if the statement is a CREATE DATABASE one.
As pointed out by unutbu one way to approach this is using the psycopg2.sql submodule and use identifiers to build the statement avoiding SQL injection.
The code:
import psycopg2
from psycopg2 import sql
conn = psycopg2.connect(host="...", dbname="...",
user="...", password="...", port='...')
cursor = conn.cursor()
query = ''' CREATE DATABASE {} ;'''
name = 'stackoverflow_example_db'
conn.autocommit = True
cursor.execute(sql.SQL(query).format(
sql.Identifier(name)))
cursor.close()
conn.close()
Other aditional observations:
format() do not work with %s, use {} instead
Autocommit mode is a must for this statement to work
The specified connection user needs creation privileges
I am trying to update the data from 'Active' to 'Retired by loop through a list of devices from the specific text file.
Somehow, however, it does not filter the list of devices from the text file and update the corresponding data, making no changes to the database at all.
Could it have something to do with my for statement, or mysql statement that I came up with? Regardless of how many times I fix MYSQL, it still results the same.
What could be the problem?
Please take a look at the code below and see if there is any mistake I have made with regards to MYSQL-wise or Python-wise.
Thank you in advance for your great help. Much appreciated.
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=############;'
'Database=########;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT id, device_id, model_number, serial_number_1,\
status_1, user_name_1 FROM [Footprint].[fpscdb001_cmdb_004].[desktop]')
results = []
with open('H:\list.txt') as inputfile:
results = inputfile.read().splitlines()
SQL = """UPDATE [Footprint].[fpscdb001_cmdb_004].[desktop]
SET status_1 = "Retired"
WHERE device_id == %s"""
try:
for i in results:
cursor.execute(SQL, results[i])
cursor.commit()
# print(rowcount)
except:
conn.rollback()
finally:
conn.close()
It looks like the problem is both your SQL and your Python.
There is a problem with your SQL at this part: WHERE device_id == %s. In SQL, there is no ==. Instead, you use a single = to both set and check values. You should use WHERE device_id = ?.
In addition, you're using %s as a placeholder in your query. I'm not familiar with pyodbc, but a quick check of the docs looks like you should be using the ? as a placeholder.
So try this:
SQL = """UPDATE [Footprint].[fpscdb001_cmdb_004].[desktop]
SET status_1 = "Retired"
WHERE device_id = ?"""
Building on the answer that #RToyo wrote, you may be able to do this a little more quickly
we can build a list of "?" placeholders in the SQL, and then pass each item safely to the ODBC holder, using the * notation to explode the array of device id's into the ODBC execute() function. This allows you to both execute only one query, and do it securely, too
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=############;'
'Database=########;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT id, device_id, model_number, serial_number_1,\
status_1, user_name_1 FROM [Footprint].[fpscdb001_cmdb_004].[desktop]')
results = []
with open('H:\list.txt') as inputfile:
results = inputfile.read().splitlines()
SQL = """UPDATE [Footprint].[fpscdb001_cmdb_004].[desktop]
SET status_1 = "Retired"
WHERE device_id in ({})""".format(("?, " * len(results))[0:-2])
try:
if len(results) > 0:
cursor.execute(SQL, *results)
except:
conn.rollback()
finally:
conn.close()
Hope this helps someone.
I'm currently working on a real-time json via python
my idea :
- extract and wrangle json via pandas python (did it : about 3720 x 15 df)
- convert dataframe to tables in mysql (using pymysql on windows)
my code :
dfc=dfc.values.tolist()
connection = pymysql.connect(host='localhost',
user='root',
password='',
db="decaux",
charset='utf8mb4')
with connection.cursor() as cursor:
for rows in dfc:
sql7="insert into contracts(id,contract_name,commercial_name) values(%s, %s, %s)"
data = (rows[0],rows[1],rows[2])
cursor.execute(sql7,data)
but nothing happen...
what did I do wrong ?
Is there a better idea ?
Here is my code:
from urllib.request import urlopen
from bs4 import BeautifulSoup as bs
import re
import pymysql
resp = urlopen("https://en.wikipedia.org/wiki/Main_Page").read().decode("utf-8")
soup = bs(resp ,"html.parser")
listUrls = soup.findAll("a", href=re.compile("^/wiki/"))
for url in listUrls:
if not re.search('\.(jpg|JPG)$', url['href']):
conn = pymysql.connect(
host='127.0.0.1',
user='root',
password='',
db='wikiurl',
charset='utf8mb4'
)
try:
with conn.cursor() as cursor:
sql = "insert into 'wikiurl'('urlname','urlhref') VALUES (%s , %s)"
cursor.execute(sql,(url.get_text(), "https://en.wikipedia.org" + url["href"]))
conn.commit()
finally:
conn.close()
Error:
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''wikiurl'('urlname','urlhref') VALUES ('Wikipedia' , 'https://en.wikipedia.org/w' at line 1")
First of all, I recommend giving whitespace the utmost attention to detail.
Try this:
sql = "INSERT INTO wikiurl (urlname, urlhref) VALUES (%s, %s)"
Also notice that single quotation marks are not necessary around the table name. See: MySQL Insert documentation.
Edit: And you don't need quotation marks around the column names.
I think your sql syntax has some error,but it is not easy to debug it.
I recommend you use this method to print what the real sql string that is sent to mysql server.pymysql manual above:
mogrify(self, query, args=None)
'''Returns the exact string that is sent to the database by calling the
execute() method.
This method follows the extension to the DB API 2.0 followed by Psycopg.'''
eg:
you can use
print cursor.mogrify(sql,(url.get_text(), "https://en.wikipedia.org" + url["href"]))
good luck!