I've got a MySql database (running using the stock Docker image) and it contains a table defined like this:
CREATE TABLE `Edits` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`context` varchar(255) NOT NULL,
`field` varchar(255) NOT NULL,
`value` text,
`username` varchar(255) DEFAULT NULL,
`requested` datetime DEFAULT NULL,
PRIMARY KEY (`id`,`context`,`field`)
) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=latin1
I'm connecting to it in a WSGI application, and my connection code looks like this:
import contextlib
import MySQLdb
mysql = MySQLdb.connect(host = 'host', db = 'db')
def _cursor():
mysql.ping(True)
return contextlib.closing(mysql.cursor())
def _exec(sql, params = None):
with _cursor() as c:
c.execute(sql, params)
def save_edits(id, context, field, value, username):
return _exec('REPLACE INTO Edits SET id = %(id)s, context = %(context)s,
field = %(field)s, value = %(value)s, username = %(username)s,
requested = UTC_TIMESTAMP()', {
'id': id,
'context': context,
'field': field,
'value': value,
'username': username,
})
When I call the save_edits function, it doesn't throw an exception, but it fails to update the database. Furthermore, attempting to run the query REPLACE INTO Edits SET id = 1, context = 'x', field = 'y', value = 'z', username = 'foo', requested = UTC_TIMESTAMP(); through a mysql shell afterwards fails with an ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction error. If I wait about 5 minutes, however, that error goes away and I can run the REPLACE INTO query against the MySql database again.
What's going on? How can I fix this error?
It turns out that autocommit is false by default, and closing the cursor without closing the connection will keep the transaction open. I changed the method to:
def _exec(sql, params = None):
with _cursor() as c:
c.execute(sql, params)
mysql.commit()
And that fixed the problem.
Related
Error Message
You have an error in your SQL syntax; check the manual that
corresponds to your MariaDB server version for the right syntax to use
near '%s' at line 1
MySQL Database Table
CREATE TABLE `tblorders` (
`order_id` int(11) NOT NULL,
`order_date` date NOT NULL,
`order_number` varchar(50) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
ALTER TABLE `tblorders`
ADD PRIMARY KEY (`order_id`),
ADD UNIQUE KEY `order_number` (`order_number`);
ALTER TABLE `tblorders`
MODIFY `order_id` int(11) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=4;
Code
mydb = mysql.connector.connect(host = "localhost", user = "root", password = "", database = "mydb")
mycursor = mydb.cursor()
sql = "Select order_id from tblorders where order_number=%s"
val = ("1221212")
mycursor.execute(sql, val)
Am I missing anything?
You must pass a list or a tuple as the arguments, but a tuple of a single value is just a scalar in parentheses.
Here are some workarounds to ensure that val is interpreted as a tuple or a list:
sql = "Select order_id from tblorders where order_number=%s"
val = ("1221212",)
mycursor.execute(sql, val)
sql = "Select order_id from tblorders where order_number=%s"
val = ["1221212"]
mycursor.execute(sql, val)
This is a thing about Python that I always find weird, but it makes a kind of sense.
In case you want to insert data you have to modify your SQL. Use INSERT instead of SELECT like this:
INSERT INTO tblorders (order_number) VALUES ("122121");
That statement will add new record to the table. Besides, in MariaDB you need to use ? instead of %s that works on Mysql database.
sql = "INSERT INTO tblorders (order_number) VALUES (?);"
val = "1231231"
mycursor.execute(sql, [val])
I have a flask server that needs to register an account. This account will have an ID as the primary key. I set the primary key to be
usr_id INTEGER NOT NULL PRIMARY KEY GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1)
when I try to save the info to the db (using python), I try
save_data = 'INSERT INTO myTable(name, email, psw) VALUES(?, ?, ?)'
param = name, email, psw #these are python vars
stmt = ibm_db.prepare(conn, save_data)#conn is the connection to db2 in python
ibm_db.execute(stmt, param)
My problem is, 'myTable' has that userid that auto increments. When I tried registering for the first time I got an error saying I was giving too few values. I figured that the missing one would be userid, so I specified which values I was giving as shown in the code above. Now I get a new error. I don't understand what I should do to make the db2 generate the number and I am not sure if that error still relates to the missing value I talked about
the error message I get in the terminal is
Exception: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0302N The value of a host variable in the EXECUTE or OPEN statemen SQLCODE=-302ange for its corresponding use. SQLSTATE=22001
with a table defined like this:
create table so.my_table(id int not null primary key generated always as identity,
name varchar(10),
email varchar(20),
psw varchar(16))";
this works just fine:
save_data_sql = 'insert into so.my_table(name, email, psw) values(?, ?, ?)'
params = 'kkuduk', 'kkuduk#ibm', 'strongPassword'
stmt = ibm_db.prepare(conn, save_data_sql)
ibm_db.execute(stmt, params)
i.e.
In [16]: ibm_db.execute(stmt, params)
In [17]: stmt = ibm_db.exec_immediate(conn,"select * from so.my_table")
In [18]: result = ibm_db.fetch_assoc(stmt)
In [19]: result
Out[19]: {'ID': 1, 'NAME': 'kkuduk', 'EMAIL': 'kkuduk#ibm', 'PSW': 'strongPassword'}
other option is to bind parametes on-by-one
stmt = ibm_db.prepare(conn, save_data_sql)
ibm_db.bind_param(stmt, 1, 'kkuduk')
ibm_db.bind_param(stmt, 2, 'kkuduk#ibm')
ibm_db.bind_param(stmt, 3, 'strongPassword')
ibm_db.execute(stmt)
An error SQL0302N could be returned e.g. if I would try a value out for range, so likely the issue is not your usr_id column, but one of 3 that you are actually trying to explicitly insert.
I want to run a parametrized mysql UPDATE SET statement in flask_sqlalchemy. Since I do not know which columns should be updated I wrote a helper function to help in writing the statement.
My model
class User(db.Model):
id = db.Column(db.Integer, primary_key=True, autoincrement= True)
username = db.Column(db.String(80), nullable=False)
email = db.Column(db.String(120), nullable=False)
def __repr__(self):
return '<User %r>' % self.username
Helper function
def run_sql(params):
# e.g: params = {'username': "testing", "email": "testing#testing.ts", "id": 1}
id = params.pop("id")
# params = {'username': "testing", "email": "testing#testing.ts"}
sets = list(map(lambda col: f"{col}=:{col}", params.keys()))
# sets = ["username=:username", "email=:email"]
sets = ", ".join(sets)
# sets = "username=:username, email=:email"
params["id"] = id
# params = {'username': "testing", "email": "testing#testing.ts", "id": 1}
sql_statement = f"""UPDATE User SET {sets} WHERE id=:id LIMIT 1"""
# sql_statement = UPDATE User SET username=:username, email=:email WHERE id=:id LIMIT 1
return sql_statement
Calling helper function
if __name__ == "__main__":
conn = engine.connect()
params = {'username': "testing", "email": "testing#testing.ts", "id": 1}
sql_statement = run_sql(params)
conn.execute(sql_statement, params)
Running the previous code generates the following exception
"sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ':username, email=:email WHERE id=:id LIMIT 1' at line 1") [SQL: 'UPDATE User SET username=:username, email=:email WHERE id=:id LIMIT 1'] [parameters: {'username': 'testing', 'email': 'testing#testing.ts', 'id': 1}] (Background on this error at: http://sqlalche.me/e/f405)"
The SQL statement looks fine to me, so the parameters. Am I missing something?
The bind params aren't MySQL style, and as you're passing in plain text to engine.execute(), SQLAlchemy isn't applying a dialect to the query before executing it.
Try this:
engine.execute("SELECT :val", {"val": 1}) # will fail, same as your query
...and then this:
engine.execute("SELECT %(val)s", {"val": 1}) # will execute
Wrapping the query with text() will let SQLAlchemy handle the proper bind style:
from sqlalchemy import text # or you can use db.text w/ flask-sqlalchemy
engine.execute(text("SELECT :val"), {"val": 1})
One other thing to note is that SQLAlchemy will automatically handle construction of the UPDATE query for you, respecting the values in the parameter dict, e.g.:
id_ = 1
params = {'username': "testing", "email": "testing#testing.ts"}
User.__table__.update().values(params).where(id=id_)
# UPDATE user SET username=%(username)s, email=%(email)s WHERE user.id = %(id_1)s
params = {'username': "testing"}
User.__table__.update().values(params).where(id=id_)
# UPDATE user SET username=%(username)s WHERE user.id = %(id_1)s
params = {'username': "testing", "unexpected": "value"}
User.__table__.update().values(params).where(id=id_)
# sqlalchemy.exc.CompileError: Unconsumed column names: unexpected
If you want to use the :named_parameter form for your SQL statement you'll need to use SQLAlchemy's text method and then call the execute method of a Connection object:
import sqlalchemy as sa
# ...
with engine.begin() as conn:
conn.exec_driver_sql("CREATE TEMPORARY TABLE temp (id varchar(10) PRIMARY KEY)")
sql = sa.text("INSERT INTO temp (id) VALUES (:id)")
params = {'id': 'foo'}
conn.execute(sql, params)
print(conn.execute(sa.text("SELECT * FROM temp")).fetchall()) # [('foo',)]
This code uses pymysql, however when i try to insert the variable title into the sql query it comes out with 'title' for example when i set title to = test the database created is 'test' is there a way to create the table without the extra quotes
import pymysql
connection = pymysql.connect(
host='localhost',
user='root',
password='',
db='comments',
)
c= connection.cursor()
sql ='''CREATE TABLE IF NOT EXISTS `%s` (
`comment_id` int(11) NOT NULL,
`parent_comment_id` int(11) NOT NULL,
`comment` varchar(200) NOT NULL,
`comment_sender_name` varchar(40) NOT NULL,
`date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8; '''
c.execute(sql, (title))
In my case I just rewrite the escape method in class 'pymysql.connections.Connection', which obviously adds "'" arround your string.
I don't know whether it is a bad idea, but there seems no better ways, if anyone knows, just let me know.
Here's my code:
from pymysql.connections import Connection, converters
class MyConnect(Connection):
def escape(self, obj, mapping=None):
"""Escape whatever value you pass to it.
Non-standard, for internal use; do not use this in your applications.
"""
if isinstance(obj, str):
return self.escape_string(obj) # by default, it is :return "'" + self.escape_string(obj) + "'"
if isinstance(obj, (bytes, bytearray)):
ret = self._quote_bytes(obj)
if self._binary_prefix:
ret = "_binary" + ret
return ret
return converters.escape_item(obj, self.charset, mapping=mapping)
config = {'host':'', 'user':'', ...}
conn = MyConnect(**config)
cur = conn.cursor()
I have usecase in whch I have to read rows having status = 0 from mysql.
Table schema:
CREATE TABLE IF NOT EXISTS in_out_analytics(
id INT AUTO_INCREMENT PRIMARY KEY,
file_name VARCHAR(255),
start_time BIGINT,
end_time BIGINT,
duration INT,
in_count INT,
out_count INT,
status INT
)
I am using this below code to read data from mysql.
persistance.py
import mysql
import mysql.connector
import conf
class DatabaseManager(object):
# global vars to storing db connection details
connection = None
def __init__(self):
self.ip = conf.db_ip
self.user_name = conf.db_user
self.password = conf.db_password
self.db_name = conf.db_name
# Initialize database only one time in application
if not DatabaseManager.connection:
self.connect()
self.cursor = DatabaseManager.connection.cursor()
self.create_schema()
def connect(self):
try:
DatabaseManager.connection = mysql.connector.connect(
host= self.ip,
database = self.db_name,
user = self.user_name,
password = self.password
)
print(f"Successfully connected to { self.ip } ")
except mysql.connector.Error as e:
print(str(e))
def create_schema(self):
# Create database
# sql = f"CREATE DATABASE { self.db_name} IF NOT EXIST"
# self.cursor.execute(sql)
# Create table
sql = """
CREATE TABLE IF NOT EXISTS in_out_analytics(
id INT AUTO_INCREMENT PRIMARY KEY,
file_name VARCHAR(255),
start_time BIGINT,
end_time BIGINT,
duration INT,
in_count INT,
out_count INT,
status INT
)"""
self.cursor.execute(sql)
def read_unprocessed_rows(self):
sql = "SELECT id, start_time, end_time FROM in_out_analytics WHERE status=0;"
self.cursor.execute(sql)
result_set = self.cursor.fetchall()
rows = []
for row in result_set:
id = row[0]
start_time = row[1]
end_time = row[2]
details = {
'id' : id,
'start_time' : start_time,
'end_time' : end_time
}
rows.append(details)
return rows
test.py
import time
from persistance import DatabaseManager
if __name__ == "__main__":
# Rows which are inserted after application is started do not get processed if
# 'DatabaseManager' is defined here
# dm = DatabaseManager()
while True:
# Rows which are inserted after application is started do get processed if
# 'DatabaseManager' is defined here
dm = DatabaseManager()
unprocessed_rows = dm.read_unprocessed_rows()
print(f"unprocessed_rows: { unprocessed_rows }")
time.sleep(2)
Problem:
The problem is, when I define database object dm = DatabaseManager() above the while loop, then any new row which is inserted after the application is started do not get processed and if I define the dm = DatabaseManager() inside the while loop then the rows which are inserted even after application is started gets processed.
What is the problem with the above code?
Ideally, we should make only one object of DatabaseManager as this class is creating a connection with MySQL. Hence creating a connection with any database should be the ideal case.
Making an assumption here, as I cannot test it myself.
tl;dr: Add DatabaseManager.connection.commit() to your read_unprocessed_rows
When you execute your SELECT statement, a transaction is created implicitly, using the default isolation level REPEATABLE READ. That creates a snapshot of the database at that point in time and all consecutive reads in that transaction will read from the snapshot established during the first read. The effects of different isolation levels are described here. To refresh the snapshot in REPEATABLE READ, you can commit your current transaction before executing the next statement.
So, when you instantiate your DatabaseManager inside your loop, each SELECT starts a new transaction on a new connection, hence has a fresh snapshot every time. When instantiating your Databasemanager outside the loop, the transaction created by the first SELECT keeps the same snapshot for all consecutive SELECTs and updates from outside that transaction remain invisible.