I'm really new with Python and currently I'm trying to understand how it works and I'm building small codes for practising.
I have a table contains the details of the users: called luzer.
I want to update the password field (called: JELSZO in the table) is a varbinary (1000).
The MysqlDB connection works via a pool.
So, I use this def to execute Sql queries and commit updates etc...
def execute(self, sql, args=None, commit=False):
""" Execute a sql """
# get connection form connection pool.
conn = self.pool.get_connection()
cursor = conn.cursor()
if args:
cursor.execute(sql, args)
else:
cursor.execute(sql)
if commit is True:
conn.commit()
self.close(conn, cursor)
return None
else:
res = cursor.fetchall()
self.close(conn, cursor)
return res
And this is how I try to update the password field (JELSZO)
sql_update_query = "Update luzer SET JELSZO = %s where AZON = %s" #the AZON is the userid in the table.
pas2 = testing(MySQLPool, sql_update_query, (jelszoid1, loginid, ), True) #if the commit = True then it should run the conn.commit() above.
It runs without any error but when I try to check if it commited the update succesfully then I see that nothing happend.
The password is a binary string (generated using Fernet Key).
I would really appreciate if you have any idea what could go wrong here?
I solved this bloody issue. I used a different named def for it.
def update(request, bid, *args, **kwargs):
mysql_pool = MySQLPool()
query = sql_update_query
result = mysql_pool.execute(query, (jelszoid1, loginid,), True)
print('RESULT : \n', result)
return result
while the original (def testing now I renamed to 'selection') function stayed as it was:
def selection(request, bid, *args, **kwargs):
mysql_pool = MySQLPool()
query = query1
result = mysql_pool.execute(query, (loginid,), False)
print('RESULT : \n', result)
return result
As you can see the only difference is between the parameters of the function call.
I was stupid because I disregarded the basic sql rule that the select queries do not need commit.
Related
I'm new in Python and Mysql.
I'm just trying to learn and understand how SQL and Python works together.
I'm trying to build up a database connection via SqlPool. I found the related code here:
class MySQLPool(object):
.
.
.
def execute(self, sql, args=None, commit=False):
""" Execute a sql """
# get connection form connection pool.
conn = self.pool.get_connection()
cursor = conn.cursor()
if args:
cursor.execute(sql, args) #so if there is args in the sql select then it should be run like this: (select * from table , args)
else:
cursor.execute(sql) #if there is no args then it will run the simple select statement
if commit is True:
conn.commit()
self.close(conn, cursor)
return None
else:
res = cursor.fetchall() #else it will fetch all the records
self.close(conn, cursor)
return res
Then in the main.py I'm trying to use this def:
l = 'saxonb'
def testing(request, bid, *args, **kwargs):
mysql_pool = MySQLPool()
query = """select AZON, LOGIN, JELSZO from tmrecords.luzer WHERE LOGIN='?'"""
result = mysql_pool.execute(query)
print('RESULT : \n', result)
query = """select AZON, LOGIN, JELSZO from tmrecords.luzer WHERE LOGIN='?'"""
testing(MySQLPool, query, l)
It should be found 1 record but it is coming back with an empty result.
I have tried several placeholders like ? or %s but it does not work properly and I can't see what the problem is.
It might be a del calling problem? I mean I try to call the def incorrectly?
Or placeholder problem?
Can anyone help me in this? I would really appreciate any helps. Thanks. Hexa
I recently had a Python 2.7x project where I needed to use mysql.connector to execute multiple, semicolon delineated statements in one query. This is explained nicely in the this post..
However, I needed to use mysql.connector with Twisted for my current project, which means using Twisted's excellent enterprise.adbapi module to make my new blocking database connection non-blocking.
config = {"user": username, "password": password, "host": hostname,
"database": database_name, "raise_on_warnings": True}
cp = adbapi.ConnectionPool("mysql.connector", **config)
my test statements are defined below. I apologize that they are a bit of a frivolous example, but I know the results that I expect, and it should be enough to verify that I'm getting results for multiple statements.
statement1 = "SELECT * FROM queue WHERE id = 27;"
statement2 = "SELECT * FROM order WHERE id = 1;"
statement_list = [statement1, statement2]
statements = " ".join(statement_list)
The problem comes when I now try to execute the ConnectionPool method .runQuery()
def _print_result(result):
if result:
print("this is a result")
print(result)
else:
print("no result")
reactor.stop()
d = cp.runQuery(statements, multi=True)
d.addBoth(_print_result)
this gets me the following result:
this is a result [Failure instance: Traceback: : No result set to fetch from.
How can I use Twisted's adbapi module to get the results that I know are there?
So, it turns out that when using adbapi.ConnectionPool.runQuery(), the default behavior is to send the result of the database interrogation to the cursor.fetchall() method. However, when using mysql.connector, this doesn't work, even without twisted. Instead one needs to iterate over the result set, and call fetchall() on each member of the set.
So, the way I solved this was with the following subclass.
from twisted.enterprise import adbapi
class NEWadbapiConnectionPool(adbapi.ConnectionPool):
def __init__(self, dbapiName, *connargs, **connkw):
adbapi.ConnectionPool.__init__(self, dbapiName, *connargs, **connkw)
def runMultiQuery(self, *args, **kw):
return self.runInteraction(self._runMultiQuery, *args, **kw)
def _runMultiQuery(self, trans, *args, **kw):
result = trans.execute(*args, **kw)
result_list = []
for item in result:
if item.with_rows:
result_list.append(item.fetchall())
return result_list
so now I create the following:
def _print_result(result):
if result:
print("this is a result")
print(result)
else:
print("no result")
reactor.stop()
cp = NEWadbapiConnectionPool("mysql.connector", **config)
d = cp.runMultiQuery(statements, multi=True)
d.addBoth(_print_result)
and get a list of the results for each statement.
I hope someone else finds this useful.
RunQuery always expects results. The right way to do it is to call runOperation() which does not fetch results.
If you want to use .runQuery, it expects results to fetch so you need to return something
dbpool.runQuery(
"UPDATE something SET col1=true WHERE some_id=123 RETURNING *"
)
.runOperation does not expect results
dbpool.runOperation(
"UPDATE something SET col1=true WHERE some_id=123"
)
I am using the SQL Alchemy core version 1.1 and I can't seem to be able to get transaction working inside my falcon (python) application. I have followed the documentation to my knowledge correctly.
edit: database postgresql -> psycopg2cffi
def __init__(self, *args, **kwargs):
self.__conn_url__ = settings.get_db_url()
self.db_engine = create_engine(self.__conn_url__)
self.db_engine.echo = False
self.metadata = MetaData(self.db_engine)
self.connection = self.db_engine.connect()
self.organization_types_table = Table('organization_types', self.metadata, autoload=True)
self.organization_type_names_table = Table('organization_type_names', self.metadata, autoload=True)
def post(self, json_data):
transaction = self.connection.begin()
print(transaction)
try:
results = self.organization_types_table.insert().\
values(date_created=datetime.datetime.now()).\
execute()
organization_types_id = results.inserted_primary_key
results = self.organization_type_names_table.insert().\
values(organization_types_id=organization_types_id[0],
lang=json_data['lang'],
name=json_data['name']).\
execute()
transaction.commit()
print("I didn't rollback")
return results
except:
transaction.rollback()
print("I rollback :D")
raise
If I run this code multiple times inserting the same objects. It should only work for the first time due to a index constraint not allowing duplications.
The results I will get according to my print() statements:
Transaction object -> I didn't rollback
Transaction object -> I rollback :D
Transaction object -> I rollback :D
If I look inside my tables I can clearly see that organization_types_table contains 3 records while organization_type_names contains 1. Why isn't organization_types_table not rolling back according to the transaction?
You need to do
self.connection.execute(query)
instead of
query.execute()
query.execute() executes query on the engine, which acquires a new connection, instead of using the one on which you have a transaction open.
I am trying to write a multi-threaded Python application in which a single SQlite connection is shared among threads. I am unable to get this to work. The real application is a cherrypy web server, but the following simple code demonstrates my problem.
What change or changes to I need to make to run the sample code, below, successfully?
When I run this program with THREAD_COUNT set to 1 it works fine and my database is updated as I expect (that is, letter "X" is added to the text value in the SectorGroup column).
When I run it with THREAD_COUNT set to anything higher than 1, all threads but 1 terminate prematurely with SQLite related exceptions. Different threads throw different exceptions (with no discernible pattern) including:
OperationalError: cannot start a transaction within a transaction
(occurs on the UPDATE statement)
OperationalError: cannot commit - no transaction is active
(occurs on the .commit() call)
InterfaceError: Error binding parameter 0 - probably unsupported type.
(occurs on the UPDATE and the SELECT statements)
IndexError: tuple index out of range
(this one has me completely puzzled, it occurs on the statement group = rows[0][0] or '', but only when multiple threads are running)
Here is the code:
CONNECTION = sqlite3.connect('./database/mydb', detect_types=sqlite3.PARSE_DECLTYPES, check_same_thread = False)
CONNECTION.row_factory = sqlite3.Row
def commands(start_id):
# loop over 100 records, read the SectorGroup column, and write it back with "X" appended.
for inv_id in range(start_id, start_id + 100):
rows = CONNECTION.execute('SELECT SectorGroup FROM Investment WHERE InvestmentID = ?;', [inv_id]).fetchall()
if rows:
group = rows[0][0] or ''
msg = '{} inv {} = {}'.format(current_thread().name, inv_id, group)
print msg
CONNECTION.execute('UPDATE Investment SET SectorGroup = ? WHERE InvestmentID = ?;', [group + 'X', inv_id])
CONNECTION.commit()
if __name__ == '__main__':
THREAD_COUNT = 10
for i in range(THREAD_COUNT):
t = Thread(target=commands, args=(i*100,))
t.start()
It's not safe to share a connection between threads; at the very least you need to use a lock to serialize access. Do also read http://docs.python.org/2/library/sqlite3.html#multithreading as older SQLite versions have more issues still.
The check_same_thread option appears deliberately under-documented in that respect, see http://bugs.python.org/issue16509.
You could use a connection per thread instead, or look to SQLAlchemy for a connection pool (and a very efficient statement-of-work and queuing system to boot).
I ran into the SqLite threading problem when writing a simple WSGI server for fun and learning.
WSGI is multi-threaded by nature when running under Apache.
The following code seems to work for me:
import sqlite3
import threading
class LockableCursor:
def __init__ (self, cursor):
self.cursor = cursor
self.lock = threading.Lock ()
def execute (self, arg0, arg1 = None):
self.lock.acquire ()
try:
self.cursor.execute (arg1 if arg1 else arg0)
if arg1:
if arg0 == 'all':
result = self.cursor.fetchall ()
elif arg0 == 'one':
result = self.cursor.fetchone ()
except Exception as exception:
raise exception
finally:
self.lock.release ()
if arg1:
return result
def dictFactory (cursor, row):
aDict = {}
for iField, field in enumerate (cursor.description):
aDict [field [0]] = row [iField]
return aDict
class Db:
def __init__ (self, app):
self.app = app
def connect (self):
self.connection = sqlite3.connect (self.app.dbFileName, check_same_thread = False, isolation_level = None) # Will create db if nonexistent
self.connection.row_factory = dictFactory
self.cs = LockableCursor (self.connection.cursor ())
Example of use:
if not ok and self.user: # Not logged out
# Get role data for any later use
userIdsRoleIds = self.cs.execute ('all', 'SELECT role_id FROM users_roles WHERE user_id == {}'.format (self.user ['id']))
for userIdRoleId in userIdsRoleIds:
self.userRoles.append (self.cs.execute ('one', 'SELECT name FROM roles WHERE id == {}'.format (userIdRoleId ['role_id'])))
Another example:
self.cs.execute ('CREATE TABLE users (id INTEGER PRIMARY KEY, email_address, password, token)')
self.cs.execute ('INSERT INTO users (email_address, password) VALUES ("{}", "{}")'.format (self.app.defaultUserEmailAddress, self.app.defaultUserPassword))
# Create roles table and insert default role
self.cs.execute ('CREATE TABLE roles (id INTEGER PRIMARY KEY, name)')
self.cs.execute ('INSERT INTO roles (name) VALUES ("{}")'.format (self.app.defaultRoleName))
# Create users_roles table and assign default role to default user
self.cs.execute ('CREATE TABLE users_roles (id INTEGER PRIMARY KEY, user_id, role_id)')
defaultUserId = self.cs.execute ('one', 'SELECT id FROM users WHERE email_address = "{}"'.format (self.app.defaultUserEmailAddress)) ['id']
defaultRoleId = self.cs.execute ('one', 'SELECT id FROM roles WHERE name = "{}"'.format (self.app.defaultRoleName)) ['id']
self.cs.execute ('INSERT INTO users_roles (user_id, role_id) VALUES ({}, {})'.format (defaultUserId, defaultRoleId))
Complete program using this construction downloadable at:
http://www.josmith.org/
N.B. The code above is experimental, there may be (fundamental) issues when using this with (many) concurrent requests (e.g. as part of a WSGI server). Performance is not critical for my application. The simplest thing probably would have been to just use MySql, but I like to experiment a little, and the zero installation thing about SqLite appealed to me. If anyone thinks the code above is fundamentally flawed, please react, as my purpose is to learn. If not, I hope this is useful for others.
I'm guessing here, but it looks like the reason why you are doing this is a performance concern.
Python threads aren't performant in any meaningful way for this use case. Instead, use sqlite transactions, which are super fast.
If you do all your updates in a transaction, you'll find an order of magnitude speedup.
I want to execute the sql_query given in below function and doing this way :
def get_globalid(self):
cursor = self.connect()
sql_query = "REPLACE INTO globalid (stub) VALUES ('a'); SELECT LAST_INSERT_ID() as id"
cursor = self.query(sql_query, cursor)
for row in cursor:
actionid = row[0]
return actionid
connect() function I am using to make a connection and it is defined separately .The query function i am using above is to execute any query passed to it . It is defined as :
def query(self,query,cursor):
cursor.execute(query)
self.close()
return cursor
It is not working properly . Is there anything I am missing ? Is there any mysqli function for python like that in php (multi_query) ?
mysql-python can't execute semicolon separated multiple query statement.
If you are looking for last_insert_id you can try with this:
conmy = MySQLdb.connect(host, user, passwd, db)
cursor = conmy.cursor()
sql_query = "REPLACE INTO globalid (stub) VALUES ('a')"
cursor.execute(sql)
last_insert_id = conmy.insert_id()
conmy.close()