pyodbc not committing changes to db2 database - python

I am trying to update my db2 database using pyodbc in python. The sql statement runs normally without errors on the database directly. when I run the code below, I get no errors and the code executes successfully but when I query the database, the changes did not save.
try:
conn2 = pyodbc.connect("DRIVER={iSeries Access ODBC Driver};SYSTEM="+ Config_Main.iseriesServer +";DATABASE="+ Config_Main.iseriesDB +";UID="+ Config_Main.iseriesUser +";PWD=" + Config_Main.iseriesPass)
db2 = conn2.cursor()
for row in encludeData:
count = len(str(row[2]))
srvid = row[2]
if count < 10:
sql3 = "UPDATE SVCEN2DEV.SRVMAST SET svbrch = ? WHERE svtype != '*DCS-' AND svacct = ? AND svcid LIKE '%?' and svbrch = ?"
db2.execute(sql3, (row[4],row[1],"%" + str(srvid),row[5]))
else:
sql3 = "UPDATE SVCEN2DEV.SRVMAST SET svbrch = ? WHERE svtype != '*DCS-' AND svacct = ? AND svcid = ? and svbrch = ?"
db2.execute(sql3, (row[4],row[1],srvid,row[5]))
conn2.commit()
except pyodbc.Error as e:
logging.error(e)
I have tried setting conn2.autocommit = True. and I have also tried moving the conn2.commit() inside of the for loop to commit after each iteration. I also tried a different driver {IBM i Access ODBC Driver}
EDIT:
Sample of encludeData
['4567890001','4567890001','1234567890','1234567890','foo','bar']
After changing the except statement to grab general errors, the code above now produces this error:
IntegrityError('23000', '[23000] [IBM][System i Access ODBC Driver][DB2 for i5/OS]SQL0803 - Duplicate key value specified. (-803) (SQLExecDirectW)')

As OP found out, the application layer language, Python, may not raise specific database exceptions such as duplicate index or foreign key issues and hence will silently fail or will be logged on server side. Usually errors that affect actual SQL queries to run like incorrect identifiers and syntax errors will raise an error on client side.
Therefore, as best practice in programming it is necessary to use exception handling like Python's try/except/finally or the equivalent in other general purpose languages that interface with any external API like database connections in order to catch and properly handle runtime issues.
Below will print any exception on statements raised in the try block including connection and query execution. And regardless of success or fail will run the finally statements.
try:
conn2 = pyodbc.connect(...)
db2 = conn2.cursor()
sql = "..."
db2.execute(sql, params)
conn2.commit()
except Exception as e:
print(e)
finally:
db2.close()
conn2.close()

Related

psycopg2 why is the schema not created?

im trying to create a schema in postgres database using psycopg2.
For some reason the schema is not created and later on the code crashes because it tries to refer to the missing schema. The connection is set to auto commit mode, which definetly works because i can create a database with this specific connection.
For debugging purposes i have wrapped every step in it's own try/except statement.
Code is below, as it is right there, it does not raise any exceptions, just the follow up crashes because the schema is missing.
def createDB(dbName, connString):
conn = psycopg2.connect(connString)
conn.set_session(autocommit =True) # autocommit must be True sein, else CREATE DATABASE will fail https://www.psycopg.org/docs/usage.html#transactions-control
cursor = conn.cursor()
createDB = sql.SQL('CREATE DATABASE {};').format(
sql.Identifier(dbName)
)
createSchema = sql.SQL('CREATE SCHEMA IF NOT EXISTS schema2;')
searchpath = sql.SQL('ALTER DATABASE {} SET search_path TO public, schema2;').format(
sql.Identifier(dbName)
)
dropDB = sql.SQL('DROP DATABASE IF EXISTS {};').format(
sql.Identifier(dbName)
)
try:
cursor.execute(dropDB)
except Exception as e:
print('drop DB failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(createDB)
except Exception as e:
print('create DB failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(createSchema)
print('schema created')
except Exception as e:
print('create schema failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(searchpath)
except Exception as e:
print('set searchpath failed')
logging.error(e)
conn.close()
exit()
conn.close()
Adding an explicit commit does not do the trick either.
What am i missing?
EDIT
I have added a small screenshot with the console logs. As you can see, the code below gets executed.
EDIT 2
Out of sheer curiosity, i have tried to execute this very SQL statement in pgadmin:
CREATE SCHEMA IF NOT EXISTS schema2
and it works just fine, which shows, that my SQL is not wrong, so back to square one.
EDIT 3 -- Solution
So i have come up with a solution, thank to you #jjanes for pointing me in the right direction. This function does not connect to a specific database, but the server as a whole, since im using it to create new databases, hence the connection string looks something like this :
user=postgres password=12345 host=localhost port=5432
Which allows me to perform server level operations like create and drop database. But schemas are a Database level operation. Moving the exact same logic to the part of the code which is connected to the newly created database works like a charm.
You create the schema in the original database specified by the connect string. Once you create the new database, you need to connect to it in order to work in it. Otherwise, you are just working in the old database.

mysql.connector.cursor.execute() proceeds silently but makes no changes despite commit()

I need to execute certain MySQL commands in a python script, which is a straight-forward task. For testing purposes I have boiled down the commands to this:
import mysql.connector
script = """
CREATE DATABASE `new_project`;
CREATE TABLE `new_project`.`category` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
UNIQUE KEY `unq_name` (`name`),
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4;
"""
connection = mysql.connector.connect(
host="localhost",
port="3306",
user="root",
passwd="somepassword",
)
cursor = connection.cursor()
try:
print("begin execution")
cursor.execute(script, multi=True)
warnings = cursor.fetchwarnings()
if warnings:
for warning in warnings:
print(warning)
connection.commit()
cursor.close()
connection.close()
print("connection closed")
except mysql.connector.Error as err:
print(err.msg)
The user credentials are replaced with the correct information when running the script.
The output of this script is
begin execution
connection closed
with no errors, warnings or other output. The database new_project is not created. When I run the same MySQL commands in another interface, they work as expected and create the database and the table.
I must be overlooking something very simple.
The documentation of execute mentions that the method returns an iterator with the results for each query when multi=True. It seems the queries are not doing anything until the iterator has been processed, regardless of commit(). However, CREATE statements do not produce any results and trying to iterate over the return value of execute results in an exception: generator raised StopIteration. This is related to a bug in the connector module and has been fixed in version 8.0.13 with support of python 3.7.
The solution is now to always iterate over the return value of execute, even if no return data is expected, and to upgrade the connector module. If the upgrade is not feasible it is possible to catch the failed iteration and continue.
The fixed code (including the part for earlier versions of the connector module) now looks something like this:
try:
results = cursor.execute(script, multi=True)
try:
for result in results:
pass
except Exception as e:
pass
warnings = cursor.fetchwarnings()
if warnings:
for warning in warnings:
# handle warning
connection.commit()
cursor.close()
connection.close()
except mysql.connector.Error as err:
# handle error
Try using password="somepassword" instead of passwd="somepassword". And remove multi=True. It'll create a warning but will still execute both of your statements.

CALL multiple procedures with python

I have a linux server and I would like to run a python script every day to run mysql procedures but I do not know how to run multiple procedures and put a condition if there is an error that it sends me an email with the description of the error. Here is my script with only one procedure:
#!/usr/bin/python
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","user","password","bddname" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# execute SQL query using execute() method.
cursor.execute("CALL proc_commande_clts_detail7();")
# Fetch a single row using fetchone() method.
data = cursor.fetchone()
print "Database version : %s " % data
# disconnect from server
db.close()
Thank you for your help.
You can use callproc method to execute MySQL procedures
for proc_name in proc_name_list:
try:
result_args = cursor.callproc(proc_name, args=())
except Exception as e:
send_mail(str(e))
If you want to call multiple procedures, you can put callproc in some kind of loop and use try...catch for error handling.
wrapping them in try/except block and trigger email in except block?
Scheduling can be done through cron job.
import traceback
try:
cursor.execute("CALL proc_commande_clts_detail7();")
catch Exception as e:
email_msg = traceback.format_exc()
#send email logic

pyodbc.ProgrammingError: No results. Previous SQL was not a query, when executing multiple statements at once

i am dealing with sql server database,
where i have a table named 'table1' containing 1 column and 1 row
exp_num
0
I am trying to update the 0 value exp_num column to +1 and also return old experiment and updated experiment.
For this i am using declare statements.
DECLARE #UpdateOutput1 table (Oldexp_num int,Newexp_num int);
UPDATE get_exp_num
SET exp_num = exp_num+1
OUTPUT
DELETED.exp_num,
INSERTED.exp_num
INTO #UpdateOutput1;
select * from #UpdateOutput1
When i'm running this in SQL editor i am getting the results.
Oldexp_num Newexp_num
0 1
but if i make this same as a query, and try to use pyodbc package i am getting error.
import pyodbc
connection = pyodbc.connect() # i am getting a connection
query = "DECLARE #UpdateOutput1 table (Oldexp_num int,Newexp_num int);UPDATE get_exp_num SET exp_num = exp_num+1 OUTPUT DELETED.exp_num, INSERTED.exp_num INTO #UpdateOutput1; select Newexp_num from #UpdateOutput1;"
cursor = connection.cursor()
cursor.execute(query)
cursor.fetchone()
When im doing cursor.fetchone() , i am getting following error.
File "<ipython-input-1398-bdaba305080c>", line 1, in <module>
cursor.fetchone()
ProgrammingError: No results. Previous SQL was not a query.
Is there any error in pyodbc package? or in my query
The problem was solved by adding SET NOCOUNT ON; to the beginning of the anonymous code block. That statement suppresses the record count values generated by DML statements like UPDATE ... and allows the result set to be retrieved directly.
Whenever the server generates some informative messages this scenario may occur. The thing is that pyodbc is not ready to handle multiple result sets at the same time that it is receiving "messages" from the server. By setting "NOCOUNT ON/OFF" you may get rid of just one kind of this "messages". The server could also yield some warnings or some procedure may PRINT something and those would "break" the SQL provoking the same error.
So a more generalist solution would be to iterate over the result sets while also checking if there are more sets to retrieve and inspecting if the server has sent any messages in between. For instance:
def process_query(self, query):
try:
self.cursor.execute(query)
rowlist = []
rows = self.__extract_resultset()
while rows or self.__has_next():
if rows:
rowlist.append(rows)
rows = self.__extract_resultset()
self.cursor.commit()
return rowlist
except pyodbc.ProgrammingError as e:
raise CustomException()
except Exception as e:
raise CustomException()
def __has_next(self):
try:
has_next = self.cursor.nextset()
if self.cursor.messages:
print(f'Info Message: {self.cursor.messages}', 'info')
except pyodbc.ProgrammingError as err:
has_next = False
print(f'ProgrammingError: {err}', 'error')
return has_next
def __extract_resultset(self):
data = []
try:
records = self.cursor.fetchall()
headers = [x[0] for x in self.cursor.description]
for record in records:
data.append(dict(zip(headers, record)))
except pyodbc.ProgrammingError as err:
print(f'ProgrammingError: {err}', 'error')
return data
Also some exception handling is more likely to be mandatory since both cursor.fetchall() and cursor.nextset() are very prone to fail because we don't know before hand when a message from the server will appear and any time they do, then the fetch* operations will have been failed. In the other hand nextset will fail (instead of just returning False) when no more result sets are available.
Hope this helps!

pyMySQL: How to check if connection is already opened or close

I am getting the error InterfaceError (0, ''). Is there way in Pymysql library I can check whether connection or cursor is closed. For cursor I am already using context manager like that:
with db_connection.cursor() as cursor:
....
You can use Connection.open attribute.
The Connection.open field will be 1 if the connection is open and 0 otherwise. So you can say
if conn.open:
# do something
The conn.open attribute will tell you whether the connection has been
explicitly closed or whether a remote close has been detected.
However, it's always possible that you will try to issue a query and
suddenly the connection is found to have given out - there is no way
to detect this ahead of time (indeed, it might happen during the
process of issuing the query), so the only truly safe thing is to wrap
your calls in a try/except block
Use conn.connection in if statement.
import pymysql
def conn():
mydb=pymysql.Connect('localhost','root','password','demo_db',autocommit=True)
return mydb.cursor()
def db_exe(query,c):
try:
if c.connection:
print("connection exists")
c.execute(query)
return c.fetchall()
else:
print("trying to reconnect")
c=conn()
except Exception as e:
return str(e)
dbc=conn()
print(db_exe("select * from users",dbc))
This is how I did it, because I want to still run the query even if the connection goes down:
def reconnect():
mydb=pymysql.Connect(host='localhost',user='root',password='password',database='demo_db',ssl={"fake_flag_to_enable_tls":True},autocommit=True)
return mydb.cursor()
try:
if (c.connection.open != True):
c=reconnect() # reconnect
if c.connection.open:
c.execute(query)
return c.fetchall()
except Exception as e:
return str(e)
I think the try and catch might do the trick instead of checking cursor only.
try:
c = db_connection.cursor()
except OperationalError:
connected = False
else:
connected = True
#code here
I initially went with the solution from AKHIL MATHEW to call conn.open but later during testing found that sometimes conn.open was returning positive results even though the connection was lost. To be certain, I found I could call conn.ping() which actually tests the connection. The function also accepts an optional parameter (reconnect=True) which will cause it to automatically reconnect if the ping fails.
Of course there is a cost to this - as implied by the name, ping actually goes out to the server and tests the connection. You don't want to do this before every query, but in my case I have an AWS lambda spinning up on warm start and trying to reuse the connection, so I think I can justify testing my connection once on each warm start and reconnecting if it's been lost.

Categories