We Keep Coding

Execute a multi-select SQL query using pyodbc

I'm trying to execute a multi-select SQL query using Pyodbc, but getting errors either about no results or no scalar variables. Since I need to create SQL variables that are used at different locations in the query, how could I get this to run in pyodbc?
Would this be feasible if I converted my SQL into a stored procedure?
It is not likely that I will be able to create the logic as a stored procedure as I do not have write access to the database.
Is there any possible way to get this type of query to run in python, or does it need to be modified in some way?
| ID | LNAME | FNAME | EMAIL |
| ----- | -------- | -------- | ------- |
| 1 | Smith | Bob | s#a.com |
| 2 | Davidson | Mike | d#a.com |
| 1 | Campbell | Brian | c#a.com |
This is what I tried so far but keep running into errors.
q = """
set ANSI_WARNINGS OFF;
declare #html varchar(MAX)
decalre #dedupedemails varchar(MAX)
decalre #esc_seq int
set #esc_seq = 5;
if object_id('tempdb.dbo.##dedupemail', 'U') is not null
drop table ##dedupemail;
with sub1 as (
select p.ID, p.LNAME, p.FNAME, p.EMAIL
from dbo.person p
),
sub2 as (
select
s1.*,
case when s1.ID = 1
then 'Yes'
else 'No'
end as IS_ADMIN
from sub1 s1
)
select distinct s2.* into ##dedupemail
from sub2 s2
where s2.IS_ADMIN = 'Yes'
set #html = 'abc';
select #dedupedemails = ltrim(stuff((
select '; ' + d.email
from ##dedupemail d
for xml path('')), 1,1,''));
select #dedupedemails as EMAIL_LIST, #html as EMAIL_BODY"""
try:
cnxn = pyodbc.connect(cnxn_str)
cursor = cnxn()
cursor.execute(q)
result = cursor.fetchall()
del cnxn
except pyodbc.Error as e:
print("Error caught: ", e)
The error:
Error caught: No results. Previous SQL was not a query.
Another error that I get refers to unknown scalars, but it always gives errors.

Why am I getting this Not all parameters were used in the SQL statement error using python?

I have a mysql database with one table that I'm trying to import from CSV using python.
The error I'm getting is:
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
But I have only 1 field in the table, so I am only using 1 parameter.
This is the table in MySQL:
desc billing_info;
+-----------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+---------+------+-----+---------+-------+
| InvoiceId | int(11) | NO | | NULL | |
+-----------+---------+------+-----+---------+-------+
This is my code:
import mysql.connector
import csv
source_dir = 'source_files/aws_bills/'
source_file = 'test_data.csv'
source = source_dir + source_file
mydb = mysql.connector.connect(user='xxxx', password='xxxx',
host='xxxx',
database='aws_bill')
cursor = mydb.cursor()
csv_data = csv.reader(source)
sql = "INSERT INTO billing_info (InvoiceId) VALUES (%i)"
for row in csv_data:
cursor.execute(sql, row)
#close the connection to the database.
mydb.commit()
cursor.close()

Your row variable has more than one value in it, maybe you meant:
for row in csv_data:
cursor.execute(sql, (row[0],)) # a one-tuple with the first element in the row..
also, the mysql connector usually wants you to use %s for any type of parameter, i.e.:
sql = "INSERT INTO billing_info (InvoiceId) VALUES (%s)"
update: your second issue is that you haven't opened the file, i.e.:
import os
import mysql.connector
import csv
# source_dir = 'source_files/aws_bills/'
# source_file = 'test_data.csv'
# source = source_dir + source_file
source = os.path.join('source_files', 'aws_bills', 'test_data.csv')
sql = "INSERT INTO billing_info (InvoiceId) VALUES (%s)"
mydb = mysql.connector.connect(user='xxxx', password='xxxx', host='xxxx', database='aws_bill')
cursor = mydb.cursor()
try:
with open(source, 'rb') as fp:
for row in csv.reader(fp):
cursor.execute(sql, (row[0],))
cursor.close()
mydb.commit()
except:
mydb.rollback()
finally:
mydb.close()

MySQL connector Python variables not registering

I'm currently using the mysql-connector-python package to execute database actions on Flask. It's been working so well until suddenly the variables don't seem to working correctly anymore. My code is here:
#bp.route('/addcart', methods=('OPTIONS', 'POST'))
def addcart():
...
userID = session.get("user_id")
reqDict = request.get_json()
itemCode = str(reqDict['itemCode'])
itemAmt = reqDict['itemAmt']
if userID is not None:
db = get_db()
cursor = db.cursor()
query = ('SELECT %s FROM cartdata WHERE id = %s')
cursor.execute(query, (itemCode, userID))
currentNum = cursor.fetchone()[0]
if currentNum is None:
stmt = ('UPDATE cartdata SET %s = 1 WHERE id = %s')
cursor.execute(stmt, (itemCode, userID))
else:
currentNum = int(currentNum) + int(itemAmt)
stmt = ('UPDATE cartdata SET %s = %s WHERE id = %s')
cursor.execute(stmt, (itemCode, currentNum, userID))
....
For some reason, I seem to having trouble with the itemCode variable. When I use it properly, like in the execution of 'query' or 'stmt', it doesn't work. Typically I will get an error saying
" You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near ''p1' = 1 WHERE id = 21'".
However, if I do this:
query = ('SELECT ' + itemCode + ' FROM cartdata WHERE id = %s')
...
stmt = ('UPDATE cartdata SET '+ itemCode +' = 1 WHERE id = %s')
...
It works properly as intended.
EDIT: I've checked my backend, and apparently the UPDATE statement does not actually update anything. So now I'm at a complete loss.
I don't understand why the connector suddenly breaks now for variables. I've checked this variables and its types, but they were the expected types. Any insight would be helpful.
My table schema for 'cartdata' looks something like this:
+-------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| p1 | int(8) | YES | | NULL | |
| p2 | int(8) | YES | | NULL | |
| p3 | int(8) | YES | | NULL | |
| p4 | int(8) | YES | | NULL | |
| p5 | int(8) | YES | | NULL | |
+-------+---------+------+-----+---------+-------+

That's because when MySQL connector injects your variables into the SQL statement, it formats them according to their type.
You can actually see it in the error message that you get:
"p1' = 1 WHERE id = 21'"
^
So probably, your SQL query looks like this:
SELECT 'p1' FROM cartdata WHERE id = someId
Which is syntactically invalid SQL...
Your second option however seems okay. Btw, it seems weird to adapt the column you want to select depending on the user's input... I'd highly recommend to validate this value with something efficient...
Details
You cannot use %s for column names since this injects a string value in your SQL query and this results in a non valid SQL syntax (column names are not string values).
As above:
SET %s = ...
Generates:
SET 'colName' = ...
which is not valid because you are attempting to affect a value to another value...
That would be the same as trying to do the following in python:
'foo' = 'bar'
or
'foo' = 4
You can use %s when setting values (using SET colName = %s) or filtering values (using WHERE colName = %s) because the type of the values in the column colName is actually a string.
As above:
WHERE colName = %s
Generates:
WHERE colName = 'fooBar'
which is valid because you filter on the values that are equal to the string fooBar.
By the way, you might want to check what
SELECT %s FROM cartdata WHERE id = %s
gives you as a result. That could result problems... Actually MySQL won't tell you anything, but you result will probably be exactly the value of itemCode. (it is valid SQL SELECT 'hello', it just returns 'hello').

How to check if table exists in DataBase using Robot Framework

I am trying to check whether table-name exists in database and it is throwing me that schema doesn't exist.I have tried to get the values from table and it is successful.Following is the code that I am trying.
***Settings***
Library DatabaseLibrary
Library Collections
***Testcases***
Connect to Vertica and Check if table exist
Connect To Database Using Custom Params vertica_python database='pmdb',user='dbadmin', password='warehouse', host='10.166.12.242', port=5433
Table Must Exist DCA_ITOC_RESOURCE_D
#${tableName} Query select table_name from tables where table_schema='OBR' AND table_name='DCA_ITOC_RESOURCE_D'
#List Should Contain Value ${tableName} DCA_ITOC_RESOURCE_D
Test result
root#hyi01lr0bsaehost92:/var/robot-tests# pybot database-tests.robot
==============================================================================
Database-Tests
==============================================================================
Connect to Vertica and Check if table exist | FAIL |
MissingSchema: Severity: ERROR, Message: Schema "information_schema" does not exist, Sqlstate: 3F000, Routine: RangeVarGetObjid, File: /scratch_a/release/svrtar1291/vbuild/vertica/Catalog/Namespace.cpp, Line: 288, SQL: u"SELECT * FROM information_schema.tables WHERE table_name='DCA_ITOC_RESOURCE_D'"
------------------------------------------------------------------------------
Database-Tests | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: /var/robot-tests/output.xml
Log: /var/robot-tests/log.html
Report: /var/robot-tests/report.html

This has worked for me after adding vertica query in assertion.py in databaselibrary module
def table_must_exist(self, tableName, sansTran=False):
"""
Check if the table given exists in the database. Set optional input `sansTran` to True to run command without an
explicit transaction commit or rollback.
For example, given we have a table `person` in a database
When you do the following:
| Table Must Exist | person |
Then you will get the following:
| Table Must Exist | person | # PASS |
| Table Must Exist | first_name | # FAIL |
Using optional `sansTran` to run command without an explicit transaction commit or rollback:
| Table Must Exist | person | True |
"""
logger.info('Executing : Table Must Exist | %s ' % tableName)
if self.db_api_module_name in ["cx_Oracle"]:
selectStatement = ("SELECT * FROM all_objects WHERE object_type IN ('TABLE','VIEW') AND owner = SYS_CONTEXT('USERENV', 'SESSION_USER') AND object_name = UPPER('%s')" % tableName)
elif self.db_api_module_name in ["sqlite3"]:
selectStatement = ("SELECT name FROM sqlite_master WHERE type='table' AND name='%s' COLLATE NOCASE" % tableName)
elif self.db_api_module_name in ["ibm_db", "ibm_db_dbi"]:
selectStatement = ("SELECT name FROM SYSIBM.SYSTABLES WHERE type='T' AND name=UPPER('%s')" % tableName)
else:
selectStatement = ("SELECT * FROM v_catalog.columns WHERE table_schema='OBR' AND table_name='%s'" % tableName)
#else:
# selectStatement = ("SELECT * FROM information_schema.tables WHERE table_name='%s'" % tableName)
num_rows = self.row_count(selectStatement, sansTran)
if num_rows == 0:
raise AssertionError("Table '%s' does not exist in the db" % tableName)

Psycopg request hangs even if cursor is closed

We use an object that keeps connection to PostgreSQL database and creates new cursors to serve requests. I observed strange behavior: even when the response was read and the cursor is closed, the request is still hanging in the database, preventing updating the table etc etc.
When the connection is closed, it disappears.
I know about ORM frameworks and maybe will end up using one of them, but I just want to understand what's happening here. Why the request is still there?
Here's the python code:
import psycopg2
def main():
conn = psycopg2.connect("dbname=tmpdb password=1 host=localhost")
cur = conn.cursor()
cur.execute("SELECT 1;")
items = cur.fetchall()
cur.close()
#uncommenting the following line solves the problem
#conn.close()
print items
while True:
pass
main()
Here's how to start the code:
>python test_loop.py
[(1,)]
Here's how to observe hanging request:
tmpdb=# SELECT datname,usename,pid,client_addr,waiting,query_start,query FROM pg_stat_activity ;
datname | usename | pid | client_addr | waiting | query_start | query
---------+----------+-------+-------------+---------+-------------------------------+------------------------------------------------------------------------------------------
tmpdb | savenkov | 530 | ::1 | f | 2013-08-12 13:56:32.652996+00 | SELECT 1;
tmpdb | savenkov | 88351 | | f | 2013-08-12 13:56:35.331442+00 | SELECT datname,usename,pid,client_addr,waiting,query_start,query FROM pg_stat_activity ;
(2 rows)

Why do you think it is blocking?
Create the table
create table t (i integer);
Now run it:
import psycopg2
def main():
conn = psycopg2.connect("dbname=cpn")
cur = conn.cursor()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
raw_input('Enter to insert')
cur.execute("insert into t (i) values (1) returning i;")
items = cur.fetchall()
conn.commit()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
raw_input('Enter to update')
cur.execute("update t set i = 2 returning i")
items = cur.fetchall()
conn.commit()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
cur.close()
while True:
pass
main()
Notice that you need to connection.commit() for it to be commited.
With that said don't do connection management. In instead use a connection pooler like Pgbouncer. It will save you from lots of complexity and frustration.
If the application runs on the same machine as the db then don't even bother. Just always close the connection as frequently as necessary. If both are in a fast intranet it is also not worth the added complexity of a connection pooler unless there is a really huge number of queries.