We Keep Coding

Executing a postgresql query with plpg-sql from sqlalchemy

I can't find examples using plpg-sql in raw SQL to be executed by sqlAlchemy these were the closest but no plpg-sql:
How to execute raw SQL in Flask-SQLAlchemy app
how to set autocommit = 1 in a sqlalchemy.engine.Connection
I've done research and I'm not sure if this is possible. I'm trying to either INSERT or UPDATE a record and there is no error. It must fail silently because there's no record created/updated in the database and I've explicitly set AutoCommit=True.
Python:
engine = db.create_engine(connstr, pool_size=20, max_overflow=0)
Session = scoped_session(sessionmaker(bind=engine, autocommit=True))
s = Session()
query = """DO $$
declare
ppllastActivity date;
percComplete numeric;
begin
select lastactivity into ppllastActivity FROM feeds WHERE email = :e and courseName=:c and provider = :prov;
IF COALESCE (ppllastActivity, '1900-01-01') = '1900-01-01' THEN
INSERT INTO feeds (email, courseName, completedratio, lastActivity, provider) VALUES (:e, :c, :p, :l, :prov);
ELSEIF ppllastActivity < :l THEN
UPDATE feeds set completedratio = :p,lastActivity = :l WHERE email = :e and courseName = :c and provider = :prov;
END if;
end; $$"""
params = {'e' : item.get('email').replace("'", "''").lower(), 'c' : item.get('courseName').replace("'", "''"), 'p' : item.get('progress'), 'l' : item.get('lastActivity'),'prov' : "ACG" }
result = s.execute(text(query),params)
I'm unable to troubleshoot since it doesn't give me any errors. Am I going down the wrong path? Should I just use psql.exe or can you do plpg-sql in raw SQL with sqlAlchemy?

While typing this question up I found a solution or a bug.
The automcommit=True doesn't work, you have to begin a transaction:
with s.begin():
result = s.execute(text(query),params)

ORA-00933: SQL command not properly ended with psycopg2

So I've been trying to get this query to run from python with psycopg2 but it gives me the error
ORA-00933: SQL command not properly ended
While if I try to run the query in the oracle DB it goes through without a problem.
All it does is get the count from one table and compare it with the count of another and returns PASS or FAIL.
If anyone could help identify what could be the cause I would be very grateful.
Here's the query:
'WITH QA AS ( SELECT COUNT(*) C FROM QAAUTO.FactInternetSales)
,DEV AS (SELECT COUNT (*) C FROM qaauto.LoyaltyProgramA)
SELECT "Records count" AS TEST_NAME
,QA.C AS QA_RECORDS
,DEV.C AS DEV_RECORDS
,CASE WHEN DEV.C > 0 AND DEV.C = QA.C THEN "PASS" ELSE "FAIL" END AS RESULT
FROM QA,DEV ; '
python code that runs it:
def generate_results(self, driver):
for tce in self.testCaseExecutors:
testExecutorId = tce["testcaseexecutorid"]
script = tce["script"]
failOutputScript = tce["failoutputscript"]
result = run_Script(driver, script)
output = None
if result == 'PASS':
stateid = 'PASS'
elif result == 'FAIL':
stateid = 'FAIL'
tcResult = (testExecutorId, stateid, output)
self.testCaseResults.append(tcResult)
def run_Script(driver, script):
result = driver.fetchall(script, {})
return result[0]
in my case script contains the query posted above

Apparently the semi-colon at the end of the query was causing the error.

Getting error 'value of type dict not supported' when calling an Oracle Stored Procedure in Python

When calling an Oracle Stored Procedure using cx_oracle, I getting the following error Python value of type dict not supported.
My code is as follows -
try:
cursor = connection.cursor()
cursor.callproc("my_proc", [fname, hash, json])
cursor.close()
connection.commit()
finally:
connection.close()
In my stored procedure I am passing in two strings (varchar2 in Oracle) and json data which in Oracle in a Clob.
Please note I am new to Python and still getting to grips with it.

It seems to work for me. If you update your question with more detail we might be able to spot your issue
cursor = connection.cursor()
cursor.execute("""
create or replace procedure my_proc (
p1 in varchar2,
p2 in varchar2,
p3 in clob,
p4 out varchar2
) as
begin
p4 := p1 || p2 || p3;
end;
""")
hash = "abc"
fname = "def"
json = "ghi"
with connection.cursor() as cursor:
myout = cursor.var(cx_Oracle.STRING)
cursor.callproc("my_proc", [fname, hash, json, myout])
print(myout.getvalue())
Output is:
defabcghi

Python MySQL escape special characters

I am using python to insert a string into MySQL with special characters.
The string to insert looks like so:
macaddress_eth0;00:1E:68:C6:09:A0;macaddress_eth1;00:1E:68:C6:09:A1
Here is the SQL:
UPGRADE inventory_server
set server_mac = macaddress\_eth0\;00\:1E\:68\:C6\:09\:A0\;macaddress\_eth1\;00\:1E\:68\:C6\:09\:A1'
where server_name = 'myhost.fqdn.com
When I execute the update, I get this error:
ERROR 1064 (42000):
You have an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near 'UPGRADE inventory_server
set server_mac = 'macaddress\_eth0\;00\:1E\:68\:C6\:09\' at line 1
The python code:
sql = 'UPGRADE inventory_server set server_mac = \'%s\' where server_name = \'%s\'' % (str(mydb.escape_string(macs)),host)
print sql
try:
con = mydb.connect(DBHOST,DBUSER,DBPASS,DB);
with con:
cur = con.cursor(mydb.cursors.DictCursor)
cur.execute(sql)
con.commit()
except:
return False
How can I insert this text raw?

This is one of the reasons you're supposed to use parameter binding instead of formatting the parameters in Python.
Just do this:
sql = 'UPGRADE inventory_server set server_mac = %s where server_name = %s'
Then:
cur.execute(sql, macs, host)
That way, you can just deal with the string as a string, and let the MySQL library figure out how to quote and escape it for you.
On top of that, you generally get better performance (because MySQL can compile and cache one query and reuse it for different parameter values) and avoid SQL injection attacks (one of the most common ways to get yourself hacked).

Welcome to the world of string encoding formats!
tl;dr - The preferred method for handling quotes and escape characters when storing data in MySQL columns is to use parameterized queries and let the MySQLDatabase driver handle it. Alternatively, you can escape quotes and slashes by doubling them up prior to insertion.
Full example at bottom of link
standard SQL update
# as_json must have escape slashes and quotes doubled
query = """\
UPDATE json_sandbox
SET data = '{}'
WHERE id = 1;
""".format(as_json)
with DBConn(*client.conn_args) as c:
c.cursor.execute(query)
c.connection.commit()
parameterized SQL update
# SQL Driver will do the escaping for you
query = """\
UPDATE json_sandbox
SET data = %s
WHERE id = %s;
"""
with DBConn(*client.conn_args) as c:
c.cursor.execute(query, (as_json, 1))
c.connection.commit()
Invalid JSON SQL
{
"abc": 123,
"quotes": "ain't it great",
"multiLine1": "hello\nworld",
"multiLine3": "hello\r\nuniverse\r\n"
}
Valid JSON SQL
{
"abc": 123,
"quotes": "ain''t it great",
"multiLine1": "hello\\nworld",
"multiLine3": "hello\\r\\nuniverse\\r\\n"
}
Python transform:
# must escape the escape characters, so each slash is doubled
# Some MySQL Python libraries also have an escape() or escape_string() method.
as_json = json.dumps(payload) \
.replace("'", "''") \
.replace('\\', '\\\\')
Full example
import json
import yaml
from DataAccessLayer.mysql_va import get_sql_client, DBConn
client = get_sql_client()
def encode_and_store(payload):
as_json = json.dumps(payload) \
.replace("'", "''") \
.replace('\\', '\\\\')
query = """\
UPDATE json_sandbox
SET data = '{}'
WHERE id = 1;
""".format(as_json)
with DBConn(*client.conn_args) as c:
c.cursor.execute(query)
c.connection.commit()
return
def encode_and_store_2(payload):
as_json = json.dumps(payload)
query = """\
UPDATE json_sandbox
SET data = %s
WHERE id = %s;
"""
with DBConn(*client.conn_args) as c:
c.cursor.execute(query, (as_json, 1))
c.connection.commit()
return
def retrieve_and_decode():
query = """
SELECT * FROM json_sandbox
WHERE id = 1
"""
with DBConn(*client.conn_args) as cnx:
cursor = cnx.dict_cursor
cursor.execute(query)
rows = cursor.fetchall()
as_json = rows[0].get('data')
payload = yaml.safe_load(as_json)
return payload
if __name__ == '__main__':
payload = {
"abc": 123,
"quotes": "ain't it great",
"multiLine1": "hello\nworld",
"multiLine2": """
hello
world
""",
"multiLine3": "hello\r\nuniverse\r\n"
}
encode_and_store(payload)
output_a = retrieve_and_decode()
encode_and_store_2(payload)
output_b = retrieve_and_decode()
print("original: {}".format(payload))
print("method_a: {}".format(output_a))
print("method_b: {}".format(output_b))
print('')
print(output_a['multiLine1'])
print('')
print(output_b['multiLine2'])
print('\nAll Equal?: {}'.format(payload == output_a == output_b))

Python example how to insert raw text:
Create a table in MySQL:
create table penguins(id int primary key auto_increment, msg VARCHAR(4000))
Python code:
#!/usr/bin/env python
import sqlalchemy
from sqlalchemy import text
engine = sqlalchemy.create_engine(
"mysql+mysqlconnector://yourusername:yourpassword#yourhostname.com/your_database")
db = engine.connect()
weird_string = "~!##$%^&*()_+`1234567890-={}|[]\;':\""
sql = text('INSERT INTO penguins (msg) VALUES (:msg)')
insert = db.execute(sql, msg=weird_string)
db.close()
Run it, examine output:
select * from penguins
1 ~!##$%^&*()_+`1234567890-={}|[]\;\':"
None of those characters were interpreted on insert.

Although I also think parameter binding should be used, there is also this:
>>> import MySQLdb
>>> example = r"""I don't like "special" chars ¯\_(ツ)_/¯"""
>>> example
'I don\'t like "special" chars \xc2\xaf\\_(\xe3\x83\x84)_/\xc2\xaf'
>>> MySQLdb.escape_string(example)
'I don\\\'t like \\"special\\" chars \xc2\xaf\\\\_(\xe3\x83\x84)_/\xc2\xaf'

SQL_CALC_FOUND_ROWS in mysql stored procedure with python

See this stored procedure
-- --------------------------------------------------------------------------------
-- Routine DDL
-- --------------------------------------------------------------------------------
DELIMITER $$
CREATE DEFINER=`root`#`localhost` PROCEDURE `get_followers`(in _user_id int,in _topic_id int,in _type int)
MAIN:BEGIN
SELECT SQL_CALC_FOUND_ROWS follow.user_id
-- COUNT(follow.user_id) AS _topic_idcount
FROM
follow
WHERE
follow.followee_user_id = _user_id
AND (topic_id = _topic_id OR topic_id = 0)
GROUP BY follow.user_id;
SELECT FOUND_ROWS() AS count;
END
When I am use test call to this stored procedure function in mysql workbench it is giving expected result as number of count.
But When I execute python code and dump the json out put of this query it is giving following result.
[{"user_id": 3}, {"user_id": 4}, {"user_id": 5}]
According to my view it is not considering SELECT FOUND_ROWS() AS count; this statement when I call SP form python code as fallow
results = execute_sp("get_followers", [user_id, topic_id, type])
here execut is my custom function.
def execute_sp( sp_name, paramaters ):
#helper method to run sp's and return results
db = Db()
cursor = db.cursor()
cursor.callproc(sp_name,paramaters)
results = cursor.fetchallDict()
cursor.close()
return results
pleas help me to solve this.....

You'll have to try this to see if it works - I can't test it at the moment...
results = cursor.fetchallDict() is returning the first result set, as far as mysqldb is concerned. I.e. the result from the first SELECT. Try adding a nextset() call, like this:
def execute_sp( sp_name, paramaters ):
#helper method to run sp's and return results
db = Db()
cursor = db.cursor()
cursor.callproc(sp_name,paramaters)
cursor.nextset() #Need the second result set in the proc
results = cursor.fetchallDict()
cursor.close()
return results
Let me know if it doesn't work.