We Keep Coding

How to use PostgreSQL extensions in SQLAlchemy (specifically Flask-SQLAlchemy)?

In the website I'm making with Flask, I'm trying to "upgrade" my search bar by using the pg_trgm PostgreSQL extension to match words even after being spelled wrong. However, I'm having trouble figuring out how to use the extension's custom functions alongside SQLAlchemy.
Will I have to use raw SQL to properly perform the queries, or is there a way to do this more cleanly?

Assuming the extension is configured correctly, these functions should be accessible like any other database function. For example, here's how the similarity function might be called using SQLAlchemy core:
import sqlalchemy as sa
engine = sa.create_engine('postgresql+psycopg2:///test', echo=True, future=True)
metadata = sa.MetaData()
tbl = sa.Table('t70546926', metadata,
sa.Column('c1', sa.String, primary_key=True),
sa.Column('c2', sa.String))
tbl.drop(engine, checkfirst=True)
tbl.create(engine)
ins = sa.insert(tbl).values(c1='Alice', c2='Alison')
with engine.begin() as conn:
conn.execute(ins)
query = sa.select(sa.func.similarity(tbl.c.c1, tbl.c.c2).label('similarity_result'))
with engine.connect() as conn:
rows = conn.execute(query)
for row in rows:
print(row.similarity_result)
For Flask-SQLAlchemy, you might do something like this:
result = dbsession.query(sa.func.similarity(val1, val2)).all()

postgres pg8000 create database

I am trying to create a database with pg8000 driver of postgressql , but unable to create. Creating a db manually and then connecting to it works fine with me, but i need to create db with my code. I am getting error "sqlalchemy.exc.ProgrammingError: (pg8000.ProgrammingError)". I have tried below code for creating a db.
from sqlalchemy import create_engine
dburl = "postgresql+pg8000://user:pswd#myip:5432/postgres/"
engine = create_engine(dburl)
conn = engine.connect()
conn.execute("COMMIT")
conn.execute("CREATE DATABASE qux")
I also tried with below -
from sqlalchemy import create_engine
from sqlalchemy.engine import url
settings ={"drivername" : "postgresql+pg8000", "host" : "myip","port" : 5432,"username" : "user","password" : "pswd","database" : "MyTestDB"}
db=create_engine(url.URL(**settings))
db.execute("commit")
This is the exact Error i am getting """sqlalchemy.exc.ProgrammingError: (pg8000.ProgrammingError) ('ERROR', '25001', 'CREATE DATABASE cannot run inside a transaction block') [SQL: 'create database workDB']""""
Please suggest as to how i can create this db...

Here's a solution:
from sqlalchemy import create_engine
dburl = "postgresql+pg8000://user:pswd#myip:5432/postgres/"
engine = create_engine(dburl)
conn = engine.connect()
con.rollback() # Make sure we're not in a transaction
con.autocommit = True # Turn on autocommit
conn.execute("CREATE DATABASE qux")
con.autocommit = False # Turn autocommit back off again
The docs talk about this problem of executing commands that can't be run in a transaction. The thing is that pg8000 automatically executes a begin transaction before any execute() if there isn't already a transaction in progress. This is fine until you come to execute a command that can't be executed inside a transaction. In that case you have to enter autocommit mode, which implicitly starts a transaction before the statement and commits it afterwards, but automatically avoids doing this if (like CREATE DATABASE) the statement can't be executed within a transaction.

Calling MSSQL stored procedure from SqlAlchemy

It does not seem like SqlAlchemy supports calling stored procedures. Did anybody find a workaround for this that works with SQL Server?
Sample procedure:
CREATE PROCEDURE list_lock_set #name varchar (5), #requester varchar(30)
AS
BEGIN
SET NOCOUNT ON;
INSERT INTO list_lock (name, requester, acquired) values (#name, #requester, GETDATE())
RETURN 0
END
GO
This works:
import pyodbc
dbh = pyodbc.connect(driver=''{SQL Server}'', server=srv, database=db, uid=uid, pwd=pwd)
dbc = dbh.cursor()
dbc.execute("list_lock_set ?, ?", ['bbc', 'pyodbc'])
dbc.commit()
This does not produce an error but also but does not work:
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQL Server', echo=True)
engine.execute("list_lock_set ?, ?", ['bbc', 'sqlalchemy'])
Thank you.
EDIT: It appears the best solution is to fish out pyodbc cursor from the engine:
cursor = engine.raw_connection().cursor()
cursor.execute("list_lock_set ?, ?", ['bbc', 'using cursor'])
cursor.commit()
I can also obtain pyodbc Connection:
engine.raw_connection().connection
and set autocommit=True, but that might interfere with engine logic. Many thanks to #Batman.

To have it working in sqlalchemy I managed to do it this way:
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQL Server', echo=True)
with engine.begin() as conn:
conn.execute("exec dbo.your_proc")

I remember this giving me grief before too. From memory either session.execute() or connection.execute() worked for me. There's also a callproc() method, which is probably the right way to go. http://docs.sqlalchemy.org/en/latest/core/connections.html
Also, I've had issues in the past with MSSQL which seemed to be due to something asynchronous happening where the method was returning before the procedure was finished, which resulted in unpredictable results on the database. I found that putting a time.sleep(1) (or whatever the appropriate number is) right after the call fixed this.

Yes, Even I was facing the same issue. SQLAlchemy was executing the procedure but no actions were happening.
So I tweaked it by adding connection.execute("exec <Procedure_name>")

Following on from #MATEITA LUCIAN:
Add input parameters to the stored procedure
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQLServer', echo=True)
with engine.begin() as conn:
conn.execute('EXEC dbo.your_proc #textinput_param = "sometext", #intinput_param = 4')
For those that aren't sure about how or what to refer to with the '....//usr:passw#srv/db?driver=SQL server', echo=True... part of creating the engine I would suggest reading this and choosing Option 1 - Provide a DSN.
Retrieve output from the stored procedure
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQLServer', echo=True)
with engine.begin() as conn:
result = conn.execute('EXEC dbo.your_proc #textinput_param = "sometext", #intinput_param = 4')
for row in result:
print('text_output: %s \t int_output: %d' % (row['text_field'], row['int_field']))

sqlAlchemy does not recognise changes to DB made outside of session

Something peculiar I've noticed is that any changes committed to the DB outside of the session (such as ones made in MySQL's Workbench) are not recognised in the sqlAlchemy session. I have to close and open a new session for sqlAlchemy to recognise it.
For example, a row I deleted manually is still fetched from sqlAlchemy.
This is how I initialise the session:
engine = create_engine('mysql://{}:{}#{}/{}'.format(username, password, host, schema), pool_recycle=3600)
Session = sessionmaker(bind=engine)
session = Session()
metadata = MetaData()
How can I get sqlAlchemy to recognise them?
My sqlAlchemy version is 0.9.4 and my MySQL version is 5.5.34. We use only sqlAlchemy's Core (no ORM).

To be able to read committed data from others transactions you'll need to set transaction isolation level to READ COMMITTED. For sqlalchemy and mysql:
To set isolation level using create_engine():
engine = create_engine(
"mysql://scott:tiger#localhost/test",
isolation_level="READ COMMITTED")
To set using per-connection execution options:
connection = engine.connect()
connection = connection.execution_options(
isolation_level="READ COMMITTED")
source

SQL Alchemy: SQL syntax error on add with session

I'm currently stuck with the following error when I try to perform an add using a session with SQLAlchemy.
sqlalchemy.exc.ProgrammingError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s)' at line 1") b'INSERT INTO ctr_status (`Name`) VALUES (%s)' ('Test',)
Here is the code:
from sqlalchemy import MetaData, Table
from sqlalchemy import create_engine, orm
# Login omitted...
url = 'mysql+mysqldb://{user}:{password}#{host}:{port}/{database}'.format(user=user, password=password, host=host, port=port, database=database)
e = create_engine(url, echo=True)
metadata = MetaData()
metadata.bind = e
metadata.create_all()
ctr_status = Table('ctr_status', metadata, autoload=True)
class CTRStatus(object):
pass
orm.mapper(CTRStatus, ctr_status)
new_status = CTRStatus()
new_status.Name = 'Test'
session_maker = orm.sessionmaker(bind=e, autoflush=True, autocommit=False, expire_on_commit=True)
session = orm.scoped_session(session_maker)
session.add(new_status)
# Crash here at flush
session.flush()
I'm confused as to what I'm doing wrong. I'm currently using MySQL 5.5, Python 3.3.2 and SQLAlchemy-0.8.2.win32-py3.3.
Here are some of the links I have been following:
SQL Alchemy - Tutorial
SQL Alchemy - Sessions

I managed to get it working by switching the adapter to OurSQL and changing the url to use mysql+oursql.
I would like to add that it was surprisingly non-trivial to get OurSQL working on my setup for Python 3.3.
I had to delete oursqlx/oursql.c from the initial downloaded files such that it be auto-generated by Cython.
I also had to manually modify the setup.py because it was pointing to the wrong registry path for the mysql_root variable in setup_windowsish(). Essentially, the current implementation doesn't seem to support MySQL Server 5.5 nor does it support if the registry path is in HKEY_CURRENT_USER instead of HKEY_LOCAL_MACHINE.