I'm totally new using sqlalchemy and postgresql. I read this tutorial to build the following piece of code :
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy import engine
def connect(user, password, db, host='localhost', port=5432):
'''Returns a connection and a metadata object'''
# We connect with the help of the PostgreSQL URL
# postgresql://federer:grandestslam#localhost:5432/tennis
url = 'postgresql://{}:{}#{}:{}/{}'
url = url.format(user, password, host, port, db)
# The return value of create_engine() is our connection object
con = sqlalchemy.create_engine(url, client_encoding='utf8')
# We then bind the connection to MetaData()
meta = sqlalchemy.MetaData(bind=con, reflect=True)
return con, meta
con, meta = connect('federer', 'grandestslam', 'tennis')
con
engine('postgresql://federer:***#localhost:5432/tennis')
meta
MetaData(bind=Engine('postgresql://federer:***#localhost:5432/tennis'))
When running it I have this error :
File "test.py", line 22, in <module>
engine('postgresql://federer:***#localhost:5432/tennis')
TypeError: 'module' object is not callable
what should I do ? thanks !
So, your problem is happening because you've made this call:
from sqlalchemy import engine
And then you've used this later in the file:
engine('postgresql://federer:***#localhost:5432/tennis')
Strangely, in that section, you have some statements that are just con and meta with no assignments or calls or anything. I'm not sure what you're doing there. I would suggest that you check out SQLalchemy's page on engine and connection use to help get you sorted.
It will of course depend on exactly how you've set up your database. I used the declarative_base module in one of my projects, so my process of setting up a session to connect to my DB looks like this:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# Connect to Database and create database session
engine = create_engine('postgresql://catalog:catalog#localhost/menus')
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
And in my database setup file, I've assigned:
Base = declarative_base()
But you'll have to customize it a bit to your particular setup. I hope that helps.
Edit: I see now where those calls to con and meta were coming from, as well as your other confusing lines, it's part of the tutorial you linked to. What he was doing in that tutorial was using the Python interpreter in command line. I'll explain a few of the things he did there in the hope that it helps you some more. Lines beginning with >>> are what he enters in as commands. The other lines are the output he receives back.
>>> con, meta = connect('federer', 'grandestslam', 'tennis') # he creates the connection and meta objects
>>> con # now he calls the connection by itself to have it show that it's connected to his DB
Engine(postgresql://federer:***#localhost:5432/tennis)
>>> meta # here he calls his meta object to show how it, too, is connected
MetaData(bind=Engine(postgresql://federer:***#localhost:5432/tennis))
Related
I'm fairly new to the SQLAlchemy ORM. Im using a mySQL database whose schema I imported in a .sql file. I created the engine, connected to the database. I bound both the MetaData and the Session objects to the engine. But when I ran:
for t in metadata.tables:
print(t.name)
I got the following error:
fkey["referred_table"] = rec["TABLENAME"]
KeyError: 'TABLENAME'
So what am I doing wrong here? It is something elementary?
Below is the full code:
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy import *
engine = create_engine('mysql://sunnyahlawat:miq182#localhost/sqsunny')
engine.connect()
Session = sessionmaker(bind=engine)
session = Session()
metadata = MetaData(bind = engine, reflect = True)
#metadata.reflect(bind = engine)
for t in metadata.tables:
print(t.name)
#print(engine.table_names())
If the database being referred is a data dump and the table in question has foreign keys linked to an external database which has not been exported and is not on the same server, this error can come up.
The foreign key constraint fails in such a case.
A possible solution is to drop the constraint - if this is being tried out just in a test environment.
I have an issue with in my Flask app concerning SQLAlchemy and MySQL.
In one of my file: connection.py I have a function that creates a DB connection and set it as a global variable:
db = None
def db_connect(force=False):
global db
db = pymysql.connect(.....)
def makecursor():
cursor = db.cursor(pymysql.cursors.DictCursor)
return db, cursor
And then I have a User Model created with SQL ALchemy models.ppy
class User(Model):
id = column........
It inherits from Model which is a class that I create in another file orm.py
import connection
Engine = create_engine(url, creator=lambda x: connection.makecursor()[0], pool_pre_ping=True)
session_factory = sessionmaker(bind=Engine, autoflush=autoflush)
Session = scoped_session(session_factory)
class _Model:
query = Session.query_property()
Model = declarative_base(cls=_Model, constructor=model_constructor)
In my application I can have long script running so the DB timeout. So I have a function that "reconnect" my DB (it actually only create a new connexion and replace the global DB variable)
My goal is to be able to catch the close of my DB and reconnect it instantly. I tried with SQLAlchemy events but it never worked. (here)
Here is some line that reproduces the error:
res = User.query.filter_by(username="myuser#gmail.com").first()
connection.db.close()
# connection.reconnect() # --> SOLUTION
res = User.query.filter_by(username="myuser#gmail.com").first()
If you guys have any ideas of how to achieve that, let me know 🙏🏻
Oh and I forgot, this application is still running with python2.7.
I'm utilizing Flask and SqlAlchemy. The database I've created for SqlAlchemy seems to mess up when I try to run my website and will pop up with the error stating that there's a thread error. I'm wondering if it's because I haven't dropped my table from my previous schema. I'm using a linux server to try and run the "python3" and the file to set up my database.
I've tried to physically delete the table from my local drive and the re run it but I still up this error.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import scoped_session
from database_setup import Base, Category, Item
engine = create_engine('sqlite:///database_tables.db')
Base.metadata.bind = engine
Session = sessionmaker()
Session.bind = engine
session = Session()
brushes = Category(id = 1, category_name = 'Brushes')
session.add(brushes)
session.commit()
pencils = Category(id = 2, category_name = 'Pencils')
session.add(pencils)
session.commit()
When I am in debug mode using Flask, I click the links I've made using these rows, but after three clicks I get the error
"(sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140244909291264 and this is thread id 140244900898560 [SQL: SELECT category.id AS category_id, category.category_name AS category_category_name FROM category] [parameters: [{}]] (Background on this error at: http://sqlalche.me/e/f405)"
you can use for each thread a session, by indexing them using the thread id _thread.get_ident():
import _thread
engine = create_engine('sqlite:///history.db', connect_args={'check_same_thread': False})
...
Base.metadata.create_all(engine)
sessions = {}
def get_session():
thread_id = _thread.get_ident() # get thread id
if thread_id in sessions:
return sessions[thread_id]
session_factory = sessionmaker(bind=engine)
Session = scoped_session(session_factory)
sessions[thread_id] = Session()
return sessions[thread_id]
then use get_session() where it is needed, in your case:
get_session().add(brushes)
get_session().commit()
I would like to verify the SSL connection that SQLAlchemy sets up when using create_engine to connect to a PostgreSQL database. For example, if I have the following Python 3 code:
from sqlalchemy import create_engine
conn_string = "postgresql+psycopg2://myuser:******#someserver:5432/somedb"
conn_args = {
"sslmode": "verify-full",
"sslrootcert": "/etc/ssl/certs/ca-certificates.crt",
}
engine = create_engine(conn_string, connect_args=conn_args)
I know that I can print the contents of engine.__dict__, but it doesn't contain any information about the SSL settings (TLS version, cipher suite, etc) that it's using to connect:
{
'_echo': False,
'dialect': <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7f988a217978>,
'dispatch': <sqlalchemy.event.base.ConnectionEventsDispatch object at 0x7f988938e788>,
'engine': Engine(postgresql+psycopg2://myuser:******#someserver:5432/somedb),
'logger': <Logger sqlalchemy.engine.base.Engine (DEBUG)>,
'pool': <sqlalchemy.pool.impl.QueuePool object at 0x7f988a238c50>,
'url': postgresql+psycopg2://myuser:******#someserver:5432/somedb
}
I know I can do something like SELECT * FROM pg_stat_ssl;, but does the SQLAlchemy engine store this kind of information as a class attribute / method?
Thank you!
I don't use postgres so hopefully this holds true for you.
SQLAlchemy takes the info that you provide in the url and passes it down to the underlying dbapi library that is also specified in the url, in your case it's psycopg2.
Your engine instance only connects to the database when needed, and sqlalchemy just passes the connection info along to the driver specified in the url which returns a connection that sqlalchemy uses.
Forgive that this is mysql, but should be fundamentally the same for you:
>>> engine
Engine(mysql+mysqlconnector://test:***#localhost/test)
>>> conn = engine.connect()
>>> conn
<sqlalchemy.engine.base.Connection object at 0x000001614ACBE2B0>
>>> conn.connection
<sqlalchemy.pool._ConnectionFairy object at 0x000001614BF08630>
>>> conn.connection.connection
<mysql.connector.connection_cext.CMySQLConnection object at 0x000001614AB7E1D0>
Calling engine.connect() returns a sqlalchemy.engine.base.Connection instance that has a connection property for which the docstring says:
The underlying DB-API connection managed by this Connection.
However, you can see from above that it actually returns a sqlalchemy.pool._ConnectionFairy object which from it's docstring:
Proxies a DBAPI connection...
Here is the __init__() method of the connection fairy, and as you can see it has a connection attribute that is the actual underlying dbapi connection.
def __init__(self, dbapi_connection, connection_record, echo):
self.connection = dbapi_connection
self._connection_record = connection_record
self._echo = echo
As to what info is available on the dbapi connection object, it depends on the implementation of that particular driver. E.g psycopg2 connection objects have an info attribute:
A ConnectionInfo object exposing information about the native libpq
connection.
That info object has attributes such as ssl_in_use:
True if the connection uses SSL, False if not.
And ssl_attribute:
Returns SSL-related information about the connection.
So you don't have to dig too deep to get at the actual db connection to see what is really going on.
Also, if you want to ensure that all client connections are ssl, you can always force them to.
Here´s a quick and dirty of what SuperShoot spelled out in detail:
>>> from sqlalchemy import create_engine
>>> db_string = "postgresql+psycopg2://myuser:******#someserver:5432/somedb"
>>> db = create_engine(db_string)
>>> conn = db.connect()
>>> conn.connection.connection.info.ssl_in_use
Should return True if using SSL.
In case someone is looking for PostgreSQL and pg8000, see the pg8000 docs.
For SSL defaults, it is:
import sqlalchemy
sqlalchemy.create_engine(url, connect_args={'ssl_context':True})
I have the following set up for which on session.query() SqlAlchemy returns stale data:
Web application running on Flask with Gunicorn + supervisor.
one of the services is composed in this way:
app.py:
#app.route('/api/generatepoinvoice', methods=["POST"])
#auth.login_required
def generate_po_invoice():
try:
po_id = request.json['po_id']
email = request.json['email']
return jsonify(response=POInvoiceGenerator.get_invoice(po_id, email))
except Exception as ex:
app.logger.error("generate_po_invoice(): " + ex.message)
in another folder i have the database related stuff:
DatabaseModels (folder)
|-->Model.py
|-->Connection.py
that's what is contained in the connection.py file:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine(DB_BASE_URI, isolation_level="READ COMMITTED")
Session = scoped_session(sessionmaker(bind=engine))
session = Session()
Base = declarative_base()
and thats an extract of the model.py file:
from DatabaseModels.Connection import Base
from sqlalchemy import Column, String, etc...
class Po(Base):
__tablename__ = 'PLC_PO'
id = Column("POId", Integer, primary_key=True)
code = Column("POCode", String(50))
etc...
Then i have another file POInvoiceGenerator.py
that contains the call to the database for fetching some data:
import DatabaseModels.Connection as connection
import DatabaseModels.model as model
def get_invoice(po_code, email):
try:
po_code = po_code.strip()
PLCConnection.session.expire_all()
po = connection.session.query(model.Po).filter(model.Po.code == po_code).first()
except Exception as ex:
logger.error("get_invoice(): " + ex.message)
in subsequent users calls to this service sometimes i start to get errors like: could not find data in the db for that specific code and so on. Like if the data are stale and so on.
My first approach was to add isolation_level="READ COMMITTED" to the engine declaration and then to create a scoped session, but the stale data reading keeps appening.
Is there anyone that had any idea if my setup is wrong (the session and the model are reused among multiple methods and files)
Thanks in advance.
even if the solution pointed by #TonyMountax seems valid and made me discover something that i didn't know about SqlAlchemy, In the end i opted for something different.
I figured out that the connection established by SqlAlchemy was durable since it was created from a pool of connection everytime, this somehow was causing the data to be stale.
i added a NullPool to my code:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.pool import NullPool
engine = create_engine(DB_URI, isolation_level="READ COMMITTED", poolclass=NullPool)
Session = scoped_session(sessionmaker(bind=engine))
session = Session()
and then i'm calling a session close for every query that i make:
session.query("some query..")
session.close()
this will cause SqlAlchemy to create a new connection every time and get fresh data from the db.
Hope that this is the correct way to use it and that might be useful to someone else.
The way you instantiate your database connections means that they are reused for the next request, and they have some state left from the previous request. SQLAlchemy uses a concept of sessions to interact with the database, so that your data does not abruptly change in a single request even if you happen to perform the same query twice. This makes sense when you are using the ORM query features. For instance, if you were to query len(User.friendlist) twice during the same session, but a friend request was accepted during the request, then it will still show the same number in both locations.
To fix this, you must set up the session on first request, then you must tear it down when the request is finished. To do so is not trivial, but there is a well-established project that does it already: Flask-SQLAlchemy. It's from Pallets, the people behind Flask itself and Jinja2.