I'm working with sqlalchemy and postgresql using python modules.
There is a database module which creates the objects that I need in order to work with postgresql tables. The init method is the following:
class Database:
is_instantiated = False
def __init__(self):
if not Database.is_instantiated:
self.engine = create_engine("postgresql+psycopg2://name:pwd#localhost:5432/udemy")
self.metadata = MetaData()
self.create_table()
Database.is_instantiated = True
print("Database connection success")
else:
print("Already connected")
The create_table() method just creates tables objects like:
def create_table(self):
self.tbl_employee = Table("employee", self.metadata,
Column("id", Integer(), primary_key=True),
Column("first_name", String(), nullable=False),
Column("last_name", String(), nullable=False),
Column("birthday", String(10), nullable=False),
Column("department_name", String(), nullable=False)
)
Now, there is a start module with a button that if pressed it opens another window using the following method:
def manage_employees(self):
self.hide()
self.employee_window = EmployeeMenu(self)
self.employee_window.show()
The employee module, when instanciated, populate the QTableWidget using this method:
def populate_tbl(self):
self.db = Database()
headers, data = self.db.get_data_for_tab()
tbl_nr_rows = len(data)
tbl_nr_columns = len(headers)
self.tbl_center.setRowCount(tbl_nr_rows)
self.tbl_center.setColumnCount(tbl_nr_columns)
self.tbl_center.setHorizontalHeaderLabels(tuple(headers))
self.tbl_center.setSelectionMode(qw.QAbstractItemView.SelectionMode.SingleSelection)
self.tbl_center.horizontalHeader().setSectionResizeMode(qw.QHeaderView.Stretch)
self.tbl_center.setSelectionBehavior(qw.QAbstractItemView.SelectRows)
for row in range(tbl_nr_rows):
for col in range(tbl_nr_columns):
self.tbl_center.setItem(row, col, qw.QTableWidgetItem(str(data[row][col])))
The self.db.get_data_for_tab() simply takes data from the postgresql in order to populate the table widget.
The first time that I open the employee window everything works fine, but I have also a button that can bring me back to the start menu:
def show_start(self):
self.hide()
self.start_menu.show()
If I use this method and then try to reopen the employee window (using the manage_employees), it will raise an error because the Database.is_instantiated attribute is True now and therefore it will not run the code for creating the engine, metadata, ...
My first question is if it is correct to try to create the engine and others database's objects only one time, and in that case how can I set up that properly in order to avoid this kind of issues.
If it is not necessary, I have seen that if I remove the initial check in the init method of the Database class, the employee table will be populated properly, but I don't know if this is the right way to work with sqlalchmey objects.
Many thanks in advance for your help.
Andrea
Related
Description of the problem
I wanted to use ormar with SQLite for my project but ran into the problem that ormar doesn't save changes to the database. Although everything seems to be done according to the documentation. (Additionally, I used the faker to generate unique names to fill db and loguru to logging)
My code
import asyncio
from databases import Database
from faker import Faker
from loguru import logger as log
from ormar import ModelMeta, Model, Integer, String
from sqlalchemy import create_engine, MetaData
fake = Faker()
DB_PATH = 'sqlite:///db.sqlite3'
database = Database(DB_PATH)
metadata = MetaData()
class BaseMeta(ModelMeta):
database = database
metadata = metadata
class User(Model):
class Meta(BaseMeta):
tablename = 'users'
id: int = Integer(primary_key=True)
name: str = String(max_length=64)
# Also I tried without `with_connect` function, but it also doesn't work
async def with_connect(function):
async with database:
await function()
async def create():
return f"User created: {await User(name=fake.name()).save()}"
# Also I tried this: `User.objects.get_or_create(name=fake.name())`
# but it also doesn't work:
async def read():
return f"All data from db: {await User.objects.all()}"
async def main():
log.info(await create())
log.info(await create())
log.info(await read())
if __name__ == '__main__':
engine = create_engine(DB_PATH)
metadata.drop_all(engine)
metadata.create_all(engine)
try:
asyncio.run(with_connect(main))
finally:
metadata.drop_all(engine)
Results
As a result, I expected that after each run of the code, data would be printed during previous runs. That is so that the created objects are saved to the file db.sqlite3.
The actual result is that after each run of the code, only the data generated during that run is printed.
Conclusion
Why is the data not saved to the database file? Maybe I misunderstood how ORMs work?
I am using Flask-SQLAlchemy, with autocommit set to False and autoflush set to True. It's connecting to a mysql database.
I have 3 methods like this:
def insert_something():
insert_statement = <something>
db.session.execute(insert_statement);
db.session.commit()
def delete_something():
delete_statement = <something>
db.session.execute(delete_statement);
db.session.commit()
def delete_something_else():
delete_statement = <something>
db.session.execute(delete_statement);
db.session.commit()
Sometimes I want to run these methods individually; no problems there — but sometimes I want to run them together in a nested transaction. I want insert_something to run first, and delete_something to run afterwards, and delete_something_else to run last. If any of those methods fail then I want everything to be rolled back.
I've tried the following:
db.session.begin_nested()
insert_something()
delete_something()
delete_something_else()
db.session.commit()
This doesn't work, though, because insert_something exits the nested transaction (and releases the savepoint). Then, when delete_something runs db.session.commit() it actually commits the deletion to the database because it is in the outermost transaction.
That final db.session.commit() in the code block above doesn't do anything..everything is already committed by that point.
Maybe I can do something like this, but it's ugly as hell:
db.session.begin_nested()
db.session.begin_nested()
db.session.begin_nested()
db.session.begin_nested()
insert_something()
delete_something()
delete_something_else()
db.session.commit()
There's gotta be a better way to do it without touching the three methods..
Edit:
Now I'm doing it like this:
with db.session.begin_nested():
insert_something()
with db.session.begin_nested():
delete_something()
with db.session.begin_nested():
delete_something_else()
db.session.commit()
Which is better, but still not great.
I'd love to be able to do something like this:
with db.session.begin_nested() as nested:
insert_something()
delete_something()
delete_something_else()
nested.commit() # though I feel like you shouldn't need this in a with block
The docs discuss avoiding this pattern in arbitrary-transaction-nesting-as-an-antipattern and session-faq-whentocreate.
But there is an example in the docs that is similar to this but it is for testing.
https://docs.sqlalchemy.org/en/14/orm/session_transaction.html?highlight=after_transaction_end#joining-a-session-into-an-external-transaction-such-as-for-test-suites
Regardless, here is a gross transaction manager based on the example that "seems" to work but don't do this. I think there are a lot of gotchas in here.
import contextlib
from sqlalchemy import (
create_engine,
Integer,
String,
)
from sqlalchemy.schema import (
Column,
MetaData,
)
from sqlalchemy.orm import declarative_base, Session
from sqlalchemy import event
from sqlalchemy.sql import delete, select
db_uri = 'postgresql+psycopg2://username:password#/database'
engine = create_engine(db_uri, echo=True)
metadata = MetaData()
Base = declarative_base(metadata=metadata)
class Device(Base):
__tablename__ = "devices"
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String(50))
def get_devices(session):
return [d.name for (d,) in session.execute(select(Device)).all()]
def create_device(session, name):
session.add(Device(name=name))
session.commit()
def delete_device(session, name):
session.execute(delete(Device).filter(Device.name == name))
session.commit()
def almost_create_device(session, name):
session.add(Device(name=name))
session.flush()
session.rollback()
#contextlib.contextmanager
def force_nested_transaction_forever(session, commit_on_complete=True):
"""
Keep re-entering a nested transaction everytime a transaction ends.
"""
d = {
'nested': session.begin_nested()
}
#event.listens_for(session, "after_transaction_end")
def end_savepoint(session, transaction):
# Start another nested trans if the prior one is no longer active.
if not d['nested'].is_active:
d['nested'] = session.begin_nested()
try:
yield
finally:
# Stop trapping us in perpetual nested transactions.
# Is this the right place for this ?
event.remove(session, "after_transaction_end", end_savepoint)
# This seems like it would be error prone.
if commit_on_complete and d['nested'].is_active:
d.pop('nested').commit()
if __name__ == '__main__':
metadata.create_all(engine)
with Session(engine) as session:
with session.begin():
# THIS IS NOT RECOMMENDED
with force_nested_transaction_forever(session):
create_device(session, "0")
create_device(session, "a")
delete_device(session, "a")
almost_create_device(session, "a")
create_device(session, "b")
assert len(get_devices(session)) == 2
assert len(get_devices(session)) == 2
I'm trying to figure out how to chain class methods to improve a utility class I've been writing - for reasons I'd prefer not to get into :)
Now suppose I wanted to chain a chain class methods on a class instance (in this case for setting the cursor) e.g.:
# initialize the class instance
db = CRUD(table='users', public_fields=['name', 'username', 'email'])
#the desired interface class_instance.cursor(<cursor>).method(...)
with sql.read_pool.cursor() as c:
db.cursor(c).get(target='username', where="omarlittle")
The part that's confusing is I would prefer the cursor not to persist as a class attribute after .get(...) has been called and has returned, I'd like to require that .cursor(cursor) must be first called.
class CRUD(object):
def __init__(self, table, public_fields):
self.table = table
self.public_fields = public_fields
def fields(self):
return ', '.join([f for f in self.public_fields])
def get(self, target, where):
#this is strictly for illustration purposes, I realize all
#the vulnerabilities this leaves me exposed to.
query = "SELECT {fields} FROM {table} WHERE {target} = {where}"
query.format(fields=self.fields, table=self.table, target=target,
where=where)
self.cursor.execute(query)
def cursor(self, cursor):
pass # this is where I get lost.
If I understand what you're asking, what you want is for the cursor method to return some object with a get method that works as desired. There's no reason the object it returns has to be self; it can instead return an instance of some cursor type.
That instance could have a back-reference to self, or it could get its own copy of whatever internals are needed to be a cursor, or it could be a wrapper around an underlying object from your low-level database library that knows how to be a cursor.
If you look at the DB API 2.0 spec, or implementations of it like the stdlib's sqlite3, that's exactly how they do it: A Database or Connection object (the thing you get from the top-level connect function) has a cursor method that returns a Cursor object, and that Cursor object has an execute method.
So:
class CRUDCursor(object):
def __init__(self, c, crud):
self.crud = crud
self.cursor = however_you_get_an_actual_sql_cursor(c)
def get(self, target, where):
#this is strictly for illustration purposes, I realize all
#the vulnerabilities this leaves me exposed to.
query = "SELECT {fields} FROM {table} WHERE {target} = {where}"
query.format(fields=self.crud.fields, table=self.crud.table,
target=target, where=where)
self.cursor.execute(query)
# you may want this to return something as well?
class CRUD(object):
def __init__(self, table, public_fields):
self.table = table
self.public_fields = public_fields
def fields(self):
return ', '.join([f for f in self.public_fields])
# no get method
def cursor(self, cursor):
return CRUDCursor(self, cursor)
However, there still seems to be a major problem with your example. Normally, after you execute a SELECT statement on a cursor, you want to fetch the rows from that cursor. You're not keeping the cursor object around in your "user" code, and you explicitly don't want the CRUD object to keep its cursor around, so… how do you expect to do that? Maybe get is supposed to return self.cursor.fetch_all() at the end or something?
How can I perform post processing on my SQL3 database via python? The following code doesn't work, but what I am trying to do is first create a new database if not exists already, then insert some data, and finally execute the query and close the connection. But I what to do so separately, so as to add additional functionality later on, such as delete / updater / etc... Any ideas?
class TitlesDB:
# initiate global variables
conn = None
c = None
# perform pre - processing
def __init__(self, name):
import os
os.chdir('/../../')
import sqlite3
conn = sqlite3.connect(name)
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS titles (title VARCHAR(100) UNIQUE)')
# insert a bunch of new titles
def InsertTitles(self, list):
c.executemany('INSERT OR IGNORE INTO titles VALUES (?)', list)
# perform post - processing
def __fina__(self):
conn.commit()
conn.close()
You could create a context manager to do the pre- and postprocessing.
import contextlib
#contextlib.contextmanager
def titles_cursor():
# perform pre - processing
conn = sqlite3.connect(name)
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS titles (title VARCHAR(100) UNIQUE)')
yield c
# perform post - processing
conn.commit()
conn.close()
Use it in a with statement:
with titles_cursor() as c:
c.executemany('INSERT OR IGNORE INTO titles VALUES (?)', list)
First, wouldn't it be better to avoid having the sql connection inside the __init__?
You will have a problem if you want to use this class in the same instance after using __fina__.
You could have it in another method and call it and call the connection closing method when needed and commit after each method is executed.
Here is what I use : Create a class method that connects to the db and executes a query from an argument,commits and closes connection and you pass any query you want executed as an argument of that method.You can simply call this method anytime you want.
And the best about this is that you can create a method that passes multiple queries as arguments before closing db connection.
This is specially usefull if have to use sql connections to the same db in another class without using a set of methods each time you need to execute a sql query.
Here is a little example I used with MySQLdb module, It's pretty simple but it worked.
import MySQLdb
class DbQuery:
'''Here is the class I talked about'''
def __init__(self):
'''You can define the main queries here but it's not necessary
They can be global variables
If you don't have class dependency from which you get variables
you might not even need to define __init__'''
def Sql_Connect(self):
self.db = MySQLdb.connect("localhost","root","","data_db" )
self.cursor = db.cursor()
def Sql_Commit(self):
self.db.commit()
print "Info : Database updated"
except:
self.db.rollback()
print "Error : Database rollback"
self.db.close()
def Query(self,query):
self.Sql_Connect()
try :
self.cursor.execute(query)
self.Sql_Commit()
The only thing important is to remember the query structure.
In my application I'm using SQLAlchemy for storing most persistent data across app restarts. For this I have a db package containing my mapper classes (like Tag, Group etc.) and a support class creating a single engine instance using create_engine and a single, global, Session factory using sessionmaker.
Now my understanding of how to use SQLAlchemys sessions is, that I don't pass them around in my app but rather create instances using the global factory whenever I need database access.
This leads to situations were a record is queried in one session and then passed on to another part of the app, which uses a different session instance. This gives me exceptions like this one:
Traceback (most recent call last):
File "…", line 29, in delete
session.delete(self.record)
File "/usr/lib/python3.3/site-packages/sqlalchemy/orm/session.py", line 1444, in delete
self._attach(state, include_before=True)
File "/usr/lib/python3.3/site-packages/sqlalchemy/orm/session.py", line 1748, in _attach
state.session_id, self.hash_key))
sqlalchemy.exc.InvalidRequestError: Object '<Group at 0x7fb64c7b3f90>' is already attached to session '1' (this is '3')
Now my question is: did I get the usage of Session completly wrong (so I should use one session only at a time and pass that session around to other components together with records from the database) or could this result from actual code issue?
Some example code demonstrating my exact problem:
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base, declared_attr
Base = declarative_base()
class Record(Base):
__tablename__ = "record"
id = Column(Integer, primary_key=True)
name = Column(String)
def __init__(self, name):
self.name = name
def __repr__(self):
return "<%s('%s')>" % (type(self).__name__, self.name)
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
s1 = Session()
record = Record("foobar")
s1.add(record)
s1.commit()
# This would be a completly different part of app
s2 = Session()
record = s2.query(Record).filter(Record.name == "foobar").first()
def delete_record(record):
session = Session()
session.delete(record)
session.commit()
delete_record(record)
For now I switched over to using a single, global session instance. That's neither nice nor clean in my opinion, but including lots and lots of boiler plate code to expunge objects from one session just to add them back to their original session after handing it over to some other application part was no realistic option, either.
I suppose this will completely blow up if I start using multiple threads to access the database via the very same session…