I just want to run python code before the database inserts the data, to be able to validate a string. And since I'm using the https://github.com/encode/databases library I can only use the Core of SQLAlchemy, so I can't declare my table using ORM.
I have the following table:
application_permissions_table = Table(
"application_permissions",
metadata,
Column("id", Integer, primary_key=True),
Column("acl", String, nullable=False, unique=True),
)
I created an event to check the validity of acl. I want to be able to use this with PostgreSQL, SQLite and maybe another database.
#event.listens_for(application_permissions_table.c.acl, "set")
def receive_set(target, value, oldvalue, initiator):
regex = r"^(Allow|Deny)\|(\w+:\w+|Authenticated)\|\w+$"
if not re.match(regex, value):
raise ValueError("ACL don't match the format")
Unfortunately this doesn't work and gives the following error: sqlalchemy.exc.InvalidRequestError: No such event 'set' for target 'application_permissions.acl'
Related
I'm very new to FastAPI and SqlAlchemy.
I've a table like this
import sqlalchemy
from sqlalchemy import Integer, Column, String
Agents = sqlalchemy.Table(
'agents',
metadata,
sqlalchemy.Column('id', Integer, primary_key=True),
sqlalchemy.Column('name', String(500))
)
Now, when the values are saved into the name column, we save the encrypted version of the name using Pgcrypto module of PostgreSQL. If this was sqlalchemy.orm based table, then I could use column_property to encrypt and decrypt on the fly when saving and selecting. Is there anything similar for such table based object? Otherwise, for now, I've to manually decrypt value on each select like this
from sqlalchemy import select,func, LargeBinary,cast
select([Agents, func.pgp_sym_decrypt(cast(Agents.c.name, LargeBinary), '**SECRET_KEY**').label('name')])
You can see that I've to decrypt every time. Is there any way that I could just decrypt on the fly whenever I use that Agents table?
Thank you in advance for any help.
You can create custom type:
from sqlalchemy import String, func, type_coerce, TypeDecorator
from sqlalchemy.dialects.postgresql import BYTEA
class PGPString(TypeDecorator):
impl = BYTEA
cache_ok = True
def __init__(self, passphrase):
super(PGPString, self).__init__()
self.passphrase = passphrase
def bind_expression(self, bindvalue):
return func.pgp_sym_encrypt(
type_coerce(bindvalue, String),
self.passphrase
)
def column_expression(self, col):
return func.pgp_sym_decrypt(col, self.passphrase)
Agents = sqlalchemy.Table(
'agents',
metadata,
sqlalchemy.Column('id', Integer, primary_key=True),
sqlalchemy.Column('name', PGPString("**SECRET_KEY**"))
)
see more examples here
In my postgres server we have a database database with 2 schemas: public and api.
public has several tables, and I need to create a table in api with a foreign key to a table in public called model.
So it's:
-Schemas
--public
---tables
----models
--api
---tables
Using SQLAlchemy I have the following class:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
However upon launch I receive the following error:
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'training.model_id' could not find table 'public.model' with which to generate a foreign key to target column 'id'
It seems it does look for the correct thing but can't find it? I'm very confused as to why this is happening, especially because the error refers to not finding "public", which is created by default by postgres, rather than "api" which I created myself in pgAdmin.
Am I missing some cruicial config?
The error you are getting means that you are trying to create a foreign key referencing a table that SQLAlchemy does not know about. You can tell sqlalchemy about it by creating a Table associated with the same MetaData describing the referenced table. You can also do this using sqlalchemy's reflection capabilities. For example:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
metadata.reflect(schema="public", only=["model"])
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
By default, MetaData.create_all() will check for the existence of tables first, before creating them, but you can also specify the exact tables to create: metadata.create_all(tables=[self.__table])
I have a table that already exists:
USERS_TABLE = Table("users", META_DATA,
Column("id", Integer, Sequence("user_id_seq"), primary_key=True),
Column("first_name", String(255)),
Column("last_name", String(255))
)
I created this table by running this:
CONN = create_engine(DB_URL, client_encoding="UTF-8")
META_DATA = MetaData(bind=CONN, reflect=True)
# ... table code
META_DATA.create_all(CONN, checkfirst=True)
the first time it worked and I was able to create the table. However, the 2nd time around I got this error:
sqlalchemy.exc.InvalidRequestError: Table 'users' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
which makes sense since the table users already exists. I'm able to see if the table exists like so:
TABLE_EXISTS = CONN.dialect.has_table(CONN, "users")
However, how do I actually get the existing table object? I can't find this anywhere in the documentation. Please help.
We have 3 different approaches here:
assume that required tables have been created already, reflecting them and getting with MetaData.tables dictionary field like
from sqlalchemy import MetaData, create_engine
CONN = create_engine(DB_URL, client_encoding="UTF-8")
META_DATA = MetaData(bind=CONN, reflect=True)
USERS_TABLE = META_DATA.tables['users']
removing reflect flag from MetaData object initialization, because we don't use it and moreover – trying to create tables that've been already reflected:
from sqlalchemy import (MetaData, Table, Column, Integer, String, Sequence,
create_engine)
CONN = create_engine('sqlite:///db.sql')
META_DATA = MetaData(bind=CONN)
USERS_TABLE = Table("users", META_DATA,
Column("id", Integer, Sequence("user_id_seq"),
primary_key=True),
Column("first_name", String(255)),
Column("last_name", String(255)))
META_DATA.create_all(CONN, checkfirst=True)
assuming that we are keeping reflected table if it was previously created by setting in Table object initializer keep_existing flag to True:
from sqlalchemy import (MetaData, Table, Column, Integer, String, Sequence,
create_engine)
CONN = create_engine('sqlite:///db.sql')
META_DATA = MetaData(bind=CONN, reflect=True)
USERS_TABLE = Table("users", META_DATA,
Column("id", Integer, Sequence("user_id_seq"),
primary_key=True),
Column("first_name", String(255)),
Column("last_name", String(255)),
keep_existing=True)
META_DATA.create_all(CONN, checkfirst=True)
Which one to choose? Depends on your use case, but I prefer second one since it looks like you aren't using reflection, also it is simplest modification: just removing flag from MetaData initializer.
P. S.
we can always make reflection after initialization of MetaData object with MetaData.reflect method:
META_DATA.reflect()
also we can specify which tables to reflect with only parameter (may be any iterable of str objects):
META_DATA.reflect(only=['users'])
and many more.
This works for me pretty well -
import sqlalchemy as db
engine = db.create_engine("your_connection_string")
meta_data = db.MetaData(bind=engine)
db.MetaData.reflect(meta_data)
USERS = meta_data.tables['users']
# View the columns present in the users table
print(USERS.columns)
# You can run sqlalchemy queries
query = db.select([
USERS.c.id,
USERS.c.first_name,
USERS.c.last_name,
])
result = engine.execute(query).fetchall()
Note that using reflect parameter in Metadata(bind=engine, reflect=True) is deprecated and will be removed in a future release. Above code takes care of it.
__table_args__ = {'extend_existing': True}
right below __tablename__
If you're using async Sqlalchemy, you can use
metadata = MetaData()
async with engine.connect() as conn:
await conn.run_sync(metadata.reflect, only=["harshit_table"])
harshit_table = Table("harshit_table", metadata, autoload_with=engine)
print("tables: ", harshit_table, type(harshit_table))
I'm quite new to this, but what worked for me was this (variables are declared in the original question)
USERS_TABLE_NEW = Table("users", META_DATA, autoload_with=CONN)
Considering Storm, a python ORM, I would like to automatically generate the schema for a (mysql) database. The home page states
"Storm works well with existing database schemas." ( https://storm.canonical.com/FrontPage ),
hence I was hoping to not having to create model classes. However, the 'getting started' tutorial ( https://storm.canonical.com/Tutorial ) suggests that a class, like the one below, needs to be created manually for each table, and each field needs to be manually specified:
class Person(object):
__storm_table__ = "person"
id = Int(primary=True)
name = Unicode()
Alternatively, SQLAlchemy doesn't seem to support a reverse engineering feature either, but does need a schema like this one:
user = Table('user', metadata,
Column('user_id', Integer, primary_key = True),
Column('user_name', String(16), nullable = False),
Column('email_address', String(60)),
Column('password', String(20), nullable = False)
)
Of course, these classes/schemas make sense, since each table will likely represent some 'object of interest' and one can extend them with all kinds of functionality. However, they are tedious to create, and their (initial) content is straight forward when a database already exists.
One ORM that does allow for reverse engineering is:
http://docs.doctrine-project.org/en/2.0.x/reference/tools.html
Are there similar reverse engineering tools for Storm or SQLAlchemy or any python ORM or python database fancyfier?
I am not aware of how Storm manages this process, but you can certainly reflect tables in a database with sqlalchemy. For example, below is a basic example using a SQL Server instance that I have access to at the moment.
AN ENTIRE DATABASE
>>> from sqlalchemy import create_engine, MetaData
>>> engine = create_engine('mssql+pyodbc://<username>:<password>#<host>/<database>') # replace <username> with user name etc.
>>> meta = MetaData()
>>> meta.reflect(bind=engine)
>>> funds_table = meta.tables['funds'] # tables are stored in meta.tables dict
>>> funds_table # now stores database schema object
Table(u'funds', MetaData(bind=None), Column(u'fund_token', INTEGER(), table=<funds>, primary_key=True, nullable=False), Column(u'award_year_token', INTEGER(), ForeignKey(u'award_year_defn.award_year_token'), table=<funds>, nullable=False), ... Column(u'fin_aid_disclosure_category', VARCHAR(length=3, collation=u'SQL_Latin1_General_CP1_CI_AS'), table=<funds>), Column(u'report_as_additional_unsub', BIT(), table=<funds>, server_default=DefaultClause(<sqlalchemy.sql.expression.TextClause object at 0x000000000545B6D8>, for_update=False)), schema=None)
If you merely want to reflect one table at a time, you can use the following code instead.
ONE TABLE AT A TIME (much faster)
>>> from sqlalchemy import Table, create_engine, MetaData
>>> engine = create_engine('mssql+pyodbc://<username>:<password>#<host>/<database>')
>>> meta = MetaData()
>>> funds_table = Table('funds', meta, autoload=True, autoload_with=engine) # indicate table name (here 'funds') with a string passed to Table as the first argument
>>> funds_table # now stores database schema object
Table(u'funds', MetaData(bind=None), Column(u'fund_token', INTEGER(), table=<funds>, primary_key=True, nullable=False), Column(u'award_year_token', INTEGER(), ForeignKey(u'award_year_defn.award_year_token'), table=<funds>, nullable=False), ... Column(u'fin_aid_disclosure_category', VARCHAR(length=3, collation=u'SQL_Latin1_General_CP1_CI_AS'), table=<funds>), Column(u'report_as_additional_unsub', BIT(), table=<funds>, server_default=DefaultClause(<sqlalchemy.sql.expression.TextClause object at 0x000000000545B6D8>, for_update=False)), schema=None)
As you can probably imagine, you can then save the relevant tables' data for accessing the tables more quickly again in the future.
I have a simple data structure, where a film table has a foreign key to country table.
In order to retrieve all the films that are from the same country, I have this property 'same_country_films', a self-referential relationship.
It almost does the job correctly, however, it also includes the film itself in the list. How can I exclude it and just have other films?
Many thanks!
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import mapper, relationship
metadata = MetaData()
country_table = Table('country', metadata,
Column('id', String, primary_key=True),
Column('name', String),
)
film_table = Table('film', metadata,
Column('id', Integer, primary_key=True),
Column('title', String),
Column('year', Integer),
Column('country_id', Integer, ForeignKey('country.id'))
)
class Country(object):
pass
class Film(object):
pass
mapper(Country, country_table)
mapper(Film, film_table,
properties={
'country':relationship(
Country,
backref='films'),
'same_country_films':relationship(
Film,
primaryjoin=film_table.c.country_id==\
film_table.c.country_id,
foreign_keys=[
film_table.c.country_id,
]
)
}
)
The simplest solution is to code this property yourself instead of relation:
class Film(object):
#property
def same_country_films(self):
return [f for f in self.country.films if f!=self]
This solution won't do separate query for this property when both film.same_country_films and country.films are accessed during session life. The property can't be updated as you can usualy do with relation, but I doubt it's realy needed.
The bad thing is that it's evaluated for each access (not so much work). You can change property decorator to chaching one (like cached_property in werkzeug), but then the property won't reflect changes in country.films after first access to it.
This should do it I think, (though I haven't actually tested it):
primaryjoin=sqlalchemy.and_(
film_table.c.country_id==film_table.c.country_id,
film_table.c.id!=film_table.c.id)