I'm very new to FastAPI and SqlAlchemy.
I've a table like this
import sqlalchemy
from sqlalchemy import Integer, Column, String
Agents = sqlalchemy.Table(
'agents',
metadata,
sqlalchemy.Column('id', Integer, primary_key=True),
sqlalchemy.Column('name', String(500))
)
Now, when the values are saved into the name column, we save the encrypted version of the name using Pgcrypto module of PostgreSQL. If this was sqlalchemy.orm based table, then I could use column_property to encrypt and decrypt on the fly when saving and selecting. Is there anything similar for such table based object? Otherwise, for now, I've to manually decrypt value on each select like this
from sqlalchemy import select,func, LargeBinary,cast
select([Agents, func.pgp_sym_decrypt(cast(Agents.c.name, LargeBinary), '**SECRET_KEY**').label('name')])
You can see that I've to decrypt every time. Is there any way that I could just decrypt on the fly whenever I use that Agents table?
Thank you in advance for any help.
You can create custom type:
from sqlalchemy import String, func, type_coerce, TypeDecorator
from sqlalchemy.dialects.postgresql import BYTEA
class PGPString(TypeDecorator):
impl = BYTEA
cache_ok = True
def __init__(self, passphrase):
super(PGPString, self).__init__()
self.passphrase = passphrase
def bind_expression(self, bindvalue):
return func.pgp_sym_encrypt(
type_coerce(bindvalue, String),
self.passphrase
)
def column_expression(self, col):
return func.pgp_sym_decrypt(col, self.passphrase)
Agents = sqlalchemy.Table(
'agents',
metadata,
sqlalchemy.Column('id', Integer, primary_key=True),
sqlalchemy.Column('name', PGPString("**SECRET_KEY**"))
)
see more examples here
Related
I'd like to log whether or not SQLAlchemy has to create any database tables when create_all() is called, however I don't see any documentation on a return value from create_all(). How do I go about this?
I've tried setting up a simple in-memory database using the following code and called create_all(). With echo set to true, I can see that the table is created as expected, but myreturn has a type of NoneType. If I call create_all() a second time, the table isn't created, and myreturn is still NoneType.
from sqlalchemy import create_engine, Column, Integer, String, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
Base = declarative_base()
class User(Base):
__tablename__ = 'testtable'
id = Column('id', Integer, primary_key=True)
name = Column('name', String, unique=True)
engine = create_engine('sqlite:///:memory:', echo=True)
myreturn = Base.metadata.create_all(bind=engine)
Is there a way to identify whether the tables are created by create_all() or do I need to create additional logic to verify this in the database directly before invoking create_all() ?
You can use the after_create event. The event handler will be passed a keyword arg called tables which is a collection of any tables that were created within the create_all() method call.
from sqlalchemy import event
#event.listens_for(Base.metadata, 'after_create')
def receive_after_create(target, connection, tables, **kw):
"listen for the 'after_create' event"
if tables:
print('A table was created')
else:
print('A table was not created')
I am using SQLAlchemy as ORM for a python project. I have created few models/schema and it is working fine. Now I need to query a existing MySQL database, no insert/update just the select statement.
How can I create a wrapper around the tables of this existing database? I have briefly gone through the sqlalchemy docs and SO but couldn't find anything relevant. All suggest execute method, where I need to write the raw sql queries, while I want to use the SQLAlchemy query method in same way as I am using with the SA models.
For example if the existing db has table name User then I want to query it using the dbsession ( only the select operation, probably with join)
You seem to have an impression that SQLAlchemy can only work with a database structure created by SQLAlchemy (probably using MetaData.create_all()) - this is not correct. SQLAlchemy can work perfectly with a pre-existing database, you just need to define your models to match database tables. One way to do that is to use reflection, as Ilja Everilä suggests:
from sqlalchemy import Table
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyClass(Base):
__table__ = Table('mytable', Base.metadata,
autoload=True, autoload_with=some_engine)
(which, in my opinion, would be totally fine for one-off scripts but may lead to incredibly frustrating bugs in a "real" application if there's a potential that the database structure may change over time)
Another way is to simply define your models as usual taking care to define your models to match the database tables, which is not that difficult. The benefit of this approach is that you can map only a subset of database tables to you models and even only a subset of table columns to your model's fields. Suppose you have 10 tables in the database but only interested in users table from where you only need id, name and email fields:
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column(sa.String)
email = sa.Column(sa.String)
(note how we didn't need to define some details which are only needed to emit correct DDL, such as the length of the String fields or the fact that the email field has an index)
SQLAlchemy will not emit INSERT/UPDATE queries unless you create or modify models in your code. If you want to ensure that your queries are read-only you may create a special user in the database and grant that user SELECT privileges only. Alternatively/in addition, you may also experiment with rolling back the transaction in your application code.
You can access an existing table using the automap extension:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
Base = automap_base()
Base.prepare(engine, reflect=True)
Users = Base.classes.users
session = Session(engine)
res = session.query(Users).first()
Create a table with autoload enabled that will inspect it. Some example code:
from sqlalchemy.sql import select
from sqlalchemy import create_engine, MetaData, Table
CONN_STR = '…'
engine = create_engine(CONN_STR, echo=True)
metadata = MetaData()
cookies = Table('cookies', metadata, autoload=True,
autoload_with=engine)
cols = cookies.c
with engine.connect() as conn:
query = (
select([cols.created_at, cols.name])
.order_by(cols.created_at)
.limit(1)
)
for row in conn.execute(query):
print(row)
Other answers don't mention what to do if you have a table with no primary key, so I thought I would address this. Assuming a table called Customers that has columns for CustomerId, CustomerName, CustomerLocation you could do;
from sqlalchemy.ext.automap import automap_base
from sqlalchemy import create_engine, MetaData, Column, String, Table
from sqlalchemy.orm import Session
Base = automap_base()
conn_str = '...'
engine = create_engine(conn_str)
metadata = MetaData()
# you only need to define which column is the primary key. It can automap the rest of the columns.
customers = Table('Customers',metadata, Column('CustomerId', String, primary_key=true), autoload=True, autoload_with=engine)
Base.prepare()
Customers= Base.classes.Customers
session = Session(engine)
customer1 = session.query(Customers).first()
print(customer1.CustomerName)
Assume we have a Postgresql database named accounts. And we already have a table named users.
import sqlalchemy as sa
psw = "verysecret"
db = "accounts"
# create an engine
pengine = sa.create_engine('postgresql+psycopg2://postgres:' + psw +'#localhost/' + db)
from sqlalchemy.ext.declarative import declarative_base
# define declarative base
Base = declarative_base()
# reflect current database engine to metadata
metadata = sa.MetaData(pengine)
metadata.reflect()
# build your User class on existing `users` table
class User(Base):
__table__ = sa.Table("users", metadata)
# call the session maker factory
Session = sa.orm.sessionmaker(pengine)
session = Session()
# filter a record
session.query(User).filter(User.id==1).first()
Warning: Your table should have a Primary Key defined. Otherwise, Sqlalchemy won't like it.
This example shows how to use it with "non-declarative" - http://docs.sqlalchemy.org/en/latest/core/ddl.html#sqlalchemy.schema.DDL
How can I use it with the ORM declarative syntax?
For example, with this structure:
Base = declarative_base(bind=engine)
class TableXYZ(Base):
__tablename__ = 'tablexyz'
Silly example, but think this is what you're looking for, should get you going:
from sqlalchemy import event
from sqlalchemy.engine import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import create_session
from sqlalchemy.schema import Column, DDL
from sqlalchemy.types import Integer
Base = declarative_base()
engine = create_engine('sqlite:////tmp/test.db', echo=True)
class TableXYZ(Base):
__tablename__ = 'tablexyz'
id = Column(Integer, primary_key=True)
#event.listen(
# Base.metadata, 'after_create',
# DDL("""
# alter table TableXYZ add column name text
# """)
event.listen(
TableXYZ.__table__, 'after_create',
DDL("""
alter table TableXYZ add column name text
""")
)
Base.metadata.create_all(engine)
Running the above results in - note "name text" for the added column:
sqlite> .schema tablexyz
CREATE TABLE tablexyz (
id INTEGER NOT NULL, name text,
PRIMARY KEY (id)
);
I have my code in declarative and use the event.listen to add triggers and other stored procedures. Seems to work well.
It should be the same with "non-declarative" and "declarative".
You register your event by specifying (with your class and the doc's event & function) :
event.listen(TableXYZ, 'before_create', DDL('DROP TRIGGER users_trigger'))
Syntax is something like:
event.listen(Class, 'name_of_event', function)
The docs for Camelot say that it uses Elixir models. Since SQLAlchemy has included declarative_base for a while, I had used that instead of Elixir for another app. Now I would like to use the SQLAlchemy/declarative models directly in Camelot.
There is a post on Stackoverflow that says Camelot is not tied to Elixir and that using different models would be possible but it doesn't say how.
Camelot's original model.py only has this content:
import camelot.types
from camelot.model import metadata, Entity, Field, ManyToOne, OneToMany, Unicode, Date, Integer, using_options
from camelot.view.elixir_admin import EntityAdmin
from camelot.view.forms import *
__metadata__ = metadata
I added my SQLAlchemy model and changed model.py to this:
import camelot.types
from camelot.model import metadata, Entity, Field, ManyToOne, OneToMany, Unicode, Date, using_options
from camelot.view.elixir_admin import EntityAdmin
from camelot.view.forms import *
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
__metadata__ = metadata
Base = declarative_base()
class Test(Base):
__tablename__ = "test"
id = Column(Integer, primary_key=True)
text = Column(String)
It didn't work. When I start main.py, I can see the GUI and Test in the sidebar, but can't see any rows. This is the tail of the traceback:
File "/usr/lib/python2.6/dist-packages/camelot/view/elixir_admin.py", line 52, in get_query
return self.entity.query
AttributeError: type object 'Test' has no attribute 'query'
This is the elixir_admin.py code for line 46-52:
#model_function
def get_query(self):
""":return: an sqlalchemy query for all the objects that should be
displayed in the table or the selection view. Overwrite this method to
change the default query, which selects all rows in the database.
"""
return self.entity.query
If this code is causing the problem, how do I overwrite the method to change the default query to make it work?
How can you use SQLAlchemy/declarative models in Camelot?
Here is some sample code on using Declarative to define a Movie model for Camelot, some explanation can be found here.
import sqlalchemy.types
from sqlalchemy import Column
from sqlalchemy.ext.declarative import ( declarative_base,
_declarative_constructor )
from camelot.admin.entity_admin import EntityAdmin
from camelot.model import metadata
import camelot.types
from elixir import session
class Entity( object ):
def __init__( self, **kwargs ):
_declarative_constructor( self, **kwargs )
session.add( self )
Entity = declarative_base( cls = Entity,
metadata = metadata,
constructor = None )
class Movie( Entity ):
__tablename__ = 'movie'
id = Column( sqlalchemy.types.Integer, primary_key = True )
name = Column( sqlalchemy.types.Unicode(50), nullable = False )
cover = Column( camelot.types.Image(), nullable = True )
class Admin( EntityAdmin ):
list_display = ['name']
form_display = ['name', 'cover']
Which version of Camelot are you using ?
With the current version of Camelot (11.12.30) it is possible to use Declarative through some
hacks. The upcoming version will make it much easier, while after this, the examples will be
ported to Declarative as well.
I am giving Pylons a try with SQLAlchemy, and I love it, there is just one thing, is it possible to print out the raw SQL CREATE TABLE data generated from Table().create() before it's executed?
from sqlalchemy.schema import CreateTable
print(CreateTable(table))
If you are using declarative syntax:
print(CreateTable(Model.__table__))
Update:
Since I have the accepted answer and there is important information in klenwell answer, I'll also add it here.
You can get the SQL for your specific database (MySQL, Postgresql, etc.) by compiling with your engine.
print(CreateTable(Model.__table__).compile(engine))
Update 2:
#jackotonye Added in the comments a way to do it without an engine.
print(CreateTable(Model.__table__).compile(dialect=postgresql.dialect()))
You can set up you engine to dump the metadata creation sequence, using the following:
def metadata_dump(sql, *multiparams, **params):
# print or write to log or file etc
print(sql.compile(dialect=engine.dialect))
engine = create_engine(myDatabaseURL, strategy='mock', executor=metadata_dump)
metadata.create_all(engine)
One advantage of this approach is that enums and indexes are included in the printout. Using CreateTable leaves this out.
Another advantage is that the order of the schema definitions is correct and (almost) usable as a script.
I needed to get the raw table sql in order to setup tests for some existing models. Here's a successful unit test that I created for SQLAlchemy 0.7.4 based on Antoine's answer as proof of concept:
from sqlalchemy import create_engine
from sqlalchemy.schema import CreateTable
from model import Foo
sql_url = "sqlite:///:memory:"
db_engine = create_engine(sql_url)
table_sql = CreateTable(Foo.table).compile(db_engine)
self.assertTrue("CREATE TABLE foos" in str(table_sql))
Something like this? (from the SQLA FAQ)
http://docs.sqlalchemy.org/en/latest/faq/sqlexpressions.html
It turns out this is straight-forward:
from sqlalchemy.dialects import postgresql
from sqlalchemy.schema import CreateTable
from sqlalchemy import Table, Column, String, MetaData
metadata = MetaData()
users = Table('users', metadata,
Column('username', String)
)
statement = CreateTable(users)
print(statement.compile(dialect=postgresql.dialect()))
Outputs this:
CREATE TABLE users (
username VARCHAR
)
Going further, it can even support bound parameters in prepared statements.
Reference
How do I render SQL expressions as strings, possibly with bound parameters inlined?
...
or without an Engine:
from sqlalchemy.dialects import postgresql
print(statement.compile(dialect=postgresql.dialect()))
SOURCE: http://docs.sqlalchemy.org/en/latest/faq/sqlexpressions.html#faq-sql-expression-string
Example: Using SQL Alchemy to generate a user rename script
#!/usr/bin/env python
import csv
from sqlalchemy.dialects import postgresql
from sqlalchemy import bindparam, Table, Column, String, MetaData
metadata = MetaData()
users = Table('users', metadata,
Column('username', String)
)
renames = []
with open('users.csv') as csvfile:
for row in csv.DictReader(csvfile):
renames.append({
'from': row['sAMAccountName'],
'to': row['mail']
})
for rename in renames:
stmt = (users.update()
.where(users.c.username == rename['from'])
.values(username=rename['to']))
print(str(stmt.compile(dialect=postgresql.dialect(),
compile_kwargs={"literal_binds": True})) + ';')
When processing this users.csv:
sAMAccountName,mail
bmcboatface,boaty.mcboatface#example.com
ndhyani,naina.dhyani#contoso.com
Gives output like this:
UPDATE users SET username='boaty.mcboatface#example.com' WHERE users.username = 'bmcboatface';
UPDATE users SET username='naina.dhyani#contoso.com' WHERE users.username = 'ndhyani';users.username = 'ndhyani';
Why a research vessel has an email address is yet to be determined. I have been in touch with Example Inc's IT team and have had no response.
May be you mean echo parameter of sqlalchemy.create_engine?
/tmp$ cat test_s.py
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Department(Base):
__tablename__ = "departments"
department_id = sa.Column(sa.types.Integer, primary_key=True)
name = sa.Column(sa.types.Unicode(100), unique=True)
chief_id = sa.Column(sa.types.Integer)
parent_department_id = sa.Column(sa.types.Integer,
sa.ForeignKey("departments.department_id"))
parent_department = sa.orm.relation("Department")
engine = sa.create_engine("sqlite:///:memory:", echo=True)
Base.metadata.create_all(bind=engine)
/tmp$ python test_s.py
2011-03-24 15:09:58,311 INFO sqlalchemy.engine.base.Engine.0x...42cc PRAGMA table_info("departments")
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc ()
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc
CREATE TABLE departments (
department_id INTEGER NOT NULL,
name VARCHAR(100),
chief_id INTEGER,
parent_department_id INTEGER,
PRIMARY KEY (department_id),
UNIQUE (name),
FOREIGN KEY(parent_department_id) REFERENCES departments (department_id)
)
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc ()
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc COMMIT