We Keep Coding

Data in JSON Column as normal column

I have a lot of unstructured data in Column(JSONB).
I would like to "standardize" this data and access it like normal Columns.
This actually does work:
class Order(Base):
__tablename__ = "orders"
id_= Column(Integer, primary_key=True, nullable=False)
type = Column(String)
data = Column(JSONB)
#hybrid_property
def email(self):
return self.data['email']
#email.setter
def email(self, value):
self.data['email'] = value
#email.expression
def email(cls):
return cls.data['email'].astext
However data can have different keys, eg: email, e_mail, email_address, etc..
The solution I'm thinking about is to "extend" main class with another class depending
on value in "type":
class Parser1:
#hybrid_property
def email(self):
return self.data['email']
#email.setter
def email(self, value):
self.data['email'] = value
#email.expression
def email(cls):
return cls.data['email'].astext
class Parser2:
#hybrid_property
def email(self):
return self.data['email_address']
#email.setter
def email(self, value):
self.data['email_address'] = value
#email.expression
def email(cls):
return cls.data['email_address'].astext
class Order(Base):
__tablename__ = "orders"
id_= Column(Integer, primary_key=True, nullable=False)
type = Column(String)
data = Column(JSONB)
if type == 'one':
extend with class "Parser1"
elif type == 'two':
extend with class "Parser2"
Would this work or is it stupid idea?
Do you have any better solution?
Any hints very welcome.

Update function for a Flask-SQLAlchemy model

I often use an update function when working with Flask-SQLAlchemy models:
from app import db
class User(db.Model):
__tablename__ = 'User'
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(255))
name = db.Column(db.String(255))
def update(self, email=None, name=None):
if email is not None:
self.email = email
if name is not None:
self.name = name
def dump(self):
return dict([(k, v) for k, v in vars(self).items() if not k.startswith("_")])
This allows me to directly update an object with a json body:
user.update(**body)
But with a table containing a lot of columns, writing this function can be really annoying.
Do you know a more concise approach?

You can iterate over dict fields and use setattr to update:
for field, value in body.items():
if value is not None:
setattr(self, field, value)

SQLAlchemy how to create a relationship or mapping of a table's "type" row to the table model or mixin

I need to set the entity_type_id as a column value when I persist a row to a generic table of various entity_types. I should be able to load the entity_type_id for every specific instance at instantiation time because it is accessible via a simple select statement. I'd like to have that id automatically retrieved/set at the class (or instance) level without executing a query and/or manually setting to every time I persist a row of an "entity_type".
I tried an entity_type_id #property on the mixin that returns the id of the entity_type using the object_session but for reasons I don't fully understand the orm still inserts null as the entity_type_id value when I commit/flush the session. (my guess is having the "property" itself isn't the same thing as setting the attribute value on the instance and/or causing an issue because the column name from the base class has the same name)
Here's a slimmed down version of the relevant models in my schema:
class EntityType(Base):
__tablename__ = 'entity_type'
id = Column(UUID(as_uuid=True), primary_key=True, server_default=FetchedValue())
table_name = Column(String, nullable=False)
ui_label = Column(Text, unique=True, nullable=False)
entry_key = Column(Text, unique=True, nullable=False)
Base class model:
class TrackedEntity(Base):
#declared_attr
def __tablename__(cls):
return convert(cls.__name__)
__table_args__ = (
UniqueConstraint('entity_type_id', 'label'),
)
id = Column(UUID(as_uuid=True), primary_key=True, server_default=FetchedValue())
entity_type_id = Column('entity_type_id', ForeignKey('entity_type.id'))
label = Column('label', String, nullable=False)
entity_type = relationship('EntityType')
polymorphic_discriminator = column_property(select([EntityType.table_name]).where(EntityType.id == entity_type_id).as_scalar())
#declared_attr
def entity_type_label(cls):
return association_proxy('entity_type', 'label')
#declared_attr
def __mapper_args__(cls):
if cls.__name__ == 'TrackedEntity':
return {
"polymorphic_on": cls.polymorphic_discriminator,
"polymorphic_identity": cls.__tablename__
}
else:
return {"polymorphic_identity": cls.__tablename__}
Children class mixin:
class TrackedEntityMixin(object):
# noinspection PyMethodParameters
#declared_attr
def id(cls) -> Column:
return Column(ForeignKey('tracked_entity.id'), primary_key=True)
#gets me the id but isn't very helpful like this, still needs to be manually set like child.entity_type_id = child._entity_type_id
#property
def _entity_type_id(self):
return object_session(self). \
scalar(
select([EntityType.id]).
where(EntityType.table_name == self.__tablename__)
)
A child class model:
class DesignedMolecule(TrackedEntityMixin, TrackedEntity):
extra = Column('extra', String)
parents = relationship('TrackedEntity', secondary='mix_dm_parent_entity')

How to insert many-to-many relationship data using SQLAlchemy with enum column?

I'm using flask with SQLAlchemy, there is a User class with a many-to-many relation to Role. Role has a name property which is an enum (RoleType).
When I try to insert a new user with a default User role:
with client.session() as session:
r = session.query(Role).filter(RoleType.USER.name == Role.name).first()
user.roles = [r]
session.add(user)
session.commit()
I get the following exception:
'RoleType' object has no attribute 'translate'
Here are the User, Role and RoleType classes:
from sqlalchemy import ForeignKey, Column, String, Integer, Boolean, Table
from sqlalchemy.orm import relationship, backref
user_role_association_table = Table('user_role', BaseRelation.metadata,
Column('user_id', Integer, ForeignKey('user.id')),
Column('role_name', Integer, ForeignKey('role.name')))
class User(BaseRelation):
id = Column(String, primary_key=True, nullable=False)
email = Column(String, nullable=False)
password = Column(String, nullable=False)
active = Column(Boolean, default=False, nullable=False)
roles = relationship(Role,
lazy='subquery',
secondary=user_role_association_table)
from enum import auto, Enum
class Role(BaseRelation):
name = Column(Enum(RoleType), primary_key=True)
def __init__(self, **kwargs):
name = kwargs.get("name", None)
if (name is not None) and isinstance(name, str):
kwargs["name"] = RoleType.get_type(name)
super(Role, self).__init__(**kwargs)
def __eq__(self, other):
if isinstance(other, dict):
return self.name == other.get('name', None)
return self.name == other.name
def __hash__(self):
return hash(self.name)
def __json__(self):
return ['name']
class RoleType(Enum):
USER = auto()
ADMIN = auto()

Combine SQLAlchemy hybrid_property with native property construction

I have a User class in SQLAlchemy. I want to be able to encrypt the user's email address attribute in the database but still make it searchable through the filter query.
My problem is that if I use #hybrid_property my query theoretically works, but my construction doesn't, and if I use #property my construction works but my query doesn't
from cryptography.fernet import Fernet # <- pip install cryptography
from werkzeug.security import generate_password_hash
class User(db.Model):
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
email_hash = db.Column(db.String(184), unique=True, nullable=False)
password_hash = db.Column(db.String(128))
# #property # <- Consider this as option 2...
#hybrid_property # <- Consider this as option 1...
def email(self):
f = Fernet('SOME_ENC_KEY')
value = f.decrypt(self.email_hash.encode('utf-8'))
return value
#email.setter
def email(self, email):
f = Fernet('SOME_ENC_KEY')
self.email_hash = f.encrypt(email.encode('utf-8'))
#property
def password(self):
raise AttributeError('password is not a readable attribute.')
#password.setter
def password(self, password):
self.password_hash = generate_password_hash(password)
def __init__(self, **kwargs):
super(User, self).__init__(**kwargs)
# other checks and modifiers
For option 1: When I attempt to construct a user with User(email='a#example.com',password='secret') I receive the traceback,
~/models.py in __init__(self, **kwargs)
431 # Established role assignment by default class initiation
432 def __init__(self, **kwargs):
--> 433 super(User, self).__init__(**kwargs)
434 if self.role is None:
435 _default_role = Role.query.filter_by(default=True).first()
~/lib/python3.6/site-packages/sqlalchemy/ext/declarative/base.py in _declarative_constructor(self, **kwargs)
697 raise TypeError(
698 "%r is an invalid keyword argument for %s" %
--> 699 (k, cls_.__name__))
700 setattr(self, k, kwargs[k])
701 _declarative_constructor.__name__ = '__init__'
TypeError: 'email' is an invalid keyword argument for User
For option 2: If instead I change #hybrid_property to #property the construction is fine but then my query User.query.filter_by(email=form.email.data.lower()).first() fails and returns None.
What should I change to get it working as required?
==============
Note I should say that I have tried to avoid using dual attributes since I didn't want to make extensive edits to the underlying codebase. so I have explicitly tried to avoid separating creation with querying in terms of User(email_input='a#a.com', password='secret') and User.query.filter_by(email='a#a.com').first():
class User(db.Model):
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
email_hash = db.Column(db.String(184), unique=True, nullable=False)
password_hash = db.Column(db.String(128))
#hybrid_property
def email(self):
f = Fernet('SOME_ENC_KEY')
value = f.decrypt(self.email_hash.encode('utf-8'))
return value
#property
def email_input(self):
raise AttributeError('email_input is not a readable attribute.')
#email_input.setter
def email_input(self, email):
f = Fernet('SOME_ENC_KEY')
self.email_hash = f.encrypt(email.encode('utf-8'))
#property
def password(self):
raise AttributeError('password is not a readable attribute.')
#password.setter
def password(self, password):
self.password_hash = generate_password_hash(password)
def __init__(self, **kwargs):
super(User, self).__init__(**kwargs)
# other checks and modifiers

In your hybrid_property, email, the line self.f.decrypt(self.email_hash.encode('utf-8')) is fine if self.email_hash is a str type, however, as email is a hybrid_property, when SQLAlchemy uses it to generate SQL self.email_hash is actually a sqlalchemy.orm.attributes.InstrumentedAttribute type.
From the docs regarding hybrid properties:
In many cases, the construction of an in-Python function and a
SQLAlchemy SQL expression have enough differences that two separate
Python expressions should be defined.
And so you can define an hybrid_property.expression method which is what SQLAlchemy will use to generate sql, allowing you to keep your string treatment intact in your hybrid_property method.
Here is the code I ended up with that worked for me given your example. I've stripped quite a bit out of your User model for simplicity but all the important parts are there. I also had to make up implementations for other functions/classes that were called in your code but not supplied (see MCVE):
class Fernet:
def __init__(self, k):
self.k = k
def encrypt(self, s):
return s
def decrypt(self, s):
return s
def get_env_variable(s):
return s
def generate_password_hash(s):
return s
class User(db.Model):
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
email_hash = db.Column(db.String(184), unique=True, nullable=False)
f = Fernet(get_env_variable('FERNET_KEY'))
#hybrid_property
def email(self):
return self.f.decrypt(self.email_hash.encode('utf-8'))
#email.expression
def email(cls):
return cls.f.decrypt(cls.email_hash)
#email.setter
def email(self, email):
self.email_hash = self.f.encrypt(email.encode('utf-8'))
if __name__ == '__main__':
db.drop_all()
db.create_all()
u = User(email='a#example.com')
db.session.add(u)
db.session.commit()
print(User.query.filter_by(email='a#example.com').first())
# <User 1>
Unfortunately, the code above only works because the mock Fernet.decrypt method returns the exact object that was passed in. The problem with storing a Fernet encoded hash of the user's email addresses is that Fernet.encrypt does not return the same fernet token from one execution to the next, even with the same key. E.g.:
>>> from cryptography.fernet import Fernet
>>> f = Fernet(Fernet.generate_key())
>>> f.encrypt('a#example.com'.encode('utf-8')) == f.encrypt('a#example.com'.encode('utf-8'))
False
So, you want to query a database for a record, but with no way of knowing what the stored value of field that you are querying actually is at query time. You could build a classmethod that queries the entire users table and loop through each record, decrypting it's stored hash and comparing it to the clear text email. Or you can build a hashing function that will always return the same value, hash new users emails using that function and query the email_hash field directly with the hash of the email string. Of those, the first would be very inefficient given lots of users.
The Fernet.encrypt function is:
def encrypt(self, data):
current_time = int(time.time())
iv = os.urandom(16)
return self._encrypt_from_parts(data, current_time, iv)
So, you could define static values of current_time and iv and directly call Fermat._encrypt_from_parts yourself. Or you could use python's built in hash and just set a fixed seed so that it is deterministic. You could then hash the email string that you want to query and first and directly query Users.email_hash. As long as you didn't do any of the above for password fields!