Lets say that I have a database structure with three tables that look like this:
items
- item_id
- item_handle
attributes
- attribute_id
- attribute_name
item_attributes
- item_attribute_id
- item_id
- attribute_id
- attribute_value
I would like to be able to do this in SQLAlchemy:
item = Item('item1')
item.foo = 'bar'
session.add(item)
session.commit()
item1 = session.query(Item).filter_by(handle='item1').one()
print item1.foo # => 'bar'
I'm new to SQLAlchemy and I found this in the documentation (http://www.sqlalchemy.org/docs/05/mappers.html#mapping-a-class-against-multiple-tables):
j = join(items, item_attributes, items.c.item_id == item_attributes.c.item_id). \
join(attributes, item_attributes.c.attribute_id == attributes.c.attribute_id)
mapper(Item, j, properties={
'item_id': [items.c.item_id, item_attributes.c.item_id],
'attribute_id': [item_attributes.c.attribute_id, attributes.c.attribute_id],
})
It only adds item_id and attribute_id to Item and its not possible to add attributes to Item object.
Is what I'm trying to achieve possible with SQLAlchemy? Is there a better way to structure the database to get the same behaviour of "dynamic columns"?
This is called the entity-attribute-value pattern. There is an example about this under the SQLAlchemy examples directory: vertical/.
If you are using PostgreSQL, then there is also the hstore contrib module that can store a string to string mapping. If you are interested then I have some code for a custom type that makes it possible to use that to store extended attributes via SQLAlchemy.
Another option to store custom attributes is to serialize them to a text field. In that case you will lose the ability to filter by attributes.
The link to vertical/vertical.py is broken. The example had been renamed to dictlike-polymorphic.py and dictlike.py.
I am pasting in the contents of dictlike.py:
"""Mapping a vertical table as a dictionary.
This example illustrates accessing and modifying a "vertical" (or
"properties", or pivoted) table via a dict-like interface. These are tables
that store free-form object properties as rows instead of columns. For
example, instead of::
# A regular ("horizontal") table has columns for 'species' and 'size'
Table('animal', metadata,
Column('id', Integer, primary_key=True),
Column('species', Unicode),
Column('size', Unicode))
A vertical table models this as two tables: one table for the base or parent
entity, and another related table holding key/value pairs::
Table('animal', metadata,
Column('id', Integer, primary_key=True))
# The properties table will have one row for a 'species' value, and
# another row for the 'size' value.
Table('properties', metadata
Column('animal_id', Integer, ForeignKey('animal.id'),
primary_key=True),
Column('key', UnicodeText),
Column('value', UnicodeText))
Because the key/value pairs in a vertical scheme are not fixed in advance,
accessing them like a Python dict can be very convenient. The example below
can be used with many common vertical schemas as-is or with minor adaptations.
"""
class VerticalProperty(object):
"""A key/value pair.
This class models rows in the vertical table.
"""
def __init__(self, key, value):
self.key = key
self.value = value
def __repr__(self):
return '<%s %r=%r>' % (self.__class__.__name__, self.key, self.value)
class VerticalPropertyDictMixin(object):
"""Adds obj[key] access to a mapped class.
This is a mixin class. It can be inherited from directly, or included
with multiple inheritence.
Classes using this mixin must define two class properties::
_property_type:
The mapped type of the vertical key/value pair instances. Will be
invoked with two positional arugments: key, value
_property_mapping:
A string, the name of the Python attribute holding a dict-based
relationship of _property_type instances.
Using the VerticalProperty class above as an example,::
class MyObj(VerticalPropertyDictMixin):
_property_type = VerticalProperty
_property_mapping = 'props'
mapper(MyObj, sometable, properties={
'props': relationship(VerticalProperty,
collection_class=attribute_mapped_collection('key'))})
Dict-like access to MyObj is proxied through to the 'props' relationship::
myobj['key'] = 'value'
# ...is shorthand for:
myobj.props['key'] = VerticalProperty('key', 'value')
myobj['key'] = 'updated value']
# ...is shorthand for:
myobj.props['key'].value = 'updated value'
print myobj['key']
# ...is shorthand for:
print myobj.props['key'].value
"""
_property_type = VerticalProperty
_property_mapping = None
__map = property(lambda self: getattr(self, self._property_mapping))
def __getitem__(self, key):
return self.__map[key].value
def __setitem__(self, key, value):
property = self.__map.get(key, None)
if property is None:
self.__map[key] = self._property_type(key, value)
else:
property.value = value
def __delitem__(self, key):
del self.__map[key]
def __contains__(self, key):
return key in self.__map
# Implement other dict methods to taste. Here are some examples:
def keys(self):
return self.__map.keys()
def values(self):
return [prop.value for prop in self.__map.values()]
def items(self):
return [(key, prop.value) for key, prop in self.__map.items()]
def __iter__(self):
return iter(self.keys())
if __name__ == '__main__':
from sqlalchemy import (MetaData, Table, Column, Integer, Unicode,
ForeignKey, UnicodeText, and_, not_)
from sqlalchemy.orm import mapper, relationship, create_session
from sqlalchemy.orm.collections import attribute_mapped_collection
metadata = MetaData()
# Here we have named animals, and a collection of facts about them.
animals = Table('animal', metadata,
Column('id', Integer, primary_key=True),
Column('name', Unicode(100)))
facts = Table('facts', metadata,
Column('animal_id', Integer, ForeignKey('animal.id'),
primary_key=True),
Column('key', Unicode(64), primary_key=True),
Column('value', UnicodeText, default=None),)
class AnimalFact(VerticalProperty):
"""A fact about an animal."""
class Animal(VerticalPropertyDictMixin):
"""An animal.
Animal facts are available via the 'facts' property or by using
dict-like accessors on an Animal instance::
cat['color'] = 'calico'
# or, equivalently:
cat.facts['color'] = AnimalFact('color', 'calico')
"""
_property_type = AnimalFact
_property_mapping = 'facts'
def __init__(self, name):
self.name = name
def __repr__(self):
return '<%s %r>' % (self.__class__.__name__, self.name)
mapper(Animal, animals, properties={
'facts': relationship(
AnimalFact, backref='animal',
collection_class=attribute_mapped_collection('key')),
})
mapper(AnimalFact, facts)
metadata.bind = 'sqlite:///'
metadata.create_all()
session = create_session()
stoat = Animal(u'stoat')
stoat[u'color'] = u'reddish'
stoat[u'cuteness'] = u'somewhat'
# dict-like assignment transparently creates entries in the
# stoat.facts collection:
print stoat.facts[u'color']
session.add(stoat)
session.flush()
session.expunge_all()
critter = session.query(Animal).filter(Animal.name == u'stoat').one()
print critter[u'color']
print critter[u'cuteness']
critter[u'cuteness'] = u'very'
print 'changing cuteness:'
metadata.bind.echo = True
session.flush()
metadata.bind.echo = False
marten = Animal(u'marten')
marten[u'color'] = u'brown'
marten[u'cuteness'] = u'somewhat'
session.add(marten)
shrew = Animal(u'shrew')
shrew[u'cuteness'] = u'somewhat'
shrew[u'poisonous-part'] = u'saliva'
session.add(shrew)
loris = Animal(u'slow loris')
loris[u'cuteness'] = u'fairly'
loris[u'poisonous-part'] = u'elbows'
session.add(loris)
session.flush()
q = (session.query(Animal).
filter(Animal.facts.any(
and_(AnimalFact.key == u'color',
AnimalFact.value == u'reddish'))))
print 'reddish animals', q.all()
# Save some typing by wrapping that up in a function:
with_characteristic = lambda key, value: and_(AnimalFact.key == key,
AnimalFact.value == value)
q = (session.query(Animal).
filter(Animal.facts.any(
with_characteristic(u'color', u'brown'))))
print 'brown animals', q.all()
q = (session.query(Animal).
filter(not_(Animal.facts.any(
with_characteristic(u'poisonous-part', u'elbows')))))
print 'animals without poisonous-part == elbows', q.all()
q = (session.query(Animal).
filter(Animal.facts.any(AnimalFact.value == u'somewhat')))
print 'any animal with any .value of "somewhat"', q.all()
# Facts can be queried as well.
q = (session.query(AnimalFact).
filter(with_characteristic(u'cuteness', u'very')))
print 'just the facts', q.all()
metadata.drop_all()
Related
Hi I have a table in 3NF form
ftype_table = Table(
'FTYPE',
Column('ftypeid', Integer, primary_key=True),
Column('typename', String(50)),
base.metadata,
schema='TEMP')
file_table = Table(
'FILE',
base.metadata,
Column('fileid', Integer, primary_key=True),
Column('datatypeid', Integer, ForeignKey(ftype_table.c.datatypeid)),
Column('size', Integer),
schema='TEMP')
and mappers
class File(object): pass
class FileType(object): pass
mapper(File, file_table, properties={'filetype': relation(FileType)})
mapper(FileType, file_table)
suppose Ftype table contains 1:TXT 2:AVI 3:PPT
what i would like to do is the following if i create a File object like this:
file=File()
file.size=10
file.filetype= FileType('PPT')
Session.save(file)
Session.flush()
is that the File table contains fileid:xxx,size:10, datatypeid:3
Unfortunately an entry gets added to the FileType table and this id gets propagated to the File table.
Is there a smart way to do achieve the above with sqlalchemy witout the need to do a query on the FileType table to see if the entry exist or not
Thanks
the UniqueObject recipe is the standard answer here: http://www.sqlalchemy.org/trac/wiki/UsageRecipes/UniqueObject . The idea is to override the creation of File using either __metaclass__.call() or File.__new__() to return the already-existing object, from the DB or from cache (the initial DB lookup, if the object isn't already present, is obviously unavoidable unless something constructed around MySQL's REPLACE is used).
edit: since I've been working on the usage recipes, I've rewritten the unique object recipe to be more portable and updated for 0.5/0.6.
Just create a cache of FileType objects, so that the database lookup occurs only the first time you use a given file type:
class FileTypeCache(dict):
def __missing__(self, key):
obj = self[key] = Session.query(FileType).filter_by(typename=key).one()
return obj
filetype_cache = FileTypeCache()
file=File()
file.size=10
file.filetype= filetype_cache['PPT']
should work, modulo typos.
Since declarative_base and zzzeek code does not work with sqlalchemy 0.4, I
used the following cache so that new objects also stay unique if they are not present in the db
class FileTypeCache(dict):
def __missing__(self, key):
try:
obj = self[key] = Session.query(FileType).filter_by(typename=key).one()
return obj
except InvalidRequestError:
return obj=self[key]= FileType(key)
return obj
override eq of FileType
class FileType(object):
def __init__(self, typename)
self.typename=typename
def __eq__(self):
if isinstance(other, FileType):
return self.typename == other.typename
else:
return False
I'm trying to model an entity that as one or more one-to-many relationships, such that it's last_modified attribute is updated, when
a child is added or removed
a child is modified
the entity itself is modified
I've put together the following minimal example:
class Config(Base):
__tablename__ = 'config'
ID = Column('ID', Integer, primary_key=True)
name = Column('name', String)
last_modified = Column('last_modified', DateTime, default=now, onupdate=now)
params = relationship('ConfigParam', backref='config')
class ConfigParam(Base):
__tablename__ = 'config_params'
ID = Column('ID', Integer, primary_key=True)
ConfigID = Column('ConfigID', Integer, ForeignKey('config.ID'), nullable=False)
key = Column('key', String)
value = Column('value', Float)
#event.listens_for(Config.params, 'append')
#event.listens_for(Config.params, 'remove')
def receive_append_or_remove(target, value, initiator):
target.last_modified = now()
#event.listens_for(ConfigParam.key, 'set')
#event.listens_for(ConfigParam.value, 'set')
def receive_attr_change(target, value, oldvalue, initiator):
if target.config:
# don't act if the parent config isn't yet set
# i.e. during __init__
target.config.last_modified = now()
This seems to work, but I'm wondering if there's a better way to do this?
Specifically, this becomes very verbose since my actual ConfigParam implementation has more attributes and I'm having multiple one-to-many relations configured on the parent Config class.
Take this with a huge grain of salt, it "seems" to work, could explode:
def rel_listener(t, v, i):
t.last_modified = now()
def listener(t, v, o, i):
if t.config:
t.config.last_modified = now()
from sqlalchemy import inspect
for rel in inspect(Config).relationships:
event.listen(rel, 'append', rel_listener)
event.listen(rel, 'remove', rel_listener)
for col in inspect(ConfigParam).column_attrs:
event.listen(col, 'set', listener)
Problem is that the inspections make no exceptions and columns such as 'ID' and 'ConfigID' will be bound to event listeners.
Another perhaps slightly less tedious form would be to just use a list of attributes to bind events to in a similar fashion:
for attr in ['key', 'value']:
event.listen(getattr(ConfigParam, attr), 'set', listener)
This gives you control over what is bound to events and what is not.
I need to implement a "related items" feature, i.e. to allow items from the same table to be arbitrarily linked to each other in a many-to-many fashion. Something similar to how news websites show related articles.
Also, I need the relationship to be bi-directional, something like this:
a = Item()
b = Item()
a.related.append(b)
assert a in b.related # True
Now, on SQL level I imagine this could be solved by modifying the "standard" many-to-many relationship so 2 records are inserted into the association table each time an association is made, so (a -> b) and (b -> a) are two separate records.
Alternatively, the join condition for the many-to-many table could somehow check both sides of the association, so roughly instead of ... JOIN assoc ON a.id = assoc.left_id ... SQLAlchemy would produce something like ... JOIN assoc ON a.id = assoc.left_id OR a.id = assoc.right_id ...
Is there a way to configure this with SQLAlchemy so the relation works similar to a "normal" many-to-many relationship?
It's likely that I'm just don't know the correct terminology - everything I came up with - "self-referential", "bidirectional", "association" - is used to describe something else in SQLAlchemy.
Using Attribute Events should do the job. See the sample code below, where little ugly piece of code is solely for the purpose of avoid endless recursion:
class Item(Base):
__tablename__ = "item"
id = Column(Integer, primary_key=True)
name = Column(String(255), nullable=False)
# relationships
related = relationship('Item',
secondary = t_links,
primaryjoin = (id == t_links.c.from_id),
secondaryjoin = (id == t_links.c.to_id),
)
_OTHER_SIDE = set()
from sqlalchemy import event
def Item_related_append_listener(target, value, initiator):
global _OTHER_SIDE
if not((target, value) in _OTHER_SIDE):
_OTHER_SIDE.add((value, target))
if not target in value.related:
value.related.append(target)
else:
_OTHER_SIDE.remove((target, value))
event.listen(Item.related, 'append', Item_related_append_listener)
# ...
a = Item()
b = Item()
a.related.append(b)
assert a in b.related # True
For completeness sake, here's the code I ended up with; the listener method is slightly different to avoid using a global variable, an also there's a listener for remove event.
import sqlalchemy as sa
related_items = sa.Table(
"related_items",
Base.metadata,
sa.Column("id", sa.Integer, primary_key=True),
sa.Column("from_id", sa.ForeignKey("items.id")),
sa.Column("to_id", sa.ForeignKey("items.id")),
)
class Item(Base):
__tablename__ = 'items'
...
related = sa.orm.relationship('Item',
secondary = related_items,
primaryjoin = (id == related_items.c.from_id),
secondaryjoin = (id == related_items.c.to_id),
)
def item_related_append_listener(target, value, initiator):
if not hasattr(target, "__related_to__"):
target.__related_to__ = set()
target.__related_to__.add(value)
if target not in getattr(value, "__related_to__", set()):
value.related.append(target)
sa.event.listen(Item.related, 'append', item_related_append_listener)
def item_related_remove_listener(target, value, initiator):
if target in value.related:
value.related.remove(target)
sa.event.listen(Item.related, 'remove', item_related_remove_listener)
I have several classes, and want to work with their collections like with a DB (or Django ORM, but simpler). Working with a DB would be a huge overhead, so I'd prefer having similar functionality in memory:
>>> node = Nodes.create(lat=X,lon=Y)
>>> node.id
1
>>> Nodes.all()
[<Node 1>]
>>> Nodes[1] # by id
[<Node 1>]
>>> way_nodes = map(Nodes.create, ((X, Y), (Z, W)))
>>> way = Ways.create(way_nodes)
>>> way.nodes
[<Node 2>, <Node 3>]
>>> way.id
1
This is basically all what I need. Is there anything similar in Python or in custom packages?
If none, and I have to write my own, what's the best choice to inherit from?
Those seem like pretty basic requirements, why not code them into your classes?
from itertools import count, starmap
class Node(object):
nextid = count(1).next
nodes = {}
def __init__(self, lat, lon):
self.lat, self.lon = lat, lon
self.id = self.nextid()
self.nodes[self.id] = self
#classmethod
def all(cls):
return cls.nodes.values()
def __repr__(self):
return '<Node %s>' % self.id
#classmethod
def byid(cls, id):
return cls.nodes[id]
class Way(object):
nextid = count(1).next
def __init__(self, nodes):
self.nodes = nodes[:]
self.id = self.nextid()
node = Node(lat='X',lon='Y')
print node.id
print Node.all()
print Node.byid(1)
way_nodes = list(starmap(Node, (('X', 'Y'), ('Z', 'W'))))
way = Way(way_nodes)
print way.nodes
print way.id
Working with a proper relational database is not necessarily a huge overhead. Are you aware of SQLite? (python module sqlite3) It can work with an in-memory database.
import sqlite3
db_conn = sqlite3.connect(":memory:")
db_conn.execute("CREATE TABLE nodes (id INTEGER PRIMARY KEY AUTOINCREMENT, lat FLOAT, long FLOAT);")
But for what you want to do, I'm not sure you need any relational semantics at all. (All you seem to be doing is creating indexable lists of Nodes - would a dict of node_id : Node objects do for you?
In passing, map() does exist in Python, but it's considered more "Pythonic" to use a list comprehension. The reasons for using list comprehensions are:
The syntax is more intuitive
List comprehensions combine the features of both map() and filter().
Instead of saying
way_nodes = map(Nodes.create, ((X,Y),(Z,W))
we would say
way_nodes = [Nodes.create(*coord_pair) for coord_pair in ((X,Y),(Z,W))]
Both are equally correct, but the second form is preferred.
You could use the builtin list type, if node is a list:
node[0] # get by id (id == 0)
node # .all()
If you'd like to create special classes to implement desired interface then by writing about the same amount of code as you would for your custom classes you could get full-power of relational databases using SQLAlchemy:
# get node id
node.id # -> 1
# get all nodes
Node.query.all() # -> [<Node 1>, <Node 2>, <Node 3>, <Node 4>]
# get all nodes associated with the `way`
way.nodes # -> [<Node 2>, <Node 3>, <Node 4>]
# get `way` with id == 1
Way.query.get(1) # -> <Way 1>
# get all `way`s the node belongs to
node.ways.all() # -> []
# get all nodes with longitude == Y
Node.query.filter_by(lon=Y).all() # -> [<Node 1>, <Node 2>]
Here's the code that implements it:
from sqlalchemy import create_engine, Column, Integer, Float, Table, ForeignKey
from sqlalchemy.orm import scoped_session, sessionmaker, relationship, backref
from sqlalchemy.ext.declarative import declarative_base, declared_attr
# declare models
class Base(declarative_base()):
__abstract__ = True
# provide id for each object
id = Column(Integer, primary_key=True)
#classmethod
def create(cls, *args, **kwargs):
obj = cls(*args, **kwargs)
Session.add(obj) # add to db session
return obj
#declared_attr
def __tablename__(cls):
"""Use lowercased classname for table name."""
return cls.__name__.lower()
def __repr__(self):
return "<{name} {id}>".format(name=self.__class__.__name__, id=self.id)
class Node(Base):
lat = Column(Float)
lon = Column(Float)
def __init__(self, lat, lon):
self.lat = lat
self.lon = lon
# define many-to-many relationship
nodes = Table(
'nodes', Base.metadata,
Column('node_id', Integer, ForeignKey('node.id')),
Column('way_id', Integer, ForeignKey('way.id'))
)
class Way(Base):
nodes = relationship(Node, secondary=nodes,
backref=backref('ways', lazy='dynamic'))
def __init__(self, nodes):
self.nodes = nodes
Test
# create in memory db
engine = create_engine('sqlite://')
# one db session per thread
Session = scoped_session(sessionmaker(bind=engine))
Base.query = Session.query_property() # allow queries via Model.query
#
Base.metadata.create_all(bind=engine)
X, Y, Z, W = range(4)
node = Node.create(lat=X, lon=Y)
way_nodes = [Node.create(*args) for args in [(X, Y), (Z, W)]]
way = Way.create(way_nodes)
way.nodes.append(Node(10, 11))
Session.commit()
assert node.id == 1
assert node.ways.all() == []
assert Node.query.filter_by(lon=Y).count() == 2
assert way.id == 1
assert any(x.lat == 10 and x.lon == 11 for x in way.nodes)
assert Node.query.filter_by(lat=10, lon=11).one()
assert Way.query.get(1) == way
# remove context (thread) -local session
Session.remove()
Hi I have a table in 3NF form
ftype_table = Table(
'FTYPE',
Column('ftypeid', Integer, primary_key=True),
Column('typename', String(50)),
base.metadata,
schema='TEMP')
file_table = Table(
'FILE',
base.metadata,
Column('fileid', Integer, primary_key=True),
Column('datatypeid', Integer, ForeignKey(ftype_table.c.datatypeid)),
Column('size', Integer),
schema='TEMP')
and mappers
class File(object): pass
class FileType(object): pass
mapper(File, file_table, properties={'filetype': relation(FileType)})
mapper(FileType, file_table)
suppose Ftype table contains 1:TXT 2:AVI 3:PPT
what i would like to do is the following if i create a File object like this:
file=File()
file.size=10
file.filetype= FileType('PPT')
Session.save(file)
Session.flush()
is that the File table contains fileid:xxx,size:10, datatypeid:3
Unfortunately an entry gets added to the FileType table and this id gets propagated to the File table.
Is there a smart way to do achieve the above with sqlalchemy witout the need to do a query on the FileType table to see if the entry exist or not
Thanks
the UniqueObject recipe is the standard answer here: http://www.sqlalchemy.org/trac/wiki/UsageRecipes/UniqueObject . The idea is to override the creation of File using either __metaclass__.call() or File.__new__() to return the already-existing object, from the DB or from cache (the initial DB lookup, if the object isn't already present, is obviously unavoidable unless something constructed around MySQL's REPLACE is used).
edit: since I've been working on the usage recipes, I've rewritten the unique object recipe to be more portable and updated for 0.5/0.6.
Just create a cache of FileType objects, so that the database lookup occurs only the first time you use a given file type:
class FileTypeCache(dict):
def __missing__(self, key):
obj = self[key] = Session.query(FileType).filter_by(typename=key).one()
return obj
filetype_cache = FileTypeCache()
file=File()
file.size=10
file.filetype= filetype_cache['PPT']
should work, modulo typos.
Since declarative_base and zzzeek code does not work with sqlalchemy 0.4, I
used the following cache so that new objects also stay unique if they are not present in the db
class FileTypeCache(dict):
def __missing__(self, key):
try:
obj = self[key] = Session.query(FileType).filter_by(typename=key).one()
return obj
except InvalidRequestError:
return obj=self[key]= FileType(key)
return obj
override eq of FileType
class FileType(object):
def __init__(self, typename)
self.typename=typename
def __eq__(self):
if isinstance(other, FileType):
return self.typename == other.typename
else:
return False