We have a need to create SQLAlchemy classes to access multiple external data sources that will increase in number over time. We use the declarative base for our core ORM models and I know we can manually specify new ORM classes using the autoload=True to auto generate the mapping.
The problem is that we need to be able generate them dynamically taking something like this:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
stored={}
stored['tablename']='my_internal_table_name'
stored['objectname']='MyObject'
and turning it into something like this dynamically:
class MyObject(Base):
__tablename__ = 'my_internal_table_name'
__table_args__ = {'autoload':True}
We don't want the classes to persist longer than necessary to open a connection, perform the queries, and then closing the connection. Therefore, ideally, we can put the items in the "stored" variable above into a database and pull them as needed. The other challenge is that the object name (e.g. "MyObject") may be used on different connections so we cannot define it once and keep it around.
Any suggestions on how this might be accomplished would be greatly appreciated.
Thanks...
You can dynamically create MyObject using the 3-argument call to type:
type(name, bases, dict)
Return a new type object. This is essentially a dynamic form of the
class statement...
For example:
mydict={'__tablename__':stored['tablename'],
'__table_args__':{'autoload':True},}
MyObj=type(stored['objectname'],(Base,),mydict)
print(MyObj)
# <class '__main__.MyObject'>
print(MyObj.__base__)
# <class '__main__.Base'>
print(MyObj.__tablename__)
# my_internal_table_name
print(MyObj.__table_args__)
# {'autoload': True}
Related
Code/Example
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
If I run print(Base) then I get result:
<class 'sqlalchemy.ext.declarative.api.Base'>
(I.e Base is a class created by the function declarative_base).
If I run print(Base.metadata) then I get
Metadata(bind=engine(postgres//:user:password#host/database))
But exactly how does print(Base.metadata) even run if Base is a class and metadata is an object?
How are the two connected?
I could understand if metadata was a class attribute but from what I understand it is not so how does this work?
Is there something in python syntax that I have missed?
Looking in metadata code:
from sqlalchemy import MetaData
metadata = MetaData()
What is here the definition of metadata and MetaData?
Are they both objects?
Is MetaData() a function or something else?
According to SQLAlchemy documentation MetaData is:
a container object that keeps together many different features of a database (or multiple databases) being described.
But if it is an object so then how can it be run as something similar to a function: Metadata()?
Summary:
How are Base and metadata connected if one is a class and the other is an object?
How can the command (Base.metadata) be run?
What is the difference between metadata and MetaData and what is the MetaData() definition?
You are overthinking this. In Python everything is an object. Classes, functions, instances, numbers, strings, lists, dictionaries, modules, ..., everything.
The SQLAlchemy objects you are looking at are no different, there really is nothing special about these.
Yes, the declarative_base() function returns a new class object, one you assigned to the name Base. That class has attributes, and one of those attributes is named metadata. All Python classes have attributes, they can point to any object you like:
class Foo:
def __init__(self, name):
self.name = name
def print(self):
print(f"Hello, {self.name}!")
class Bar:
spam = Foo("World")
The above class Bar has an attribute spam that is an instance of the class Foo. You can call methods on it:
>>> Bar.spam.print()
Hello, World!
Note that Bar is still a class, and Bar.spam is an instance of the Foo class:
>>> Bar
<class '__main__.Bar'>
>>> Bar.spam
<__main__.Foo object at 0x10d43c650>
>>> type(Bar.spam)
<class '__main__.Foo'>
Because Bar is just another object, you can also pass it around, put it in a list, or return it from a function. declarative_base() constructed a class object for you, with attributes, and returned it to the caller.
Similarly, Base.metadata is just an instance of the MetaData class:
>>> type(Base.metadata)
<class 'sqlalchemy.sql.schema.MetaData'>
and so is your metadata = MetaData() object.
Note that if you only run Base = declarative_base() and nothing else, the MetaData instance is not actually showing any database connection information:
>>> Base.metadata
MetaData(bind=None)
That's because you didn't actually connect the metadata to a database session. Once you start connecting a database with an engine and start using the models you define by subclassing Base, the metadata object at some point will need to know more about the specifics of the database (such as what kind of SQL dialect is supported) and it'll be bound by SQLAlchemy.
Assume I have a class Interaction. My program processes interactions, in a way that each interaction updates a score table. Interaction is declared as an exact mapping of the database table, but I also want it to have a reference to the relevant instance of ScoreTable. ScoreTable is a class that holds the scores, and controls the business logic to update scores:
class Interaction(Base):
__tablename__ = 'interactions'
#Mirror the table's structure
anomaly_id = Column(Integer, ForeignKey('anomalies.id'), primary_key=True)
user_id = Column(Integer, primary_key=True) # ForeignKey('users.id'),
feedback_score = Column(Integer)
#scoreUpdater is another (non-ORM) object I want this instance to be aware of
myScoreUpdater= ...
All instances of Interaction fetched by a query() will share the same ScoreUpdater.So when I run the query() to get all my instances of Interactions, can I somehow tell the query() to set the scoreUpdater to a certain value, in the same process? Else, can I give query() a half-built template instance if Interaction to clone and populate the ORM data into?
I read that I could modify the standard constructor to perform certain tasks, but don't know how to pass extra arguments (such as the instance of ScoreUpdater) to the constructor via the query()
I guess the other way is to run the query first and let it populate the ORM-related fields, and then in a second step, iterate over the query results to set the non-OM fields (i.e. the right instance of scoreUpdater)?
I'm new to SQLalchemy ... and converting from java to python. So if you think my approach is fundamentally wrong, let me know!
The relevant documentation on constructing objects says:
The SQLAlchemy ORM does not call __init__ when recreating objects from database rows. The ORM’s process is somewhat akin to the Python standard library’s pickle module, invoking the low level __new__ method and then quietly restoring attributes directly on the instance rather than calling __init__.
If you need to do some setup on database-loaded instances before they’re ready to use, there is an event hook known as InstanceEvents.load() which can achieve this; it is also available via a class-specific decorator called orm.reconstructor(). When using orm.reconstructor(), the mapper will invoke the decorated method with no arguments every time it loads or reconstructs an instance of the class. This is useful for recreating transient properties that are normally assigned in __init__
So if I've understood you correctly, you could define a reconstructor for Interaction that populates the non-ORM fields:
from sqlalchemy.orm import reconstructor
class Interaction(Base):
__tablename__ = 'interactions'
# Mirror the table's structure
anomaly_id = Column(Integer, ForeignKey('anomalies.id'), primary_key=True)
user_id = Column(Integer, primary_key=True) # ForeignKey('users.id'),
feedback_score = Column(Integer)
# myScoreUpdater is another (non-ORM) object I want this instance to be aware of
#reconstructor
def init_on_load(self):
# Do what you must to populate score updater
self.myScoreUpdater = ...
Note that you'll probably want to share the logic between the reconstructor and __init__, so either just decorate __init__ as the reconstructor, if it can be called without arguments, or move initialization of score updater to a method.
Finally, if your ScoperUpdater does not actually need to know what instance it is bound to, you could just have it as a class attribute shared between all instances – like static attributes in Java.
I'm using SQLAlchemy in Python 3 and am confused why the following code works - it looks to me like what should be an object level variable is acting as a class variable. I have seen Why is instance variable behaving like a class variable in Python? which doesn't look related to my question.
I have the following declarations in a db module in which I create an instance of Base and set a query variable on it that points to the SessionFactory query_property()
import sqlalchemy as sa
import sqlalchemy.ext.declarative as sa_ed
Base = sa_ed.declarative_base()
engine = sa.create_engine('connection string')
session_factory = session or sa_orm.scoped_session(sa_orm.sessionmaker(autocommit=False, autoflush=False, bind=engine))
Base.query = session_factory.query_property() # make the query_property available in models for querying
My model classes are then declared as follows:
import db
class MyModel(db.Base):
id = Column(Integer)
# more model stuff
I can then run queries by accessing the query variable as follows:
return MyModel.query.filter(MyModel.id == 22).first()
This does work, but it looks as though the query variable exists at the class level and not at the object instance level as I am able to access it directly through the class.
What am I not understanding?
You put the query property on the parent class earlier:
Base.query = session_factory.query_property() # make the query_property available in models for querying
So query is most definitely a member of a class (Base). And since MyModel inherits from Base, MyModel should also have a query member (due to the magic of inheritance).
I'm building a webapp that has optional Facebook Login. The users created through the Facebook API are handled differently at several points in my application. I want to encapsulate these differences in a subclass of Person that overrides methods.
class Person(Model):
def get_profile_picture(self):
return profile_pictures.url(self.picture)
class FacebookPerson(Person):
def get_profile_picture(self):
return 'http:/.../%s.jpg' % self.graph_id
I would like to avoid the nasty if self.graph_id and just query the Person model and get the right object for each user.
I've thought of hacking the metaclass to add the FacebookPerson as a base. Obviously I would like to avoid such voodoo.
I'm using Flask and Flask-SQLAlchemy.
The general idea would be to store the model's class name as metadata in each row, and when you instantiate the object, do something like:
def query(self):
# stuff
return model_class(data)
To do this in SQLAlchemy, you might look at making Person the base class to something like BasicPerson and FacebookPerson, and in Person.init(), use the metadata to initialize to the proper subclass.
For example, the idea would be than when this query returns, user will have been initialized to the proper subclass:
user = session.query(Person).filter_by(name='james').first()
You will probably need to modify this concept a bit for SQLAlchemy (I haven't used it in a while), but that's the general idea.
Or, you could do something like store the metadata in a cookie with the user_id, and then when they log in again, use the metadata to pass the proper class to the user query:
user = session.query(FacebookPerson).filter_by(name='james').first()
If you want this to be generic so that the metatdata is meaningful to non-Python clients, instead of storing the model's class name, store the model's "object_type" and have something in each client library that maps object_types to classes.
In the low-level Java api, I used to be able to make up entity names at runtime.
In python, it seems that I have to pre-define a class name that is also my entity name:
class SomeKindaData(db.Expando):
pass
sKD = SomeKindaData(key_name='1')
...
Is there a way to make entity names up at run time in App Engine for Python?
I don't know much about App Engine, but you can define classes at run time like this:
def get_my_class(name):
return type(name, (db.Expando,), {})
So type takes three arguments:
name of class
tuple of classes to inherit from
dictionary of class attributes
Entities themselves don't have names. In the datastore, entities are identified by keys, which can have names or IDs. You're setting your entity's key name to "1" in your sample code. Entities are also classified by kind, in this case SomeKindaData.
db.Model and db.Expando provide a local ORM abstraction around the datastore. When you use these, your entity's kind name is set to your model class name by default. If you don't want to define a model class before creating an entity, you can use the low-level datastore API:
from google.appengine.api import datastore
sKD = datastore.Entity(kind='SomeKindaData', name='1')
sKD['SomeProperty'] = 'SomeValue'
datastore.Put(sKD)