code :
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
DBSession = sessionmaker()
DBSession.bind = engine
session = DBSession()
engine = create_engine('mysql://root:password#localhost/db', echo=True)
Base = declarative_base(engine)
class Accounts(Base):
__tablename__ = 'accounts'
__table_args__ = {'autoload': True}
I am trying to store sqlalchemy record object into memcache
from pymemcache.client.base import Client
client = Client(('localhost', 11211))
client.set('testkey', session.query(Users).get(1))
It is storing string object instead of User object
output : '<__main__.Users object at 0x105b69b10>'
Any help ?
Thanks advance
The issue isn't so much about how to store a SQLAlchemy object but rather how to store any Object instance.
This is from docstring of the pymemcache Client class that you've imported:
Values must have a str() method to convert themselves to a byte
string.
You haven't included a definition of the Users class that you are querying your database with so I can only assume you haven't overridden __str__. Therefore, when pymemcache tries to convert your object into a byte string, it is calling python's default __str__ implementation and storing that.
As #SuperShoot mentions in their answer, pymemcache expects to store a str representation. Since your SQLAlchemy model instance (record) is not natively a str representation, pymemcache tries to call its default __str__() method, which produces an undesired result.
What you need is an interim step that will serialize your SQLAlchemy model instance to a str of one structure or another. This makes your logical round-trippable flow:
Serialize your model instance to a str of one structure or another.
Store the serialized str using pymemcache.
When retrieving the str from pymemcache, de-serialize it into a SQLAlchemy model instance to continue working easily with it in your Python code.
This can be a bit complicated with SQLAlchemy models, which is why I recommend using the SQLAthanor library (full disclosure: I'm the author). It lets you serialize your SQLAlchemy model to CSV, JSON, or YAML - which you can then persist to memcache. And it also lets you easily de-serialize a CSV, JSON, or YAML string into a SQLAlchemy model instance as well, letting you easily maintain the whole flow described above.
There's a lot more functionality to the library which you can read about on the docs page: https://sqlathanor.readthedocs.io/en/latest/
The important bit to remember is that when using SQLAthanor, you'll need to decide what format you want to store your data in (I recommend JSON or YAML), and then explicitly configure the columns/attributes you want to have serialized to that format (this is a security feature in the library). Because your code snippet shows that you're using Declarative Reflection, you'll probably want to look at the following sections of the documentation for how to configure SQLAthanor:
https://sqlathanor.readthedocs.io/en/latest/using.html#using-declarative-reflection-with-sqlathanor
https://sqlathanor.readthedocs.io/en/latest/quickstart.html#using-sqlathanor-with-sqlalchemy-reflection
Hope this helps!
Related
New to sqlalchemy and somewhat novice with programing and python. I had wanted to query a table. It seems I can use the all() function when querying but cannot filter without creating a class.
1.) Can I filter without creating a class and using the declarative api? Is the filtering example stated below incorrect?
2.) When would it be appropriate to use declarative api in sqlalchemy and when would it not be appropriate?
import sqlalchemy as sql
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import sessionmaker
db = sql.create_engine('postgresql://postgres:password#localhost:5432/postgres')
engine = db.connect()
meta = MetaData(engine)
session = sessionmaker(bind=engine)
session = session()
files = Table('files',meta,
Column('file_id',Integer,primary_key=True),
Column('file_name',String(256)),
Column('query',String(256)),
Column('results',Integer),
Column('totalresults',Integer),
schema='indeed')
session.query(files).all() #ok
session.query(files).filter(files.file_name = 'test.json') #not ok
If you want to filter by a Table construct, it should be:
session.query(files).filter(files.c.file_name == 'test.json')
You need to create mapped classes if you want to use the ORM features of SQLAlchemy. For example, with the code you currently have, in order to do an update you have to do
session.execute(files.update().values(...))
As opposed to:
file = session.query(File).first()
file.file_name = "new file name"
session.commit()
The declarative API happens to be the easiest way of constructing mapped classes, so use it if you want to use the ORM.
Filter using declarative api this way:
session.query(files).filter(files.file_name == 'test.json').all()
You can also use raw sql queries (docs).
Whether using declarative api or not may depend on your queries complexity, because sometimes sqlalchemy doesn't optimize them right way.
Why I always need to do that in 2 steps in SqlAlchemy?
import sqlalchemy as sa
import sqlalchemy.orm as orm
engine = sa.create_engine(<dbPath>, echo=True)
Session = orm.sessionmaker(bind=engine)
my_session = Session()
Why I cannot do it in one shot like (it's could be more simple, no?) :
import sqlalchemy as sa
import sqlalchemy.orm as orm
engine = sa.create_engine(<dbPath>, echo=True)
Session = orm.Session(bind=engine)
The reason sessionmaker() exists is so that the various "configurational" arguments it requires only need to be set up in one place, instead of repeating "bind=engine, autoflush=False, expire_on_commit=False", etc. over and over again. Additionally, sessionmaker() provides an "updateable" interface, such that you can set it up somewhere in your application:
session = sessionmaker(expire_on_commit=False)
but then later, when you know what database you're talking to, you can add configuration to it:
session.configure(bind=create_engine("some engine"))
It also serves as a "callable" to pass to the very common scoped_session() construct:
session = scoped_session(sessionmaker(bind=engine))
With all of that said, these are just conventions that the documentation refers to so that a consistent "how to use" story is presented. There's no reason you can't use the constructor directly if that is more convenient, and I use the Session() constructor all the time. It's just that in a non-trivial application, you will probably end up sticking that constructor call to Session() inside some kind of callable function anyway, sessionmaker() serves as a default for that callable.
In the most general sense, the Session establishes all conversations with the database and represents a “holding zone” for all the objects which you’ve loaded or associated with it during its lifespan. It provides the entrypoint to acquire a Query object, which sends queries to the database using the Session object’s current database connection, populating result rows into objects that are then stored in the Session, inside a structure called the Identity Map - a data structure that maintains unique copies of each object, where “unique” means “only one object with a particular primary key”.
Try to pprint and see whats inside;
import pprint
pprint.pprint(my_session)
Here's the rest of the story: http://docs.sqlalchemy.org/ru/latest/orm/session.html
I'm a beginner in SQLAlchemy and found query can be done in 2 method:
Approach 1:
DBSession = scoped_session(sessionmaker())
class _Base(object):
query = DBSession.query_property()
Base = declarative_base(cls=_Base)
class SomeModel(Base):
key = Column(Unicode, primary_key=True)
value = Column(Unicode)
# When querying
result = SomeModel.query.filter(...)
Approach 2
DBSession = scoped_session(sessionmaker())
Base = declarative_base()
class SomeModel(Base):
key = Column(Unicode, primary_key=True)
value = Column(Unicode)
# When querying
session = DBSession()
result = session.query(SomeModel).filter(...)
Is there any difference between them?
In the code above, there is no difference. This is because, in line 3 of the first example:
the query property is explicitly bound to DBSession
there is no custom Query object passed to query_property
As #petr-viktorin points out in the answer here, there must be a session available before you define your model in the first example, which might be problematic depending on the structure of your application.
If, however, you need a custom query that adds additional query parameters automatically to all queries, then only the first example will allow that. A custom query class that inherits from sqlalchemy.orm.query.Query can be passed as an argument to query_property. This question shows an example of that pattern.
Even if a model object has a custom query property defined on it, that property is not used when querying with session.query, as in the last line in the second example. This means something like the first example the only option if you need a custom query class.
I see these downsides to query_property:
You cannot use it on a different Session than the one you've configured (though you could always use session.query then).
You need a session object available before you define your schema.
These could bite you when you want to write tests, for example.
Also, session.query fits better with how SQLAlchemy works; query_property looks like it's just added on top for convenience (or similarity with other systems?).
I'd recommend you stick to session.query.
An answer (here) to a different SQLAlchemy question might help. That answer starts with:
You can use Model.query, because the Model (or usually its base class, especially in cases where declarative extension is used) is assigned Session.query_property. In this case the Model.query is equivalent to Session.query(Model).
I store the MySQL Compress function to insert compressed blob data to the database.
In a previous question I was instructed to to use
func.compress
( mysql Compress() with sqlalchemy )
The problem now is that I want to read also the data from the database.
In mysql I would have done
SELECT UNCOMPRESS(text) FROM ...
probably I should use a getter in the class.
I tried to do somethin like:
get_html(self):
return func.uncompress(self.text)
but this does not work. It returns an sqlalchemy.sql.expression.Function and not the string.
Moreover I could not find which functions contains sqlalchemy's func.
Any ideas on how i could write a getter in the object so I get back the uncompressed data.
func is actually a really fancy factory object for special function objects which are rendered to SQL at query time - you cannot evaluate them in Python since Python would not know how your database implements compress(). That's why it doesn't work.
SQLAlchemy lets you map SQL expressions to mapped class attributes. If you're using the declarative syntax, extend your class like so (it's untested, but I'm confident this is the way to go):
from sqlalchemy.orm import column_property
class Demo(...):
data_uncompressed = column_property(func.uncompress(data))
Now whenever SQLAlchemy loads an instance from the database, the SELECT query will contain SELECT ..., UNCOMPRESS(demotable.data), ... FROM demotable.
Edit by Giorgos Komninos:
I used the
http://docs.sqlalchemy.org/en/rel_0_7/orm/mapper_config.html#using-a-plain-descriptor
and it worked.
Because of legacy data which is not available in the database but some external files, I want to create a SQLAlchemy object which contains data read from the external files, but isn't written to the database if I execute session.flush()
My code looks like this:
try:
return session.query(Phone).populate_existing().filter(Phone.mac == ident).one()
except:
return self.createMockPhoneFromLicenseFile(ident)
def createMockPhoneFromLicenseFile(self, ident):
# Some code to read necessary data from file deleted....
phone = Phone()
phone.mac = foo
phone.data = bar
phone.state = "Read from legacy file"
phone.purchaseOrderPosition = self.getLegacyOrder(ident)
# SQLAlchemy magic doesn't seem to work here, probably because we don't insert the created
# phone object into the database. So we set the id fields manually.
phone.order_id = phone.purchaseOrderPosition.order_id
phone.order_position_id = phone.purchaseOrderPosition.order_position_id
return phone
Everything works fine except that on a session.flush() executed later in the application SQLAlchemy tries to write the created Phone object to the database (which fortunately doesn't succeed, because phone.state is longer than the data type allows), which breaks the function which issues the flush.
Is there any way to prevent SQLAlchemy from trying to write such an object?
Update
While I didn't find anything on
using_mapper_options(save_on_init=False)
in the Elixir documentation (maybe you can provide a link?), it seemed to me worth a try (I would have preferred a way to prevent a single instance from being written instead of the whole entity).
At first it seemed that the statement has no effect and I suspected my SQLAlchemy/Elixir versions to be too old, but then I found out that the connection to the PurchaseOrderPosition entity (which I didn't modify) made with
phone.purchaseOrderPosition = self.getLegacyOrder(ident)
causes the phone object to be written again. If I remove the statement, everything seems to be fine.
You need to do
import elixir
elixir.options_defaults['mapper_options'] = { 'save_on_init': False }
to prevent Entity instances which you instantiate being auto-added to the session. Ideally, this should be done as early as possible in your code. You can also do this on a per-entity basis, via using_mapper_options(save_on_init=False) - see the Elixir documentation for more details.
Update:
See this post on the Elixir mailing list indicating that this is the solution.
Also, as Ants Aasma points out, you can use cascade options on the Elixir relationship to set up cascade options in SQLAlchemy. See this page for more details.
Well, sqlalchemy doesn't, by default.
Consider the following self-contained example code.
from sqlalchemy import Column, Integer, Unicode, create_engine
from sqlalchemy.orm import create_session
from sqlalchemy.ext.declarative import declarative_base
e = create_engine('sqlite://')
Base = declarative_base(bind=e)
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(Unicode(50))
# create the empty table and a session
Base.metadata.create_all()
s = create_session(bind=e, autoflush=False, autocommit=False)
# assert the table is empty
assert s.query(User).all() == []
# create a new User instance but don't save it to database:
u = User()
u.name = 'siebert'
# I could run s.add(u) here but I won't
s.flush()
s.commit()
# assert the table is still empty
assert s.query(User).all() == []
So I'm not sure what's implicity adding your instances to the session. Normally you have to manually call s.add(u) to make it go to the session. I'm not familiar with elixir so perhaps this is some elixir trickery... Maybe you could remove it from the session, by using session.expunge().
Old post but I came across a similar issue, in my case in sqlalchemy it was caused by cascading on backrefs:
http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html#backref-cascade
Turn it off on your backrefs so that you have to explicitly add things to the session