I do understand, that a sqlalchemy.orm.scoping.scoped_session uses a session_factory to create a session and also possesses a registry to return an already present session through the __call__() call.
But one can also directly call the .query method upon scoped_session and that completely confuses me, since scoped_session:
1. does not have this method
2. is not a dynamic wrapper of a sqlalchemy.orm.session.Session and
3. is not a subclass of sqlalchemy.orm.session.Session.
How is scoped_session able to dispatch a query? I just don't see any indirection or abstraction that would allow for this.. yet it works.
from sqlalchemy.orm import sessionmaker,scoped_session
from sqlalchemy import create_engine
user, password, server, dbname = "123","123","123", "123"
s = 'oracle://%s:%s#%s/%s' % (user, password, server, dbname)
some_engine = create_engine(s)
_sessionmaker = sessionmaker(bind=some_engine)
sc_sess = scoped_session(_sessionmaker) # here sc_sess is an isntance of "sqlalchemy.orm.scoping.scoped_session"
sc_sess.query(...) # works! but why?
# the following is what i expect to work and to be normal workflow
session = sc_sess() # returns an instance of sqlalchemy.orm.session.Session
session.query(...)
This behaviour is described in the SqlAlchemy Documentation:
Implicit Method Access
The job of the scoped_session is simple; hold onto a Session for all who ask for it. As a means of producing more transparent access to this Session, the scoped_session also includes proxy behavior, meaning that the registry itself can be treated just like a Session directly; when methods are called on this object, they are proxied to the underlying Session being maintained by the registry:
Session = scoped_session(some_factory)
# equivalent to:
#
# session = Session()
# print(session.query(MyClass).all())
#
print(Session.query(MyClass).all())
The above code accomplishes the same task as that of acquiring the current Session by calling upon the registry, then using that Session.
So this behaviour is normal, but how is it implemented? (not proxy in general, but precisely in this example)
Thanks.
You've obviously had a good look at the sqlalchemy.orm.scoping.scoped_session class, and if you look just a little further down in the same module, you'll find the following snippet (link):
def instrument(name):
def do(self, *args, **kwargs):
return getattr(self.registry(), name)(*args, **kwargs)
return do
for meth in Session.public_methods:
setattr(scoped_session, meth, instrument(meth))
If we dissect that from the bottom up, we've first got the for meth in Session.public_methods: loop, where Session.public_methods is simply a tuple of the names of methods that a Session exposes, and the string "query" is one of those:
class Session(_SessionClassMethods):
...
public_methods = (
...,
"query",
...,
)
Each of those names (meth) in Session.public_methods is passed to the setattr call inside the loop:
setattr(scoped_session, meth, instrument(meth))
The value that is assigned to the name of the method on the scoped_session is the return value of the call to instrument(meth), which is a closure called, do(). That function calls the scoped_session.registry to get the registered Session object, gets the named method (meth), and calls it with the *args & **kwargs that were passed to do().
As the for meth in Session.public_methods: loop is defined in the global namespace of the module, it is executed at compile time, before anything else has a chance to use the scoped_session. So by the time your code can get a hold of a scoped_session instance, those methods have already been monkey patched on there.
Related
I have the following problem. I have multiple unittests, each one having a context manager to open a browser and do some selenium testing.
I want to ensure that I can run the tests in parallel, and that the browser window is closed in case of error, so I use a context manager:
def test_xxx():
with webapp() as p:
file_loader = FileLoader(p, id="loader").upload(path)
As you can see, I use a class FileLoader, which takes the context manager web application (basically a wrapper of the selenium driver) and uses it to encapsulate the rigmarole required to upload the file.
My objective would be not to have to specify the p parameter to FileLoader(), so that I could write
def test_xxx():
with webapp():
file_loader = FileLoader(id="loader").upload(path)
I could use a global that is assigned when the context manager is opened, but this would prevent any isolation when tests run in parallel. Suppose that one test connects to site A, and another test connects to site B. I need two drivers, each connected to a different site.
In other words, how can I design FileLoader to be aware of its enclosing context manager without passing the context variable?
By using the inspect module, a code can read the local variables of its caller. It is a rather unusual if not dangerous use, because it actually boils down to a non standard and non conventional way of passing a parameter to a function. But if you really want to go that way, this other SO question gives a possible way.
Demo:
class ContextAware:
""" Class that will copy the x local variable of its caller if any"""
def __init__(self):
# uncomment next line for debugging
# print("ContextAware", inspect.currentframe().f_back.f_locals)
self.x = inspect.currentframe().f_back.f_locals.get('x')
def foo(x):
# the value of x is not explicitely passed, yet it will be used...
return ContextAware()
The object created in foo is aware of the x variable of its caller:
>>> a = foo(4)
>>> a.x
4
>>> a = foo(6)
>>> a.x
6
That means you you could write something close to:
def test_xxx():
with webapp() as ctx_p:
file_loader = FileLoader(id="loader").upload(path)
provided the __init__ method of FileLoader spies on the ctx_p local variable of its caller.
I'm trying to introduce type hints into an existing codebase, but I'm running into an issue when I attempt to type my query.
from sqlalchemy.orm.query import Query
class DbContext:
def __init__(self, db_host, db_port, db_name, db_user, db_password):
engine = create_engine(...)
session = sessionmaker(bind=engine)
self.Session: Session = session(bind=engine)
...
def fetch(context: DbContext, filters: ...):
sub_query: Query = context.Session.query(...)
Before I added type hints, filtering dynamically was simply a matter of:
if filters.name is not None:
sub_query = sub_query.filter(
Person.name.ilike(f"%{filters.name}%"))
However, now with hinting I'm getting this error:
Expression of type "None" cannot be assigned to declared type "Query"
Sure enough, filter appears to return None:
(method) filter: (*criterion: Unknown) -> None
I navigated to the source and it appears the method does indeed not return anything.
def filter(self, *criterion):
for criterion in list(criterion):
criterion = expression._expression_literal_as_text(criterion)
criterion = self._adapt_clause(criterion, True, True)
if self._criterion is not None:
self._criterion = self._criterion & criterion
else:
self._criterion = criterion
There's obviously a disconnect somewhere, as assigning None to sub_query should result in an error which the hint is warning against, but I need to perform the assignment for the filtering to actually work:
# Does NOT work, filtering is not applied
if filters.name is not None:
sub_query.filter(
Person.name.ilike(f"%{filters.name}%"))
# Works but Pylance complains
if filters.name is not None:
sub_query = sub_query.filter(
Person.name.ilike(f"%{filters.name}%"))
This is my first foray into Python, would love some guidance as to what is going on here!
You are missing two things:
You need to install typing stubs for SQLAlchemy.
The Query.filter() method has a decorator that defines what is returned.
Typing stubs for SQLAlchemy
You want to install the sqlalchemy-stubs project, it provides stubs for the SQLAlchemy API.
Note that even with this stub installed you still will see issues with Pyright (the checking tool underpinning the Pylance extension), because static stubs cannot fully represent the dynamic nature of some parts of the SQLAlchemy API, such as model column definitions (e.g. if your Person model has a column called name, defined with name = Column(String), then the stubs can't tell Pyright that name will be a string). The sqlalchemy-stubs project includes a plugin for the mypy type checker to handle the dynamic parts better, but such plugins can't be used with other type checkers.
With the stubs installed, Pylance can tell you about filter:
Query.filter() decorator details
The Query.filter() method implementation is not actually operating on the original instance object; it has been annotated with a decorator:
#_generative(_no_statement_condition, _no_limit_offset)
def filter(self, *criterion):
...
The #_generative(...) part is significant here; the definition of the decorator factory shows that the filter() method is essentially replaced by this wrapper method:
def generate(fn, *args, **kw):
self = args[0]._clone()
for assertion in assertions:
assertion(self, fn.__name__)
fn(self, *args[1:], **kw)
return self
Here, fn is the original filter() method definition, and args[0] is the reference to self, the initial Query instance. So self is replaced by calling self._clone() (basically, a new instance is created and the attributes copied over), it runs the declared assertions (here, _no_statement_condition and _no_limit_offset are such assertions), before running the original function on the clone.
So, what the filter() function does, is alter the cloned instance in place, and so doesn't have to return anything; that's taken care of by the generate() wrapper. It is this trick of swapping out methods with utility wrappers that confused Pyright into thinking None is returned, but with stubs installed it knows that another Query instance is returned instead.
I'm aware that methods are just objects that can be accessed via getattr(obj, 'method_name'). Is the method does not exist, this will trigger obj.__getattr__(method_name). However, is it possible in the __getattr__ implementation to distinct whether the attribute is called directly by the user or not? It seems to me that descriptors might allow this, but I'm not entirely sure.
My motivation is a proxy class that forwards both attribute access and method calls to a wrapped object to which communication is slow. For attribute access, we necessarily have to block and wait for the result. But for method access, I'd like to inject a _blocking parameter that allows to receive a non-blocking promise object:
proxy = Proxy(Inner())
proxy.value # Block and wait for inner.value
promise = proxy.method(arg1, args2, _blocking=False) # Non-blocking
promise() # Wait for the return value of inner.method(arg1, arg2)
I have a variable that points to a specific class instance's method.
Lets say there is a class called Client that implements a method get.
A single instance called client is created for Client, and a variable get_func is assigned with client's get.
For example, lets assume I have the following simplified code:
class Client:
def get(self):
print("This is the get function!")
client = Client()
get_func = client.get
I have little actual control over get_func and how it's used, I cannot change that.
I would now want to make sure get_func has the Client.get.
The trivial test get_func == Client.get does not work, as get_func is a bound method of a specific Client instance.
I also cannot get the client instance directly (but a way to get the self of a bound method is a valid option, if only I knew how to do that)
Client.get is either a function object (Python 3), or an unbound method (Python 2). Client().get on the other hand, is a bound method. Methods are wrappers around a function object, recording the instance they are bound to (if there is an instance) to pass in as the self argument. See the Python descriptor How-to as to how Python produces methods from functions.
You can unwrap both a bound method and an unbound method to get the underlying function object that they wrap, and test that, with the __func__ attribute:
get_func.__func__ is Client.get # Python 3
get_func.__func__ is Client.get.__func__ # Python 2, unwrap the unbound method
If you need to make your code compatible with both Python 2 and 3, you could create a simple helper function:
def unwrap_method(f):
return getattr(f, '__func__', f)
and use that:
unwrap_method(get_func) is unwrap_method(Client.get)
I am using Flask-RESTful and trying to have my REST endpoints using the technique showed here
The main code is
def authenticate(func):
#wraps(func)
def wrapper(*args, **kwargs):
if not getattr(func, 'authenticated', True):
return func(*args, **kwargs)
acct = basic_authentication() # custom account lookup function
if acct:
return func(*args, **kwargs)
restful.abort(401)
return wrapper
class Resource(restful.Resource):
method_decorators = [authenticate] # applies to all inherited resources
I do the same way and it seems to work, but I am not sure what happens with #wraps?
It seems magic to me at the moment and I did not understand the following
a.) It seems function which is wrapped with #wraps is passed to the wrapper, then what is the wrapper returning?
Possible answer: Everything that was passed to the function initially?
If yes, how can I pass more information like the acct object with everything so that my function receives the account object and I don't have to do a database fetch for it?
UPDATE
Based on the example, my rest endpoint looks like
class UserResource(RestResource):
def get(self, uuid):
return {'method': 'get user -> ' + uuid}
and I call it like
curl -X GET http://127.0.0.1:5000/users/validUUID
Now when my every request is authenticated, I see if a valid acct object exists and if it exists, I delegate the control to endpoint
Question:
Since I am actually making one database call to find out acct object, is it possible to pass in that to the endpoint when a valid acct is located?
This way two things happen
a.) I know the call is authenticated
b.) I reuse the acct object which I can use for my further work, rather than making the DB call again and get the acct object from validUUID again
How can I achieve this ?
authenticate is a decorator -- it takes a function and returns a modified version of that function (which is usually implemented by wrapping the function and wrapping it).
Now, the problem with wrappers is that they often don't act exactly like the original function in some ways -- they may be missing docstrings, have the wrong __name__ (wrapper instead of what it should be called), and other blemishes. This might be important if some other code is using that extra information. functools.wraps is a simple function that adds this information from the original function (here, func) to the wrapper function, so it behaves more like the original function. (Technically, it is itself a decorator, which is the confusing part, but you don't have to worry about that detail. Just know that it is a nice tool that copies attributes from a wrapped function to a wrapper function).
Thus, when you write
new_function = authenticate(old_function)
or more commonly
#authenticate
def function(...)
new_function will look more like old_function.