I'm trying to introduce type hints into an existing codebase, but I'm running into an issue when I attempt to type my query.
from sqlalchemy.orm.query import Query
class DbContext:
def __init__(self, db_host, db_port, db_name, db_user, db_password):
engine = create_engine(...)
session = sessionmaker(bind=engine)
self.Session: Session = session(bind=engine)
...
def fetch(context: DbContext, filters: ...):
sub_query: Query = context.Session.query(...)
Before I added type hints, filtering dynamically was simply a matter of:
if filters.name is not None:
sub_query = sub_query.filter(
Person.name.ilike(f"%{filters.name}%"))
However, now with hinting I'm getting this error:
Expression of type "None" cannot be assigned to declared type "Query"
Sure enough, filter appears to return None:
(method) filter: (*criterion: Unknown) -> None
I navigated to the source and it appears the method does indeed not return anything.
def filter(self, *criterion):
for criterion in list(criterion):
criterion = expression._expression_literal_as_text(criterion)
criterion = self._adapt_clause(criterion, True, True)
if self._criterion is not None:
self._criterion = self._criterion & criterion
else:
self._criterion = criterion
There's obviously a disconnect somewhere, as assigning None to sub_query should result in an error which the hint is warning against, but I need to perform the assignment for the filtering to actually work:
# Does NOT work, filtering is not applied
if filters.name is not None:
sub_query.filter(
Person.name.ilike(f"%{filters.name}%"))
# Works but Pylance complains
if filters.name is not None:
sub_query = sub_query.filter(
Person.name.ilike(f"%{filters.name}%"))
This is my first foray into Python, would love some guidance as to what is going on here!
You are missing two things:
You need to install typing stubs for SQLAlchemy.
The Query.filter() method has a decorator that defines what is returned.
Typing stubs for SQLAlchemy
You want to install the sqlalchemy-stubs project, it provides stubs for the SQLAlchemy API.
Note that even with this stub installed you still will see issues with Pyright (the checking tool underpinning the Pylance extension), because static stubs cannot fully represent the dynamic nature of some parts of the SQLAlchemy API, such as model column definitions (e.g. if your Person model has a column called name, defined with name = Column(String), then the stubs can't tell Pyright that name will be a string). The sqlalchemy-stubs project includes a plugin for the mypy type checker to handle the dynamic parts better, but such plugins can't be used with other type checkers.
With the stubs installed, Pylance can tell you about filter:
Query.filter() decorator details
The Query.filter() method implementation is not actually operating on the original instance object; it has been annotated with a decorator:
#_generative(_no_statement_condition, _no_limit_offset)
def filter(self, *criterion):
...
The #_generative(...) part is significant here; the definition of the decorator factory shows that the filter() method is essentially replaced by this wrapper method:
def generate(fn, *args, **kw):
self = args[0]._clone()
for assertion in assertions:
assertion(self, fn.__name__)
fn(self, *args[1:], **kw)
return self
Here, fn is the original filter() method definition, and args[0] is the reference to self, the initial Query instance. So self is replaced by calling self._clone() (basically, a new instance is created and the attributes copied over), it runs the declared assertions (here, _no_statement_condition and _no_limit_offset are such assertions), before running the original function on the clone.
So, what the filter() function does, is alter the cloned instance in place, and so doesn't have to return anything; that's taken care of by the generate() wrapper. It is this trick of swapping out methods with utility wrappers that confused Pyright into thinking None is returned, but with stubs installed it knows that another Query instance is returned instead.
Related
I am playing around with python metaclasses, and trying to write some sort of metaclasses that changes or adds methods dynamically for its subclasses.
For example, here is a metaclass that its purpose is to find async methods in the subclass (that their name also ends with the string "_async") and add an additional "synchronized" version of this method:
class AsyncClientMetaclass(type):
#staticmethod
def async_func_to_sync(func):
return lambda *_args, **_kwargs: run_synchronized(func(*_args, **_kwargs))
def __new__(mcs, *args, **kwargs):
cls = super().__new__(mcs, *args, **kwargs)
_, __, d = args
for key, value in d.items():
if asyncio.iscoroutinefunction(value) and key.endswith('_async'):
sync_func_name = key[:-len('_async')]
if sync_func_name in d:
continue
if isinstance(value, staticmethod):
value = value.__func__
setattr(cls, sync_func_name, mcs.async_func_to_sync(value))
return cls
# usage
class SleepClient(metaclass=AsyncClientMetaclass):
async def sleep_async(self, seconds):
await asyncio.sleep(seconds)
return f'slept for {seconds} seconds'
c = SleepClient()
res = c.sleep(2)
print(res) # prints "slept for 2 seconds"
This example works great, the only problem is that the python linter warns about using the non async method that the metaclass has created (for the example above, the warning is Unresolved attribute reference 'sleep' for class 'SleepClient')
For now, I am adding pylint: disable whenever I am using a sync method created by the metaclass, but I am wondering if is there any way to add a custom linter rule with the metaclass, so the linter will know those methods will be created dynamically.
And are you think there is a better way to achieve this purpose rather than using metaclass?
Thanks!
As put by Chepner: no static code analyser can know about these methods - not linters nor type-annotation checking tools like MyPy, unless you give then a hint.
Maybe there is one way out: static type annotators will consume a parallel ".pyi" stub file, put side by side to the correspondent ".py" file that can list class interfaces, and, I may be wrong, but whatever it findes there will supersede what the toll "sees" on the actual Py file.
So, you could instrument your metaclass to, aside from generating the actual methods, render their signature and the signature for the "real" methods and attributes of the class as source code, and record those as the proper "pyi" file. You will have to run this code once, before the linter can find its way - but it is the only workaround I can think of.
In other words, to be clear:
make a mechanism called by the metaclass that will check for the existence and time-stamp of the appropriate ".pyi" file for the classes it is modifying, and generate them. By checking the timestamp, or generating this file only when some "--build" variable is active, there should be no runtime penalties, and static-type checkers (and possibly some linters), should be pleased.
I have a wrapper class for a database cursor like this:
namedtuple_def = TypeVar("namedtuple_def")
class Wrapper:
def fetchall(defn: Optional[namedtuple_def] = None) -> List[namedtuple_def]:
return self.cursor.fetchall()
The underlying cursor always returns namedtuples, but the nametuple returned depends on the query run. So I'm trying to get it to take a nametuple type as an unused parameter to hint exactly what types are getting returned. If you call it without the defn parameter the return type is partially erased, but if you pass the equivalent namedtuple class your query results have fully specified types.
My problem with the code above is that the typechecker thinks I'm returning the type variable/class for the namedtuple argument, not the instantiated namedtuples - or in other words, List[Type[my_namedtuple]] instead of List[my_namedtuple]. How do I fix the type signature?
n.b. I am using the PyCharm typechecker, not MyPy, but I don't expect them to vary here.
SQLAlchemy backrefs tend to cause circular imports as they get complex, so I came up with a way to "re-open" a Python class like in Ruby:
def reopen(cls):
"""
Moves the contents of the decorated class into an existing class and returns that.
Usage::
from .other_module import ExistingClass
#reopen(ExistingClass)
class ExistingClass:
#property
def new_property(self):
pass
This is equivalent to::
def new_property(self):
pass
ExistingClass.new_property = property(new_property)
"""
def decorator(temp_cls):
for attr, value in temp_cls.__dict__.items():
# Skip the standard Python attributes, process the rest
if attr not in ('__dict__', '__doc__', '__module__', '__weakref__'):
setattr(cls, attr, value)
return cls
return decorator
This is a simplified version; the full implementation has more safety checks and tests.
This code works fairly well for inserting bi-directional SQLAlchemy relationships and helper methods wherever SQLAlchemy's existing relationship+backref mechanism doesn't suffice.
However:
mypy raises error "name 'ExistingClass' already defined (possibly by an import)"
pylint raises E0102 "class already defined"
I could ignore both errors by having an annotation on the line (# type: ignore and # skipcq), but overuse of those annotations can let bugs slip through. It'll be nice to tell mypy and pylint that this use is okay.
How can I do that?
As a workaround, I've switched to this pattern:
#reopen(ExistingClass)
class __ExistingClass:
#property
def new_property(self):
pass
>>> assert __ExistingClass is ExistingClass
True
Pylint and Mypy no longer complain about redefinition. However, Mypy will not recognise ExistingClass.new_property as an addition to the class, so it kicks the problem downstream to code that uses it.
My impression is that such class extension hacks, while discussed multiple times, have not gained enough traction as a coding pattern, and in Mypy's case this will need a custom plugin.
I do understand, that a sqlalchemy.orm.scoping.scoped_session uses a session_factory to create a session and also possesses a registry to return an already present session through the __call__() call.
But one can also directly call the .query method upon scoped_session and that completely confuses me, since scoped_session:
1. does not have this method
2. is not a dynamic wrapper of a sqlalchemy.orm.session.Session and
3. is not a subclass of sqlalchemy.orm.session.Session.
How is scoped_session able to dispatch a query? I just don't see any indirection or abstraction that would allow for this.. yet it works.
from sqlalchemy.orm import sessionmaker,scoped_session
from sqlalchemy import create_engine
user, password, server, dbname = "123","123","123", "123"
s = 'oracle://%s:%s#%s/%s' % (user, password, server, dbname)
some_engine = create_engine(s)
_sessionmaker = sessionmaker(bind=some_engine)
sc_sess = scoped_session(_sessionmaker) # here sc_sess is an isntance of "sqlalchemy.orm.scoping.scoped_session"
sc_sess.query(...) # works! but why?
# the following is what i expect to work and to be normal workflow
session = sc_sess() # returns an instance of sqlalchemy.orm.session.Session
session.query(...)
This behaviour is described in the SqlAlchemy Documentation:
Implicit Method Access
The job of the scoped_session is simple; hold onto a Session for all who ask for it. As a means of producing more transparent access to this Session, the scoped_session also includes proxy behavior, meaning that the registry itself can be treated just like a Session directly; when methods are called on this object, they are proxied to the underlying Session being maintained by the registry:
Session = scoped_session(some_factory)
# equivalent to:
#
# session = Session()
# print(session.query(MyClass).all())
#
print(Session.query(MyClass).all())
The above code accomplishes the same task as that of acquiring the current Session by calling upon the registry, then using that Session.
So this behaviour is normal, but how is it implemented? (not proxy in general, but precisely in this example)
Thanks.
You've obviously had a good look at the sqlalchemy.orm.scoping.scoped_session class, and if you look just a little further down in the same module, you'll find the following snippet (link):
def instrument(name):
def do(self, *args, **kwargs):
return getattr(self.registry(), name)(*args, **kwargs)
return do
for meth in Session.public_methods:
setattr(scoped_session, meth, instrument(meth))
If we dissect that from the bottom up, we've first got the for meth in Session.public_methods: loop, where Session.public_methods is simply a tuple of the names of methods that a Session exposes, and the string "query" is one of those:
class Session(_SessionClassMethods):
...
public_methods = (
...,
"query",
...,
)
Each of those names (meth) in Session.public_methods is passed to the setattr call inside the loop:
setattr(scoped_session, meth, instrument(meth))
The value that is assigned to the name of the method on the scoped_session is the return value of the call to instrument(meth), which is a closure called, do(). That function calls the scoped_session.registry to get the registered Session object, gets the named method (meth), and calls it with the *args & **kwargs that were passed to do().
As the for meth in Session.public_methods: loop is defined in the global namespace of the module, it is executed at compile time, before anything else has a chance to use the scoped_session. So by the time your code can get a hold of a scoped_session instance, those methods have already been monkey patched on there.
I am working within a Python web framework that uses Python 3 type annotations for validation and dependency injection.
So I am looking for a way to generate functions with type annotations from a parameters given to the generating function:
def gen_fn(args: Dict[str, Any]) -> Callable:
def new_fn(???):
pass
return new_fn
so that
inspect.signature(gen_fn({'a': int}))
will return
<Signature (a:int)>
Is there something I cam put instead of the ??? that will do the thing I need.
I also looked at Signature.replace() in the inspect module, but did not find a way to attach the new signature to a new or existing function.
I am hesitant to use ast because:
The abstract syntax itself might change with each Python release
So my question is: What (if any) is a reasonable way to generate a function with Python 3 type annotation based on a dict passed to the generating function?
Edit: while #Aran-Fey's solution answer my question correctly, it appears that my assumption was wrong. Changing the signature doesn't allow calling the new_fn using the new signature. That is gen_fn({'a': int})(a=42) raises a TypeError: ... `got an unexpected keyword argument 'a'.
Instead of creating a function with annotations, it's easier to create a function and then set the annotations manually.
inspect.signature looks for the existence of a __signature__ attribute before it looks at the function's actual signature, so we can craft an appropriate inspect.Signature object and assign it there:
params = [inspect.Parameter(param,
inspect.Parameter.POSITIONAL_OR_KEYWORD,
annotation=type_)
for param, type_ in args.items()]
new_fn.__signature__ = inspect.Signature(params)
typing.get_type_hints does not respect __signature__, so we should update the __annotations__ attribute as well:
new_fn.__annotations__ = args
Putting them both together:
def gen_fn(args: Dict[str, Any]) -> Callable:
def new_fn():
pass
params = [inspect.Parameter(param,
inspect.Parameter.POSITIONAL_OR_KEYWORD,
annotation=type_)
for param, type_ in args.items()]
new_fn.__signature__ = inspect.Signature(params)
new_fn.__annotations__ = args
return new_fn
print(inspect.signature(gen_fn({'a': int}))) # (a:int)
print(get_type_hints(gen_fn({'a': int}))) # {'a': <class 'int'>}
Note that this doesn't make your function callable with these arguments; all of this is just smoke and mirrors that makes the function look like it has those parameters and annotations. Implementing the function is a separate issue.
You can define the function with varargs to aggregate all the arguments into a tuple and a dict:
def new_fn(*args, **kwargs):
...
But that still leaves you with the problem of implementing the function body. You haven't said what the function should do when it's called, so I can't help you with that. You can look at this question for some pointers.