How can I override a mongoengine's queryset method?
Specifically, I want to override .order_by(), but the closest I can get is to add another method ordered that would conditionally call .order_by():
class TransactionQuerySet(QuerySet):
def ordered(self, *args, target_data=None, **kwargs):
if target_data is None:
result = self.order_by(*args, **kwargs)
return result
else:
return 'transactions sorted by target data'
Ideally I would like this new method to be named the same as mongoengine's method - order_by - but if I do this I will exceed recursion depth when the queryset manager is called like Transaction.objects.order_by.
How can I do this?
To avoid recursion, you should call order_by method of the parent class. TransactionQuerySet should look like the following.
class TransactionQuerySet(QuerySet):
def order_by(self, *args, target_data=None):
if target_data is None:
return super().order_by(*args)
return "transactions sorted by target data"
Now if you call order_by on TransactionQuerySet object it won't fall in recursion.
class Transaction(Document):
title = StringField(max_length=100, required=True)
last_modified = DateTimeField(default=datetime.utcnow)
def __str__(self):
return self.title
meta = {"collection": "transaction_collection"}
with MongoClient("mongodb://localhost:27017") as client:
connection = client.get_database("transaction")
collection = connection.transaction_collection
transaction_set = TransactionQuerySet(Transaction, collection)
print(transaction_set.order_by("-title"))
OUTPUT
[<Transaction: ETHUSTD>, <Transaction: BTCUSTD>, ...]
I'm trying to initialize an objects field with a class that needs to know the type that is using it:
class Device(Model):
objects = AbstractManager(Device)
# the rest of the class here
This is how AbstractManager is defined:
class AbstractManager:
def __init__(self, cls: type):
self.cls = cls
def all(self):
result = []
for cls in self._get_subclasses():
result.extend(list(cls.objects.all()))
return result
def _get_subclasses(self):
return self.cls.__subclasses__()
So I can later call this and returns all() from all subclasses:
Device.objects.all()
The issue here is that I cannot use Device while initializing Device.objects, since Device is still not initialized.
As a work-around I'm initializing this outside of the class, but there's gotta be a better way:
class Device(Model):
objects = None
# the rest of the class here
Device.objects = AbstractManager(Device)
PD: I have a C#/C++ background, so maybe I'm thinking too much about this in a static-typing mindset, can't tell
You don't need to add any additional logic for this. Django allows you to access model class from manager using self.model attribute:
def _get_subclasses(self):
return self.model.__subclasses__()
You do not have to do that. Django will automatically call the contribute_to_class method, where it will pass the model, and for a manager, it will be stored in self.model. You can thus simply implement this as:
from django.db.models.manager import ManagerDescriptor
class AbstractManager(models.Manager):
def all(self):
result = []
for cls in self._get_subclasses():
result.extend(list(cls.objects.all()))
return result
def contribute_to_class(self, model, name):
self.name = self.name or name
self.model = model
setattr(model, name, AbstractManagerDescriptor(self))
model._meta.add_manager(self)
def _get_subclasses(self):
return self.model.__subclasses__()
class AbstractManagerDescriptor(ManagerDescriptor):
def __get__(self, instance, cls=None):
if instance is not None:
raise AttributeError("Manager isn't accessible via %s instances" % cls.__name__)
if cls._meta.swapped:
raise AttributeError(
"Manager isn't available; '%s.%s' has been swapped for '%s'" % (
cls._meta.app_label,
cls._meta.object_name,
cls._meta.swapped,
)
)
return cls._meta.managers_map[self.manager.name]
and add the manager as:
class Device(models.Model):
objects = AbstractManager()
That being said, I'm not sure that this is a good idea for two reasons:
you are returning a list, and normally .all() returns a QuerySet, you thus here "destroy" the laziness of the queryset, which can result in expensive querying; and
if one would use Device.objects.filter() for example, it would simply circumvent.
You might want to subclass the queryset, and then aim to implement that differently.
I'm trying to inject dependencies into my Django view (controller?). Here's some background.
Normally, the urls.py file is what handles the routing. It is usually something like this:
urlpatterns = [
path("", views.get_all_posts, name="get_all_posts"),
path("<int:post_id>", views.get_post, name="get_post"),
path("create", views.create_post, name="create_post"),
]
The problem with this, is that once you get to create_post for instance, you might have a dependency on a service that creates posts:
# views.py
...
def create_post(self):
svc = PostCreationService()
svc.create_post()
This kind of pattern is difficult to test. While I know python testing libraries have tools to mock this sort of thing, I'd rather inject the dependency into the view. Here's what I came up with.
A Controller class that has a static method, export(deps) that takes in a list of dependencies and returns a list of url pattern objects:
class ApiController(object):
#staticmethod
def export(**deps):
ctrl = ApiController(**deps)
return [
path("", ctrl.get_all_posts, name="get_all_posts"),
path("<int:post_id>", ctrl.get_post, name="get_post"),
path("create", ctrl.create_post, name="create_post"),
]
def __init__(self, **deps):
self.deps = deps
def get_all_posts():
pass
...
This looks janky, but I'm not aware of any other way to do what I'm trying to do. The controller needs to return a list of url patterns, and it also needs to take in a list of dependencies. Using the above technique, I can do this in urls.py:
urlpatterns = ApiController.export(foo_service=(lambda x: x))
I am now free to use foo_service in any of the methods of ApiController.
Note:
One alternative would be for the constructor to return the list of urls, but I don't see that as a huge improvement over this. In fact, it strikes me as being more confusing because the class constructor would return a list instead of an instance of the class.
Note 2:
I'm aware that python has mocking tools for mocking class members. Please don't suggest using them. I'd like to use DI as the way to control and manage dependencies.
Any ideas on what the best way to do this is?
Consider injecting using decorators:
from functools import wraps
class ServiceInjector:
def __init__(self):
self.deps = {}
def register(self, name=None):
name = name
def decorator(thing):
"""
thing here can be class or function or anything really
"""
if not name:
if not hasattr(thing, "__name__"):
raise Exception("no name")
thing_name = thing.__name__
else:
thing_name = name
self.deps[thing_name] = thing
return thing
return decorator
def inject(self, func):
#wraps(func)
def decorated(*args, **kwargs):
new_args = args + (self.deps, )
return func(*new_args, **kwargs)
return decorated
# usage:
si = ServiceInjector()
# use func.__name__, registering func
#si.register()
def foo(*args):
return sum(args)
# we can rename what it's been registered as, here, the class is registered
# with name `UpperCase` instead of the class name `UpperCaseRepresentation`
#si.register(name="UpperCase")
class UpperCaseRepresentation:
def __init__(self, value):
self.value = value
def __str__(self):
return self.value.upper()
#register float
si.register(name="PI")(3.141592653)
# inject into functions
#si.inject
def bar(a, b, c, _deps): # the last one in *args would be receiving the dependencies
UpperCase, PI, foo = _deps['UpperCase'], _deps['PI'], _deps['foo']
print(UpperCase('abc')) # ABC
print(PI) # 3.141592653
print(foo(a, b, c, 4, 5)) # = 15
bar(1, 2, 3)
# inject into class methods
class Foo:
#si.inject
def my_method(self, a, b, _deps, kwarg1=30):
return _deps['foo'](a, b, kwarg1)
print(Foo().my_method(1, 2, kwarg1=50)) # = 53
You could take a look at https://github.com/ets-labs/python-dependency-injector, but that is a pretty big setup.
You could also create something small like a Service factory
# services.py
class ServiceFactory:
def __init__(self):
self.__services = {}
def register(self, name, service_class):
# Maybe add some validation
self.__services[name] = service_class
def create(self, name, *args, **kwargs):
# Maybe add some error handling or fallbacks
return self.__services[name](*args, **kwargs)
factory = ServiceFactory()
# In your settings.py for example
from services import factory
factory.register('post_creation', PostCreationService)
# Or maybe in apps.ready do auto_load that will loop all apps and get config from services.py
# In your views.py
from services import factory
def create_post(self):
svc = factory.create('post_creation')
svc.create_post()
# In your tests.py
from services import factory
def setUp(self):
factory.register('post_creation', FakePostCreationService)
While reading Dependency Injection Principles, Practices, and Patterns and trying to apply the examples to a django app I came up with the following:
# views.py
class IndexView(View):
# Must include this to bypass django's validation
product_service: IProductService = None
# Init method not necessary but more explicit
def __init__(self, product_service: IProductService):
self.product_service = product_service
def get(self, request):
self.product_service.do_stuff()
...
# urls.py
# Construct dependencies. I guess this is the closest to the entry-point we can get
# with Django.
repo = DjangoProductRepository()
product_service = ProductService(repo)
urlpatterns = [
path('admin/', admin.site.urls),
path("",
IndexView.as_view(product_service=product_service),
name="index"),
]
This is only an updated version of rabbit.aaron reply above. My idea is to be able to specify which dependencies to inject instead of getting a dictionary with all registered dependencies.
from functools import wraps
class ServiceInjector:
deps = {}
def register(self, name=None):
name = name
def decorator(thing):
"""
thing here can be class or function or anything really
"""
if not name:
if not hasattr(thing, '__name__'):
raise Exception('no name')
thing_name = thing.__name__
else:
thing_name = name
self.__class__.deps[thing_name] = thing
return thing
return decorator
class inject:
def __init__(self, *args):
self.selected_deps = args
def __call__(self, func):
#wraps(func)
def decorated(*args, **kwargs):
selected_deps = {k: v for k, v in ServiceInjector.deps.items() if k in self.selected_deps}
new_kwargs = {**kwargs, **selected_deps}
return func(*args, **new_kwargs)
return decorated
Usage:
si = ServiceInjector()
# use func.__name__, registering func
#si.register()
def foo(*args):
return sum(args)
Custom naming still works
#si.register(name='uppercase')
class UpperCaseRepresentation:
def __init__(self, value):
self.value = value
def __str__(self):
return self.value.upper()
Register float
si.register(name="PI")(3.141592653)
Inject into functions
#si.inject('foo', 'PI', 'uppercase')
def bar(a, b, c, uppercase: UpperCaseRepresentation, **kwargs):
"""
You can specify dependencies as keyword arguments and add typehint annotation.
"""
UpperCase, foo = kwargs['UpperCase'], kwargs['foo']
print(uppercase('abc')) # ABC
print(PI) # 3.141592653
print(foo(a, b, c, 4, 5)) # = 15
bar(1, 2, 3)
Inject into class methods
class Bar:
#si.inject('foo')
def my_method(self, a, b, foo, kwarg1=30):
return foo(a, b, kwarg1)
print(Bar().my_method(1, 2, kwarg1=50)) # = 53
You could go the flask route and export a class instance with a property that initializes and caches the service on first access. E.g:
def default_factory():
pass
# service.py
class ServiceProvider:
def __init__(self, create_instance=default_factory):
self.create_instance = create_instance
_instance = None
#property
def service(self):
if self._instance:
return self._instance
self._instance = self.create_instance()
return self._instance
service_provider = ServiceProvider()
from .service import service_provider
# views.py
def view(request):
service_provider.service.do_stuff()
# etc.
This has the advantages of being easy to mock and not having any magic.
The most boring solution I could come up with involves using class variables:
# Module services.post_service
def default_create_post():
return "foo"
class Provider:
create_post = default_create_post
Then you could import and use normally in a view or elsewhere:
from services import post_service
post_service.Provider.create_post()
# Should return "foo"
And when testing it could be swapped out before being called:
from django.test import TestCase
from services import post_service
from unittest.mock import patch
class MyTestCase(TestCase):
#patch('services.post_service.default_create_post')
def test_some_view(self, mock_create_post):
mock_create_post.return_value = "bar"
post_service.Provider.create_post = mock_create_post
# Now when calling post_service.Provider.create_post it should just return "bar"
I have a serializer as:
class DataSerializer(serializers.Serializer):
skip_place = True
name = serializers.CharField(max_length=None)
place = serializers.CharField(max_length=None,required=False)
def validate_place(self,value):
if not skip_place and len(value)<=0:
raise serializers.ValidationError("Place is required.")
Now observe skip_place. If I call the DataSerializer instance as:
DataSerializer(data=data, skip_place=False)
Then it should validate over place as in the code.
But I was not able to pass the argument skip_place=True. I get an error: TypeError: __init__() got an unexpected keyword argument 'skip_place'
You can send it with including extra context.
In view;
DataSerializer(data=data, context={'skip_place': True}) # or False
In serializer;
class DataSerializer(serializers.Serializer):
name = serializers.CharField(max_length=None)
place = serializers.CharField(max_length=None,required=False)
def validate_place(self,value):
skip_place = self.context.get("skip_place") # <- here we use self.context to get extra args
if not skip_place and len(value)<=0:
raise serializers.ValidationError("Place is required.")
Hi You can override the init method of serializer, pop the value from there and assign into variable.
class DataSerializer(serializers.Serializer):
skip_place = True
name = serializers.CharField(max_length=None)
place = serializers.CharField(max_length=None,required=False)
def __init__(self, *args, **kwargs):
self.skip_place = kwargs.pop('skip_place ', False)
super().__init__(*args, **kwargs)
def validate_place(self,value):
if not self.skip_place and len(value)<=0:
raise serializers.ValidationError("Place is required.")
I'm new to ItemLoaders.
I've a set seen_ids in which I add all the product_ids which I scrape so that I can check if there is any duplicate and skip it at earliest.
The problem is, I want to do this in __init__. If it's a duplicate, I don't want any reference to be returned, and I can't explicitly return None from __init__. How would I do that?
seen_ids = set()
def __init__(self, item=None, selector=None, response=None, parent=None, product_id=None, **context):
if product_id in self.seen_ids:
return None
self.seen_ids.add(product_id)
super(GarmentLoader, self).__init__(item, selector, response, parent, **context)
item['retailer_sku'] = product_id
But it's giving error on None, and if I don't return anything, it returns object's reference and further checks fail.
It wouldn't work because constructor basically doesn't return anything else than instance and because instances wouldn't share seen_ids.
You can use class method instead:
class CustomItemLoader(ItemLoader):
seen_ids = set()
#classmethod
def with_product_id(cls, **kwargs):
product_id = kwargs.pop('product_id', None)
if product_id in cls.seen_ids:
return None
cls.seen_ids.add(product_id)
return cls(**kwargs)
Then create loader's instance using it:
loader = CustomItemLoader.with_product_id(response=response, product_id=product_id, ...)