We Keep Coding

Indirect way of usine Model.objects.all() in a formset

I'm using something like this to populate inlineformsets for an update view:
formset = inline_formsetfactory(Client, Groupe_esc, form=GroupEscForm, formset=BaseGroupEscInlineFormset, extra=len(Groupe.objects.all()))
(Basically I need as many extra form as there are entries in that table, for some special processing I'm doing in class BaseGroupEscInlineFormset(BaseInlineFormset)).
That all works fine, BUT if I pull my code & try to makemigrations in order to establish a brand new DB, that line apparently fails some django checks and throws up a "no such table (Groupe)" error and I cannot makemigrations. Commenting that line solves the issues (then I can uncomment it after making migration). But that's exactly best programming practices.
So I would need a way to achieve the same result (determine the extra number of forms based on the content of Groupe table)... but without triggering that django check that fails. I'm unsure if the answer is django-ic or pythonic.
E.g. perhaps I could so some python hack that allows me to specific the classname without actually importing Groupe, so I can do my_hacky_groupe_import.Objects.all(), and maybe that wouldn't trigger the error?
EDIT:
In forms.py:
from .models import Client, Groupe
class BaseGroupEscInlineFormset(BaseInlineFormSet):
def get_form_kwargs(self, index):
""" this BaseInlineFormset method returns kwargs provided to the form.
in this case the kwargs are provided to the GroupEsForm constructor
"""
kwargs = super().get_form_kwargs(index)
try:
group_details = kwargs['group_details'][index]
except Exception as ex: # likely this is a POST, but the data is already in the form
group_details = []
return {'group_details':group_details}
GroupeEscFormset = inlineformset_factory(Client, Groupe_esc,
form=GroupeEscForm,
formset=BaseGroupEscInlineFormset,
extra=len(Groupe.objects.all()),
can_delete=False)

The issue as already outlined is that your code is written at the module level and it executes a query when the migrations are not yet done, giving you an error.
One solution as I already pointed in the comment would be to write the line to create the formset class in a view, example:
def some_view(request):
GroupeEscFormset = inlineformset_factory(
Client,
Groupe_esc,
form=GroupeEscForm,
formset=BaseGroupEscInlineFormset,
extra=len(Groupe.objects.all()),
can_delete=False
)
Or if you want some optimization and want to keep this line at the module level to not keep recreating this formset class, you can override the __init__ method and accept extra as an argument (basically your indirect way to call Model.objects.all()):
class BaseGroupEscInlineFormset(BaseInlineFormSet):
def __init__(self, *args, extra=3, **kwargs):
self.extra = extra
super().__init__(*args, **kwargs)
...
GroupeEscFormset = inlineformset_factory(Client, Groupe_esc,
form=GroupeEscForm,
formset=BaseGroupEscInlineFormset,
can_delete=False)
# In your views:
def some_view(request):
formset = GroupeEscFormset(..., extra=Groupe.objects.count()) # count is better if the queryset is needed only to get a count

How Does Django Manager Orchestrate QuerySet?

This is a follow-up question from this post. For the sake of completeness and self-containment, I will include all necessary bits in this question.
Consider a very simple example:
# Models
class Cart(models.Model):
name = models.CharField(max_length=20)
class Item(models.Model):
name = models.CharField(max_length=20)
cart = models.ForeignKey(Cart, on_delete=models.CASCADE)
# Execution
class Test(TestCase):
#classmethod
def setUpTestData(cls):
# Save a new cart
c = Cart(name="MyCart")
c.save()
# Save two items to the cart
t = Item(name="item1", cart=c)
t.save()
Item.objects.create(name="item2", cart=c)
def simpleTest(self):
i = Item.objects.get(id=2) # <--- breakpoint
# Do something, but not important for this question
I am using PyCharm to try to understand the inner workings of Django. If we put a breakpoint at the marked line and step into the method call get(). After we get a Manager from __get__ in ManagerDescriptor (due to .objects), the execution now goes to this class method in BaseManager:
#classmethod
def _get_queryset_methods(cls, queryset_class):
def create_method(name, method):
def manager_method(self, *args, **kwargs):
return getattr(self.get_queryset(), name)(*args, **kwargs) # <--- Starts here
manager_method.__name__ = method.__name__
manager_method.__doc__ = method.__doc__
return manager_method
The execution stops at the marked line above. If we step into it, get_queryset() is first called, which returns a new QuerySet object, as expected. Note that the new QuerySet object initializes self._result_cache to None. This attribute caches the result of the QuerySet. After the object is initialized, the get() method is finally called. However, if we inspect the calling QuerySet object (which is the object that is just initialized), its _result_cache has already been populated with the two Item objects that I have saved. I suspect something is probably happening in a different thread, perhaps? Can someone experienced in Django point out the relevant code that is executed between the end of get_queryset() and before the beginning of get()? I can't seem to find any documentation myself.
Thanks!

Do I need a manager when creating and saving objects in Django?

I'm using Django with Python 3.7 and PyCharm. I'm following this tutorial for learning how to create model and save them into the database -- https://docs.djangoproject.com/en/dev/topics/db/queries/#creating-objects . The tutorial refers to manager for retrieving objects, but not for setting them, so I'm confused as to why I get the below error
Article.objects.create_article(main page, '/path', 'url', 10, 22)
Traceback (most recent call last):
File "<input>", line 1, in <module>
AttributeError: 'Manager' object has no attribute 'create_article'
when I'm trying to create and save an object. Below is how I define my object in my models.py file ...
class Article(models.Model):
mainpage = models.ForeignKey(MainPage, on_delete=models.CASCADE,)
path = models.CharField(max_length=100)
url = models.TextField
time = models.DateTimeField(default=datetime.now)
votes = models.IntegerField(default=0)
comments = models.IntegerField(default=0)
def __str__(self):
return self.path
#classmethod
def create(cls, mainpage, path, url, votes, comments):
article = cls(mainpage=mainpage,path=path,url=url,votes=votes,comments=comments)
return article
I'm sure I'm missing something really obvious, but I don't know what it is.
Edit: Many have suggested using the "Article.objects.create" method but below is the output from the console ...
Article.objects.create(mainpage=mainpage, path='/path', url='url', votes=10, comments=22)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Users/davea/Documents/workspace/mainarticles_project/venv/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/davea/Documents/workspace/mainarticles_project/venv/lib/python3.7/site-packages/django/db/models/query.py", line 411, in create
obj = self.model(**kwargs)
File "/Users/davea/Documents/workspace/mainarticles_project/venv/lib/python3.7/site-packages/django/db/models/base.py", line 485, in __init__
raise TypeError("'%s' is an invalid keyword argument for this function" % kwarg)
TypeError: 'url' is an invalid keyword argument for this function

When you define a Model Class, Django jumps into the fray and adds a default Manager to your model under the "objects" property. You can customize the Model and the Manager for different purposes.
Generally methods on your Model should deal with a single instance, or conceptually a row in your database. Model.save(), Model.delete() all act on a single instance or row.
Methods on your manager should usually work with your table as a whole, such as filter(), get(), aggregate(), because these functions perform operations against your table. You also have the create() method on your manager because it adds a row to your table. Further, you can define custom Manager's and assign them to different properties on your model. For example:
class EngineerManager(Manager):
def get_queryset(self):
return super().get_queryset().filter(employee_type="engineer")
class ManagerManager(Manager):
def get_queryset(self):
return super().get_queryset().filter(employee_type="manager")
class Employee(Model):
employee_type = models.CharField()
engineers = EngineerManager()
managers = ManagerManager()
objects = Manager() # This is always added to your models to list all objects
Employee.engineers.filter() # Will only return engineering employees
Employee.objects.filter() # Will return ALL employees
Now to your problem, Article.objects.create_article(...) does not seem to exist, you should probably use Article.objects.create(...) because the default Manager does have a create() method.
Using Article.objects.create(...) will persist a new Article to the database and return it. Otherwise, you could technically use your Article.create(...) method to create an in-memory instance of Article, however it has not been saved to the database so keep in mind you will have to call save() on the instance before it's persisted to your database.
https://docs.djangoproject.com/en/2.1/topics/db/models/
https://docs.djangoproject.com/en/2.1/topics/db/managers/

To create an article in database use manager's create method (params should be named, not positioned, as in your example)
Article.objects.create(mainpage=mainpage, path=path, url=url,
votes=votes, comments=comments)
or you can initialise class instance and then save it. Here params could be positioned or named, no matter.
article = Article(mainpage, path, url, votes, comments)
article.save()
Your method def create(cls, mainpage, path, url, votes, comments): makes no sense, because it duplicates call (__call__ method) of a class, like in my second example. If you want to add some extra logic to object creation, you should define custom Manager class and add method there and then link your model's objects property to you custom manager class like objects = CustomManager()

Django : how could I overload Q in django.db.models.query QuerySet to use for a special purpose in my Manager

First of all, since this is my first question/post, I would like to thank you all for this great community, and amazing service, as -- like many developers around the world -- stackoverflow -- is my main resource when it comes to code issues.
Notice :
This post is a bit long (sorry), covers two different -- but related -- aspects of the situation, and is organised as follows :
Background / Context
A design issue, on which i would like to receive your advice.
The solution I'm trying to implement to solve 2.
The actual issue, and question, related to 3.
1. Some context :
I have two different Model objects (Report and Jobs) from an existing implementation that has a poor design.
Fact is that both objects are quite similar in purpose, but were probably implemented in two different time frames.
A lot of processing happen on these objects, and since the the system has to evolve, I started to write a metaclass/interface, from which both will be subclasses. Currently both Models use different Fields names for same purpose, like author and juser to denote User (which is very stupid) and so on.
Since I cannot afford to just change the columns names in the database, and then go through thousands of lines of code to change every references to these fields (even though I could, thanks to Find Usages feature of modern IDEs), and also because theses object might be used somewhere else, I used the db_column= feat. to be able in each model to have the same field name and ultimately handle both object alike (instead of having thousands of line of duplicated code to do the same stuff).
So, I have something like that :
from django.db import models
class Report(Runnable):
_name = models.CharField(max_length=55, db_column='name')
_description = models.CharField(max_length=350, blank=True, db_column='description')
_author = ForeignKey(User, db_column='author_id')
# and so on
class Jobs(Runnable):
_name = models.CharField(max_length=55, db_column='jname')
_description = models.CharField(max_length=4900, blank=True, db_column='jdetails')
_author = ForeignKey(User, db_column='juser_id')
# and so on
As i said earlier, to avoid rewriting object's client code, I used properties that shadows the fields :
from django.db import models
class Runnable(models.Model):
objects = managers.WorkersManager() # The default manager.
#property # Report
def type(self):
return self._type
#type.setter # Report
def type(self, value):
self._type = value
# for backward compatibility, TODO remove both
#property # Jobs
def script(self):
return self._type
#script.setter # Jobs
def script(self, value):
self._type = value
# and so on
2. The design issue :
This is nice and it's kind of what I wanted, except now using Report.objects.filter(name='something') or Jobs.objects.filter(jname='something') won't work, obviously due to Django design (and so on with .get(), .exclude() etc...), and the client code is sadly full of those.
I'm of course planning to replace them all with methods of my newly created WorkersManager
Aparté :
Wait ... what ? "newly created WorkersManager" ??
Yes, after two years and thousands of line of code, there was no Manager in here, crazy right ?But guess what ? that's the least of my concerns; and cheer up, since most of the code still lies in view.py and assosiated files (instead of being properly inside the objects it is suposed to manipulate), basicaly somewhat "pure" imperative python...
Great right ?
3. My solution :
After a lot of reading (here and there) and research about that, I found out that :
Trying to subclass Field, was not a solution
I could actually overload QuerySet.
So I did :
from django.db.models.query_utils import Q as __originalQ
class WorkersManager(models.Manager):
def get_queryset(self):
class QuerySet(__original_QS):
"""
Overloads original QuerySet class
"""
__translate = _translate # an external fonction that changes the name of the keys in kwargs
def filter(self, *args, **kwargs):
args, kwargs = self.__translate(*args, **kwargs)
super(QuerySet, self).filter(args, kwargs)
# and many more others [...]
return QuerySet(self.model, using=self._db)
And this is quite fine.
4. So what's wrong ?
The problem is that Django internally uses Q inside db.model.query, using its own imports, and nowhere Q is exposed or referenced, so it could be overloaded.
>>> a =Report.objects.filter(name='something')
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/venv/local/lib/python2.7/site-packages/django/db/models/manager.py", line 143, in filter
return self.get_query_set().filter(*args, **kwargs)
File "/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 624, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 642, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1250, in add_q
can_reuse=used_aliases, force_having=force_having)
File "/venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1122, in add_filter
process_extras=process_extras)
File "/venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1316, in setup_joins
"Choices are: %s" % (name, ", ".join(names)))
FieldError: Cannot resolve keyword 'name' into field. Choices are: _author, _description, _name, # and many more
But I do remember reading something about how Django only loads the first occurrence of a Model, and how you could trick it by redefining such a Model before using import (well obviously this doesn't apply to python)
So ultimately I tried to overload Q, by redefining it before importing relevant class, or after, but I cannot possibly figure it out.
Here is what I tried :
from django.db.models.query_utils import Q as __originalQ
__translation = {'name': '_name',} # has much more, just for exemple
def _translate(*args, **kwargs):
for key in kwargs:
if key in __translation.keys():
kwargs[__translation[key]] = kwargs[key]
del kwargs[key]
return args, kwargs
class Q(__originalQ):
"""
Overloads original Q class
"""
def __init__(self, *args, **kwargs):
super(Q, self).__init__(_translate(*args, **kwargs))
# now import QuerySet which should use the new Q class
from django.db.models.query import QuerySet as __original_QS
class QuerySet(__original_QS):
"""
Overloads original QuerySet class
"""
__translate = _translate # writing shortcut
def filter(self, *args, **kwargs):
args, kwargs = self.__translate(*args, **kwargs)
super(QuerySet, self).filter(args, kwargs)
# and some others
# now import QuerySet which should use the new QuerySet class
from django.db import models
class WorkersManager(models.Manager):
def get_queryset(self):
# might not even be required if above code was successful
return QuerySet(self.model, using=self._db)
This of course has no effect, as Q gets re-imported from django.db.model.query in the definition of _filter_or_exclude.
So of course, an intuitive solution would be to overload _filter_or_exclude, and copy its original code without calling the superBut here is the catch : I'm using an old version of Django, that might be updated someday, and I don't want to mess with Django implementation specifics, as I already did with get_queryset, but I guess this is kind of ok since it's (as far as I understand) a placeholder for overloading, and it was also the only way.
So here I am, and my question is :
Is there no other way to do it ? is there no way for me to overload Q inside of a Django module ?
Thank you very much for reading all the way :)
here is a potato (Oups, wrong website, sorry :) )
EDIT :
So, after trying to overload _filter_or_exclude, it seems that it has no effect.
I'm probably missing something about the call stack order or something alike... I'm continue tomorrow, and let you know.

Yeay ! I found the solution.
Turns out that first, forgot to have a return in my functions, like :
def filter(self, *args, **kwargs):
args, kwargs = self.__translate(*args, **kwargs)
super(QuerySet, self).filter(args, kwargs)
Instead of :
def filter(self, *args, **kwargs):
args, kwargs = self.__translate(*args, **kwargs)
return super(QuerySet, self).filter(args, kwargs)
and I also had :
args, kwargs = self.__translate(*args, **kwargs)
Instead of :
args, kwargs = self.__translate(args, kwargs)
which cause unpack on fucntion call, and thus eveything from original kwargs ended up in args, thus preventing translate to have any effect.
But even worst, I failed to understand that I could directly overload filter , get and so on, directly from my custom manager...
Which saves me the effort of dealing with QuerySet and Q.
In the end the following code is working as expected :
def _translate(args, kwargs):
for key in kwargs.keys():
if key in __translation.keys():
kwargs[__translation[key]] = kwargs[key]
del kwargs[key]
return args, kwargs
class WorkersManager(models.Manager):
def filter(self, *args, **kwargs):
args, kwargs = _translate(args, kwargs)
return super(WorkersManager, self).filter(*args, **kwargs)
# etc...

keyerror inside django model class __init__

Here's a Django model class I wrote. This class gets a keyerror when I call get_object_or_404 from Django (I conceive that keyerror is raised due to no kwargs being passed to __init__ by the get function, arguments are all positional). Interestingly, it does not get an error when I call get_object_or_404 from console.
I wonder why, and if the below code is the correct way (ie, using init to populate the link field) to construct this class.
class Link(models.Model)
event_type = models.IntegerField(choices=EVENT_TYPES)
user = models.ForeignKey(User)
created_on = models.DateTimeField(auto_now_add = True)
link = models.CharField(max_length=30)
isActive = models.BooleanField(default=True)
def _generate_link(self):
prelink = str(self.user.id)+str(self.event_type)+str(self.created_on)
m = md5.new()
m.update(prelink)
return m.hexdigest()
def __init__(self, *args, **kwargs):
self.user = kwargs['user'].pop()
self.event_type = kwargs['event_type'].pop()
self.link = self._generate_link()
super(Link,self).__init__(*args,**kwargs)

self.user = kwargs['user'].pop()
self.event_type = kwargs['event_type'].pop()
You're trying to retrieve an entry from the dictionary, and then call its pop method. If you want to remove and return an object from a dictionary, call dict.pop():
self.user = kwargs.pop('user')
Of course, this will fail with a KeyError when "user" is not present in kwargs. You'll want to provide a default value to pop:
self.user = kwargs.pop('user', None)
This means "if "user" is in the dictionary, remove and return it. Otherwise, return None".
Regarding the other two lines:
self.link = self._generate_link()
super(Link,self).__init__(*args,**kwargs)
super().__init__() will set link to something, probably None. I would reverse the lines, to something like this:
super(Link,self).__init__(*args,**kwargs)
self.link = self._generate_link()
You might want to add a test before setting the link, to see if it already exists (if self.link is not None: ...). That way, links you pass into the constructor won't be overwritten.

There's no reason to write your own __init__ for Django model classes. I think you'll be a lot happier without it.
Almost anything you think you want to do in __init__ can be better done in save.

I don't think you need the __init__ here at all.
You are always calculating the value of link when the class is instantiated. This means you ignore whatever is stored in the database. Since this is the case, why bother with a model field at all? You would be better making link a property, with the getter using the code from _generate_link.
#property
def link(self):
....

wonder why, and if the below code is the correct way (ie, using __init__ to populate the link field) to construct this class.
I once got some problems when I tried to overload __init__
In the maillist i got this answer
It's best not to overload it with your own
__init__. A better option is to hook into the post_init signal with a
custom method and in that method do your process() and
make_thumbnail() calls.
In your case the post_init-signal should do the trick and implementing __init__ shouldn't be necessary at all.
You could write something like this:
class Link(models.Model)
event_type = models.IntegerField(choices=EVENT_TYPES)
user = models.ForeignKey(User)
created_on = models.DateTimeField(auto_now_add = True)
link = models.CharField(max_length=30)
isActive = models.BooleanField(default=True)
def create_link(self):
prelink = str(self.user.id)+str(self.event_type)+str(self.created_on)
m = md5.new()
m.update(prelink)
return m.hexdigest()
def post_link_init(sender, **kwargs):
kwargs['instance'].create_link()
post_init.connect(post_link_init, sender=Link)
>>> link = Link(event_type=1, user=aUser, created_on=datetime.now(), link='foo', isActive=True)
providing keyword unique for link = models.CharField(max_length=30, unique=True) could be helpful, too. If it is not provided, get_object_or_404 may won't work in case the same value in the link-field exists several times.
signals and unique in the django-docs