I'm new with django and I'm trying to make a unit test where I want to compare a QuerySet before and after a batch editing function call.
def test_batchEditing_9(self):
reset() #reset database for test
query = Game.objects.all()
query_old = Game.objects.all()
dict_value = {'game_code' : '001'}
Utility.batchEditing(Game, query, dict_value)
query_new = Game.objects.all()
self.assertTrue(compareQuerySet(query_old, query_new))
My problem is that query_old will be updated after batchEditing is called. Therefor, both querysets will be the same.
It seem that QuerySet is bound to the current state of the database.
Is this normal?
Is there a way to unbind a QuerySet from the database?
I have tried queryset.values, list(queryset) but it still updates the value.
I'm actually thinking about iterating on the queryset and creating a list of dictionaries by myself, but I want to know if there is an easier way.
Here is batchEditing (didn't paste input validity check)
def batchEditing(model, query, values):
for item in query:
if isinstance(item, model):
for field, val in values.iteritems():
if val is not None:
setattr(item, field, val)
item.save()
Here is compareQuerySet
def compareQuerySet(object1, object2):
list_val1 = object1.values_list()
list_val2 = object2.values_list()
for i in range(len(list_val1)):
if list_val1[i] != list_val2[i]:
return False
return True
A Queryset is essentially just generating SQL and only on evaluating it, the database is hit. As far as I remember, that happens on iterating over the Queryset. For instance,
gamescache = list(Game.objects.all())
or
for g in Game.objects.all():
...
hit the database.
Following code should work:
def test_batchEditing_9(self):
reset() #reset database for test
query = Game.objects.all()
query_old = set(query)
dict_value = {'game_code' : '001'}
Utility.batchEditing(Game, query, dict_value)
query_new = set(query)
self.assertEqual(query_old, query_new)
This is because Game.objects.all() is not hitting database, but just creates object that stores query parameters.
BTW. If you will use order_by in query and order is important you can use list rather than set.
Related
I'm trying to get a queryset from the cache, but am unsure if this even has a point.
I have the following method (simplified) inside a custom queryset:
def queryset_from_cache(self, key: str=None, timeout: int=60):
# Generate a key based on the query.
if key is None:
key = self.__generate_key # ()
# If the cache has the key, return the cached object.
cached_object = cache.get(key, None)
# If the cache doesn't have the key, set the cache,
# and then return self (from DB) as cached_object
if cached_object is None:
cached_object = self
cache.set(key, cached_object , timeout=timeout)
return cached_object
The usage is basically to append it to a django QuerySet method, for example:
queryset = MyModel.objects.filter(id__range=[0,99]).queryset_from_cache()
My question:
Would usage like this work?
Or would it call MyModel.objects.filter(id__range=[0,99]) from the database no matter what?
Since normally caching would be done like this:
cached_object = cache.get(key, None)
if cached_object is None:
cached_object = MyModel.objects.filter(id__range=[0,99])
#Only now call the query
cache.set(key, cached_object , timeout=timeout)
And thus the queryset filter() method only gets called when the key is not present in the cache, as opposed to always calling it, and then trying to get it from the cache with the queryset_from_cache method.
This is a really cool idea, but I'm not sure if you can Cache full-on Objects.. I think it's only attributes
Now this having a point. Grom what I'm seeing from the limited code I've seen idk if it does have a point, unless filtering for Jane and John (and only them) is very common. Very narrow.
Maybe just try caching ALL the users or just individual Users, and only the attributes you need
Update
Yes! you are completetly correct, you can cache full on objects- how cool!
I don't think your example method of queryset = MyModel.objects.filter(id__range=[0,99]).queryset_from_cache() would work.
but you can do something similar by using Model Managers and do something like: queryset = MyModel.objects.queryset_from_cache(filterdict)
Models
Natually you can return just the qs, this is just for the example to show it actually is from the cache
from django.db import models
class MyModelManager(models.Manager):
def queryset_from_cache(self, filterdict):
from django.core.cache import cache
cachekey = 'MyModelCache'
qs = cache.get(cachekey)
if qs:
d = {
'in_cache': True,
'qs': qs
}
else:
qs = MyModel.objects.filter(**filterdict)
cache.set(cachekey, qs, 300) # 5 min cache
d = {
'in_cache': False,
'qs': qs
}
return d
class MyModel(models.Model):
name = models.CharField(max_length=200)
#
# other attributes
#
objects = MyModelManager()
Example Use
from app.models import MyModel
filterdict = {'pk__range':[0,99]}
r = MyModel.objects.queryset_from_cache(filterdict)
print(r['qs'])
While it's not exactly what you wanted, it might be close enough
One of my views needs to filter a queryset by a method value, example:
invoices_ids = list(map(lambda inv: inv.id, filter(lambda inv: inv.status().lower() == request['status'], invoices)))
invoices = invoices.filter(id__in = invoices_ids)
The status method comes from something like this:
class Invoice(models.Model):
(...)
def status(self):
if self.canceled:
return 'Canceled'
elif self.passed_date:
return 'Passed'
elif self.req_date:
return 'Requested'
return 'Inserted'
Problem is this kind of filtering gives me an OperationalError "too many SQL variables".. I guess there are too many invoices with a specific status, and the filter id__in gets a gigantic list.
How can i overcome this? (To filter by status without having to save into another model variable)
I have a SQLAlchemy model:
class Ticket(db.Model):
__tablename__ = 'ticket'
id = db.Column(INTEGER(unsigned=True), primary_key=True, nullable=False,
autoincrement=True)
cluster = db.Column(db.VARCHAR(128))
#classmethod
def get(cls, cluster=None):
query = db.session.query(Ticket)
if cluster is not None:
query = query.filter(Ticket.cluster==cluster)
return query.one()
If I add a new column and would like to extend the get method, I have to add one if xxx is not None like this below:
#classmethod
def get(cls, cluster=None, user=None):
query = db.session.query(Ticket)
if cluster is not None:
query = query.filter(Ticket.cluster==cluster)
if user is not None:
query = query.filter(Ticket.user==user)
return query.one()
Is there any way I could make this more efficient? If I have too many columns, the get method would become so ugly.
As always, if you don't want to write something repetitive, use a loop:
#classmethod
def get(cls, **kwargs):
query = db.session.query(Ticket)
for k, v in kwargs.items():
query = query.filter(getattr(table, k) == v)
return query.one()
Because we're no longer setting the cluster=None/user=None as defaults (but instead depending on things that weren't specified by the caller simply never being added to kwargs), we no longer need to prevent filters for null values from being added: The only way a null value will end up in the argument list is if the user actually asked to search for a value of None; so this new code is able to honor that request should it ever take place.
If you prefer to retain the calling convention where cluster and user can be passed positionally (but the user can't search for a value of None), see the initial version of this answer.
I'm trying to take an object, look up a queryset, find the item in that queryset, and find the next one.
#property
def next_object_url(self):
contacts = Model.objects.filter(owner=self.owner).order_by('-date')
place_in_query = list(contacts.values_list('id', flat=True)).index(self.id)
next_contact = contacts[place_in_query + 1]
When I add this to the model and run it, here's what I get for each variable for one instance.
CURRENT = Current Object
NEXT = Next Object
contacts.count = 1114
self.id = 3533 #This is CURRENT.id
place_in_query = 36
contacts[place_in_query] = NEXT
next_contact = CURRENT
What am i missing / what dumb mistake am i making?
In your function, contacts is a QuerySet. The actual objets are not fetched in the line:
contacts = Model.objects.filter(owner=self.owner).order_by('-date')
because you don’t use a function like list(), you don’t iterate the QuerySet yet... It is evaluated later. This is probably the reason of your problem.
Since you need to search an ID in the list of contacts and the find the next object in that list, I think there is no way but fetch all the contact and use a classic Python loop to find yours objects.
#property
def next_object_url(self):
contacts = list(Model.objects.filter(owner=self.owner).order_by('-date').all())
for curr_contact, next_contact in zip(contacts[:-1], contacts[1:]):
if curr_contact.id == self.id:
return next_contact
else:
# not found
raise ContactNotFoundError(self.id)
Another solution would be to change your database model in order to add a notion of previous/next contact at database level…
I want to log every action what will be done with some SQLAlchemy-Models.
So, I have a after_insert, after_delete and before_update hooks, where I will save previous and current representation of model,
def keep_logs(cls):
#event.listens_for(cls, 'after_delete')
def after_delete_trigger(mapper, connection, target):
pass
#event.listens_for(cls, 'after_insert')
def after_insert_trigger(mapper, connection, target):
pass
#event.listens_for(cls, 'before_update')
def before_update_trigger(mapper, connection, target):
prev = cls.query.filter_by(id=target.id).one()
# comparing previous and current model
MODELS_TO_LOGGING = (
User,
)
for cls in MODELS_TO_LOGGING:
keep_logs(cls)
But there is one problem: when I'm trying to find model in before_update hook, SQLA returns modified (dirty) version.
How can I get previous version of model before updating it?
Is there a different way to keep model changes?
Thanks!
SQLAlchemy tracks the changes to each attribute. You don't need to (and shouldn't) query the instance again in the event. Additionally, the event is triggered for any instance that has been modified, even if that modification will not change any data. Loop over each column, checking if it has been modified, and store any new values.
#event.listens_for(cls, 'before_update')
def before_update(mapper, connection, target):
state = db.inspect(target)
changes = {}
for attr in state.attrs:
hist = attr.load_history()
if not hist.has_changes():
continue
# hist.deleted holds old value
# hist.added holds new value
changes[attr.key] = hist.added
# now changes map keys to new values
I had a similar problem but wanted to be able to keep track of the deltas as changes are made to sqlalchemy models instead of just the new values. I wrote this slight extension to davidism's answer to do that along with slightly better handling of before and after, since they are lists sometimes or empty tuples other times:
from sqlalchemy import inspect
def get_model_changes(model):
"""
Return a dictionary containing changes made to the model since it was
fetched from the database.
The dictionary is of the form {'property_name': [old_value, new_value]}
Example:
user = get_user_by_id(420)
>>> '<User id=402 email="business_email#gmail.com">'
get_model_changes(user)
>>> {}
user.email = 'new_email#who-dis.biz'
get_model_changes(user)
>>> {'email': ['business_email#gmail.com', 'new_email#who-dis.biz']}
"""
state = inspect(model)
changes = {}
for attr in state.attrs:
hist = state.get_history(attr.key, True)
if not hist.has_changes():
continue
old_value = hist.deleted[0] if hist.deleted else None
new_value = hist.added[0] if hist.added else None
changes[attr.key] = [old_value, new_value]
return changes
def has_model_changed(model):
"""
Return True if there are any unsaved changes on the model.
"""
return bool(get_model_changes(model))
If an attribute is expired (which sessions do by default on commit) the old value is not available unless it was loaded before being changed. You can see this with the inspection.
state = inspect(entity)
session.commit()
state.attrs.my_attribute.history # History(added=None, unchanged=None, deleted=None)
# Load history manually
state.attrs.my_attribute.load_history()
state.attrs.my_attribute.history # History(added=(), unchanged=['my_value'], deleted=())
In order for attributes to stay loaded you can not expire entities by settings expire_on_commit to False on the session.