I want to have some kind of timestsamp for updating of models. The problem is that mysql timestamp is only in 1 sec resolution, so if I use DateTime with auto_add_now this is not fine enough for me.
I thought of using float field and to assign it to time.time() on every update. This is a model where I use update using QuerySet.update()
model MyModel:
last_update = models.FloatField(default=time.time)
I then take care of updating last_update before any save (all my calls goes through central location, so I can do this explicitly in the creation), e.g something like:
m = MyModel(..,last_update=time.time())
or when I do bulk update,
m = MyModel.objects.filter(....).update(last_update_time=time.time())
Are there any issues with this approach of saving float and order by it?
Just to clarify, I'm aware of DateTimeField option with auto_now=True, the problem is that this gives me 1 second resolution since it is mapped to date time field in MYSQL which is 1 sec resolution (I'm using MYSQL 5.5)
I think you can use DateField.auto_now, and the field will be updated each time the object gets saved. For example last_update = models.DateTimeField(auto_now = True).
You can read the documentation here, as you can see that Field option is intended to archive exactly what you want.
Also you can override the save method as you can see there and update the last_update field as you prefer.
Personally I recommend the first approach because of simplicity, there is also a third option using signals but I guess that can get so confusing.
from datetime import datetime
class Model(models.Model):
date_created = models.DateTimeField()
date_modified = models.DateTimeField()
def save(self):
if self.date_created == None:
self.date_created = datetime.now()
self.date_modified = datetime.now()
super(Model, self).save()
Related
TLDR
When creating a new object using Django ORM, can I, in a transactionally safe / race-condition-free manner, set a field's value based on an already existing object's value, say F('sequence_number') + 1 where F('sequence_number') refers not to the current object (which does not exist yet) but to the most recent object with that prefix in the table?
Longer version
I have a model Issue with properties sequence_number and sequence_prefix. There is a unique constraint on (sequence_prefix, sequence_number) (e.g. DATA-1).
class Issue(models.Model):
created_at = models.DateTimeField(auto_now_add=True)
sequence_prefix = models.CharField(blank=True, default="", max_length=32)
sequence_number = models.IntegerField(null=False)
class Meta:
constraints = [
models.UniqueConstraint(
fields=["sequence_prefix", "sequence_number"], name="unique_sequence"
)
]
The idea is that issues —for auditing purposes— have unique sequence numbers for each variable (user-determined) prefix: when creating an issue the user selects a prefix, e.g. REVIEW or DATA, and the sequence number is the incremented value of the previous issue with that same sequence. So it's like an AutoField but dependent on the value of another field for its value. There can not be two issues DATA-1, but REVIEW-1 and DATA-1 and OTHER-1 all may exist at the same time.
How can I tell Django when creating an Issue, that it must find the most recent object for that given sequence_prefix, take the sequence_number + 1 and use that for the new object's sequence_number value, in a way that is safe of any race-condition?
A good way to archive this is to override the save() method of the Issue model.
For example:
class Issue(models.Model):
created_at = models.DateTimeField(auto_now_add=True)
sequence_prefix = models.CharField(blank=True, default="", max_length=32)
sequence_number = models.IntegerField(null=False)
def save(self, *args, **kwargs):
max_id_by_prefix = Issue.objects.filter(sequence_prefix=self.sequence_prefix).max().id
self.sequence_number = max_id_by_prefix + 1
super(Issue, self).save(*args, **kwargs)
class Meta:
constraints = [
models.UniqueConstraint(
fields=["sequence_prefix", "sequence_number"], name="unique_sequence"
)
]
In this way, before saving the object, you can take the max sequence_number of the sequence_prefix that you are saving.
Unless you want to use database sequences (AutoField), I believe you will need to implement something on your own. There are two options
Prevent concurrent inserts per specific sequence_prefix with some locking mechanism (I would use Redis for a distributed lock, to support multi-processing setup)
Implement your own sequencing (again, Redis is a perfect choices), which will provide you with auto-incrementing sequence_number per prefix. For example:
sequence_number = redis_client.incr('sequence:REVIEW')
I have a model Student with manager StudentManager as given below. As property gives the last date by adding college_duration in join_date. But when I execute this property computation is working well, but for StudentManager it gives an error. How to write manager class which on the fly computes some field using model fields and which is used to filter records.
The computed field is not in model fields. still, I want that as filter criteria.
class StudentManager(models.Manager):
def passed_students(self):
return self.filter(college_end_date__lt=timezone.now())
class Student(models.Model):
join_date = models.DateTimeField(auto_now_add=True)
college_duration = models.IntegerField(default=4)
objects = StudentManager()
#property
def college_end_date(self):
last_date = self.join_date + timezone.timedelta(days=self.college_duration)
return last_date
Error Django gives. when I tried to access Student.objects.passed_students()
django.core.exceptions.FieldError: Cannot resolve keyword 'college_end_date' into field. Choices are: join_date, college_duration
Q 1. How alias queries done in Django ORM?
By using the annotate(...)--(Django Doc) or alias(...) (New in Django 3.2) if you're using the value only as a filter.
Q 2. Why property not accessed in Django managers?
Because the model managers (more accurately, the QuerySet s) are wrapping things that are being done in the database. You can call the model managers as a high-level database wrapper too.
But, the property college_end_date is only defined in your model class and the database is not aware of it, and hence the error.
Q 3. How to write manager to filter records based on the field which is not in models, but can be calculated using fields present in the model?
Using annotate(...) method is the proper Django way of doing so. As a side note, a complex property logic may not be re-create with the annotate(...) method.
In your case, I would change college_duration field from IntegerField(...) to DurationField(...)--(Django Doc) since its make more sense (to me)
Later, update your manager and the properties as,
from django.db import models
from django.utils import timezone
class StudentManager(models.Manager):
<b>def passed_students(self):
default_qs = self.get_queryset()
college_end = models.ExpressionWrapper(
models.F('join_date') + models.F('college_duration'),
output_field=models.DateField()
)
return default_qs \
.annotate(college_end=college_end) \
.filter(college_end__lt=timezone.now().date())</b>
class Student(models.Model):
join_date = models.DateTimeField()
college_duration = models.DurationField()
objects = StudentManager()
#property
def college_end_date(self):
# return date by summing the datetime and timedelta objects
return <b>(self.join_date + self.college_duration).date()
Note:
DurationField(...) will work as expected in PostgreSQL and this implementation will work as-is in PSQL. You may have problems if you are using any other databases, if so, you may need to have a "database function" which operates over the datetime and duration datasets corresponding to your specific database.
Personally, I like this solution,
To quote #Willem Van Olsem's comment:
You don't. The database does not know anything about properties, etc. So it can not filter on this. You can make use of .annotate(..) to move the logic to the database side.
You can either do the message he shared, or make that a model field that auto calculates.
class StudentManager(models.Manager):
def passed_students(self):
return self.filter(college_end_date__lt=timezone.now())
class Student(models.Model):
join_date = models.DateTimeField(auto_now_add=True)
college_duration = models.IntegerField(default=4)
college_end_date = models.DateTimeField()
objects = StudentManager()
def save(self, *args, **kwargs):
# Add logic here
if not self.college_end_date:
self.college_end_date = self.join_date + timezone.timedelta(days-self.college_duration)
return super.save(*args, **kwargs)
Now you can search it in the database.
NOTE: This sort of thing is best to do from the start on data you KNOW you're going to want to filter. If you have pre-existing data, you'll need to re-save all existing instances.
Problem
You’re attempting to query on a row that doesn’t exist in the database. Also, Django ORM doesn’t recognize a property as a field to register.
Solution
The direct answer to your question would be to create annotations, which could be subsequently queried off of. However, I would reconsider your table design for Student as it introduces unnecessary complexity and maintenance overhead.
There’s much more framework/db support for start date, end date idiosyncrasy than there is start date, timedelta.
Instead of storing duration, store end_date and calculate duration in a model method. This makes more not only makes more sense as students are generally provided a start date and estimated graduation date rather than duration, but also because it’ll make queries like these much easier.
Example
Querying which students are graduating in 2020.
Students.objects.filter(end_date__year=2020)
I have the following two models (just for a test):
class IdGeneratorModel(models.Model):
table = models.CharField(primary_key=True, unique=True,
null=False, max_length=32)
last_created_id = models.BigIntegerField(default=0, null=False,
unique=False)
#staticmethod
def get_id_for_table(table: str) -> int:
try:
last_id_set = IdGeneratorModel.objects.get(table=table)
new_id = last_id_set.last_created_id + 1
last_id_set.last_created_id = new_id
last_id_set.save()
return new_id
except IdGeneratorModel.DoesNotExist:
np = IdGeneratorModel()
np.table = table
np.save()
return IdGeneratorModel.get_id_for_table(table)
class TestDataModel(models.Model):
class Generator:
#staticmethod
def get_id():
return IdGeneratorModel.get_id_for_table('TestDataModel')
id = models.BigIntegerField(null=False, primary_key=True,
editable=False, auto_created=True,
default=Generator.get_id)
data = models.CharField(max_length=16)
Now I use the normal Django Admin site to create a new Test Data Set element. What I expected (and maybe I'm wrong here) is, that the method Generator.get_id() is called exactly one time when saving the new dataset to the database. But what really happens is, that the Generator.get_id() method is called three times:
First time when I click the "add a Test Data Set" button in the admin area
A second time shortly after that (no extra interaction from the user's side)
And a third time when finally saving the new data set
The first time could be OK: This would be the value pre-filled in a form field. Since the primary key field is not displayed in my form, this may be an unnecessary call.
The third time is also clear: It's done before saving. When it's really needed.
The code above is only an example and it is a test for me. In the real project I have to ask a remote system for an ID instead from another table model. But whenever I query that system, the delivered ID gets locked there - like the get_id_for_table() method counts up.
I'm sure there are better ways to get an ID from a method only when really needed - the method should be called exactly one time - when inserting the new dataset. Any idea how to achieve that?
Forgot the version: It's Django 1.8.5 on Python 3.4.
This is not an answer to your question, but could be a solution to your problem
I believe this issue is very complicated. Especially because you want a transaction that spans a webservice call and a database insert... What I would use in this case: generate a uuid locally. This value is practially guaranteed to be unique in the 4d world (time + location) and use that as id. Later, when the save is done, sync with your remote services.
class Record(ndb.Model):
notes = ndb.TextProperty()
last_updated = ndb.DateTimeProperty(auto_now=True)
Part of Unit Test setup:
record2 = Record()
# trying to set the last_updated timestamp to a previous date
record2.last_updated = previous_date
record2.put()
#after saving it, the timestamp is back to today's date
Hence I can't emulate an old record for my unit testing. How do I override that field without having to change the model?
From the docs
It is possible to override the value for a property with auto_now_add=True, but not for one with auto_now=True. The automatic value is not generated until the entity is written; that is, these options don't provide dynamic defaults. (These details differ from the old db API.)
I have the below db model:
from datetime import datetime
class TermPayment(models.Model):
# I have excluded fields that are irrelevant to the question
date = models.DateTimeField(default=datetime.now(), blank=True)
I add a new instance by using the below:
tp = TermPayment.objects.create(**kwargs)
My issue: all records in database have the same value in date field, which is the date of the first payment. After the server restarts, one record has the new date and the other records have the same as the first. It looks as if some data is cached, but I can't find where.
database: mysql 5.1.25
django v1.1.1
it looks like datetime.now() is being evaluated when the model is defined, and not each time you add a record.
Django has a feature to accomplish what you are trying to do already:
date = models.DateTimeField(auto_now_add=True, blank=True)
or
date = models.DateTimeField(default=datetime.now, blank=True)
The difference between the second example and what you currently have is the lack of parentheses. By passing datetime.now without the parentheses, you are passing the actual function, which will be called each time a record is added. If you pass it datetime.now(), then you are just evaluating the function and passing it the return value.
More information is available at Django's model field reference
Instead of using datetime.now you should be really using from django.utils.timezone import now
Reference:
Documentation for django.utils.timezone.now
so go for something like this:
from django.utils.timezone import now
created_date = models.DateTimeField(default=now, editable=False)
From the documentation on the django model default field:
The default value for the field. This can be a value or a callable object. If callable it will be called every time a new object is created.
Therefore following should work:
date = models.DateTimeField(default=datetime.now,blank=True)
David had the right answer. The parenthesis () makes it so that the callable timezone.now() is called every time the model is evaluated. If you remove the () from timezone.now() (or datetime.now(), if using the naive datetime object) to make it just this:
default=timezone.now
Then it will work as you expect:
New objects will receive the current date when they are created, but the date won't be overridden every time you do manage.py makemigrations/migrate.
I just encountered this. Much thanks to David.
The datetime.now() is evaluated when the class is created, not when new record is being added to the database.
To achieve what you want define this field as:
date = models.DateTimeField(auto_now_add=True)
This way the date field will be set to current date for each new record.
datetime.now() is being evaluated once, when your class is instantiated. Try removing the parenthesis so that the function datetime.now is returned and THEN evaluated. I had the same issue with setting default values for my DateTimeFields and wrote up my solution here.
From the Python language reference, under Function definitions:
Default parameter values are evaluated when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that that same “pre-computed” value is used for each call.
Fortunately, Django has a way to do what you want, if you use the auto_now argument for the DateTimeField:
date = models.DateTimeField(auto_now=True)
See the Django docs for DateTimeField.
The answer to this one is actually wrong.
Auto filling in the value (auto_now/auto_now_add isn't the same as default). The default value will actually be what the user sees if its a brand new object. What I typically do is:
date = models.DateTimeField(default=datetime.now, editable=False,)
Make sure, if your trying to represent this in an Admin page, that you list it as 'read_only' and reference the field name
read_only = 'date'
Again, I do this since my default value isn't typically editable, and Admin pages ignore non-editables unless specified otherwise. There is certainly a difference however between setting a default value and implementing the auto_add which is key here. Test it out!
In Django 3.0 auto_now_add seems to work with auto_now
reg_date=models.DateField(auto_now=True,blank=True)
if you need only DateField try this
date = models.DateField(auto_now=False, auto_now_add=False, null=True, blank=True)
if you need Both Date and Time try this
date = models.DateTimeField(auto_now_add=True, null=True, blank=True)