I've an personal ecommerce site.
I'm using the ID of the model as the Order Number. Just because it seemed logic, and I was expecting ID would increment just by 1 everytime.
However, I'm noticing that the ID of my Orders (of my Order model) had jumped twice:
a) From 54 to 86 (32 of difference).
b) From 99 to 132 (33 of difference).
Don't know why, don't know if I should use a custom field instead of the models ID.
I'm using Django 3.0 and hosting my project on Heroku.
models.py:
class Order(models.Model):
ORDER_STATUS = (
('recibido_pagado', 'Recibido y pagado'),
('recibido_no_pagado', 'Recibido pero no pagado'),
('en_proceso', 'En proceso'),
('en_camino', 'En camino'),
('entregado', 'Entregado'),
('cancelado', 'Cancelado por no pagar' )
)
token = models.CharField(max_length=100, blank=True, null=True)
first_name = models.CharField(max_length=50, blank=True, null=True)
last_name = models.CharField(max_length=50, blank=True, null=True)
phone_number = models.CharField(max_length=30, blank=True)
total = models.DecimalField(max_digits=10, decimal_places=2)
stickers_price = models.DecimalField(max_digits=10, decimal_places=2)
discount = models.DecimalField(max_digits=10, decimal_places=2, default=Decimal('0.00'))
shipping_cost = models.DecimalField(max_digits=10, decimal_places=2)
email = models.EmailField(max_length=250, blank = True, verbose_name= 'Correo electrónico')
last_four = models.CharField(max_length=100, blank=True, null=True)
created = models.DateTimeField(auto_now_add=True)
shipping_address = models.CharField(max_length=100, blank=True, null=True)
shipping_address1 = models.CharField(max_length=100, blank=True, null=True)
reference = models.CharField(max_length=100, blank=True, null=True)
shipping_department = models.CharField(max_length=100, blank=True, null=True)
shipping_province = models.CharField(max_length=100, blank=True, null=True)
shipping_district = models.CharField(max_length=100, blank=True, null=True)
reason = models.CharField(max_length=400, blank=True, null=True, default='')
status = models.CharField(max_length=20, choices=ORDER_STATUS, default='recibido_pagado')
comments = models.CharField(max_length=400, blank=True, null=True, default='')
cupon = models.ForeignKey('marketing.Cupons', blank=True, null=True, default=None, on_delete=models.SET_NULL)
class Meta:
db_table = 'Order'
ordering = ['-created']
def __str__(self):
return str(self.id)
def igv(self):
igv = int(self.total) * 18/100
return igv
def shipping_date(self):
shipping_date = self.created + datetime.timedelta(days=10)
return shipping_date
def deposit_payment_date(self):
deposit_payment_date = self.created + datetime.timedelta(days=2)
return
View that creates the order:
#csrf_exempt
def cart_charge_deposit_payment(request):
amount = request.POST.get('amount')
email = request.user.email
shipping_address = request.POST.get('shipping_address')
shipping_cost = request.POST.get('shipping_cost')
discount = request.POST.get('discount')
stickers_price = request.POST.get('stickers_price')
comments = request.POST.get('comments')
last_four = 1111
transaction_amount = amount
first_name = request.user.first_name
last_name = request.user.last_name
phone_number = request.user.profile.phone_number
current_time = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
shipping_address1 = request.user.profile.shipping_address1
reference = request.user.profile.reference
shipping_department = request.user.profile.shipping_department
shipping_province = request.user.profile.shipping_province
shipping_district = request.user.profile.shipping_district
order_details = Order.objects.create(
token='Random',
first_name=first_name,
last_name=last_name,
phone_number=phone_number,
email=email, # Using email entered in Culqi module, NOT user.email. Could be diff.
total=transaction_amount,
stickers_price = stickers_price,
discount = discount,
shipping_cost=shipping_cost,
last_four=last_four,
created=current_time,
shipping_address=shipping_address,
shipping_address1=shipping_address1,
reference=reference,
shipping_department=shipping_department,
shipping_province=shipping_province,
shipping_district=shipping_district,
status='recibido_no_pagado',
cupon=cupon,
comments=comments
)
...
If you need consecutive numbering without holes you should not use Django's autogenerated id field as your order number.
In order to guarantee uniqueness even under concurrency Django creates a database sequence which is an object in the database that produces a new value each time it is consulted. Note that the sequence consumes the value produced even if it is not saved to the database anywhere.
What happens then is that whenever you try to create an instance and this operation fails at the database level, a number from the sequence is consumed anyway. So let's say you create your first Order successfully, it will have the ID number 1. Then let's say that you try to create a second Order, but the INSERT in the database fails (for example for some integrity check, or whatever). Afterwards you successfully create a third Order, you would expect that this order has the ID number 2, but it will actually have ID number 3, because the number 2 was consumed from the sequence even if it was not saved.
So no, you cannot use the id if you need to ensure there are no holes in your order numbers.
Now in order to have consecutive numeration you could simply add a column
order_number = models.PositiveIntegerField(unique=True, null=True)
question is how to properly set its value. So in an ideal world where there is no concurrency (two processes running queries against the same database) you could simply get the maximum order number so far, add 1 and then save this value into order_number. Thing is if you do this naively you will end up having duplicates (actually integrity errors, because unique=True will prevent duplicates).
One way to solve this would be to lock your table (see this SO question) while you compute and update your order number.
As I assume you don't care that the order number faithfully reflects the order in which orders where created but only that it is sequential and without holes what you can do is to run a query like the following inside a transaction (assuming your Order model lives inside an orders django app):
UPDATE orders_order SET order_number = (SELECT COALESCE(MAX(order_number), 0) FROM orders_order) + 1 WHERE id = [yourid] AND order_number IS NULL
Now even with this query you could have concurrency issues, since Django uses postgres default isolation level by default. So in order to make this query safe you will need to change isolation level. Refer to this SO question for a way on having two separate connections with two different isolation levels. What you need to make this query safe is to set the isolation level to SERIALIZABLE.
Assuming you were able to solve the isolation level issue then is the thing on how to run this query
from django.db import connections, transaction
with transaction.atomic(using='your_isolated_db_alias'):
with connections['your_isolated_db_alias'].cursor() as cursor:
cursor.execute('UPDATE orders_order SET order_number = (SELECT COALESCE(MAX(order_number), 0) FROM orders_order) + 1 WHERE id = %s AND order_number IS NULL', order.id)
The snippet above assumes you have the order for which you want to set the order number in a variable called order. If your isolation is right then you should be safe.
Now there is a third alternative which is to use select_for_update() as a table locking mechanism (although it is not intended for that but for row level locking). So the idea is simple, in the same way as before you first create your order and then update it to set the order number. So in order to guarantee that you won't end up with duplicate (aka IntegrityError) order numbers what you do is issue a query that selects all the Orders in your DB and then use select_for_update() in the following way:
from django.db import transaction
with transaction.atomic():
# This locks every row in orders_order until the end of the transaction
Order.objects.all().select_for_update() # pointless query just to lock the table
max_on = Order.objects.aggregate(max_on=Max('order_number'))['max_on']
Order.objects.filter(id=order.id).update(order_number=max_on + 1)
As long as you are sure that you have at least 1 order before entering the code block above AND that you always do the full select_for_update() first, then you should also be safe.
And these are the ways I can think of how to solve the consecutive numbering. I'd love to see an out of the box solution for this, but unfortunately I do not know any.
This will not answer your question directly, but still might be useful for you or somebody with a similar problem.
From the data integrity point of view, deleting potentially useful data such as customer order in production can be a really bad idea. Even if you don't need this data at the moment, you may come to a point in future when you want to analyze all of your orders, even failed / cancelled ones.
What I would suggest here, is to ensure that deleting not so important related models doesn't cause deleting orders. You can easily achieve this by passing PROTECT argument to your ForeignKey field. This will raise ProtectedError when trying to delete related model. Another useful options are SET_NULL and SET_DEFAULT whose names speak for themselves.
By following this approach, you will never need to worry about the broken id counter.
Let's leave Django, Python.
That is DB topic. Say - you start transaction, with new row in particular table. That means new ID. If you commit that amount of work - new ID is visible. If rollback happens ID is lost. From DB perspective there is no way to reuse that number.
Be aware that select max(id) + 1 is bad practice - what if two transactions do that at the same time?
Other option is lock. I can see 3 solutions here:
Lock all rows in the table - that means - your insert time depends on table size :)
As a side note. If you go one by one to lock, be sure to sort all rows in the table to be sure there is no deadlock. Say you use Postgres, edit means row can be moved at the end... so order depends on what is going on with the data. If so two transactions can lock rows in different order, and deadlock is a matter of time. During tests, under low load - everything goes just fine...
Lock whole table. Better, since not depends on rows, but you block against edits as well.
Separate table for generators - each generator has row in that table - you lock that row, take next value, at the end of transaction row is released.
To all points. That means - you need short transactions. In web apps that is general rule. Just be sure create order is light, and most heavy things are performed as separate transaction. Why? Lock is released at the end of transaction.
Hope it explains the case.
In Django. Let's create model:
class Custom_seq(models.Model):
name = models.CharField(max_length=100, blank=False, null=False)
last_number = models.IntegerField(default=0)
Query for next id:
seq = Custom_seq.objects.filter(name='order sequence').select_for_update(no_wait=False).first()
new_order_id = seq.last_number + 1
seq.last_number = new_order_id
seq.save()
Why it works? Please note that at one time you are creating one order. It can be committed - so used, or rolled back - cancelled... both cases are supported.
It is database internal behavior: https://www.postgresql.org/docs/current/functions-sequence.html
Important
To avoid blocking concurrent transactions that obtain numbers from the
same sequence, a nextval operation is never rolled back; that is, once
a value has been fetched it is considered used and will not be
returned again. This is true even if the surrounding transaction later
aborts, or if the calling query ends up not using the value. For
example an INSERT with an ON CONFLICT clause will compute the
to-be-inserted tuple, including doing any required nextval calls,
before detecting any conflict that would cause it to follow the ON
CONFLICT rule instead. Such cases will leave unused “holes” in the
sequence of assigned values. Thus, PostgreSQL sequence objects cannot
be used to obtain “gapless” sequences.
I would like to set the "order" IntegerField of my Achievement model to the current count of objects in Achievement. The order field is used to order achievements, and users can change it. For now I have 1 as default.
class Achievement(models.Model):
title = models.CharField(max_length=50, blank=True)
description = models.TextField()
order = models.IntegerField(default=1) #Get the number of achievement objects
class Meta:
db_table = 'achievement'
ordering = ['order', 'id']
For example, if I already have one achievement in my database with whatever order the next one should get order=2 by default.
As far as I understood, you want to have a default value of 1 to the order integer field and increment it with each entry of Achievment (same functionality as the id), but also allow users change it.
For this purpose you can use Django's AutoField:
An IntegerField that automatically increments according to available IDs. You usually won’t need to use this directly; a primary key field will automatically be added to your model if you don’t specify otherwise.
Like this:
class Achievement(models.Model):
...
order = models.AutoField(default=1, primary_key=False)
# Also specify that this autofield is *not* a ^ primary key
class Meta:
ordering = ['order', 'id']
I have the following model as an inline field in another model:
class route_ordering(models.Model):
route_id = models.ForeignKey(route, on_delete=models.CASCADE)
activity_id = models.ForeignKey(activity, on_delete=models.CASCADE)
day = models.IntegerField()
order = models.IntegerField()
And in the admin.py:
class RouteAdmin(admin.ModelAdmin):
inlines = (RouteOrderingInline,)
I would like to make "order" self-incrementing from one, so it will be auto filled when I go to the Django admin panel (in the first line order=1 , then order-2 etc.)
I know you can use Default to set an autofilled value, but I want it to increment by itself.
How can I do this?
I can't guarantee this will work, and I'm not able to test out right now, but I think you can pass in a generator to your default value arg.
define increment():
return range(1, 1000)
And wherever your passing in a default just call next(increment())
Sorry I can't provide a more detailed example, I'm writing this on my phone XD, but I think that should work.
I was programming a function in python that required a group of objects of a model to be ordered by a specific parameter (the codename of a problem), this parameter resulted to be a string. When I sorted it inside the Problems table, it worked correctly, however, when I sorted it inside ProblemUser model (using order_by) the sort was completely incorrect respecting the real results.
I managed to work it around by sorting the elements AFTER i got it from the model using sorted() function, however, i am still with the doubt on how to sort a foreign key in the order_by function.
class Problem(models.Model):
problem_name = models.CharField(max_length=256,default='Unnamed')
source = models.CharField(max_length=200)
url = models.CharField(max_length=200)
codename = models.CharField(max_length=200, primary_key=True)
difficulty = models.CharField(max_length=32,default='UNRATED')
class ProblemUser(models.Model):
problem = models.ForeignKey(Problem)
handler = models.ForeignKey(Handler)
solved = models.BooleanField(default=True)
voted = models.BooleanField(default=False)
entry = models.DateField()
t = ProblemUser.objects.filter(solved=1, handler=Handler.objects.get(id=1)).order_by('problem')
t[0].problem.codename < t[42].problem.codename
False
t[1].problem.codename < t[42].problem.codename
True
I also tried order_by('problem__codename')
Some codenames examples are (these are outputs that i remember to have seen when i ordered by the foreign key):
S_1_T
S_1000_E
S_1001_F
.
.
.
S_2_P
Thank you for your time and help! :).
try it like that please.
t = ProblemUser.objects.filter(solved=1, handler=Handler.objects.get(id=1)).order_by('problem__codename')
When getting members based on Unit, I only want to get the ones who are actually in that unit as of now.
I've got a model looking like this:
class Member(models.Model):
name = models.CharField(max_length=256)
unit = models.ManyToManyField(Unit, through='Membership')
class Membership(models.Model):
member = models.ForeignKey(Member)
unit = models.ForeignKey(Unit)
start = models.DateField(default=date.today)
stop = models.DateField(blank=True, null=True)
class Unit(models.Model):
name = models.CharField(max_length=256)
As you can see, members can have a "fake" membership in unit, that is only history and should not be considered in the searches and listings of the admin. They should be shown in the change-page for a single object though.
The admin looks like this:
class MembershipInline(admin.TabularInline):
model = Membership
extra = 1
class MemberAdmin(admin.ModelAdmin):
list_filter = ('unit',)
inlines = [MembershipInline,]
So how can I (if at all possible this way), when filtering on unit only get those units whose membership__stop__isnull=True?
I tried Managers, I can make them work on the model in the admin itself, but not on the filtering/searches. There is also a def queryset(self) method that is overrideable, but I can't wrap my head around how to use it to fix my problem.
Edit, how this is used: A member has only one membership in a unit, however, they could be members from before, but they are ended (with stop). So I only want to filter (and show, in the list view) those members who have an open-ended membership (like, that they are members of that unit now).
Any ideas?
So you're trying to get the members of a specific Unit, right?
unit = Unit.objects.select_related().get(id=some_id)
This will pull the unit out of the database for you, along with the Memberships and Users that belong to it. You can access and filter the users by:
for member in unit.membership__set.filter(stop__isnull=True):
print member.name
I hope this helps? I may be wrong, I haven't tested this.
One way to certainly achieve this is by adding a denormalized field for
has_open_ended_membership.
To do this just add a BooleaneField like that to the Member and make sure it's consistent.
From the django documentation this seems to be the only way without writing specialized code in the ModelAdmin object:
Set list_filter to activate filters in
the right sidebar of the change list
page of the admin. This should be a
list of field names, and each
specified field should be either a
BooleanField, CharField, DateField,
DateTimeField, IntegerField or
ForeignKey.
I'm curious about other approaches - list_filter certainly is limited.
I fixed it with putting in a denormalized field in member, with a foreign-key to the active unit. Then, to make it work and be automatically updated in the admin, I made the specialized save-function for Membership.
class Member(models.Model):
name = models.CharField(max_length=256)
unit = models.ManyToManyField(Unit, through='Membership')
unit_denorm = models.ForeignKey(Unit)
class Membership(models.Model):
member = models.ForeignKey(Member)
unit = models.ForeignKey(Unit)
start = models.DateField(default=date.today)
stop = models.DateField(blank=True, null=True)
def save(self, *args, **kwargs):
if not self.stop:
self.member.unit_denorm = self.unit
self.member.save()
super(Membership, self).save(*args, **kwargs)
class Unit(models.Model):
name = models.CharField(max_length=256)
And with list_filter = ('unit_denorm',) in the admin, it does exactly what I want.
Great! Of course, there should only be one field with stop__isnull=True. I haven't figured out how to make that restriction. but people using the system know they shouldn't do that anyway.