We Keep Coding

Django update_or_create with potentially null idenitfier

I have a function in Django that accepts a post request. If that post request includes an id, I want to update that object. If the post request sends a blank/null id, I'd like to create a new object and have Django set the ID (primary key) of the model.
Here is what I have:
def save_user_roster_to_db():
name = request.data['name']
id = request.data['id'] # this could be null/blank or be an ID
try:
user_roster = SavedUserRoster.objects.update_or_create(
id=id,
defaults={
"name": name,
}
)
return user_roster.pk
except:
raise Exception
However, this seems to try to assign the blank/null value as the ID when creating a new object. How can I tell Django to set its own ID (which is the primary key) if it's creating a new object?

You can't use update_or_create with the id field. Because in the current situation, id can have a value of None, and the Django model can't create the object with an id of None.
So I think you can try like the following.
def save_user_roster_to_db():
name = request.data['name']
id = request.data['id'] # this could be null/blank or be an ID
try:
if id:
user_roster = SavedUserRoster.objects.get(pk = id).update(name = name)
else:
user_roster = SavedUserRoster.objects.create(name = name)
return user_roster.pk
except:
raise Exception

Traditionally id is auto generated and always unique. In this case replacing None id will create exception when you create record for first time.
There are two possible options.
OPTION 1:
Create another unique_identified i.e username or email.
def save_user_roster_to_db():
name = request.data['name']
unique_identified = request.data['unique_identified'] # this could be null/blank or be an ID
user_roster, is_created = SavedUserRoster.objects.update_or_create(
unique_identified=unique_identified,
defaults={
"name": name,
}
)
return user_roster.pk
OPTION 2:
For Create:
Get the last id from database and add 1 with last id, so that It will be the next id value, It will avoid the None exception.
For Update:
It will update the existence record against id
def save_user_roster_to_db():
name = request.data['name']
id = request.data['id'] # this could be null/blank or be an ID
if id is None:
id = int(MyUserModel.objects.all().last().id)+1
user_roster, is_created = SavedUserRoster.objects.update_or_create(
id=id,
defaults={
"name": name,
}
)
return user_roster.pk

How to restrict a Element with a name getting created to the Model that already have an element with the same name? Odoo 12

a = kw.get('a') #text getting from the user input
models = request.env['htpmodel']
for model in models:
if str(a) != str(model.name):
h = model.create({
'name': a,
})
If the str of user input(a) is not in the model's name needs to be created otherwise duplicate element needs not to be created

I would suggest searching for the record first using the name you are supplied:
a = kw.get("a")
models = request.env["htpmodel"]
# search to see if a record with that name already exists
record = models.search([("name", "=", a)], limit=1)
if not record:
# the record doesn't exist
h = model.create({
"name": a,
})
A query to the model would be better practice than to loop through the results.
Let me know if you need anything clarifying,
Thanks,

How to update object returned in query

So I'm a flask/sqlalchemy newbie but this seems like it should be a pretty simple. Yet for the life of me I can't get it to work and I can't find any documentation for this anywhere online. I have a somewhat complex query I run that returns me a list of database objects.
items = db.session.query(X, func.count(Y.x_id).label('total')).filter(X.size >= size).outerjoin(Y, X.x_id == Y.x_id).group_by(X.x_id).order_by('total ASC')\
.limit(20).all()
after I get this list of items I want to loop through the list and for each item update some property on it.
for it in items:
it.some_property = 'xyz'
db.session.commit()
However what's happening is that I'm getting an error
it.some_property = 'xyz'
AttributeError: 'result' object has no attribute 'some_property'
I'm not crazy. I'm positive that the property does exist on model X which is subclassed from db.Model. Something about the query is preventing me from accessing the attributes even though I can clearly see they exist in the debugger. Any help would be appreciated.
class X(db.Model):
x_id = db.Column(db.Integer, primary_key=True)
size = db.Column(db.Integer, nullable=False)
oords = db.relationship('Oords', lazy=True, backref=db.backref('x', lazy='joined'))
def __init__(self, capacity):
self.size = size

Given your example your result objects do not have the attribute some_property, just like the exception says. (Neither do model X objects, but I hope that's just an error in the example.)
They have the explicitly labeled total as second column and the model X instance as the first column. If you mean to access a property of the X instance, access that first from the result row, either using index, or the implicit label X:
items = db.session.query(X, func.count(Y.x_id).label('total')).\
filter(X.size >= size).\
outerjoin(Y, X.x_id == Y.x_id).\
group_by(X.x_id).\
order_by('total ASC').\
limit(20).\
all()
# Unpack a result object
for x, total in items:
x.some_property = 'xyz'
# Please commit after *all* the changes.
db.session.commit()
As noted in the other answer you could use bulk operations as well, though your limit(20) will make that a lot more challenging.

You should use the update function.
Like that:
from sqlalchemy import update
stmt = update(users).where(users.c.id==5).\
values(name='user #5')
Or :
session = self.db.get_session()
session.query(Organisation).filter_by(id_organisation = organisation.id_organisation).\
update(
{
"name" : organisation.name,
"type" : organisation.type,
}, synchronize_session = False)
session.commit();
session.close()
The sqlAlchemy doc : http://docs.sqlalchemy.org/en/latest/core/dml.html

Sqlalchemy not committing object changes to postgres DB

I am using the following passage of code:
#app.route('/budget_item/<int:budget_id>/edit', methods=['GET', 'POST'])
def budget_item_edit(budget_id):
budget_item = session.query(Budget).filter_by(id=budget_id).one()
print "Start EDIT sequence"
# Return form data from HTML initial load form
elif request.method == 'POST':
budget_amount_reallocated_total = budget_item.budget_amount_reallocated_total
#ORIGINAL BUDGET
if request.form['transaction_type'] == 'Original Budget':
#amount
if request.form['amount'] == "":
amount = 0
else:
amount = float(str(request.form['amount']))
budget_item = Budget(
#created_date = "",
budget_transaction_type = request.form['transaction_type'],
budget_line = request.form['budget_line'],
amount = amount,
description = request.form['description']
#date_received = request.form['date_received']
)
try:
count = 1
while count < 10000:
count += 1
#budget_line
setattr(budget_item,'budget_line'+str(count),request.form['budget_line'+str(count)])
#amount
setattr(budget_item,'amount'+str(count),float(request.form['amount'+str(count)]))
budget_amount_reallocated_total += float(request.form['amount'+str(count)])
setattr(budget_item, 'budget_amount_reallocated_total', budget_amount_reallocated_total)
#description
setattr(budget_item,'description'+str(count), request.form['description'+str(count)])
#date_received
setattr(budget_item,'date_received'+str(count),request.form['date_received'+str(count)])
session.commit()
except:
session.commit()
return redirect(url_for('budget_master'))
else:
print "I'm done! This is not a post request"
This block of code is setup to pass data from an HTML via a POST request an then update a corresponding object in the Postgres DB. I can confirm that the object queried from the DB "budget_item" is being updated by settattr. At the end of the passage, I use commit() to update the object; however, the database doesn't reflect the changes. Just to test to make sure things are flowing, I've tried session.add(budget_item) followed by session.commit() to make sure the connect to the DB is OK. That works. How do i update this budget_item object into the database? Any help is much appreciated.

i think that a simple
budget_item.budget_amount_reallocated_total = budget_amount_reallocated_total
session.add(budget_item)
session.commit()
is the right way to do it

To answer your question, to update the budget_item that already exists in the database you need to update the Budget instance that you retrieved from the database, i.e.
budget_item = session.query(Budget).filter_by(id=budget_id).one()
not the one that you have newly created with:
budget_item = Budget(...)
Here the first budget_item represents the row in the database, so this is the one to update. To that end you can replace the code that creates the second Budget instance with this:
budget_item.budget_transaction_type = request.form['transaction_type']
budget_item.budget_line = request.form['budget_line']
budget_item.amount = amount
budget_item.description = request.form['description']
Once you have finished updating the Budget instance you can call session.commit() to flush it to the database.
As mentioned in my comment to your question, it appears that you are trying to add a large number of additional attributes to budget_item all of which will be ignored by sqlalchemy unless they are defined in the mapping between the Budget instance and the Budget table.

What is an efficient way of inserting thousands of records into an SQLite table using Django?

I have to insert 8000+ records into a SQLite database using Django's ORM. This operation needs to be run as a cronjob about once per minute.
At the moment I'm using a for loop to iterate through all the items and then insert them one by one.
Example:
for item in items:
entry = Entry(a1=item.a1, a2=item.a2)
entry.save()
What is an efficient way of doing this?
Edit: A little comparison between the two insertion methods.
Without commit_manually decorator (11245 records):
nox#noxdevel marinetraffic]$ time python manage.py insrec
real 1m50.288s
user 0m6.710s
sys 0m23.445s
Using commit_manually decorator (11245 records):
[nox#noxdevel marinetraffic]$ time python manage.py insrec
real 0m18.464s
user 0m5.433s
sys 0m10.163s
Note: The test script also does some other operations besides inserting into the database (downloads a ZIP file, extracts an XML file from the ZIP archive, parses the XML file) so the time needed for execution does not necessarily represent the time needed to insert the records.

You want to check out django.db.transaction.commit_manually.
http://docs.djangoproject.com/en/dev/topics/db/transactions/#django-db-transaction-commit-manually
So it would be something like:
from django.db import transaction
#transaction.commit_manually
def viewfunc(request):
...
for item in items:
entry = Entry(a1=item.a1, a2=item.a2)
entry.save()
transaction.commit()
Which will only commit once, instead at each save().
In django 1.3 context managers were introduced.
So now you can use transaction.commit_on_success() in a similar way:
from django.db import transaction
def viewfunc(request):
...
with transaction.commit_on_success():
for item in items:
entry = Entry(a1=item.a1, a2=item.a2)
entry.save()
In django 1.4, bulk_create was added, allowing you to create lists of your model objects and then commit them all at once.
NOTE the save method will not be called when using bulk create.
>>> Entry.objects.bulk_create([
... Entry(headline="Django 1.0 Released"),
... Entry(headline="Django 1.1 Announced"),
... Entry(headline="Breaking: Django is awesome")
... ])
In django 1.6, transaction.atomic was introduced, intended to replace now legacy functions commit_on_success and commit_manually.
from the django documentation on atomic:
atomic is usable both as a decorator:
from django.db import transaction
#transaction.atomic
def viewfunc(request):
# This code executes inside a transaction.
do_stuff()
and as a context manager:
from django.db import transaction
def viewfunc(request):
# This code executes in autocommit mode (Django's default).
do_stuff()
with transaction.atomic():
# This code executes inside a transaction.
do_more_stuff()

Bulk creation is available in Django 1.4:
https://django.readthedocs.io/en/1.4/ref/models/querysets.html#bulk-create

Have a look at this. It's meant for use out-of-the-box with MySQL only, but there are pointers on what to do for other databases.

You might be better off bulk-loading the items - prepare a file and use a bulk load tool. This will be vastly more efficient than 8000 individual inserts.

To answer the question particularly with regard to SQLite, as asked, while I have just now confirmed that bulk_create does provide a tremendous speedup there is a limitation with SQLite: "The default is to create all objects in one batch, except for SQLite where the default is such that at maximum 999 variables per query is used."
The quoted stuff is from the docs--- A-IV provided a link.
What I have to add is that this djangosnippets entry by alpar also seems to be working for me. It's a little wrapper that breaks the big batch that you want to process into smaller batches, managing the 999 variables limit.

You should check out DSE. I wrote DSE to solve these kinds of problems ( massive insert or updates ). Using the django orm is a dead-end, you got to do it in plain SQL and DSE takes care of much of that for you.
Thomas

def order(request):
if request.method=="GET":
cust_name = request.GET.get('cust_name', '')
cust_cont = request.GET.get('cust_cont', '')
pincode = request.GET.get('pincode', '')
city_name = request.GET.get('city_name', '')
state = request.GET.get('state', '')
contry = request.GET.get('contry', '')
gender = request.GET.get('gender', '')
paid_amt = request.GET.get('paid_amt', '')
due_amt = request.GET.get('due_amt', '')
order_date = request.GET.get('order_date', '')
print(order_date)
prod_name = request.GET.getlist('prod_name[]', '')
prod_qty = request.GET.getlist('prod_qty[]', '')
prod_price = request.GET.getlist('prod_price[]', '')
print(prod_name)
print(prod_qty)
print(prod_price)
# insert customer information into customer table
try:
# Insert Data into customer table
cust_tab = Customer(customer_name=cust_name, customer_contact=cust_cont, gender=gender, city_name=city_name, pincode=pincode, state_name=state, contry_name=contry)
cust_tab.save()
# Retrive Id from customer table
custo_id = Customer.objects.values_list('customer_id').last() #It is return
Tuple as result from Queryset
custo_id = int(custo_id[0]) #It is convert the Tuple in INT
# Insert Data into Order table
order_tab = Orders(order_date=order_date, paid_amt=paid_amt, due_amt=due_amt, customer_id=custo_id)
order_tab.save()
# Insert Data into Products table
# insert multiple data at a one time from djanog using while loop
i=0
while(i<len(prod_name)):
p_n = prod_name[i]
p_q = prod_qty[i]
p_p = prod_price[i]
# this is checking the variable, if variable is null so fill the varable value in database
if p_n != "" and p_q != "" and p_p != "":
prod_tab = Products(product_name=p_n, product_qty=p_q, product_price=p_p, customer_id=custo_id)
prod_tab.save()
i=i+1

I recommend using plain SQL (not ORM) you can insert multiple rows with a single insert:
insert into A select from B;
The select from B portion of your sql could be as complicated as you want it to get as long as the results match the columns in table A and there are no constraint conflicts.

def order(request):
if request.method=="GET":
# get the value from html page
cust_name = request.GET.get('cust_name', '')
cust_cont = request.GET.get('cust_cont', '')
pincode = request.GET.get('pincode', '')
city_name = request.GET.get('city_name', '')
state = request.GET.get('state', '')
contry = request.GET.get('contry', '')
gender = request.GET.get('gender', '')
paid_amt = request.GET.get('paid_amt', '')
due_amt = request.GET.get('due_amt', '')
order_date = request.GET.get('order_date', '')
prod_name = request.GET.getlist('prod_name[]', '')
prod_qty = request.GET.getlist('prod_qty[]', '')
prod_price = request.GET.getlist('prod_price[]', '')
# insert customer information into customer table
try:
# Insert Data into customer table
cust_tab = Customer(customer_name=cust_name, customer_contact=cust_cont, gender=gender, city_name=city_name, pincode=pincode, state_name=state, contry_name=contry)
cust_tab.save()
# Retrive Id from customer table
custo_id = Customer.objects.values_list('customer_id').last() #It is return Tuple as result from Queryset
custo_id = int(custo_id[0]) #It is convert the Tuple in INT
# Insert Data into Order table
order_tab = Orders(order_date=order_date, paid_amt=paid_amt, due_amt=due_amt, customer_id=custo_id)
order_tab.save()
# Insert Data into Products table
# insert multiple data at a one time from djanog using while loop
i=0
while(i<len(prod_name)):
p_n = prod_name[i]
p_q = prod_qty[i]
p_p = prod_price[i]
# this is checking the variable, if variable is null so fill the varable value in database
if p_n != "" and p_q != "" and p_p != "":
prod_tab = Products(product_name=p_n, product_qty=p_q, product_price=p_p, customer_id=custo_id)
prod_tab.save()
i=i+1
return HttpResponse('Your Record Has been Saved')
except Exception as e:
return HttpResponse(e)
return render(request, 'invoice_system/order.html')