We Keep Coding

Extract only the data from a Django Query Set

I am working to learn Django, and have built a test database to work with. I have a table that provides basic vendor invoice information, so, and I want to simply present a user with the total value of invoices that have been loaded to into the database. I found that the following queryset does sum the column as I'd hoped:
total_outstanding: object = Invoice.objects.aggregate(Sum('invoice_amount'))
but the result is presented on the page in the following unhelpful way:
Total $ Outstanding: {'invoice_amount__sum': Decimal('1965')}
The 1965 is the correct total for the invoices that I populated the database with, so the queryset is pulling what I want it to, but I just want to present that portion of the result to the user, without the other stuff.
Someone else asked a similar question (basically the same) here: how-to-extract-data-from-django-queryset, but the answer makes no sense to me, it is just:
k = k[0] = {'name': 'John'}
Queryset is list .
Can anyone help me with a plain-English explanation of how I can extract just the numerical result of that query for presentation to a user?

What you here get is a dictionary that maps the name of the aggregate to the corresponding value. You can use subscripting to obtain the corresponding value:
object = Invoice.objects.aggregate(
Sum('in,voice_amount')
)['invoice_amount__sum']

Python-Eve: Prevent inserting duplicates without using unique fields

I am trying to prevent inserting duplicate documents by the following approach:
Get a list of all documents from the desired endpoint which will contain all the documents in JSON-format. This list is called available_docs.
Use a pre_POST_<endpoint> hook in order to handle the request before inserting to the data. I am not using the on_insert hook since I need to do this before validation.
Since we can access the request object use request.json to get the payload JSON-formatted
Check if request.json is already contained in available_docs
Insert new document if it's not a duplicate only, abort otherwise.
Using this approach I got the following snippet:
def check_duplicate(request):
if not request.json in available_sims:
print('Not a duplicate')
else:
print('Duplicate')
flask.abort(422, description='Document is a duplicate and already in database.')
The available_docs list looks like this:
available_docs = [{'foo': ObjectId('565e12c58b724d7884cd02bb'), 'bar': [ObjectId('565e12c58b724d7884cd02b9'), ObjectId('565e12c58b724d7884cd02ba')]}]
The payload request.json looks like this:
{'foo': '565e12c58b724d7884cd02bb', 'bar': ['565e12c58b724d7884cd02b9', '565e12c58b724d7884cd02ba']}
As you can see, the only difference between the document which was passed to the API and the document already stored in the DB is the datatype of the IDs. Due to that fact, the if-statement in my above snippet evaluates to True and judges the document to be inserted not being a duplicate whereas it definitely is a duplicate.
Is there a way to check if a passed document is already in the database? I am not able to use unique fields since the combination of all document fields needs to be unique only. There is an unique identifier (which I left out in this example), but this is not suitable for the desired comparison since it is kind of a time stamp.
I think something like casting the given IDs at the keys foo and bar as ObjectIDs would do the trick, but I do not know how to to this since I do not know where to get the datatype ObjectID from.

You approach would be much slower than setting a unique rule for the field.
Since, from your example, you are going to compare objectids, can't you simply use those as the _id field for the collection? In Mongo (and Eve of course) that field is unique by default. Actually, you typically don't even define it. You would not need to do anything at all, as a POST of a document with an already existing id would fail right away.
If you can't go that way (maybe you need to compare a different objectid field and still, for some reason, you can't simply set a unique rule for the field), I would look at querying the db for the field value instead than getting all the documents from the db and then scanning them sequentially in code. Something like db.find({db_field: new_document_field_value}). If that returns true, new document is a duplicate. Make sure db_field is indexed (which usually holds true also for fields tagged with unique rule)
EDIT after the comments. A trivial implementation would probable be something like this:
def pre_POST_callback(resource, request):
# retrieve mongodb collection using eve connection
docs = app.data.driver.db['docs']
if docs.find_one({'foo': <value>}):
flask.abort(422, description='Document is a duplicate and already in database.')
app = Eve()
app.run()

Here's my approach on preventing duplicate records:
def on_insert_subscription(items):
c_subscription = app.data.driver.db['subscription']
user = decode_token()
if user:
for item in items:
if c_subscription.find_one({
'topic': ObjectId(item['topic']),
'client': ObjectId(user['user_id'])
}):
abort(422, description="Client already subscribed to this topic")
else:
item['client'] = ObjectId(user['user_id'])
else:
abort(401, description='Please provide proper credentials')
What I'm doing here is creating subscriptions for clients. If a client is already subscribed to a topic I throw 422.
Note: the client ID is decoded from the JWT token.

How to find not null variables from URL query string? or can we get the variable in URL query string from request in python

I want to filter data based on multiple fields in Django python. The scenario is, if a GET request to webserver comes as /searchStudent/?firstName=&lastName=xyzage=23&class=&city= and we dont know what can be the possible parameter in query string, some of them can come with value and some variable doesnot comes with value. The question is, is there any to get the variable which have only value or which dont have values from request. I know we can simply getting value by using request.GET.get('city') but I am finding way for getting the non null variable from query string, is there any way to find the non value variable from query string? In above scenario city, class, and firstName doesn't have values and I dont want to add it on filter. what will be the right approach? please suggest the right way.

To get a dict of non-empty query parameters use this code:
non_empty_params = dict((field, value)
for field, value in request.GET.iteritems() if value)
But do not build queryset this way. You should never ever trust data from the user input. So you have to have a list of fields to search and use this list to filter out incorrect field names:
fields = ('firstName', 'lastName', 'age', 'class', 'city', )
filter_params = dict((field, value)
for field, value in request.GET.iteritems()
if value and field in fields)
students = Student.objects.filter(**filter_params)

I did this way and I get answer for filtering dynamically
filter_args={}
if request.GET.get('firstName') :
filter_args['firstName']=request.GET.get('firstName')
if request.GET.get('lastName') :
filter_args['lastName']=request.GET.get('lastName')
if request.GET.get('city') :
filter_args['city']=request.GET.get('city')
Student.object.filter(**filter_args)
This is know as Dynamic Queries, I was not aware with this. Thanks #catavaran, for suggesting concept. also refered this link Dynamic Django Queries

unique python/django issue with variable column/field name in form

I have quite unique problem with django.
Im providing website users interface for editing large data. Each row on this data represents a row in database. Or one object of certain Type.
Users click on cells in the table and form opens where they can edit this fields/column value.
In essence it works like this:
1) based on where user clicks, query is sent to server containting object id and the field that he is editing.
2) based on this information form is created on the fly:
class FieldEditorForm(forms.ModelForm):
class Meta:
model = MyObject
fields = ['id', field ]
Notice the field there is Variable not name of the field.
3) this field passes its own modelform validation and all is fine. in save method Model.save() is enough to update the value.
But now to the problem. Sometimes empty value is sent to server in this form. Empy value such as u'' or almost emtpty like u' '. I want to repace this with None so NULL would be saved to database.
There are two places where i could do that. In field validation modifying the cleaned_data or in form save method.
Both approaches raise unique problem as i dont know how to create variable function names.
def clean_%(field)s():
or in case of form save method
r.%(field)s = None
is what i need, but those methods dont work. So how can i create method name which is variable or set objects variable parameter to something. Is it even possible or do i have to rethink my approach there?
Alan

In the latter case, setattr(r, field + 's', None).

Mongoengine integer id, and User creating

MongoDB is using string(hash) _id field instead of integer; so, how to get classic id primary key? Increment some variable each time I create my class instance?
class Post(Document):
authors_id = ListField(IntField(required=True), required=True)
content = StringField(max_length=100000, required=True)
id = IntField(required=True, primary_key=True)
def __init__(self):
//what next?
Trying to create new user raises exception:
mongoengine.queryset.OperationError: Tried to save duplicate unique keys
(E11000 duplicate key error index: test.user.$_types_1_username_1
dup key: { : "User", : "admin" })
Code:
user = User.create_user(username='admin', email='example#mail.com',
password='pass')
user.is_superuser = True
user.save()
Why?

There is the SequenceField which you could use to provide this. But as stated incrementing id's dont scale well and are they really needed? Can't you use ObjectId or a slug instead?

If you want to use an incrementing integer ID, the method to do it is described here:
http://www.mongodb.org/display/DOCS/How+to+Make+an+Auto+Incrementing+Field
This won't scale for a vary large DB/app but it works well for small or moderate application.

1) If you really want to do it you have to override the mongoengine method saving your documents, to make it look for one document with the highest value for your id and save the document using that id+1. This will create overhead (and one additional read every write), therefore I discourage you to follow this path. You could also have issues of duplicated IDs (if you save two records at the exactly same time, you'll read twice the last id - say 1 and save twice the id 1+1 = 2 => that's really bad - to avoid this issue you'd need to lock the entire collection at every insert, by losing performances).
2) Simply you can't save more than one user with the same username (as the error message is telling you) - and you already have a user called "admin".