I have a simple Django application were I try to aggregate multiple values into an annotation for easier processing on the client side.
Basically, I need to sum the values of multiple columns into one.
For this I'm trying to use annotate with F functions:
qs = TimeReport.objects \
.filter(year=year, term=term) \
.annotate(
created_by_first_name=F('created_by__first_name'),
created_by_last_name=F('created_by__last_name'),
total_hours = F('master_thesis_supervision_hours')
+ F('semester_project_supervision_hours')
+ F('other_job_hours')
+ F('MAN_hours')
+ F('exam_proctoring_and_grading_hours')
+ F('class_teaching_exam_hours')
+ F('class_teaching_practical_work_hours')
+ F('class_teaching_preparation_hours')
+ F('class_teaching_teaching_hours'),
) \
.all().values()
Suprisingly, when I inspect the content of the calculated field, it does not contain anything:
list(qs)[0]['total_hours']
None
Trying to cast the result of the F function does not either:
...
total_hours = int(F('master_thesis_supervision_hours'))
+int(F('semester_project_supervision_hours'))
+ ...
I also tried to update the models.py to add a property:
#property
def total_hours(self):
return self.master_thesis_supervision_hours + self.class_teaching_total_hours + self.semester_project_supervision_hours + self.other_job_hours + self.MAN_hours + self.exam_proctoring_and_grading_hours
and update the views.py accordingly:
qs = TimeReport.objects \
.filter(year=year, term=term) \
.annotate(
created_by_first_name=F('created_by__first_name'),
created_by_last_name=F('created_by__last_name'),
total_hours = F('total_hours'),
) \
.all().values()
But I get the following error:
django.core.exceptions.FieldError: Cannot resolve keyword 'total_hours' into field.
What would be the correct way to do this?
Related
I am trying to get buy and sell orders from binance api(python-binance) which has a limit of 500 values.(500 ask,500 buy). I already can get this with creating 500 variables with index numbers but it seems to me there has to be a better way than to write 500 lines of code.
This is the code I am trying to make it happen.
#!/usr/bin/python
# -*- coding: utf-8 -*-
from binance.client import Client
user_key = ''
secret_key = ''
binance_client = Client(user_key, secret_key)
while True:
alis = binance_client.futures_order_book(symbol='XRPUSDT')
binance_buy = alis['bids'][0]
binance_buy1 = alis['bids'][1]
binance_buy2 = alis['bids'][2]
binance_buy3 = alis['bids'][3]
binance_buy4 = alis['bids'][4]
binance_buy5 = alis['bids'][5]
binance_buy6 = alis['bids'][6]
binance_buy7 = alis['bids'][7]
binance_buy8 = alis['bids'][8]
binance_buy9 = alis['bids'][9]
binance_buy10 = alis['bids'][10]
binance_buy11 = alis['bids'][11]
binance_buy12 = alis['bids'][12]
binance_buy13 = alis['bids'][13]
binance_buy14 = alis['bids'][14]
binance_buy15 = alis['bids'][15]
binance_buy16 = alis['bids'][16]
binance_buy17 = alis['bids'][17]
binance_buy18 = alis['bids'][18]
binance_buy19 = alis['bids'][19]
binance_buy20 = alis['bids'][20]
binance_sell = alis['asks'][0]
binance_sell1 = alis['asks'][1]
binance_sell2 = alis['asks'][2]
binance_sell3 = alis['asks'][3]
binance_sell4 = alis['asks'][4]
binance_sell5 = alis['asks'][5]
binance_sell6 = alis['asks'][6]
binance_sell7 = alis['asks'][7]
binance_sell8 = alis['asks'][8]
binance_sell9 = alis['asks'][9]
binance_sell10 = alis['asks'][10]
binance_sell11 = alis['asks'][11]
binance_sell12 = alis['asks'][12]
binance_sell13 = alis['asks'][13]
binance_sell14 = alis['asks'][14]
binance_sell15 = alis['asks'][15]
binance_sell16 = alis['asks'][16]
binance_sell17 = alis['asks'][17]
binance_sell18 = alis['asks'][18]
binance_sell19 = alis['asks'][19]
binance_sell20 = alis['asks'][20]
binance_buy_demand = float(binance_buy[1]) + float(binance_buy1[1]) \
+ float(binance_buy2[1]) + float(binance_buy3[1]) \
+ float(binance_buy4[1]) + float(binance_buy5[1]) \
+ float(binance_buy6[1]) + float(binance_buy7[1]) \
+ float(binance_buy8[1]) + float(binance_buy9[1]) \
+ float(binance_buy10[1]) + float(binance_buy11[1]) \
+ float(binance_buy12[1]) + float(binance_buy13[1]) \
+ float(binance_buy14[1]) + float(binance_buy15[1]) \
+ float(binance_buy16[1]) + float(binance_buy17[1]) \
+ float(binance_buy18[1]) + float(binance_buy19[1]) \
+ float(binance_buy20[1])
for i in range(0, 500):
print (alis['asks'][i][0], alis['asks'][i][1])
binance_sell_demand = float(binance_sell[1]) \
+ float(binance_sell1[1]) + float(binance_sell2[1]) \
+ float(binance_sell3[1]) + float(binance_sell4[1]) \
+ float(binance_sell5[1]) + float(binance_sell6[1]) \
+ float(binance_sell7[1]) + float(binance_sell8[1]) \
+ float(binance_sell9[1]) + float(binance_sell10[1]) \
+ float(binance_sell11[1]) + float(binance_sell12[1]) \
+ float(binance_sell13[1]) + float(binance_sell14[1]) \
+ float(binance_sell15[1]) + float(binance_sell16[1]) \
+ float(binance_sell17[1]) + float(binance_sell18[1]) \
+ float(binance_sell19[1]) + float(binance_sell20[1])
there is 500 bids and 500 asks
after getting data I sum bids ands asks like this
I tried to make for loop but only could print this values cant sum it in for loop this is the code I tried:
sample output:
0.9315 18328.6
0.9316 18201.2
0.9317 23544.0
0.9318 260.4
0.9319 689.5
0.9320 20410.5
0.9321 47.7
0.9322 294.2
0.9323 446.6
0.9324 104.0
0.9325 3802.3
0.9326 100.1
0.9327 20122.9
0.9328 1410.0
0.9329 7745.1
0.9330 9094.4
0.9331 10389.9
0.9332 248.5
0.9333 71559.7
0.9334 18024.1
0.9335 7404.5
0.9336 1366.6
0.9337 21972.4
0.9338 1224.8
0.9339 49.9
0.9340 17590.5
0.9341 17967.1
0.9342 272.3
0.9343 704.4
0.9344 3581.7
0.9345 3896.8
the first items are price second is quantity
I am trying to make a function that sum and divide avg.price and also want to know how many bids total and asks total.
Use the sum() function with a generator that gets the appropriate element of each list item.
binance_buy_demand = sum(float(x[1]) for x in alis['bids'])
binance_sell_demand = sum(float(x[1]) for x in alis['asks'])
You can sum in a for loop like this:
total_ask_volume = 0
total_bid_volume = 0
for i in range(0, 500):
total_ask_volume += float(alis["asks"][i][1])
total_bid_volume += float(alis["bids"][i][1])
print(total_ask_volume, total_bid_volume)
Another option is to skip the i index, and go through the values directly:
total_ask_volume = 0
for ask in alis["asks"]:
total_ask_volume += float(ask[1])
total_bid_volume = 0
for bid in alis["bids"]:
total_bid_volume += float(bid[1])
print(total_ask_volume, total_bid_volume)
This can sometimes be clearer, particularly in situations where you'd otherwise have multiple indexes (i, j, k and so on) which could get confusing.
As a side note, float often rounds in unintuitive ways; it probably doesn't matter in this particular circumstance, but in most situations involving money you probably want to use Decimal instead (or multiply by 100 and count whole numbers of cents).
I need dynamic filtering in DRF that should allow using parenthesis for defining operations precedence and use any combination of the available fields in model.
Operations are: and, or, eq (equal), ne (not equal), gt (greater than), lt (less than)
example: "(date eq '2016-05-01') AND ((number_of_calories gt 20) OR (number_of_calories lt 10))"
How can I achieve this? what is best way?
Currently I have below solution but it's not good approach as it's vulnerable to SQL Injection:
utils.py
mappings = {
' eq ': ' = ',
' ne ': ' != ',
' gt ': ' > ',
' lt ': ' < ',
' gte ': ' >= ',
' lte ': ' <= ',
}
def convert_string(query: str) -> Optional[str]:
if query and isinstance(query, str):
pattern_drop = re.compile(r"drop\s+table\s*\w*")
pattern_alter = re.compile(r"alter\s+table\s+\w+")
pattern_delete = re.compile(r"delete\s+from\s+\w+")
pattern_update = re.compile(r"update\s+\w+\s+set\s+\w+")
pattern_insert = re.compile(r"insert\s+into\s+\w+")
pattern_select = re.compile(r"select\s+\w+\s+from\s+")
query_lower = query.lower()
if '--' in query_lower or '/*' in query_lower or \
pattern_drop.match(query_lower) or pattern_alter.match(query_lower) or \
pattern_update.match(query_lower) or pattern_insert.match(query_lower) or \
pattern_delete.match(query_lower) or pattern_select.match(query_lower):
return None
for expression, operation in mappings.items():
query = query.replace(expression, operation)
return query
views.py
def get_queryset(self):
q_string = self.request.data['query']
# q_string = "(date eq '2016-05-01') AND ((number_of_calories gt 20) OR (number_of_calories lt 10))"
query = convert_string(q_string)
# just replace 'eq' with '=', 'ne' with '!=', and so on ...
# query = "(date = '2016-05-01') AND ((number_of_calories > 20) OR (number_of_calories < 10))"
users = Users.objects.raw('SELECT * FROM Users WHERE ' + query)
return users
For parsing a query string like:
string = "((num_of_pages gt 20) OR (num_of_pages lt 10)) AND (date gt '2016-05-01')"
you can use the pyparsing package (not an expert but very powerful library) with django Q objects:
parsing code:
import pyparsing as pp
import operator as op
from django.db.models import Q
word = pp.Word(pp.alphas, pp.alphanums + "_-*'")
operator = pp.oneOf('lt gt eq').setResultsName('operator')
number = pp.pyparsing_common.number()
quoted = pp.quotedString().setParseAction(pp.removeQuotes)
term = (word | number | quoted)
key = term.setResultsName('key')
value = term.setResultsName('value')
group = pp.Group(key + operator + value)
def q_item(item):
"""Helper for create django Q() object"""
k = f'{item.key}__{item.operator}'
v = item.value
return Q(**{k: v})
class BaseBinary:
def __init__(self, tokens):
self.args = tokens[0][0::2]
def __repr__(self):
return f'{self.__class__.__name__}({self.symbol}):{self.args}'
def evaluate(self):
a = q_item(self.args[0]) if not isinstance(self.args[0], BaseBinary) else self.args[0].evaluate()
b = q_item(self.args[1]) if not isinstance(self.args[1], BaseBinary) else self.args[1].evaluate()
return self.op(a, b)
class BoolNotOp(BaseBinary):
symbol = 'NOT'
op = op.not_
def __init__(self, tokens):
super().__init__(tokens)
self.args = tokens[0][1]
def evaluate(self):
a = q_item(self.args) if not isinstance(self.args, BaseBinary) else self.args.evaluate()
return ~a
class BoolAndOp(BaseBinary):
symbol = 'AND'
op = op.and_
class BoolOrOp(BaseBinary):
symbol = 'OR'
op = op.or_
expr = pp.infixNotation(group,
[('NOT', 1, pp.opAssoc.RIGHT, BoolNotOp),
('AND', 2, pp.opAssoc.LEFT, BoolAndOp),
('OR', 2, pp.opAssoc.LEFT, BoolOrOp)])
Now given a string like:
string = "(date gt '2016-05-01') AND ((num_of_pages gt 20) OR (num_of_pages lt 10))"
to the parser:
parser = expr.parseString(string)[0]
print(parser.evaluate())
give us our Q objects:
(AND: ('date__gt', '2016-05-01'), (OR: ('num_of_pages__gt', 20), ('num_of_pages__lt', 10)))
ready to be filtered
class Book(models.Model):
title = models.CharField(max_length=200)
counter = models.PositiveIntegerField(default=0)
date = models.DateField(auto_now=True)
num_of_pages = models.PositiveIntegerField(default=0)
qs = Book.objects.filter(parser.evaluate())
print(qs.query)
SELECT "core_book"."id", "core_book"."title", "core_book"."counter", "core_book"."date", "core_book"."num_of_pages" FROM "core_book" WHERE ("core_book"."date" > 2016-05-01 AND ("core_book"."num_of_pages" > 20 OR "core_book"."num_of_pages" < 10))
P.S not fully tested.
I currently use the Q object extensively in a project I have that is using the users get parameters to filter a search result.
Here is a snippet
some_initial_query_object = Model.objects.all()
qs_result_dates = []
qs_result_dates.append(
Q(
event_date__start_date_time__gte='2021-08-01',
event_date__start_date_time__lt='2021-09-01' + datetime.timedelta(days=1)
)
)
some_initial_query_object = some_initial_query_object.filter(qs_result_dates)
In your scenario you can use | for OR and & for AND
Q(date='2016-05-01')
&
Q(number_of_calories__gt=20, number_of_calories__lt=10)
Here is an example of dynamic filtering using DRF by overriding the get_queryset method on the ModelViewSet that I use in all of my projects. Using this method I can leverage the full power of Django-ORM framework from the frontend.
views.py
def BaseAPIView(...):
''' base view for other views to inherit '''
def get_queryset(self):
queryset = self.queryset
# get filter request from client:
filter_string = self.request.query_params.get('filter')
# apply filters if they are passed in:
if filters:
filter_dictionary = json.loads(filter_string)
queryset = queryset.filter(**filter_dictionary)
return queryset
The request url will now look like, for example: my_website.com/api/users?filter={"first_name":"John"}
Which can be built like:
script.js
// using ajax as an example:
var filter = JSON.stringify({
"first_name" : "John"
});
$.ajax({
"url" : "my_website.com/api/users?filter=" + filter,
"type" : "GET",
...
});
Some advantages:
no need to specify which fields can be filtered on each view class
write it once, use it everywhere
front end filtering looks exactly like django filtering
can do the same with exclude
Some disadvantages:
potential security risks if you want some fields to be non-filterable
less intuitive front-end code to query a table
Overall, this approach has been far more useful for me than any packages out there.
I am trying to create a JSON string that I can send in a PUT request but build it dynamically. For example, once I am done, I'd like the string to look like this:
{
"request-id": 1045058,
"db-connections":
[
{
"db-name":"Sales",
"table-name":"customer"
}
]
}
I'd like to use constants for the keys, for example, for request-id, I'd like to use CONST_REQUEST_ID. So the keys will be,
CONST_REQUEST_ID = "request-id"
CONST_DB_CONNECTIONS = "db_connections"
CONST_DB_NAME = "db-name"
CONST_TABLE_NAME = "table-name"
The values for the keys will be taken from various variables. For example we can take value "Sales" from a var called dbname with the value "Sales".
I have tried json.load and getting exception.
Help would be appreciated please as I am a bit new to Python.
Create a normal python dictionary, then convert it to JSON.
raw_data = {
CONST_DB_NAME: getDbNameValue()
}
# ...
json_data = json.dumps(raw_data)
# Use json_data in your PUT request.
You can concat the string using + with variables.
CONST_REQUEST_ID = "request-id"
CONST_DB_CONNECTIONS = "db_connections"
CONST_DB_NAME = "db-name"
CONST_TABLE_NAME = "table-name"
request_id = "1045058"
db_name = "Sales"
table_name = 'customer'
json_string = '{' + \
'"' + CONST_REQUEST_ID + '": ' + request_id \
+ ',' + \
'"db-connections":' \
+ '[' \
+ '{' \
+ '"' + CONST_DB_NAME +'":"' + db_name + '",' \
+ '"' + CONST_TABLE_NAME + '":"' + table_name + '"' \
+ '}' \
+ ']' \
+ '}'
print json_string
And this is the result
python st_ans.py
{"request-id": 1045058,"db-connections":[{"db-name":"Sales","table-name":"customer"}]}
I have a condition in which I have joined a table with itself. For getting result rows I need to have two separate attributes in the result rows each of which contain model items.
order_sq1: Order = self._query_from(Order). \
filter(Order.status == OrderStatus.REGISTERED.value,
Order.dispatch_date + Order.dispatch_time > current_time,
Order.who_added_role != UserRole.SHIPPER.value).subquery('Order1')
order_sq2: Order = self._query_from(Order). \
filter(Order.status == OrderStatus.REGISTERED.value,
Order.dispatch_date + Order.dispatch_time > current_time,
Order.who_added_role != UserRole.SHIPPER.value).subquery('Order2')
query = self._query_from(order_sq1).join(
order_sq2,
and_(order_sq1.c.tracking_code != order_sq2.c.tracking_code,
func.ST_DistanceSphere(order_sq1.c.destination_location,
order_sq2.c.source_location) < 0.15 *
order_sq1.c.distance
, func.ST_DistanceSphere(order_sq2.c.destination_location,
order_sq1.c.source_location) < 0.15 *
order_sq1.c.distance
, between(((order_sq1.c.dispatch_date + order_sq1.c.dispatch_time) -
(order_sq2.c.dispatch_date + order_sq2.c.dispatch_time) -
(order_sq1.c.distance / 60) * timedelta(hours=1)),
timedelta(hours=2),
timedelta(hours=20))
, order_sq1.c.source_region_id != order_sq1.c.destination_region_id
, order_sq2.c.source_region_id != order_sq2.c.destination_region_id
, order_sq1.c.vehicle_type == order_sq2.c.vehicle_type
)).add_columns(order_sq2)
I need the result in the form of (Order1, Order2) fields, each as Order class.
I tried many ways with no success!
Here is my model:
class SystemConfiguration(models.Model):
idSystemConfiguration = models.AutoField(primary_key=True)
mail_movement_subject = models.CharField(max_length=50, default="Movement Triggered")
mail_snapped_subject = models.CharField(max_length=50, default="Image Snapped")
mail_carbon_subject = models.CharField(max_length=50, default="Carbon Monoxide Triggered")
mail_door_subject = models.CharField(max_length=50, default="Door Activity Triggered")
door1_name = models.CharField(max_length=20, default="Garage Door")
door2_name = models.CharField(max_length=20, default="Garage Door 2")
door3_name = models.CharField(max_length=20, default="Garage Door 3")
metric = models.BooleanField()
mail_temp_subject = models.CharField(max_length=50, default="Temperature Triggered")
min_temp = models.DecimalField(decimal_places=1,max_digits=4)
max_temp = models.DecimalField(decimal_places=1,max_digits=4)
mail_humidity_subject = models.CharField(max_length=50, default="Humidity Triggered")
mail_to_user = models.CharField(max_length=50, default="")
min_humidity = models.DecimalField(decimal_places=1,max_digits=3)
max_humidity = models.DecimalField(decimal_places=1,max_digits=3)
def setDefaults(self):
print "Creating SystemConfig..."
self.mail_movement_subject = "Movement Triggered"
self.mail_snapped_subject = "Image Snapped"
self.mail_carbon_subject = "Carbon Monoxide Triggered"
self.mail_door_subject = "Door Activity Triggered"
self.door1_name = "Garage Door"
self.door2_name = "Garage Door 2"
self.door3_name = "Garage Door 3"
self.metric = False
self.mail_temp_subject = "Temperature Triggered"
self.min_temp = 28
self.max_temp = 100
self.mail_humidity_subject = "Humidity Triggered"
self.min_humidity = 40
self.max_humidity = 50
self.mail_to_user = ""
self.save()
def __str__(self):
return "SystemConfiguration:" + str(self.idSystemConfiguration) \
+ ",mail_movement_subject:" + str(self.mail_movement_subject) \
+ ",mail_snapped_subject:" + str(self.mail_snapped_subject) \
+ ",mail_carbon_subject:" + str(self.mail_carbon_subject) \
+ ",mail_door_subject:" + str(self.mail_door_subject) \
+ ",Door1Name:" + str(self.door1_name) \
+ ",Door2Name:" + str(self.door2_name) \
+ ",Door3Name:" + str(self.door3_name) \
+ ",Metric:" + str(self.metric) \
+ ",mail_temp_subject:" + str(self.mail_temp_subject) \
+ ",minTemp:" + str(self.min_temp) \
+ ",maxTemp:" + str(self.max_temp) \
+ ",mail_humidity_subject:" + str(self.mail_humidity_subject) \
+ ",minHumid:" + str(self.min_humidity) \
+ ",maxHumid:" + str(self.max_humidity) \
+ ",mail_to_user:" + str(self.mail_to_user) \
+ ",wan_port:" + str(self.wan_port) \
+ ",wan_video_port:" + str(self.wan_video_port) \
+ "\n"
Here is where I am trying to access it (note that I am running a script outside of django, and hence the sys.path.apend(...) and os.environ[...] shenanigans:
import sys
import os
sys.path.append(os.path.abspath("/home/pi/garageMonitor/django/garageMonitor"))
os.environ['DJANGO_SETTINGS_MODULE'] = 'garageMonitor.settings'
from garageMonitor.models import SystemConfiguration
def handleMailFile(file):
config = SystemConfiguration.objects.filter(idSystemConfiguration=1)
config = config[0]
for x in config.__dict__:
print x
print len(config.__dict__)
Here is the results of the run:
mail_snapped_subject
mail_humidity_subject
mail_carbon_subject
max_humidity
door2_name
max_temp
_state
idSystemConfiguration
min_humidity
mail_door_subject
min_temp
door3_name
mail_temp_subject
mail_movement_subject
metric
door1_name
16
Note that all of the fields except mail_to_user are printed in the for loop! I checked the database and sure enough, the mail_to_user field is there with the correct length and type.
Any suggestions would be most appreciated.
I tried adding another field identical to one that was actually working (changed the name.) Django created the field in the database, but it did not show up in the list of dict elements either?!?
I added another column in the database with django code and re-ran the setup. The coded added the column (with the default value), but still when I run the following:
import sys
import os
sys.path.append(os.path.abspath("/home/pi/garageMonitor/django/garageMonitor"))
os.environ['DJANGO_SETTINGS_MODULE'] = 'garageMonitor.settings'
from garageMonitor.models import SystemConfiguration
config = SystemConfiguration.objects.filter(idSystemConfiguration=1)
config = config[0]
for x in config.__dict__:
print x
print len(config.__dict__)
I only get 16 entries in the dict object and the new column is not there. I removed all .pyc files thinking there may be some artifact laying around, but that did not help either.
So, I tried another approach. I created a new model (Jordan) by copying the original one and added it to the models.py file. I ran the setup code. It created the table correctly in the database, but when I tried to access the model, I get the following error:
ImportError: cannot import name Jordan
Arrrgggghhhhh!!!!!
tried moving to another directory (django) and ran the following:
python...
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'garageMonitor.settings'
from garageMonitor.models import Jordan
config = Jordan.objects.filter(idSystemConfiguration=1)
config = config[0]
for x in config.__dict__:
print x
And here is my output:
_known_related_objects
_sticky_filter
_db
_result_cache
_prefetch_done
_for_write
query
model
_prefetch_related_lookups
_hints
(not remotely like what the database table looks like!)