I have created the following SQL query to find all users within a mile and it seems to work fine:
SELECT * FROM user
WHERE ST_DWithin(
user.location,
ST_MakePoint(-2.242631, 53.480759)::geography, 1609)
);
However I want to convert this into a flask/sqlalchemy/geoalchemy query?
Try something like this:
DISTANCE = 100 #100 meters
db.session.query(User).filter(func.ST_DWithin(User.location, cast(funct.ST_SetSRID(func.ST_MakePoint(-2.242631, 53.480759), 1609), Geography), DISTANCE)).all()
Related
I am trying to create a website using Django and Python3 where users can add there MTG cards to their inventory. I have two tables, inventory_cards - 12,000 Rows - that holds the users inventory data and pricing_cards - 65,000 rows - that holds the pricing data for each card.
inventory_cards table contains user_id, card_id, nonfoil, foil
pricing_cards tables contains card_id, date, nonfoil, foil
I am trying to work out the total worth of a users inventory, however my code is either slow or super heavy on the database.
method 1:
The method takes around 6 minutes but only hits the database twice. With this data being displayed of a website, it is not viable to have the page take over 6 minutes to load.
user_inventory = list(inventory_cards.objects.filter(user_id=request.user.id).values_list('card_id', 'nonfoil', 'foil'))
pricing = list(pricing_cards.objects.order_by('card_id', '-date').distinct('card_id').values_list('card_id', 'nonfoil', 'foil'))
combined_list = [x + y[1:] for x in user_inventory for y in pricing if x[0] == y[0]]
for i in combined_list:
inventory_value = Decimal(inventory_value) + ((Decimal(i[1]) * Decimal(i[3])) + (Decimal(i[2]) * Decimal(i[4])))
method 2:
The method takes around 15 seconds but the database shows over 25,00 transactions. This is still too long for the page to load, and can imagine the strain on the database with multiple users.
user_inventory = inventory_cards.objects.filter(user_id=request.user.id).values('card_id', 'nonfoil', 'foil')
for i in user_inventory:
nonfoil_value = i['nonfoil'] * float(pricing_cards.objects.filter(card_id=i['card_id']).values_list('nonfoil').get()[0])
foil_value = i['foil'] * float(pricing_cards.objects.filter(card_id=i['card_id']).values_list('foil').get()[0])
inventory_value = inventory_value + (nonfoil_value + foil_value)
is there a better way to perform this calculation? is there python package I can install to perform this calculation better.
bb1950328 pointed me in the right direction. The below code run a SQL query without using the Django models. The webpage loads almost instantly.
with connections['default'].cursor() as cursor:
cursor.execute("SELECT SUM((inventory_cards.nonfoil * pricing_cards.nonfoil) + (inventory_cards.foil * pricing_cards.foil)) FROM inventory_cards INNER JOIN pricing_cards ON pricing_cards.card_id = inventory_cards.card_id WHERE inventory_cards.user_id = 3")
inventory_value = cursor.fetchone()[0]
Scenario
I have a table student. it has following attributes
name,
age,
school_passout_date,
college_start_date
I need a report to know what is the avg number of days student get free between the passing the school and starting college.
Current approach
Currently i am irritating over the range of values finding days for each student and getting its avg.
Problem
That is highly inefficient when the record set gets bigger.
Question
Is there any ability in the Django ORM that gives me totals days between the two dates?
Possibility
I am looking for something like this.
Students.objects.filter(school_passed=True, started_college=True).annotate(total_days_between=Count('school_passout_date', 'college_start_date'), Avg_days=Avg('school_passout_date', 'college_start_date'))
You can do this like so:
Model.objects.annotate(age=Cast(ExtractDay(TruncDate(Now()) - TruncDate(F('created'))), IntegerField()))
This lets you work with the integer value, eg you could then do something like this:
from django.db.models import IntegerField, F
from django.db.models.functions import Cast, ExtractDay, TruncDate
qs = (
Model
.objects
.annotate(age=Cast(ExtractDay(TruncDate(Now()) - TruncDate(F('created'))), IntegerField()))
.annotate(age_bucket=Case(
When(age__lt=30, then=Value('new')),
When(age__lt=60, then=Value('current')),
default=Value('aged'),
output_field=CharField(),
))
)
This question is very old but Django ORM is much more advanced now.
It's possible to do this using F() functions.
from django.db.models import Avg, F
college_students = Students.objects.filter(school_passed=True, started_college=True)
duration = college_students.annotate(avg_no_of_days=Avg( F('college_start_date') - F('school_passout_date') )
Mathematically, according to the (expected) fact that the pass out date is allway later than the start date, you can just get an average off all your start date, and all your pass out date, and make the difference.
This gives you a solution like that one
from django.db.models import Avg
avg_start_date = Students.objects.filter(school_passed=True, started_college=True).aggregate(Avg('school_start_date'))
avg_passout_date = Students.objects.filter(school_passed=True, started_college=True).aggregate(Avg('school_passout_date'))
avg_time_at_college = avg_passout_date - avg_start_date
Django currently only accept aggregation for 4 function : Max, Min, Count, et Average, so this is a little tricky to do.
Then the solution is using the method extra . That way:
Students.objects.
extra(select={'difference': 'school_passout_date' - 'college_start_date'}).
filter('school_passed=True, started_college=True)
But then, you still have to do the average on the server side
I have this sqlalchemy query:
query = session.query(Store).options(joinedload('salesmen').
joinedload('comissions').
joinedload('orders')).\
filter(Store.store_code.in_(selected_stores))
stores = query.all()
for store in stores:
for salesman in store.salesmen:
for comission in salesman.comissions:
#generate html for comissions for each salesman in each store
#print html document using PySide
This was working perfectly, however I added two new filter queries:
filter(Comissions.payment_status == 0).\
filter(Order.order_date <= self.dateEdit.date().toPython())
If I add just the first filter the application hangs for a couple of seconds, if I add both the application hangs indefinitely
What am I doing wrong here? How do I make this query fast?
Thank you for your help
EDIT: This is the sql generated, unfortunately the class and variable names are in Portuguese, I just translated them to English so it would be easier to undertand,
so Loja = Store, Vendedores = Salesmen, Pedido = Order, Comission = Comissao
Query generated:
SELECT "Loja"."CodLoja", "Vendedores_1"."CodVendedor", "Vendedores_1"."NomeVendedor", "Vendedores_1"."CodLoja", "Vendedores_1"."PercentualComissao",
"Vendedores_1"."Ativo", "Comissao_1"."CodComissao", "Comissao_1"."CodVendedor", "Comissao_1"."CodPedido",
"Pedidos_1"."CodPedido", "Pedidos_1"."CodLoja", "Pedidos_1"."CodCliente", "Pedidos_1"."NomeCliente", "Pedidos_1"."EnderecoCliente", "Pedidos_1"."BairroCliente",
"Pedidos_1"."CidadeCliente", "Pedidos_1"."UFCliente", "Pedidos_1"."CEPCliente", "Pedidos_1"."FoneCliente", "Pedidos_1"."Fone2Cliente", "Pedidos_1"."PontoReferenciaCliente",
"Pedidos_1"."DataPedido", "Pedidos_1"."ValorProdutos", "Pedidos_1"."ValorCreditoTroca",
"Pedidos_1"."ValorTotalDoPedido", "Pedidos_1"."Situacao", "Pedidos_1"."Vendeu_Teflon", "Pedidos_1"."ValorTotalTeflon",
"Pedidos_1"."DataVenda", "Pedidos_1"."CodVendedor", "Pedidos_1"."TipoVenda", "Comissao_1"."Valor", "Comissao_1"."DataPagamento", "Comissao_1"."StatusPagamento"
FROM "Comissao", "Pedidos", "Loja" LEFT OUTER JOIN "Vendedores" AS "Vendedores_1" ON "Loja"."CodLoja" = "Vendedores_1"."CodLoja"
LEFT OUTER JOIN "Comissao" AS "Comissao_1" ON "Vendedores_1"."CodVendedor" = "Comissao_1"."CodVendedor" LEFT OUTER JOIN "Pedidos" AS "Pedidos_1" ON "Pedidos_1"."CodPedido" = "Comissao_1"."CodPedido"
WHERE "Loja"."CodLoja" IN (:CodLoja_1) AND "Comissao"."StatusPagamento" = :StatusPagamento_1 AND "Pedidos"."DataPedido" <= :DataPedido_1
Your FROM clause is producing a Cartesian product and includes each table twice, once for filtering the result and once for eagerly loading the relationship.
To stop this use contains_eager instead of joinedload in your options. This will look for the related attributes in the query's columns instead of constructing an extra join. You will also need to explicitly join to the other tables in your query, e.g.:
query = session.query(Store)\
.join(Store.salesmen)\
.join(Store.commissions)\
.join(Store.orders)\
.options(contains_eager('salesmen'),
contains_eager('comissions'),
contains_eager('orders'))\
.filter(Store.store_code.in_(selected_stores))\
.filter(Comissions.payment_status == 0)\
.filter(Order.order_date <= self.dateEdit.date().toPython())
I would like to insert records into a sqlite database with fields such that every query that specifies a value for that field does not disqualify the record.
Make Model Engine Parameter
Ford * * 1
Ford Taurus * 2
Ford Escape * 3
So a query = (database.table.Make == Ford') & (database.table.Model == 'Taurus') would return the first two records
EDIT: thanks to woot, I decided to use the following: (database.table.Make.belongs('Ford','')) & (database.table.Model.belongs('Taurus','')) which is the syntax for the IN operator in web2py
Are you looking for something like this? It won't perform well due to the ORs if you have a lot of rows.
SELECT *
FROM Cars
WHERE ( Cars.Make = 'Ford' OR Cars.Make = '*' )
AND ( Cars.Model = 'Taurus' OR Cars.Model = '*' )
Here is a SQL Fiddle example.
If you meant to use NULL, you can just replace that and replace the OR condition with OR Cars.Make IS NULL, etc.
Or to make it maybe a little less verbose:
SELECT *
FROM Cars
WHERE Cars.Make IN ('Ford','*')
AND Cars.Model IN ('Taurus','*')
But you wouldn't be able to use NULL in this case and would have to use the * token.
SQL Fiddle
I just installed PostGIS with GeoDjango. Everything worked fine, but now I have a problem and cant find out the reason for this.
I have model like this:
from django.contrib.gis.db import models
class Shop(models.Model):
name = models.CharField(max_length=80)
point = models.PointField(null=True, blank=True)
objects = models.GeoManager()
And I set its point to this position (49.794254,9.927489). Then i create a point like this:
pnt = fromstr('POINT(50.084068 8.238381)')
The distance between this points should be about ~ 125 km, but when i do this:
results = Shop.objects.distance(pnt)
print results[0].distance.km
I'm getting always about 60 km too much in my result, so it returns 190 km! My SRIDs of both points are 4326... probably something wrong with that?
And maybe another interesting fact, when i do this:
pnt.distance(shop.point)
it returns 1.713790... as a result.
What am I doing wrong? Any alternatives for me to use with python + django? If there is a better solution I would not need to use PostGIS.
Hope you can help me!
Chris
I just ran this query in postgis :
select round(CAST(ST_Distance_Sphere(ST_GeomFromText('POINT(49.794254 9.927489)',4326), ST_GeomFromText('POINT(50.084068 8.238381)',4326)) As numeric)/1000.0,2) as distance_km;
distance_km
-------------
190.50
the result is in fact 190.50, so it seems there's nothing wrong with your 190 km result
same result with this awesome page, there is a brief explanation of how to calculate this distances.
the 1.713790... result seems to be in the same units of the srid, or in other words that number is not in meters.
EDIT
Ooohh I just saw your problem you misplaced the lat and lon, in the WKT format, Longitude comes first so the real query should be:
select round(CAST(ST_Distance_Sphere(ST_GeomFromText('POINT(9.927489 49.794254)',4326), ST_GeomFromText('POINT(8.238381 50.084068)',4326)) As numeric)/1000.0,2) as distance_km;
distance_km
-------------
125.10
so the points should be created like this
POINT(9.927489 49.794254)
POINT(8.238381 50.084068)