Related
How do I properly write the filter code so it returns only the Animals that are not sold out.
I'm using POSTGRES db, python3.6 and Django 2.1.7 (currently there are v2.2a1,v2.2b1 pre-release versions)
My questioin is an extension to Django JSONField filtering
which filters on a hard coded value in the filter.
My Case requires an annotated value in the filter.
models.py I know that the models can be optimized, but I already have huge amount of records since more than 3 years
from django.db import models
from django.contrib.postgres.fields import JSONField
class Animal(models.Model):
data = models.JSONField(verbose_name=_('data'), blank=True)
class Sell(models.Model):
count = models.IntegerField(verbose_name=_('data'), blank=True)
animal = models.ForeignKey('Animal',
on_delete=models.CASCADE,
related_name="sales_set",
related_query_name="sold"
)
in my api I want to return only the animals that still have something left for selling
animal = Animal(data={'type':'dog', 'bread':'Husky', 'count':20})
What I want to filter should be similar to animal.data['count'] > sum(animal.sales_set__count
Animal.objects.annotate(animals_sold=Sum('sales_set__count'))
.filter(data__contains=[{'count__gt': F('animals_sold')}])
with the code above i get builtins.TypeError
TypeError: Object of type 'F' is not JSON serializable
if I remove the F it won't filter on the value of the animals_sold, but on the text 'animals_sold' and it doesn't do any help.
Animal.objects.annotate(animals_sold=Sum('sales_set__count'))
.filter(data__contains=[{'count__gt': F('animals_sold')}])
Edit 1:
There is one more topic here that can be linked:
Postgres: values query on json key with django
Edit 2:
here is some additional code with custom transform classes as suggested in related django ticket
from django.db.models.constants import LOOKUP_SEP
from django.db.models import F, Q, Prefetch, Sum
from django.db.models import IntegerField, FloatField, ExpressionWrapper
from django.db.models.functions import Cast
from django.contrib.postgres.fields import JSONField
from django.contrib.postgres.fields.jsonb import KeyTransform, KeyTextTransform
class KeyIntegerTransform(KeyTransform): # similar to KeyTextTransform
""" trasnform the data.count to integer """
operator = '->>'
nested_operator = '#>>'
output_field = IntegerField()
class KeyIntTransformFactory:
""" helper class for the JSONF() """
def __init__(self, key_name):
self.key_name = key_name
def __call__(self, *args, **kwargs):
return KeyIntegerTransform(self.key_name, *args, **kwargs)
class JSONF(F):
""" for filtering on JSON Fields """
def resolve_expression(self, query=None, allow_joins=True, reuse=None, summarize=False, for_save=False):
rhs = super().resolve_expression(query, allow_joins, reuse, summarize, for_save)
field_list = self.name.split(LOOKUP_SEP)
for name in field_list[1:]:
rhs = KeyIntegerTransform(name)(rhs)
return rhs
queryset filtering I tried so far:
q = q.filter(data__contains={'count__gt':JSONF('sold_count_sum')})
# err: Object of type 'JSONF' is not JSON serializable
q = q.filter(sold_count_sum__lt=Cast(JSONF('data_count'), IntegerField()))
# err: operator does not exist: text ->> unknown
q = q.filter(sold_count_sum__lt=Cast(JSONF('data__count'), IntegerField()))
# err: 'KeyIntegerTransform' takes exactly 1 argument (0 given)
q = q.filter(sold_count_sum__lt=KeyIntegerTransform('count', 'data'))
# err: operator does not exist: text ->> unknown
q = q.filter(sold_count_sum__lt=F('data__count'))
# err: operator does not exist: text ->> unknown
q = q.filter(sold_count_sum__lt=F('data_count'))
# err: operator does not exist: text ->> unknown
q = q.filter(sold_count_sum__lt=JSONF('data_count'))
# err: operator does not exist: text ->> unknown
q = q.filter(sold_count_sum__lt=JSONF('data__count'))
# err: 'KeyIntegerTransform' takes exactly 1 argument (0 given)
q = q.filter(sold_count_sum__lt=JSONF('data', 'count'))
# err: JSONF.__init__() takes 2 params
queryset = Animal.objects.annotate(
json=Cast(F('data'), JSONField()),
sold_count_sum = Sum('sold__count'),
sold_times = Count('sold'),
).filter(
Q(sold_times=0) | Q(sold_count_sum__lt=Cast(
KeyTextTransform('count', 'json'), IntegerField())
),
# keyword filtering here ...
# client = client
)
this is what works for me, but it can be optimized with a good JSONF field probably
we can also (re)move the json annotation and use casted version of data (may have some performance improvement):
queryset = Animal.objects.annotate(
sold_count_sum = Sum('sold__count'),
sold_times = Count('sold'),
).filter(
Q(sold_times=0) | Q(sold_count_sum__lt=Cast(
KeyTextTransform('count', Cast(
F('data'), JSONField())), IntegerField()
)
),
# keyword filtering here ...
# client = client
)
How about something like this:
from django.db.models import Sum, F
from django.contrib.postgres.fields.jsonb import KeyTransform
Animal.objects.annotate(animals_sold=Sum('sales_set__count'), data_count=KeyTransform('count', 'data')).filter(data_count__gt=F('animals_sold'))
The F class doesn't support a JSONField at this time, but you might try making your own custom expression as described in the related ticket.
I'm want to populate a WTForms SelectField with the rows returned by this query:
cur.execute("SELECT length FROM skipakke_alpin_ski WHERE stock > 0")
The query returns rows with the ski length of different types of skis. cur.fetchall() returns the following tuple:
[(70,), (75,), (82,), (88,), (105,), (115,), (125,), (132,), (140,), (150,), (160,), (170,)]
How would I go about to do add these numbers to a SelectField, so that each ski length would be its own selectable choice? If I had done this manually, I would have done the following:
ski_size = SelectField('Ski size', choices=['70', '70', '75', '75'])
... And so on for all of the different lengths.
In one of the projects I have used like below:
models
class PropertyType(db.Model):
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.String(255), nullable=False)
def __repr__(self):
return str(self.id)
and in forms
from wtforms.ext.sqlalchemy.fields import QuerySelectField
class PropertyEditor(Form):
property_type = QuerySelectField(
'Property Type',
query_factory=lambda: models.PropertyType.query,
allow_blank=False
)
//Other remaining fields
Hope this helps.
The solution might look like the code below.
Let's assume you have two files: routes.py and views.py
In routes.py file you put this
from flask_wtf import FlaskForm
from wtforms import SelectField
# Here we have class to render ski
class SkiForm(FlaskForm):
ski = SelectField('Ski size')
In views.py file you put this
# Import your SkiForm class from `routes.py` file
from routes import SkiForm
# Here you define `cur`
cur = ...
# Now let's define a method to return rendered page with SkiForm
def show_ski_to_user():
# List of skies
cur.execute("SELECT length FROM skipakke_alpin_ski WHERE stock > 0")
skies = cur.fetchall()
# create form instance
form = SkiForm()
# Now add ski length to the options of select field
# it must be list with options with (key, value) data
form.ski.choices = [(ski, ski) for ski in skies]
# If you want to use id or other data as `key` you can change it in list generator
if form.validate():
# your code goes here
return render_template('any_file.html', form=form)
Remember that by default key value is unicode. If you want to use int or other data type use coerce argument in SkiForm class, like this
class SkiForm(FlaskForm):
ski = SelectField('Ski size', coerce=int)
Solution:
I had quite simmilar problem, and here is my workaround
class SkiForm(FlaskForm):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
cur.execute("SELECT length FROM skipakke_alpin_ski WHERE stock > 0")
skies = cur.fetchall()
self.ski_size.choices = [
(ski , ski) for ski in skies
]
ski_size = SelectField("Ski size")
Explanation:
We modified original FlaskForm, so that it executs database query each time when it is being created.
So SkiForm field data choices always stays up to date.
I managed to solve this by doing the following:
def fetch_available_items(table_name, column):
with sqlite3.connect('database.db') as con:
cur = con.cursor()
cur.execute("SELECT length FROM skipakke_alpin_ski WHERE stock > 0")
return cur.fetchall()
class SkipakkeForm(Form):
alpin_ski = SelectField('Select your ski size', choices=[])
#app.route('/skipakke')
def skipakke():
form = SkipakkeForm
# Clear the SelectField on page load
form.alpin_ski.choices = []
for row in fetch_available_items('skipakke_alpin_ski', 'length'):
stock = str(row[0])
form.alpin_ski.choices += [(stock, stock + ' cm')]
Is it possible to call
tasks = models.Conference.objects.filter(location_id=key)
data = serializers.serialize("json", tasks)
and have it return the verbose field names rather than the variable names?
One way to accomplish this, is by monkey patching the methods within the django.core.serializers.python.Serializer class to return each fields verbose_name opposed to the standard name attribute.
Take for example the following code...
models.py
from django.db import models
class RelatedNode(models.Model):
name = models.CharField(max_length=100, verbose_name="related node")
class Node(models.Model):
name = models.CharField(max_length=100, verbose_name="verbose name")
related_node = models.ForeignKey(RelatedNode, verbose_name="verbose fk related node", related_name="related_node")
related_nodes = models.ManyToManyField(RelatedNode, verbose_name="verbose related m2m nodes", related_name="related_nodes")
I create these model objects within the database...
RelatedNode.objects.create(name='related_node_1')
RelatedNode.objects.create(name='related_node_2')
RelatedNode.objects.create(name='related_node_fk')
Node.objects.create(name='node_1', related_node=RelatedNode.objects.get(name='related_node_fk'))
Node.objects.all()[0].related_nodes.add(RelatedNode.objects.get(name='related_node_1'))
Node.objects.all()[0].related_nodes.add(RelatedNode.objects.get(name='related_node_2'))
views.py
from testing.models import Node
from django.utils.encoding import smart_text, is_protected_type
from django.core.serializers.python import Serializer
from django.core import serializers
def monkey_patch_handle_field(self, obj, field):
value = field._get_val_from_obj(obj)
# Protected types (i.e., primitives like None, numbers, dates,
# and Decimals) are passed through as is. All other values are
# converted to string first.
if is_protected_type(value):
self._current[field.verbose_name] = value
else:
self._current[field.verbose_name] = field.value_to_string(obj)
def monkey_patch_handle_fk_field(self, obj, field):
if self.use_natural_foreign_keys and hasattr(field.rel.to, 'natural_key'):
related = getattr(obj, field.name)
if related:
value = related.natural_key()
else:
value = None
else:
value = getattr(obj, field.get_attname())
self._current[field.verbose_name] = value
def monkey_patch_handle_m2m_field(self, obj, field):
if field.rel.through._meta.auto_created:
if self.use_natural_foreign_keys and hasattr(field.rel.to, 'natural_key'):
m2m_value = lambda value: value.natural_key()
else:
m2m_value = lambda value: smart_text(value._get_pk_val(), strings_only=True)
self._current[field.verbose_name] = [m2m_value(related)
for related in getattr(obj, field.name).iterator()]
Serializer.handle_field = monkey_patch_handle_field
Serializer.handle_fk_field = monkey_patch_handle_fk_field
Serializer.handle_m2m_field = monkey_patch_handle_m2m_field
serializers.serialize('json', Node.objects.all())
This outputs for me...
u'[{"fields": {"verbose fk related node": 3, "verbose related m2m nodes": [1, 2], "verbose name": "node_1"}, "model": "testing.node", "pk": 1}]'
As we could see, this actually gives us back the verbose_name of each field as keys in the returned dictionaries.
I am using the Products.FacultyStaffDirectory product to manage contacts in my website.
One of the content types, i.e. FacultyStaffDirectory, has the description field in the 'categorization' tab when you are in edit mode. Now I need to remove the description field and put it in the default tab.
To do this, I'm using archetypes.schemaextender product to edit the field. In particular, I am using ISchemaModifier.
I have implemented the code but the field is still being shown in the categorization tab. May be something I missed. Below is the code:
This is the class that contains the description field that I want to modify:
# -*- coding: utf-8 -*-
__author__ = """WebLion <support#weblion.psu.edu>"""
__docformat__ = 'plaintext'
from AccessControl import ClassSecurityInfo
from Products.Archetypes.atapi import *
from zope.interface import implements
from Products.FacultyStaffDirectory.interfaces.facultystaffdirectory import IFacultyStaffDirectory
from Products.FacultyStaffDirectory.config import *
from Products.CMFCore.permissions import View, ManageUsers
from Products.CMFCore.utils import getToolByName
from Products.ATContentTypes.content.base import ATCTContent
from Products.ATContentTypes.content.schemata import ATContentTypeSchema, finalizeATCTSchema
from Products.membrane.at.interfaces import IPropertiesProvider
from Products.membrane.utils import getFilteredValidRolesForPortal
from Acquisition import aq_inner, aq_parent
from Products.FacultyStaffDirectory import FSDMessageFactory as _
schema = ATContentTypeSchema.copy() + Schema((
LinesField('roles_',
accessor='getRoles',
mutator='setRoles',
edit_accessor='getRawRoles',
vocabulary='getRoleSet',
default = ['Member'],
multiValued=1,
write_permission=ManageUsers,
widget=MultiSelectionWidget(
label=_(u"FacultyStaffDirectory_label_FacultyStaffDirectoryRoles", default=u"Roles"),
description=_(u"FacultyStaffDirectory_description_FacultyStaffDirectoryRoles", default=u"The roles all people in this directory will be granted site-wide"),
i18n_domain="FacultyStaffDirectory",
),
),
IntegerField('personClassificationViewThumbnailWidth',
accessor='getClassificationViewThumbnailWidth',
mutator='setClassificationViewThumbnailWidth',
schemata='Display',
default=100,
write_permission=ManageUsers,
widget=IntegerWidget(
label=_(u"FacultyStaffDirectory_label_personClassificationViewThumbnailWidth", default=u"Width for thumbnails in classification view"),
description=_(u"FacultyStaffDirectory_description_personClassificationViewThumbnailWidth", default=u"Show all person thumbnails with a fixed width (in pixels) within the classification view"),
i18n_domain="FacultyStaffDirectory",
),
),
))
FacultyStaffDirectory_schema = OrderedBaseFolderSchema.copy() + schema.copy() # + on Schemas does only a shallow copy
finalizeATCTSchema(FacultyStaffDirectory_schema, folderish=True)
class FacultyStaffDirectory(OrderedBaseFolder, ATCTContent):
"""
"""
security = ClassSecurityInfo()
implements(IFacultyStaffDirectory, IPropertiesProvider)
meta_type = portal_type = 'FSDFacultyStaffDirectory'
# Make this permission show up on every containery object in the Zope instance. This is a Good Thing, because it easy to factor up permissions. The Zope Developer's Guide says to put this here, not in the install procedure (http://www.zope.org/Documentation/Books/ZDG/current/Security.stx). This is because it isn't "sticky", in the sense of being persisted through the ZODB. Thus, it has to run every time Zope starts up. Thus, when you uninstall the product, the permission doesn't stop showing up, but when you actually remove it from the Products folder, it does.
security.setPermissionDefault('FacultyStaffDirectory: Add or Remove People', ['Manager', 'Owner'])
# moved schema setting after finalizeATCTSchema, so the order of the fieldsets
# is preserved. Also after updateActions is called since it seems to overwrite the schema changes.
# Move the description field, but not in Plone 2.5 since it's already in the metadata tab. Although,
# decription and relateditems are occasionally showing up in the "default" schemata. Move them
# to "metadata" just to be safe.
if 'categorization' in FacultyStaffDirectory_schema.getSchemataNames():
FacultyStaffDirectory_schema.changeSchemataForField('description', 'categorization')
else:
FacultyStaffDirectory_schema.changeSchemataForField('description', 'metadata')
FacultyStaffDirectory_schema.changeSchemataForField('relatedItems', 'metadata')
_at_rename_after_creation = True
schema = FacultyStaffDirectory_schema
# Methods
security.declarePrivate('at_post_create_script')
def at_post_create_script(self):
"""Actions to perform after a FacultyStaffDirectory is added to a Plone site"""
# Create some default contents
# Create some base classifications
self.invokeFactory('FSDClassification', id='faculty', title='Faculty')
self.invokeFactory('FSDClassification', id='staff', title='Staff')
self.invokeFactory('FSDClassification', id='grad-students', title='Graduate Students')
# Create a committees folder
self.invokeFactory('FSDCommitteesFolder', id='committees', title='Committees')
# Create a specialties folder
self.invokeFactory('FSDSpecialtiesFolder', id='specialties', title='Specialties')
security.declareProtected(View, 'getDirectoryRoot')
def getDirectoryRoot(self):
"""Return the current FSD object through acquisition."""
return self
security.declareProtected(View, 'getClassifications')
def getClassifications(self):
"""Return the classifications (in brains form) within this FacultyStaffDirectory."""
portal_catalog = getToolByName(self, 'portal_catalog')
return portal_catalog(path='/'.join(self.getPhysicalPath()), portal_type='FSDClassification', depth=1, sort_on='getObjPositionInParent')
security.declareProtected(View, 'getSpecialtiesFolder')
def getSpecialtiesFolder(self):
"""Return a random SpecialtiesFolder contained in this FacultyStaffDirectory.
If none exists, return None."""
specialtiesFolders = self.getFolderContents({'portal_type': 'FSDSpecialtiesFolder'})
if specialtiesFolders:
return specialtiesFolders[0].getObject()
else:
return None
security.declareProtected(View, 'getPeople')
def getPeople(self):
"""Return a list of people contained within this FacultyStaffDirectory."""
portal_catalog = getToolByName(self, 'portal_catalog')
results = portal_catalog(path='/'.join(self.getPhysicalPath()), portal_type='FSDPerson', depth=1)
return [brain.getObject() for brain in results]
security.declareProtected(View, 'getSortedPeople')
def getSortedPeople(self):
""" Return a list of people, sorted by SortableName
"""
people = self.getPeople()
return sorted(people, cmp=lambda x,y: cmp(x.getSortableName(), y.getSortableName()))
security.declareProtected(View, 'getDepartments')
def getDepartments(self):
"""Return a list of FSDDepartments contained within this site."""
portal_catalog = getToolByName(self, 'portal_catalog')
results = portal_catalog(portal_type='FSDDepartment')
return [brain.getObject() for brain in results]
security.declareProtected(View, 'getAddableInterfaceSubscribers')
def getAddableInterfaceSubscribers():
"""Return a list of (names of) content types marked as addable using the
IFacultyStaffDirectoryAddable interface."""
return [type['name'] for type in listTypes() if IFacultyStaffDirectoryAddable.implementedBy(type['klass'])]
security.declarePrivate('getRoleSet')
def getRoleSet(self):
"""Get the roles vocabulary to use."""
portal_roles = getFilteredValidRolesForPortal(self)
allowed_roles = [r for r in portal_roles if r not in INVALID_ROLES]
return allowed_roles
#
# Validators
#
security.declarePrivate('validate_id')
def validate_id(self, value):
"""Ensure the id is unique, also among groups globally."""
if value != self.getId():
parent = aq_parent(aq_inner(self))
if value in parent.objectIds():
return _(u"An object with id '%s' already exists in this folder") % value
groups = getToolByName(self, 'portal_groups')
if groups.getGroupById(value) is not None:
return _(u"A group with id '%s' already exists in the portal") % value
registerType(FacultyStaffDirectory, PROJECTNAME)
Below is the portion of code from the class that I have implemented to change the description field's schemata:
from Products.Archetypes.public import ImageField, ImageWidget, StringField, StringWidget, SelectionWidget, TextField, RichWidget
from Products.FacultyStaffDirectory.interfaces.facultystaffdirectory import IFacultyStaffDirectory
from archetypes.schemaextender.field import ExtensionField
from archetypes.schemaextender.interfaces import ISchemaModifier, ISchemaExtender, IBrowserLayerAwareExtender
from apkn.templates.interfaces import ITemplatesLayer
from zope.component import adapts
from zope.interface import implements
class _ExtensionImageField(ExtensionField, ImageField): pass
class _ExtensionStringField(ExtensionField, StringField): pass
class _ExtensionTextField(ExtensionField, TextField): pass
class FacultyStaffDirectoryExtender(object):
"""
Adapter to add description field to a FacultyStaffDirectory.
"""
adapts(IFacultyStaffDirectory)
implements(ISchemaModifier, IBrowserLayerAwareExtender)
layer = ITemplatesLayer
fields = [
]
def __init__(self, context):
self.context = context
def getFields(self):
return self.fields
def fiddle(self, schema):
desc_field = schema['description'].copy()
desc_field.schemata = "default"
schema['description'] = desc_field
And here is the code from my configure.zcml:
<adapter
name="apkn-FacultyStaffDirectoryExtender"
factory="apkn.templates.extender.FacultyStaffDirectoryExtender"
provides="archetypes.schemaextender.interfaces.ISchemaModifier"
/>
Is there something that I'm missing in this?
def fiddle(self, schema):
schema['description'].schemata = 'default'
should be sufficient. The copy() operation does not make any sense here.
In order to check if the fiddle() method is actually used: use pdb or add debug print statements.
It turns out that there was nothing wrong with the code.
I got it to work by doing the following:
Restarted the plone website
Re-installed the product
Clear and rebuilt my catalog
This got it working somehow.
I have a Django project containing two apps, Expenses and Sales which both have models named Item. I'm using django-autocomplete-light to ease the selection of Item. This works for either Expenses or Sales depending on which channel I register last but the other one wrongly shows the same Items.
autocomplete_light_registry.py
from sales.models import Item as SalesItem
from expenses.models import Item as ExpenseItem
class ExpenseChannel(autocomplete_light.ChannelBase):
def query_filter(self, results):
q = self.request.GET.get('q', None)
if q:
if results.model == ExpenseItem:
results = results.filter(
Q(name__icontains=q)
return results
class SalesChannel(autocomplete_light.ChannelBase):
def query_filter(self, results):
q = self.request.GET.get('q', None)
if q:
if results.model == SalesItem:
results = results.filter(
Q(name__icontains=q)
return results
autocomplete_light.register(ExpenseItem, ExpenseChannel, placeholder='Select an item (e)')
autocomplete_light.register(SalesItem, SalesChannel, placeholder='Select an item (s)')
admin.py
For sales app, similar in expenses
import autocomplete_light
class SalesItemInline(admin.TabularInline):
fields = ('item', )
model = SalesItem
form = autocomplete_light.modelform_factory(SalesItem)
Checking the log when using the autocomplete fields i see the same url being fetched from both views.
"GET /autocomplete/channel/ItemChannel/?q= HTTP/1.1" 200 1416
How do I configure this so list of sales.Item is returned in Admin Sales view and list of expenses.Item is returned in Admin Expenses view?
What's happening is that the channel class is generated in most cases and it's name is generated too. However, you can avoid channel class generation and channel name generation (hopefully, or this would really suck).
From the registry documentation:
Three cases are possible:
specify model class and ModelNameChannel will be generated extending ChannelBase, with attribute model=model
specify a model and a channel class that does not have a model attribute, and a ModelNameChannel will be generated, with attribute
model=model
specify a channel class with a model attribute, and the channel is directly registered
The solution to avoid channel class generation is to be in the third case: register a model and channel class with a model attribute.
autocomplete_light_registry.py
from sales.models import Item as SalesItem
from expenses.models import Item as ExpenseItem
class ExpenseChannel(autocomplete_light.ChannelBase):
placeholder='Select an item (e)'
model = ExpenseItem
def query_filter(self, results):
q = self.request.GET.get('q', None)
if q:
if results.model == ExpenseItem:
results = results.filter(
Q(name__icontains=q)
return results
class SalesChannel(autocomplete_light.ChannelBase):
model = SalesItem
placeholder = 'Select an item (s)'
def query_filter(self, results):
q = self.request.GET.get('q', None)
if q:
if results.model == SalesItem:
results = results.filter(
Q(name__icontains=q)
return results
autocomplete_light.register(ExpenseChannel)
autocomplete_light.register(SalesChannel)
That would work up to 0.7rc2.
Starting 0.7rc3 (to be released when the pending issue is closed), register() has a new keyword argument, channel_name, which you may use.
But you should be careful with your code, it seems like the query_filter() implementation from your classes is the same as the default implementation ...