We Keep Coding

How to Handle When Request Returns None

I have a list of IDs which corresponds to a set of records (opportunities) in a database. I then pass this list as a parameter in a RESTful API request where I am filtering the results (tickets) by ID. For each match, the query returns JSON data pertaining to the individual record. However, I want to handle when the query does not find a match. I would like to assign some value for this case such as the string "None", because not every opportunity has a ticket. How can I make sure there exists some value in presales_tickets for every ID in opportunity_list? Could I provide a default value in the request for this case?
views.py
opportunities = cwObj.get_opportunities()
temp = []
opportunity_list = []
cw_presales_engineers = []
for opportunity in opportunities:
temp.append(str(opportunity['id']))
opportunity_list = ','.join(temp)
presales_tickets = cwObj.get_tickets_by_opportunity(opportunity_list)
for opportunity in opportunities:
try:
if opportunity['id'] == presales_tickets[0]['opportunity']['id']:
try:
for presales_ticket in presales_tickets:
cw_engineer = presales_ticket['owner']['name']
cw_presales_engineers.append(cw_engineer)
except:
pass
else:
cw_engineer = 'None'
cw_presales_engineers.append(cw_engineer)
except AttributeError:
cw_engineer = ''
cw_presales_engineers.append(cw_engineer)

So, lets say you have a Ticket model and Opportunity model. Connected via a foreign key.
class Opportunity(models.Model):
... some fields here ...
class Ticket(models.Model):
opportunity = models.ForeignKey(Opportunity)
and in your view, you get a list of opportunity ids
def some_view(request):
ids = request.GET['ids']
It sounds, like what you want is to fetch all the tickets for the supplied opportunities and add some default processing for the opportunities that do not have tickets. If that is the case, why not do something like
def some_view(request):
ids = request.GET['ids']
tickets = Ticket.objects.filter(opportunity__id__in=ids)
results = []
for ticket in tickets:
result = ... do your thing here ...
results.append(result)
# now handle missing opportunities
good_ids = tickets.values_list('opportunity__id', flat=True).distinct()
for id in ids:
if id not in good_ids:
result = ... do your default processing ...
results.append(result)
Is that what you are trying to do?

How do I produce nested JSON from database query with joins? Using Python / SQLAlchemy

I have a specify use case but my question pertains to the best way of doing this in general.
I have three tables
Order - primary key order_id
OrderLine - Linking table with order_id, product_id and quantity. An order has 1 or more order lines
Product - primary key product_id, each order line has one product
In sqlachemy / python how do I generate nested JSON along the lines of:
{
"orders": [
{
"order_id": 1
"some_order_level_detail": "Kansas"
"order_lines": [
{
"product_id": 1,
"product_name": "Clawhammer",
"quantity": 5
},
...
]
},
...
]
}
Potential Ideas
Hack away doing successive queries
First idea which I want to get away from if possible is using list comprehesion and a brute force approach.
def get_json():
answer = {
"orders": [
{
"order_id": o.order_id,
"some_order_level_detail": o.some_order_level_detail,
"order_lines": [
{
"product_id": 1,
"product_name": Product.query.get(o_line.product_id).product_name,
"quantity": 5
}
for o_line in OrderLine.query.filter(order_id=o.order_id).all()
]
}
for o in Order.query.all()
]
}
This gets hard to maintain mixing the queries with json. Ideally I'd like to do a query first...
Get joined results first, somehow manipulate later
The second idea is to do a join query to join the three tables showing per row in OrderLine the order and product details.
My question to pythonista out there is is there a nice way to convert this to nested json.
Another way?
This really seems like such a common requirement I'm really wondering whether there is a book method for this sort of thing?
Is there an SQLAchemy version of this

Look into marshmallow-sqlalchemy, as it does exactly what you're looking for.
I strongly advise against baking your serialization directly into your model, as you will eventually have two services requesting the same data, but serialized in a different way (including fewer or more nested relationships for performance, for instance), and you will either end up with either (1) a lot of bugs that your test suite will miss unless you're checking for literally every field or (2) more data serialized than you need and you'll run into performance issues as the complexity of your application scales.
With marshmallow-sqlalchemy, you'll need to define a schema for each model you'd like to serialize. Yes, it's a bit of extra boilerplate, but believe me - you will be much happier in the end.
We build applications using flask-sqlalchemy and marshmallow-sqlalchemy like this (also highly recommend factory_boy so that you can mock your service and write unit tests in place of of integration tests that need to touch the database):
# models
class Parent(Base):
__tablename__ = 'parent'
id = Column(Integer, primary_key=True)
children = relationship("Child", back_populates="parent")
class Child(Base):
__tablename__ = 'child'
id = Column(Integer, primary_key=True)
parent_id = Column(Integer, ForeignKey('parent.id'))
parent = relationship('Parent', back_populates='children',
foreign_keys=[parent_id])
# schemas. Don't put these in your models. Avoid tight coupling here
from marshmallow_sqlalchemy import ModelSchema
import marshmallow as ma
class ParentSchema(ModelSchema):
children = ma.fields.Nested(
'myapp.schemas.child.Child', exclude=('parent',), many=True)
class Meta(ModelSchema.Meta):
model = Parent
strict = True
dump_only = ('id',)
class ChildSchema(ModelSchema):
parent = ma.fields.Nested(
'myapp.schemas.parent.Parent', exclude=('children',))
class Meta(ModelSchema.Meta):
model = Child
strict = True
dump_only = ('id',)
# services
class ParentService:
'''
This service intended for use exclusively by /api/parent
'''
def __init__(self, params, _session=None):
# your unit tests can pass in _session=MagicMock()
self.session = _session or db.session
self.params = params
def _parents(self) -> typing.List[Parent]:
return self.session.query(Parent).options(
joinedload(Parent.children)
).all()
def get(self):
schema = ParentSchema(only=(
# highly recommend specifying every field explicitly
# rather than implicit
'id',
'children.id',
))
return schema.dump(self._parents()).data
# views
#app.route('/api/parent')
def get_parents():
service = ParentService(params=request.get_json())
return jsonify(data=service.get())
# test factories
class ModelFactory(SQLAlchemyModelFactory):
class Meta:
abstract = True
sqlalchemy_session = db.session
class ParentFactory(ModelFactory):
id = factory.Sequence(lambda n: n + 1)
children = factory.SubFactory('tests.factory.children.ChildFactory')
class ChildFactory(ModelFactory):
id = factory.Sequence(lambda n: n + 1)
parent = factory.SubFactory('tests.factory.parent.ParentFactory')
# tests
from unittest.mock import MagicMock, patch
def test_can_serialize_parents():
parents = ParentFactory.build_batch(4)
session = MagicMock()
service = ParentService(params={}, _session=session)
assert service.session is session
with patch.object(service, '_parents') as _parents:
_parents.return_value = parents
assert service.get()[0]['id'] == parents[0].id
assert service.get()[1]['id'] == parents[1].id
assert service.get()[2]['id'] == parents[2].id
assert service.get()[3]['id'] == parents[3].id

I would add a .json() method to each model, so that they call each other. It's essentially your "hacked" solution but a bit more readable/maintainable. Your Order model could have:
def json(self):
return {
"id": self.id,
"order_lines": [line.json() for line in self.order_lines]
}
Your OrderLine model could have:
def json(self):
return {
"product_id": self.product_id,
"product_name": self.product.name,
"quantity": self.quantity
}
Your resource at the top level (where you're making the request for orders) could then do:
...
orders = Order.query.all()
return {"orders": [order.json() for order in orders]}
...
This is how I normally structure this JSON requirement.

Check my answer in this thread Flask Sqlalchmey - Marshmallow Nested Schema fails for joins with filter ( where ) conditions and using the Marshmallow package you include in your schema something like this:
name = fields.Nested(Schema, many=True)

Filtering objects in Django based on optional arguments

Many times I find myself writing code similar to:
query = MyModel.objects.all()
if request.GET.get('filter_by_field1'):
query = query.filter(field1 = True)
if request.GET.get('filter_by_field2'):
query = query.filter(field2 = False)
field3_filter = request.GET.get('field3'):
if field3_filter is not None:
query = query.filter(field3 = field3_filter)
if field4_filter:
query = query.filter(field4 = field4_filter)
# etc...
return query
Is there a better, more generic way of building queries such as the one above?

If the only things that are ever going to be in request GET are potential query arguments, you could do this:
query = MyModel.objects.filter(**request.GET)

Separating "user-owned" from "other" data in Django template

I have an Openstack-powered, Django-modified application that shows the disk images and snapshots available for a user to launch. The user currently sees both snapshots they created and ones they did not. I would like to separate the current table into two based on whether they are owned by the user or not.
My two table definitions are as follows (note I altered row_actions accordingly):
class UserSnapshotsTable(OldSnapshotsTable):
cloud = tables.Column(get_cloud, verbose_name=_("Cloud"))
class Meta:
name = "usersnapshots"
verbose_name = _("User Snapshots")
table_actions = (DeleteSnapshot,)
row_actions = (LaunchSnapshot, LaunchCluster, EditImage, DeleteSnapshot)
pagination_param = "snapshot_marker"
row_class = UpdateRow
status_columns = ["status"]
class OtherSnapshotsTable(OldSnapshotsTable):
cloud = tables.Column(get_cloud, verbose_name=_("Cloud"))
class Meta:
name = "othersnapshots"
verbose_name = _("Other Snapshots")
table_actions = (DeleteSnapshot,)
row_actions = (LaunchSnapshot, LaunchCluster)
pagination_param = "snapshot_marker"
row_class = UpdateRow
status_columns = ["status"]
I have altered the HTML template to pull the "UserSnapshotsTable" and "OtherSnapshotsTable" tables (I copied the original table and renamed both), but both full tables still generate under the respective headings. There are two functions generating the data:
def get_usersnapshots_data(self):
req = self.request
marker = req.GET.get(UserSnapshotsTable._meta.pagination_param, None)
try:
usersnaps, self._more_snapshots = api.snapshot_list_detailed(req,
marker=marker)
except:
usersnaps = []
exceptions.handle(req, _("Unable to retrieve user-owned snapshots."))
return usersnaps
def get_othersnapshots_data(self):
req = self.request
marker = req.GET.get(OtherSnapshotsTable._meta.pagination_param, None)
try:
othersnaps, self._more_snapshots = api.snapshot_list_detailed(req,
marker=marker)
except:
othersnaps = []
exceptions.handle(req, _("Unable to retrieve non-user-owned snapshots."))
return othersnaps
There are also Edit/Delete options defined for images, and imported for snapshots, that seem to have a key comparison. Here's the "Delete" one (line 7):
class DeleteImage(tables.DeleteAction):
data_type_singular = _("Image")
data_type_plural = _("Images")
def allowed(self, request, image=None):
if image:
return image.owner == request.user.tenant_id
# Return True to allow table-level bulk delete action to appear.
return True
def delete(self, request, obj_id):
api.image_delete(request, obj_id)
How can I separate those tables out? This is my first time asking a question here, so please let me know if I can provide further information. Apologies for the length of it.

As far as I see you are using glanceclient. If that so you can use extra_filters parameter of snapshot_list_detailed() to filter only user images like this:
usersnaps, self._more_snapshots = api.snapshot_list_detailed(
req,
marker = marker,
extra_filters = {"owner": "user_name"}
)
Under cover snapshot_list_detailed uses GET images of Openstack Image Service API.

Union and Intersect in Django

class Tag(models.Model):
name = models.CharField(maxlength=100)
class Blog(models.Model):
name = models.CharField(maxlength=100)
tags = models.ManyToManyField(Tag)
Simple models just to ask my question.
I wonder how can i query blogs using tags in two different ways.
Blog entries that are tagged with "tag1" or "tag2":
Blog.objects.filter(tags_in=[1,2]).distinct()
Blog objects that are tagged with "tag1" and "tag2" : ?
Blog objects that are tagged with exactly "tag1" and "tag2" and nothing else : ??
Tag and Blog is just used for an example.

You could use Q objects for #1:
# Blogs who have either hockey or django tags.
from django.db.models import Q
Blog.objects.filter(
Q(tags__name__iexact='hockey') | Q(tags__name__iexact='django')
)
Unions and intersections, I believe, are a bit outside the scope of the Django ORM, but its possible to to these. The following examples are from a Django application called called django-tagging that provides the functionality. Line 346 of models.py:
For part two, you're looking for a union of two queries, basically
def get_union_by_model(self, queryset_or_model, tags):
"""
Create a ``QuerySet`` containing instances of the specified
model associated with *any* of the given list of tags.
"""
tags = get_tag_list(tags)
tag_count = len(tags)
queryset, model = get_queryset_and_model(queryset_or_model)
if not tag_count:
return model._default_manager.none()
model_table = qn(model._meta.db_table)
# This query selects the ids of all objects which have any of
# the given tags.
query = """
SELECT %(model_pk)s
FROM %(model)s, %(tagged_item)s
WHERE %(tagged_item)s.content_type_id = %(content_type_id)s
AND %(tagged_item)s.tag_id IN (%(tag_id_placeholders)s)
AND %(model_pk)s = %(tagged_item)s.object_id
GROUP BY %(model_pk)s""" % {
'model_pk': '%s.%s' % (model_table, qn(model._meta.pk.column)),
'model': model_table,
'tagged_item': qn(self.model._meta.db_table),
'content_type_id': ContentType.objects.get_for_model(model).pk,
'tag_id_placeholders': ','.join(['%s'] * tag_count),
}
cursor = connection.cursor()
cursor.execute(query, [tag.pk for tag in tags])
object_ids = [row[0] for row in cursor.fetchall()]
if len(object_ids) > 0:
return queryset.filter(pk__in=object_ids)
else:
return model._default_manager.none()
For part #3 I believe you're looking for an intersection. See line 307 of models.py
def get_intersection_by_model(self, queryset_or_model, tags):
"""
Create a ``QuerySet`` containing instances of the specified
model associated with *all* of the given list of tags.
"""
tags = get_tag_list(tags)
tag_count = len(tags)
queryset, model = get_queryset_and_model(queryset_or_model)
if not tag_count:
return model._default_manager.none()
model_table = qn(model._meta.db_table)
# This query selects the ids of all objects which have all the
# given tags.
query = """
SELECT %(model_pk)s
FROM %(model)s, %(tagged_item)s
WHERE %(tagged_item)s.content_type_id = %(content_type_id)s
AND %(tagged_item)s.tag_id IN (%(tag_id_placeholders)s)
AND %(model_pk)s = %(tagged_item)s.object_id
GROUP BY %(model_pk)s
HAVING COUNT(%(model_pk)s) = %(tag_count)s""" % {
'model_pk': '%s.%s' % (model_table, qn(model._meta.pk.column)),
'model': model_table,
'tagged_item': qn(self.model._meta.db_table),
'content_type_id': ContentType.objects.get_for_model(model).pk,
'tag_id_placeholders': ','.join(['%s'] * tag_count),
'tag_count': tag_count,
}
cursor = connection.cursor()
cursor.execute(query, [tag.pk for tag in tags])
object_ids = [row[0] for row in cursor.fetchall()]
if len(object_ids) > 0:
return queryset.filter(pk__in=object_ids)
else:
return model._default_manager.none()

I've tested these out with Django 1.0:
The "or" queries:
Blog.objects.filter(tags__name__in=['tag1', 'tag2']).distinct()
or you could use the Q class:
Blog.objects.filter(Q(tags__name='tag1') | Q(tags__name='tag2')).distinct()
The "and" query:
Blog.objects.filter(tags__name='tag1').filter(tags__name='tag2')
I'm not sure about the third one, you'll probably need to drop to SQL to do it.

Please don't reinvent the wheel and use django-tagging application which was made exactly for your use case. It can do all queries you describe, and much more.
If you need to add custom fields to your Tag model, you can also take a look at my branch of django-tagging.

This will do the trick for you
Blog.objects.filter(tags__name__in=['tag1', 'tag2']).annotate(tag_matches=models.Count(tags)).filter(tag_matches=2)