We Keep Coding

GAE NDB Confused about Models and duplicating attributes

I'm trying to learn Google App Engine's NDB and I'm confused about the structure of models.
My situation is similar to a CMS platform with Post Types (like in WordPress), so I have "Blogs" and "Pages". All of these Post Types require the same set of attributes: Parent, Name, Slug, Template, Content, Status, and Date.
So far, I gather that I need to create a Model for these like this:
class Post(ndb.Expando):
parent = ndb.StringProperty()
name = ndb.StringProperty()
slug = ndb.StringProperty()
template = ndb.StringProperty()
content = ndb.StringProperty(indexed=False)
status = ndb.StringProperty()
date = ndb.DateTimeProperty(auto_now_add=True)
(I'm using Expando because I will be adding "unknown" attributes in my application)
But with this structure, all of my posts (in every Post Type) will be within the same "kind", so queries will take longer (if I'm not mistaken).
How can I create many Models (kinds) with the same attributes?
Do I copy & paste the above Model under different class names?
Is it possible to create new Models dynamically (similar to "Custom Post Types" in WordPress)? Does it work if I use ndb.Key('Blog', blogid) instead of declaring a Model?
Do I create a Model called class PostType(ndb.Model) that stores the "Post Types" and give them ancestors of Posts? (If I'm not mistaken, this would cause problems because updating a Post would "lock" the entire ancestor tree for a second or so)
My primary goal is efficiency. Thanks!
Updates:
As written by Dan and mgilson, adding sub-classes of the main Post class Model is a good way to solve this:
class Post(ndb.Expando):
parent = ndb.StringProperty()
name = ndb.StringProperty()
slug = ndb.StringProperty()
template = ndb.StringProperty()
content = ndb.StringProperty(indexed=False)
status = ndb.StringProperty()
date = ndb.DateTimeProperty(auto_now_add=True)
class Blog(Post):
pass
However, this requires writing the Models statically. Is there a way to accomplish this dynamically (without declaring them as Models beforehand)?
Update:
Following the advice given below, I decided to keep all of my entities under the same kind. I might decide later on to change this to subclasses (separate kinds for each "Post Type") if my queries get messy. Thank you all for your great advice!

How can I create many Models with the same attributes?
You can subclass:
class SpecialPost(Post):
"""Special post type that is a different kind than Post."""
Though it's often easy enough to use the same kind and just add an extra field that represents the kind of post which you can filter on in queries.
Is it possible to create new Models dynamically (similar to "Custom Post Types" in WordPress)? Does it work if I use ndb.Key('Blog', blogid) instead of declaring a Model?
I'm not 100% sure that I understand what you're asking here. You can dynamically create models the same way you can dynamically create classes in python (using type), but you probably don't want to be doing this. Getting those dynamically created models (and keeping track of their names) will probably end up giving you serious headaches.

Basically a simple example of subclassing, which #mgilson mentioned already.
class Post(ndb.Expando):
parent = ndb.StringProperty()
name = ndb.StringProperty()
slug = ndb.StringProperty()
template = ndb.StringProperty()
content = ndb.StringProperty(indexed=False)
status = ndb.StringProperty()
date = ndb.DateTimeProperty(auto_now_add=True)
class Blog(Post):
someint = ndb.IntegerProperty()
blog = Blog(status='new', someint=2)
key = blog.put()
print key.kind()
As for dynamically creating models, from the Model's Constructor doc:
An application won't normally call Model(), but is likely to call the
constructor of a class that inherits from Model. This creates a new
instance of this model, also known as an entity.
Even if possible (I didn't dig too deep inside ndb/models.py to say with certainty that it's not) it doesn't appear a clear thing. Personally I'd stay away from that and instead re-think the need for such dynamically created models.

How can I create an entity in the GAE datastore of a type determined at runtime?

I have a URL route that captures the name of the model for which the entity will be created.(e.g. '/New<model>/') Now, I have the model name as a string. How can I take that string and use it to create a new entity of this model to put in the datastore?
I'm building a custom admin interface for GAE with the ability to register new models and have forms built dynamically based on registered models. I need a way to then be able to take the posted data submitted by the admin user and put it into the datastore using the correct model which will be in the URL the form is posted to. I'm currently building a handler to process requests coming into that URL scheme but have hit this roadblock. Any help is greatly appreciated.

Although you could access the model class with
globals()[modelname]
if the model class is defined at the module level, as #TimHoffman points out in the comments, it is generally not a good idea to allow (potentially malicious) users arbitrary access to your globals.
Since you are generating model classes dynamically, a safer way would be to save those model classes in a dict:
model = { 'MyModel': MyModelClass, ... }
and then given the string name modelname, you could access the class with
model[modelname]

I didn't use globals to get at my class constructor. Instead, I made a variable that points to the current module and used getattr to capture my class constructor. Here's my handler in case anyone else wants to see what I did.
class NewThingHandler(BaseRequestHandler):
def get(self, thing):
self.redirect('/admin/')
def post(self, thing):
this_module = sys.modules[__name__]
ThisModel = getattr(this_module, thing)
arguments = {}
for property in ThisModel.properties().keys():
if type(ThisModel._properties[property]) is db.DateProperty:
this_date = map(int, self.request.get(property).split('/'))
this_date = datetime.date(this_date[2], this_date[0], this_date[1])
arguments[property] = this_date
continue
arguments[property] = self.request.get(property)
new_thing = ThisModel(**arguments)
new_thing.put()
self.redirect('/admin/')
thing is captured from the URL.

ForeignKey to abstract class (generic relations)

I'm building a personal project with Django, to train myself (because I love Django, but I miss skills). I have the basic requirements, I know Python, I carefully read the Django book twice if not thrice.
My goal is to create a simple monitoring service, with a Django-based web interface allowing me to check status of my "nodes" (servers). Each node has multiple "services". The application checks the availability of each service for each node.
My problem is that I have no idea how to represent different types of services in my database. I thought of two "solutions" :
single service model, with a "serviceType" field, and a big mess with the fields. (I have no great experience in database modeling, but this looks... "bad" to me)
multiple service models. i like this solution, but then I have no idea how I can reference these DIFFERENT services in the same field.
This is a short excerpt from my models.py file : (I removed everything that is not related to this problem)
from django.db import models
# Create your models here.
class service(models.Model):
port = models.PositiveIntegerField()
class Meta:
abstract = True
class sshService(service):
username = models.CharField(max_length=64)
pkey = models.TextField()
class telnetService(service):
username = models.CharField(max_length=64)
password = models.CharField(max_length=64)
class genericTcpService(service):
pass
class genericUdpService(service):
pass
class node(models.Model):
name = models.CharField(max_length=64)
# various fields
services = models.ManyToManyField(service)
Of course, the line with the ManyToManyField is bogus. I have no idea what to put in place of "*Service". I honestly searched for solutions about this, I heard of "generic relations", triple-join tables, but I did'nt really understand these things.
Moreover, English is not my native language, so coming to database structure and semantics, my knowledge and understanding of what I read is limited (but that's my problem)

For a start, use Django's multi-table inheritance, rather than the abstract model you have currently.
Your code would then become:
from django.db import models
class Service(models.Model):
port = models.PositiveIntegerField()
class SSHService(Service):
username = models.CharField(max_length=64)
pkey = models.TextField()
class TelnetService(Service):
username = models.CharField(max_length=64)
password = models.CharField(max_length=64)
class GenericTcpService(Service):
pass
class GenericUDPService(Service):
pass
class Node(models.Model):
name = models.CharField(max_length=64)
# various fields
services = models.ManyToManyField(Service)
On the database level, this will create a 'service' table, the rows of which will be linked via one to one relationships with separate tables for each child service.
The only difficulty with this approach is that when you do something like the following:
node = Node.objects.get(pk=node_id)
for service in node.services.all():
# Do something with the service
The 'service' objects you access in the loop will be of the parent type.
If you know what child type these will have beforehand, you can just access the child class in the following way:
from django.core.exceptions import ObjectDoesNotExist
try:
telnet_service = service.telnetservice
except (AttributeError, ObjectDoesNotExist):
# You chose the wrong child type!
telnet_service = None
If you don't know the child type beforehand, it gets a bit trickier. There are a few hacky/messy solutions, including a 'serviceType' field on the parent model, but a better way, as Joe J mentioned, is to use a 'subclassing queryset'. The InheritanceManager class from django-model-utils is probably the easiest to use. Read the documentation for it here, it's a really nice little bit of code.

I think one approach that you might consider is a "subclassing queryset". Basically, it allows you to query the parent model and it will return instances of the child models in the result queryset. It would let you do queries like:
models.service.objects.all()
and have it return to you results like the following:
[ <sshServiceInstance>, <telnetServiceInstance>, <telnetServiceInstance>, ...]
For some examples on how to do this, check out the links on the blog post linked below.
http://jazstudios.blogspot.com/2009/10/django-model-inheritance-with.html
However, if you use this approach, you shouldn't declare your service model as abstract as you do in the example. Granted, you will be introducing an extra join, but overall I've found the subclassing queryset to work pretty well for returning a mixed set of objects in a queryset.
Anyway, hope this helps,
Joe

If you are looking for generic foreign key relations you should check the Django contenttypes framework (built into Django). The docs pretty much explain how to use it and how to work with generic relations.

An actual service can only be on one node, right? In that case when not have a field
node = models.ForeignKey('node', related_name='services')
in the service class?

How to create Form from a Model which has a ListProperty

I am currently using Django forms with the Google App Engine and I have a model which is as follows:
class Menu(db.Model):
name = db.StringProperty(required=True)
is_special = db.BooleanProperty()
menu_items = db.ListProperty(MenuItem)
I have a MenuForm which is the following:
class MenuForm(djangoforms.ModelForm):
class Meta:
model = Menu
exclude = ['added_by','menu_items']
When I run this I get the following error:
Exception Type: ValueError
Exception Value: Item type MenuItem is not acceptable
I want to crate the form and have it omit the menu_items property as for one I don't think there is an in built control for the multiple choice, like a group of check boxes. Either way I cannot understand with this property in the exclude items why it is throwing this error.
TIA
Andrew

Your problem comes well before the "create a form" task begins: ListProperty does not allow a list of model entities (although I can't find this clearly documented in the app engine docs, I'm still looking in the docs for a good, clear, unambiguous statement about that). Try changing it into (say) a list of strings, and you'll see everything works (I believe a dropdown is what you get if you don't exclude such a property).
Edit: found the spot in the docs where the issue is mentioned, although it's quaintly phrased -- quoting with added emphasis:
The list can contain values of any of
the value types supported by the
datastore.
...point is, you can have in the list objects of any of the value types... not reference ones, i.e., entities that are instances of some model.
Could you use a list of key strings, instead...?

Django labels and translations - Model Design

Lets say I have the following Django model:
class StandardLabel(models.Model):
id = models.AutoField(primary_key=True)
label = models.CharField(max_length=255)
abbreviation = models.CharField(max_length=255)
Each label has an ID number, the label text, and an abbreviation. Now, I want to have these labels translatable into other languages. What is the best way to do this?
As I see it, I have a few options:
1: Add the translations as fields on the model:
class StandardLabel(models.Model):
id = models.AutoField(primary_key=True)
label_english = models.CharField(max_length=255)
abbreviation_english = models.CharField(max_length=255)
label_spanish = models.CharField(max_length=255)
abbreviation_spanish = models.CharField(max_length=255)
This is obviously not ideal - adding languages requires editing the model, the correct field name depends on the language.
2: Add the language as a foreign key:
class StandardLabel(models.Model):
id = models.AutoField(primary_key=True)
label = models.CharField(max_length=255)
abbreviation = models.CharField(max_length=255)
language = models.ForeignKey('languages.Language')
This is much better, now I can ask for all labels with a certain language, and throw them into a dict:
labels = StandardLabel.objects.filter(language=1)
labels = dict((x.pk, x) for x in labels)
But the problem here is that the labels dict is meant to be a lookup table, like so:
x = OtherObjectWithAReferenceToTheseLabels.object.get(pk=3)
thelabel = labels[x.labelIdNumber].label
Which doesn't work if there is a row per label, possibly with multiple languages for a single label. To solve that one, I need another field:
class StandardLabel(models.Model):
id = models.AutoField(primary_key=True)
group_id = models.IntegerField(db_index=True)
label = models.CharField(max_length=255)
abbreviation = models.CharField(max_length=255)
language = models.ForeignKey('languages.Language')
class Meta:
unique_together=(("group_id", "language"),)
#and I need to group them differently:
labels = StandardLabel.objects.filter(language=1)
labels = dict((x.group_id, x) for x in labels)
3: Throw label text out into a new model:
class StandardLabel(models.Model):
id = models.AutoField(primary_key=True)
text = models.ManyToManyField('LabelText')
class LabelText(models.Model):
id = models.AutoField(primary_key=True)
label = models.CharField(max_length=255)
abbreviation = models.CharField(max_length=255)
language = models.ForeignKey('languages.Language')
labels = StandardLabel.objects.filter(text__language=1)
labels = dict((x.pk, x) for x in labels)
But then this doesn't work, and causes a database hit every time I reference the label's text:
x = OtherObjectWithAReferenceToTheseLabels.object.get(pk=3)
thelabel = labels[x.labelIdNumber].text.get(language=1)
I've implemented option 2, but I find it very ugly - i don't like the group_id field, and I can't think of anything better to name it. In addition, StandardLabel as i'm using it is an abstract model, which I subclass to get different label sets for different fields.
I suppose that if option 3 /didn't/ hit the database, it's what I'd choose. I believe the real problem is that the filter text__language=1 doesn't cache the LabelText instances, and so the DB is hit when I text.get(language=1)
What are your thoughts on this? Can anyone recommend a cleaner solution?
Edit: Just to make it clear, these are not form labels, so the Django Internationalization system doesn't help.

Another option you might consider, depending on your application design of course, is to make use of Django's internationalization features. The approach they use is quite common to the approach found in desktop software.
I see the question was edited to add a reference to Django internationalization, so you do know about it, but the intl features in Django apply to much more than just Forms; it touchs quite a lot, and needs only a few tweaks to your app design.
Their docs are here: http://docs.djangoproject.com/en/dev/topics/i18n/#topics-i18n
The idea is that you define your model as if there was only one language. In other words, make no reference to language at all, and put only, say, English in the model.
So:
class StandardLabel(models.Model):
abbreviation = models.CharField(max_length=255)
label = models.CharField(max_length=255)
I know this looks like you've totally thrown out the language issue, but you've actually just relocated it. Instead of the language being in your data model, you've pushed it to the view.
The django internationalization features allow you to generate text translation files, and provides a number of features for pulling text out of the system into files. This is actually quite useful because it allows you to send plain files to your translator, which makes their job easier. Adding a new language is as easy as getting the file translated into a new language.
The translation files define the label from the database, and a translation for that language. There are functions for handling the language translation dynamically at run time for models, admin views, javascript, and templates.
For example, in a template, you might do something like:
<b>Hello {% trans "Here's the string in english" %}</b>
Or in view code, you could do:
# See docs on setting language, or getting Django to auto-set language
s = StandardLabel.objects.get(id=1)
lang_specific_label = ugettext(s.label)
Of course, if your app is all about entering new languages on the fly, then this approach may not work for you. Still, have a look at the Internationalization project as you may either be able to use it "as is", or be inspired to a django-appropriate solution that does work for your domain.

I would keep things as simple as possible. The lookup will be faster and the code cleaner with something like this:
class StandardLabel(models.Model):
abbreviation = models.CharField(max_length=255)
label = models.CharField(max_length=255)
language = models.CharField(max_length=2)
# or, alternately, specify language as a foreign key:
#language = models.ForeignKey(Language)
class Meta:
unique_together = ('language', 'abbreviation')
Then query based on abbreviation and language:
l = StandardLabel.objects.get(language='en', abbreviation='suite')

I'd much prefer to add a field per language than a new model instance per language. It does require schema alteration when you add a new language, but that isn't hard, and how often do you expect to add languages? In the meantime, it'll give you better database performance (no added joins or indexes) and you don't have to muck up your query logic with translation stuff; keep it all in the templates where it belongs.
Even better, use a reusable app like django-transmeta or django-modeltranslation that makes this stupid simple and almost completely transparent.

Although I would go with Daniel's solution, here is an alternative from what I've understood from your comments:
You can use an XMLField or JSONField to store your language/translation pairs. This would allow your objects referencing your labels to use a single id for all translations. And then you can have a custom manager method to call a specific translation:
Label.objects.get_by_language('ru', **kwargs)
Or a slightly cleaner and slightly more complicated solution that plays well with admin would be to denormalize the XMLField to another model with many-to-one relationship to the Label model. Same API, but instead of parsing XML it could query related models.
For both suggestions there's a single object where users of a label will point to.
I wouldn't worry about the queries too much, Django caches queries and your DBMS would probably have superior caching there as well.