I'm relatively new to Python, coming from the PHP world. In PHP, I would routinely fetch an row, which would correspond to and object from the database, say User, and add properties to it before passing the user object to my view page.
For example, the user has properties email, name and id.
I get 5 users from the database and in a for loop, I assign a dynamic property to the user, say image.
This doesn't seem to work in Python/Google App Engine datastore models (I think it has to do more with the datastore model than python) in a for loop. It works within the for loop (meaning I can reference user.image within the for loop, but once the for loop ends, all of the objects seem to not have the new attribute image anymore.
Here is a code example:
# Model
Class User(ndb.Model):
email = ndb.StringProperty()
name = ndb.StringProperty()
# And then a function that returns a list of users
users = User.get_users()
user_list = []
# For loop
for user in user:
# For example, get image
user.image = Image.get_image(user.key)
user_list.append(user)
# If I print or log this user in the for loop, I see a result
logging.info(user.image) # WORKS!
for ul in user_list:
print ul.image # Results in None/ATTR Error
Can anyone explain to me why this is happening and how to achieve this goal?
I've searched the forms, but I couldn't find anything.
Try using Expando Model
Sometimes you don't want to declare your properties ahead of time. A
special model subclass, Expando, changes the behavior of its entities
so that any attribute assigned (as long as it doesn't start with an
underscore) is saved to the Datastore.
Related
As title suggests, I want to remove all contents of the database. I'm making an application in Python using Flask and MongoAlchemy for database. I have the class People which models the documents. The structure is:
class People(db.Document):
Name = db.StringField()
Age = db.IntField()
Password = db.StringField()
Vms = db.AnythingField()
With the remove method I can delete only one document at once, based on the name or other detail, but I want to be able to delete them all at once.
This is my function for deleting a document, based on name.
def delete_document(name):
x = People.query.filter(People.Name == name).first()
x.remove()
Where People is the reference to the model class.
Apparently there is a easy way, using one of the built-in methods from Query class, remove_all. With the remove_all method you can get all the results from database in a list, and after that you can call the remove method on every instance. The code is posted below. Not sure if there is a more efficient way of doing this, so feel free to post it.
def deleteAll():
x = People.query.all()
for i in range(0, len(x)):
x[i].remove()
I have the User model in my datastore which contains some attributes:
I need to query all users filtering by the company attribute.
So, as I would normally do, I do this:
from webapp2_extras.appengine.auth.models import User
employees = User.query().filter(User.company == self.company_name).fetch()
This gives me:
AttributeError: type object 'User' has no attribute 'company'
And when I do:
employees = User.query().filter().fetch()
It gives me no error and shows the list with all the Users.
How do I query by field? Thanks
Your question is a bit misdirected. You ask how to query by field, which you are already doing with correct syntax. The problem, as Jeff O'Neill noted, is your User model does not have that company field, so your query-by-field attempt results in an error. (Here is some ndb documentation that you should definitely peruse and bookmark if you haven't already.) There are three ways to remedy your missing-field problem:
Subclass the User model, as Jeff shows in his answer. This is quick and simple, and may be the best solution for what you want.
Create your own User model, completely separate from the webapp2 one. This is probably overkill for what you want, just judging from your question, because you would have to write most of your own authentication code that the webapp2 user already handles for you.
Create a new model that contains extra user information, and has a key property containing the corresponding user's key. That would look like this:
class UserProfile(ndb.Expando):
user_key = ndb.KeyProperty(kind='User', required=True)
company = ndb.StringProperty()
# other possibilities: profile pic? address? etc.
with queries like this:
from models.user_profile import UserProfile
from webapp2_extras.appengine.auth.models import User
from google.appengine.ext import ndb
# get the employee keys
employee_keys = UserProfile.query(UserProfile.company == company_name).fetch(keys_only=True)
# get the actual user objects
employees = ndb.get_multi(employee_keys)
What this solution does is it separates your User model that you use for authentication (webapp2's User) from the model that holds extra user information (UserProfile). If you want to store profile pictures or other relatively large amounts of data for each user, you may find this solution works best for you.
Note: you can put your filter criteria in the .query() parentheses to simplify things (I find I rarely use the .filter() method):
# long way
employees = User.query().filter(User.company == self.company_name).fetch()
# shorter way
employees = User.query(User.company == self.company_name).fetch()
You've imported a User class defined by webapp2. This User class does not have an attribute called company so that is why you are getting the error from User.company.
You probably want to do create your own User model by subclassing the one provided by webapp2:
from webapp2_extras.appengine.auth.models import User as Webapp2_User
class User(Webapp2_User):
company = ndb.StringProperty()
Then your query should work.
One caveat, I've never used webapp2_extras.appengine.auth.models so I don't know what that is exactly.
I am currently trying to create a CustomUser entity in my app engine project upon a user signing in for the first time. I would like CustomUser entities to be unique, and I would like to prevent the same entity from being created more than once. This would be fairly easy to do if I can supply it with an ancestor upon entity creation, as this will make the transaction strongly consistent.
Unfortunately, this is not the case, due to the fact that a CustomUser entity is a root entity, and it will thus be eventually consistent, not strongly consistent. Because of this, there are instances when the entity is created twice, which I would like to prevent as this will cause problems later on.
So the question is, is there a way I can prevent the entity from being created more than once? Or at least make the commit of the ancestor entity strongly consistent to prevent duplication? Here's my code, and interim (hacky) solution.
import time
import logging
from google.appengine.ext import ndb
# sample Model
class CustomUser(ndb.Model):
user_id = ndb.StringProperty(required=True)
some_data = ndb.StringProperty(required=True)
some_more_data = ndb.StringProperty(required=True)
externally_based_user_id = "id_taken_from_somewhere_else"
# check if this id already exists in the Model.
# If it does not exist yet, create it
user_entity = CustomUser.query(
CustomUser.user_id == externally_based_user_id,
ancestor=None).get()
if not user_entity:
# prepare the entity
user_entity = CustomUser(
user_id=externally_based_user_id,
some_data="some information",
some_more_data="even more information",
parent=None
)
# write the entity to ndb
user_key = user_entity.put()
# inform of success
logging.info("user " + str(user_key) + " created")
# eventual consistency workaround - loop and keep checking if the
# entity has already been created
#
# I understand that a while loop may not be the wisest solution.
# I can also use a for loop with n range to avoid going around the loop infinitely.
# Both however seem to be band aid solutions
while not entity_check:
entity_check = CustomUser.query(
CustomUser.user_id == externally_based_user_id,
ancestor=None).get()
# time.sleep to prevent the instance from consuming too much processing power and
# memory, although I'm not certain if this has any real effect apart from
# reducing the number of loops
if not entity_check:
time.sleep(0.5)
EDIT: Solution I ended up using based on both of Daniel Roseman's suggestions. This can be further simplified by using get_or_insert as suggested by voscausa. I've stuck to using the usual way of doing things to make things clearer.
import logging
from google.appengine.ext import ndb
# ancestor Model
# we can skip the creation of an empty class like this, and just use a string when
# retrieving a key
class PhantomAncestor(ndb.Model):
pass
# sample Model
class CustomUser(ndb.Model):
# user_id now considered redundance since we will be
# user_id = ndb.StringProperty(required=True)
some_data = ndb.StringProperty(required=True)
some_more_data = ndb.StringProperty(required=True)
externally_based_user_id = "id_taken_from_somewhere_else"
# construct the entity key using information we know.
# entity_key = ndb.Key(*arbitrary ancestor kind*, *arbitrary ancestor id*, *Model*, *user_id*)
# we can also use the string "PhantomAncestor" instead of passing in an empty class like so:
# entity_key = ndb.Key("SomeRandomString", externally_based_user_id, CustomUser, externally_based_user_id)
# check this page on how to construct a key: https://cloud.google.com/appengine/docs/python/ndb/keyclass#Constructors
entity_key = ndb.Key(PhantomAncestor, externally_based_user_id, CustomUser, externally_based_user_id)
# check if this id already exists in the Model.
user_entity = entity_key.get()
# If it does not exist yet, create it
if not user_entity:
# prepare the entity with the desired key value
user_entity = CustomUser(
# user_id=externally_based_user_id,
some_data="some information",
some_more_data="even more information",
parent=None,
# specify the custom key value here
id=externally_based_user_id
)
# write the entity to ndb
user_key = user_entity.put()
# inform of success
logging.info("user " + str(user_key) + " created")
# we should also be able to use CustomUser.get_and_insert to simplify the code further
A couple of things here.
First, note that the ancestor doesn't have to actually exist. If you want a strongly consistent query, you can use any arbitrary key as an ancestor.
A second option would be to use user_id as your key. Then you can do a key get, rather than a query, which again is strongly consistent.
I am using appengine with python 2.7 and webapp2 framework. I am not using ndb.model.
I have the following model:
class Story(db.Model);
name = db.StringProperty()
class UserProfile(db.Model):
name = db.StringProperty()
user = db.UserProperty()
class Tracking(db.Model):
user_profile = db.ReferenceProperty(UserProfile)
story = db.ReferenceProperty(Story)
upvoted = db.BooleanProperty()
flagged = db.BoolenProperty()
A user can upvote and/or flag a story but only once. Hence I came up with the above model.
Now when a user clicks on the upvote link, on the database I try to see if the user has not already voted it, hence I do try to do the following:
get the user instance with his id as up = db.get(db.Key.from_path('UserProfile', uid))
then get the story instance as follows s_ins = db.get(db.Key.from_path('Story', uid))
Now it is the turn to check if a Tracking based on these two exist, if yes then don't allow voting, else allow him to vote and update the Tracking instance.
What is the most convenient way to fetch a Tracking instance given an id(db.key().id()) of user_profile and story?
What is the most convenient way to save a Tracking model having given a user profile id and an story id?
Is there a better way to implement tracking?
You can try tracking using lists of keys versus having a separate entry for track/user/story:
class Story(db.Model);
name = db.StringProperty()
class UserProfile(db.Model):
name = db.StringProperty()
user = db.UserProperty()
class Tracking(db.Model):
story = db.ReferenceProperty(Story)
upvoted = db.ListProperty(db.Key)
flagged = db.ListProperty(db.Key)
So when you want to see if a user upvoted for a given story:
Tracking.all().filter('story =', db.Key.from_path('Story', uid)).filter('upvoted =', db.Key.from_path('UserProfile', uid)).get(keys_only=True)
Now the only problem here is the size of the upvoted/flagged lists can't grow too large (I think the limit is 5000), so you'd have to make a class to manage this (that is, when adding to the upvoted/flagged lists, detect if X entries exists, and if so, start a new tracking object to hold additional values). You will also have to make this transactional and with HR you have a 1 write per second threshold. This may or may not be an issue depending on your expected use case. A way around the write threshold would be to implement upvotes/flags using pull-queues and to have a cron job that pulls and batch updates tracking objects as needed.
This method has its pros/cons. The most obvious cons are the ones I just listed. The pros, however, may be worth it. You can get a full list of users who upvoted/flagged a story from a single list (or multiple depending on how popular the story is). You can get a full list of users with a lot fewer queries to the datastore. This method should also take less storage, index, and metadata space. Additionally, adding a user to a tracking object will be cheaper, instead of writing a new object + 2 writes for each property, you would just be charged 1 write for the object + 2 writes for the entry to the list (9 vs 3 writes for adding users to a pre-existing tracked story, or 9 vs 7 for untracked stories)
What you propose sounds reasonable.
Don't use the app engine generated key for Tracking. Because the combination of story/user should be unique, create your own key as a combination of the story/user. Something like
tracking = Tracking.get_or_insert(str(story.id) + "-" + str(user.id), **params)
If you know the story/user, then you can always fetch the tracking by key name.
I'm trying to solve a bug. The following functions adds a Key object to a user's attribute called tabs. For some reason, after calling put() on the user entity, the newly added key isn't saved. Couldn't figure out why. Maybe there is some delay that prevents the changes from appearing immediately? In that case is memcache the solution?
class User(GeoModel):
tabs = db.ListProperty(db.Key)
#db.transactional
def add_tab_transaction(self, user_key, tab_key):
user = models.User.get(user_key)
user.tabs.append(tab_key)
user.put()
logging.debug('tabs this user has:')
logging.debug(user.tabs) # prints the list with the new value
user = models.User.get(user_key)
logging.debug('rechecking the same thing:')
logging.debug(user.tabs) # prints the list without the new value
This behaviour is explained in Isolation and Consistency.
"In a transaction, all reads reflect the current, consistent state of the Datastore at the time the transaction started. This does not include previous puts and deletes inside the transaction."