I have a GAE app using NDB datastore and python which assigns tasks to employees. I have Task Entities and Employee Entities which have arrays of Tasks (storing the tasks' keys). I am trying to implement a "cascading delete" where I can pass my delete function the key of the task to delete, and have it "cascade" to employee entities to clean up references to that task. Right now my delete task function works fine but it does not cascade correctly. When I delete a task and check out an employee who has been assigned that task, its key value still shows. I would greatly appreciate any pointers anyone can provide!
My entity definitions are in a db_models file, with Task entities (consisting only of name as a string) and Employee entities which have arrays of tasks:
class Employee(ndb.Model):
name = ndb.StringProperty(required=True)
title = ndb.StringProperty(required=True)
tasks = ndb.KeyProperty(repeated=True)
def to_dict(self):
d = super(Employee, self).to_dict()
d['tasks'] = [m.id() for m in d['tasks']]
return d
My delete function, which I am passing the 'did' or the key of the Task entity to delete
class TaskDelete(webapp2.RequestHandler):
def get(self, **kwargs):
if 'application/json' not in self.request.accept:
webapp2.abort(406, details="Not Acceptable, API only supports application/json MIME type")
return
if 'did' in kwargs:
entity = ndb.Key(db_models.Task, int(kwargs['did'])).delete()
q = db_models.Employee.query()
key = q.fetch(keys_only=True)
for x in key:
employee = ndb.Key(db_models.Employee, int(x.id())).get()
for task in employee.tasks:
if 'did' == task:
task.delete()
employee.put()
First of all, you are requesting Employees one at a time and that is very slow. Instead of:
q = db_models.Employee.query()
key = q.fetch(keys_only=True)
for x in key:
employee = ndb.Key(db_models.Employee, int(x.id())).get()
use:
for employee in db_models.Employee.query():
Now you simply need to update your employee.tasks property:
for task in employee.tasks:
if 'did' == task:
task.delete()
employee.tasks.remove(task) # add this line
employee.put()
break # add this line too
Related
In google app engine, say I have a Parent and a Child Entity:
class Parent(ndb.Model):
pass
class Child(ndb.Model):
parent_key = ndb.KeyProperty(indexed = True)
... other properties I don't need to fetch ...
I have a list of parents' keys, say parents_list, and I'm trying to answer efficiently: what parent in parents_list has a child.
Ideally, I would run this query:
children_list = Child.query().filter(Child.parent_key = parents_list).fetch(projection = 'parent_key')
It does not work because of the projection property (parent_key) being in the equality filter. So I would have to retrieve all properties, which seems inefficient.
Is there a way to efficiently solve this?
Your child model should actually be
class Child(ndb.Model):
parent_key = ndb.KeyProperty(kind="Parent", indexed = True)
If you were doing this in Python2, you could use an ndb_tasklet (see code below; note that I haven't executed this code myself so it's not guaranteed to work; it's just here to serve as a guide but I have used tasklets in the past before). If python3, try and create async queries
class Parent(ndb.Model):
#classmethod
def parentsWithChildren(cls, parents_list):
#ndb.tasklet
def child_callback(parent_key):
q = Child.query(Child.parent_key == parent_key)
output = yield q.fetch_async(projection = 'parent_key')
raise ndb.Return ((parentKey, output))
# For each key in parents_list, invoke the callback - child_callback which returns a result only if the parent_key matches
# ndb.tasklet is asynchronous so code is run in parallel
final_output = [ child_callback(a) for a in parents_list]
return final_output
I have a task that creates a task history record every time a new task is processed. I instantiate a new TaskHistory instance at the beginning of the task function. For some reason TaskHistory class attributes assigned in a previous task are being assigned to TaskHistory class attributes in subsequent tasks. For example, if the task succeeds I assign:
task_history.meta['success'] = 'Successfully processed {} rows '.format(row_count)'
In a subsequent task it might fail and I assign:
task_history.meta['error'] = 'Processed {} rows '.format(row_count) + str(e)
The subsequent task should only assign meta['error'] but it's assigning the previous meta['success'] value as well even though the task_history was reinstantiated.
Below is the code that calls the task:
args = [file_ids]
kwargs = {'requester': request.user.profile}
csv_import.apply_async(args=args, kwargs=kwargs)
Below is the task function:
#task
def csv_import(file_ids, requester=None):
task_history = TaskHistory()
task_history.requester = requester
task_history.status = 'pending'
task_history.started = timezone.now()
task_history.save()
row_count = 0
try:
//main logic goes here
task_history.status = 'completed'
task_history.completed = now()
task_history.meta['success'] = 'Successfully processed {} rows '.format(row_count)
task_history.save()
except Exception as e:
task_history.status = 'failed'
task_history.completed = timezone.now()
task_history.meta['error'] = 'Processed {} rows '.format(row_count) + str(e)
task_history.save()
raise Exception
In the end this had nothing to do with celery but with the way I set the default value for my meta field in my model. The meta field is a JSONfield() and I set the default value to default={} which creates a mutable instance and is shared between all instances of JSONfield. I fixed this by setting it to a callable dict. The new field now appears as followed:
meta = JSONField(default=dict)
More information about this can be found here https://docs.djangoproject.com/es/1.9/ref/contrib/postgres/fields/#jsonfield. This is the same standard that should be used for all model default values.
This is the code:
def create_game(user_id):
game = classes.Games(user_id = user_id)
game.put()
def get_game(user_id):
game_query = classes.Games.gql('WHERE user_id = :1', user_id)
game = False
for item in game_query:
game = item
if not game:
create_game(user_id)
get_game(user_id)
else:
return game
def render_page(self):
message = 'this is a game page<br>'.decode('utf-8')
user = creditentials.get_user(self)
if not user:
self.redirect('/')
return
game = get_game(user.key().id())
message += 'current game ID: '.decode('utf-8') + str(game.key().id())
self.response.write(message)
I expect it to create just one instance of the game, instad it creates 10! Appearantly GQL query is perfromed asynchronously, and starting from the 3rd (?) instance of the get_game(user_id) it just skips game_query = classes.Games.gql('WHERE user_id = :1', user_id) line.
Am I right? How do I avoid this?
Queries aren't immediately consistent, so an entity that you've only just created won't be returned in a query performed right-away, you need to wait a bit.
In your case, you don't need to query for the entity - you just created it, so you know it exists, and can use it. Change your create_game function to return the new game, and then use that.
If you expect your user_id to be unique (and given your query, this seems to be the case) you could use it as the entity-id, then you can get-by-id instead of querying, which will strongly-consistent.
Check the "Data Consistency" section on this docs page for more detail on how queries work.
From google documentation:
"A model instance's key includes the instance's entity kind along with a unique identifier. The identifier may be either a key name string, assigned explicitly by the application when the instance is created, or an integer numeric ID, assigned automatically by App Engine when the instance is written (put) to the Datastore. "
so in the example:
name = "John"
idd = 11
person = Person(name, idd)
person.put()
How do i get the "integer numeric ID, assigned automatically by App Engine"?
if you are using ndb put() returns the new key... call the id function on the key:
name = "John"
idd = 11
person = Person(name, idd)
new_key = person.put()
auto_assigned_id = new_key.id()
from https://developers.google.com/appengine/docs/python/ndb/entities :
To store the object as a persistent entity in the Datastore, use the
put() method. This returns a key for retrieving the entity from the
Datastore later:
sandy_key = sandy.put()
and:
https://developers.google.com/appengine/docs/python/ndb/keyclass#Key_id
Have you tried the
print person.id()
or if you have provided the unique identifuer
print person.id_or_name()
Also the put() method returns the key
key = person.put()
I'm trying to make an appraisal system
This is my class
class Goal(db.Expando):
GID = db.IntegerProperty(required=True)
description = db.TextProperty(required=True)
time = db.FloatProperty(required=True)
weight = db.IntegerProperty(required=True)
Emp = db.UserProperty(auto_current_user=True)
Status = db.BooleanProperty(default=False)
Following things are given by employee,
class SubmitGoal(webapp.RequestHandler):
def post(self):
dtw = simplejson.loads(self.request.body)
try:
maxid = Goal.all().order("-GID").get().GID + 1
except:
maxid = 1
try:
g = Goal(GID=maxid, description=dtw[0], time=float(dtw[1]), weight=int(dtw[2]))
g.put()
self.response.out.write(simplejson.dumps("Submitted"))
except:
self.response.out.write(simplejson.dumps("Error"))
Now, here Manager checks the goals and approve it or not.. if approved then status will be stored as true in datastore else false
idsta = simplejson.loads(self.request.body)
try:
g = db.Query(Goal).filter("GID =", int(idsta[0])).get()
if g:
if idsta[1]:
g.Status=True
try:
del g.Comments
except:
None
else:
g.Status=False
g.Comments=idsta[2]
db.put(g)
self.response.out.write(simplejson.dumps("Submitted"))
except:
self.response.out.write(simplejson.dumps("Error"))
Now, this is where im stuck..."filter('status=',True)".. this is returning all the entities which has status true.. means which are approved.. i want those entities which are approved AND which have not been assessed by employee yet..
def get(self):
t = []
for g in Goal.all().filter("Status = ",True):
t.append([g.GID, g.description, g.time, g.weight, g.Emp])
self.response.out.write(simplejson.dumps(t))
def post(self):
idasm = simplejson.loads(self.request.body)
try:
g = db.Query(Goal).filter("GID =", int(idasm[0])).get()
if g:
g.AsmEmp=idasm[1]
db.put(g)
self.response.out.write(simplejson.dumps("Submitted"))
except:
self.response.out.write(simplejson.dumps("Error"))
How am I supposed to do this? as I know that if I add another filter like "filter('AsmEmp =', not None)" this will only return those entities which have the AsmEmp attribute what I need is vice versa.
You explicitly can't do this. As the documentation states:
It is not possible to query for entities that are missing a given property.
Instead, create a property for is_assessed which defaults to False, and query on that.
could you not simply add another field for when employee_assessed = db.user...
and only populate this at the time when it is assessed?
The records do not lack the attribute in the datastore, it's simply set to None. You can query for those records with Goal.all().filter('status =', True).filter('AsmEmp =', None).
A few incidental suggestions about your code:
'Status' is a rather unintuitive name for a boolean.
It's generally good Python style to begin properties and attributes with a lower-case letter.
You shouldn't iterate over a query directly. This fetches results in batches, and is much less efficient than doing an explicit fetch. Instead, fetch the number of results you need with .fetch(n).
A try/except with no exception class specified and no action taken when an exception occurs is a very bad idea, and can mask a wide variety of issues.
Edit: I didn't notice that you were using an Expando - in which case #Daniel's answer is correct. There doesn't seem to be any good reason to use Expando here, though. Adding the property to the model (and updating existing entities) would be the easiest solution here.