I've added a list property to an entity model with a large number of existing instances.
class MyModel(db.Model):
new_property = db.ListProperty(item_type=str, default=None)
Upon deployment to the live environment the app runs without issues for a short time and then starts throwing BadValueError error's as it tries to retrieve records from the datastore.
The code throwing the error is just a straight call to the datastore:
app_item = db.get(app_item_key)
I'm using 1.7.5. of the Python 2.7 runtime.
Any ideas on what I can do to prevent this, or at least trap it so that I can get data from the store?
Traceback (most recent call last):
File "/base/data/home/apps/app/4-15.365909351579418812/app.py", line 1739, in app_get
app_item = db.get(app_item_key)
File "/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 1533, in get
return get_async(keys, **kwargs).get_result()
File "/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
return self.__get_result_hook(self)
File "/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1459, in __get_hook
entities = rpc.user_data(entities)
File "/python27_runtime/python27_lib/versions/1/google/appengine/api/datastore.py", line 600, in local_extra_hook
return extra_hook(result)
File "/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 1503, in extra_hook
model = cls1.from_entity(entity)
File "/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 1438, in from_entity
return cls(None, _from_entity=entity, **entity_values)
File "/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 970, in __init__
prop.__set__(self, value)
File "/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
value = self.validate(value)
File "/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 3460, in validate
value = super(ListProperty, self).validate(value)
File "/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 641, in validate
raise BadValueError('Property %s is required' % self.name)
BadValueError: Property new_property is required
For those that follow:
As per Aaron D's suggestion, changing the default value to an empty list resolved this issue, so:
new_property = db.ListProperty(item_type=str, default=None)
Should read:
new_property = db.ListProperty(item_type=str, default=[])
Looking at the source code of the Google App Engine in the __init__.py referenced in your traceback, you can see a comment in the ListProperty doc comments (line 3428) that says:
Note that the only permissible value for 'required' is True.
So, even though you are not providing it, it looks like line 3442 is setting it automatically:
self._require_parameter(kwds, 'required', True)
If you look further into the source code (line 3500), you can see the definition of empty() for a ListProperty:
def empty(self, value):
"""Is list property empty.
[] is not an empty value.
Returns:
True if value is None, else false.
"""
return value is None
I could think of two issues that might cause error but I haven't verified through testing.
1) If for some reason, you already have data in that field (perhaps you are reusing the new_property name?) and it was empty, then it seems likely to generate the error you have. I am not sure how to fix this problem, except to suggest that you use a unique name for your new_property instead. The post I referenced in my comment explains how to "fix" the data.
2) Since you already have records, your code is trying to populate those using your default value of None, which matches the empty() test and then throws the exception. In that case, if you just provide a default value of [] instead, it should work.
I am pretty sure you example code here is not what you are using. I would bet you have required=True in the new property. You are then retrieving an old record which doesn't have a value for the required property. Just dropping 'required=True` will make those errors go away. If you need to have that value required then you need to add the default value to the field before enforcing the constraint.
* removed some complete garbage about None not being a valid default value for ListProperty *
So I tried to replicate the situation based on the information you have supplied
and I have the answer. I can generate the problem by first creating a model that has a name new_property of type StringProperty with a default of None. put() the record with no value for new_property getting the default of None written, then change the model definition of new_property toListProperty`, and the fetch the record. We get the same stack trace. See shell log below.
s~lightning-catfish> class MyModel(db.Model):
... pass
...
s~lightning-catfish> x = MyModel()
s~lightning-catfish> x.put()
datastore_types.Key.from_path(u'MyModel', 1001L, _app=u's~lightning-catfish')
s~lightning-catfish> class MyModel(db.Model):
... new_property = db.ListProperty(item_type=str,default=None)
...
s~lightning-catfish> y = db.get(x.key())
s~lightning-catfish> y
<MyModel object at 0x9e09dcc>
s~lightning-catfish> y.new_property
[]
s~lightning-catfish> new_property = db.StringProperty(defaul
KeyboardInterrupt
s~lightning-catfish> class MyModel(db.Model):
... new_property = db.StringProperty(default=None)
...
s~lightning-catfish> z = MyModel()
s~lightning-catfish> z.put()
datastore_types.Key.from_path(u'MyModel', 2001L, _app=u's~lightning-catfish')
s~lightning-catfish> class MyModel(db.Model):
... new_property = db.ListProperty(item_type=str,default=None)
...
s~lightning-catfish> a1 = db.get(z.key())
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/timh/google_appengine/google/appengine/ext/db/__init__.py", line 1533, in get
return get_async(keys, **kwargs).get_result()
File "/home/timh/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
return self.__get_result_hook(self)
File "/home/timh/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1459, in __get_hook
entities = rpc.user_data(entities)
File "/home/timh/google_appengine/google/appengine/api/datastore.py", line 600, in local_extra_hook
return extra_hook(result)
File "/home/timh/google_appengine/google/appengine/ext/db/__init__.py", line 1503, in extra_hook
model = cls1.from_entity(entity)
File "/home/timh/google_appengine/google/appengine/ext/db/__init__.py", line 1438, in from_entity
return cls(None, _from_entity=entity, **entity_values)
File "/home/timh/google_appengine/google/appengine/ext/db/__init__.py", line 970, in __init__
prop.__set__(self, value)
File "/home/timh/google_appengine/google/appengine/ext/db/__init__.py", line 614, in __set__
value = self.validate(value)
File "/home/timh/google_appengine/google/appengine/ext/db/__init__.py", line 3460, in validate
value = super(ListProperty, self).validate(value)
File "/home/timh/google_appengine/google/appengine/ext/db/__init__.py", line 641, in validate
raise BadValueError('Property %s is required' % self.name)
BadValueError: Property new_property is required
s~lightning-catfish>
To fix the data you will need to access it at a low level and change the data types stored in the record.
I have code for fetching and putting entities without using models if you want it.
* last thing you should try *
Try using the following code or something like it to fetch objects without using the model.
You get the underlying data back, with types etc.. in dicts. That will show you what is in the datastore.
from google.appengine.api import datastore
from google.appengine.api import datastore_errors
def get_entities(keys):
rpc = datastore.GetRpcFromKwargs({})
keys, multiple = datastore.NormalizeAndTypeCheckKeys(keys)
entities = None
try:
entities = datastore.Get(keys, rpc=rpc)
except datastore_errors.EntityNotFoundError:
assert not multiple
return entities
x = get_entities([some_key])
Related
I have two SQLAlchemy class declarations that refer to each other so the first one gives an error because the second one is not yet declared.
The records in Key with key_type_id == 4 are references from one Entity to a parent Entity via the parent_entity_id.
To define the relationship in Entity that collects all the child entities I need to add a backref, but that references key which is declared later.
class Entity(db.Model):
__tablename__ = 'entity'
entity_id = db.Column(db.INTEGER, primary_key=True)
...
children = db.relationship(
'Entity', secondary=key,
primaryjoin="and_(key.c.entity_id == entity_id, "
"key.c.key_type_id == 4)",
secondaryjoin=(key.c.parent_entity_id == entity_id),
backref=db.backref('key', lazy='dynamic'), lazy='dynamic')
class Key(db.Model):
__tablename__ = 'key'
...
entity_id = db.Column(db.ForeignKey('entity.entity_id'), nullable=False,
index=True)
...
key_type_id = db.Column(db.ForeignKey('key_type.key_type_id'), index=True)
# 4 for a foreign key
...
parent_entity_id = db.Column(db.INTEGER, index=True)
...
The error traceback is.
ipdb> Traceback (most recent call last):
File "<ipython-input-1-a3063c2d9856>", line 1, in <module>
debugfile('C:/Users/Mark Kortink/Dropbox/Python/projects/metapplica/_dev/Scraps/ooClass2DBs.py', wdir='C:/Users/Mark Kortink/Dropbox/Python/projects/metapplica/_dev/Scraps')
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 856, in debugfile
debugger.run("runfile(%r, args=%r, wdir=%r)" % (filename, args, wdir))
File "C:\ProgramData\Anaconda3\lib\bdb.py", line 585, in run
exec(cmd, globals, locals)
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "c:/users/mark kortink/dropbox/python/projects/metapplica/_dev/scraps/ooclass2dbs.py", line 207, in <module>
class Entity(db.Model):
File "c:/users/mark kortink/dropbox/python/projects/metapplica/_dev/scraps/ooclass2dbs.py", line 254, in Entity
'Entity', secondary=key,
NameError: name 'key' is not defined
First am i declaring the backref relationship correctly?
Second how do I break the deadlock?
The name key doesn't exist in the module you are using. Even if you define your Key class first, there would never actually be a reference to the Table object named key in the scope of your module. In recognition of this, you can pass the name of the table as a string to the arguments of relationship() that would otherwise require a reference to that name in your module scope.
From the docs:
When using the Declarative extension, the Declarative initializer
allows string arguments to be passed to relationship(). These string
arguments are converted into callables that evaluate the string as
Python code, using the Declarative class-registry as a namespace. This
allows the lookup of related classes to be automatic via their string
name, and removes the need for related classes to be imported into the
local module space before the dependent classes have been declared.
That means secondary=key, can be declared as secondary="key". SQLAlchemy will then only try to resolve the string "key" to the object that it references after all of the classes have been mapped and the table will exist in the internal registry that SQLAlchemy uses to track such objects. Similarly, secondaryjoin=(key.c.parent_entity_id == entity_id) can be defined as secondaryjoin="key.c.parent_entity_id == entity_id".
setattr() for an item in Redmine issues, is failing, with the following error.
Traceback (most recent call last):
File "E:\test\get_redmine_data.py", line 47, in <module>
print (item.assigned_to)
File "C:\Python27\lib\site-packages\redminelib\resources\standard.py", line 150, in __getattr__
return super(Issue, self).__getattr__(attr)
File "C:\Python27\lib\site-packages\redminelib\resources\base.py", line 164, in __getattr__
attr, encoded = self.encode(attr, decoded, self.manager)
File "C:\Python27\lib\site-packages\redminelib\resources\base.py", line 266, in encode
return attr, manager.new_manager(cls._resource_map[attr]).to_resource(value)
File "C:\Python27\lib\site-packages\redminelib\managers\base.py", line 29, in to_resource
return self.resource_class(self, resource)
File "C:\Python27\lib\site-packages\redminelib\resources\base.py", line 130, in __init__
self._decoded_attrs = dict(dict.fromkeys(relations_includes), **attributes)
TypeError: type object argument after ** must be a mapping, not str
I am trying to set some default assignee, for issues where the assignee is not set. The code fails at the line, where I print the attribute I just set. My code is given below:
redmine = Redmine('http://redmine_url', username='uname', password='pwd')
project = redmine.project.get('proj_name')
work_items = project.issues
for item in work_items:
assignee_not_set = getattr(item,'assigned_to',True)
if assignee_not_set == True:
print item.id
setattr(item,'assigned_to','Deepak')
print (item.assigned_to)
I also tried using the update() method,
redmine.project.update(item.id, assigned_to='Deepak')
That also fails with another error - redminelib.exceptions.ResourceNotFoundError: Requested resource doesn't exist.
I verifed that the issue id exists in Redmine.
You have several problems here:
The attribute name is assigned_to_id and not assigned_to
It accepts user id which is int and not a username which is str
No need to use setattr() here, just use item.assigned_to_id = 123
You need to call item.save() after setting assigned_to_id otherwise it won't be saved to Redmine
When you're trying to use update() method, you're using in on a Project resource and not on Issue resource, this is why you're getting ResourceNotFoundError
All this information is available in the docs: https://python-redmine.com/resources/issue.html
Is it possible to extract a sub-key from a JSONField field and annotate the Queryset with its value? I'm trying to extract the value within the query rather than post-processing in the Python code.
Model architecture is:
Django 1.10
Model has a django.contrib.postgres.fields.JSONFieldcalleddata` to store an API response. This example is Twitter.
Other fields are profile_id and screen_name. The rest of the data lives within the data field so it can be queried ad-hoc.
I thought I'd be able to combine annotate and django.models.F but I'm getting the following error:
> models.TwitterUser.objects.annotate(foll_count=F("data__followers_count"))
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/query.py", line 914, in annotate
clone.query.add_annotation(annotation, alias, is_summary=False)
File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 971, in add_annotation
summarize=is_summary)
File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/expressions.py", line 463, in resolve_expression
return query.resolve_ref(self.name, allow_joins, reuse, summarize)
File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1462, in resolve_ref
self.get_initial_alias(), reuse)
File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1402, in setup_joins
names, opts, allow_many, fail_on_missing=True)
File "/Virtualenv/env_name/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1370, in names_to_path
" not permitted." % (names[pos + 1], name))
django.core.exceptions.FieldError: Cannot resolve keyword 'followers_count' into field. Join on 'data' not permitted.
This isn't explicitly documented anywhere so I'm attempting to reverse engineer it using the double underscores used elsewhere in Django. I've separately tried accessing the key as if it was a native Python dict (F("data")[followers_count"]) but that didn't work either.
Any direct answers or pointers towards other areas would be appreciated.
I couldn't use F() at the time of writing so had to fallback to a RawSQL call to access the field.
Based on the prior work of How to aggregate (min/max etc.) over Django JSONField data? and José San Gil on his blog.
qs = models.TwitterUser.objects.annotate(followers=RawSQL(
# The Postgres SQL query to access a JSON object field.
# `data` is the name of the JSONField in my schema.
# More: https://www.postgresql.org/docs/9.3/static/functions-json.html
"((data->%s))",
# The parameter to insert into the query and replace '%s'
# This could be a variable
# and the query adapted into a reusable universal function.
("followers_count",) # In my example, this is the JSON sub-key I'm after
)
)
Note: I've bracketed and indented to hopefully aid comprehension.
You can also add a .filter(data__has_key="insert_key_here") before the annotate if you only want to return items that contain the field in question. This is a nice native method for JSON within the ORM and hopefully in time we'll have similar ways of accessing JSON sub-fields directly through the ORM.
I have the following two models:
Class Foo(models.model):
param1 = ...
param2 = ...
...
paramN = ...
Class Bar(models.model):
foo = models.ForeignKey(Foo)
...
...
GOAL: Compute a QuerySet of all instances of Foo such that more than 1 Bar instance is connected to it
I have been looking for a solution and this seems to work for everybody else
Foo.objects.annotate(num_bar=Count('bar')).filter(num_bar__gt=1)
This gave me a FieldError saying that 'bar' was not a possible field for Foo, I then tried 'bar_set' and also got the same error
Is there a chance I am implementing them wrong, or because they are old they are depreciated now? Any help would be appreciated!
traceback
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 794, in annotate
obj.query.add_annotation(annotation, alias, is_summary=False)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 982, in add_annotation
summarize=is_summary)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/aggregates.py", line 20, in resolve_expression
c = super(Aggregate, self).resolve_expression(query, allow_joins, reuse, summarize)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/expressions.py", line 491, in resolve_expression
c.source_expressions[pos] = arg.resolve_expression(query, allow_joins, reuse, summarize, for_save)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/expressions.py", line 448, in resolve_expression
return query.resolve_ref(self.name, allow_joins, reuse, summarize)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1532, in resolve_ref
self.get_initial_alias(), reuse)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1471, in setup_joins
names, opts, allow_many, fail_on_missing=True)
File "/home/ryan/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1396, in names_to_path
"Choices are: %s" % (name, ", ".join(available)))
FieldError: Cannot resolve keyword 'bar' into field. Choices are: param1, param2, param3, ..., paramN
version
my django version is 1.8.3
There can be multiple reasons for this error, I'll try to tell the probable causes:
Recently upgrading your django version may cause the problem, clear
migrations, rerun.
Moving from local server to production server can cause this problem
sometimes.
Your app name can cause the problem, if it starts with "__".
Try other simpler queries, if they work, try to change the query so
that you don't use the num_model.
also check if you can get the count of them ( the dirty way :) ):
for foo in Foo.objects.all():
if foo.bar_set.count() < 2:
#do sth like : foo.bar_set.get() or temp = temp + 1
as of your short description of model (not your main code), cannot find other causes. Your query should work.
So after trying a lot, this was a solution that worked:
Bar.objects.values("foo_id").annotate(Count("foo_id")).filter(pk__count__gt=1)
Not exactly sure why this worked and the other didn't, but it essentially just gets the count of Bar objects with the same foo_id and makes sure there is more than 1.
If somebody would like to explain a potential reason why this works and the other did not, that would be appreciated.
I had the same issue by import mistake. This is solution for my use case
# from django.db.models.sql.aggregates import Count # wrong import
from django.db.models import Count # correct one
I am integrating django with a legacy system and database and have a model that looks like this
class Label(models.Model)
iLabelID = models.IntegerField(db_column='LABELID', primary_key=True)
organization = models.OneToOneField(Organization, related_name='labels', db_column='ORGANIZATION')
sLabelText = models.CharField(max_length=42)
Using this notation (more or less hungarian notation) is a requirement of the project.
The following will work in django:
>>> Label.objects.get(organization_id=1)
But I want to be able to write this:
>>> Label.objects.get(iOrganizationID=1)
I tried subclassing models.OneToOneField with
class MyOneToOneField(models.OneToOneField):
def get_attname(self):
# default is:
# return '%s_id' % self.name
return 'i%s%sID' % (self.name[0].upper(), self.name[1:])
But this is the error I get when trying to use it:
>>> Label.objects.get(iOrganizationID=1)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "django/db/models/manager.py", line 151, in get
return self.get_queryset().get(*args, **kwargs)
File "django/db/models/query.py", line 301, in get
num = len(clone)
File "django/db/models/query.py", line 77, in __len__
self._fetch_all()
File "django/db/models/query.py", line 854, in _fetch_all
self._result_cache = list(self.iterator())
File "django/db/models/query.py", line 230, in iterator
obj = model(*row_data)
File "django/db/models/base.py", line 347, in __init__
setattr(self, field.attname, val)
AttributeError: can't set attribute
EDIT: here's another pain point:
I wish to generate some JSON. This JSON will be fed to another part of the system on which I have no control (not possible to change names). I wish I could do the following:
json.dumps(list(Label.objects.values('iLabelID', 'iOrganizationID', 'sAnotherValue')))
But this is not possible. I have to do this
list(Label.objects.values('iLabelID', 'organization_id', 'sAnotherValue'))
And then manually map organization_id to iOrganizationID, although this is not a problem for the label's ID. It makes code more difficult to maintain, to read, and slower to execute.
Note that this is not specific to hungarian notation, you may need to suffix with _identifier or _pk or whatever instead of _id.
EDIT2: I must have made another error because as lanzz pointed out get_attname does work -_-
I solved it using some modifications of your code:
class CustomAttrNameForeignKey(djangoFields.related.ForeignKey):
def __init__(self, *args, **kwargs):
attname = kwargs.pop('attrname', None)
super(CustomAttrNameForeignKey, self).__init__(*args, **kwargs)
self.attname = attname or super(CustomAttrNameForeignKey, self).get_attname()
def get_attname(self):
return self.attname
class AModelUsingThis(djangoModels.Model):
snapshot = CustomAttrNameForeignKey(
ParentalModel, db_column="timestamp", to_field="timestamp",
attrname='timestamp', related_name="children")
In this case we get in the model an attribute for FK with no suffix at all, but with the name given. I haven't tested it yet on DB, but just tried to instantiate model also with this custom FK given -- it works fine.