I have the following MongonEngine models:
from app import db
from datetime import datetime
from mongoengine import signals
class PathEmbedded(db.EmbeddedDocument):
"""
To be embedded.
"""
_id = db.ObjectIdField(required=False)
distance = db.IntField(required=False, min_value=0, default=0)
meta = {
"allow_inheritance": True,
}
def __unicode__(self):
return "Path '%s': %d m" % (self.id, self.distance)
class Path2(PathEmbedded, db.Document):
"""
Same as above, but standalone version to be stored in its own collection.
"""
_id = db.ObjectIdField()
orig = db.ObjectIdField(required=True)
dest = db.ObjectIdField(required=True)
updateStamp = db.DateTimeField(required=True)
ok_to_use = db.BooleanField(required=True, default=False)
meta = {
'indexes': [
{
'fields': ['ok_to_use', 'orig', 'dest'],
'cls': False, # does this affect performance?!
},
],
}
#classmethod
def pre_save(cls, sender, document, **kwargs):
document.updateStamp = datetime.utcnow()
def to_embedded(self):
"""
Converts the standalone Path instance into an embeddadle PathEmbedded instance.
"""
import json
temp = json.loads(self.to_json())
#remove the {"_cls": "Location"} key.
#If we don't do this, the output will be a 'Location' instance, not a 'LocationEmbedded' instace
temp.pop('_cls')
return PathEmbedded().from_json(json.dumps(temp))
def get_from_gmaps(self):
"""
Get distance from Google maps using the directions API and append to the 'paths' list.
Return False on error or True on success.
"""
try:
self.distance = 10,
self.save()
except Exception, e:
print str(e)
return False
else:
return True
# connect event hooks:
signals.pre_save.connect(Path2.pre_save, sender=Path2)
So, at some point I'm updating a path instance by calling get_from_gmaps():
from app.models.Path2 import Path2 as P
from bson import ObjectId
p=P(orig=ObjectId(), dest=ObjectId())
p.save()
p.get_from_gmaps()
which raises:
>>> p.get_from_gmaps()
ValidationError (Path2:54d34b97362499300a6ec3be) (10 could not be converted to int: ['distance'])
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "[...]app/models/Path2/get_from_gmaps.py", line 18, in get_from_gmaps
self.save()
File "[...]venv/local/lib/python2.7/site-packages/mongoengine/document.py", line 224, in save
self.validate(clean=clean)
File "[...]venv/local/lib/python2.7/site-packages/mongoengine/base/document.py", line 323, in validate
raise ValidationError(message, errors=errors)
ValidationError: ValidationError (Path2:54d34b97362499300a6ec3be) (10 could not be converted to int: ['distance'])
Originally I was storing an integer parsed from some json and converted to int, and thought somthing was wrong there, but i replaced it with an int value for debugging and now get this. I really don't know where to start o.O
EDIT: expanded code to provide complete [non]working example.
There's an extra comma after the 10:
self.distance = 10,
^
You are setting distance to a tuple containing an int, instead of an int.
HINT: The reason why your are seeing such an unhelpful message is that MongoEngine is using %s format string improperly. In fact, the result of "%s" % something depends on the type of something, as tuples are special cased. Compare:
>>> '%s' % 10
'10'
>>> '%s' % (10,)
'10'
>>> '%s' % (10, 11)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: not all arguments converted during string formatting
>>> '%s' % ((10,),) # This is the correct way of formatting strings
'(10,)' # when the type of the argument is unknown.
This is a MongoEngine's problem of course, but if you want to avoid the same kind of mistake in your code, remember to always use tuples at the right of the % operator, or even better use the .format() method.
Are you sure the self model you send is the right one?
This ValidationError is thrown when you have declare a ReferenceField in a document, and you try to save this document before saving the referenced document (Mongoengine represents a reference field in MongoDB as an dictionnay containing the class and the ObjectId of the reference).
Related
I am using voluptuous a lot to validate yaml description files. Often the errors are cumbersome to decipher, especially for regular users.
I am looking for a way to make the error a bit more readable. One way is to identify which line in the YAML file is incrimined.
from voluptuous import Schema
import yaml
from io import StringIO
Validate = Schema({
'name': str,
'age': int,
})
data = """
name: John
age: oops
"""
data = Validate(yaml.load(StringIO(data)))
In the above example, I get this error:
MultipleInvalid: expected int for dictionary value # data['age']
I would rather prefer an error like:
Error: validation failed on line 2, data.age should be an integer.
Is there an elegant way to achieve this?
The problem is that on the API boundary of yaml.load, all representational information of the source has been lost. Validate gets a Python dict and does not know where it originated from, and moreover the dict does not contain this information.
You can, however, implement this yourself. voluptuous' Invalid error carries a path which is a list of keys to follow. Having this path, you can parse the YAML again into nodes (which carry representation information) and discover the position of the item:
import yaml
def line_from(path, yaml_input):
node = yaml.compose(yaml_input)
for item in path:
for entry in node.value:
if entry[0].value == item:
node = entry[1]
break
else: raise ValueError("unknown path element: " + item)
return node.start_mark.line
# demostrating this on more complex input than yours
data = """
spam:
egg:
sausage:
spam
"""
print(line_from(["spam", "egg", "sausage"], data))
# gives 4
Having this, you can then do
try:
data = Validate(yaml.load(StringIO(data)))
except Invalid as e:
line = line_from(e.path, data)
path = "data." + ".".join(e.path)
print(f"Error: validation failed on line {line} ({path}): {e.error_message}")
I'll go this far for this answer as it shows you how to discover the origin line of an error. You will probably need to extend this to:
handle YAML sequences (my code assumes that every intermediate node is a MappingNode, a SequenceNode will have single nodes in its value list instead of a key-value tuple)
handle MultipleInvalid to issue a message for each inner error
rewrite expected int to should be an integer if you really want to (no idea how you'd do that)
abort after printing the error
With the help of flyx I found ruamel.yaml which provide the line and col of a parsed YAML file. So one can manage to get the wanted error with:
from voluptuous import Schema
from ruamel.yaml import load, RoundTripLoader
from io import StringIO
Validate = Schema({
'name': {
'firstname': str,
'lastname': str
},
'age': int,
})
data = """
name:
firstname: John
lastname: 12.0
age: 42
"""
class Validate:
def __init__(self, stream):
self._yaml = load(stream, Loader=RoundTripLoader)
return self.validate()
def validate(self):
try:
self.data = Criteria(self._yaml)
except Invalid as e:
node = self._yaml
for key in e.path:
if (hasattr(node[key], '_yaml_line_col')):
node = node[key]
else:
break
path = '/'.join(e.path)
print(f"Error: validation failed on line {node._yaml_line_col.line}:{node._yaml_line_col.col} (/{path}): {e.error_message}")
else:
return self.data
data = Validate(StringIO(data))
With this I get this error message:
Error: validation failed on line 2:4 (/name): extra keys not allowed
I have the following api call to tda-api
orders = client.get_account(config.account_id,fields=['positions'])
Gives the Error:
File "/opt/anaconda3/lib/python3.7/site-packages/tda/client/base.py", line 361, in get_account
fields = self.convert_enum_iterable(fields, self.Account.Fields)
File "/opt/anaconda3/lib/python3.7/site-packages/tda/utils.py", line 66, in convert_enum_iterable
self.type_error(value, required_enum_type)
File "/opt/anaconda3/lib/python3.7/site-packages/tda/utils.py", line 41, in type_error
possible_members_message))
ValueError: expected type "Fields", got type "str". (initialize with enforce_enums=False to disable this checking)
the documentation follows:
Client.get_account(account_id, *, fields=None)
and if i replace with:
client.get_account(config.account_id,fields=positions)
'positions' is not defined
And if i look into the api the code for the get_account() function looks like:
class Fields(Enum):
'''Account fields passed to :meth:`get_account` and
:meth:`get_accounts`'''
POSITIONS = 'positions'
ORDERS = 'orders'
def get_account(self, account_id, *, fields=None):
fields = self.convert_enum_iterable(fields, self.Account.Fields)
params = {}
if fields:
params['fields'] = ','.join(fields)
Most likely not the correct way to go about this but I found a fix for now.
I commented out the following that checks for the type in the utils.py file for the tda package.
def convert_enum_iterable(self, iterable, required_enum_type):
if iterable is None:
return None
if isinstance(iterable, required_enum_type):
return [iterable.value]
values = []
for value in iterable:
if isinstance(value, required_enum_type):
values.append(value.value)
# elif self.enforce_enums:
# self.type_error(value, required_enum_type)
else:
values.append(value)
return values
I believe if you create your own client you can set this property to false as well.
And then I am able to use the following to get my current positions:
data = client.get_account(config.account_id,fields=['positions'])
At one point, I had these working but haven't tried in a while. I do recall it being difficult to decipher what a field type was. Depending on your import statements, you might need the entire tda.client.Client.Account.Fields.POSITIONS, for example.
r_acct_orders = client.get_account(config.ACCOUNT_ID, fields=tda.client.Client.Account.Fields.ORDERS).json() # field options None, ORDERS, POSITIONS
r = client.get_account(config.ACCOUNT_ID, fields=tda.client.Client.Account.Fields.POSITIONS).json() # field options None, ORDERS, POSITIONS
r = client.get_accounts(fields=tda.client.Client.Account.Fields.ORDERS).json() # field options None, ORDERS, POSITIONS
r = client.get_accounts(fields=None).json() # field options None, ORDERS, POSITIONS
Also, I use a config.py for account_id but you can just use your syntax as needed.
I got the following error message:
$ python tmp2.py
why??
Traceback (most recent call last):
File "tmp2.py", line 15, in <module>
test._id = ObjectId(i[0])
File "/home/mictadlo/.virtualenvs/unisnp/lib/python2.7/site-packages/bson/objectid.py", line 92, in __init__
self.__validate(oid)
File "/home/mictadlo/.virtualenvs/unisnp/lib/python2.7/site-packages/bson/objectid.py", line 199, in __validate
raise InvalidId("%s is not a valid ObjectId" % oid)
bson.errors.InvalidId: test1 is not a valid ObjectId
with this code:
from bson.objectid import ObjectId
from mongoengine import *
class Test(Document):
_id = ObjectIdField(required=True)
tag = StringField(required=True)
if __name__ == "__main__":
connect('dbtest2')
print "why??"
for i in [('test1', "a"), ('test2', "b"), ('test3', "c")]:
test = Test()
test._id = ObjectId(i[0])
test.char = i[1]
test.save()
How is it possible to use its own ids which are unique too?
According to the documentation: http://docs.mongoengine.org/apireference.html#fields, ObjectIdField is 'A field wrapper around MongoDB’s ObjectIds.'. So it cannot accept a string test1 as an object id.
You may have to change the code to something like this:
for i in [(bson.objectid.ObjectId('test1'), "a"), (bson.objectid.ObjectId('test2'), "b"), (bson.objectid.ObjectId('test3'), "c")]:
for your code to work (Assuming test1 etc are valid id)
Two things:
ObjectId receives a 24 hex string, you can't initialize it with that string. For instance, instead of using 'test1' you can use a string such as '53f6b9bac96be76a920e0799' or '111111111111111111111111'. You don't even need to initialize an ObjectId, you could do something like this:
...
test._id = '53f6b9bac96be76a920e0799'
test.save()
...
I don't know what are you trying to accomplish by using _id. If you are trying to produce and id field or "primary key" for you document, it's not necessary because one is generated automatically. Your code would be:
class Test(Document):
tag = StringField(required=True)
for i in [("a"), ("b"), ("c")]:
test = Test()
test.char = i[0]
test.save()
print(test.id) # would print something similar to 53f6b9bac96be76a920e0799
If you insist in using a field named _id you must know that your id will be the same, because internally, MongoDB calls it _id. If you still want to use string1 as identifier you should do:
class Test(Document):
_id = StringField(primary_key=True)
tag = StringField(required=True)
I have following piece of code to patch the Folder:
ATFolderSchema = ATContentTypeSchema.copy() + \
ConstrainTypesMixinSchema.copy() + NextPreviousAwareSchema.copy()
finalizeATCTSchema(ATFolderSchema, folderish=True, moveDiscussion=False)
field = StringField("rafal_shortdescription",
schemata = "default",
widget = StringWidget(
label = _(u"label_shortdescription",
default=u"Short Description"),
description = _(u"help_shortdescription",
default=u"Used in tabs."),
),
),
ATFolderSchema.addField(field)
and last line throws:
File "/home/rafal/projects/vidensportalen_v2/eggs/Products.Archetypes-1.6.4-py2.6.egg/Products/Archetypes/Schema/__init__.py", line 198, in _validateOnAdd
raise ValueError, "Object doesn't implement IField: %r" % field
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/rafal/projects/vidensportalen_v2/parts/instance/etc/site.zcml", line 12.2-12.39
ZopeXMLConfigurationError: File "/home/rafal/projects/vidensportalen_v2/eggs/Plone-4.0.2-py2.6.egg/Products/CMFPlone/meta.zcml", line 39.4-43.10
ValueError: Object doesn't implement IField: <Field rafal_shortdescription(string:rw)>
Any idea why?
I'd advise you to use archetypes.schemaextender instead of using patches to alter Archetypes content types.
The package includes documentation on how to implement your additional field.
As for your error, you created a tuple with one element, a field:
>>> example = 1,
>>> print example
(1,)
Delete the trailing comma and your code should work as intended.
I'm trying to get results from a SOAP service called Chrome ADS (for vehicle data). They provided php and Java samples, but I need python (our site is in Django). My question is:
What should I be passing as a request to the SOAP service when using wsdl2py-generated classes?
Following the examples I'm using a DataVersionsRequest object as the request parameter, but the code generated by wsdl2py seems to want a getDataVersions object, and there's something like that defined at the bottom of the generated _client.py file. But that too seems to throw an error. So I'm not sure what I should be passing as the request obj. Any suggestions?
$sudo apt-get install python-zsi
$wsdl2py http://platform.chrome.com/***********
$python
>>> url = "http://platform.chrome.com/***********"
>>> from AutomotiveDescriptionService6_client import *
>>> from AutomotiveDescriptionService6_types import *
>>> locator = AutomotiveDescriptionService6Locator()
>>> service = locator.getAutomotiveDescriptionService6Port()
>>> locale = ns0.Locale_Def('locale')
>>> locale._country="US"
>>> locale._language="English"
>>> acctInfo = ns0.AccountInfo_Def('accountInfo')
>>> acctInfo._accountNumber=*****
>>> acctInfo._accountSecret="*****"
>>> acctInfo._locale = locale
>>> dataVersionsRequest = ns0.DataVersionsRequest_Dec()
>>> dataVersionsRequest._accountInfo = acctInfo
>>> service.getDataVersions(dataVersionsRequest)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "AutomotiveDescriptionService6_client.py", line 36, in getDataVersions
raise TypeError, "%s incorrect request type" % (request.__class__)
TypeError: <class 'AutomotiveDescriptionService6_types.DataVersionsRequest_Dec'> incorrect request type
>>> dataVersionsRequest = getDataVersions
>>> dataVersionsRequest._accountInfo = acctInfo
>>> service.getDataVersions(dataVersionsRequest)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "AutomotiveDescriptionService6_client.py", line 36, in getDataVersions
raise TypeError, "%s incorrect request type" % (request.__class__)
AttributeError: class DataVersionsRequest_Holder has no attribute '__class__'
>>> quit()
$cat AutomotiveDescriptionService6_client.py
.....
# Locator
class AutomotiveDescriptionService6Locator:
AutomotiveDescriptionService6Port_address = "http://platform.chrome.com:80/AutomotiveDescriptionService/AutomotiveDescriptionService6"
def getAutomotiveDescriptionService6PortAddress(self):
return AutomotiveDescriptionService6Locator.AutomotiveDescriptionService6Port_address
def getAutomotiveDescriptionService6Port(self, url=None, **kw):
return AutomotiveDescriptionService6BindingSOAP(url or AutomotiveDescriptionService6Locator.AutomotiveDescriptionService6Port_address, **kw)
# Methods
class AutomotiveDescriptionService6BindingSOAP:
def __init__(self, url, **kw):
kw.setdefault("readerclass", None)
kw.setdefault("writerclass", None)
# no resource properties
self.binding = client.Binding(url=url, **kw)
# no ws-addressing
# op: getDataVersions
def getDataVersions(self, request, **kw):
if isinstance(request, getDataVersions) is False:
raise TypeError, "%s incorrect request type" % (request.__class__)
# no input wsaction
self.binding.Send(None, None, request, soapaction="", **kw)
# no output wsaction
response = self.binding.Receive(getDataVersionsResponse.typecode)
return response
.....
getDataVersions = GED("urn:description6.kp.chrome.com", "DataVersionsRequest").pyclass
Also, as an aside, I'm not sure that the strings I'm passing to the pname parameter are correct, I assume that those are the ones I see inside the XML when I explore the service with SOAP UI, right?
It looks like you might be passing a class to service.getDataVersions() the second time instead of an instance (it can't be an instance if it doesn't have __class__).
What's happening is isinstance() returns false, and in the process of trying to raise a type error, an attribute error gets raised instead because it's trying to access __class__ which apparently doesn't exist.
What happens if you try:
>>> dataVersionsRequest = getDataVersions**()**
>>> dataVersionsRequest._accountInfo = acctInfo
>>> service.getDataVersions(dataVersionsRequest)
?
Based on the line:
if isinstance(request, getDataVersions) is False:
raise TypeError, "%s incorrect request type" % (request.__class__)
it definitely looks like you should be passing an instance of getDataVersions, so you're probably on the right track.
You probably need to be instantiating your definition objects and then populating them. Look for type == pyclass_type objects associated with the request you're wanting to make and instantiate them.
e.g. (just guessing)
>>> versionrequest = getDataVersions()
>>> versionrequest.AccountInfo = versionrequest.new_AccountInfo()
>>> versionrequest.AccountInfo.accountNumber = "123"
>>> versionrequest.AccountInfo.accountSecret = "shhhh!"
>>> service.getDataVersions(versionrequest)
I found that the code generated by wsdl2py was too slow for my purposes. Good luck.