I'm using Django 2, Python 3.7 and the django-address module (https://pypi.org/project/django-address/). I'm trying to insert some seed data. I have this YAML ...
- model: address.locality
pk: 1
fields:
name: "Chicago"
postal_code: "60053"
state:
name: IL
country:
- United States
When I run my seed command
python manage.py loaddata maps/fixtures/seed_data.yaml
I get this error ...
localhost:web davea$ python manage.py loaddata maps/fixtures/seed_data.yaml
Traceback (most recent call last):
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 923, in to_python
return int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'dict'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/serializers/python.py", line 157, in Deserializer
data[field.attname] = model._meta.get_field(field_name).to_python(field_value)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 928, in to_python
params={'value': value},
django.core.exceptions.ValidationError: ["'{'name': 'IL', 'country': ['United States']}' value must be an integer."]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 113, in loaddata
self.load_label(fixture_label)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 168, in load_label
for obj in objects:
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/serializers/pyyaml.py", line 73, in Deserializer
yield from PythonDeserializer(yaml.load(stream, Loader=SafeLoader), **options)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/serializers/python.py", line 159, in Deserializer
raise base.DeserializationError.WithData(e, d['model'], d.get('pk'), field_value)
django.core.serializers.base.DeserializationError: Problem installing fixture '/Users/davea/Documents/workspace/chicommons/maps/web/maps/fixtures/seed_data.yaml': ["'{'name': 'IL', 'country': ['United States']}' value must be an integer."]: (address.locality:pk=1) field_value was '{'name': 'IL', 'country': ['United States']}'
I have added this in the file maps/monkey_patching.py to help with auto-creating the entities based on unique identifiers ...
from address import State
from address import Country
def country_get_by_natural_key(self, name):
return self.get_or_create(name=name)[0]
def state_get_by_natural_key(self, name, country_id):
return self.get_or_create(name=name, country_id=country_id)[0]
Country.add_to_class("get_by_natural_key",country_get_by_natural_key)
State.add_to_class("get_by_natural_key",state_get_by_natural_key)
Edit: including stack trace per Shivam's request ...
(venv) localhost:maps davea$ python web/manage.py loaddata web/maps/fixtures/seed_data.yaml
Traceback (most recent call last):
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 923, in to_python
return int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/serializers/python.py", line 157, in Deserializer
data[field.attname] = model._meta.get_field(field_name).to_python(field_value)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 928, in to_python
params={'value': value},
django.core.exceptions.ValidationError: ["'['IL', 'United States']' value must be an integer."]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "web/manage.py", line 21, in <module>
main()
File "web/manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 113, in loaddata
self.load_label(fixture_label)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 168, in load_label
for obj in objects:
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/serializers/pyyaml.py", line 73, in Deserializer
yield from PythonDeserializer(yaml.load(stream, Loader=SafeLoader), **options)
File "/Users/davea/Documents/workspace/chicommons/maps/web/venv/lib/python3.7/site-packages/django/core/serializers/python.py", line 159, in Deserializer
raise base.DeserializationError.WithData(e, d['model'], d.get('pk'), field_value)
django.core.serializers.base.DeserializationError: Problem installing fixture '/Users/davea/Documents/workspace/chicommons/maps/web/maps/fixtures/seed_data.yaml': ["'['IL', 'United States']' value must be an integer."]: (address.locality:pk=1) field_value was '['IL', 'United States']'
Issue with natural key
get_by_natural_key method is available on the manager level but you are adding method on model level. So instead of above, you need to do like this:
from django.db import models
from address.models import State
class CustomManager(models.Manager):
def get_by_natural_key(self, name):
return self.get_or_create(name=name)[0]
State.add_to_class('objects', CustomManager())
Issue with fixtures
Nested data of fixtures should only contain args and not kwargs. Check Is there a way in a seed_data.yaml file to autogenerate models on which first model is dependent upon? for more detail. And after that, you need to design your get_by_natural_key function accordingly. So for locality model fixtures would look like
- model: address.locality
pk: 1
fields:
name: "Chicago"
postal_code: "60053"
state: ['I', 'United States']
And overrided manager would be
class CustomManager(models.Manager):
def get_by_natural_key(self, state_name, country):
country = Country.objects.get_or_create(name=country)[0]
return State.objects.get_or_create(name=state_name, country=country)[0]
setattr(State._meta, 'default_manager', CustomManager())
I would offer an alternative approach, which is just to write your own management command to import this data, instead of trying to make loaddata work - I think this is potentially much less effort.
This command works for the sample data you've provided - it may need some minor adjustments if you have more than just address.locality objects in your YAML file:
import yaml
from django.core.management.base import BaseCommand
from address.models import Country, State, Locality
class Command(BaseCommand):
def add_arguments(self, parser):
parser.add_argument('file', nargs=1, type=str)
def handle(self, *args, **options):
f = options['file'][0]
with open(f, 'r') as yamlfile:
data = yaml.load(yamlfile)
for row in data:
if row['model'] == 'address.locality':
state_name = row['fields']['state']['name']
country_name = row['fields']['state']['country'][0]
country_obj, _ = Country.objects.get_or_create(name=country_name)
state_obj, _ = State.objects.get_or_create(country=country_obj, name=state_name)
Locality.objects.get_or_create(
pk=row['pk'],
state=state_obj,
postal_code=row['fields']['postal_code'],
name=row['fields']['name']
)
(Side note - the format of your YAML is a little weird, as it provides a list for country. I would have thought a state can only belong to a single country?).
In the source code of this package, Locality.state field is related to State model by a foreignKey. So, your data fixture state field must be an integer of the related table name.
See django-address source code
PS: In your case you must build all the related models that Locality model is related to. If not you'll have a data integrity error ... And all bunch of other errors.
Related
I'm in the process of upgrading to python 3.10 and in that context I thought it would be nice to also upgrade packages used. Right now, the problem is with the marshmallow packages and at this point I can't even run their example code for NewType anymore.
This is my code (taken from the comment in the NewType definition):
from marshmallow_dataclass import NewType, dataclass, List
import marshmallow.validate
IPv4 = NewType('IPv4', str, validate=marshmallow.validate.Regexp(r'^([0-9]{1,3}\\.){3}[0-9]{1,3}$'))
#dataclass
class MyIps:
ips: List[IPv4]
MyIps.Schema().load({"ips": ["0.0.0.0", "grumble grumble"]})
I only added the first line to have the necessary commands available.
When I run this I receive the following error-message:
/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py:373: UserWarning: ****** WARNING ****** marshmallow_dataclass was called on the class <function NewType.<locals>.new_type at 0x7fb5a077fd90>, which is not a dataclass. It is going to try and convert the class into a dataclass, which may have undesirable side effects. To avoid this message, make sure all your classes and all the classes of their fields are either explicitly supported by marshmallow_dataclass, or define the schema explicitly using field(metadata=dict(marshmallow_field=...)). For more information, see https://github.com/lovasoa/marshmallow_dataclass/issues/51 ****** WARNING ******
warnings.warn(
Traceback (most recent call last):
File "/usr/lib/python3.10/dataclasses.py", line 1197, in fields
fields = getattr(class_or_instance, _FIELDS)
AttributeError: 'function' object has no attribute '__dataclass_fields__'. Did you mean: '__dataclass_params__'?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 370, in _internal_class_schema
fields: Tuple[dataclasses.Field, ...] = dataclasses.fields(clazz)
File "/usr/lib/python3.10/dataclasses.py", line 1199, in fields
raise TypeError('must be called with a dataclass type or instance')
TypeError: must be called with a dataclass type or instance
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 384, in _internal_class_schema
created_dataclass: type = dataclasses.dataclass(clazz)
File "/usr/lib/python3.10/dataclasses.py", line 1185, in dataclass
return wrap(cls)
File "/usr/lib/python3.10/dataclasses.py", line 1176, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
File "/usr/lib/python3.10/dataclasses.py", line 909, in _process_class
for b in cls.__mro__[-1:0:-1]:
AttributeError: 'function' object has no attribute '__mro__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/username/testscript.python", line 9, in <module>
MyIps.Schema().load({"ips": ["0.0.0.0", "grumble grumble"]})
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/lazy_class_attribute.py", line 33, in __get__
setattr(cls, self.name, self.func())
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 356, in class_schema
return _internal_class_schema(clazz, base_schema, clazz_frame)
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 402, in _internal_class_schema
attributes.update(
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 405, in <genexpr>
field_for_schema(
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 695, in field_for_schema
generic_field = _field_for_generic_type(typ, base_schema, typ_frame, **metadata)
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 503, in _field_for_generic_type
child_type = field_for_schema(
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 729, in field_for_schema
or _internal_class_schema(typ, base_schema, typ_frame)
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 387, in _internal_class_schema
raise TypeError(
TypeError: IPv4 is not a dataclass and cannot be turned into one.
My current install is python 3.10.6 and marshmallow_dataclasses 8.5.8. (The code was working with python 3.9 and marshmallow dataclasses 8.3.0)
Does anyone know why this is happening and how to solve this (other than downgrading, obviously)
Does anyone else have this issue? Is this new behaviour and I should adapt our code (if so, how?)? Is this a bug?
I've had a similar issue with a similar bit of code that is raising an error since trying to upgrade marshmallow_dataclasses.
import dataclasses
import marshmallow_dataclass
ARGON2_PASSWORD = marshmallow_dataclass.NewType(
"ARGON2_PASSWORD",
str,
validate=validate.Regexp(
r"[$]argon2(?:id|i)[$]v=\d{1,3}[$]m=\d{3,20},t=\d{1,4},"
r"p=\d{1,2}[$][^$]{1,100}[$][^$]{1,768}"))
#marshmallow_dataclass.add_schema
#dataclasses.dataclass
class Argon2Password:
password: ARGON2_PASSWORD
The detailed root cause is currently discussed in https://github.com/lovasoa/marshmallow_dataclass/issues/206 afaics.
For the time being, downgrading to typing-inspect==0.7.1 worked for me as a hotfix.
When I try to override a method that has an argument with Literal as its type hint, I get a RecursionError from the overrides module (see stack trace below). I'm not sure why this is happening, or if it is possible to override such methods.
edit: The overrides module I'm talking about is this third party module: github.com/mkorpela/overrides
Here is a toy example that reproduces my error:
from typing import Literal
from overrides import overrides
class Base:
def foo(self, mode: Literal["train"]) -> None:
raise NotImplementedError()
class Child(Base):
#overrides
def foo(self, mode: Literal["train"]) -> None:
pass
Removing the #overrides decorator is a workaround since it doesn't change my code's behavior, but I would prefer to keep it if possible. My plan is to change the arg type to an Enum which is probably a better design decision anyway. But I'm curious why this combination of #overrides and Literal doesn't work.
Here is the stack trace:
Traceback (most recent call last):
File "model-autotraining/temp.py", line 12, in <module>
class ChildModel(BaseClass):
File "model-autotraining/temp.py", line 14, in ChildModel
def foo(self, mode: Literal["train"]) -> None:
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/overrides/overrides.py", line 88, in overrides
return _overrides(method, check_signature, check_at_runtime)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/overrides/overrides.py", line 114, in _overrides
_validate_method(method, super_class, check_signature)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/overrides/overrides.py", line 135, in _validate_method
ensure_signature_is_compatible(super_method, method, is_static)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/overrides/signature.py", line 94, in ensure_signature_is_compatible
ensure_all_kwargs_defined_in_sub(
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/overrides/signature.py", line 153, in ensure_all_kwargs_defined_in_sub
and not _issubtype(super_type_hints[name], sub_type_hints[name])
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/overrides/signature.py", line 42, in _issubtype
return issubtype(left, right)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 428, in issubtype
return _is_normal_subtype(normalize(left), normalize(right), forward_refs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 251, in normalize
args = _normalize_args(args)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 232, in _normalize_args
return tuple(_normalize_args(type_) for type_ in tps)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 232, in <genexpr>
return tuple(_normalize_args(type_) for type_ in tps)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 232, in _normalize_args
return tuple(_normalize_args(type_) for type_ in tps)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 232, in <genexpr>
... repeated many times ...
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 232, in _normalize_args
return tuple(_normalize_args(type_) for type_ in tps)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 232, in <genexpr>
return tuple(_normalize_args(type_) for type_ in tps)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/site-packages/typing_utils/__init__.py", line 231, in _normalize_args
if isinstance(tps, collections.abc.Sequence):
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/abc.py", line 98, in __instancecheck__
return _abc_instancecheck(cls, instance)
File "/opt/homebrew/Caskroom/miniforge/base/envs/worker/lib/python3.8/abc.py", line 102, in __subclasscheck__
return _abc_subclasscheck(cls, subclass)
RecursionError: maximum recursion depth exceeded in comparison
As others have noted in the comments, this behavior is a bug in the third party overrides module I'm using https://github.com/mkorpela/overrides). It is a known bug: https://github.com/mkorpela/overrides/issues/94
Bug in the package is now fixed.
I am trying to use a parent class as a blueprint for new classes.
E.g. the FileValidator contains all generic attributesand methods for a generic file. Then I want to create for example a ImageValidator inheriting everything from the FileValidator but with additional, more specific attribtues, methods. etc. In this example the child class is called: FileValidatorPlus
My understanding was, that if I inherit the parent class I can just plug-in more attributes/methods without repeating anything, like just adding min_size. But the following code gives: TypeError: FileValidatorPlus.__init__() got an unexpected keyword argument 'max_size'
class File:
def __init__(self, file_size=100):
self.file_size = file_size
class FileValidator(object):
error_messages = {'max_size': 'some error msg'}
def __init__(self, max_size=None):
self.max_size = max_size
def __call__(self, file):
print(self.max_size > file.file_size)
class FileValidatorPlus(FileValidator):
error_messages = {'min_size': 'some other error msg'}
def __init__(self, min_size=None):
super(FileValidatorPlus, self).__init__()
self.error_messages.update(super(FileValidatorPlus, self).error_messages)
self.min_size = min_size
validator = FileValidatorPlus(max_size=10, min_size=20)
print(validator.error_messages)
print(validator.max_size)
print(validator.min_size)
Full Traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
TypeError: FileValidatorPlus.__init__() got an unexpected keyword argument 'max_size'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py", line 108, in add_exec
more, exception_occurred = self.do_add_exec(code_fragment)
File "/opt/.pycharm_helpers/pydev/pydevconsole.py", line 90, in do_add_exec
command.run()
File "/opt/.pycharm_helpers/pydev/_pydev_bundle/pydev_console_types.py", line 35, in run
self.more = self.interpreter.runsource(text, '<input>', symbol)
File "/usr/local/lib/python3.10/code.py", line 74, in runsource
self.runcode(code)
File "/usr/local/lib/python3.10/code.py", line 94, in runcode
self.showtraceback()
File "/usr/local/lib/python3.10/code.py", line 148, in showtraceback
sys.excepthook(ei[0], ei[1], last_tb)
File "/opt/.pycharm_helpers/pydev/pydevconsole.py", line 112, in info
traceback.print_exception(type, value, tb)
NameError: name 'traceback' is not defined
Traceback (most recent call last):
File "/opt/.pycharm_helpers/pydev/pydevconsole.py", line 284, in process_exec_queue
interpreter.add_exec(code_fragment)
File "/opt/.pycharm_helpers/pydev/_pydev_bundle/pydev_code_executor.py", line 132, in add_exec
return more, exception_occurred
UnboundLocalError: local variable 'exception_occurred' referenced before assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/.pycharm_helpers/pydev/pydevconsole.py", line 511, in <module>
pydevconsole.start_server(port)
File "/opt/.pycharm_helpers/pydev/pydevconsole.py", line 407, in start_server
process_exec_queue(interpreter)
File "/opt/.pycharm_helpers/pydev/pydevconsole.py", line 292, in process_exec_queue
traceback.print_exception(type, value, tb, file=sys.__stderr__)
UnboundLocalError: local variable 'traceback' referenced before assignment
Process finished with exit code 1
You need to add max_size as a parameter in your FileValidatorPlus constructor.
If you want FileValidatorPlus to inherit this field from the super's constructor, you must not define a __init__ method.
In other words, if you want FileValidatorPlus to inherit max_size parameter from FileValidator, you should delete __init__ because the subclass only inherits constructor properties when they don't have their own constructor.
But in your case I think that it would be a better solution just to add min_size=None to FileValidatorPlus.__init__:
def __init__(self, max_size=None, min_size=None):
super(FileValidatorPlus, self).__init__(max_size)
I got also different error :
TypeError: __init__() got an unexpected keyword argument 'max_size'
and the solution would be :
def __init__(self, max_size=None, min_size=None):
super(FileValidatorPlus, self).__init__(max_size)
Output:
{'min_size': 'some other error msg', 'max_size': 'some error msg'}
10
20
SDK version: 1.0.43
To minimize clicking and compare accuracy between PipelineRuns, I'd like to log a metric from inside a PythonScriptStep to the parent PipelineRun. I thought I could do this like:
from azureml.core import Run
run = Run.get_context()
foo = 0.80
run.parent.log("accuracy",foo)
however I get this error.
Traceback (most recent call last):
File "get_metrics.py", line 62, in <module>
run.parent.log("geo_mean", top3_runs)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/run.py", line 459, in parent
return None if parent_run_id is None else get_run(self.experiment, parent_run_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/run.py", line 1713, in get_run
return next(runs)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/run.py", line 297, in _rehydrate_runs
yield factory(experiment, run_dto)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/run.py", line 325, in _from_dto
return PipelineRun(experiment=experiment, run_id=run_dto.run_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/run.py", line 74, in __init__
service_endpoint=_service_endpoint)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_graph_context.py", line 46, in __init__
service_endpoint=service_endpoint)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_aeva_provider.py", line 118, in create_provider
service_endpoint=service_endpoint)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_aeva_provider.py", line 133, in create_service_caller
service_endpoint = _AevaWorkflowProvider.get_endpoint_url(workspace, experiment_name)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_aeva_provider.py", line 153, in get_endpoint_url
workspace_name=workspace.name, workspace_id=workspace._workspace_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/workspace.py", line 749, in _workspace_id
self.get_details()
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/workspace.py", line 594, in get_details
self._subscription_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/_project/_commands.py", line 507, in show_workspace
AzureMachineLearningWorkspaces, subscription_id).workspaces,
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/authentication.py", line 112, in _get_service_client
all_subscription_list, tenant_id = self._get_all_subscription_ids()
TypeError: 'NoneType' object is not iterable
Update
On further investigation, I tried just printing the parent attribute of the run with the line below and got the same Traceback
print("print run parent attribute", run.parent)
The get_properties() method the below. I'm guessing that azureml just uses the azureml.pipelinerunid property for pipeline tree hierarchy, and that the parent attribute has been left for any user-defined hierarchies.
{
"azureml.runsource": "azureml.StepRun",
"ContentSnapshotId": "45bdecd3-1c43-48da-af5c-c95823c407e0",
"StepType": "PythonScriptStep",
"ComputeTargetType": "AmlCompute",
"azureml.pipelinerunid": "e523d575-c373-46d2-a4bc-1717f5e34ec2",
"_azureml.ComputeTargetType": "batchai",
"AzureML.DerivedImageName": "azureml/azureml_dfd7f4f952ace529f986fe919909c3ec"
}
Please upgrade your SDK to the latest version. Seems like this issue was fixed sometime after 1.0.43.
Was run defined? Can you try ...
run = Run.get_context()
run.parent.log('metric1', 0.80)
I'm unable to repro on 1.0.60 (latest) or 1.0.43 (as in post)
I can do run.parent.log()
run = Run.get_context()
run.parent.log('metric1', 0.80)
in my step
and then I can do
r = Run(ws.experiments["expName"], runid)
r.get_metrics()
and can see metric on pipeline run.
Unclear if i miss something.
I have the following factory:
class ContactFactory(DjangoModelFactory):
name = Faker('company')
industry = Iterator(Industry.objects.all())
class Meta:
model = 'sales.contact'
#post_generation
def requested_devices(self, create, extracted, **kwargs):
if create:
self.requested_devices.add(MSize.objects.first())
And I'm trying to write a test such as:
class TestUserCanAskQuestion(TestCase):
#classmethod
def setUpTestData(cls):
call_command('insert_initial_data')
def setUp(self):
self.contact = ContactFactory()
But everytime I run the test, it results in "StopIteration" error.
Here is the full stack trace:
ERROR: test_dummy (comminquiry.tests.test_views.TestUserCanAskQuestion)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/mnt/c/Users/Alire/Projects/mm/mm-bpmui/src/comminquiry/tests/test_views.py", line 32, in test_dummy
a = ContactFactory.build()
File "/mnt/c/Users/Alire/Projects/mm/venv/lib/python3.5/site-packages/factory/base.py", line 546, in build
return cls._generate(enums.BUILD_STRATEGY, kwargs)
File "/mnt/c/Users/Alire/Projects/mm/venv/lib/python3.5/site-packages/factory/base.py", line 500, in _generate
return step.build()
File "/mnt/c/Users/Alire/Projects/mm/venv/lib/python3.5/site-packages/factory/builder.py", line 272, in build
step.resolve(pre)
File "/mnt/c/Users/Alire/Projects/mm/venv/lib/python3.5/site-packages/factory/builder.py", line 221, in resolve
self.attributes[field_name] = getattr(self.stub, field_name)
File "/mnt/c/Users/Alire/Projects/mm/venv/lib/python3.5/site-packages/factory/builder.py", line 375, in __getattr__
extra=context,
File "/mnt/c/Users/Alire/Projects/mm/venv/lib/python3.5/site-packages/factory/declarations.py", line 196, in evaluate
value = next(iter(self.iterator))
File "/mnt/c/Users/Alire/Projects/mm/venv/lib/python3.5/site-packages/factory/utils.py", line 136, in __iter__
value = next(self.iterator)
StopIteration
----------------------------------------------------------------------
Ran 1 test in 2.433s
If I move the ContactFactory() outside of class, the error disappears.
I'm I missing something? or it's a bug in factory boy or django?
(I'm using factory_boy==2.11.1 and django==2.1.2)
As #dirkgroten had suggested, one the fields was returning an empty queryset. This was the root cause of the error.