When I save a pipeline, which has an ExecutionContext associated with it, and try to load it again, I get the error shown below.
from neuraxle.base import ExecutionContext, Identity
from neuraxle.pipeline import Pipeline
PIPELINE_NAME = 'saved_pipeline_name'
cache_folder = 'cache_folder'
pipeline = Pipeline([
Identity()
]).with_context(ExecutionContext(cache_folder))
pipeline.set_name(PIPELINE_NAME).save(ExecutionContext(cache_folder), full_dump=True)
loaded_pipeline = ExecutionContext(cache_folder).load(PIPELINE_NAME)
Error message:
Traceback (most recent call last):
File "save_example.py", line 12, in <module>
loaded_pipeline = ExecutionContext(cache_folder).load(PIPELINE_NAME)
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 555, in load
).load(context_for_loading, True)
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 3621, in load
return loaded_self.load(context, full_dump)
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 1708, in load
return self._load_step(context, savers)
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 1717, in _load_step
loaded_self = saver.load_step(loaded_self, context)
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 3644, in load_step
step.apply('_assert_has_services', context=context)
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 2316, in apply
results: RecursiveDict = self._apply_childrens(results=results, method=method, ra=ra)
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 2327, in _apply_childrens
for children in self.get_children():
File ".env/lib/python3.7/site-packages/neuraxle/base.py", line 2530, in get_children
return [self.wrapped]
AttributeError: 'StepWithContext' object has no attribute 'wrapped'
Without the with_context(ExecutionContext(cache_folder)) the loading works fine. Is this expected behaviour, or is it a bug? What would be the best practice for saving pipelines, when working with execution contexts?
There was an erroneous call to a function in StepWithContext's saver. There will be a hotfix pushed on Neuraxle's main repository in the next day or so. If you can wait until then your code should execute with no problems.
If not, I'd suggest you to bypass StepWithContext by calling the save directly on StepWithContext's wrapped step (i.e. your pipeline instance):
pipeline.wrapped.set_name(PIPELINE_NAME).save(ExecutionContext(cache_folder), full_dump=True)
loaded_pipeline = ExecutionContext(cache_folder).load(PIPELINE_NAME)
You'll then have to re-wrap the loaded_pipeline instance with a StepWithContext using the .with_context() call.
When the hotfix will be available, keep in my mind that ExecutionContext instances are not getting saved at all and that, on loading, StepWithContext's context attribute is getting replace with whatever context is used for the loading.
Feel free to ask me any other questions! I'll be glad to answer them.
Cheers
Related
I saved a bunch of stuff in one single pickle file with dill and some of the data are objects. Unfortunately, I didn't realize that if I changed my code structure e.g., heavy refactoring that dill wouldn't be able to find my code anymore for loading the objects - or at least that is my guess from the print statement:
------- Main Resume from Checkpoint --------
Traceback (most recent call last):
File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/torch/serialization.py", line 607, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/torch/serialization.py", line 882, in _load
result = unpickler.load()
File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/torch/serialization.py", line 875, in find_class
return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'meta_learning'
python-BaseException
however, it's ok if it cannot find them. There is other data there that I still need or can use to restore everything.
How can I "force" dill to open things even if it cannot find the code? Or I can point to it to the paths for the locations of the new code if that is what it needs.
I saw this error being posted a lot and often it was due to the file not being closed properly after opening. But since I'm using the integrated torch.load() function, I'm not sure what I could do different.
First the saving part:
torch.save({
'model_state_dict': agent.dqn.state_dict(),
...
'loss_history': agent.losshistory
}, modelpath)
and here the loading part, where I also get the error message:
if os.path.exists(modelpath):
checkpoint = torch.load(modelpath)
agent.dqn.load_state_dict(checkpoint['model_state_dict'])
...
agent.losshistory = checkpoint['loss_history']
and here the error:
Traceback (most recent call last):
File "c:/Users/levin/Desktop/programming/main.py", line 33, in <module>
checkpoint = torch.load(modelpath)
File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 529, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 702, in _legacy_load
result = unpickler.load()
EOFError: Ran out of input
One more thing I want to mention is that I used this exact code several times without a problem. I can't remember changing anything that could have caused the error.
According to this thread it seems to raise an exception when reading an empty file, so please check the size of the document before reading it and post a response if it is not solved.
SDK version: 1.0.43
To minimize clicking and compare accuracy between PipelineRuns, I'd like to log a metric from inside a PythonScriptStep to the parent PipelineRun. I thought I could do this like:
from azureml.core import Run
run = Run.get_context()
foo = 0.80
run.parent.log("accuracy",foo)
however I get this error.
Traceback (most recent call last):
File "get_metrics.py", line 62, in <module>
run.parent.log("geo_mean", top3_runs)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/run.py", line 459, in parent
return None if parent_run_id is None else get_run(self.experiment, parent_run_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/run.py", line 1713, in get_run
return next(runs)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/run.py", line 297, in _rehydrate_runs
yield factory(experiment, run_dto)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/run.py", line 325, in _from_dto
return PipelineRun(experiment=experiment, run_id=run_dto.run_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/run.py", line 74, in __init__
service_endpoint=_service_endpoint)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_graph_context.py", line 46, in __init__
service_endpoint=service_endpoint)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_aeva_provider.py", line 118, in create_provider
service_endpoint=service_endpoint)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_aeva_provider.py", line 133, in create_service_caller
service_endpoint = _AevaWorkflowProvider.get_endpoint_url(workspace, experiment_name)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/pipeline/core/_aeva_provider.py", line 153, in get_endpoint_url
workspace_name=workspace.name, workspace_id=workspace._workspace_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/workspace.py", line 749, in _workspace_id
self.get_details()
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/workspace.py", line 594, in get_details
self._subscription_id)
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/_project/_commands.py", line 507, in show_workspace
AzureMachineLearningWorkspaces, subscription_id).workspaces,
File "/azureml-envs/azureml_ffecfef6fbfa1d89f72d5af22e52c081/lib/python3.6/site-packages/azureml/core/authentication.py", line 112, in _get_service_client
all_subscription_list, tenant_id = self._get_all_subscription_ids()
TypeError: 'NoneType' object is not iterable
Update
On further investigation, I tried just printing the parent attribute of the run with the line below and got the same Traceback
print("print run parent attribute", run.parent)
The get_properties() method the below. I'm guessing that azureml just uses the azureml.pipelinerunid property for pipeline tree hierarchy, and that the parent attribute has been left for any user-defined hierarchies.
{
"azureml.runsource": "azureml.StepRun",
"ContentSnapshotId": "45bdecd3-1c43-48da-af5c-c95823c407e0",
"StepType": "PythonScriptStep",
"ComputeTargetType": "AmlCompute",
"azureml.pipelinerunid": "e523d575-c373-46d2-a4bc-1717f5e34ec2",
"_azureml.ComputeTargetType": "batchai",
"AzureML.DerivedImageName": "azureml/azureml_dfd7f4f952ace529f986fe919909c3ec"
}
Please upgrade your SDK to the latest version. Seems like this issue was fixed sometime after 1.0.43.
Was run defined? Can you try ...
run = Run.get_context()
run.parent.log('metric1', 0.80)
I'm unable to repro on 1.0.60 (latest) or 1.0.43 (as in post)
I can do run.parent.log()
run = Run.get_context()
run.parent.log('metric1', 0.80)
in my step
and then I can do
r = Run(ws.experiments["expName"], runid)
r.get_metrics()
and can see metric on pipeline run.
Unclear if i miss something.
I am getting a dill/pickle memory error when loading a serialized object file. I am not quite sure what is happening and I am unsure on how to fix it.
When I call:
stat_bundle = train_batch_iterator(clf, TOTAL_TRAINED_EVENTS)
The code traces to the train_batch_iterator function in which it loads a serialized object and trains the classifier with the data within the object. This is the code:
def train_batch_iterator(clf, tte):
plot_data = [] # initialize plot data array
for file in glob.glob('./SerializedData/Batch8172015_19999/*'):
with open(file, 'rb') as stream:
minibatch_train = dill.load(stream)
clf.partial_fit(minibatch_train.data[1], minibatch_train.target,
classes=np.array([11, 111]))
tte += len(minibatch_train.target)
plot_data.append((test_batch_iterator(clf), tte))
return plot_data
Here is the error:
Traceback (most recent call last):
File "LArSoftSGD-version2.0.py", line 154, in <module>
stat_bundle = train_batch_iterator(clf, TOTAL_TRAINED_EVENTS)
File "LArSoftSGD-version2.0.py", line 118, in train_batch_iterator
minibatch_train = dill.load(stream)
File "/home/jdoe/.local/lib/python3.4/site-packages/dill/dill.py", line 199, in load
obj = pik.load()
File "/home/jdoe/.local/lib/python3.4/pickle.py", line 1038, in load
dispatch[key[0]](self)
File "/home/jdoe/.local/lib/python3.4/pickle.py", line 1184, in load_binbytes
self.append(self.read(len))
File "/home/jdoe/.local/lib/python3.4/pickle.py", line 237, in read
return self.file_read(n)
MemoryError
I have no idea what could be going wrong. The error seems to be in the line minibatch_train = dill.load(stream) and the only thing I can think of is that the serialized data file is too large, however the file is exactly 1161 MB which doesn't seem to big/big enough to cause a memory error.
Does anybody know what might be going wrong?
I'm testing ebaysdk Python library that lets you connect to ebay. Now I'm trying examples from: https://github.com/timotheus/ebaysdk-python/
So far I got stuck at this example:
from ebaysdk.shopping import Connection as Shopping
shopping = Shopping(domain="svcs.sandbox.ebay.com", config_file="ebay.yaml")
response = shopping.execute('FindPopularItems',
{'QueryKeywords': 'Python'})
print response.disct()
When I run it. It gives me this error:
Traceback (most recent call last):
File "ebay-test.py", line 13, in <module>
{'QueryKeywords': 'Python'})
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/connection.py", line 123, in execute
self.error_check()
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/connection.py", line 193, in error_check
estr = self.error()
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/connection.py", line 305, in error
error_array.extend(self._get_resp_body_errors())
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/shopping/__init__.py", line 188, in _get_resp_body_errors
dom = self.response.dom()
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/response.py", line 229, in dom
return self._dom
File "/usr/local/lib/python2.7/dist-packages/ebaysdk-2.1.0-py2.7.egg/ebaysdk/response.py", line 216, in __getattr__
return getattr(self._obj, name)
AttributeError: 'Response' object has no attribute '_dom'
Am I missing something here or it could be some kind of bug in library?
Do you have a config file? I had a lot of problems getting started with this SDK. To get the yaml config file to work, I had to specify the directory that it was in. So in your example, it would be:
shopping = Shopping(domain="svcs.sandbox.ebay.com", config_file=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'ebay.yaml'))
You should also be able to specify debug=true in your Shopping() declaration as in Shopping(debug=True).
Make sure if you have not, to specify your APP_ID and other necessary values in the config file.
You have the wrong domain, it should be open.api.sandbox.ebay.com. See this page on the ebaysdk github.