Problem
Let's say I have this module named custom_module:
class CustomClass:
pass
And I use this class in a script that serializes an object of the class I defined in custom_module:
import cloudpickle
import custom_module as cm
custom_object = cm.CustomClass()
with open('filename.pkl', 'wb') as file:
cloudpickle.dump(custom_object, file)
I copy this pickle file to another environment and load it with another script:
import cloudpickle
with open('filename.pkl', 'rb') as file:
custom_object = cloudpickle.load(file)
That would yield this error:
Traceback (most recent call last):
File "loader.py", line 4, in <module>
custom_object = cloudpickle.load(file)
ModuleNotFoundError: No module named 'custom_module'
Awkward solution
As a workaround it is possible to read and execute everything that is in custom_module in my script:
exec(open(path.join('custom_module.py')).read())
but that looks really weird and I can't use CustomClass as cm.CustomClass. Is there any other solution that doesn't involve copying all the code in the first environment to the second?
You can solve the problem by reimplement CustomClass in custom_module the following way:
def __CustomClass():
class CustomeClass:
... # Your implementaion here
return CustomeClass
CustomClass = __CustomClass()
The error will gone if you are lucky enough. If not, you need to dive into CustomClass to find out other functions or classes that are defined in other local modules, and reimplement them using the same method.
You can find more detail in this question. You may also use cloudpickle.register_pickle_by_value to solve the issue, but it is labeled as an experimental feature.
Related
Most of the tutorials and books about Django or Flask import specific classes from files instead of importing the whole file.
For example, importing DataRequiered validator from wrtforms.validators is done via from wtforms import validators instead of importing it via import wtforms.validators as valids and then accessing DataRequiered with valids.DataRequiered.
My question is: Is there an reason for this ?
I thought to something like avoiding the loading a whole module for computation/memory optimization (is it really relevant?) ? Or is it simply to make the code more readable ?
My question is: Is there an reason for this ?
from module_or_package import something is the canonical pythonic idiom (when you only want to import something in your current namespace of course).
Also, import module_or_package.something only works if module_or_package is a package and something a submodule, it raises an ImportError(No module named something) if something is a function, class or whatever object defined in module_or_package, as can be seen in the stdlib with os.path (which is a submodule of the os.package) vs datetime.date (which is a class defined in the datetime module):
>>> import os.path as p
>>> p
<module 'posixpath' from '/home/bruno/.virtualenvs/blook/lib/python2.7/posixpath.pyc'>
vs
>>>import datetime.date as d
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named date
thought to something like avoiding the loading a whole module for computation/memory optimization (is it really relevant?)
Totally irrelevant - importing a given name from a module requires importing the whole module. Actually, this:
from module_or_package import something
is only syntactic sugar for
import module_or_package
something = module_or_package.something
del module_or_package
EDIT: You mention in a comment that
Right, but importing the whole module means loading it to the memory, which can be a reason for importing only a submodule/class
so it seems I failed to make the point clear: in Python, you can not "import only a submodule/class", period.
In Python, import, class and def are all executable statements (and actually just syntactic sugar for operation you can do 'manually' with functions and classes). Importing a module actually consists in executing all the code at the module's top-level (which will instanciate function and class objects) and create a module object (instance of module type) which attributes will be all names defined at the top-level via import, def and class statements or via explicit assignment. It's only when all this has been done that you can access any name defined in the module, and this is why, as I stated above,
from module import obj
is only syntactic sugar for
import module
obj = module.obj
del module
But (unless you do something stupid like defining a terabyte-huge dict or list in your module) this doesn't actually take that much time nor eat much ram, and a module is only effectively executed once per process the first time it's imported - then it's cached in sys.modules so subsequent imports only fetch it from cache.
Also, unless you actively prevents it, Python will cache the compiled version of the module (the .pyc files) and only recompile it if the .pyc is missing or older than the source .py file.
wrt/ packages and submodules, importing a submodule will also execute the package's __init__.py and build a module instance from it (IOW, at runtime, a package is also a module). Package initializer are canonically rather short, and actually quite often empty FWIW...
It depends, in the tutorial that was probably done for readability
Usually if you use most of the classes in a file, you import the file. If the files contains many classes but you only need a few, just import those.
It's both a matter of readability and optimization.
I am trying to create some tests for a Python program. I am using Python version 2.7.12 so I had to install mock using sudo pip install mock.
According to the documentation and several other sites, I should be able to use the following code to use patch:
import mock # Also could use from mock import patch
import unittest
class TestCases(unittest.TestCase):
#mock.patch('MyClass.ImportedClass') # If using from mock import patch should just be #patch
def test_patches(self, mock_ImportedClass):
# Test code...
However, the above code throws the following error:
AttributeError: 'module' object has no attribute 'patch'
After some experimenting in the terminal, it seems quite a few things don't work. For example
>>> import mock
>>> mock.MagicMock
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'MagicMock'
>>> mock.patch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'patch'
>>> mock.Mock
<class mock.Mock at 0x7fdff56b0600>
Interestingly, I can access something called patch using mock.Mock(), but I don't know if this works in the same way and cannot find anything about it in the documentation.
My experience with Python is fairly limited so I'm not sure what might be wrong. I'm fairly certain the right package was installed. Any help on how I can get this working?
Just for the sake of completeness, I wanted to add the answer, but the credit goes to #user2357112.
If you run into this issue check to make sure you don't have another file named mock.py (or whatever file you're trying to import) either in the same directory or in the search path for python. To confirm you've loaded the correct file, you can check mock.__file_, which is an attribute added automatically by python to your imported files. If you're using the terminal, you can just do something like:
>>> import mock
>>> mock.__file__
'/usr/lib/python2.7/...'
If the file path doesn't match what you expect, you should remove the file being imported from your search path or modify your search path so that the file you want is found first. If neither of those are an option, you need to change the name of the file you're importing (if it's one you define and not a library like mock).
What is the "correct" way to import/use classmethods when my class is part of a module?
I created a Python utility for my co-workers, basically using my Java knowledge, Stack Overflow and Google. It works fine, but an experienced Python person reviewed everything and suggested to improve the pythonicity of the code.
Initially, I just used sys.path.append() to add sub-directories that would contain many .py files (basically one class within each .py).
Now I am trying to get the thing working with packages and modules:
I added (empty) __init__.py files in my root directory, and all sub-directories;
I turned all my classes into "modules", by adding __name__ = "whatever-filename" at line 1.
And, well, imports work. In my main script, I can do
from classes import MyHelper
(where classes refers to a sub-directory, and MyHelper to a module within). But:
MyHelper.some_class_method()
gives me:
Traceback (most recent call last):
File "./xyz", line 12, in <module>
MyHelper.some_class_method()
AttributeError: 'module' object has no attribute 'some_class_method'
But I didn't change MyHelper - it still has all the #classmethods that I could use before introducing packages/modules.
Edit: MyHelper looks like this:
__name__ = "MyHelper"
...
class MyHelper(object):
"""This class..."""
...
MyHelper refers to a module name when imported, not the class name which is the same. You can try the following import instead to import a class from a module in a child directory.
from classes.MyHelper import MyHelper
MyHelper.calling_my_class_method_here()
from X import Y
is really importing a class or function Y from the module X into the current module's namespace, and you can use Y directly.
Alternatively, if you just imported the module X, eg
import X
then you would call the function (or class) Y as X.Y. So in your case, try just
import MyHelper
MyHelper.some_class_method()
I'm very new to Python and I have a code like this:
class Configuration:
#staticmethod
def test():
return "Hello World"
When I call the method test from other python code like this:
import test
test.Configuration.test()
I get an error like this:
Traceback (most recent call last):
File "example.py", line 3, in <module>
test.Configuration.test()
AttributeError: 'module' object has no attribute 'test'
where I'm making the mistake?
Edit:
My directory structure:
root
--example.py
--test
----__init.py__
----Configuration.py
Python module names and the classes they contain are separate. You need use the full path:
import test
print test.Configuration.Configuration.test()
Your test package has a module named Configuration, and inside that module is your Configuration class.
Note that Python, unlike Java, lets you define methods outside classes too, no need to make this a static method. Nor do you need to use a separate file per class.
Try to rename your module to something other than 'test', since this is the name of a standard library module (http://docs.python.org/2/library/test.html) and probably you're importing that module instead of your own. Another option is to add the directory containing your test module into the PYTHONPATH environment variable, so that python may find it instead of the standard library module (but this is not advised as it shadows the standard module and you won't be able to import it later).
To check which file you're importing from, do:
import test
print test
I'm looking into creating a Plugin structure for a program and making it so even the core library is treated as plugins. in my research I came across this code that is dynamically importing modules.
def __initialize_def(self, module):
"""Attempt to load the definition"""
# Import works the same for py files and package modules so strip!
if module.endswith(".py"):
name = module [:-3]
else:
name = module
# Do the actual import
__import__(name)
definition = sys.modules[name]
# Add the definition only if the class is available
if hasattr(definition, definition.info["class"]):
self.definitions[definition.info["name"]] = definition
logging.info("Loaded %s" % name)
I have tried to understand what this code is doing and I've succeeded to a point. However, I simply can't understand the latter part of the code, specifically these two lines:
if hasattr(definition, definition.info["class"]):
self.definitions[definition.info["name"]] = definition
I can't figure out what definition.info["<key>"] is referring to.
What is this .info[] dictionary and what does it hold? Is it common to all Python objects or only module objects? What is it useful for?
py> import sys,os
py> sys.modules["os"].info["class"]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'info'
So this info attribute must be specific to modules that can be used as plugins in this program.
Reserved names in Python generally begin with two underscores. You just stumbled on some piece of a larger codebase, that gives info module-scope values a special meaning. I don't think its authors chose a particularly good name for these, anyway; $FRAMEWORK_MODULE_INFO would be more explicit.