I want add a method to a class, without extending another class (for the import statements should be unaltered).
Conceptually, my current approach is:
add_method.py
def new_method():
pass
MyObject.new_method = new_method
main.py
from package import MyObject
import add_method
ob = MyObject()
ob.new_method()
This does the job, however, PyCharm does not recognize that the import add_method import statement is acutally used: "Unused import statement". Is there an elegant way to obtain the same effect with PyCharm recognizing the import?
Related
If I have a python file as follows
import time
class abc:
def func1(self):
while True:
do something
time.sleep(5)
xyz = abc()
xyz.func1()
The above works exactly as expected
If I move the class definition out to a seperate .py file and import it
import time
from test1 import abc
xyz = abc()
xyz.func1()
Works as expected until it hits the time.sleep(5) when it errors
saying time is not defined
I have tried adding the time import into the class definition but
still seem to get the same problem
What am I missing.
Moving the import to the class definition file doesn't seem to resolve the problem, either adding it at the top or in the init .
Take a look at the python documentation about the import statement.
Quote from there:
define a name or names in the local namespace for the scope where the import statement occurs.
So your toplevel import is bound on module level and the name time is accessible from wherever in that module.
If you move the import inside the method, it will be accessible in the local scope of the method and won't be visible outside of that method.
So, you don't want to move the import in your other module, as you don't use it there. To make things work, you should preserve the import of time where it is now and only move your function calls outside:
# test1.py
import time
class ABC:
def func1(self):
while True:
# do something
time.sleep(5)
print('slept 5 secs')
# main.py
from test1 import ABC
abc = ABC()
abc.func1()
[Not related to your question tip] You should also take a look at PEP8 and follow it's conventions.
what is the easiest way to solve the following problem in extending/altering the functionality of a third party library?
The library offers a class LibraryClass with a function func_to_be_changed. This function has a local variable internal_variable which is the instance of another class SimpleCalculation of that library. I created a new class BetterCalculation in my own module and now want LibraryClass.func_to_be_changed to use an instance of this new class.
# third party library
from third_party_library.utils import SimpleCalculation
class LibraryClass:
def func_to_be_changed(self, x):
# many complicated things go on
internal_variable = SimpleCalculation(x)
# many more complicated things go on
The easiest solution would be, to just copy the code from the third party library, subclass the LibraryClass and overwrite the function func_to_be_changed:
# my module
from third_party_library import LibraryClass
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
"""This is an exact copy of LibraryClass.func_to_be_changed."""
# many complicated things go on
internal_variable = BetterCalculation(x) # Attention: this line has been changed!!!
# many more complicated things go on
However, this involves copying of many lines of code. When a new version of the third party class improves on code that was copied without modification, this modifications need to be incorporated manually by another copying step.
I tried to use unittest.mock.patch as I know that the following two snippets work:
# some script
from unittest.mock import patch
import third_party_library
from my_module import BetterCalculation
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = third_party_library.utils.SimpleCalculation(x) # indeed uses `BetterCalculation`
def foo(x):
return third_party_library.utils.SimpleCalculation(x)
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = foo(x) # indeed uses `BetterCalculation`
However, the following does not work:
# some script
from unittest.mock import patch
from third_party_library.utils import SimpleCalculation
from my_module import BetterCalculation
def foo(x):
return SimpleCalculation(x)
with patch('third_party_library.utils.SimpleCalculation', BetterCalculation):
local_ = foo(x) # does not use `BetterCalculation`
# this works again
with patch('__main__.SimpleCalculation', BetterCalculation):
local_ = foo(x) # indeed uses `BetterCalculation`
Therefore, the following won`t work either:
# my module
from unittest.mock import patch
from third_party_library import LibraryClass
from my_module import BetterCalculation
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
with patch(
'third_party_library.utils.SimpleCalculation',
BetterCalculation
):
super().func_to_be_changed(x)
Is there an elegant pythonic way to do this? I guess this boils down to the question: What is the equivaltent of __main__ in the last code snippet that needs to replace third_party_library.utils?
Some context
The first string argument in the patch function can have two different meanings depending on the situation. In the first situation the described object has not been imported and is unavailable to the program which would, therefore, result in a NameError without the mocking. However, in the question, an object needs to be overwritten. Therefore, it is available to the program and not using patch would not result in an error.
Disclaimer
I might have used the complete wrong language in here, as for sure there are precise python terms for all the described notions.
Overwriting an object
As shown in the question, the locally imported SimpleCalculation can be overwritten with __main__.SimpleCalculation. Therefore, it is important to remember that you need to tell patch the path to the local object and not how that same object would be imported in the current script.
Let's assume the following module:
# thirdpartymodule/__init__.py
from .utils import foo
def local_foo():
print("Hello local!")
class Bar:
def __init__(self):
foo()
local_foo()
and
# thirdpartymodule/utils.py
def foo():
print("third party module")
We want to override the functions foo and local_foo. But we don't want to override any functions, we want to override the functions foo and local_foo in the context of the file thirdpartymodule/__init__.py. It is unimportant that the function foo enters the context of the file via an import statement, while local_foo is defined locally. So we want to override the functions in the context of thirdpartymodule.foo and thirdpartymodule.local_foo. The context thirdpartymodule.utils.foo is not important here and won't help us. The following snippet illustrates that:
from unittest.mock import patch
from thirdpartymodule import Bar
bar = Bar()
# third party module
# Hello local!
def myfoo():
print("patched function")
with patch("thirdpartymodule.foo", myfoo):
bar = Bar()
# patched function
# Hello local!
# will not work!
with patch("thirdpartymodule.utils.foo", myfoo):
bar = Bar()
# third party module
# Hello local!
with patch("thirdpartymodule.local_foo", myfoo):
bar = Bar()
# third party module
# patched function
In the hypothetical module of the question we first need to assume that the class LibraryClass is defined in the file third_party_library/library_class.py. Then, we want to override third_party_library.library_class.SimpleCalculation and the correct patch would be:
# my module
from unittest.mock import patch
from third_party_library import LibraryClass
from my_module import BetterCalculation
class BetterLibraryClass(LibraryClass):
def func_to_be_changed(self, x):
with patch(
'third_party_library.library_class.SimpleCalculation',
BetterCalculation
):
super().func_to_be_changed(x)
Python's docs say that there's an alternative to local imports to prevent loading the module on startup:
https://docs.python.org/3/library/unittest.mock-examples.html#mocking-imports-with-patch-dict
...to prevent “up front costs” by delaying the import.
This can also be solved in better ways than an unconditional local
import (store the module as a class or module attribute and only do
the import on first use).
I don't understand the explanation in brackets. How do I do this? However I think about it, I seem to end up with local imports anyway.
The documentation likely refers to the use of importlib.import_module, which exposes Python's import functionality:
import importlib
class Example():
TO_IMPORT = "os" # the name of the module to lazily import
__module = None
def __init__(self):
if self.__module is None:
self.__module = importlib.import_module(self.TO_IMPORT)
Note that this way the module is only imported once when the class is first instantiated and is not available in global namespace.
Further, it allows you to change which module is imported, which could be useful e.g. in cases where the same class is used as an interface to different backends:
import importlib
class Example():
def __init__(self, backend="some_module"):
self.module = importlib.import_module(backend)
Does it matter where modules are loaded in a code?
Or should they all be declared at the top, since during load time the external modules will have to be loaded regardless of where they are declared in the code...?
Example:
from os import popen
try:
popen('echo hi')
doSomethingIllegal;
except:
import logging #Module called only when needed?
logging.exception("Record to logger)
or is this optimized by the compiler the same way as:
from os import popen
import logging #Module will always loaded regardless
try:
popen('echo hi')
doSomethingIllegal;
except:
logging.exception("Record to logger)
This indicates it may make a difference:
"import statements can be executed just about anywhere. It's often useful to place them inside functions to restrict their visibility and/or reduce initial startup time. Although Python's interpreter is optimized to not import the same module multiple times, repeatedly executing an import statement can seriously affect performance in some circumstances."
These two OS questions, local import statements? and import always at top of module? discuss this at length.
Finally, if you are curious about your specific case you could profile/benchmark your two alternatives in your environment.
I prefer to put all of my import statements at the top of the source file, following stylistic conventions and for consistency (also it would make changes easier later w/o having to hunt through the source file looking for import statements scattered throughout)
The general rule of thumb is that imports should be at the top of the file, as that makes code easier to follow, and that makes it easier to figure out what a module will need without having to go through all the code.
The Python style guide covers some basic guidelines for how imports should look: http://www.python.org/dev/peps/pep-0008/#imports
In practice, though, there are times when it makes sense to import from within a particular function. This comes up with imports that would be circular:
# Module 1
from module2 import B
class A(object):
def do_something(self):
my_b = B()
...
# Module 2
from module1 import A
class B(object):
def do_something(self):
my_a = A()
...
That won't work as is, but you could get around the circularity by moving the import:
# Module 1
from module2 import B
class A(object):
def do_something(self):
my_b = B()
...
# Module 2
class B(object):
def do_something(self):
from module1 import A
my_a = A()
...
Ideally, you would design the classes such that this would never come up, and maybe even include them in the same module. In that toy example, having each import the other really doesn't make sense. However, in practice, there are some cases where it makes more sense to include an import for one method within the method itself, rather than throwing everything into the same module, or extracting the method in question out to some other object.
But, unless you have good reason to deviate, I say go with the top-of-the-module convention.
I want to fake a package in python. I want to define something so that the code can do
from somefakepackage.morefakestuff import somethingfake
And somefakepackage is defined in code and so is everything below it. Is that possible? The reason for doing this is to trick my unittest that I got a package ( or as I said in the title, a module ) in the python path which actually is just something mocked up for this unittest.
Sure. Define a class, put the stuff you need inside that, assign the class to sys.modules["classname"].
class fakemodule(object):
#staticmethod
def method(a, b):
return a+b
import sys
sys.modules["package.module"] = fakemodule
You could also use a separate module (call it fakemodule.py):
import fakemodule, sys
sys.modules["package.module"] = fakemodule
Yes, you can make a fake module:
from types import ModuleType
m = ModuleType("fake_module")
import sys
sys.modules[m.__name__] = m
# some scripts may expect a file
# even though this file doesn't exist,
# it may be used by Python for in error messages or introspection.
m.__file__ = m.__name__ + ".py"
# Add a function
def my_function():
return 10
m.my_function = my_function
Note, in this example its using an actual module (of ModuleType) since some
Python code may expect modules, (instead of a dummy class).
This can be made into a utility function:
def new_module(name, doc=None):
import sys
from types import ModuleType
m = ModuleType(name, doc)
m.__file__ = name + '.py'
sys.modules[name] = m
return m
print(new_module("fake_module", doc="doc string"))
Now other scripts can run:
import fake_module
I took some of the ideas from the other answers and turned them into a Python decorator #modulize which converts a function into a module. This module can then be imported as usual. Here is an example.
#modulize('my_module')
def my_dummy_function(__name__): # the function takes one parameter __name__
# put module code here
def my_function(s):
print(s, 'bar')
# the function must return locals()
return locals()
# import the module as usual
from my_module import my_function
my_function('foo') # foo bar
The code for the decorator is as follows
import sys
from types import ModuleType
class MockModule(ModuleType):
def __init__(self, module_name, module_doc=None):
ModuleType.__init__(self, module_name, module_doc)
if '.' in module_name:
package, module = module_name.rsplit('.', 1)
get_mock_module(package).__path__ = []
setattr(get_mock_module(package), module, self)
def _initialize_(self, module_code):
self.__dict__.update(module_code(self.__name__))
self.__doc__ = module_code.__doc__
def get_mock_module(module_name):
if module_name not in sys.modules:
sys.modules[module_name] = MockModule(module_name)
return sys.modules[module_name]
def modulize(module_name, dependencies=[]):
for d in dependencies: get_mock_module(d)
return get_mock_module(module_name)._initialize_
The project can be found here on GitHub. In particular, I created this for programming contests which only allow the contestant to submit a single .py file. This allows one to develop a project with multiple .py files and then combine them into one .py file at the end.
You could fake it with a class which behaves like somethingfake:
try:
from somefakepackage.morefakestuff import somethingfake
except ImportError:
class somethingfake(object):
# define what you'd expect of somethingfake, e.g.:
#staticmethod
def somefunc():
...
somefield = ...
TL;DR
Patch sys.modules using unittest.mock:
mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
Explanation
Other answers correctly recommend to fix sys.modules but a proper way to do it is by patching it using mock.patch. Meaning replacing it temporarily (only for when tests are run) with a fake object that optionally imitates the desired behaviour. And restoring it back once tests are finished to not affect other test cases.
The code in TL;DR section will simply make your missing package not raise ImportError. To provide fake package with contents and imitate desired behaviour, initiate mock.Mock(…) with proper arguments (e.g. add attributes via Mock's **kwargs).
Full code example
The code below temporarily patches sys.modules so that it includes somefakepackage and makes it importable from the dependent modules without ImportError.
import sys
import unittest
from unittest import mock
class SomeTestCase(unittest.TestCase):
def test_smth(self):
# implement your testing logic, for example:
self.assertEqual(
123,
somefakepackage_dependent.some_func(),
)
#classmethod
def setUpClass(cls): # called once before all the tests
# define what to patch sys.modules with
cls._modules_patcher = mock.patch.dict(
sys.modules,
{'somefakepackage': mock.Mock()},
)
# actually patch it
cls._modules_patcher.start()
# make the package globally visible and import it,
# just like if you have imported it in a usual way
# placing import statement at the top of the file,
# but relying on a patched dependency
global somefakepackage_dependent
import somefakepackage_dependent
#classmethod # called once after all tests
def tearDownClass(cls):
# restore initial sys.modules state back
cls._modules_patcher.stop()
To read more about setUpClass/tearDownClass methods, see unittest docs.
unittest's built-in mock subpackage is actually a very powerful tool. Better dive deeper into its documentation to get a better understanding.