Is it possible to overwrite the definition of a class so that all calls (inside my program or other .py files) receive the overwrite? For example:
# package.py
class orgClass:
def some_method(x):
return x * x
# my_prog.py
import other_package
from package import orgClass
class orgClass(orgClass):
def some_method(x):
return x + x
if __name__ == '__main__':
other_package.run()
# other_package.py
from package import orgClass
def run():
o = orgClass()
x = 5
print (o.some_method(x)) # would print 10 not 25; however it prints 25
I need to alter the implementation of the orgClass for this specific program, but I don't want to change source code (of the original package; seems like a bad idea anyhow) so I figured I could just "rewrite" the class implementation in my_prog.py and that would take affect to all calls that came from the __main__.thread.
You have a few options here.
Option 1. You can modify the class object directly (monkey-patching):
import package
def new_some_method(self, x):
return x + x
package.orgClass.some_method = new_some_method
(Variants on this include monkey-patching the whole orgClass object, to similar effect.)
Option 2. You could create a replacement module, and then modify sys.path to force it to load first:
# replacement-packages/package.py
class orgClass:
def some_method(self, x):
return x + x
# my_prog.py
sys.path = ['path/to/replacement-packages'] + sys.path
import package # loads replacement-packages/package.py
(Variants on this include using path files or various shims in __init__.py, or modifying sys.modules directly.)
Option 3. You could fork the whole package project, and modify it to suit your needs. Then you could use a virtualenv or some other set-up to only install the modified package as a dependency of your my_prog program, while other programs would use the default python interpreter which would load the original package.
To the best of my knowledge, you have 2 options:
Edit the original file package.py.
Make every file that requires the new method import my_prog.py.
Since you don't want the first option, the answer is No - you cannot do that.
If I understand correctly you want to
1) import orgClass
2) adjust orgClass
#my_classes.py
import orgClass
class orgClass(orgClass):
def some_method(x):
return x+x
edit:
To then use this function use
from my_classes import orgClass
edit:
Now I see your importing is different, then it might be done using argument such as:
# package.py
class orgClass:
def some_method(x):
return x * x
# my_prog.py
import other_package
from package import orgClass
class orgClass(orgClass):
def some_method(x):
return x + x
if __name__ == '__main__':
other_package.run(orgClass)
# other_package.py
def run(o):
x = 5
print (o.some_method(x)) # would print 10 not 25; however it prints 25
Related
I want to build a well-modularized python project, where all alternative modules should be registed and acessed via a function named xxx_builder.
Taking data class as an example:
register.py:
def register(key, module, module_dict):
"""Register and maintain the data classes
"""
if key in module_dict:
logger.warning(
'Key {} is already pre-defined, overwritten.'.format(key))
module_dict[key] = module
data_dict = {}
def register_data(key, module):
register(key, module, data_dict)
data.py:
from register import register_data
import ABCDEF
class MyData:
"""An alternative data class
"""
pass
def call_my_data(data_type):
if data_type == 'mydata'
return MyData
register_data('mydata', call_my_data)
builder.py:
import register
def get_data(type):
"""Obtain the corresponding data class
"""
for func in register.data_dict.values():
data = func(type)
if data is not None:
return data
main.py:
from data import MyData
from builder import get_data
if __name__ == '__main__':
data_type = 'mydata'
data = get_data(type=data_type)
My problem
In main.py, to register MyData class into register.data_dict before calling the function get_data, I need to import data.py in advance to execute register_data('mydata', call_my_data).
It's okay when the project is small, and all the data-related classes are placed according to some rules (e.g. all data-related class should be placed under the directory data) so that I can import them in advance.
However, this registeration mechanism means that all data-related classes will be imported, and I need to install all packages even if I won't use it actually. For example, when the indicator data_type in main.py is not mydata I still need to install ABCDEF package for the class MyData.
So is there any good idea to avoid importing all the packages?
Python's packaging tools come with a solution for this: entry points. There's even a tutorial about how to use entry points for plugins (which seems like what you're doing) (in conjunction with this Setuptools tutorial).
IOW, something like this (nb. untested), if you have a plugin package that has defined
[options.entry_points]
myapp.data_class =
someplugindata = my_plugin.data_classes:SomePluginData
in setup.cfg (or pypackage.toml or setup.py, with their respective syntaxes), you could register all of these plugin classes (here shown with an example with a locally registered class too).
from importlib.metadata import entry_points
data_class_registry = {}
def register(key):
def decorator(func):
data_class_registry[key] = func
return func
return decorator
#register("mydata")
class MyData:
...
def register_from_entrypoints():
for entrypoint in entry_points(group="myapp.data_class"):
register(entrypoint.name)(entrypoint.load())
def get_constructor(type):
return data_class_registry[type]
def main():
register_from_entrypoints()
get_constructor("mydata")(...)
get_constructor("someplugindata")(...)
I'm new to packaging up Python so not sure what search term to use.
Inside my package, there is a _checksum.py
# _checksum.py
class Add():
def __init__(self, x, y):
self.x = x
self.y = y
def answer(self):
return self.x + self.y
So to use them, I'd have to import the file name
import MYPACKAGE
import MYPACKAGE._checksum as checksum
test = checksum.Add(3, 4)
test.answer() #7
So my question is is there a way to set alias to MYPACKAGE._checksum, maybe something like from MYPACKAGE import checksum?
Python relies on filename a lot for module importing, just to make it more intuitive. However, if it's for the package users, you can probably do
# MYPACKAGE/__init__.py
import ._checksum as checksum
So when your users using your package, they can do
# Application code
from MYPACKAGE import checksum
I'm writing unit tests to validate my project functionalities. I need to replace some of the functions with mock function and I thought to use the Python mock library. The implementation I used doesn't seem to work properly though and I don't understand where I'm doing wrong. Here a simplified scenario:
root/connector.py
from ftp_utils.py import *
def main():
config = yaml.safe_load("vendor_sftp.yaml")
downloaded_files = []
downloaded_files = get_files(config)
for f in downloaded_files:
#do something
root/utils/ftp_utils.py
import os
import sys
import pysftp
def get_files(config):
sftp = pysftp.Connection(config['host'], username=config['username'])
sftp.chdir(config['remote_dir'])
down_files = sftp.listdir()
if down_files is not None:
for f in down_files:
sftp.get(f, os.path.join(config['local_dir'], f), preserve_mtime=True)
return down_files
root/tests/connector_tester.py
import unittest
import mock
import ftp_utils
import connector
def get_mock_files():
return ['digital_spend.csv', 'tv_spend.csv']
class ConnectorTester(unittest.TestCase)
#mock.patch('ftp_utils.get_files', side_effect=get_mock_files)
def test_main_process(self, get_mock_files_function):
# I want to use a mock version of the get_files function
connector.main()
When I debug my test I expect that the get_files function called inside the main of connector.py is the get_mock_files(), but instead is the ftp_utils.get_files(). What am I doing wrong here? What should I change in my code to properly call the get_mock_file() mock?
Thanks,
Alessio
I think there are several problems with your scenario:
connector.py cannot import from ftp_utils.py that way
nor can connector_tester.py
as a habit, it is better to have your testing files under the form test_xxx.py
to use unittest with patching, see this example
In general, try to provide working minimal examples so that it is easier for everyone to run your code.
I modified rather heavily your example to make it work, but basically, the problem is that you patch 'ftp_utils.get_files' while it is not the reference that is actually called inside connector.main() but probably rather 'connector.get_files'.
Here is the modified example's directory:
test_connector.py
ftp_utils.py
connector.py
test_connector.py:
import unittest
import sys
import mock
import connector
def get_mock_files(*args, **kwargs):
return ['digital_spend.csv', 'tv_spend.csv']
class ConnectorTester(unittest.TestCase):
def setUp(self):
self.patcher = mock.patch('connector.get_files', side_effect=get_mock_files)
self.patcher.start()
def test_main_process(self):
# I want to use a mock version of the get_files function
connector.main()
suite = unittest.TestLoader().loadTestsFromTestCase(ConnectorTester)
if __name__ == "__main__":
unittest.main()
NB: what is called when running connector.main() is 'connector.get_files'
connector.py:
from ftp_utils import *
def main():
config = None
downloaded_files = []
downloaded_files = get_files(config)
for f in downloaded_files:
print(f)
connector/ftp_utils.py unchanged.
Is there any way to make an implicit initializer for modules (not packages)?
Something like:
#file: mymodule.py
def __init__(val):
global value
value = 5
And when you import it:
#file: mainmodule.py
import mymodule(5)
The import statement uses the builtin __import__ function.
Therefore it's not possible to have a module __init__ function.
You'll have to call it yourself:
import mymodule
mymodule.__init__(5)
These things often are not closed as duplicates, so here's a really nice solution from Pass Variable On Import. TL;DR: use a config module, configure that before importing your module.
[...] A cleaner way to do it which is very useful for multiple configuration
items in your project is to create a separate Configuration module
that is imported by your wrapping code first, and the items set at
runtime, before your functional module imports it. This pattern is
often used in other projects.
myconfig/__init__.py :
PATH_TO_R_SOURCE = '/default/R/source/path'
OTHER_CONFIG_ITEM = 'DEFAULT'
PI = 3.14
mymodule/__init__.py :
import myconfig
PATH_TO_R_SOURCE = myconfig.PATH_TO_R_SOURCE
robjects.r.source(PATH_TO_R_SOURCE, chdir = True) ## this takes time
class SomeClass:
def __init__(self, aCurve):
self._curve = aCurve
if myconfig.VERSION is not None:
version = myconfig.VERSION
else:
version = "UNDEFINED"
two_pi = myconfig.PI * 2
And you can change the behaviour of your module at runtime from the
wrapper:
run.py :
import myconfig
myconfig.PATH_TO_R_SOURCE = 'actual/path/to/R/source'
myconfig.PI = 3.14159
# we can even add a new configuration item that isn't present in the original myconfig:
myconfig.VERSION="1.0"
import mymodule
print "Mymodule.two_pi = %r" % mymodule.two_pi
print "Mymodule.version is %s" % mymodule.version
Output:
> Mymodule.two_pi = 6.28318
> Mymodule.version is 1.0
I have two python scripts in different locations. I try to import the functions of the second one in such a way that they are integrated with the globals of the first one. That works fine. However when I call functions in the first script from the second one they cant be found.
foo.py
def run():
glob = {}
loc = {}
execfile("/path/to/bar.py", glob, loc)
currmodule = globals()
currmodule["func_in_bar"] = glob["func_in_bar"]
func_in_bar()
def func_in_foo_A():
print("fooA")
def func_in_foo_B():
print("fooB")
if __name__ == "__main__":
run()
bar.py
def func_in_bar():
func_in_foo_A()
func_in_foo_B()
When foo is run it fails with: NameError: global name 'func_in_foo_A' is not defined
In bar.py you need to add an import foo and then reference foo.func_in_foo(), etc.
Alternatively, use the form: from foo import func_in_foo