I'm new to packaging up Python so not sure what search term to use.
Inside my package, there is a _checksum.py
# _checksum.py
class Add():
def __init__(self, x, y):
self.x = x
self.y = y
def answer(self):
return self.x + self.y
So to use them, I'd have to import the file name
import MYPACKAGE
import MYPACKAGE._checksum as checksum
test = checksum.Add(3, 4)
test.answer() #7
So my question is is there a way to set alias to MYPACKAGE._checksum, maybe something like from MYPACKAGE import checksum?
Python relies on filename a lot for module importing, just to make it more intuitive. However, if it's for the package users, you can probably do
# MYPACKAGE/__init__.py
import ._checksum as checksum
So when your users using your package, they can do
# Application code
from MYPACKAGE import checksum
Related
I want to build a well-modularized python project, where all alternative modules should be registed and acessed via a function named xxx_builder.
Taking data class as an example:
register.py:
def register(key, module, module_dict):
"""Register and maintain the data classes
"""
if key in module_dict:
logger.warning(
'Key {} is already pre-defined, overwritten.'.format(key))
module_dict[key] = module
data_dict = {}
def register_data(key, module):
register(key, module, data_dict)
data.py:
from register import register_data
import ABCDEF
class MyData:
"""An alternative data class
"""
pass
def call_my_data(data_type):
if data_type == 'mydata'
return MyData
register_data('mydata', call_my_data)
builder.py:
import register
def get_data(type):
"""Obtain the corresponding data class
"""
for func in register.data_dict.values():
data = func(type)
if data is not None:
return data
main.py:
from data import MyData
from builder import get_data
if __name__ == '__main__':
data_type = 'mydata'
data = get_data(type=data_type)
My problem
In main.py, to register MyData class into register.data_dict before calling the function get_data, I need to import data.py in advance to execute register_data('mydata', call_my_data).
It's okay when the project is small, and all the data-related classes are placed according to some rules (e.g. all data-related class should be placed under the directory data) so that I can import them in advance.
However, this registeration mechanism means that all data-related classes will be imported, and I need to install all packages even if I won't use it actually. For example, when the indicator data_type in main.py is not mydata I still need to install ABCDEF package for the class MyData.
So is there any good idea to avoid importing all the packages?
Python's packaging tools come with a solution for this: entry points. There's even a tutorial about how to use entry points for plugins (which seems like what you're doing) (in conjunction with this Setuptools tutorial).
IOW, something like this (nb. untested), if you have a plugin package that has defined
[options.entry_points]
myapp.data_class =
someplugindata = my_plugin.data_classes:SomePluginData
in setup.cfg (or pypackage.toml or setup.py, with their respective syntaxes), you could register all of these plugin classes (here shown with an example with a locally registered class too).
from importlib.metadata import entry_points
data_class_registry = {}
def register(key):
def decorator(func):
data_class_registry[key] = func
return func
return decorator
#register("mydata")
class MyData:
...
def register_from_entrypoints():
for entrypoint in entry_points(group="myapp.data_class"):
register(entrypoint.name)(entrypoint.load())
def get_constructor(type):
return data_class_registry[type]
def main():
register_from_entrypoints()
get_constructor("mydata")(...)
get_constructor("someplugindata")(...)
Is it possible to overwrite the definition of a class so that all calls (inside my program or other .py files) receive the overwrite? For example:
# package.py
class orgClass:
def some_method(x):
return x * x
# my_prog.py
import other_package
from package import orgClass
class orgClass(orgClass):
def some_method(x):
return x + x
if __name__ == '__main__':
other_package.run()
# other_package.py
from package import orgClass
def run():
o = orgClass()
x = 5
print (o.some_method(x)) # would print 10 not 25; however it prints 25
I need to alter the implementation of the orgClass for this specific program, but I don't want to change source code (of the original package; seems like a bad idea anyhow) so I figured I could just "rewrite" the class implementation in my_prog.py and that would take affect to all calls that came from the __main__.thread.
You have a few options here.
Option 1. You can modify the class object directly (monkey-patching):
import package
def new_some_method(self, x):
return x + x
package.orgClass.some_method = new_some_method
(Variants on this include monkey-patching the whole orgClass object, to similar effect.)
Option 2. You could create a replacement module, and then modify sys.path to force it to load first:
# replacement-packages/package.py
class orgClass:
def some_method(self, x):
return x + x
# my_prog.py
sys.path = ['path/to/replacement-packages'] + sys.path
import package # loads replacement-packages/package.py
(Variants on this include using path files or various shims in __init__.py, or modifying sys.modules directly.)
Option 3. You could fork the whole package project, and modify it to suit your needs. Then you could use a virtualenv or some other set-up to only install the modified package as a dependency of your my_prog program, while other programs would use the default python interpreter which would load the original package.
To the best of my knowledge, you have 2 options:
Edit the original file package.py.
Make every file that requires the new method import my_prog.py.
Since you don't want the first option, the answer is No - you cannot do that.
If I understand correctly you want to
1) import orgClass
2) adjust orgClass
#my_classes.py
import orgClass
class orgClass(orgClass):
def some_method(x):
return x+x
edit:
To then use this function use
from my_classes import orgClass
edit:
Now I see your importing is different, then it might be done using argument such as:
# package.py
class orgClass:
def some_method(x):
return x * x
# my_prog.py
import other_package
from package import orgClass
class orgClass(orgClass):
def some_method(x):
return x + x
if __name__ == '__main__':
other_package.run(orgClass)
# other_package.py
def run(o):
x = 5
print (o.some_method(x)) # would print 10 not 25; however it prints 25
Is there any way to make an implicit initializer for modules (not packages)?
Something like:
#file: mymodule.py
def __init__(val):
global value
value = 5
And when you import it:
#file: mainmodule.py
import mymodule(5)
The import statement uses the builtin __import__ function.
Therefore it's not possible to have a module __init__ function.
You'll have to call it yourself:
import mymodule
mymodule.__init__(5)
These things often are not closed as duplicates, so here's a really nice solution from Pass Variable On Import. TL;DR: use a config module, configure that before importing your module.
[...] A cleaner way to do it which is very useful for multiple configuration
items in your project is to create a separate Configuration module
that is imported by your wrapping code first, and the items set at
runtime, before your functional module imports it. This pattern is
often used in other projects.
myconfig/__init__.py :
PATH_TO_R_SOURCE = '/default/R/source/path'
OTHER_CONFIG_ITEM = 'DEFAULT'
PI = 3.14
mymodule/__init__.py :
import myconfig
PATH_TO_R_SOURCE = myconfig.PATH_TO_R_SOURCE
robjects.r.source(PATH_TO_R_SOURCE, chdir = True) ## this takes time
class SomeClass:
def __init__(self, aCurve):
self._curve = aCurve
if myconfig.VERSION is not None:
version = myconfig.VERSION
else:
version = "UNDEFINED"
two_pi = myconfig.PI * 2
And you can change the behaviour of your module at runtime from the
wrapper:
run.py :
import myconfig
myconfig.PATH_TO_R_SOURCE = 'actual/path/to/R/source'
myconfig.PI = 3.14159
# we can even add a new configuration item that isn't present in the original myconfig:
myconfig.VERSION="1.0"
import mymodule
print "Mymodule.two_pi = %r" % mymodule.two_pi
print "Mymodule.version is %s" % mymodule.version
Output:
> Mymodule.two_pi = 6.28318
> Mymodule.version is 1.0
I have a directory structure like this
main.py
markdown-extensions/
__init__.py
doc_extension.py
Here is my doc_extension.py (it's intended to be a bare bones markdown post processor):
from markdown.postprocessors import Postprocessor
class DocsPostProcessor(Postprocessor):
def run(self, text):
return "<h1>hello world</h1>"
class DocsExtension:
def extendMarkdown(self,md):
postProcessor = DocsPostProcessor()
postProcessor.md = md
md.postprocessors.add(postProcessor)
How do I go about importing it into my main.py? I've tried variations on the following to no avail:
import markdown-extensions.doc_extension
import markdown-extensions.*
import markdown-extensions.doc_extension
The - sign is not a valid character for a Python name (also known as identifier), whether it is a module or not. See here.
from markdown-extensions.doc_extension import *
but rather be explicit, as * will import all global variables, methods and classes. So:
from markdown-extensions.doc_extension import DocsPostProcessor, DocsExtension
*edit
And yes besides that you can't have "-"s, I mistook it for a "_".
Either it's lack of sleep but I feel silly that I can't get this. I have a plugin, I see it get loaded but I can't instantiate it in my main file:
from transformers.FOMIBaseClass import find_plugins, register
find_plugins()
Here's my FOMIBaseClass:
from PluginBase import MountPoint
import sys
import os
class FOMIBaseClass(object):
__metaclass__ = MountPoint
def __init__(self):
pass
def init_plugins(self):
pass
def find_plugins():
plugin_dir = os.path.dirname(os.path.realpath(__file__))
plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith("Transformer.py")]
sys.path.insert(0, plugin_dir)
for plugin in plugin_files:
mod = __import__(plugin)
Here's my MountPoint:
class MountPoint(type):
def __init__(cls,name,bases,attrs):
if not hasattr(cls,'plugins'):
cls.plugins = []
else:
cls.plugins.append(cls)
I see it being loaded:
# /Users/carlos/Desktop/ws_working_folder/python/transformers/SctyDistTransformer.pyc matches /Users/carlos/Desktop/ws_working_folder/python/transformers/SctyDistTransformer.py
import SctyDistTransformer # precompiled from /Users/carlos/Desktop/ws_working_folder/python/transformers/SctyDistTransformer.pyc
But, for the life of me, I can't instantiate the 'SctyDistTransformer' module from the main file. I know I'm missing something trivial. Basically, I want to employ a class loading plugin.
To dymically load Python modules from arbitrary folders use imp module:
http://docs.python.org/library/imp.html
Specifically the code should look like:
mod = imp.load_source("MyModule", "MyModule.py")
clz = getattr(mod, "MyClassName")
Also if you are building serious plug-in architecture I recommend using Python eggs and entry points:
http://wiki.pylonshq.com/display/pylonscookbook/Using+Entry+Points+to+Write+Plugins
https://github.com/miohtama/vvv/blob/master/vvv/main.py#L104