Is there a way (using only python. i.e.: without a bash script nor another language code) to call a specific function in every script inside a folder without needing to import all of them explicitly.
For example, let's say that this is my structure:
main.py
modules/
module1.py
module2.py
module3.py
module4.py
and every moduleX.py has this code:
import os
def generic_function(caller):
print('{} was called by {}'.format(os.path.basename(__file__), caller))
def internal_function():
print('ERROR: Someone called an internal function')
while main.py has this code:
import modules
import os
for module in modules.some_magic_function():
module.generic_function(os.path.basename(__file__))
So if I run main.py, I should get this output:
module1.py was called by main.py
module2.py was called by main.py
module3.py was called by main.py
module4.py was called by main.py
*Please note that internal_function() shouldn't be called (unlike this question). Also, I don't want to declare explicitly every module file even on a __init__.py
By the way, I don't mind to use classes for this. In fact it could be even better.
You can use exec or eval to do that. So it would go roughly this way (for exec):
def magic_execute():
import os
import glob
for pyfl in glob.glob(os.path(MYPATH, '*.py'):
with open(pyfl, 'rt') as fh:
pycode = fh.read()
pycode += '\ngeneric_function({})'.format(__file__)
exec(pycode)
The assumption here is that you are not going to import the modules at all.
Please note, that there are numerous security issues related to using exec in such a non-restricted manner. You can increase security a bit.
While sophros' approach is quickly and enough for implicitly importing the modules, you could have issues related to controlling every module or with complex calls (like having conditions for each calls). So I went with another approeach:
First I created a class with the function(s) (now methods) declared. With this I can avoid checking if the method exists as I can use the default one if I didn't declare it:
# main.py
class BaseModule:
def __init__(self):
# Any code
def generic_function(self, caller):
# This could be a Print (or default return value) or an Exception
raise Exception('generic_function wasn\'t overridden or it was used with super')
Then I created another class that extends the BaseModule. Sadly I wasn't able to get a good way for checking inherence without knowing the name of the child class so I used the same name for every module:
# modules/moduleX.py
from main import BaseModule
class GenericModule(BaseModule):
def __init__(self):
BaseModule.__init__(self)
# Any code
def generic_function(self, caller):
print('{} was called by {}'.format(os.path.basename(__file__), caller))
Finally, in my main.py, I used the importlib for importing the modules dynamically and saving an instance for each one, so I can use them later (for sake of simplicity I didn't save them in the following code, but it's easy as using a list and appending every instance on it):
# main.py
import importlib
import os
if __name__ == '__main__':
relPath = 'modules' # This has to be relative to the working directory
for pyFile in os.listdir('./' + relPath):
# just load python (.py) files except for __init__.py or similars
if pyFile.endswith('.py') and not pyFile.startswith('__'):
# each module has to be loaded with dots instead of slashes in the path and without the extension. Also, modules folder must have a __init___.py file
module = importlib.import_module('{}.{}'.format(relPath, pyFile[:-3]))
# we have to test if there is actually a class defined in the module. This was extracted from [1]
try:
moduleInstance = module.GenericModule(self)
moduleInstance.generic_function(os.path.basename(__file__)) # You can actually do whatever you want here. You can save the moduleInstance in a list and call the function (method) later, or save its return value.
except (AttributeError) as e:
# NOTE: This will be fired if there is ANY AttributeError exception, including those that are related to a typo, so you should print or raise something here for diagnosting
print('WARN:', pyFile, 'doesn\'t has GenericModule class or there was a typo in its content')
References:
[1] Check for class existence
[2] Import module dynamically
[3] Method Overriding in Python
Related
I am developing a Python package which does work by taking in user-defined objects all of which are instances of a class which I wrote. The way I have designed is, user passes his/her objects as defined in one or more python scripts (see example below).
I want to access the objects which user defines in the scripts. How can I do that?
I looked at import by filename but to no avail. I even went on to use imp.load_source but didn't solve.
Some typical user-defined objects
Assume for the sake of the problem, all methods are defined in Base. I understand what I am asking for leads to arbitrary code execution, so I am open to suggestions wherein users can pass their instances of the Base class arbitrarily but safely.
foo.py has the following code:
from package import Base
foo = Base('foo')
foo.AddBar('bar', 'bar')
foo.AddCow('moo')
ooo.py :
from package import Base
ooo = Base('ooo')
ooo.AddBar('ooo','ooo')
ooo.AddO(12)
And I run my main program as,
main_program -p foo.py ooo.py
I want to be able to access foo, ooo in the main_program body.
Tried:
I am using python2.7 I know I am using older Python, I will make the move soon
importlib
Tried importlib.import_module but it throws ImportError: Import by filename is not supported.
__import__
I tried using __import__('/path/to/file.py') but it throws the same ImportError: Import by filename is not supported.
At this point, any solution which lets me use objects defined in user-input scripts works.
If you are okay with skipping the .py in the filename, this can be solved by asking the user to pass the module name (basically the file name without the py) extension
Referring to this answer and this book, here is an example
tester.py
class A:
def write(self):
print("hello")
obj = A()
Now we want to dynamically access obj from a file called test.py, so we do
python test.py tester
And what does test.py do? It imports the module based on name and access it methods. Note that this assumes you are not concerned about the order in which the user passes the objects
test.py
import sys
# Get all parameters (sys.argv[0] is the file name so skipping that)
module_names = sys.argv[1:]
# I believe you can also do this using importlib
modules = list(map(__import__, module_names))
# modules[0] is now "tester"
modules[0].obj.write()
Mapping this to your example, I think this should be
foo.py
from package import Base
foo = Base('foo')
foo.AddBar('bar', 'bar')
foo.AddCow('moo')
ooo.py
from package import Base
ooo = Base('ooo')
ooo.AddBar('ooo','ooo')
ooo.AddO(12)
And run main program as
python main_program.py foo ooo
Have you tried
from (whatever the file name is) import *
Also don’t include .py in the file name.
I'm trying to dynamically update code during runtime by reloading modules using importlib.reload. However, I need a specific module variable to be set before the module's code is executed. I could easily set it as an attribute after reloading but each module would have already executed its code (e.g., defined its default arguments).
A simple example:
# module.py
def do():
try:
print(a)
except NameError:
print('failed')
# main.py
import module
module.do() # prints failed
module.a = 'succeeded'
module.do() # prints succeeded
The desired pseudocode:
import_module_without_executing_code module
module.initialise(a = 'succeeded')
module.do()
Is there a way to control module namespace initialisation (like with classes using metaclasses)?
It's not usually a good idea to use reload other than for interactive debugging. For example, it can easily create situations where two objects of type module.A are not the same type.
What you want is execfile. Pass a globals dictionary (you don't need an explicit locals dictionary) to keep each execution isolated; anything you store in it ahead of time acts exactly like the "pre-set" variables you want. If you do want to have a "real" module interface change, you can have a wrapper module that calls (or just holds as an attribute) the most recently loaded function from your changing file.
Of course, since you're using Python 3, you'll have to use one of the replacements for execfile.
Strictly speaking, I don't believe there is a way to do what you're describing in Python natively. However, assuming you own the module you're trying to import, a common approach with Python modules that need some initializing input is to use an init function.
If all you need is some internal variables to be set, like a in you example above, that's easy: just declare some module-global variables and set them in your init function:
Demo: https://repl.it/MyK0
Module:
## mymodule.py
a = None
def do():
print(a)
def init(_a):
global a
a = _a
Main:
## main.py
import mymodule
mymodule.init(123)
mymodule.do()
mymodule.init('foo')
mymodule.do()
Output:
123
foo
Where things can get trickier is if you need to actually redefine some functions because some dynamic internal something is dependent on the input you give. Here's one solution, borrowed from https://stackoverflow.com/a/1676860. Basically, the idea is to grab a reference to the current module by using the magic variable __name__ to index into the system module dictionary, sys.modules, and then define or overwrite the functions that need it. We can define the functions locally as inner functions, then add them to the module:
Demo: https://repl.it/MyHT/2
Module:
## mymodule.py
import sys
def init(a):
current_module = sys.modules[__name__]
def _do():
try:
print(a)
except NameError:
print('failed')
current_module.do = _do
I have a directory structure like this:
foo/
__init__.py
printsum.py
'''
def printsum(x,y):
print("sum is " + str(x+y))
'''
bar.py
'''
def printsum(x,y):
print("sum is " + str(x+y))
'''
example.py
'''
#!/usr/bin/env python3
import foo.printsum
import bar
foo.printsum.printsum(1,2)
bar.printsum(3,4)
'''
The file example.py is meant to be run as a script, while foo and bar are meant to be imported modules. I would like to make the namespace of foo like bar. That is - I don't want the double printsum.printsum. I don't want all of the foo module to be saved in one big monolithic file, like bar.py. I want the foo.printsum() method to be saved in a file by itself, but I want printsum() to exist at the foo. namespace level.
Can this be done somehow?
Import the functions in __init__py.
foo/__init__py:
from .printsum import printsum
(If you are using python2, you might need to remove the dot or use full import path. Rules for relative import paths are stricter in python 3)
Then you can call the function directly from foo:
import foo
foo.printsum(...)
The foo import will run __init__. You can add some prints in there as well if you want proof. This is a perfectly normal way to expose functionality in package modules. You can also do * imports, but that is generally not recommended. Import strictly what you want to expose.
__init__ in other words is a place you can glue stuff to the package, but also do automatic initialisation that triggers in the import.
Also look up the __all__ keyword. It can be extremely useful to hide code that is not relevant for a user of the package or module.
parent/__init__.py:
favorite_numbers = [1]
def my_favorite_numbers():
for num in favorite_numbers:
num
my_favorite_numbers()
from .child import *
my_favorite_numbers()
parent/child.py:
print favorite_numbers
favorite_numbers.append(7)
I then created a file one directory up from parent directory named tst.py:
import parent
So the directory structure looks like this:
parent (directory)
__init__.py (file)
child.py (file)
tst.py (file)
And I get this error upon execution:
NameError: name 'favorite_numbers' is not defined
How can I add a value to favorite_numbers within child.py so that when I execute the my_favorite_numbers() function, I get 1 and 7.
In Python, each module has its own separate globals. That's actually the whole point of modules (as opposed to, say, C preprocessor-style text inserts).
When you do from .child import *, that imports .child, then copies all of its globals into the current module's globals. They're still separate modules, with their own globals.
If you want to pass values between code in different modules, you probably want to wrap that code up in functions, then pass the values as function arguments and return values. For example:
parent/__init__.py:
from .child import *
favorite_numbers = [1]
def my_favorite_numbers():
for num in favorite_numbers:
num
my_favorite_numbers()
child_stuff(favorite_numbers)
my_favorite_numbers()
parent/child.py:
def child_stuff(favorite_numbers):
print favorite_numbers
favorite_numbers.append(7)
In fact, you almost always want to wrap up any code besides initialization (defining functions and classes, creating constants and other singletons, etc.) in a function anyway. When you import a module (including from … import), that only runs its top-level code the first time. If you import again, the module object already exists in memory (inside sys.modules), so Python will just use that, instead of running the code to build it again.
If you really want to push a value into another module's namespace, you can, but you have to do it explicitly. And this means you have to have the module object available by importing it, not just importing from it:
from . import child
child.favorite_numbers = favorite_numbers
But this is rarely a good idea.
Did you ever run setup.py or a way of "building" your library?
I would create a setup.py file and likely run it in develop mode. Python setup.py develop vs install
Suppose you have the following structure in your src folder:
conf.py
./drivers
mod1.py --> contains mod1Class
mod2.py --> contains mod2Class
What I'd like to have is a snippet of code in conf.py to automatically instantiate the classes in mod*.py so that if one day I'll add mod3.py --> mod3Class this will be automatically instantiated in conf.py without adding any line of code.
I tried, without success:
from drivers import *
but I'm not able to import, I receive a NameError. So I'm stuck at the very first step. Also suppose I'm able to perform the import successfully, how can I do:
mod1Class_instance = mod1.mod1Class() (in a cycle, one instance for every file in drivers)
in an automatic way? I cannot use strings to make the instance of a class so I cannot get the names of the files in drivers and use strings. What's the right way to do this operation?
Thanks
Maybe, it's what you need:
from types import ModuleType
import drivers
for driver_module in dir(drivers):
if not isinstance(driver_module, ModuleType):
continue # not real module driver
for cls in dir(driver_module):
if not isinstance(cls, SomeBaseClass):
continue # not real mod class
# create new variable with name as lower class name
locals()[cls.__name__.lower()] = cls()
And, also, you should create __init__.py file in your drivers folder. This will mean that your folder is a python-module.
On the other hand, I recommend manually describe all imports. This simple approach makes your code more clear.
This loads classes in modules in a directory drivers, which is in the same directory of the current module, and does not need to make drivers a package:
from collections import defaultdict
import os
import pkgutil
def getclasses(module):
"""Get classes from a driver module."""
# I'd recommend another way to find classes; for example, in defuz's
# answer, classes in driver modules would have one base class.
try:
yield getattr(module, module.__name__ + "Class")
except AttributeError:
pass
instances = defaultdict(list)
drivers_dir = os.path.join(os.path.dirname(__file__), 'drivers')
for module_loader, name, ispkg in pkgutil.iter_modules([drivers_dir]):
module = module_loader.find_module(name).load_module(name)
for cls in getclasses(module):
# You might want to use the name of the module as a key instead of the
# module object, or, perhaps, to append all instances to a same list.
instances[module].append(cls())
# I'd recommend not putting instances in the module namespace,
# and just leaving them in that dictionary.
for mod_instances in instances.values():
for instance in mod_instances:
locals()[type(instance).__name__ + "_instance"] = instance