I have a dozen scripts where I'm repeating this
# caller.py
THIS_FILE_DIR = os.path.dirname(os.path.realpath(__file__))
relative_path = lambda *x: os.path.join(THIS_FILE_DIR, *x)
I'd like to move this into a module as follows:
# module.py
def relative_path(*x):
this_file_dir = os.path.dirname(os.path.realpath(__file__))
return os.path.join(this_file_dir, *x)
However I'd like to get __file__ in the scope of the calling script (caller.py) rather than the module (module.py) itself. Is there a way to do this without having the calling script pass in __file__ (so my relative_path() method only needs *x)?
It is not possible for an imported function to access global variables of the module where it [function] is imported.
relative_path() function can not access __file__ of caller.py.
You may pass it as a parameter:
def relative_path(*x, __file__):
this_file_dir = os.path.dirname(os.path.realpath(__file__))
return os.path.join(this_file_dir, *x)
It actually is possible. You can get path of caller like this
import inspect
def relative_path(*x):
caller_path = inspect.stack()[1].filename
You then have string of absolute path (recommend using pathlib instead of os if necessary)
Passing file is not necessary and can be confusing...
Related
I have to run the main file that depends on the rest of a relatively large project. Project structure can be seen as such
main.py
opts.py
--models \
--model1.py
--model2.py
...
--schedulers
--scheduler1.py
--scheduler2.py
...
...
The problem is when I have to pass arguments to each component (using argparse). A simple way would be to define all parameters in a single place for each component. This worked so far for me (in opts.py), but I would like to make something more elegant. In my parse function for each component parse_models or parse_scheduler I would like to iterate through each submodule of models and schedulers and let them define their own arguments by calling a function define_arguments that each of them has where they create their own sub parser.
All in all, how do I iterate through the submodules and call their define_arguments function from within opts.py?
You can iterate over all the python files using the glob module. You can find the correct path with the parent module's __path__ attribute. Import the modules using importlib.import_module. The imported module then contains the define_arguments function that you can pass a parser per submodule to define the arguments on:
from glob import glob
import os, importlib
def load_submodule_parsers(parent_module, parser, help=None):
if help is None:
help = parent_module.__name__ + " modules"
modules = glob(os.path.join(parent_module.__path__, "*.py"))
subparsers = parser.add_subparsers(help=help)
for module_file in modules:
module_name = os.path.basename(module_file)[:-3]
if module == "__init__":
continue
module = importlib.import_module(module_name, package=parent_module.__name__)
if "define_arguments" not in module.__dict__:
raise ImportError(parent_module.__name__ + " submodule '" + module_name + "' does not have required 'define_arguments' function.")
parser = subparsers.add_parser(module_name)
module.define_arguments(parser)
Pass the function the parent module object:
import argparse, models, schedulers
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
models_parser = subparsers.add_parser("models")
load_submodule_parsers(models, models_parser)
schedulers_parser = subparsers.add_parser("schedulers")
load_submodule_parsers(schedulers, schedulers_parser)
Untested code but I think you can refine it from here
here is what I am basically trying to do:
import sh, os
with sh.cd('/tmp'):
print os.getcwd()
print os.getcwd()
I get the following error though
line 3, in <module>
with sh.cd('/tmp'):
AttributeError: __exit__
What am I missing here? Are there alternative solutions to change directory within a context?
You can't use just any class/function as a context manager, it has to actually explicitly be implemented that way, using either the contextlib.contextmanager decorator on a function, or in the case of a class, by defining the __enter__ and __exit__ instance methods.
The sh.cd function you're using is simply a wrapper around os.chdir:
>>> import sh
>>> sh.cd
<bound method Environment.b_cd of {}>
b_cd is defined as:
def b_cd(self, path):
os.chdir(path)
As you can see, it's just a normal function; it can't be used as a context manager.
The link whereswalden provided shows a good way of implementing the behavior you want as a class. It could similarly be implemented as a function like this:
import contextlib
import os
#contextlib.contextmanager
def cd(path):
old_path = os.getcwd()
os.chdir(path)
try:
yield
finally:
os.chdir(old_path)
Sample usage:
print(os.getcwd())
with cd("/"):
print os.getcwd()
print(os.getcwd())
Output:
'/home/dan'
'/'
'/home/dan'
sh now has the pushd() function that can be used as a context manager to change the current directory temporarily:
import sh
with sh.pushd("/tmp"):
sh.touch("a_file")
See https://amoffat.github.io/sh/sections/command_class.html?highlight=pushd#pushd
I have a data structure that looks like this in python 2.7
myfile.py
--parsers
--folder1
file1.py
def filemethod(data=None)
pass
Under the folder parsers, I can add many subfolders
I will always know then name of the function I want to call however
How do I do an import the parser directory so I can find the methods in each of the sub directory and accessible from myfile.py. I use getattr to convert a name to a function object. This is necessary because I get the name of the function to call from a remote call to a redis queue.
import ??????
methodToCall = getattr('filemethod', 'file1')
methodToCall(data)
A good way to do dynamic imports is using imp.load_source():
import imp
module = imp.load_source( 'mymodule', module_full_path )
in your case it will be something like:
module = imp.load_source( 'file1', '.\parsers\file1.py')
methodToCall = getattr( module, 'filemethod' )
methodToCall( data )
Make sure you replace 'file1' and '.\parsers\file1.py' with your desired module name and the correct path to its source file.
Another way is to first import the subdirectories from parsers/__init__.py.
parsers/__init__.py:
import folder1
import folder2
...
then:
import parsers
foldername = 'folder1' # for example
mod = getattr(parsers, foldername)
methodToCall = getattr(mod, 'filemethod')
methodToCall(data)
If I want the path of the current module, I'll use __file__.
Now let's say I want a function to return that. I can't do:
def get_path():
return __file__
Because it will return the path of the module the function has been declared in.
I need it to work even if the function is not called at the root of the module but at any level of nesting.
This is how I would do it:
import sys
def get_path():
namespace = sys._getframe(1).f_globals # caller's globals
return namespace.get('__file__')
Get it from the globals dict in that case:
def get_path():
return globals()['__file__']
Edit in response to the comment: given the following files:
# a.py
def get_path():
return 'Path from a.py: ' + globals()['__file__']
# b.py
import a
def get_path():
return 'Path from b.py: ' + globals()['__file__']
print get_path()
print a.get_path()
Running this will give me the following output:
C:\workspace>python b.py
Path from b.py: b.py
Path from a.py: C:\workspace\a.py
Next to the absolute/relative paths being different (for brevity, lets leave that out), it looks good to me.
I found a way to do it with the inspect module. I'm ok with this solution, but if somebody find a way to do it without dumping the whole stacktrace, it would be cleaner and I would accept his answer gratefully:
def get_path():
frame, filename, line_number, function_name, lines, index =\
inspect.getouterframes(inspect.currentframe())[1]
return filename
Assuming my_function() is located in my_apps.views I would like to import my_function dynamically without using something like exec or eval.
Is there anyway to accomplish this. I'm looking to do something similar to:
my_function = import_func("my_apps.views.my_function")
my_function()
... code is executed
you want
my_function = getattr(__import__('my_apps.views'), 'my_function')
If you happen to know the name of the function at compile time, you can shorten this to
my_function = __import__('my_apps.views').my_function
This will load my_apps.views and then assign its my_function attribute to the local my_function.
If you are sure that you only want one function, than this is acceptable. If you want more than one attribute, you can do:
views = __import__('my_apps.views')
my_function = getattr(views, 'my_function')
my_other_function = getattr(views, 'my_other_function')
my_attribute = getattr(views, 'my_attribute')
as it is more readable and saves you some calls to __import__. again, if you know the names, the code can be shortened as above.
You could also do this with tools from the imp module but it's more complicated.
Note that Python 2.7 added the importlib module,
convenience wrappers for __import__() and a backport of 3.1 feature.
This module is a minor subset of what is available in the more full-featured package of the same name from Python 3.1 that provides a complete implementation of import. What is here has been provided to help ease in transitioning from 2.7 to 3.1.
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be specified to the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod). The specified module will be inserted into sys.modules and returned.
def import_by_string(full_name):
module_name, unit_name = full_name.rsplit('.', 1)
return getattr(__import__(module_name, fromlist=['']), unit_name)
exists = import_by_string("os.path.exists")
I just wrote this code and seems what a lot of people need, so even if later i show it
def my_import(module_name,func_names = [],cache = False):
if module_name in globals() and cache:
return True
try:
m = __import__(module_name, globals(), locals(), func_names, -1)
if func_names:
for func_name in func_names:
globals()[func_name] = getattr(m,func_name)
else:
globals()[module_name] = m
return True
except ImportError:
return False
def my_imports(modules):
for module in modules:
if type(module) is tuple:
name = module[0]
funcs = module[1]
else:
name = module
funcs = []
if not my_import(name, funcs):
return module
return ''
def checkPluginsImports(plugin,modules):
c = my_imports(modules)
if c:
print plugin +" has errors!: module '"+c+"' not found"
# example: file test.py with "x" function
def d():
checkPluginsImports('demoPlugin',[('test',['x'])])
d()
x()
Use the standard library pkg_resources
from pkg_resources import EntryPoint
my_function = EntryPoint.parse("my_function=my_apps.views:my_function").load(require=False)
We have four cases separated by the fact whether the module and/or the function fixed or not:
module name is a fixed string, function name is a fixed string:
my_function = __import__('my_apps.views', fromlist=['my_function'].my_function
(altough in this case it is much more simple to use from my_app.views import my_function)
module name is a fixed string, function name is variable:
function_name = ...
.
.
.
my_function = getattr(__import__('my_apps.views', fromlist=[function_name]),
function_name)
module name is variable, function name is fixed string:
module_name = ...
.
.
.
my_function = __import__(module_name, fromlist=['my_function']).my_function
module name is variable, function name is variable:
module_name = ...
.
.
.
function_name = ...
.
.
.
my_function = getattr(__import__(module_name, fromlist=[function_name]),
function_name)
Note: For an empty list (that's the default value) as __import__ keyword argument fromlist not the module, but the package root is returned. For all non-empty lists the actual module returned.
Sources and further information:
Python documentation -> Built-in Functions -> __import__
Python documentation -> Built-in Functions -> getattr