let's say I wanted to make a core library for a project, with functions like:
def foo(x):
"""common, useful function"""
and I want to make these functions globally available in my project so that when I call them in a file, I don't need to import them. I have a virtualenv, so I feel like I should be able to modify my interpreter to make them globally available, but wasn't sure if there was any established methodologies behind this. I am aware it defies some pythonic principles.
It is possible to create a custom "launcher" that sets up some global variables and executes the code in a python file:
from sys import argv
# we read the code of the file passed as the first CLI argument
with open(argv[1]) as fin:
code = fin.read()
# just an example: this will be available in the executed python file
def my_function():
return "World"
global_variables = {
'MY_CONSTANT': "Hello", # prepare a global variable
'my_function': my_function # prepare a global function
}
exec(code, global_variables) # run the file with new global variables
Use it like this: python launcher.py my_dsl_file.py.
Example my_dsl_file.py:
# notice: no imports at all
print(MY_CONSTANT)
print(my_function())
Interestingly Python (at least CPython) uses a different way to setup some useful functions like help. It runs a file called site.py that adds some values to the builtins module.
import builtins
def my_function():
return "World"
builtins.MY_CONSTANT = "Hello"
builtins.my_function = my_function
# run your file like above or simply import it
import <your file>
I wouldn't recommend either of these ways. A simple from <your library> import * is a much better approach.
The downside of the first two variants is that no tool will know anything about your injected globals. E.g. mypy, flake8 and all IDEs i know of will fail.
Related
i am trying to build an infrastructure which uses a text file to dynamically setup an environment in which code should run. the text file will contain paths of python modules to load AND variables which other parts of my code will need. thus, i am trying to use one mechanism for loading modules and variables (versus virtualenv just for modules and another scheme for variables)
my code would look like:
my_script_which_uses_env.py:
import envcontrol_module
import some_other_module_whose_path_is_unknown_at_run_time
def main():
my_envcontrol = envcontrol_module.EnvControlClass()
my_variable = my_envcontrol.get_value("SOME_VARIABLE")
those using my infrastructure would be required to "import envcontrol_module" as the first line in their code. after that import is done, then the syspath has already been modified to include the paths of the modules to load, ie when import envcontrol_module returns, the sys path has the path for some_other_module_whose_path_is_unknown_at_run_time.
at a high level, i think the following is needed:
1. need "import envcontrol_module" to cause a reading of my config text file. this would create a dict with the variable names and values. here i could modify sys.path
2. somehow keep this dict around such that when an instance of EnvControlClass is created, this class is able to access the dict
envcontrol_module.py
def main():
config_dict = read_config_file()
modify_sys_path(config_dict)
class EnvControl():
self.config_dict = config_dict
how can i do the above in a python friendly way:
1. config dict is not global to envcontrol_module
2. maintain the need for this to be transparent to programmers, they simply do "import envcontrol_module" first and then do subsequent imports normally
i know i can do the following, but then the programmers would do the imports in their main() rather than at top of file, which will be different/not as nice/not python-esque:
my_script_which_uses_env.py:
import envcontrol_module
def main():
my_envcontrol = envcontrol_module.EnvControlClass()
import some_other_module_whose_path_is_unknown_at_run_time
my_variable = my_envcontrol.get_value("SOME_VARIABLE")
envcontrol_module.py
class EnvControl():
def __init__(self):
self.config_dict = read_config_file()
modify_sys_path(config_dict)
not sure if i explained this well. if i haven't, let me know and i can clarify more.
thanks!
To illustrate what I am trying to do, let's say I have a module testmod that lives in ./testmod.py. The entire contents of this module is
x = test
I would like to be able to successfully import this module into Python, using any of the tools available in importlib or any other built in library.
Obviously doing a simple import testmod statement from the current directory results in an error: NameError: name 'test' is not defined.
I thought that maybe passing either globals or locals to __import__ correctly would modify the environment inside the script being run, but it does not:
>>> testmod = __import__('testmod', globals={'test': 'globals'}, locals={'test': 'locals'})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/jfoxrabi/testmod.py", line 1, in <module>
x = test
NameError: name 'test' is not defined
I was setting the value of test differently so I could see which dict testmod.x came from if this worked.
Since neither of these seems to work, I am stuck. Is it even possible to accomplish what I am trying to do? I would guess that yes, since this is Python, not Sparta.
I am using Python 3.5 on Anaconda. I would very much prefer not to use external libraries.
Update: The Why
I am importing a module into my program as a configuration file. The reason that I am not using JSON or INI is that I would like to have the full scope of Python's interpreter available to compute the values in the config from expressions. I would like to have certain values that I compute before-hand in the program available to do those calculations.
While I am aware of the fact that this is about as bad as calling eval (I do that too in my program), I am not concerned with the security aspect for the time being. I am, however, quite willing to entertain better solutions should this indeed turn out to be a case of XY.
I came up with a solution based on this answer and the importlib docs. Basically, I have access to the module object before it is loaded by using the correct sequence of calls to importlib:
from importlib.util import spec_from_file_location, module_from_spec
from os.path import splitext, basename
def loadConfig(fileName):
test = 'This is a test'
name = splitext(basename(fileName))[0]
spec = spec_from_file_location(name, fileName)
config = module_from_spec(spec)
config.test = test
spec.loader.exec_module(config)
return config
testmod = loadConfig('./testmod.py')
This is a bit better than modifying builtins, which may have unintended consequences in other parts of the program, and may also restrict the names I can pass in to the module.
I decided to put all the configuration items into a single field accessible at load time, which I named config. This allows me to do the following in testmod:
if 'test' in config:
x = config['test']
The loader now looks like this:
from importlib.util import spec_from_file_location, module_from_spec
from os.path import splitext, basename
def loadConfig(fileName, **kwargs):
name = splitext(basename(fileName))[0]
spec = spec_from_file_location(name, fileName)
config = module_from_spec(spec)
config.config = kwargs
spec.loader.exec_module(config)
return config
testmod = loadConfig('./testmod.py', test='This is a test')
After finding myself using this a bunch of times, I finally ended up adding this functionality to the utility library I maintain, haggis. haggis.load.load_module loads a text file as a module with injection, while haggis.load.module_as_dict does a more advanced version of the same that loads it as a potentially nested configuration file into a dict.
You could screw with Python's builtins to inject your own fake built-in test variable:
import builtins # __builtin__, no s, in Python 2
builtins.test = 5 # or whatever other placeholder value
import testmod
del builtins.test # clean up after ourselves
I want to import a python module without adding its containing folder to the python path. I would want the import look like
from A import B as C
Due to the specific path that shall be used, the import looks like
import imp
A = imp.load_source('A', 'path')
C = A.B
This is quite unhandy with long paths and module names. Is there an easier way? Is there A way, where the module is not added to the local variables (no A)?
If you just don't want A to be visible at a global level, you could stick the import (imp.load_source) inside a function. If you actually don't want a module object at all in the local scope, you can do that too, but I wouldn't recommend it.
If module A is a python source file you could read in the file (or even just the relevant portion that you want) and run an exec on it.
source.py
MY_GLOBAL_VAR = 1
def my_func():
print 'hello'
Let's say you have some code that wants my_func
path = '/path/to/source.py'
execfile(path)
my_func()
# 'hello'
Be aware that you're also going to get anything else defined in the file (like MY_GLOBAL_VAR). Again, this will work, but I wouldn't recommend it
Someone looking at your code won't be able to see where my_func came from.
You're essentially doing the same thing as a from A import * import, which is generally frowned upon in python, because , you could be importing all sorts of things into your namespace that you didn't want. And even if it works now, if the source code changes, it could import names that shadow your own global symbols.
It's potentially a security hole, since you could be exec'ing an untrusted source file.
It's way more verbose than a regular python import.
I'd like to save complex configuration by writing some Python code with variable assignments and functions which I will import later by a ConfigReader class.
Basically so far I've written my config file:
a=1
And a class that works like
c=ConfigReader("C:\my_file")
print(c.a)
For that I used exec(), but now I also want that the config file knows its own filename (since the directory provides information for some variables)
So I need a config file like:
a=parse_project_number_from_dir(__file__)
It seems I need to replace exec() by some imp module loading in Python 3? Is that the easiest way to execute a simple file making it aware of its path?
Moreover I'd like my ConfigReader class to read all variables into a dictionary. (With exec I just looked at locals() ). What should I do now and can I not import the auxiliary imports in config file (like the parse_project_number_from_dir function), but just what I actually define there (i.e. a=)?
When you exec a Python script, there is no filename, because you are giving exec a string or a compiled object. The exec command has no way to know where you got this string.
But there is a way to share the knowledge with the config file that you are passing to exec. For example...
Let's say you have a config file that says
print myname
Then run the following Python script:
ns = {}
ns["myname"] = "sample.cfg"
s = open("sample.cfg","r")
exec s in ns # in 3.x use exec(s,ns) instead
This creates a new namespace, and in that namespace creates a variable named myname. Since the script knows the name of the file that it is about to open, it can assign this to a variable. The effect is the same as exec 'myname = "sample.cfg"' in ns.
Then, when you exec the config file, you share your script's knowledge by telling it to use the ns namespace. If this seems a bit confusing, do a little reading about globals and locals and namespaces.
import sys
sys.path.insert(0,"path to your file")
from yourFile import *
I'm working on pypreprocessor which is a preprocessor that takes c-style directives and I've been able to make it work like a traditional preprocessor (it's self-consuming and executes postprocessed code on-the-fly) except that it breaks library imports.
The problem is: The preprocessor runs through the file, processes it, outputs to a temporary file, and exec() the temporary file. Libraries that are imported need to be handled a little different, because they aren't executed, but rather they are loaded and made accessible to the caller module.
What I need to be able to do is: Interrupt the import (since the preprocessor is being run in the middle of the import), load the postprocessed code as a tempModule, and replace the original import with the tempModule to trick the calling script with the import into believing that the tempModule is the original module.
I have searched everywhere and so far and have no solution.
This Stack Overflow question is the closest I've seen so far to providing an answer:
Override namespace in Python
Here's what I have.
# Remove the bytecode file created by the first import
os.remove(moduleName + '.pyc')
# Remove the first import
del sys.modules[moduleName]
# Import the postprocessed module
tmpModule = __import__(tmpModuleName)
# Set first module's reference to point to the preprocessed module
sys.modules[moduleName] = tmpModule
moduleName is the name of the original module, and tmpModuleName is the name of the postprocessed code file.
The strange part is this solution still runs completely normal as if the first module completed loaded normally; unless you remove the last line, then you get a module not found error.
Hopefully someone on Stack Overflow know a lot more about imports than I do, because this one has me stumped.
Note: I will only award a solution, or, if this is not possible in Python; the best, most detailed explanation of why this is not impossible.
Update: For anybody who is interested, here is the working code.
if imp.lock_held() is True:
del sys.modules[moduleName]
sys.modules[tmpModuleName] = __import__(tmpModuleName)
sys.modules[moduleName] = __import__(tmpModuleName)
The 'imp.lock_held' part detects whether the module is being loaded as a library. The following lines do the rest.
Does this answer your question? The second import does the trick.
Mod_1.py
def test_function():
print "Test Function -- Mod 1"
Mod_2.py
def test_function():
print "Test Function -- Mod 2"
Test.py
#!/usr/bin/python
import sys
import Mod_1
Mod_1.test_function()
del sys.modules['Mod_1']
sys.modules['Mod_1'] = __import__('Mod_2')
import Mod_1
Mod_1.test_function()
To define a different import behavior or to totally subvert the import process you will need to write import hooks. See PEP 302.
For example,
import sys
class MyImporter(object):
def find_module(self, module_name, package_path):
# Return a loader
return self
def load_module(self, module_name):
# Return a module
return self
sys.meta_path.append(MyImporter())
import now_you_can_import_any_name
print now_you_can_import_any_name
It outputs:
<__main__.MyImporter object at 0x009F85F0>
So basically it returns a new module (which can be any object), in this case itself. You may use it to alter the import behavior by returning processe_xxx on import of xxx.
IMO: Python doesn't need a preprocessor. Whatever you are accomplishing can be accomplished in Python itself due to it very dynamic nature, for example, taking the case of the debug example, what is wrong with having at top of file
debug = 1
and later
if debug:
print "wow"
?
In Python 2 there is the imputil module that seems to provide the functionality you are looking for, but has been removed in python 3. It's not very well documented but contains an example section that shows how you can replace the standard import functions.
For Python 3 there is the importlib module (introduced in Python 3.1) that contains functions and classes to modify the import functionality in all kinds of ways. It should be suitable to hook your preprocessor into the import system.