We Keep Coding

Possible to import a python file without knowing it's name? [duplicate]

I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__() function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.
Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
See also: How do I import a module given the full path?

With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module for Python 2 and Python 3.
Or using __import__ you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.

The recommended way for Python 2.7 and 3.1 and later is to use importlib module:
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')

Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required.
Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst

Using importlib
Importing a source file
Here is a slightly adapted example from the documentation:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Verify contents of the module:
print(dir(module))
From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().
To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:
sys.modules[module_name] = module
from pluginX import hello
hello()
Importing a package
To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
print(dir(pkg))

Use the imp module, or the more direct __import__() function.

You can use exec:
exec("import myapp.commands.%s" % command)

If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()

Similar as #monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod

The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'

The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".

Pickle a GdkPixbuf.Pixbuf object in Python3

I want to pickle and unpickle a GdkPixbuf.Pixbuf in Python3. To be more specific the multiprocessing package of Python3 need to do it because I share such objects between process via a Queue.
The problem is that the object changes from
<GdkPixbuf.Pixbuf object at 0x7f8b9e9cfb88 (GdkPixbuf at 0x563b61725c60)>
to
<GdkPixbuf.Pixbuf object at 0x7f8b9e9eaea0 (uninitialized at 0x(nil))>
That is the minimal working example.
>>> import gi
>>> from gi.repository import GdkPixbuf
__main__:1: PyGIWarning: GdkPixbuf was imported without specifying a version first. Use gi.require_version('GdkPixbuf', '2.0') before import to ensure that the right version gets loaded.
>>> pf = GdkPixbuf.Pixbuf.new_from_file('_icon.png')
>>> pf
<GdkPixbuf.Pixbuf object at 0x7f8b9e9cfb88 (GdkPixbuf at 0x563b61725c60)>
>>> import pickle
>>> pickle.dump(pf, open('p', 'wb'))
>>> pb2 = pickle.load(open('p', 'rb'))
>>> pb2
<GdkPixbuf.Pixbuf object at 0x7f8b9e9eaea0 (uninitialized at 0x(nil))>
I see no other way to pickle. The icon need to be loaded in a separate process (on a different CPU core then the applications main/first process) and then should be transfered to the main process. This is done via a Queue which pickles all data.

My solution is holding the "icon" not as a Pixbuf object in memory but as raw bytes I read from the file.
After unpickling this bytes I convert them to a Pixbuf.
>>> import gi
>>> from gi.repository import GdkPixbuf, Gio, GLib
__main__:1: PyGIWarning: GdkPixbuf was imported without specifying a version first. Use gi.require_version('GdkPixbuf', '2.0') before import to ensure that the right version gets loaded.
>>> with open('_icon.png', 'rb') as f:
... icon_bytes = f.read()
...
>>>
>>> import pickle
>>> pickle.dump(icon_bytes, open('p', 'wb'))
>>>
>>> pb = pickle.load(open('p', 'rb'))
>>> pb = GLib.Bytes(pb)
>>> pb = GdkPixbuf.Pixbuf.new_from_stream(Gio.MemoryInputStream.new_from_bytes(pb))
>>> pb
<GdkPixbuf.Pixbuf object at 0x7fc0858ac5e8 (GdkPixbuf at 0x55e0d8d08b60)>

Python os.environ throws key error?

I'm accessing an environment variable in a script with os.environ.get and it's throwing a KeyError. It doesn't throw the error from the Python prompt. This is running on OS X 10.11.6, and is Python 2.7.10.
What is going on?
$ python score.py
Traceback (most recent call last):
File "score.py", line 4, in <module>
setup_logging()
File "/score/log.py", line 29, in setup_logging
config = get_config()
File "/score/log.py", line 11, in get_config
environment = os.environ.get('NODE_ENV')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py", line 23, in __getitem__
raise KeyError(key)
KeyError: 'NODE_ENV'
$ python -c "import os; os.environ.get('NODE_ENV')"
$
As requested, here's the source code for score.py
from __future__ import print_function
from log import get_logger, setup_logging
setup_logging()
log = get_logger('score')
And here's log.py
import json
import os
import sys
from iron_worker import IronWorker
from logbook import Logger, Processor, NestedSetup, StderrHandler, SyslogHandler
IRON_IO_TASK_ID = IronWorker.task_id()
def get_config():
environment = os.environ.get('NODE_ENV')
if environment == 'production':
filename = '../config/config-production.json'
elif environment == 'integration':
filename = '../config/config-integration.json'
else:
filename = '../config/config-dev.json'
with open(filename) as f:
return json.load(f)
def setup_logging():
# This defines a remote Syslog handler
# This will include the TASK ID, if defined
app_name = 'scoreworker'
if IRON_IO_TASK_ID:
app_name += '-' + IRON_IO_TASK_ID
config = get_config()
default_log_handler = NestedSetup([
StderrHandler(),
SyslogHandler(
app_name,
address = (config['host'], config['port']),
level = 'ERROR',
bubble = True
)
])
default_log_handler.push_application()
def get_logger(name):
return Logger(name)

Try running:
find . -name \*.pyc -delete
To delete your .pyc files.
Researching your problem I came across this question, where a user was experiencing the same thing: .get() seemingly raising a KeyError. In that case, it was caused, according to this accepted answer, by a .pyc file which contained code where a dict value was being accessed by key (i.e., mydict['potentially_nonexistent_key']), while the traceback was showing the code from the updated .py file where .get() was used. I have never heard of this happening, where the traceback references current code from a .py file, but shows an error raised by an outdated .pyc file, but it seems to have happened at least once in the history of Python...
It is a long shot, but worth a try I thought.

I encountered a similar error when I set the environment variable without exporting it. So if you do this:
me#host:/# NODE_ENV=foo
You will get this:
me#host:/# python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> node_env = os.environ['NODE_ENV']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/os.py", line 675, in __getitem__
raise KeyError(key) from None
KeyError: 'NODE_ENV'
>>>
But if you do this:
me#host:/# NODE_ENV=foo
me#host:/# export NODE_ENV
It works:
me#host:/# python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> node_env = os.environ['NODE_ENV']
>>> print(node_env)
foo
>>>

Command for windows to delete the .pyc files:
del /S *.pyc

I had the same problem. I solved that by making some corrections on the .env file:
Before:
Key = Value
After my correction:
Key=Value
without blank spaces and worked!

I was getting this error while trying to source from a .env file.
I didn't explicitly export the env vars so I had to change this.
ENVIRONMENT=DEV
to this
export ENVIRONMENT=DEV

Use export a=10 instead of a=10 while setting env variable. Add the same in ~./bashrc to reload the env var wherever you login.
Doing this resolved the issue

I'd recommend you start debugging os.py, for instance, on windows it's being used this implementation:
def get(self, key, failobj=None):
print self.data.__class__
print key
return self.data.get(key.upper(), failobj)
And if I test it with this:
import os
try:
os.environ.get('NODE_ENV')
except Exception as e:
print("-->{0}".format(e.__class__))
os.environ['NODE_ENV'] = "foobar"
try:
os.environ.get('NODE_ENV')
except Exception as e:
print("{0}".format(e.__class__))
The output will be:
<type 'dict'>
PYTHONUSERBASE
<type 'dict'>
APPDATA
<type 'dict'>
NODE_ENV
<type 'dict'>
NODE_ENV
So it makes sense the exception is not spawned reading dict.get docs.
In any case, if you don't want to mess up or debugging the python modules, try cleaning up the *.pyc files, try to set up properly NODE_ENV. And if all that don't work, restart your terminal to clear up.

How to call a function stored in another file from a Python program?

If I have a text file that contains a python function definition, how can I make the function call from another Python program. Ps: The function will be defined in the Python program that does the call.
Ways in which can be done:
Consider the python function as a module and call it. Constraint here is that I have to convert a python bare function into a module which would give errors.
Insert the code(function code) into the program that calls the function.
Which would be the better way to go about it?
Edit: Thank you for all the replies. Have shed a lot of light on the initial confusion I myself had. Another doubt would be, what if the person(Obviously, not me) has written a os.system("rm -rf"). And I end up executing it. That would mean doomsday for me, right?
Edit2: As a lot of you have asked me to use exec, I would like to point to the this thread and most particularly the namespace problem. It gives user a lot of chances to "circumvent" python. Don't y'all think?

You are looking for the exec keyword.
>>> mycode = 'print "hello world"'
>>> exec mycode
Hello world
So if you read your text file as text (assuming that it only contains the function) like:
test.txt:
def a():
print "a()"
test.py:
mycode = open('test.txt').read()
exec mycode # this will execute the code in your textfile, thus define the a() function
a() # now you can call the function from your python file
Link to doc: http://docs.python.org/reference/simple_stmts.html#grammar-token-exec%5Fstmt
You may want to look at the compile statement too: here.

compile() and eval() can do the trick:
>>> code = compile('def foo(a): return a*2', '<string>', 'exec')
>>> eval(code)
>>> foo
52: <function foo at 0x01F65F70>
>>> foo(12)
53: 24
or with file:
with open(filename) as source:
eval(compile(source.read(), filename, 'exec'))

A way like Reflection in Java? If so, Python has a module named imp to provide it.
foo.py
def foo():
return "return from function foo in file foo.py"
some code anywhere
modes = imp.get_suffixes() #got modes Explained in link below
mode = modes[-2] # because I want load a py file
with open("foo.py") as file:
m = imp.load_module("name", file, "foo.py", mode)
print(m.foo())
above mode = modes[-2] because my imp.get_suffixes() is:
>>> imp.get_suffixes()
[('.cpython-32m.so', 'rb', 3), ('module.cpython-32m.so', 'rb', 3), ('.abi3.so', 'rb', 3), ('module.abi3.so', 'rb', 3), ('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1), ('.pyc', 'rb', 2)]
here is my output:
Python 3.2.1 (default, Aug 11 2011, 01:27:29)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import imp
>>> with open("foo.py") as file:
... m = imp.load_module("foo", file, "foo.py", ('.py', 'U', 1))
...
>>> m.foo()
'return from function foo in file foo.py'
Check it here: http://docs.python.org/py3k/library/imp.html
Both python 2.7 and python 3 works:
Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import imp
>>> imp.get_suffixes()
[('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1), ('.pyc', 'rb', 2)]
>>> with open("foo.py") as file:
... m = imp.load_module("foo", file, "foo.py", ('.py', 'U', 1))
...
>>> m.foo()
'return from function foo in file foo.py'

You can use execfile:
execfile("path/example.py")
# example.py
# def example_func():
# return "Test"
#
print example_func()
# >Test
EDIT:
In case you want to execute some unsecure code, you can try to sandbox it this way,
although it is probably not very safe anyway:
def execfile_sandbox(filename):
from copy import copy
loc = globals()
bi = loc["__builtins__"]
if not isinstance(bi, dict): bi = bi.__dict__
bi = copy(bi)
# no files
del bi["file"]
# and definitely, no import
del bi["__import__"]
# you can delete other builtin functions you want to deny access to
new_locals = dict()
new_locals["__builtins__"] = bi
execfile(filename, new_locals, new_locals)
Usage:
try:
execfile_sandbox("path/example.py")
except:
# handle exception and errors here (like import error)
pass

I am not sure what is your purpose, but I suppose that you have function in one program and you do want that function run in another program. You can "marshal" function from first to second.
Example, first program:
# first program
def your_func():
return "your function"
import marshal
marshal.dump(your_func.func_code, file("path/function.bin","w"))
Second program:
# Second program
import marshal, types
code = marshal.load(file("path/function.bin"))
your_func = types.FunctionType(code, globals(), "your_func")
print your_func()
# >your function

How to save a configuration file / python file IO

I have this python code for opening a .cfg file, writing to it and saving it:
import ConfigParser
def get_lock_file():
cf = ConfigParser.ConfigParser()
cf.read("svn.lock")
return cf
def save_lock_file(configurationParser):
cf = configurationParser
config_file = open('svn.lock', 'w')
cf.write(config_file)
config_file.close()
Does this seem normal or am I missing something about how to open-write-save files? Is there a more standard way to read and write config files?
I ask because I have two methods that seem to do the same thing, they get the config file handle ('cf') call cf.set('blah', 'foo' bar) then use the save_lock_file(cf) call above. For one method it works and for the other method the write never takes place, unsure why at this point.
def used_like_this():
cf = get_lock_file()
cf.set('some_prop_section', 'some_prop', 'some_value')
save_lock_file(cf)

Just to note that configuration file handling is simpler with ConfigObj.
To read and then write a config file:
from configobj import ConfigObj
config = ConfigObj(filename)
value = config['entry']
config['entry'] = newvalue
config.write()

Looks good to me.
If both places call get_lock_file, then cf.set(...), and then save_lock_file, and no exceptions are raised, this should work.
If you have different threads or processes accessing the same file you could have a race condition:
thread/process A reads the file
thread/process B reads the file
thread/process A updates the file
thread/process B updates the file
Now the file only contains B's updates, not A's.
Also, for safe file writing, don't forget the with statement (Python 2.5 and up), it'll save you a try/finally (which you should be using if you're not using with). From ConfigParser's docs:
with open('example.cfg', 'wb') as configfile:
config.write(configfile)

Works for me.
C:\temp>type svn.lock
[some_prop_section]
Hello=World
C:\temp>python
ActivePython 2.6.2.2 (ActiveState Software Inc.) based on
Python 2.6.2 (r262:71600, Apr 21 2009, 15:05:37) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ConfigParser
>>> def get_lock_file():
... cf = ConfigParser.ConfigParser()
... cf.read("svn.lock")
... return cf
...
>>> def save_lock_file(configurationParser):
... cf = configurationParser
... config_file = open('svn.lock', 'w')
... cf.write(config_file)
... config_file.close()
...
>>> def used_like_this():
... cf = get_lock_file()
... cf.set('some_prop_section', 'some_prop', 'some_value')
... save_lock_file(cf)
...
>>> used_like_this()
>>> ^Z
C:\temp>type svn.lock
[some_prop_section]
hello = World
some_prop = some_value
C:\temp>