I have a python app which can load and run extensions. It looks like this:
import importlib, sys
sys.path.append('.')
module = importlib.import_module( 'extension' )
Extension = getattr(module, 'Extension')
instance = Extension()
instance.FunctionCall(arg1='foo')
The users can create their own loadable extension by implementing a class I defined. In this case, it is in extension.py:
class Extension:
def FunctionCall(self, arg1):
print(arg1)
This works great for now, but I would like to extend my interface to pass another argument into Extension.FunctionCall
instance.FunctionCall(arg1='foo', arg2='bar')
How can I do this while maintaining reverse compatibility with the existing interface?
Here's what happens to existing users if I make the change:
$ python3 loader.py
Traceback (most recent call last):
File "loader.py", line 9, in <module>
instance.FunctionCall(arg1='foo', arg2='bar')
TypeError: FunctionCall() got an unexpected keyword argument 'arg2'
One option is to catch the TypeError, but is this the best solution? It looks like it could get unmaintainable with several iterations of interfaces.
try:
instance.FunctionCall(arg1='foo', arg2='bar')
except TypeError:
instance.FunctionCall(arg1='foo')
Another option is to define the "interface version" as a property of the extension and then use that in a case statement, but it would be nicer to simply detect what is supported instead of pushing overhead to the users.
if hasattr(Extension, 'Standard') and callable(getattr(Extension, 'Standard')):
standard=instance.Standard()
else:
standard=0
if standard == 0:
instance.FunctionCall(arg1='foo')
elif standard == 1:
instance.FunctionCall(arg1='foo', arg2='bar')
Related
I'm trying to use a third-party lib (docutils) on Google App Engine and have a problem with this code (in docutils):
try:
import pwd
do stuff
except ImportError:
do other stuff
I want the import to fail, as it will on the actual GAE server, but the problem is that it doesn't fail on my development box (ubuntu). How to make it fail, given that the import is not in my own code?
Even easier than messing with __import__ is just inserting None in the sys.modules dict:
>>> import sys
>>> sys.modules['pwd'] = None
>>> import pwd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pwd
In your testing framework, before you cause docutils to be imported, you can perform this setup task:
import __builtin__
self.savimport = __builtin__.__import__
def myimport(name, *a):
if name=='pwd': raise ImportError
return self.savimport(name, *a)
__builtin__.__import__ = myimport
and of course in teardown put things back to normal:
__builtin__.__import__ = self.savimport
Explanation: all import operations go through __builtin__.__import__, and you can reassign that name to have such operations use your own code (alternatives such as import hooks are better for such purposes as performing import from non-filesystem sources, but for purposes such as yours, overriding __builtin__.__import__, as you see above, affords truly simple code).
I have a python package that most commonly used as a CLI tool, but I sometimes run it as a library for my own purposes. (e.g. turning it into a web-app or for unit testing)
For example, I'd like to sys.exit(1) on error to give the end-user a nice error message on exception when used as a CLI command, but raise an Exception when it's used as an imported library.
I am using an entry_points in my setup.py:
entry_points={
'console_scripts': [
'jello=jello.cli:main'
]
}
This works great, but I can't easily differentiate when the package is run at the CLI or it was imported because __name__ is always jello.cli. This is because the Entrypoint is basically importing the package as normal.
I tried creating a __main__.py file and pointing my Entrypoint there, but that did not seem to make a difference.
I'm considering checking sys.argv[0] to see if my program name exists there, but that seems to be a brittle hack. (in case the user aliases the command or something) Any other ideas or am I just doing it wrong? For now I have been passing an as_lib argument to my functions so they will behave differently based on whether they are loaded as a module or run from the CLI, but I'd like to get away from that.
This is a minimal example of a package structure that can be used as a cli and a library simultaneously.
This is the directory structure:
egpkg/
├── setup.py
└── egpkg/
├── __init__.py
├── lib.py
└── cli.py
This is the entry_points in setup.py. It's identical to yours:
entry_points={
"console_scripts": [
"egpkg_cli=egpkg.cli:main",
],
},
__init__.py:
from .lib import func
cli.py
This is where you will define your CLI and handle any issues that your functions, that you define in other python files, raise.
import sys
import argparse
from egpkg import func
def main():
p = argparse.ArgumentParser()
p.add_argument("a", type=int)
args = vars(p.parse_args())
try:
result = func(**args)
except Exception as e:
sys.exit(str(e))
print(f"Got result: {result}", file=sys.stdout)
lib.py
This is where the core of your library is defined, and you should use the library how you would expect your users to use it. When you get values/input that won't work you can raise it.
def func(a):
if a == 0:
raise ValueError("Supplied value can't be 0")
return 10 / a
Then from in the python console or a script you can:
In [1]: from egpkg import func
In [2]: func(2)
Out[2]: 5.0
In [3]: func(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/tmp/egpkg/egpkg/lib.py", line 3, in func
raise ValueError("Supplied value can't be 0")
ValueError: Supplied value can't be 0
Supplied value can't be 0
And from the CLI:
(venv) ~ egpkg_cli 2
Got result: 5.0
(venv) ~ egpkg_cli 0
Supplied value can't be 0
I believe what you want is called a 'main guard' in Python. The module knows its name at any given time. If the module is executed from the CLI, its name is __main__. So, what you can do is put all functionality you want in some function at the module level, and then you can call it differently from within the 'main guard' from how you'd call it as a library. Here's what a 'main guard' looks like:
def myfunc(thing, stuff):
if not stuff:
raise MyExc
if __name__ == '__main__':
try:
myfunc(x, y)
except MyExc:
sys.exit(1)
Here, the sys.exit is only called on error if the module is executed from the command line as a script.
I am trying to implement hostname like module and my target machine in an amazon-ec2. But When I am running the script its giving me below error:
[ansible-user#ansible-master ~]$ ansible node1 -m edit_hostname.py -a node2
ERROR! this task 'edit_hostname.py' has extra params, which is only allowed in the following modules: meta, group_by, add_host, include_tasks, import_role, raw, set_fact, command, win_shell, import_tasks, script, shell, include_vars, include_role, include, win_command
My module is like this:
#!/usr/bin/python
from ansible.module_utils.basic import *
try:
import json
except ImportError:
import simplejson as json
def write_to_file(module, hostname, hostname_file):
try:
with open(hostname_file, 'w+') as f:
try:
f.write("%s\n" %hostname)
finally:
f.close()
except Exception:
err = get_exception()
module.fail_json(msg="failed to write to the /etc/hostname file")
def main():
hostname_file = '/etc/hostname'
module = AnsibleModule(argument_spec=dict(name=dict(required=True, type=str)))
name = module.params['name']
write_to _file(module, name, hostname_file)
module.exit_json(changed=True, meta=name)
if __name__ == "__main__":
main()
I don't know where I am making the mistake. Any help will be greatly appreciated. Thank you.
When developing a new module, I would recommend to use the boilerplate described in the documentation. This also shows that you'll need to use AnsibleModule to define your arguments.
In your main, you should add something like the following:
def main():
# define available arguments/parameters a user can pass to the module
module_args = dict(
name=dict(type='str', required=True)
)
# seed the result dict in the object
# we primarily care about changed and state
# change is if this module effectively modified the target
# state will include any data that you want your module to pass back
# for consumption, for example, in a subsequent task
result = dict(
changed=False,
original_hostname='',
hostname=''
)
module = AnsibleModule(
argument_spec=module_args
supports_check_mode=False
)
# manipulate or modify the state as needed (this is going to be the
# part where your module will do what it needs to do)
result['original_hostname'] = module.params['name']
result['hostname'] = 'goodbye'
# use whatever logic you need to determine whether or not this module
# made any modifications to your target
result['changed'] = True
# in the event of a successful module execution, you will want to
# simple AnsibleModule.exit_json(), passing the key/value results
module.exit_json(**result)
Then, you can call the module like so:
ansible node1 -m mymodule.py -a "name=myname"
ERROR! this task 'edit_hostname.py' has extra params, which is only allowed in the following modules: meta, group_by, add_host, include_tasks, import_role, raw, set_fact, command, win_shell, import_tasks, script, shell, include_vars, include_role, include, win_command
As explained by your error message, an anonymous default parameter is only supported by a limited number of modules. In your custom module, the paramter you created is called name. Moreover, you should not include the .py extension in the module name. You have to call your module like so as an ad-hoc command:
$ ansible node1 -m edit_hostname -a name=node2
I did not test your module code so you may have further errors to fix.
Meanwhile, I still strongly suggest you use the default boilerplate from the ansible documentation as proposed in #Simon's answer.
I have the following folder structure:
/main
main.py
/io
__init__.py
foo.py
In Python 2.7 I would write the following in main.py:
import io.foo
or
from io.foo import *
wheareas in Python 3.5 I get an import error:
Traceback (most recent call last):
File "./main.py", line 6, in <module>
import io.foo
ImportError: No module named 'io.foo'; 'io' is not a package
I couldn't find any help so far.
io is a built-in module. Don't name your local packages the same as a built-in module.
While #ErikCederstrand's answer is correct and probably sufficient for you, I was curious as to why it failed so I went digging through cpython's source. So for any future visitors, here's what I found.
The function where it's failing is here: https://github.com/python/cpython/blob/3.4/Lib/importlib/_bootstrap.py#L2207
On line 2209, it checks to see if the parent module has been loaded:
parent = name.rpartition('.')[0] # Value of 'io'
Since it has loaded the builtin io module, it continues on like normal. After the if returns false, it goes on to assign parent module which again is set to "io":
if name in sys.modules:
return sys.modules[name]
parent_module = sys.modules[parent]
The next lines are what cause the failure, and it's because builtin modules (io anyway) don't have a __path__ instance variable. The exception you see raised here are ultimately what you're seeing when you run it:
try:
path = parent_module.__path__
except AttributeError:
msg = (_ERR_MSG + '; {!r} is not a package').format(name, parent)
raise ImportError(msg, name=name)
If you change your module name like Erik says, then step through this whole process, you can see the call to get parent_module.__path__ works like it's supposed to and everything's happy.
So, tldr: you've tricked the import system into thinking it's already loaded your custom module, but when it goes to try and use it like a custom module it fails because it's actually the builtin io.
EDIT: It looks like __path__ is set here after it goes through a normal import process in init_module_attrs:
if _override or getattr(module, '__path__', None) is None:
if spec.submodule_search_locations is not None:
try:
module.__path__ = spec.submodule_search_locations
except AttributeError:
pass
I have a module called spellnum. It can be used as a command-line utility (it has the if __name__ == '__main__': block) or it can be imported like a standard Python module.
The module defines a class named Speller which looks like this:
class Speller(object):
def __init__(self, lang="en"):
module = __import__("spelling_" + lang)
# use module's contents...
As you can see, the class constructor loads other modules at runtime. Those modules (spelling_en.py, spelling_es.py, etc.) are located in the same directory as the spellnum.py itself.
Besides spellnum.py, there are other files with utility functions and classes. I'd like to hide those files since I don't want to expose them to the user and since it's a bad idea to pollute the Python's lib directory with random files. The only way to achieve this that I know of is to create a package.
I've come up with this layout for the project (inspired by this great tutorial):
spellnum/ # project root
spellnum/ # package root
__init__.py
spellnum.py
spelling_en.py
spelling_es.py
squash.py
# ... some other private files
test/
test_spellnum.py
example.py
The file __init__.py contains a single line:
from spellnum import Speller
Given this new layout, the code for dynamic module loading had to be changed:
class Speller(object):
def __init__(self, lang="en"):
spelling_mod = "spelling_" + lang
package = __import__("spellnum", fromlist=[spelling_mod])
module = getattr(package, spelling_mod)
# use module as usual
So, with this project layout a can do the following:
Successfully import spellnum inside example.py and use it like a simple module:
# an excerpt from the example.py file
import spellnum
speller = spellnum.Speller(es)
# ...
import spellnum in the tests and run those tests from the project root like this:
$ PYTHONPATH="`pwd`:$PYTHONPATH" python test/test_spellnum.py
The problem
I cannot execute spellnum.py directly with the new layout. When I try to, it shows the following error:
Traceback (most recent call last):
...
File "spellnum/spellnum.py", line 23, in __init__
module = getattr(package, spelling_mod)
AttributeError: 'module' object has no attribute 'spelling_en'
The question
What's the best way to organize all of the files required by my module to work so that users are able to use the module both from command line and from their Python code?
Thanks!
How about keeping spellnum.py?
spellnum.py
spelling/
__init__.py
en.py
es.py
Your problem is, that the package is called the same as the python-file you want to execute, thus importing
from spellnum import spellnum_en
will try to import from the file instead of the package. You could fiddle around with relative imports, but I don't know how to make them work with __import__, so I'd suggest the following:
def __init__(self, lang="en"):
mod = "spellnum_" + lang
module = None
if __name__ == '__main__':
module = __import__(mod)
else:
package = getattr(__import__("spellnum", fromlist=[mod]), mod)