imagine you have an io heavy function like this:
def getMd5Sum(path):
with open(path) as f:
return md5(f.read()).hexdigest()
Do you think Python is flexible enough to allow code like this (notice the $):
def someGuiCallback(filebutton):
...
path = filebutton.getPath()
md5sum = $getMd5Sum()
showNotification("Md5Sum of file: %s" % md5sum)
...
To be executed something like this:
def someGuiCallback_1(filebutton):
...
path = filebutton.getPath()
Thread(target=someGuiCallback_2, args=(path,)).start()
def someGuiCallback_2(path):
md5sum = getMd5Sum(path)
glib.idle_add(someGuiCallback_3, md5sum)
def someGuiCallback_3(md5sum):
showNotification("Md5Sum of file: %s" % md5sum)
...
(glib.idle_add just pushes a function onto the queue of the main thread)
I've thought about using decoraters, but they don't allow me to access the 'content' of the function after the call. (the showNotification part)
I guess I could write a 'compiler' to change the code before execution, but it doesn't seam like the optimal solution.
Do you have any ideas, on how to do something like the above?
You can use import hooks to achieve this goal...
PEP 302 - New Import Hooks
PEP 369 - Post Import Hooks
... but I'd personally view it as a little bit nasty.
If you want to go down that route though, essentially what you'd be doing is this:
You add an import hook for an extension (eg ".thpy")
That import hook is then responsible for (essentially) passing some valid code as a result of the import.
That valid code is given arguments that effectively relate to the file you're importing.
That means your precompiler can perform whatever transformations you like to the source on the way in.
On the downside:
Whilst using import hooks in this way will work, it will surprise the life out of any maintainer or your code. (Bad idea IMO)
The way you do this relies upon imputil - which has been removed in python 3.0, which means your code written this way has a limited lifetime.
Personally I wouldn't go there, but if you do, there's an issue of the Python Magazine where doing this sort of thing is covered in some detail, and I'd advise getting a back issue of that to read up on it. (Written by Paul McGuire, April 2009 issue, probably available as PDF).
Specifically that uses imputil and pyparsing as it's example, but the principle is the same.
How about something like this:
def performAsync(asyncFunc, notifyFunc):
def threadProc():
retValue = asyncFunc()
glib.idle_add(notifyFunc, retValue)
Thread(target=threadProc).start()
def someGuiCallback(filebutton):
path = filebutton.getPath()
performAsync(
lambda: getMd5Sum(path),
lambda md5sum: showNotification("Md5Sum of file: %s" % md5sum)
)
A bit ugly with the lambdas, but it's simple and probably more readable than using precompiler tricks.
Sure you can access function code (already compiled) from decorator, disassemble and hack it. You can even access the source of module it's defined in and recompile it. But I think this is not necessary. Below is an example using decorated generator, where yield statement serves as a delimiter between synchronous and asynchronous parts:
from threading import Thread
import hashlib
def async(gen):
def func(*args, **kwargs):
it = gen(*args, **kwargs)
result = it.next()
Thread(target=lambda: list(it)).start()
return result
return func
#async
def test(text):
# synchronous part (empty in this example)
yield # Use "yield value" if you need to return meaningful value
# asynchronous part[s]
digest = hashlib.md5(text).hexdigest()
print digest
Related
I sometime would like to apply small fixes to small issues, just for testing, or for internal purposes.
For example I would like to just replace self.builder.current_docname with node.source in sphinx.writers.html5.HTML5Translator.visit_literal_block.
The (elegant) way of doing it is to copy-paste the method where the patch should be applied then modify the little tiny thing I have to modify, then override the original method with the new one. In this case I would have a lot of imports and I have to copy/paste the whole method locally (bad SSOT is bad).
Instead I would would like to write something like:
from monkey import ReMonkeyPatcher # Fake...
from sphinx.writer.html5 import HTML5Translator
# Override the class's method
ReMonkeyPatcher(HTML5Translator, 'self.builder.current_docname', 'node.source')
# Profit...
I've read here that the reflective nature of Python allows for run-time hacks with inspect and ast. So I wrote this:
def monkey_replace(function, search, replace):
source = inspect.getsource(function)
while code[0].isspace():
code = textwrap.dedent(code)
code = code.replace(search, replace)
ast_tree = ast.parse(code)
compile(ast_tree, '<string>', mode='exec')
mod = {}
exec(code, mod) # <-- Here is the issue...
method = mod[function.__name__]
replace_function(function, method) # Doesn't know how to do that yet
The main issue is exec(code, mod) in which the context is missing. To work in any/most cases, it has to be executed in the context of the original function (imports...).
Is there an elegant way of doing it?
In Python monkey-patching is extremely easy. As it says in the post you reference:
Because there are no protected variables, you can simply overwrite the function you would like to patch:
HTMLTranslator.builder.current_docname = node.source
I'm trying to put together a system that will handle lazy-loading of modules that don't explicitly exist. Basically I have an http server with a number of endpoints that I don't know ahead of time that I would like to programmatically offer for import. These modules would all have a uniform method signature, they just wouldn't exist ahead of time.
import lazy.route as test
import lazy.fake as test2
test('Does this exist?') # This sends a post request.
test2("This doesn't exist.") # Also sends a post request
I can handle all the logic I need around these imports with a uniform decorator, I just can't find any way of "decorating" imports in python, or actually interacting with them in any kind of programmatic way.
Does anyone have experience with this? I've been hunting around, and the closest thing I've found is the ast module, which would lead to a really awful kind of hacky implementation in my current under my current understanding (something like finding all import statements and manually over-writing the import function)
Not looking for a handout, just a piece of the python codebase to start looking at, or an example of someone that's done something similar.
I got a little clever in my googling and managed to find a PEP that specifically addressed this issue, it just happens to be relatively unknown, probably because the subset of reasonable uses for this is pretty narrow.
I found an excellent piece of example code showing off the new sys.meta_path implementation. I've posted it below for information on how to dynamically bootstrap your import statements.
import sys
class VirtualModule(object):
def hello(self):
return 'Hello World!'
class CustomImporter(object):
virtual_name = 'my_virtual_module'
def find_module(self, fullname, path):
"""This method is called by Python if this class
is on sys.path. fullname is the fully-qualified
name of the module to look for, and path is either
__path__ (for submodules and subpackages) or None (for
a top-level module/package).
Note that this method will be called every time an import
statement is detected (or __import__ is called), before
Python's built-in package/module-finding code kicks in."""
if fullname == self.virtual_name:
# As per PEP #302 (which implemented the sys.meta_path protocol),
# if fullname is the name of a module/package that we want to
# report as found, then we need to return a loader object.
# In this simple example, that will just be self.
return self
# If we don't provide the requested module, return None, as per
# PEP #302.
return None
def load_module(self, fullname):
"""This method is called by Python if CustomImporter.find_module
does not return None. fullname is the fully-qualified name
of the module/package that was requested."""
if fullname != self.virtual_name:
# Raise ImportError as per PEP #302 if the requested module/package
# couldn't be loaded. This should never be reached in this
# simple example, but it's included here for completeness. :)
raise ImportError(fullname)
# PEP#302 says to return the module if the loader object (i.e,
# this class) successfully loaded the module.
# Note that a regular class works just fine as a module.
return VirtualModule()
if __name__ == '__main__':
# Add our import hook to sys.meta_path
sys.meta_path.append(CustomImporter())
# Let's use our import hook
import my_virtual_module
print my_virtual_module.hello()
The full blog post is here
I am building a very basic platform in the form of a Python 2.7 module. This module has a read-eval-print loop where entered user commands are mapped to function calls. Since I am trying to make it easy to build plugin modules for my platform, the function calls will be from my Main module to an arbitrary plugin module. I'd like a plugin builder to be able to specify the command that he wants to trigger his function, so I've been looking for a Pythonic way to remotely enter a mapping in the command->function dict in the Main module from the plugin module.
I've looked at several things:
Method name parsing: the Main module would import the plugin module
and scan it for method names that match a certain format. For
example, it might add the download_file_command(file) method to its
dict as "download file" -> download_file_command. However, getting a
concise, easy-to-type command name (say, "dl") requires that the
function's name also be short, which isn't good for code
readability. It also requires the plugin developer to conform to a
precise naming format.
Cross-module decorators: decorators would let
the plugin developer name his function whatever he wants and simply
add something like #Main.register("dl"), but they would necessarily
require that I both modify another module's namespace and keep
global state in the Main module. I understand this is very bad.
Same-module decorators: using the same logic as above, I could add a
decorator that adds the function's name to some command name->function mapping local to the
plugin module and retrieve the mapping to the Main module with an
API call. This requires that certain methods always be present or
inherited though, and - if my understanding of decorators is correct - the function will only register itself the first time it is run and will unnecessarily re-register itself every subsequent time
thereafter.
Thus, what I really need is a Pythonic way to annotate a function with the command name that should trigger it, and that way can't be the function's name. I need to be able to extract the command name->function mapping when I import the module, and any less work on the plugin developer's side is a big plus.
Thanks for the help, and my apologies if there are any flaws in my Python understanding; I'm relatively new to the language.
Building or Standing on the first part of #ericstalbot's answer, you might find it convenient to use a decorator like the following.
################################################################################
import functools
def register(command_name):
def wrapped(fn):
#functools.wraps(fn)
def wrapped_f(*args, **kwargs):
return fn(*args, **kwargs)
wrapped_f.__doc__ += "(command=%s)" % command_name
wrapped_f.command_name = command_name
return wrapped_f
return wrapped
################################################################################
#register('cp')
def copy_all_the_files(*args, **kwargs):
"""Copy many files."""
print "copy_all_the_files:", args, kwargs
################################################################################
print "Command Name: ", copy_all_the_files.command_name
print "Docstring : ", copy_all_the_files.__doc__
copy_all_the_files("a", "b", keep=True)
Output when run:
Command Name: cp
Docstring : Copy many files.(command=cp)
copy_all_the_files: ('a', 'b') {'keep': True}
User-defined functions can have arbitrary attributes. So you could specify that plug-in functions have an attribute with a certain name. For example:
def a():
return 1
a.command_name = 'get_one'
Then, in your module you could build a mapping like this:
import inspect #from standard library
import plugin
mapping = {}
for v in plugin.__dict__.itervalues():
if inspect.isfunction(v) and v.hasattr('command_name'):
mapping[v.command_name] = v
To read about arbitrary attributes for user-defined functions see the docs
There are two parts in a plugin system:
Discover plugins
Trigger some code execution in a plugin
The proposed solutions in your question address only the second part.
There many ways to implement both depending on your requirements e.g., to enable plugins, they could be specified in a configuration file for your application:
plugins = some_package.plugin_for_your_app
another_plugin_module
# ...
To implement loading of the plugin modules:
plugins = [importlib.import_module(name) for name in config.get("plugins")]
To get a dictionary: command name -> function:
commands = {name: func
for plugin in plugins
for name, func in plugin.get_commands().items()}
Plugin author can use any method to implement get_commands() e.g., using prefixes or decorators — your main application shouldn't care as long as get_commands() returns the command dictionary for each plugin.
For example, some_plugin.py (full source):
def f(a, b):
return a + b
def get_commands():
return {"add": f, "multiply": lambda x,y: x*y}
It defines two commands add, multiply.
I have a third-party Python library that allows me to register a callback function that it called later.
While the code works okay with functions, when I tried to pass a method it fails, the callback is never called.
I have no control over the third party library (source code not available).
def old_callbackFunction(param, data):
print data
class MyClass():
def callbackFunction(self, param, data):
print data
myObj = MyClass()
# old_setCallback(myObj.callbackFunction, param = "x") # this would work
setCallback(myObj.callbackFunction, param = "x") # this is never called
Sorin actually figured this out himself, with help from my comment, but he indicated that he wanted me to post the original comment as an answer. I was reluctant to post this originally because I'm unsure of the precise behavior of the setCallback and callbackFunction code; use at your own risk and modify as reason dictates.
The best way to wrap a function is to use functools.partial:
from functools import partial
setCallback(partial(myObj.callbackFunction), param="x")
You may also use a lambda (but you'll lose style points with the pythonistas):
setCallback(lambda param, data: myObj.callbackFunction(param, data), param="x")
Im a coding a library including textual feedback that I need to translate.
I put the following lines in a _config.py module that I import everywhere in my app :
import gettext, os, sys
pathname = os.path.dirname(sys.argv[0])
localdir = os.path.abspath(pathname) + "/locale"
gettext.install("messages", localdir)
I have the *.mo files in ./locale/lang_LANG/LC_MESSAGES and I apply the _() function to all the strings that need to be translated.
Now I just added a feature for the user, supposedly a programmer, to be able to create his own messages. I don't want him to care about the underlying implementation, so I want him to be able to make it something straightforward like :
lib_object.message = "My message"
I used properties to make it clean, but what if my user whats to translate his own code (that uses mine) and does something like :
import gettext, os, sys
pathname = os.path.dirname(sys.argv[0])
localdir = os.path.abspath(pathname) + "/locale"
gettext.install("user_app", localdir)
lib_object.message = _("My message")
Is it a problem ? What can I do to avoid troubles without bothering my user ?
You can use the class based gettext api to isolate message catalogs. This is also what is recommended in the python gettext documentation.
The drawback is that you, or the other dev, will have to use the gettext method or define the _() method in the local scope, bound to the specific gettext class. An example of a class with its own string catalog:
import gettext
class MyClass(object):
def __init__(self, locale_for_instance):
self.lang = gettext.translation("appname", localedir, \
locale=locale_for_instance)
def some_method(self, arg):
return self.lang.gettext("You called some method")
def other_method(self, arg): # does the same thing
_ = self.lang.gettext
return _("You called some method")
You could stick the code for adding the _() in a decorator, so all the methods that need it is prefixed with something like #with_local_gettext
(Note, I've not tested the above could but It Should Work Just Fine(tm) )
If the goal is to not bother your user (and he's not very good) I guess you could use the class based approach in your code and let the user use the global one.
You can only gettext.install() once. In general it's useless for library work -- gettext.install() will only do the right thing if the module calling it is in charge of the whole program, since it will only provide you with one catalog to load from. Library code should do something akin to what Mailman does: have their own wrapper for gettext() that passes the right arguments for this module, then imports that as '_' in each module that wants to use it.