We Keep Coding

Populating DATA in the help() call

If I have the following directory structure:
handy/
- __init__.py
- utils.py
- dir1
- __init__.py
- script.py
I can populate DATA in help() by writing non-keywords into the __init__.py file, for example:
# __init__.py
hello = "xyz"
other = "z"
variables = 1
Now when I do help(handy), it shows:
DATA
hello = 'xyz'
other = 'z'
variables = 1
Are there any other ways to populate the help DATA from outside of the top-level __init__.py file, or is that the only way?

I'm not sure what you have in mind, but since handy/__init__.py is an executable script, you could do something like this:
__init__.py:
from .utils import *
hello = "xyz"
other = "z"
variables = 1
utils.py:
UTILS_CONSTANT = 42
def func():
pass
Which would result in:
>>> import handy
>>> help(handy)
Help on package handy:
NAME
handy
PACKAGE CONTENTS
utils
DATA
UTILS_CONSTANT = 42
hello = 'xyz'
other = 'z'
variables = 1
FILE
c:\stack overflow\handy\__init__.py
>>>
to what help(handy) displays.

Get imports required for re-creating an object

Given an object produced during importing the code, produce the set of imports that are needed to execute that object creation code.
Case 1:
some_obj = module.submodule.Class(42)
get_imports for_object(some_obj)
>>> "import module.submodule"
Case 2 (Sometimes the root module does not import submodules automatically (e.g. Airflow operators)):
some_obj = submodule.Class(42)
get_imports for_object(some_obj)
>>> "from module import submodule"
Case 3 (stretch goal):
some_obj = submodule.Class(sub2.Class2(42))
get_imports for_object(some_obj)
>>> ["from module import submodule", "from module2 import sub2"]
The goal is to produce import lines such that prepending them to object instantiation code will make the instantiation work.

This'll do:
def get_object_imports(obj, sub_obj_class_name=None):
sub_obj_modules = []
if sub_obj_class_name is not None:
for _, attribute_value in obj.__dict__.items():
value_str = str(getattr(attribute_value,'__class__',''))
if ('.' + sub_obj_class_name) in value_str:
sub_obj_modules.append(attribute_value.__module__)
if sub_obj_modules != []:
sub_module_imports = [('import ' + sub_obj_module) for sub_obj_module
in sub_obj_modules]
return ['import ' + obj.__module__] + sub_module_imports
else:
return 'import ' + obj.__module__
Cases (1) & (2) are equivalent, in that running either imports the same module. Note that with above, objects with same class names but different module sources will be included.
Demo:
from module import class1
from other_module import submodule
obj1 = class1()
obj2 = obj1(submodule.class2())
print(get_object_imports(obj2, 'class2'))
# ['import module', 'import other_module.submodule']

How to mock a function, in a function map/dictionary?

I am trying to patch the fun_1 function from the worker_functions dictionary and I seem to be struggling:
cli.py:
import sys
from worker_functions import (
fun_1,
fun_2,
fun_3,
)
FUNCTION_MAP = {
'run_1': fun_1,
'run_2': fun_2,
'run_3': fun_3,
}
def main():
command = sys.argv[1]
tag = sys.argv[2]
action = FUNCTION_MAP[command]
action(tag)
I've tried mocking cli.fun_1 and cli.main.action and cli.action but this is leading to failure.
test_cli.py:
from mock import patch
from cli import main
def make_test_args(tup):
sample_args = ['cli.py']
sample_args.extend(tup)
return sample_args
def test_fun_1_command():
test_args = make_test_args(['run_1', 'fake_tag'])
with patch('sys.argv', test_args),\
patch('cli.fun_1') as mock_action:
main()
mock_action.assert_called_once()
Do I seem to be missing something?

You'll need to patch the references in the FUNCTION_MAP dictionary itself. Use the patch.dict() callable to do so:
from unittest.mock import patch, MagicMock
mock_action = MagicMock()
with patch('sys.argv', test_args),\
patch.dict('cli.FUNCTION_MAP', {'run_1': mock_action}):
# ...
That's because the FUNCTION_MAP dictionary is the location that the function reference is looked up.

How do you customise lxml objectify to parse INF as a string instead of a float?

With python lxml and objectify, it converts the INF into a float. Is there a way to output INF as a string instead of a float?
from lxml import objectify
value='''
<outside>
<inside>INF</inside>
</outside>
'''
root = objectify.fromstring(value)
inside = root.inside
print inside, inside.__class__
Output:
inf <type 'lxml.objectify.FloatElement'>

Set the pytype attribute on the inside tag:
<outside xmlns:py="http://codespeak.net/lxml/objectify/pytype">
<inside py:pytype='str'>INF</inside>
</outside>
Demo:
>>> from lxml import objectify
>>> value = '''
... <outside xmlns:py="http://codespeak.net/lxml/objectify/pytype">
... <inside py:pytype='str'>INF</inside>
... </outside>
... '''
>>> root = objectify.fromstring(value)
>>> inside = root.inside
>>> print inside, inside.__class__
INF <type 'lxml.objectify.StringElement'>

How to compile a string of Python code into a module whose functions can be called?

In Python, I have a string of some Python source code containing functions like:
mySrc = '''
def foo():
print("foo")
def bar():
print("bar")
'''
I'd like to compile this string into some kind of module-like object so I can call the functions contained in the code.
Here's pseudo-code for what I'd like to do:
myMod = myCompile(mySrc)
myMod.foo()
Is this possible in Python? I've tried this, but it does not work:
myMod = compile(mySrc, '', 'exec')
myMod.foo()
This produces an error message like this:
<code object <module> at 0x104154730, file "", line 1>Traceback (most recent call last):
File "myfile.py", line XX, in run
myMod.foo()
AttributeError: 'code' object has no attribute 'foo'

You have to both compile and execute it:
myMod = compile(mySrc, '', 'exec')
exec(myMod)
foo()
You can pass dicts to exec to stop foo from “leaking” out. Combine it with a module object created using types.ModuleType:
from types import ModuleType
…
compiled = compile(mySrc, '', 'exec')
module = ModuleType("testmodule")
exec(compiled, module.__dict__)

In Python 2, you want the magical compiler package:
>>> import compiler
>>> mod = compiler.parseFile("doublelib.py")
>>> mod
Module('This is an example module.\n\nThis is the docstring.\n',
Stmt([Function(None, 'double', ['x'], [], 0,
'Return twice the argument',
Stmt([Return(Mul((Name('x'), Const(2))))]))]))
>>> from compiler.ast import *
>>> Module('This is an example module.\n\nThis is the docstring.\n',
... Stmt([Function(None, 'double', ['x'], [], 0,
... 'Return twice the argument',
... Stmt([Return(Mul((Name('x'), Const(2))))]))]))
Module('This is an example module.\n\nThis is the docstring.\n',
Stmt([Function(None, 'double', ['x'], [], 0,
'Return twice the argument',
Stmt([Return(Mul((Name('x'), Const(2))))]))]))
>>> mod.doc
'This is an example module.\n\nThis is the docstring.\n'
>>> for node in mod.node.nodes:
... print node
...
Function(None, 'double', ['x'], [], 0, 'Return twice the argument',
Stmt([Return(Mul((Name('x'), Const(2))))]))
>>> func = mod.node.nodes[0]
>>> func.code
Stmt([Return(Mul((Name('x'), Const(2))))])
And in Python 3, it's built right in.