We Keep Coding

Getting inputs to a function from a python object

I have a class. This class has a list of functions that are to be evaluated by a different program.
class SomeClass(object):
def __init__(self, context):
self.functions_to_evaluate = []
There is a function that adds functions to an instance of SomeClass, via something like:
new_function = check_number(5)
SomeClassInstance.functions_to_evaluate.append(new_function)
Where check_number is a function that will check if number is greater than 10, let's say.
If I take SomeClassInstance.functions_to_evaluate and print it, I get a bunch of python objects, like so:
<some_library.check_number object at 0x07B35B90>
I am wondering if it is possible for me to extract the input given to check_number, so something like:
SomeClassInstance.functions_to_evaluate[0].python_feature() that will return "5" or whatever the input to check_number was to me.

You can use the standard library functools.partial, which creates a new partially applied function *.
>>> from functools import partial
>>> def check_number(input):
... return input > 10
>>> fn = partial(check_number, 5)
>>> fn.args # this attribute gives you back the bound arguments, as a tuple.
(5,)
>>> fn() # calls the function with the bound arguments.
False
*: actually the partial object is not a function instance, but it is a callable, and from a duck-type perspective it's a function.

If new_function = check_number(5) is a closure, then you can extract this value using __closure__[0].cell_contents:
Example:
def foo(x):
def inn(y):
return x
return inn
s = foo(5)
print(s.__closure__[0].cell_contents)
Output:
5

I understand your confusion, but:
new_function = check_number(5)
Is calling the function, and the new_function variable gets assigned the return value of the function.
If you have this check_number function:
def check_number(input):
return input > 10
Then it will return False, and new_function will be False. Never <some_library.check_number object at 0x07B35B90>.
If you're getting <some_library.check_number object at 0x07B35B90> then your check_number() function is returning something else.

There are probably several ways to skin this cat. But I'd observe first and foremost that you're not adding python function objects to the functions_to_evaluate list, you're adding the evaluations of functions.
You could simply add a tuple of function, args to the list:
SomeClassInstace.functions_to_evaluate.append((check_number, 5))
And then you can:
for f, args in SomeClassInstance.functions_to_evaluate:
print(args)

how to assign a new value to a variable inside a function? [duplicate]

Sorry if this is a dumb question, but I've looked for a while and not really found the answer.
If I'm writing a python function, for example:
def function(in1, in2):
in1=in1+1
in2=in2+1
How do I make these changes stick?
I know why they dont, this has been addressed in many answers, but I couldn't find an answer to the question of how to actually make them do so. Without returning values or making some sort of class, is there really no way for a function to operate on its arguments in a global sense?
I also want these variables to not be global themselves, as in I want to be able to do this:
a=1
b=2
c=3
d=4
function(a,b)
function(c,d)
Is this just wishful thinking?

It can be done but I'm warning you - it won't be pretty! What you can do is to capture the caller frame in your function, then pick up the call line, parse it and extract the arguments passed, then compare them with your function signature and create an argument map, then call your function and once your function finishes compare the changes in the local stack and update the caller frame with the mapped changes. If you want to see how silly it can get, here's a demonstration:
# HERE BE DRAGONS
# No, really, here be dragons, this is strictly for demonstration purposes!!!
# Whenever you use this in code a sweet little pixie is brutally killed!
import ast
import inspect
import sys
def here_be_dragons(funct): # create a decorator so we can, hm, enhance 'any' function
def wrapper(*args, **kwargs):
caller = inspect.getouterframes(inspect.currentframe())[1] # pick up the caller
parsed = ast.parse(caller[4][0], mode="single") # parse the calling line
arg_map = {} # a map for our tracked args to establish global <=> local link
for node in ast.walk(parsed): # traverse the parsed code...
# and look for a call to our wrapped function
if isinstance(node, ast.Call) and node.func.id == funct.__name__:
# loop through all positional arguments of the wrapped function
for pos, var in enumerate(funct.func_code.co_varnames):
try: # and try to find them in the captured call
if isinstance(node.args[pos], ast.Name): # named argument!
arg_map[var] = node.args[pos].id # add to our map
except IndexError:
break # no more passed arguments
break # no need for further walking through the ast tree
def trace(frame, evt, arg): # a function to capture the wrapped locals
if evt == "return": # we're only interested in our function return
for arg in arg_map: # time to update our caller frame
caller[0].f_locals[arg_map[arg]] = frame.f_locals.get(arg, None)
profile = sys.getprofile() # in case something else is doing profiling
sys.setprofile(trace) # turn on profiling of the wrapped function
try:
return funct(*args, **kwargs)
finally:
sys.setprofile(profile) # reset our profiling
return wrapper
And now you can easily decorate your function to enable it to perform this ungodly travesty:
# Zap, there goes a pixie... Poor, poor, pixie. It will be missed.
#here_be_dragons
def your_function(in1, in2):
in1 = in1 + 1
in2 = in2 + 1
And now, demonstration:
a = 1
b = 2
c = 3
d = 4
# Now is the time to play and sing along: Queen - A Kind Of Magic...
your_function(a, b) # bam, two pixies down... don't you have mercy?
your_function(c, d) # now you're turning into a serial pixie killer...
print(a, b, c, d) # Woooo! You made it! At the expense of only three pixie lives. Savage!
# prints: (2, 3, 4, 5)
This, obviously, works only for non-nested functions with positional arguments, and only if you pass simple local arguments, feel free to go down the rabbit hole of handling keyword arguments, different stacks, returned/wrapped/chained calls, and other shenanigans if that's what you fancy.
Or, you know, you can use structures invented for this, like globals, classes, or even enclosed mutable objects. And stop murdering pixies.

If you are looking to modify the value of the variables you could have your code be
def func(a,b):
int1 = a + 2
int2 = b + 3
return int1,int2
a = 2
b = 3
a,b = func(a,b)
This allows you to actually change the values of the a and b variables with the function.

you can do:
def function(in1, in2):
return in1 + 1 , in2 + 1
a, b = function(a,b)
c, d = function(c,d)
python functions are closed -> when function(a,b) s called, a and b get reassigned to a local (to the function) references/pointers in1 and in2, which are not accessible outside of the function. provide references to those new values w/o using globals, you will need to pass that back through return.
When you pass an array or non primitive object into a function, you can modify the object's attributes and have those modifications be visible to other references for that object outside, because the object itself contain the pointers to those values, making the visible to anything else holding a pointer to that object.

Is there a programmatic way to print the definition of a dynamically created function in python? [duplicate]

Suppose I have a Python function as defined below:
def foo(arg1,arg2):
#do something with args
a = arg1 + arg2
return a
I can get the name of the function using foo.func_name. How can I programmatically get its source code, as I typed above?

If the function is from a source file available on the filesystem, then inspect.getsource(foo) might be of help:
If foo is defined as:
def foo(arg1,arg2):
#do something with args
a = arg1 + arg2
return a
Then:
import inspect
lines = inspect.getsource(foo)
print(lines)
Returns:
def foo(arg1,arg2):
#do something with args
a = arg1 + arg2
return a
But I believe that if the function is compiled from a string, stream or imported from a compiled file, then you cannot retrieve its source code.

The inspect module has methods for retrieving source code from python objects. Seemingly it only works if the source is located in a file though. If you had that I guess you wouldn't need to get the source from the object.
The following tests inspect.getsource(foo) using Python 3.6:
import inspect
def foo(arg1,arg2):
#do something with args
a = arg1 + arg2
return a
source_foo = inspect.getsource(foo) # foo is normal function
print(source_foo)
source_max = inspect.getsource(max) # max is a built-in function
print(source_max)
This first prints:
def foo(arg1,arg2):
#do something with args
a = arg1 + arg2
return a
Then fails on inspect.getsource(max) with the following error:
TypeError: <built-in function max> is not a module, class, method, function, traceback, frame, or code object

Just use foo?? or ??foo.
If you are using IPython, then you need to type foo?? or ??foo to see the complete source code. To see only the docstring in the function, use foo? or ?foo. This works in Jupyter notebook as well.
In [19]: foo??
Signature: foo(arg1, arg2)
Source:
def foo(arg1,arg2):
#do something with args
a = arg1 + arg2
return a
File: ~/Desktop/<ipython-input-18-3174e3126506>
Type: function

dis is your friend if the source code is not available:
>>> import dis
>>> def foo(arg1,arg2):
... #do something with args
... a = arg1 + arg2
... return a
...
>>> dis.dis(foo)
3 0 LOAD_FAST 0 (arg1)
3 LOAD_FAST 1 (arg2)
6 BINARY_ADD
7 STORE_FAST 2 (a)
4 10 LOAD_FAST 2 (a)
13 RETURN_VALUE

While I'd generally agree that inspect is a good answer, I'd disagree that you can't get the source code of objects defined in the interpreter. If you use dill.source.getsource from dill, you can get the source of functions and lambdas, even if they are defined interactively.
It also can get the code for from bound or unbound class methods and functions defined in curries... however, you might not be able to compile that code without the enclosing object's code.
>>> from dill.source import getsource
>>>
>>> def add(x,y):
... return x+y
...
>>> squared = lambda x:x**2
>>>
>>> print getsource(add)
def add(x,y):
return x+y
>>> print getsource(squared)
squared = lambda x:x**2
>>>
>>> class Foo(object):
... def bar(self, x):
... return x*x+x
...
>>> f = Foo()
>>>
>>> print getsource(f.bar)
def bar(self, x):
return x*x+x
>>>

To expand on runeh's answer:
>>> def foo(a):
... x = 2
... return x + a
>>> import inspect
>>> inspect.getsource(foo)
u'def foo(a):\n x = 2\n return x + a\n'
print inspect.getsource(foo)
def foo(a):
x = 2
return x + a
EDIT: As pointed out by #0sh this example works using ipython but not plain python. It should be fine in both, however, when importing code from source files.

Since this post is marked as the duplicate of this other post, I answer here for the "lambda" case, although the OP is not about lambdas.
So, for lambda functions that are not defined in their own lines: in addition to marko.ristin's answer, you may wish to use mini-lambda or use SymPy as suggested in this answer.
mini-lambda is lighter and supports any kind of operation, but works only for a single variable
SymPy is heavier but much more equipped with mathematical/calculus operations. In particular it can simplify your expressions. It also supports several variables in the same expression.
Here is how you can do it using mini-lambda:
from mini_lambda import x, is_mini_lambda_expr
import inspect
def get_source_code_str(f):
if is_mini_lambda_expr(f):
return f.to_string()
else:
return inspect.getsource(f)
# test it
def foo(arg1, arg2):
# do something with args
a = arg1 + arg2
return a
print(get_source_code_str(foo))
print(get_source_code_str(x ** 2))
It correctly yields
def foo(arg1, arg2):
# do something with args
a = arg1 + arg2
return a
x ** 2
See mini-lambda documentation for details. I'm the author by the way ;)

You can use inspect module to get full source code for that. You have to use getsource() method for that from the inspect module. For example:
import inspect
def get_my_code():
x = "abcd"
return x
print(inspect.getsource(get_my_code))
You can check it out more options on the below link.
retrieve your python code

to summarize :
import inspect
print( "".join(inspect.getsourcelines(foo)[0]))

Please mind that the accepted answers work only if the lambda is given on a separate line. If you pass it in as an argument to a function and would like to retrieve the code of the lambda as object, the problem gets a bit tricky since inspect will give you the whole line.
For example, consider a file test.py:
import inspect
def main():
x, f = 3, lambda a: a + 1
print(inspect.getsource(f))
if __name__ == "__main__":
main()
Executing it gives you (mind the indention!):
x, f = 3, lambda a: a + 1
To retrieve the source code of the lambda, your best bet, in my opinion, is to re-parse the whole source file (by using f.__code__.co_filename) and match the lambda AST node by the line number and its context.
We had to do precisely that in our design-by-contract library icontract since we had to parse the lambda functions we pass in as arguments to decorators. It is too much code to paste here, so have a look at the implementation of this function.

If you're strictly defining the function yourself and it's a relatively short definition, a solution without dependencies would be to define the function in a string and assign the eval() of the expression to your function.
E.g.
funcstring = 'lambda x: x> 5'
func = eval(funcstring)
then optionally to attach the original code to the function:
func.source = funcstring

Rafał Dowgird's answer states:
I believe that if the function is compiled from a string, stream or imported from a compiled file, then you cannot retrieve its source code.
However, it is possible to retrieve the source code of a function compiled from a string, provided that the compiling code also added an entry to the linecache.cache dict:
import linecache
import inspect
script = '''
def add_nums(a, b):
return a + b
'''
bytecode = compile(script, 'unique_filename', 'exec')
tmp = {}
eval(bytecode, {}, tmp)
add_nums = tmp["add_nums"]
linecache.cache['unique_filename'] = (
len(script),
None,
script.splitlines(True),
'unique_filename',
)
print(inspect.getsource(add_nums))
# prints:
# """
# def add_nums(a, b):
# return a + b
# """
This is how the attrs library creates various methods for classes automatically, given a set of attributes that the class expects to be initialized with. See their source code here. As the source explains, this is a feature primarily intended to enable debuggers such as PDB to step through the code.

I believe that variable names aren't stored in pyc/pyd/pyo files, so you can not retrieve the exact code lines if you don't have source files.

Python - multiple functions - output of one to the next

I know this is super basic and I have been searching everywhere but I am still very confused by everything I'm seeing and am not sure the best way to do this and am having a hard time wrapping my head around it.
I have a script where I have multiple functions. I would like the first function to pass it's output to the second, then the second pass it's output to the third, etc. Each does it's own step in an overall process to the starting dataset.
For example, very simplified with bad names but this is to just get the basic structure:
#!/usr/bin/python
# script called process.py
import sys
infile = sys.argv[1]
def function_one():
do things
return function_one_output
def function_two():
take output from function_one, and do more things
return function_two_output
def function_three():
take output from function_two, do more things
return/print function_three_output
I want this to run as one script and print the output/write to new file or whatever which I know how to do. Just am unclear on how to pass the intermediate outputs of each function to the next etc.
infile -> function_one -> (intermediate1) -> function_two -> (intermediate2) -> function_three -> final result/outfile
I know I need to use return, but I am unsure how to call this at the end to get my final output
Individually?
function_one(infile)
function_two()
function_three()
or within each other?
function_three(function_two(function_one(infile)))
or within the actual function?
def function_one():
do things
return function_one_output
def function_two():
input_for_this_function = function_one()
# etc etc etc
Thank you friends, I am over complicating this and need a very simple way to understand it.

You could define a data streaming helper function
from functools import reduce
def flow(seed, *funcs):
return reduce(lambda arg, func: func(arg), funcs, seed)
flow(infile, function_one, function_two, function_three)
#for example
flow('HELLO', str.lower, str.capitalize, str.swapcase)
#returns 'hELLO'
edit
I would now suggest that a more "pythonic" way to implement the flow function above is:
def flow(seed, *funcs):
for func in funcs:
seed = func(seed)
return seed;

As ZdaR mentioned, you can run each function and store the result in a variable then pass it to the next function.
def function_one(file):
do things on file
return function_one_output
def function_two(myData):
doThings on myData
return function_two_output
def function_three(moreData):
doMoreThings on moreData
return/print function_three_output
def Main():
firstData = function_one(infile)
secondData = function_two(firstData)
function_three(secondData)
This is assuming your function_three would write to a file or doesn't need to return anything. Another method, if these three functions will always run together, is to call them inside function_three. For example...
def function_three(file):
firstStep = function_one(file)
secondStep = function_two(firstStep)
doThings on secondStep
return/print to file
Then all you have to do is call function_three in your main and pass it the file.

For safety, readability and debugging ease, I would temporarily store the results of each function.
def function_one():
do things
return function_one_output
def function_two(function_one_output):
take function_one_output and do more things
return function_two_output
def function_three(function_two_output):
take function_two_output and do more things
return/print function_three_output
result_one = function_one()
result_two = function_two(result_one)
result_three = function_three(result_two)
The added benefit here is that you can then check that each function is correct. If the end result isn't what you expected, just print the results you're getting or perform some other check to verify them. (also if you're running on the interpreter they will stay in namespace after the script ends for you to interactively test them)
result_one = function_one()
print result_one
result_two = function_two(result_one)
print result_two
result_three = function_three(result_two)
print result_three
Note: I used multiple result variables, but as PM 2Ring notes in a comment you could just reuse the name result over and over. That'd be particularly helpful if the results would be large variables.

It's always better (for readability, testability and maintainability) to keep your function as decoupled as possible, and to write them so the output only depends on the input whenever possible.
So in your case, the best way is to write each function independently, ie:
def function_one(arg):
do_something()
return function_one_result
def function_two(arg):
do_something_else()
return function_two_result
def function_three(arg):
do_yet_something_else()
return function_three_result
Once you're there, you can of course directly chain the calls:
result = function_three(function_two(function_one(arg)))
but you can also use intermediate variables and try/except blocks if needed for logging / debugging / error handling etc:
r1 = function_one(arg)
logger.debug("function_one returned %s", r1)
try:
r2 = function_two(r1)
except SomePossibleExceptio as e:
logger.exception("function_two raised %s for %s", e, r1)
# either return, re-reraise, ask the user what to do etc
return 42 # when in doubt, always return 42 !
else:
r3 = function_three(r2)
print "Yay ! result is %s" % r3
As an extra bonus, you can now reuse these three functions anywhere, each on it's own and in any order.
NB : of course there ARE cases where it just makes sense to call a function from another function... Like, if you end up writing:
result = function_three(function_two(function_one(arg)))
everywhere in your code AND it's not an accidental repetition, it might be time to wrap the whole in a single function:
def call_them_all(arg):
return function_three(function_two(function_one(arg)))
Note that in this case it might be better to decompose the calls, as you'll find out when you'll have to debug it...

I'd do it this way:
def function_one(x):
# do things
output = x ** 1
return output
def function_two(x):
output = x ** 2
return output
def function_three(x):
output = x ** 3
return output
Note that I have modified the functions to accept a single argument, x, and added a basic operation to each.
This has the advantage that each function is independent of the others (loosely coupled) which allows them to be reused in other ways. In the example above, function_two() returns the square of its argument, and function_three() the cube of its argument. Each can be called independently from elsewhere in your code, without being entangled in some hardcoded call chain such as you would have if called one function from another.
You can still call them like this:
>>> x = function_one(3)
>>> x
3
>>> x = function_two(x)
>>> x
9
>>> x = function_three(x)
>>> x
729
which lends itself to error checking, as others have pointed out.
Or like this:
>>> function_three(function_two(function_one(2)))
64
if you are sure that it's safe to do so.
And if you ever wanted to calculate the square or cube of a number, you can call function_two() or function_three() directly (but, of course, you would name the functions appropriately).

With d6tflow you can easily chain together complex data flows and execute them. You can quickly load input and output data for each task. It makes your workflow very clear and intuitive.
import d6tlflow
class Function_one(d6tflow.tasks.TaskCache):
function_one_output = do_things()
self.save(function_one_output) # instead of return
#d6tflow.requires(Function_one)
def Function_two(d6tflow.tasks.TaskCache):
output_from_function_one = self.inputLoad() # load function input
function_two_output = do_more_things()
self.save(function_two_output)
#d6tflow.requires(Function_two)
def Function_three():
output_from_function_two = self.inputLoad()
function_three_output = do_more_things()
self.save(function_three_output)
d6tflow.run(Function_three()) # executes all functions
function_one_output = Function_one().outputLoad() # get function output
function_three_output = Function_three().outputLoad()
It has many more useful features like parameter management, persistence, intelligent workflow management. See https://d6tflow.readthedocs.io/en/latest/

This way function_three(function_two(function_one(infile))) would be the best, you do not need global variables and each function is completely independent of the other.
Edited to add:
I would also say that function3 should not print anything, if you want to print the results returned use:
print function_three(function_two(function_one(infile)))
or something like:
output = function_three(function_two(function_one(infile)))
print output

Use parameters to pass the values:
def function1():
foo = do_stuff()
return function2(foo)
def function2(foo):
bar = do_more_stuff(foo)
return function3(bar)
def function3(bar):
baz = do_even_more_stuff(bar)
return baz
def main():
thing = function1()
print thing

How do I get the return value when using Python exec on the code object of a function?

For testing purposes I want to directly execute a function defined inside of another function.
I can get to the code object of the child function, through the code (func_code) of the parent function, but when I exec it, i get no return value.
Is there a way to get the return value from the exec'ed code?

Yes, you need to have the assignment within the exec statement:
>>> def foo():
... return 5
...
>>> exec("a = foo()")
>>> a
5
This probably isn't relevant for your case since its being used in controlled testing, but be careful with using exec with user defined input.

A few years later, but the following snippet helped me:
the_code = '''
a = 1
b = 2
return_me = a + b
'''
loc = {}
exec(the_code, globals(), loc)
return_workaround = loc['return_me']
print(return_workaround) # 3
exec() doesn't return anything itself, but you can pass a dict which has all the local variables stored in it after execution. By accessing it you have a something like a return.
I hope it helps someone.

While this is the ugliest beast ever seen by mankind, this is how you can do it by using a global variable inside your exec call:
def my_exec(code):
exec('global i; i = %s' % code)
global i
return i
This is misusing global variables to get your data across the border.
>>> my_exec('1 + 2')
3
Needless to say that you should never allow any user inputs for the input of this function in there, as it poses an extreme security risk.

use eval() instead of exec(), it returns result

Something like this can work:
def outer():
def inner(i):
return i + 10
for f in outer.func_code.co_consts:
if getattr(f, 'co_name', None) == 'inner':
inner = type(outer)(f, globals())
# can also use `types` module for readability:
# inner = types.FunctionType(f, globals())
print inner(42) # 52
The idea is to extract the code object from the inner function and create a new function based on it.
Additional work is required when an inner function can contain free variables. You'll have to extract them as well and pass to the function constructor in the last argument (closure).

Here's a way to return a value from exec'd code:
def exec_and_return(expression):
exec(f"""locals()['temp'] = {expression}""")
return locals()['temp']
I'd advise you to give an example of the problem you're trying to solve. Because I would only ever use this as a last resort.

This doesn't get the return value per say, but you can provide an empty dictionary when calling exec to retrieve any variables defined in the code.
# Python 3
ex_locals = {}
exec("a = 'Hello world!'", None, ex_locals)
print(ex_locals['a'])
# Output: Hello world!
From the Python 3 documentation on exec:
The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns.
For more information, see How does exec work with locals?

Here's a solution with a simple code:
# -*- coding: utf-8 -*-
import math
x = [0]
exec("x[0] = 3*2")
print(x[0]) # 6

Since Python 3.7, dictionary are ordered. So you no longer need to agree on a name, you can just say "last item that got created":
>>> d = {}
>>> exec("def addone(i): return i + 1", d, d)
>>> list(d)
['__builtins__', 'addone']
>>> thefunction = d[list(d)[-1]]
>>> thefunction
<function addone at 0x7fd03123fe50>

if we need a function that is in a file in another directory, eg
we need the function1 in file my_py_file.py
located in /home/.../another_directory
we can use the following code:
def cl_import_function(a_func,py_file,in_Dir):
... import sys
... sys.path.insert(0, in_Dir)
... ax='from %s import %s'%(py_file,a_func)
... loc={}
... exec(ax, globals(), loc)
... getFx = loc[afunc]
... return getFx
test = cl_import_function('function1',r'my_py_file',r'/home/.../another_directory/')
test()
(a simple way for newbies...)

program = 'a = 5\nb=10\nprint("Sum =", a + b)'
program = exec(program)
print(program)