We Keep Coding

how to make new flow controls in python? [duplicate]

This question already has answers here:
Can you add new statements to Python's syntax?
(13 answers)
Closed 2 months ago.
Not sure how to explain, I mean statemtents like:
for i in l:
if a==b:
def x():
lambda x:
class spam:
while True:
basically those control statements that end with :
can I create novel ones? (like in snakemake that has a long list of new control statements)
I tried reading documentation, but could not find anything useful.
I just want to make some tools to help develop rules for snakemake.
I am currently using this:
class SM(object):
def __init__(self,**xargs):
self.items = xargs
def __getattribute__(self,attr):
return object.__getattribute__(self, "items")[attr]
input = SM(genome="Genome/genome.fa",
table="rmats/binding_strength.maxent.CLIP.csv")
table = pd.read_csv(input.table,index_col=0)
In that example I can use the class SM to emulate all the input, output, wildcard... then I can just move the code into its rule in the Snakefile without needing to manually edit all the inputs/wildcards/outputs...
However, I will still need to write the "input:".
Is there a way I could make:
input:
table="table.csv"
do
input=SM(table:"table.csv")
#or input=SM(**xargs)

Sorry, but no can do...You would have to modify the language implementation itself (the interpreter actually). When you are programming you are bound by the syntax of the language, you cannot modify the syntax "on the fly". It's not the same as e.g. defining functions, classes and whatnot.
Take a look at these:
Can you add new statements to Python's syntax?
How to make custom reserved keywords in python3
Here's the most comprehensive answer to this kind of questions imho:
https://stackoverflow.com/a/9108164/15923186

How do I evaluate a variable from a string without using eval()? [duplicate]

This question already has answers here:
How can I select a variable by (string) name?
(5 answers)
Closed 8 months ago.
Suppose I've got a set of background data measurements for different frequencies:
import numpy as np
C_bg_100kHz = 100*np.random.random(1000)
C_bg_200kHz = 200*np.random.random(1000)
Where C_bg_100kHz is some background noise for 100kHz measurements, and C_bg_200kHz is background noise for 200kHz measurements. I would like to create a function that subtracts the mean of these background data measurements from some array of measurement data, where I can specify as one of the function parameters which background data set I want to subtract. I have managed to make this fuction using eval():
def subtract(array_y,freq):
bg = eval('C_bg_' + freq)
return array_y - np.ones(len(array_y))*np.mean(bg)
>>> subtract([50,50,50],'100kHz')
array([-0.36224706, -0.36224706, -0.36224706])
>>> subtract([50,50,50],'200kHz')
array([-47.95860607, -47.95860607, -47.95860607])
Here, I can enter my data as array_y and subtract, for instance, the C_bg_100kHz dataset by passing '100kHz' as the freq input. Essentially, I want python to translate a string 'C_bg_100kHz' to the array C_bg_100kHz. However, this function uses eval(), which I've seen mentioned as something you don't want to do if you can avoid it. So, my question is whether I can avoid using eval() for this specific situation.

I've found the below approach pretty flexible for me...
Put all those methods you want to dynamically call in a class, then call inspect to get the methods within that class - that'll return a list of tuple pairs that are callable, then push them into a dictionary...
import inspect
class testClass:
def __init__(self):
# Unfortunately, need to do this to make inspect work
pass
def foo1(self):
print("Inside foo1")
def foo2(self):
print("Inside foo2")
myMethods = {row[0]:row[1] for row in inspect.getmembers(testClass(), predicate=inspect.ismethod)}
#Note, I am instantiating testClass for the above to work
# myMethods is now a dict of methods, so if you had a string coming out of your earlier work that was "foo1",
str1 = "foo1"
myMethods[str1]()
>>> "Inside foo1"
Of course, you can pass parameters in, I'd suggest wrapping them up as kwargs for flexibility.
It is of course, just a fork off what has been suggested above by timgeb - but I don't like the idea of two points of truth within code. Here, when you update the class "testClass" or whatever you call it with a new method - or renamed method, then that automatically passes through to your calling algorithm, rather than having to manually update a dict.

How to dynamically access a class and its methods from an input? [duplicate]

This question already has answers here:
How to get class object from class name string in the same module?
(3 answers)
Closed last year.
I am creating a python CLI, where user can provide an operation they want to perform, for eg:
sum 10 15
In my code, I have defined my classes as follows:
class Operation:
# common stuff
pass
class Sum(Operation):
identifier = "sum"
def perform(a, b):
return a + b
class Difference(Operation):
identifier = "diff"
def perform(a, b):
return a - b
Now, in my CLI, if I type sum 10 15 I want to return the result of Sum.perform(10, 15) and similarly if I type diff 10 15, I return the result of Difference.perform(10, 15), as sum is the identifier of class Sum and diff is the identifier of class Difference.
How do I dynamically access the class and its perform method, when I get the input directly from user input?

Classes in Python are first-class citizens, meaning they can be used as standard objects. In particular we can simply store them in a dictionary:
my_dict = {
'sum': Sum,
'diff': Difference,
}
and so on. Then when you get the operation name as string from command line you simply do
my_dict[op_name].perform(a, b)
Note that this is a very basic (and you will soon see problematic, e.g. not all operators accept two arguments) approach to what is known as parsing and abstract syntax trees. This is a huge topic, a bit hard but also very interesting. I encourage you to read about it.
// EDIT: If you want to keep identifier on the class, then you can apply a simple class decorator:
my_dict = {}
def autoregister(cls):
# It would be good idea to check whether we
# overwrite an entry here, to avoid errors.
my_dict[cls.identifier] = cls
return cls
#autoregister
class Sum(Operation):
identifier = "sum"
def perform(a, b):
return a + b
print(my_dict)
You have to remember though to import all classes before you use my_dict. In my opinion an explicit dict is easier to maintain.

Reading your comment, I think you need to interpret the input. The way I would go about this is splitting the input by spaces (based on your example), and then checking that list. For example:
# This is the place you called the input:
input_unsplit = input("Enter your command and args")
input_list = input_unsplit.split(" ")
# Check the first word to see what function we're calling
if("sum") in input_list[0].lower():
result = Sum.perform(input_list[1], input_list[2])
print(result)
# this logic can be applied to other functions as well.
This is a simple solution that could be hard to scale.
=== EDITED ===
I have more to add.
If used correctly, dir() can make a list of defined classes up to a certain point in the code. I wrote a calculator for my precal class, and in it I chose to use dir after defining all the math classes, and then if the name met certain conditions (i.e not main), it would be appended to a list of valid args to pass. You can modify your classes to include some kind of operator name property:
def class Addition:
self.op_name = "sum"
and then perform to take in an array:
def perform(numbers):
return numbers[0] + numbers [1]
To solve many of your scalability issues. Then, after declaring your classes, use dir() in a for loop to append to that valid array, like so:
valid_names = []
defined_names = dir()
for name in defined_names:
if '_' not in name:
if name not in ("sys","argparse","<any other imported module/defined var>"):
valid_names.append(name)
Note that making this step work for you is all in the placement in the script. it's a bit tedious, but works flawlessly if handled correctly (in my experience).
Then, you can use eval (safe in this context) to call the method you want:
# get input here
for name in defined_names:
if eval(name).op_name == input_list[0].lower():
eval(name).perform(input_list)
This should be a fairly easy-to-scale solution. Just watch that you keep the dir check up to date, and everything else just... works.

Python Code: Information on Execution Trace of loops/conditionals

I want to get the execution trace of a python function in terms of the loops and conditionals executed upon completion. However, I want to do this without instrumenting the original python function with additional parameters. For example:
def foo(a: int, b: int):
while a:
a = do_something()
if b:
a = do_something()
if __name__ == "__main__":
foo(a, b)
After the execution of foo() I want a execution trace something like:
[while: true, if:false, while: true, if: true, while: false, ...] which documents the sequence of conditional evaluations in the code. Is there any way to get this information automatically for an arbitrary python function?
I understand "Coverage" python module returns the "Branch coverage" information. But I am unsure how to use it in this context?

You can use as a starting point trace_conditions.py and modify it if needed.
Example
foo function that defined in the question is used in the example below:
from trace_conditions import trace_conditions
# (1) This will just print conditions
traced_foo = trace_conditions(foo)
traced_foo(a, b)
# while c -> True
# if d -> True
# ...
# (2) This will return conditions
traced_foo = trace_conditions(foo, return_conditions=True)
result, conditions = traced_foo(a, b)
# conditions = [('while', 'c', True), ('if', 'd', True), ...)]
Note: ast.unparse is used to get string representation of condition. It was introduced in Python 3.9. If you want to use older version of Python, perhaps you will want to install 3rd party package astunparse and then use it in function _condition_to_string. Otherwise trace_conditions will not return string representation of conditions.
TL;DR
Idea
Basically, we want to programmatically add catchers to the function's code. For example, print catchers could look like this:
while x > 5:
print('while x > 5', x > 5) # <-- print condition after while
# do smth
print('if x > 5', x > 5) # <-- print condition before if
if x > 5:
# do smth
So, the main idea is to use code introspection tools in python (inspect, ast, exec).
Implementation
Here I will briefly explain the code in trace_conditions.py:
Main function trace_conditions
The main function is self-explanatory and simply reflects the whole algorithm: (1) build syntactic tree; (2) inject condition catchers; (3) compile new function.
def trace_conditions(
func: Callable, return_conditions=False):
catcher_type = 'yield' if return_conditions else 'print'
tree = _build_syntactic_tree(func)
_inject_catchers(tree, catcher_type)
func = _compile_function(tree, globals_=inspect.stack()[1][0].f_globals)
if return_conditions:
func = _gather_conditions(func)
return func
The only thing that requires explanation is globals_=inspect.stack()[1][0].f_globals. In order to compile a new function we need to give python all modules that are used by that function (for example, it may use math, numpy, django, etc...). And inspect.stack()[1][0].f_globals simply takes everything what imported in the module of the calling function.
Caveat!
# math_pi.py
import math
def get_pi():
return math.pi
# test.py
from math_pi import get_pi
from trace_conditions import trace_conditions
traced = trace_conditions(get_pi)
traced() # Error! Math is not imported in this module
To solve it you can either modify code in trace_conditions.py or just add import math in test.py
_build_syntactic_tree
Here we are first getting the source code of function using inspect.getsource and then parse it in syntactic tree using ast.parse. Unfortunately, python cannot inspect source code of function if it is called from decorator, so it seems with this approach it is not possible to use convenient decorators.
_inject_catchers
In this function we traverse given syntactic tree, find while and if statements and then inject catchers before or after them. ast module has method walk, but it returns only node itself (without parent), so I implemented slightly changed version of walk that returns parent node as well. We need to know parent if we want to insert catcher before if.
def _inject_catchers(tree, catcher_type):
for parent, node in _walk_with_parent(tree):
if isinstance(node, ast.While):
_catch_after_while(node, _create_catcher(node, catcher_type))
elif isinstance(node, ast.If):
_catch_before_if(parent, node, _create_catcher(node, catcher_type))
ast.fix_missing_locations(tree)
At the end we call ast.fix_missing_locations function that helps to fill in correctly technical fields like lineno and others that required in order to compile code. Usually, you need to use it, when you modify syntactic tree.
Catching elif statement
The funny stuff is that python doesn't have in its ast grammar elif statement, so it has just if-else statements. The ast.If node has field body that contains expressions of if body and field orelse that contains expressions of else block. And elif case is simply represented by ast.If node inside orelse field. This fact reflected in the function _catch_before_if.
Catchers (and _gather_conditions)
There are several ways how you could catch conditions, the most simple is to just print it, but this approach will not work if you want handle them later in python code. One straightforward way is to have a global empty list in which you will append condition and its value during execution of function. However, I think that this solution introduces a new name in the namespace that potentially can clutter with local names inside a function, so I decided that it should be more safe to yield conditions and its information.
The function _gather_conditions is adding a wrapper around function with injected yield statements, that simply gathers all yielded conditions and returns result of function and conditions.

Alternative to exec

I'm currently trying to code a Python (3.4.4) GUI with tkinter which should allow to fit an arbitrary function to some datapoints. To start easy, I'd like to create some input-function and evaluate it. Later, I would like to plot and fit it using curve_fit from scipy.
In order to do so, I would like to create a dynamic (fitting) function from a user-input-string. I found and read about exec, but people say that (1) it is not safe to use and (2) there is always a better alternative (e.g. here and in many other places). So, I was wondering what would be the alternative in this case?
Here is some example code with two nested functions which works but it's not dynamic:
def buttonfit_press():
def f(x):
return x+1
return f
print(buttonfit_press()(4))
And here is some code that gives rise to NameError: name 'f' is not defined before I can even start to use xval:
def buttonfit_press2(xval):
actfitfunc = "f(x)=x+1"
execstr = "def {}:\n return {}\n".format(actfitfunc.split("=")[0], actfitfunc.split("=")[1])
exec(execstr)
return f
print(buttonfit_press2(4))
An alternative approach with types.FunctionType discussed here (10303248) wasn't successful either...
So, my question is: Is there a good alternative I could use for this scenario? Or if not, how can I make the code with exec run?
I hope it's understandable and not too vague. Thanks in advance for your ideas and input.
#Gábor Erdős:
Either I don't understand or I disagree. If I code the same segment in the mainloop, it recognizes f and I can execute the code segment from execstr:
actfitfunc = "f(x)=x+1"
execstr = "def {}:\n return {}\n".format(actfitfunc.split("=")[0], actfitfunc.split("=")[1])
exec(execstr)
print(f(4))
>>> 5
#Łukasz Rogalski:
Printing execstr seems fine to me:
def f(x):
return x+1
Indentation error is unlikely due to my editor, but I double-checked - it's fine.
Introducing my_locals, calling it in exec and printing in afterwards shows:
{'f': <function f at 0x000000000348D8C8>}
However, I still get NameError: name 'f' is not defined.
#user3691475:
Your example is very similar to my first example. But this is not "dynamic" in my understanding, i.e. one can not change the output of the function while the code is running.
#Dunes:
I think this is going in the right direction, thanks. However, I don't understand yet how I can evaluate and use this function in the next step? What I mean is: in order to be able to fit it, I have to extract fitting variables (i.e. a in f(x)=a*x+b) or evaluate the function at various x-values (i.e. print(f(3.14))).

The problem with exec/eval, is that they can execute arbitrary code. So to use exec or eval you need to either carefully parse the code fragment to ensure it doesn't contain malicious code (an incredibly hard task), or be sure that the source of the code can be trusted. If you're making a small program for personal use then that's fine. A big program that's responsible for sensitive data or money, definitely not. It would seem your use case counts as having a trusted source.
If all you want is to create an arbitrary function at runtime, then just use a combination of the lambda expression and eval. eg.
func_str = "lambda x: x + 1" # equates to f(x)=x+1
func = eval(func_str)
assert func(4) == 5
The reason why your attempt isn't working is that locals(), in the context of a function, creates a copy of the local namespace. Mutations to the resulting dictionary do not effect the current local namespace. You would need to do something like:
def g():
src = """
def f(x):
return x + 1
"""
exec_namespace = {} # exec will place the function f in this dictionary
exec(src, exec_namespace)
return exec_namespace['f'] # retrieve f

I'm not sure what exactly are you trying to do, i.e. what functions are allowed, what operations are permitted, etc.
Here is an example of a function generator with one dynamic parameter:
>>> def generator(n):
def f(x):
return x+n
return f
>>> plus_one=generator(1)
>>> print(plus_one(4))
5