I am trying to create functions dynamically on the fly or to create functions programmatically in python and am getting stuck at this.
def test(a,b):
print(a)
#Result:
def test(a,b,c):
print(a)
print(b,c)
I am trying it with AST module and failing to understand the AST syntax for function. I understand that a function details can be fetched using functionname.__code__.xxx. But I will be unable to make changes since it is readonly. Here is what I am trying:
I tried getting the AST dump of the above function and it didnt work.
Trying to get the string version of function definition (couldnt find reference)
I have also tried the below but feel that it may be an issue somewhere like when there is a re-assignation which makes it local:
def tests(a,b):
print(a,b)
def mytest(c):
print(a,b,c)
return mytest
def tests(a,b):
print(a,b)
def mytest(c):
print(a,b,c)
a = 10 # this will become local variable
return mytest
I am on python 3.x
Any help or any other easier way?
Related
I'm looking for a safe mechanism in Python to execute potentially unsafe script code coming from a user.
Code example:
def public(v):
print('Allowed to trigger this with '+v)
def secret():
print('Not allowed to trigger this')
unsafe_user_code = '''
def user_function(a):
if 'o' in a:
public('parameter')
a = 'hello'
user_function(a)
'''
run_code(unsafe_user_code, allowed=['public'])
This could be easily achieved with exec() but as I understand there is no way of using exec() in a safe way in Python.
Here are my requirements:
The syntax of the script should ideally be similar to Python (but could also be something like JavaScript)
Standard mechanisms like string operations, if/else/while and definition of own variables and functions need to be available in the script
The script should only be able to execute certain functions (in the example, only public() would be allowed)
I would like to rely on Python-based implementations/libraries only as I don't want to have dependencies on Python-external software
It should not introduce a security risk
The only way I found so far is to use a parsing library, where I would have to define everything by myself (e.g. this one: https://github.com/lark-parser/lark ).
Is there a better way to achieve something like this?
Thanks!
Answer
Here's the full python file if you want to copy:
from copy import copy
def public(v):
print('Allowed to trigger this with '+v)
def secret():
print('Not allowed to trigger this')
unsafe_user_code = '''
def user_function(a):
if 'o' in a:
public('parameter')
# secret()
# If you uncomment it, it'll throw an error saying 'secret' does not exist
a = 'hello'
user_function(a)
'''
def run_code(code_to_run,allowed:list[str]):
g = globals()
allowed_dict = {f"{name}": g[name] for name in allowed}
code = compile(code_to_run,'','exec')
def useless():
pass
useless.__code__ = code
g = copy(useless.__globals__)
for x in g:
del useless.__globals__[x]
for x in allowed_dict:
useless.__globals__[x] = allowed_dict[x]
useless()
run_code(unsafe_user_code, allowed=['public'])
Explaination
I don't think you need a parsing library.
You can use one of python's Built-in Functions called compile().
You can compile code like this:
text_to_compile = "print('hello world')"
code = compile(
text_to_compile, # text to compile
'file_name', # file name
'exec' # compile mode
)
more about compile modes
and then you can run it simply by:
exec(code) # prints 'hello world' in the terminal
or, you can put that code inside of a function and run the function:
def useless():
pass
useless.__code__ = code
useless() # prints 'hello world' in the terminal
now for changing the scope of the function, we can access its __global__ dictionary and remove everything from it.
then we add the functions/variables that we want to it.
# from copy import copy
def run_code(code_to_run,allowed:list[str]):
g = globals()
allowed_dict = {f"{name}": g[name] for name in allowed}
code = compile(code_to_run,'','exec')
def useless():
pass
useless.__code__ = code
g = copy(useless.__globals__)
for x in g:
del useless.__globals__[x]
for x in allowed_dict:
useless.__globals__[x] = allowed_dict[x]
useless()
alright let's talk about what's happening:
def run_code(code_to_run,allowed:list[str]):
the :list[str] is just defining the type of the variable allowed.
g = globals()
using the globals() method we are able to get all of the variables that the function run_code has access to. this includes the public and the secret functions.
allowed_dict = {f"{name}": g[name] for name in allowed}
and use the variable names that you passed down into the run_code function and get the variables from the globals of this function.
the it's pretty much the same as saying:
allowed_dict = {}
for name in allowed:
allowed_dict[name] = g[name]
alright then we compile the code.
we also make a useless function and put the compiled code into the function:
code = compile(code_to_run,'','exec')
def useless():
pass
useless.__code__ = code
after that, we copy the globals of the useless function:
g = copy(useless.__globals__)
we copy the globals in the function so that later in the code we can loop over it and delete it.
we do this because we cannot delete something from a dictionary while a for loop is iterating it.
for x in g:
del useless.__globals__[x]
we delete everything in the useless function's global.
the useless function is an empty function but it's global is filled with all sort of stuff, like the public and the secret functions.
and then we just put all the allowed variables into the globals of the function so that the code inside of the function can use the variables/functions.
for x in allowed_dict:
useless.__globals__[x] = allowed_dict[x]
then we can just run the useless function:
useless()
and that's pretty much it all.
I am using Python 3.9 and trying to write the function
def greet(request, name):
return HttpResponse(f'Hello, {name.capitalize}!')
Using f to format the string but it is not working. Any ideas on why?
capitalize is a method of the str object.
Therefore you need to add parenthesis for it to be called:
def greet(request, name):
return HttpResponse(f'Hello, {name.capitalize()}!')
Furthermore, name.capitalize is really just the reference to the function.
Try running the following inside a python interpreter:
print(str.capitalize)
You could even return this function:
def cap_str(string):
return string.capitalize
s = "programming in python"
capitalize_s = cap_str(s)
s_cap = capitalize_s()
print(s_cap)
I don't know how this would be particularly useful, but returning a function in general is pretty useful.
Can you print the error message? I suspect your error is that you want name.capitalize() rather than name.capitalize
Ah - this has already been added!
I found the following code snippet that I can't seem to make work for my scenario (or any scenario at all):
def load(code):
# Delete all local variables
globals()['code'] = code
del locals()['code']
# Run the code
exec(globals()['code'])
# Delete any global variables we've added
del globals()['load']
del globals()['code']
# Copy k so we can use it
if 'k' in locals():
globals()['k'] = locals()['k']
del locals()['k']
# Copy the rest of the variables
for k in locals().keys():
globals()[k] = locals()[k]
I created a file called "dynamic_module" and put this code in it, which I then used to try to execute the following code which is a placeholder for some dynamically created string I would like to execute.
import random
import datetime
class MyClass(object):
def main(self, a, b):
r = random.Random(datetime.datetime.now().microsecond)
a = r.randint(a, b)
return a
Then I tried executing the following:
import dynamic_module
dynamic_module.load(code_string)
return_value = dynamic_module.MyClass().main(1,100)
When this runs it should return a random number between 1 and 100. However, I can't seem to get the initial snippet I found to work for even the simplest of code strings. I think part of my confusion in doing this is that I may misunderstand how globals and locals work and therefore how to properly fix the problems I'm encountering. I need the code string to use its own imports and variables and not have access to the ones where it is being run from, which is the reason I am going through this somewhat over-complicated method.
You should not be using the code you found. It is has several big problems, not least that most of it doesn't actually do anything (locals() is a proxy, deleting from it has no effect on the actual locals, it puts any code you execute in the same shared globals, etc.)
Use the accepted answer in that post instead; recast as a function that becomes:
import sys, imp
def load_module_from_string(code, name='dynamic_module')
module = imp.new_module(name)
exec(code, mymodule.__dict__)
return module
then just use that:
dynamic_module = load_module_from_string(code_string)
return_value = dynamic_module.MyClass().main(1, 100)
The function produces a new, clean module object.
In general, this is not how you should dynamically import and use external modules. You should be using __import__ within your function to do this. Here's a simple example that worked for me:
plt = __import__('matplotlib.pyplot', fromlist = ['plt'])
plt.plot(np.arange(5), np.arange(5))
plt.show()
I imagine that for your specific application (loading from code string) it would be much easier to save the dynamically generated code string to a file (in a folder containing an __init__.py file) and then to call it using __import__. Then you could access all variables and functions of the code as parts of the imported module.
Unless I'm missing something?
I would like to invoke the following python script in C :
#!/usr/bin/python2.7
import parser
def evaluate(text):
code = parser.expr(text).compile()
return eval(code)
as explained in the following page https://docs.python.org/2/extending/embedding.html, i can call this script from C using the pwd(path) of the file.
However, i would like to know if it's possible to not load the script by calling python on a C string directly, defining the script.
For example, i would like to let's say i put :
#define PYTHON_SCRIPT ((char*)(\
import parser\
\
def evaluate(text):\
code = parser.expr(text).compile()\
return eval(code)\
))
is it possible to call the python interpreter directly on that string?
Indeed, knowing that i need to pass text as a variable, i can't use this Pyrun_SimpleString function, and i was not able to find something to answer this question.
As mentioned in the comment there is no Pyrun_SimpleString. How to execute Python functions from C is covered here. One way to do it:
Compile your script using Py_CompileString
Create a dictionary for globals/locals.
Extract the function from your globals dict by using PyDict_GetItemString(name)
Build your arguments tuple with PyArg_ParseTuple
Execute your function object by using PyObject_CallFunction.
Take a look at Weave, it allows you to include C code directly in Python code.
I was trying to speed up some code, and then I tried compiling a class and a function using cython
and WOW! I havn't measured it yet but it looks at least 10x faster.
I first looked at cython just two days ago, I'm very impressed!
However, I can't get eval() to work.
def thefirst(int a):
d = eval('1+2+a')
return d
I compile this to module1.pyd file and call it with the python file:
from module1 import thefirst
x = thefirst(2)
print x
This returns:
NameError: name 'a' is not defined.
All help is appreciated.
This is because eval has no way of examining the environment to find a. Use the locals function to pass it the environment.
def thefirst(a):
return eval('1+2+a', locals())
You may get away with cython.inline:
http://wiki.cython.org/enhancements/inline
However, keep an eye on your Python runtime's memory usage in this case. Each distinct expression that gets compiled and loaded takes up some memory. This may add up if you do this a lot.