I have a python application ready to launch.
One thing keeps tickling my mind is that the application depends on formal API of several sites to get data.
If one site got their API changed, i have to change the code and ask users to reinstall the new application. It's a lot of work if several things hang on at a time.
I came across exec, which could execute a string like a code snippet.
So, if it works good, I can save the critical code parts sqlite3 table.
In case of any change, I can ask users to do an OTA update from inside the application which will just update the sqlite3 table and code would work as usual.
But just got a hammer, return not working inside exec(), just getting return outside function exception. I don't know what are all others things that will go not working if I use exec.
Working:
def func_dyn():
if 1==1:
return 1
else:
print('test')
if __name__ == '__main__':
func_dyn();
Not Working:
global code
code="""if 1==1:
return 1
else:
print('test')
"""
def func_dyn():
global code
exec(code)
if __name__ == '__main__':
func_dyn();
How to handle the return in exec if exec was inside a function / the way it should be formatted/handled?
Why i need to put the whole code of a function into exec?
since there were many functions like this, I can't store small,small parts which could make code unreadable. So i was thinking to put whole function into string.
Why i need return?
If exceptions arise, the function should return to the caller and execute next.
Even though exec works in the current context but it doesn't seem to work in the context of the running method, so return was not possible. A workaround is set a flag inside exec and handle return outside. But exec was not a good candidate for my requirement. Instead, I have decided to properly incorporate update functionality in my code via my own standalone updater or via frameworks like pyupdater, esky, etc.
I think your approach is very difficult to debug. What is the purpose of storing code in SQL anyway? why can't you just prepare an external API file and update that if needed? This can be simply imported by adding a cache folder to your Python path programmatically and will keep your code where it belongs: a .py file.
Related
One of my favorite debug tools is a short piece of code which starts up a console. I sometimes debug by means of (roughly)
try:
#my code
...
except:
launch_console()
However, this requires anticipating where the error will be. If I enclose the entire script in a try/except, I will not have access to any variables within the function I am debugging.
It would be magnificently convenient, if I could run the program and just automatically have launch_console() run in case of an error, without losing scope. I think spyder is supposed to have this functionality, but it does not work.
Wondering if there is some nice way to do this!
Ok, so I know this has been asked before in many different threads but I find myself still trying to reduce my doubt.
I have an application that allows the user to pass a dictionary of strings and 'choose' a backend function from a library to process it. The functions are part of a 'workflow' library and loaded by the system admin on the backend. Available functions are stored in the backend in a manifest file.
The mechanics are such that the users send the dictionary as json to the web app and nominate which function from the library should process it. The function is is then loaded and executed via the python exec() or eval() functions.
Before the execution, the requested function is checked against a list of available functions (whitelist) from the manifest file.
My basic question is, can whitelisting make exec() and eval safe? Could it be made 'safer'?
If I understand it, the function is trusted by the admin and that makes it as safe as any python module you install. Just make sure that the exec part is only done on the trusted code. Here is an example where functions with the same name as their file are loaded and executed.
import json
from pathlib import Path
# files named the same as the function, no .py
FUNCTION_DIR = Path("/my/functions/are/here")
def run_func(name, data):
try:
func_namespace = {}
exec(open(FUNCTION_DIR/name).read(), func_namespace)
return func_namespace[name](json.dumps(data))
except Exception as e:
return "Hey, what kind of game are you playing here? " + str(e)
The the function is naturally whitelisted just because its in the known-safe directory.
So I have a script called controller.py that has commands that interface with a front end on a website. I am looking for a way that when a command is run, for example:
if command == 'F':
drivingSpeed = drivingSpeedActuallyUsed
for motorIndex in range(2):
runMotor(motorIndex, forward[motorIndex])
time.sleep(straightDelay)
in controller.py, it changes a line in another script called send_video.py for a few seconds then change back to the original
overlayCommand = '-vf transpose=2,transpose=2,dynoverlay=overlayfile=/home/pi/runmyrobot/images/hud.png:check_interval=500'
I need the line in send_video.py to change the directory of the image. Another issue I am having is that the send_video.py only updates on reboot, I need to tell it to update automatically.
Changing a line in code automatically is very unusual and far from what we call "best practice". This is called self-modifying code and was the topic of research done several decades ago. Since then it has been dropped almost completely and is now regarded an ugly hack in all but the very strangest circumstances.
What is typically done to achieve your goal is to pass parameters and use them in the other code.
Your wording is a bit unclear (I don't understand that part about updating a file, for example), so I'm not sure if I address your issue correctly, but I'll give it a try in a general approach:
When your code in controller.py calls the code in send_video.py it will call a function. To this function is should pass an argument which is the line. The function in send_video.py should use the passed parameter to determine which line it should use (the default one or the special one).
Feel free to improve your question by showing us more of your code or ask further questions if this does not help enough.
I'm developing a system that operates on (arbitrary) data from databases. The data may need some preprocessing before the system can work with it. To allow the user the specify possibly complex rules I though of giving the user the possibility to input Python code which is used to do this task. The system is pure Python.
My plan is to introduce the tables and columns as variables and let the user to anything Python can do (including access to the standard libs). Now to my problem:
How do I take a string (the user entered), compile it to Python (after adding code to provide the input data) and get the output. I think the easiest way would be to use the user-entered data a the body of a method and take the return value of that function a my new data.
Is this possible? If yes, how? It's unimportant that the user may enter malicious code since the worst thing that could happen is, that he screws up his own system, which is thankfully not my problem ;)
Python provides an exec() statement which should do what you want. You will want to pass in the variables that you want available as the second and/or third arguments to the function (globals and locals respectively) as those control the environment that the exec is run in.
For example:
env = {'somevar': 'somevalue'}
exec(code, env)
Alternatively, execfile() can be used in a similar way, if the code that you want executed is stored in its own file.
If you only have a single expression that you want to execute, you can also use eval.
Is this possible?
If it doesn't involve time travel, anti-gravity or perpetual motion the answer to this question is always "YES". You don't need to ask that.
The right way to proceed is as follows.
You build a framework with some handy libraries and packages.
You build a few sample applications that implement this requirement: "The data may need some preprocessing before the system can work with it."
You write documentation about how that application imports and uses modules from your framework.
You turn the framework, the sample applications and the documentation over to users to let them build these applications.
Don't waste time on "take a string (the user entered), compile it to Python (after adding code to provide the input data) and get the output".
The user should write applications like this.
from your_framework import the_file_loop
def their_function( one_line_as_dict ):
one_line_as_dict['field']= some stuff
the_file_loop( their_function )
That can actually be the entire program.
You'll have to write the_file_loop, which will look something like this.
def the_file_loop( some_function ):
with open('input') as source:
with open('output') as target:
for some_line in source:
the_data = make_a_dictionary( some_line )
some_function( the_data )
target.write( make_a_line( the_data ) )
By creating a framework, and allowing users to write their own programs, you'll be a lot happier with the results. Less magic.
2 choices:
You take his input and put it in a file, then you execute it.
You use exec()
If you just want to set some local values and then provide a python shell, check out the code module.
You can start an instance of a shell that is similar to the python shell, as well as initialize it with whatever local variables you want. This would assume that whatever functionality you want to use the resulting values is built into the classes you are passing in as locals.
Example:
shell = code.InteractiveConsole({'foo': myVar1, 'bar': myVar2})
What you actually want is exec, since eval is limited to taking an expression and returning a value. With exec, you can have code blocks (statements) and work on arbitrarily complex data, passed in as the globals and locals of the code.
The result is then returned by the code via some convention (like binding it to result).
well, you're describing compile()
But... I think I'd still implement this using regular python source files. Add a special location to the path, say '~/.myapp/plugins', and just __import__ everything there. Probably you'll want to provide some convenient base classes that expose the interface you're trying to offer, so that your users can inherit from them.
I have a lot of simple scripts that calculate some stuff or so. They consist of just a single module.
Should I write main methods for them and call them with the if __name__ construct, or just dump it all right in there?
What are the advantages of either method?
I always write a main() function (appropriately named), and put nothing but command-line parsing and a call to main() in the if __name__ == '__main__' block. That's because no matter how silly, trivial, or single-purpose I originally expect that script to be, I always end up wanting to call it from another module at some later date.
Either I take the time to make it an importable module today, or spend extra time to refactor it months later when I want to reuse it for something else.
Always.
Every time.
I've stopped fighting it and started writing my code with that expectation from the start.
Well, if you do this:
# your code
Then import your_module will execute your code. On the contrary, with this:
if __name__ == '__main__':
# your code
The import won't run the code, but targeting the interpreter at that file will.
If the only way the script is ever going to run is by manual interpreter opening, there's absolutely no difference.
This becomes important when you have a library (or reusing the definitions in the script).
Adding code to a library outside a definition, or outside the protection of if __name__ runs the code when importing, letting you initialize stuff that the library needs.
Maybe you want your library to also have some runnable functionality. Maybe testing, or maybe something like Python's SimpleHTTPServer (it comes with some classes, but you can also run the module and it will start a server). You can have that dual behaviour with the if __name__ clause.
Tools like epydoc import the module to access the docstrings, so running the code when you just want to generate HTML documentation is not really the intent.
The other answers are good, but I wanted to add an example of why you might want to be able to import it: unit testing. If you have a few functions and then the if __name__=="__main__":, you can import the module for unit testing. Maybe you're better at this than me, but I find that my "simple scripts that couldn't possibly have any bugs" tend to have bugs that I find with unit testing.
The if __name__ construct will let you easily re-use the functions and classes in the module in other Python scripts. If you don't do that, then everything in the module will run when it is imported.
If there's nothing in the script you want to reuse, then sure, just dump it all in there. I do that sometimes. If you later decide you want to reuse some code, I have found Python to be just about the easiest language in which to refactor code without breaking it.
For a good explanation of the purpose of Python's "main guard" idiom:
What does if __name__ == "__main__": do?