We Keep Coding

How can i accept and run user's code securely on my web app?

I am working on a django based web app that takes python file as input which contains some function, then in backend i have some lists that are passed as parameters through the user's function,which will generate a single value output.The result generated will be used for some further computation.
Here is how the function inside the user's file look like :
def somefunctionname(list):
''' some computation performed on list'''
return float value
At present the approach that i am using is taking user's file as normal file input. Then in my views.py i am executing the file as module and passing the parameters with eval function. Snippet is given below.
Here modulename is the python file name that i had taken from user and importing as module
exec("import "+modulename)
result = eval(f"{modulename}.{somefunctionname}(arguments)")
Which is working absolutely fine. But i know this is not the secured approach.
My question , Is there any other way through which i can run users file securely as the method that i am using is not secure ? I know the proposed solutions can't be full proof but what are the other ways in which i can run this (like if it can be solved with dockerization then what will be the approach or some external tools that i can use with API )?
Or if possible can somebody tell me how can i simply sandbox this or any tutorial that can help me..?
Any reference or resource will be helpful.

It is an important question. In python sandboxing is not trivial.
It is one of the few cases where the question which version of python interpreter you are using. For example, Jyton generates Java bytecode, and JVM has its own mechanism to run code securely.
For CPython, the default interpreter, originally there were some attempts to make a restricted execution mode, that were abandoned long time ago.
Currently, there is that unofficial project, RestrictedPython that might give you what you need. It is not a full sandbox, i.e. will not give you restricted filesystem access or something, but for you needs it may be just enough.
Basically the guys there just rewrote the python compilation in a more restricted way.
What it allows to do is to compile a piece of code and then execute, all in a restricted mode. For example:
from RestrictedPython import safe_builtins, compile_restricted
source_code = """
print('Hello world, but secure')
"""
byte_code = compile_restricted(
source_code,
filename='<string>',
mode='exec'
)
exec(byte_code, {__builtins__ = safe_builtins})
>>> Hello world, but secure
Running with builtins = safe_builtins disables the dangerous functions like open file, import or whatever. There are also other variations of builtins and other options, take some time to read the docs, they are pretty good.
EDIT:
Here is an example for you use case
from RestrictedPython import safe_builtins, compile_restricted
from RestrictedPython.Eval import default_guarded_getitem
def execute_user_code(user_code, user_func, *args, **kwargs):
""" Executed user code in restricted env
Args:
user_code(str) - String containing the unsafe code
user_func(str) - Function inside user_code to execute and return value
*args, **kwargs - arguments passed to the user function
Return:
Return value of the user_func
"""
def _apply(f, *a, **kw):
return f(*a, **kw)
try:
# This is the variables we allow user code to see. #result will contain return value.
restricted_locals = {
"result": None,
"args": args,
"kwargs": kwargs,
}
# If you want the user to be able to use some of your functions inside his code,
# you should add this function to this dictionary.
# By default many standard actions are disabled. Here I add _apply_ to be able to access
# args and kwargs and _getitem_ to be able to use arrays. Just think before you add
# something else. I am not saying you shouldn't do it. You should understand what you
# are doing thats all.
restricted_globals = {
"__builtins__": safe_builtins,
"_getitem_": default_guarded_getitem,
"_apply_": _apply,
}
# Add another line to user code that executes #user_func
user_code += "\nresult = {0}(*args, **kwargs)".format(user_func)
# Compile the user code
byte_code = compile_restricted(user_code, filename="<user_code>", mode="exec")
# Run it
exec(byte_code, restricted_globals, restricted_locals)
# User code has modified result inside restricted_locals. Return it.
return restricted_locals["result"]
except SyntaxError as e:
# Do whaever you want if the user has code that does not compile
raise
except Exception as e:
# The code did something that is not allowed. Add some nasty punishment to the user here.
raise
Now you have a function execute_user_code, that receives some unsafe code as a string, a name of a function from this code, arguments, and returns the return value of the function with the given arguments.
Here is a very stupid example of some user code:
example = """
def test(x, name="Johny"):
return name + " likes " + str(x*x)
"""
# Lets see how this works
print(execute_user_code(example, "test", 5))
# Result: Johny likes 25
But here is what happens when the user code tries to do something unsafe:
malicious_example = """
import sys
print("Now I have the access to your system, muhahahaha")
"""
# Lets see how this works
print(execute_user_code(malicious_example, "test", 5))
# Result - evil plan failed:
# Traceback (most recent call last):
# File "restr.py", line 69, in <module>
# print(execute_user_code(malitious_example, "test", 5))
# File "restr.py", line 45, in execute_user_code
# exec(byte_code, restricted_globals, restricted_locals)
# File "<user_code>", line 2, in <module>
#ImportError: __import__ not found
Possible extension:
Pay attention that the user code is compiled on each call to the function. However, it is possible that you would like to compile the user code once, then execute it with different parameters. So all you have to do is to save the byte_code somewhere, then to call exec with a different set of restricted_locals each time.
EDIT2:
If you want to use import, you can write your own import function that allows to use only modules that you consider safe. Example:
def _import(name, globals=None, locals=None, fromlist=(), level=0):
safe_modules = ["math"]
if name in safe_modules:
globals[name] = __import__(name, globals, locals, fromlist, level)
else:
raise Exception("Don't you even think about it {0}".format(name))
safe_builtins['__import__'] = _import # Must be a part of builtins
restricted_globals = {
"__builtins__": safe_builtins,
"_getitem_": default_guarded_getitem,
"_apply_": _apply,
}
....
i_example = """
import math
def myceil(x):
return math.ceil(x)
"""
print(execute_user_code(i_example, "myceil", 1.5))
Note that this sample import function is VERY primitive, it will not work with stuff like from x import y. You can look here for a more complex implementation.
EDIT3
Note, that lots of python built in functionality is not available out of the box in RestrictedPython, it does not mean it is not available at all. You may need to implement some function for it to become available.
Even some obvious things like sum or += operator are not obvious in the restricted environment.
For example, the for loop uses _getiter_ function that you must implement and provide yourself (in globals). Since you want to avoid infinite loops, you may want to put some limits on the number of iterations allowed. Here is a sample implementation that limits number of iterations to 100:
MAX_ITER_LEN = 100
class MaxCountIter:
def __init__(self, dataset, max_count):
self.i = iter(dataset)
self.left = max_count
def __iter__(self):
return self
def __next__(self):
if self.left > 0:
self.left -= 1
return next(self.i)
else:
raise StopIteration()
def _getiter(ob):
return MaxCountIter(ob, MAX_ITER_LEN)
....
restricted_globals = {
"_getiter_": _getiter,
....
for_ex = """
def sum(x):
y = 0
for i in range(x):
y = y + i
return y
"""
print(execute_user_code(for_ex, "sum", 6))
If you don't want to limit loop count, just use identity function as _getiter_:
restricted_globals = {
"_getiter_": labmda x: x,
Note that simply limiting the loop count does not guarantee security. First, loops can be nested. Second, you cannot limit the execution count of a while loop. To make it secure, you have to execute unsafe code under some timeout.
Please take a moment to read the docs.
Note that not everything is documented (although many things are). You have to learn to read the project's source code for more advanced things. Best way to learn is to try and run some code, and to see what kind function is missing, then to see the source code of the project to understand how to implement it.
EDIT4
There is still another problem - restricted code may have infinite loops. To avoid it, some kind of timeout is required on the code.
Unfortunately, since you are using django, that is multi threaded unless you explicitly specify otherwise, simple trick for timeouts using signeals will not work here, you have to use multiprocessing.
Easiest way in my opinion - use this library. Simply add a decorator to execute_user_code so it will look like this:
#timeout_decorator.timeout(5, use_signals=False)
def execute_user_code(user_code, user_func, *args, **kwargs):
And you are done. The code will never run more than 5 seconds.
Pay attention to use_signals=False, without this it may have some unexpected behavior in django.
Also note that this is relatively heavy on resources (and I don't really see a way to overcome this). I mean not really crazy heavy, but it is an extra process spawn. You should hold that in mind in your web server configuration - the api which allows to execute arbitrary user code is more vulnerable to ddos.

For sure with docker you can sandbox the execution if you are careful. You can restrict CPU cycles, max memory, close all network ports, run as a user with read only access to the file system and all).
Still,this would be extremely complex to get it right I think. For me you shall not allow a client to execute arbitrar code like that.
I would be to check if a production/solution isn't already done and use that. I was thinking that some sites allow you to submit some code (python, java, whatever) that is executed on the server.

Python: binding (otherwise) undefined names to an object

This question derives from a very specific use-case. Some clever person has put together a repl.it that allows you to use the Python version of the Processing libraries on the web:
https://repl.it/#templates/Python-Processing
The Python code looks like this:
from browser import document, window, alert
def sketch(p):
#this p is needed. it will be the processing sketch itself.
# to do things like background(0) instead do p.background(0)
def setup():
p.createCanvas(700, 410)
p.background(0)
p.rectMode(p.CENTER)
def draw():
#p.background(0)
p.fill(255,255,0,128)
p.ellipse(p.mouseX,p.mouseY,50,50)
def mousePressed(self):
p.background(0)
def keyPressed(self):
if p.key==" ":
print("Hallo")
p.setup = setup
p.draw = draw
p.mousePressed = mousePressed
p.keyPressed = keyPressed
myp5 = window.p5.new(sketch)
The issue is that I'd really like the user code to just look like this (as it would in a desktop version of the Python version of Processing):
def setup():
createCanvas(700, 410)
background(0)
rectMode(CENTER)
def draw():
fill(255,255,0,128)
ellipse(mouseX, mouseY, 50, 50)
def mousePressed(self):
background(0)
def keyPressed(self):
if key==" ":
print("Hallo")
...but as you can see in the repl, all of the calls have to be prefaced by "p." so that they are acting on the processing sketch, and wrapped in a function that takes that sketch as an argument.
I guess the question boils down to this: is there a way of wrapping things so that otherwise undefined global variables/functions (e.g. createCanvas) get bound to a particular object (in this case resolving to p.createCanvas)?
I recognize that it's kind of unpythonic, since it's about making implicit what was otherwise explicit, but in this case, it would serve to make the code more in the spirit of the Processing libraries.

Easiest way for sorting a QTreeView

I'm trying to do some basic Qt file manager app in Python 3.6 with PySide2. Code is like this:
class MainWidget(QWidget):
startDir = "."
def __init__(self):
super().__init__()
isDirselectDone = self.selectContentDir()
if isDirselectDone:
self.model = QFileSystemModel()
self.model.setRootPath(self.startDir)
self.tree = QTreeView()
self.tree.setModel(self.model)
self.tree.setSortingEnabled(True)
self.tree.show()
def selectContentDir(self):
print("Selecing game content folder")
self.startDir = QFileDialog.getExistingDirectory()
if(len(self.startDir) == 0):
print("Game content folder loading cancelled")
return False
print("Trying to load ", self.startDir)
return True
My problem is that no matter what is the contents of the chosen directory, the view does not sort the files. I can click oh the headers, and the little arrows are changing, but everything remains in the order it was loaded.
I tried to look for solutions, but the answers either say that you just have to call the setSortingEnabled() and everything magically works (which is not the case), or they describe some voodoo magic rituals involving inheriting from QAbstractTreeSuperAncientGodItemViewDelegate, then re-implement half of the Qt library and finally creating a Tartarus-like maze by connecting slots and signals all over the place.
So what is the easiest / simplest way to make this sorting work according to latest Qt standards?

I found a solution for my case, based on the following assumptions:
These three lines:
model.setRootPath(directory)
tree.setRootIndex(self.model.index(directory))
tree.setSortingEnabled(True)
Seem they have to be in this specific order, and of course the model rootpath and the tree root index has to point to the same directory (during debugging I tried to set them to different ones, just to see what happens when the tree shows just a subset of the model data, but as expected it broke multiple things in the app).
Also, any custom lines (like handling your custom columns) has to happen between the root index setting, and the sorting enable. I don't know if this is really a general rule, it seems kind of arbitrary, but in my project it turned out to be crucial to call setSortingEnabled() as a last step, to make it work.
If your custom columns do not contain simple text (or anything not easily sortable, in my case a combobox) or they are hidden, you'd better exclude that column from the sorting. This is not mandatory, but for me it resulted in faster response time for the other columns.

How to correctly pass (mutable) objects using ROS-SMACH FSM

I've been struggling with this question for a while now. My question is very specific so please don't post a link to the ROS-tutorial page with the "!Note:" paragraph showing how to pass mutable objects (unless you show me something I have missed). I would like to know if anyone has been able to correctly pass mutable objects back and forth in SMACH states, without encountering any errors.
I wrote a particularly simple and useless program to explain what I am attempting to do. I could post the code of this example (unfortunately however, not surprisingly to those that have used SMACH before, it is a long piece of code). So for now I will just try my best to explain it and include a [link] to an image of my example. I created two python scripts. Each script contains a single class and object of that class (with some basic methods). I create a publisher and subscriber in each script, one of the scripts sends messages (talks) while the other script listens to (hears) the messages. At the end the talker flags both FSMs to shutdown. If anyone would like the full code example let me know...
Code snippet below showing smach states and transitions:
# begin sm
with sm:
smach.StateMachine.add('LOAD', loadFSM(),
transitions={'LOADED':'SENDMSG'})
smach.StateMachine.add('SENDMSG', startMSG(),
transitions={'SENT':'SENDMSG',
'ENDING':'END'})
smach.StateMachine.add('END', stopFSM(),
transitions={'DONE':'complete',
'ERRED':'incomplete'})
Code snippet below showing a smach state (loadFSM):
class loadFSM(smach.State):
def __init__(self):
smach.State.__init__(self, outcomes=['LOADED'],
output_keys=['talker_obj'],
input_keys=['talker_obj'])
# Initialise our talker object
self.talker = Talk()
def execute(self, userdata):
rospy.loginfo("talker state: Loading fsm")
self.talker.init_publish()
self.talker.init_subscribe()
userdata.talker_obj = self.talker
return 'LOADED'
The errors I receive (using Ubuntu 14.04, ROS indigo and python 2.7, not too sure but I believe the same errors occur in kinetic as well) only occur during state transitions and of course the introspective server does not work (does not show state transitions). The errors are;
1. "Exception in thread sm_introViewer:status_publisher:"
2. "Could not execute transition callback: Traceback (most recent call
last): File
"/opt/ros/indigo/lib/python2.7/dist-packages/smach/container.py",
line 175, in call_transition_cbs cb(self.userdata,
self.get_active_states(), *args)"
I also need to add, that my simple finite state machine example actually works and completes successfully, even my project's 2 larger FSMs complete. However, when the FSM has many states like in my project, sometimes my simulations fail. I would like to know from anyone who has used SMACH extensively if they think these errors are the cause or if they know for a fact that I am not passing the object correctly between states.
Thanks in Advance,
Tiz

I had similar problems in the past. If I remember correctly, userdata can only handle basic types (int, string, list, dict, ...) but not arbitrary objects.
I solved it by passing the objects to the constructor of state classes, instead of using the userdata, i.e. something like:
class myState(smach.State):
def __init__(self, obj):
smach.State.__init__(self, outcomes=['foo'], ...)
self.obj = obj
...
and then initialize like follows:
obj = SomeClass()
with sm:
smach.StateMachine.add('MY_STATE', myState(obj),
transitions={'foo':'bar'})
It is not so nice as it circumvents the userdata concept but it works.

Python: how to switch between workspaces using Xlib?

How do I switch between my window manager's workspaces using Python with Xlib module?
This is my most promising attempt:
#!/usr/bin/python
from Xlib import X, display, error, Xatom, Xutil
import Xlib.protocol.event
screen = Xlib.display.Display().screen()
root = screen.root
def sendEvent(win, ctype, data, mask=None):
""" Send a ClientMessage event to the root """
data = (data+[0]*(5-len(data)))[:5]
ev = Xlib.protocol.event.ClientMessage(window=win, client_type=ctype, data=(32,(data)))
if not mask:
mask = (X.SubstructureRedirectMask|X.SubstructureNotifyMask)
root.send_event(ev, event_mask=mask)
# switch to desktop 2
sendEvent(root, Xlib.display.Display().intern_atom("_NET_CURRENT_DESKTOP"), [2])
The above code is shamelessly stolen from various places in the PyPanel source; unfortunately, it doesn't do anything, not even generate a warning / exception. Am I missing something here?
I'm using Python and PyGTK. Xlib seems to be the right choice for switching desktops. I don't intend to use wnck (buggy Python module) or similar, but I'd appreciate any pointers anyway.
I might add that this is my first attempt at writing a Python application using Xlib (or PyGTK).

Apparently you need to work on the same Display object and then flush it at the end. Something like:
display = Xlib.display.Display()
screen = display.screen()
root = screen.root
# ...
sendEvent(root, display.intern_atom("_NET_CURRENT_DESKTOP"), [1, X.CurrentTime])
display.flush()
Credit: Idea from a very similar thread (which almost works).
P.S. By the way, the desktop number starts from 0.