I'd like to build a website with a sandboxed interpreter (or compiler) either on the client side of on the server side that can take short blocks of code (python/java/c/c++ any common language would do) as input and execute it.
What I want to build is a place where given a programming question, the user can type in the solution and we can run it through some test cases, to either approve the solution or provide a test case where it breaks.
Looking for pointers to libraries, existing implementation or a general idea.
Any help much appreciated.
There are many contest websites that do something like this-- TopCoder and Timus Online Judge are two examples. They don't have much information on the technology, however.
codepad.org is the closest to what you want to do. They run programs on heavily sandboxed and firewalled EC2 servers that are periodically wiped, to prevent exploits.
Codepad is at least partially based on geordi, an IRC bot designed to run arbitrary C++ programs. It uses Haskell and traps system calls to prevent harmful activity.
Of slightly less interest, one of Google App Engine's example projects is a Python shell. It relies on GAE's server-side sandboxing to prevent malicious activity.
In terms of interface, the simplest would be to do something like the Internation Informatics Olympiad. Have people write a function with a certain name in the target language, then invoke that from your testing framework. Have simple functions that will let them request information from the framework, if necessary.
For Python you can compile PyPy in sandboxed mode which gives you a complete interpreter and full standard library but without the ability to execute arbitrary system calls. You can also limit the runtime and heap size of executed scripts.
Here's some code I wrote a while back to execute an arbitrary string containing a Python script in the pypy-sandbox binary and return the output. You can call this code from regular CPython.
Take a look at the paper An Enticing Environment for Programming which discusses building just such an environment.
Related
I am the TA for a coding class in which students will have to write Python 3 scripts to solve programming problems. An assignment consists of several problems, and for each problem the student is supposed to write a python program that will read input from standard input and write the output to the standard output. And for each problem there will be hidden test cases that we will use to evaluate their codes and grade them accordingly. So the idea is to automatize this process as much as possible. The problem is how to implement the whole framework to run students' assignments without compromising the safety of the system the assignments will be running on, which will probably be my laptop (which has Windows 10). I need to set up some kind of sandbox for Python 3, establishing limits for execution time, memory usage, disallowing access to the file system, networking, limiting imports only to safe modules from Python's standard library, etc.
Conceptually speaking I would like some kind of sand-boxed service that can receive a python script + some tests cases, the service runs the python script against the test cases in a safe environment (detecting compilation errors, time limit exceeded errors, memory limit exceeded errors, attempts to use forbidden libraries, etc.) and reporting the results back. So from Windows I can simply write a simple script that iterates over all students submissions and uses this service as a black-box to evaluate them.
Is anything like that possible on Windows 10? If so, how? My educated guess is that something like Docker or a Virtual Machine might be useful, but to be honest I'm not really sure because I lack enough expertise in these technologies, so I'm open to any suggestions.
Any advises on how to set up a secure system for automatic evaluation of untrusted Python 3 code submissions will be very appreciated.
What you are looking for a system that automatically evaluates a code using test cases.
You can use CMS to satisfy your use case. It is mainly a system to manage a programming contest, but it will be perfect for what you are trying to accomplish in your class.
I've a some analysis code that's written in Python. Currently I'm using Storm to process streams. Because Storm allows invocation of python code using message serialization I an invoke Python code from Java/Scala in my Storm bolts.
I found Pykka which is a Python implementation of the actor model. I was wondering if there is a way to invoke Python code from Akka actors? For example, is it possible to pass message from Akka actors to Pykka actors ?
I doubt that the wire protocols for the two remote actor model implementations could be easily bridged, but you could use 0MQ between the Scala code and a Python app. Akka allows message passing over 0MQ, so after some setup the Scala code could deal with the Python app just like any other actor, although I'm not sure what that would mean on the Python side.
Another possibility worth considering is to run your Python analysis code on the JVM using Jython. You could have a Scala actor call a Jython function/method. But if your Python code makes use of C extension modules, you'd have to find alternatives.
Yet another possibility (also assuming you aren't using C extension modules) is to give the converter py2scala a try; for data analysis code it might do a decent job. Presumably this would give you the most performant solution.
You could also use Jep to run embedded CPython in the actor JVM for full access to the native (C) Python packages.
See slide 5 here Mixing Python and Java.
If Python packages you need are not thread safe then can use remote Akka actors.
See https://github.com/alpinedatalabs/alpine-r/blob/35209af47f896450e4f8f8745a80595d01035ca3/server/src/main/resources/application.conf#L4
To make the Scala to Python smoother try ScalaPy - also built on top of Jep.
Technically something similar should also work for R using Renjin - GPL, unfortunately so must be careful with running embedded.
I am looking for some advice/opinions of which Python Framework to use in an implementation of multiple 'Worker' PCs co-ordinated from a central Queue Manager.
For completeness, the 'Worker' PCs will be running Audio Conversion routines (which I do not need advice on, and have standalone code that works).
The Audio conversion takes a long time, and I need to co-ordinate an arbitrary number of the 'Workers' from a central location, handing them conversion tasks (such as where to get the source files, or where to ask for the job configuration) with them reporting back some additional info, such as the runtime of the converted audio etc.
At present, I have a script that makes a webservice call to get the 'configuration' for a conversion task, based on source files located on the worker already (we manually copy the source files to the worker, and that triggers a conversion routine). I want to change this, so that we can distribute conversion tasks ("Oy you, process this: xxx") based on availability, and in an ideal world, based on pending tasks too.
There is a chance that Workers can go offline mid-conversion (but this is not likely).
All the workers are Windows based, the co-ordinator can be WIndows or Linux.
I have (in my initial searches) come across the following - and I know that some are cross-dependent:
Celery (with RabbitMQ)
Twisted
Django
Using a framework, rather than home-brewing, seems to make more sense to me right now. I have a limited timeframe in which to develop this functional extension.
An additional consideration would be using a Framework that is compatible with PyQT/PySide so that I can write a simple UI to display Queue status etc.
I appreciate that the specifics above are a little vague, and I hope that someone can offer me a pointer or two.
Again: I am looking for general advice on which Python framework to investigate further, for developing a Server/Worker 'Queue management' solution, for non-web activities (this is why DJango didn't seem the right fit).
How about using pyro? It gives you remote object capability and you just need a client script to coordinate the work.
Is there any way to disable a module from being loaded on my system? Let's say i would like to restrict my users from accessing the subprocess or popen2 module. Something like PHP's 'disabled_functions' or any similar method to achieve the same thing.
As #Thomas points out, blacklisting is a pretty poor mechanism for implementing any security mechanisms. Whitelisting is a much safer approach.
But a mechanism inside the interpreter isn't particularly excellent for any number of reasons: flaws in the interpreter that are exploitable at the source code level would allow users to walk right past any mechanisms built in at that level (and the PHP team asked Linux vendors to stop calling this a security problem, because (a) they fixed one of these every week and (b) trying to confine an untrusted user-supplied script is pretty much an impossible task -- use FastCGI or similar tools for potentially untrusted scripts).
The Python interpreter is probably not designed to handle malicious input, so don't treat it as such.
If you really want to confine what untrusted users can do with Python scripts, a few pieces of advice: Do not use mod_python or anything like it. Use FastCGI or similar tools that you let specify the user account that should run the script and won't execute the script as your webserver user. And learn how to configure SELinux or AppArmor to confine what that process can do -- an hour setting up one of these tools might save you huge headaches down the road, plus you get to laugh at all the cute little exploit attempts that fail.
This question already has answers here:
How can I sandbox Python in pure Python?
(7 answers)
Python, safe, sandbox [duplicate]
(9 answers)
Closed 9 years ago.
Let's say there is a server on the internet that one can send a piece of code to for evaluation. At some point server takes all code that has been submitted, and starts running and evaluating it. However, at some point it will definitely bump into "os.system('rm -rf *')" sent by some evil programmer. Apart from "rm -rf" you could expect people try using the server to send spam or dos someone, or fool around with "while True: pass" kind of things.
Is there a way to coop with such unfriendly/untrusted code? In particular I'm interested in a solution for python. However if you have info for any other language, please share.
If you are not specific to CPython implementation, you should consider looking at PyPy[wiki] for these purposes — this Python dialect allows transparent code sandboxing.
Otherwise, you can provide fake __builtin__ and __builtins__ in the corresponding globals/locals arguments to exec or eval.
Moreover, you can provide dictionary-like object instead of real dictionary and trace what untrusted code does with it's namespace.
Moreover, you can actually trace that code (issuing sys.settrace() inside restricted environment before any other code executed) so you can break execution if something will go bad.
If none of solutions is acceptable, use OS-level sandboxing like chroot, unionfs and standard multiprocess python module to spawn code worker in separate secured process.
You can check pysandbox which does just that, though the VM route is probably safer if you can afford it.
It's impossible to provide an absolute solution for this because the definition of 'bad' is pretty hard to nail down.
Is opening and writing to a file bad or good? What if that file is /dev/ram?
You can profile signatures of behavior, or you can try to block anything that might be bad, but you'll never win. Javascript is a pretty good example of this, people run arbitrary javascript code all the time on their computers -- it's supposed to be sandboxed but there's all sorts of security problems and edge conditions that crop up.
I'm not saying don't try, you'll learn a lot from the process.
Many companies have spent millions (Intel just spent billions on McAffee) trying to understand how to detect 'bad code' -- and every day machines running McAffe anti-virus get infected with viruses. Python code isn't any less dangerous than C. You can run system calls, bind to C libraries, etc.
I would seriously consider virtualizing the environment to run this stuff, so that exploits in whatever mechanism you implement can be firewalled one more time by the configuration of the virtual machine.
Number of users and what kind of code you expect to test/run would have considerable influence on choices btw. If they aren't expected to link to files or databases, or run computationally intensive tasks, and you have very low pressure, you could be almost fine by just preventing file access entirely and imposing a time limit on the process before it gets killed and the submission flagged as too expensive or malicious.
If the code you're supposed to test might be any arbitrary Django extension or page, then you're in for a lot of work probably.
You can try some generic sanbox such as Sydbox or Gentoo's sandbox. They are not Python-specific.
Both can be configured to restrict read/write to some directories. Sydbox can even sandbox sockets.
I think a fix like this is going to be really hard and it reminds me of a lecture I attended about the benefits of programming in a virtual environment.
If you're doing it virtually its cool if they bugger it. It wont solve a while True: pass but rm -rf / won't matter.
Unless I'm mistaken (and I very well might be), this is much of the reason behind the way Google changed Python for the App Engine. You run Python code on their server, but they've removed the ability to write to files. All data is saved in the "nosql" database.
It's not a direct answer to your question, but an example of how this problem has been dealt with in some circumstances.