Related
I am creating an app on a Raspberry Pi (if its at all relevent) which needs to access data from specific cells from a spreadsheet in order to display the settings as button text. There are multiple 'modes', settings and buttons. The button that may need to update varies and I have specified this with a Boolean variable. The text that needs to be displayed on the specified button can also vary by time and worksheet. I was able to create a variable to hold the time and then using a nested f-string was able to display data from the correct cell:
current_time = datetime.now().strftime('%H')
cell_number = int(current_time)+2
bool = True
def update_button():
if bool == True:
button.config(text=f"{ws_sheet[f'A{cell_number}'].value}")
else:
button1.config(text=f"{ws_sheet[f'B{cell_number}'].value}")
This worked fine but I also need to alter the sheet now as well based on a different variable. I tried this but it produces an fstring syntax error.
current_time = datetime.now().strftime('%H')
cell_number = int(current_time)+2
bool = True
worksheet = some_worksheet
def update_button():
if bool == True:
button.config(text=f"{{worksheet}[f'A{cell_number}'].value}")
else:
button1.config(text=f"{{worksheet}[f'B{cell_number}'].value}")
The syntax error produced:
button.config(text=f"{{worksheet}[f'A{cell_number}'].value}")
^
SyntaxError: f-string: single '}' is not allowed
This seems like a counter-intuitive error to me as well.
I am open to the fact I am probably going about this totally the wrong way and for my own ego I feel compelled to mention that I haven't been doing this very long. I know I could achieve what I want with a ton of if/else statements but I was hoping to do something a bit more elegant.
Any help is greatly appreciated.
You can pass work sheet when calling your function.
def update_button(ws_sheet):
if bool == True:
button.config(text=f"{ws_sheet[f'A{cell_number}'].value}"
else:
button1.config(text=f"{ws_sheet[f'B{cell_number}'].value}"
I'm currently building quite a complex system in Python, and when I'm debugging I often put simple print statements in several scripts. To keep an overview I often also want to print out the file name and line number where the print statement is located. I can of course do that manually, or with something like this:
from inspect import currentframe, getframeinfo
print getframeinfo(currentframe()).filename + ':' + str(getframeinfo(currentframe()).lineno) + ' - ', 'what I actually want to print out here'
Which prints something like:
filenameX.py:273 - what I actually want to print out here
To make it more simple, I want to be able to do something like:
print debuginfo(), 'what I actually want to print out here'
So I put it into a function somewhere and tried doing:
from debugutil import debuginfo
print debuginfo(), 'what I actually want to print out here'
print debuginfo(), 'and something else here'
Unfortunately, I get:
debugutil.py:3 - what I actually want to print out here
debugutil.py:3 - and something else here
It prints out the file name and line number on which I defined the function, instead of the line on which I call debuginfo(). This is obvious, because the code is located in the debugutil.py file.
So my question is actually: How can I get the filename and line number from which this debuginfo() function is called?
The function inspect.stack() returns a list of frame records, starting with the caller and moving out, which you can use to get the information you want:
from inspect import getframeinfo, stack
def debuginfo(message):
caller = getframeinfo(stack()[1][0])
print("%s:%d - %s" % (caller.filename, caller.lineno, message)) # python3 syntax print
def grr(arg):
debuginfo(arg) # <-- stack()[1][0] for this line
grr("aargh") # <-- stack()[2][0] for this line
Output:
example.py:8 - aargh
If you put your trace code in another function, and call that from your main code, then you need to make sure you get the stack information from the grandparent, not the parent or the trace function itself
Below is a example of 3 level deep system to further clarify what I mean. My main function calls a trace function, which calls yet another function to do the work.
######################################
import sys, os, inspect, time
time_start = 0.0 # initial start time
def trace_libary_init():
global time_start
time_start = time.time() # when the program started
def trace_library_do(relative_frame, msg=""):
global time_start
time_now = time.time()
# relative_frame is 0 for current function (this one),
# 1 for direct parent, or 2 for grand parent..
total_stack = inspect.stack() # total complete stack
total_depth = len(total_stack) # length of total stack
frameinfo = total_stack[relative_frame][0] # info on rel frame
relative_depth = total_depth - relative_frame # length of stack there
# Information on function at the relative frame number
func_name = frameinfo.f_code.co_name
filename = os.path.basename(frameinfo.f_code.co_filename)
line_number = frameinfo.f_lineno # of the call
func_firstlineno = frameinfo.f_code.co_firstlineno
fileline = "%s:%d" % (filename, line_number)
time_diff = time_now - time_start
print("%13.6f %-20s %-24s %s" % (time_diff, fileline, func_name, msg))
################################
def trace_do(msg=""):
trace_library_do(1, "trace within interface function")
trace_library_do(2, msg)
# any common tracing stuff you might want to do...
################################
def main(argc, argv):
rc=0
trace_libary_init()
for i in range(3):
trace_do("this is at step %i" %i)
time.sleep((i+1) * 0.1) # in 1/10's of a second
return rc
rc=main(sys.argv.__len__(), sys.argv)
sys.exit(rc)
This will print something like:
$ python test.py
0.000005 test.py:39 trace_do trace within interface func
0.001231 test.py:49 main this is at step 0
0.101541 test.py:39 trace_do trace within interface func
0.101900 test.py:49 main this is at step 1
0.302469 test.py:39 trace_do trace within interface func
0.302828 test.py:49 main this is at step 2
The trace_library_do() function at the top is an example of something that you can drop into a library, and then call it from other tracing functions. The relative depth value controls which entry in the python stack gets printed.
I showed pulling out a few other interesting values in that function, like the line number of start of the function, the total stack depth, and the full path to the file. I didn't show it, but the global and local variables in the function are also available in inspect, as well as the full stack trace to all other functions below yours. There is more than enough information with what I am showing above to make hierarchical call/return timing traces. It's actually not that much further to creating the main parts of your own source level debugger from here -- and it's all mostly just sitting there waiting to be used.
I'm sure someone will object that I'm using internal fields with data returned by the inspect structures, as there may well be access functions that do this same thing for you. But I found them in by stepping through this type of code in a python debugger, and they work at least here. I'm running python 2.7.12, your results might very if you are running a different version.
In any case, I strongly recommend that you import the inspect code into some python code of your own, and look at what it can provide you -- Especially if you can single step through your code in a good python debugger. You will learn a lot on how python works, and get to see both the benefits of the language, and what is going on behind the curtain to make that possible.
Full source level tracing with timestamps is a great way to enhance your understanding of what your code is doing, especially in more of a dynamic real time environment. The great thing about this type of trace code is that once it's written, you don't need debugger support to see it.
An update to the accepted answer using string interpolation and displaying the caller's function name.
import inspect
def debuginfo(message):
caller = inspect.getframeinfo(inspect.stack()[1][0])
print(f"{caller.filename}:{caller.function}:{caller.lineno} - {message}")
The traceprint package can now do that for you:
import traceprint
def func():
print(f'Hello from func')
func()
# File "/traceprint/examples/example.py", line 6, in <module>
# File "/traceprint/examples/example.py", line 4, in func
# Hello from func
PyCharm will automatically make the file link clickable / followable.
Install via pip install traceprint.
Just put the code you posted into a function:
from inspect import currentframe, getframeinfo
def my_custom_debuginfo(message):
print getframeinfo(currentframe()).filename + ':' + str(getframeinfo(currentframe()).lineno) + ' - ', message
and then use it as you want:
# ... some code here ...
my_custom_debuginfo('what I actually want to print out here')
# ... more code ...
I recommend you put that function in a separate module, that way you can reuse it every time you need it.
Discovered this question for a somewhat related problem, but I wanted more details re: the execution (and I didn't want to install an entire call graph package).
If you want more detailed information, you can retrieve a full traceback with the standard library module traceback, and either stash the stack object (a list of tuples) with traceback.extract_stack() or print it out with traceback.print_stack(). This was more suitable for my needs, hope it helps someone else!
I am working on a django based web app that takes python file as input which contains some function, then in backend i have some lists that are passed as parameters through the user's function,which will generate a single value output.The result generated will be used for some further computation.
Here is how the function inside the user's file look like :
def somefunctionname(list):
''' some computation performed on list'''
return float value
At present the approach that i am using is taking user's file as normal file input. Then in my views.py i am executing the file as module and passing the parameters with eval function. Snippet is given below.
Here modulename is the python file name that i had taken from user and importing as module
exec("import "+modulename)
result = eval(f"{modulename}.{somefunctionname}(arguments)")
Which is working absolutely fine. But i know this is not the secured approach.
My question , Is there any other way through which i can run users file securely as the method that i am using is not secure ? I know the proposed solutions can't be full proof but what are the other ways in which i can run this (like if it can be solved with dockerization then what will be the approach or some external tools that i can use with API )?
Or if possible can somebody tell me how can i simply sandbox this or any tutorial that can help me..?
Any reference or resource will be helpful.
It is an important question. In python sandboxing is not trivial.
It is one of the few cases where the question which version of python interpreter you are using. For example, Jyton generates Java bytecode, and JVM has its own mechanism to run code securely.
For CPython, the default interpreter, originally there were some attempts to make a restricted execution mode, that were abandoned long time ago.
Currently, there is that unofficial project, RestrictedPython that might give you what you need. It is not a full sandbox, i.e. will not give you restricted filesystem access or something, but for you needs it may be just enough.
Basically the guys there just rewrote the python compilation in a more restricted way.
What it allows to do is to compile a piece of code and then execute, all in a restricted mode. For example:
from RestrictedPython import safe_builtins, compile_restricted
source_code = """
print('Hello world, but secure')
"""
byte_code = compile_restricted(
source_code,
filename='<string>',
mode='exec'
)
exec(byte_code, {__builtins__ = safe_builtins})
>>> Hello world, but secure
Running with builtins = safe_builtins disables the dangerous functions like open file, import or whatever. There are also other variations of builtins and other options, take some time to read the docs, they are pretty good.
EDIT:
Here is an example for you use case
from RestrictedPython import safe_builtins, compile_restricted
from RestrictedPython.Eval import default_guarded_getitem
def execute_user_code(user_code, user_func, *args, **kwargs):
""" Executed user code in restricted env
Args:
user_code(str) - String containing the unsafe code
user_func(str) - Function inside user_code to execute and return value
*args, **kwargs - arguments passed to the user function
Return:
Return value of the user_func
"""
def _apply(f, *a, **kw):
return f(*a, **kw)
try:
# This is the variables we allow user code to see. #result will contain return value.
restricted_locals = {
"result": None,
"args": args,
"kwargs": kwargs,
}
# If you want the user to be able to use some of your functions inside his code,
# you should add this function to this dictionary.
# By default many standard actions are disabled. Here I add _apply_ to be able to access
# args and kwargs and _getitem_ to be able to use arrays. Just think before you add
# something else. I am not saying you shouldn't do it. You should understand what you
# are doing thats all.
restricted_globals = {
"__builtins__": safe_builtins,
"_getitem_": default_guarded_getitem,
"_apply_": _apply,
}
# Add another line to user code that executes #user_func
user_code += "\nresult = {0}(*args, **kwargs)".format(user_func)
# Compile the user code
byte_code = compile_restricted(user_code, filename="<user_code>", mode="exec")
# Run it
exec(byte_code, restricted_globals, restricted_locals)
# User code has modified result inside restricted_locals. Return it.
return restricted_locals["result"]
except SyntaxError as e:
# Do whaever you want if the user has code that does not compile
raise
except Exception as e:
# The code did something that is not allowed. Add some nasty punishment to the user here.
raise
Now you have a function execute_user_code, that receives some unsafe code as a string, a name of a function from this code, arguments, and returns the return value of the function with the given arguments.
Here is a very stupid example of some user code:
example = """
def test(x, name="Johny"):
return name + " likes " + str(x*x)
"""
# Lets see how this works
print(execute_user_code(example, "test", 5))
# Result: Johny likes 25
But here is what happens when the user code tries to do something unsafe:
malicious_example = """
import sys
print("Now I have the access to your system, muhahahaha")
"""
# Lets see how this works
print(execute_user_code(malicious_example, "test", 5))
# Result - evil plan failed:
# Traceback (most recent call last):
# File "restr.py", line 69, in <module>
# print(execute_user_code(malitious_example, "test", 5))
# File "restr.py", line 45, in execute_user_code
# exec(byte_code, restricted_globals, restricted_locals)
# File "<user_code>", line 2, in <module>
#ImportError: __import__ not found
Possible extension:
Pay attention that the user code is compiled on each call to the function. However, it is possible that you would like to compile the user code once, then execute it with different parameters. So all you have to do is to save the byte_code somewhere, then to call exec with a different set of restricted_locals each time.
EDIT2:
If you want to use import, you can write your own import function that allows to use only modules that you consider safe. Example:
def _import(name, globals=None, locals=None, fromlist=(), level=0):
safe_modules = ["math"]
if name in safe_modules:
globals[name] = __import__(name, globals, locals, fromlist, level)
else:
raise Exception("Don't you even think about it {0}".format(name))
safe_builtins['__import__'] = _import # Must be a part of builtins
restricted_globals = {
"__builtins__": safe_builtins,
"_getitem_": default_guarded_getitem,
"_apply_": _apply,
}
....
i_example = """
import math
def myceil(x):
return math.ceil(x)
"""
print(execute_user_code(i_example, "myceil", 1.5))
Note that this sample import function is VERY primitive, it will not work with stuff like from x import y. You can look here for a more complex implementation.
EDIT3
Note, that lots of python built in functionality is not available out of the box in RestrictedPython, it does not mean it is not available at all. You may need to implement some function for it to become available.
Even some obvious things like sum or += operator are not obvious in the restricted environment.
For example, the for loop uses _getiter_ function that you must implement and provide yourself (in globals). Since you want to avoid infinite loops, you may want to put some limits on the number of iterations allowed. Here is a sample implementation that limits number of iterations to 100:
MAX_ITER_LEN = 100
class MaxCountIter:
def __init__(self, dataset, max_count):
self.i = iter(dataset)
self.left = max_count
def __iter__(self):
return self
def __next__(self):
if self.left > 0:
self.left -= 1
return next(self.i)
else:
raise StopIteration()
def _getiter(ob):
return MaxCountIter(ob, MAX_ITER_LEN)
....
restricted_globals = {
"_getiter_": _getiter,
....
for_ex = """
def sum(x):
y = 0
for i in range(x):
y = y + i
return y
"""
print(execute_user_code(for_ex, "sum", 6))
If you don't want to limit loop count, just use identity function as _getiter_:
restricted_globals = {
"_getiter_": labmda x: x,
Note that simply limiting the loop count does not guarantee security. First, loops can be nested. Second, you cannot limit the execution count of a while loop. To make it secure, you have to execute unsafe code under some timeout.
Please take a moment to read the docs.
Note that not everything is documented (although many things are). You have to learn to read the project's source code for more advanced things. Best way to learn is to try and run some code, and to see what kind function is missing, then to see the source code of the project to understand how to implement it.
EDIT4
There is still another problem - restricted code may have infinite loops. To avoid it, some kind of timeout is required on the code.
Unfortunately, since you are using django, that is multi threaded unless you explicitly specify otherwise, simple trick for timeouts using signeals will not work here, you have to use multiprocessing.
Easiest way in my opinion - use this library. Simply add a decorator to execute_user_code so it will look like this:
#timeout_decorator.timeout(5, use_signals=False)
def execute_user_code(user_code, user_func, *args, **kwargs):
And you are done. The code will never run more than 5 seconds.
Pay attention to use_signals=False, without this it may have some unexpected behavior in django.
Also note that this is relatively heavy on resources (and I don't really see a way to overcome this). I mean not really crazy heavy, but it is an extra process spawn. You should hold that in mind in your web server configuration - the api which allows to execute arbitrary user code is more vulnerable to ddos.
For sure with docker you can sandbox the execution if you are careful. You can restrict CPU cycles, max memory, close all network ports, run as a user with read only access to the file system and all).
Still,this would be extremely complex to get it right I think. For me you shall not allow a client to execute arbitrar code like that.
I would be to check if a production/solution isn't already done and use that. I was thinking that some sites allow you to submit some code (python, java, whatever) that is executed on the server.
Let's assume I have two lists of variables
list a: a1 a2 a3
list b: b1 b2 b3
which I want to process in a way like this:
TEMPORARY.
SELECT IF a1=b1.
FREQUENCY someVar.
TEMPORARY.
SELECT IF a2=b2.
FREQUENCY someVar.
TEMPORARY.
SELECT IF a2=b2.
FREQUENCY someVar.
I tried tried to do this within a python loop:
BEGIN PROGRAM.
import spss
la = ['a1', 'a2', 'a3']
lb = ['b1', 'b2', 'b3']
for a, b in zip(la, lb):
spss.Submit('''
TEMPORARY.
SELECT IF %s=%s.
FREQUENCY someVar.
''' % (a, b))
END PROGRAM.
So far so good. This works except when the SELECT IF command would create an empty dataset. Outside the Python program block this would cause the following warning message in the output viewer:
No cases were input to this procedure. Either there are none in the
working data file or all of them have been filtered out. Execution of
this command stops.
But inside a Python block it causes an Error and the python script to stop.
Traceback (most recent call last):
File "", line 7, in
File "C:\PROGRA~1\ibm\SPSS\STATIS~1\23\Python\Lib\site-packages\spss\spss.py", line 1527, in Submit
raise SpssError,error spss.errMsg.SpssError: [errLevel 3] Serious error.
Is there a way to run this loop (which might produce temporary empty data sets and therefore warnings) inside of python?
Yes, if you wrap the problematic function inside a try-except construct:
for a, b in zip(la, lb):
try:
spss.Submit('''
TEMPORARY.
SELECT IF %s=%s.
FREQUENCY someVar.
''' % (a, b))
except:
pass
You could also use Python APIs to calculate how many cases ai=bi and execute conditional blocks according to this.
So for example if only less than 5 valid cases remain you may not want to produce any output (or some output suggesting no output is being produced due to low base sizes). If under 50 cases remain then you may want to run frequencies and if more than 50 cases then run descriptives ect ect. There are a number of ways you get the case count, which approach is best and most efficient perhaps depends on your data set and end goal.
See for example:
spss.GetCaseCount
Here's another approach where you can get case count like statistics from active dataset to inspire further ideas.
Adding some explanation: Statistics syntax errors have an associated severity level between 1 (lowest) and 5. You will probably never see a 5, because that means that the system has gone down in flames. When running syntax from Python via Submit, level 1 and 2 errors, which are warnings that don't stop the syntax from running, are executed normally. Level 3 and higher raise exceptions. You can handle those in your Python code via the try/except mechanism as was suggested above.
I experimented with SET MXWARNS, however neither setting it to zero, nor setting it to a very high value (e.g. 1000000) worked. The warnings were still converted into errors. So I wrote this work-around:
import codecs
import re
import sys
import spss
from spssaux import getShow
def submit_syntax(sps_filename):
output_on = spss.IsOutputOn()
spss.SetOutput("off")
unicode_mode = getShow("unicode") == u"Yes"
encoding = "utf-8-sig" if unicode_mode else getShow("locale").split(u".")[-1]
if output_on:
spss.SetOutput("on")
with codecs.open(sps_filename, encoding=encoding) as f:
syntax = f.read()
statements = re.split(ur"\. *\r\n", syntax, flags=re.MULTILINE)
for stmtno, statement in enumerate(statements, 1):
if statement.startswith(u"*"):
continue
try:
spss.Submit(statement)
except spss.SpssError:
# "no cases were input" warnings are translated into errors.
if not spss.GetCaseCount() and spss.GetLastErrorLevel() <= 3:
continue
msg = u"ERROR in statement #%d: %s [%s]"
raise RuntimeError(msg % (stmtno, statement, sys.exc_info()[1]))
Me: I am running Python 2.3.3 without possibility to upgrade and i don't have much experience with Python. My method for learning is googling and reading tons of stackoverflow.
Background: I am creating a python script whose purpose is to take two directories as arguments and then perform comparisons/diff of all the files found within the two directories. The directories have sub-directories that also have to be included in the diff.
Each directory is a List and sub-directories are nested Lists and so on...
the two directories:
oldfiles/
a_tar_ball.tar
a_text_file.txt
nest1/
file_in_nest
nest1a/
file_in_nest
newfiles/
a_tar_ball.tar
a_text_file.txt
nest1/
file_in_nest
nest1a/
Problem: Normally all should go fine as all files in oldfiles should exist in newfiles but in the above example one of the 'file_in_nest' is missing in 'newfiles/'.
I wish to print an error message telling me which file that is missing but when i'm using the code structure below the current instance of my 'compare' function doesn't know any other directories but the closest one. I wonder if there is a built in error handling that can send information about files and directory up in the recursion ladder adding info to it as we go. If i would just print the filename of the missing file i would not know which one of them it might be as there are two 'file_in_nest' in 'oldfiles'
def compare(file_tree)
for counter, entry in enumerate(file_tree[0][1:]):
if not entry in file_tree[1]
# raise "some" error and send information about file back to the
# function calling this compare, might be another compare.
elif not isinstance(entry, basestring):
os.chdir(entry[0])
compare(entry)
os.chdir('..')
else:
# perform comparison (not relevant to the problem)
# detect if "some" error has been raised
# prepend current directory found in entry[0] to file information
break
def main()
file_tree = [['/oldfiles', 'a_tar_ball.tar', 'a_text_file.txt', \
[/nest1', 'file_in_nest', [/nest1a', 'file_in_nest']], \
'yet_another_file'], \
['/newfiles', 'a_tar_ball.tar', 'a_text_file.txt', \
[/nest1', 'file_in_nest', [/nest1a']], \
'yet_another_file']]
compare(file_tree)
# detect if "some" error has been raised and print error message
This is my first activity on stackoverflow other than reading som please tell me if i should improve on the question!
// Stefan
Well, it depends whether you want to report an error as an exception or as some form of status.
Let's say you want to go the 'exception' way and have the whole program crash if one file is missing, you can define your own exception saving the state from the callee to the caller:
class PathException(Exception):
def __init__(self, path):
self.path = path
Exception.__init__(self)
def compare(filetree):
old, new = filetree
for counter, entry in enumerate(old[1:]):
if entry not in new:
raise PathException(entry)
elif not isinstance(entry, basestring):
os.chdir(entry[0])
try:
compare(entry)
os.chdir("..")
except PathException as e:
os.chdir("..")
raise PathException(os.path.join(entry, e.path))
else:
...
Where you try a recursive call, and update any incoming exception with the information of the caller.
To see it on a smaller example, let's try to deep-compare two lists, and raise an exception if they are not equal:
class MyException(Exception):
def __init__(self, path):
self.path = path
Exception.__init__(self)
def assertEq(X, Y):
if hasattr(X, '__iter__') and hasattr(Y, '__iter__'):
for i, (x, y) in enumerate(zip(X, Y)):
try:
assertEq(x, y)
except MyException as e:
raise MyException([i] + e.path)
elif X != Y:
raise MyException([]) # Empty path -> Base case
This gives us:
>>> L1 = [[[1,2,3],[4,5],[[6,7,8],[7,9]]],[3,5,[7,8]]]
>>> assertEq(L1, L1)
Nothing happens (lists are similar), and:
>>> L1 = [[[1,2,3],[4,5],[[6,7,8],[7,9]]],[3,5,[7,8]]]
>>> L2 = [[[1,2,3],[4,5],[[6,7,8],[7,5]]],[3,5,[7,8]]] # Note the [7,9] -> [7,5]
>>> try:
... assertEq(L1, L2)
... except MyException as e:
... print "Diff at",e.path
Diff at [0, 2, 1, 1]
>>> print L1[0][2][1][1], L2[0][2][1][1]
9 5
Which gives the full path.
As recursive lists or paths are basically the same thing, it is easy to adapt it to your use case.
Another simple way of solving this would be to report this difference in files as a simple diff, similar to the others: you can return it as a difference between the old file and the (non-existent) new file, or return both the list of differences in files and the list of differences of files, in which case it is easy to update recursively the values as they are returned by the recursive calls.