We Keep Coding

Why doesn't this simple Python script run?

I am writing a large program where I need to pass data/variables between functions. Note: I'm a hobbyist and OOP is out of my grasp, so just looking for a non-OOP answer!
I'm using functions to try and make the script modular and avoid having one long messy script. But the program uses a dataframe and lots of different variables which many of the functions will need to access. I don't want to specify every single variable in every function call so would like to be able to access global variables from individual functions. I can do this when the def function(): is in the same script, but I am running into a problem when I try and call global variables when importing a function from a script. Simple reprex:
from test_func import p_func
a = "yes!"
p_func()
calling p_func() from test_func.py
def p_func():
global a
print(a)
generates the error:
Traceback (most recent call last):
File "test.py", line 5, in <module>
p_func()
File "test_func.py", line 5, in p_func
print(a)
NameError: name 'a' is not defined
What am I missing?

You need to change your import line to be:
from test_func import p_func, a
Variables are imported from other modules the same way that functions are.
That said. This is really, really a bad idea as others above has said. Your best off putting all your variables into a single data structure of some sort

how to ensure safe execution of eval (and maybe exec)

I know that eval and exec are quite dangerous things, but is it possible to completely defend against possible problems? (in the case of exec - it is unlikely that everything is too problematic there, but I have hope for the safe execution of eval),
here is what i have: eval (s, {'__builtins __': {}})
but I know that, for example, this can be circumvented in several ways, for example:
s = """
(lambda fc=(
lambda n: [
c for c in
().__class__.__bases__[0].__subclasses__()
if c.__name__ == n
][0]
):
fc("function")(
fc("code")(
0,0,0,0,"KABOOM",(),(),(),"","",0,""
),{}
)()
)()
"""
for brevity I will omit the details of what is happening
In the example above, we used a list of all subclasses of the object class to create objects of the code and function classes. In exactly the same way, you can get (and instantiate) any class that exists in the program at the time you called eval ().
Here is another example of what can be done:
s = """
[
c for c in
().__class__.__bases__[0].__subclasses__()
if c.__name__ == "Quitter"
][0](0)()
"""
When I did research on this topic, I came across the protected eval execution mode in Python, which is another attempt to overcome this problem:
>>> eval("(lambda:0).func_code", {'__builtins__':{}})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in <module>
RuntimeError: function attributes not accessible in restricted mode
In short, it works as follows: if __builtins__ inside eval are different from the “official” ones, eval goes into protected mode, which denies access to some dangerous properties, such as func_code for functions.
in principle, you can get around the restriction using this code (here we get the "official" __builtins__):
s = """[
c for c in ().__class__.__base__.__subclasses__()
if c.__name__ == 'catch_warnings'
][0]()._module.__builtins__
"""
my question is: is it possible to avoid the problems with eval to the end?

How to add a global symbol that can be bound by Python compiler?

This is my first question so please be nice :) I am rather new to Python but I am very experienced in other programming languages (e.g. C++).
UPDATE 2 - Solution Found
Thanks everyone for helping :) As the solution is "hidden" in comments I will repost it here.
Instead of
file_symbols = {}
the variable local_symbol must initially be added to the file_symbols dictionary:
file_symbols = { "local_symbol" : local_symbol }
For anyone reading this: all variable / class names posted here are not to be understood as actual useful names as these examples are synthetic in nature ;)
Well... now I have to figure out the FULL meaning of:
exec compiled_code in file_symbols
So far I thought it would do nothing more than updating the dictionary file_symbols with the symbols found in compiled_code.
But it actually does a bit more as it seems! :)
UPDATE 1
Ok, my sample project below seems to be too simple to show the actual problem. Anyway, thanks for your already provided support! :)
In fact I want to first compile multiple *.py files which need access to a local symbol (class instance). All symbols coming from these compiled files shall be collected and then be used as an environment for OTHER code objects.
So I really need to do this
(note the following code shows the concept, not actual executable code):
class Functions:
(...)
global_symbols = {}
local_symbol = Functions()
# note that the external files need to access local_symbol to call its functions!
for file in external_files:
code = file.load()
compiled_code = compile(code, "<string>", "exec")
file_symbols = {}
exec compiled_code in file_symbols
global_symbols.update(file_symbols)
some_code = another_file.load()
compiled_code = compile(some_code, "<string>", "exec")
exec(compiled_code, global_symbols)
In this example the line
exec compiled_code in file_symbols
produces a NameError() - because there is no way they could access local_symbol as it is not defined anywhere in the external files although it shall be used!
So the question is how to provide access to local_symbol for the external_files so that they can call the instance's functions??
My import hook solution that some of you regard as "hack" was the only working solution so far. I would love to use a simpler one if there is any!
Thanks again :)
My initial question was this:
So here we go. What I intend to do is advanced stuff and I did not find a solution to my problem neither here nor anywhere else.
Assume the following code in Python (2.6.x / 2.7.x):
class Functions:
def __init__(self):
(...)
def func_1(...):
(...)
def func_2(...):
(...)
(...)
def func_n(...):
(...)
functions = Functions()
code = loadSomeFile(...)
compiled_code = compile(code, "<string>", "exec")
(...)
global_env = {}
local_env = {"func":functions}
exec(compiled_code, global_env, local_env)
where code in the example above is loaded from a file with a content that might look like this:
import something
(...)
def aFunction(...):
a = func.func_1(...)
b = func.func_2(...)
return a * b
(...)
aFunction()
Please note that (...) in the code above means that there might be more code that I left out for the sake of simplicity.
The problem I encounter in my example is that the compiler raises an error for this line:
compiled_code = compile(code, "<string>", "exec")
I will get this error: NameError("global name 'func' is not defined")
This error is totally understandable as the compiler can't bind to any global symbol with the name "func". But I still want to compile the code this way.
So the obvious question is:
How can I define global symbols that can be used by the compiler for compile() statements so that the compiler will bind any "unknown" symbol to an object of my choice?
In my example I would like to define a global symbol func that is bound to an instance of class Functions so that the compiler will find this symbol when compiling code which makes use of func as seen in the example above.
So how can this be accomplished?
Important:
Please note that I am aware that directly executing the code using exec(...) would fix the compilation problem because the dict local_env in the example above would provide the symbol that is required for successful execution. HOWEVER I can't do it this way because the code that shall be compiled is not small at all. It might consist of hundreds of code lines and this code is also not executed only once but many many times.
So for performance reasons I really need to compile the code rather than directly executing it.
Thanks for helping me :)

Don't provide separate globals and locals dicts to exec. That causes the executed code to behave as if it's embedded in a class definition. That means that any variable lookups in functions defined in the executed code bypass locals.
>>> exec("""
... def f():
... print a
... f()""", {}, {"a": 3})
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "<string>", line 4, in <module>
File "<string>", line 3, in f
NameError: global name 'a' is not defined
>>> exec("""
... def f():
... print a
... f()""", {"a": 3})
3
Just pass a globals dict.

The simple explanation is that you pass in func in the local namespace. aFunction does not have access to the locals you passed in (it has its own locals). All aFunction has access to is its own locals and its module's globals. func is in neither of these and so the function call fails.
Most normal modules work with their globals and locals as the same namespace (you can check this out yourself assert globals() is locals()). This is why you can define things at the module level and have them available to any defined functions (all names in a module are automatically global).
To make this work you need to make locals and globals the same dict, or just not pass locals at all. If you don't want globals to be mutated, then just copy globals and then add func to it.
src = """
def aFunction():
a = func.func_1()
b = func.func_2()
return a + b
value = aFunction()
"""
class Functions:
def func_1(self):
return "first"
def func_2(self):
return "second"
functions = Functions()
compiled_code = compile(src, "<string>", "exec")
global_env = {}
local_env = {"func":functions}
namespace = dict(global_env)
namespace.update(local_env)
exec(compiled_code, namespace)
print(namespace["value"])

it is an interesting question, thanks for posting it. I've been taking a look at how to change the globals table in compile time. Apparently, you can call the __import__() function directly and:
pass your globals in order to determine how to interpret the name in a package context.
Source: Package documentation

Well, obviously it works by simply adding (and potentially removing - if using multiple "func" instances) the instance to sys.modules:
sys.modules["func"] = functions
(...)
compiled_code = compile(code, "<string>", "exec")
The complete and working solution including an importer hook (which intercepts "import func" lines in source code files) would look like this:
import sys
class Functions:
def __init__(self):
(...)
def func_1(...):
(...)
def func_2(...):
(...)
(...)
def func_n(...):
(...)
functions = Functions()
code = loadSomeFile(...)
hook_name = "func"
class ImporterHook:
def __init__(self, path):
if path != hook_name:
raise ImportError()
def find_module(self, fullname, path=None):
return self
def load_module(self, path):
if sys.modules.has_key(path):
return sys.modules[path]
else:
sys.modules[path] = functions
return functions
sys.path_hooks.append(ImporterHook)
sys.path.insert(0, hook_name)
compiled_code = compile(code, "<string>", "exec")
(...)
exec(compiled_code)
Not so hard as it seems :) For more information see here:
https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks
and here:
https://pymotw.com/2/sys/imports.html
THANKS :)

How do I give a string object a custom method in Python?

Similar to ".format", I want to be able to automatically prefix the time before a string. I really have no idea to do it but I think it might look something like this.
>>print("Function __init__ at CLASS myClass running...".log())
Prints:
[myPrefix] Function init at CLASS myClass running...
I have no idea in the world of how I would do this.

Sadly, you can't even monkey-patch attributes onto built-in types. This:
def log(self):
print "logging "+self
str.log = log
str("hello")
print "hello".log()
Gives:
Traceback (most recent call last):
Line 3, in <module>
str.log = log
TypeError: can't set attributes of built-in/extension type 'str'
The best way to do this is to just write a logging method, like so:
def log(s):
print("my-prefix -- "+s)
log("hello")
The advantage of this is that, if at a later stage, you decide to not print your logging statements, but pipe them into a file, you only need to change the log function, not the many places you have the print statements, for example:
def log(s):
with open("my_log.txt",w) as f:
data = f.write("the time - " + s)
log("hello")
Now, all your logging statements go to the file, without having to change the actual logging call.

Problem grabbing parameters using execfile within a class

I have a problem similar to the first problem in this question, which as far as I can see went unanswered.
I have a file "config.py" which contains a lot of parameters to be used by a class (this config.py file will change), however I can't get these to propagate into the class via execfile.
In an example piece of code:
class Class():
def __init__(self):
execfile("config.py")
print x
# config.py
x = "foo"
>>> t = Class()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __init__
NameError: global name 'x' is not defined
Any help welcome, or any better methods of retrieving parameters from a file to be used in a class.
Many Thanks.

I don't get what you're trying to do (but i don't like it, and this is just me) but to fix your problem do (test in python2.6):
class Class():
def __init__(self):
execfile('config.py', locals()) # Not recommanded, maybe you want globals().
print x
But from the doc:
Note
The default locals act as described
for function locals() below:
modifications to the default locals
dictionary should not be attempted.
Pass an explicit locals dictionary if
you need to see effects of the code on
locals after function execfile()
returns. execfile() cannot be used
reliably to modify a function’s
locals.
and about :
Any help welcome, or any better
methods of retrieving parameters from
a file to be used in a class.
You can use import.

Even though it might be convenient to keep configuration settings in a Python file I would recommend against it. I think it opens up a whole set of problems that you don't really want to have do deal with. Anything could be placed in your configuration file, including malicious code.
I would use either the json module or the ConfigParser module to hold my configuration.
If you have trouble choosing between those two I would recommend the json module. Json is a simple yet flexible format for structured data.