I'm having an issue with running a python script in a python script that i simply do not understand:
Assume we have 2 files in the same directory: 'init.py' and 'text.py'
init.py:
X = 5
print("init.py was run")
test.py:
exec(open("./init.py").read())
print("X = %s" %X)
If I run test.py now, I get
init.py was run
X = 5
However, if I change test.py into:
def func_call( filename):
exec(open(filename).read())
print("X = %s" %X)
func_call("./init.py")
I get:
init.py was run
Traceback (most recent call last):
File "test.py", line 5, in
func_call("./init.py")
File "test.py", line 3, in func_call
print("X = %s" %X)
NameError: name 'X' is not defined
Can someone explain to me why this leads to different results?
Is there a workaround for this?
My goal is to initializes most of my variables by running a python script and accessing the variables set up in that python script.
According to exec_documentation:
If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.
Inside method globals() and locals() are different objects:
def method():
print(globals() == locals())
exec('X=10')
print('Method execution =', X)
method()
output:
False
NameError: name 'X' is not defined
In global level this objects are equal:
print(globals() == locals())
exec('X=99')
print('Global exec =', X)
Output:
True
Global exec = 99
So If you want to do it via method, you need to pass the same object to exec. For your code it would look like this:
def func_call(filename):
exec(open(filename).read(), globals(), globals())
print("X = %s" %X)
func_call("./init.py")
Nevertheless, as I mentioned in comment, create file with consts and import it. Try to avoid using exec/eval at all costs, unless you are 100% sure what you are doing.
Related
I'm trying to find out how much time it takes to execute a Python statement, so I looked online and found that the standard library provides a module called timeit that purports to do exactly that:
import timeit
def foo():
# ... contains code I want to time ...
def dotime():
t = timeit.Timer("foo()")
time = t.timeit(1)
print "took %fs\n" % (time,)
dotime()
However, this produces an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in dotime
File "/usr/local/lib/python2.6/timeit.py", line 193, in timeit
timing = self.inner(it, self.timer)
File "<timeit-src>", line 6, in inner
NameError: global name 'foo' is not defined
I'm still new to Python and I don't fully understand all the scoping issues it has, but I don't know why this snippet doesn't work. Any thoughts?
Change this line:
t = timeit.Timer("foo()")
To this:
t = timeit.Timer("foo()", "from __main__ import foo")
Check out the link you provided at the very bottom.
To give the timeit module access to functions you define, you can pass a setup parameter which contains an import statement:
I just tested it on my machine and it worked with the changes.
With Python 3, you can use globals=globals()
t = timeit.Timer("foo()", globals=globals())
From the documentation:
Another option is to pass globals() to the globals parameter, which
will cause the code to be executed within your current global
namespace. This can be more convenient than individually specifying
imports
You can try this hack:
import timeit
def foo():
print 'bar'
def dotime():
t = timeit.Timer("foo()")
time = t.timeit(1)
print "took %fs\n" % (time,)
import __builtin__
__builtin__.__dict__.update(locals())
dotime()
t = timeit.Timer("foo()", "from __main__ import foo")
Since timeit doesn't have your stuff in scope.
add into your setup "import thisfile; "
then when you call the setup function myfunc() use "thisfile.myfunc()"
eg "thisfile.py"
def myfunc():
return 5
def testable(par):
pass
t=timeit.timeit(stmt="testable(v)",setup="import thisfile; v=thisfile.myfunc();").repeat(10)
print( t )
In both Python2 and Python3, in the stack trace the __name__ of a function is not used, the original name (the one that is specified after def) is used instead.
Consider the example:
import traceback
def a():
return b()
def b():
return c()
def c():
print("\n".join(line.strip() for line in traceback.format_stack()))
a.__name__ = 'A'
b.__name__ = 'B'
c.__name__ = 'C'
a();
The output is:
File "test.py", line 16, in <module>
a();
File "test.py", line 4, in a
return b()
File "test.py", line 7, in b
return c()
File "test.py", line 10, in c
print("\n".join(line.strip() for line in traceback.format_stack()))
Why so? How do I change the name that is used in the stack trace? Where is the __name__ attribute used then?
So, basically every function has three things that can be considered being name of the function:
The original name of the code block
It's stored in the f.__code__.co_name (where f is the function object). If you use def orig_name to create function, orig_name is that name. For lambas it's <lambda>.
This attribute is readonly and can't be changed. So the only way to create function with the custom name in runtime I'm aware of is exec:
exec("""def {name}():
print '{name}'
""".format(name='any')) in globals()
any() # prints 'any'
(There is also more low-level way to do this that was mentioned in a comment to the question.)
The immutability of co_name actually makes sense: with that you can be sure that the name you see in the debugger (or just stack trace) is exactly the same you see in the source code (along with the filename and line number).
The __name__ attribute of the function object
It's also aliased to func_name.
You can modify it (orig_name.__name__ = 'updated name') and you surely do on a daily basis: #functools.wraps copies the __name__ of the decorated function to the new one.
__name__ is used by tools like pydoc, that's why you need #functools.wraps: so you don't see the technical details of every decorator in your documentation. Look at the example:
from functools import wraps
def decorator1(f):
def decorated(*args, **kwargs):
print 'start1'
f(*args, **kwargs)
return decorated
def decorator2(f):
#wraps(f)
def decorated(*args, **kwargs):
print 'start2'
f(*args, **kwargs)
return decorated
#decorator1
def test1():
print 'test1'
#decorator2
def test2():
print 'test2'
Here is the pydoc output:
FUNCTIONS
decorator1(f)
decorator2(f)
test1 = decorated(*args, **kwargs)
test2(*args, **kwargs)
With wraps there is no sign of decorated in the documentation.
Name of the reference
One more thing that can be called function name (though it hardly is) is the name of a variable or an attribute where reference to that function is stored.
If you create function with def name, the name attribute will be added to the current scope. In case of lambda you should assign the result to some variable: name = lambda: None.
Obviously you can create more than one reference to the same function and all that references can have different names.
The only way all that three things are connected to each other is the def foo statement that creates function object with both __name__ and __code__.co_name equal to foo and assign it to the foo attribute of the current scope. But they are not bound in any way and can be different from each other:
import traceback
def make_function():
def orig_name():
"""Docstring here
"""
traceback.print_stack()
return orig_name
globals()['name_in_module'] = make_function()
name_in_module.__name__ = 'updated name'
name_in_module()
Output:
File "my.py", line 13, in <module>
name_in_module()
File "my.py", line 7, in orig_name
traceback.print_stack()
Pydoc:
FUNCTIONS
make_function()
name_in_module = updated name()
Docstring here
I thank other people for comments and answers, they helped me to organize my thoughts and knowledge.
Tried to explore the CPython implementation, definitely not an expert. As pointed out in the comments, when the stack entry of f is printed, the attribute f.__code__.co_name is used. Also, f.__name__ is initially set to f.__code__.co_name, but when you modify the former, the latter is not modified accordingly.
Therefore, I tried to modify that directly, but it is not possible:
>>> f.__code__.co_name = 'g'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: readonly attribute
>>>
Why are there two ways to say a function's name? Well, according to the documentation, __name__ is defined for "class, function, method, descriptor, or generator instance", so in the case of functions it maps to that attribute, for other objects it will map to something else.
I have just discovered this strange scoping behaviour of both Python 2 and 3. When I'm adding a late import for a sub-module, the main import of toplevel module stops working. Viable example:
import os
def start():
import sys
print('in modules?', 'os' in sys.modules)
print('in globals?', 'os' in globals())
print('in locals?', 'os' in locals())
print('os =', os)
import os.path
os.path.exists('useless statement')
start()
The output will be:
in modules? True
in globals? True
in locals? False
Traceback (most recent call last):
File "test.py", line 15, in <module>
start()
File "test.py", line 9, in start
print('os =', os)
UnboundLocalError: local variable 'os' referenced before assignment
Any ideas?
This is nothing special about import statements. It's just how the scoping works in Python. If you're assigning a value to a label, it is local to the scope unless explicitly defined global.
Try this code -
a = 2
def start():
print a
a = 3
start()
This also fails with UnboundLocalError as your code because statement a = 3 makes the label a local to function start.
I am trying to load the function in a remote environment using cPickle. But I got the
error "the 'module' object has no attribute ..." . Where I really stuck is the namespace has
already contain that attributes , even though it fails to load
Please Help
import inspect
import cPickle as pickle
from run import run
def get_source(func):
sourcelines = inspect.getsourcelines(func)[0]
sourcelines[0] = sourcelines[0].lstrip()
return "".join(sourcelines)
def fun(f):
return f()
def fun1():
return 10
funcs = (fun, fun1)
sources = [get_source(func) for func in funcs]
funcs_serialized = pickle.dumps((fun.func_name,sources),0)
args_serialized = pickle.dumps(fun1,0)
#Creating the Environment where fun & fun1 doesnot exist
del globals()['fun']
del globals()['fun1']
r = run()
r.work(funcs_serialized,args_serialized)
Here is run.py
import cPickle as pickle
class run():
def __init__(self):
pass
def work(self,funcs_serialized,args_serialized):
func, fsources = pickle.loads(funcs_serialized)
fobjs = [compile(fsource, '<string>', 'exec') for fsource in fsources]
#After eval fun and fun1 should be there in globals/locals
for fobj in fobjs:
try:
eval(fobj)
globals().update(locals())
except:
pass
print "Fun1 in Globals: ",globals()['fun1']
print "Fun1 in locals: ",locals()['fun1']
arg = pickle.loads(args_serialized)
The error is
Fun1 in Globals: <function fun1 at 0xb7dae6f4>
Fun1 in locals: <function fun1 at 0xb7dae6f4>
Traceback (most recent call last):
File "fun.py", line 32, in <module>
r.work(funcs_serialized,args_serialized)
File "/home/guest/kathi/python/workspace/run.py", line 23, in work
arg = pickle.loads(args_serialized)
AttributeError: 'module' object has no attribute 'fun1'
I found this link helpful:
http://stefaanlippens.net/python-pickling-and-dealing-with-attributeerror-module-object-has-no-attribute-thing.html
It gives two solutions. The better solution is to add to the head of the loading module (or __main__):
from myclassmodule import MyClass
But I think a better solution should exist.
From http://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled:
Note that functions (built-in and
user-defined) are pickled by “fully
qualified” name reference, not by
value. This means that only the
function name is pickled, along with
the name of module the function is
defined in. Neither the function’s
code, nor any of its function
attributes are pickled. Thus the
defining module must be importable in
the unpickling environment, and the
module must contain the named object,
otherwise an exception will be raised.
You deleted the reference to fun1 in the module that defines fun1, thus the error.
The module name of the function is saved into the pickle, when you are doing the loads it is looking for fun1 in __main__ or whereever it was originally
try to add
from your_first_module import fun,fun1
into run.py
I'm trying to find out how much time it takes to execute a Python statement, so I looked online and found that the standard library provides a module called timeit that purports to do exactly that:
import timeit
def foo():
# ... contains code I want to time ...
def dotime():
t = timeit.Timer("foo()")
time = t.timeit(1)
print "took %fs\n" % (time,)
dotime()
However, this produces an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in dotime
File "/usr/local/lib/python2.6/timeit.py", line 193, in timeit
timing = self.inner(it, self.timer)
File "<timeit-src>", line 6, in inner
NameError: global name 'foo' is not defined
I'm still new to Python and I don't fully understand all the scoping issues it has, but I don't know why this snippet doesn't work. Any thoughts?
Change this line:
t = timeit.Timer("foo()")
To this:
t = timeit.Timer("foo()", "from __main__ import foo")
Check out the link you provided at the very bottom.
To give the timeit module access to functions you define, you can pass a setup parameter which contains an import statement:
I just tested it on my machine and it worked with the changes.
With Python 3, you can use globals=globals()
t = timeit.Timer("foo()", globals=globals())
From the documentation:
Another option is to pass globals() to the globals parameter, which
will cause the code to be executed within your current global
namespace. This can be more convenient than individually specifying
imports
You can try this hack:
import timeit
def foo():
print 'bar'
def dotime():
t = timeit.Timer("foo()")
time = t.timeit(1)
print "took %fs\n" % (time,)
import __builtin__
__builtin__.__dict__.update(locals())
dotime()
t = timeit.Timer("foo()", "from __main__ import foo")
Since timeit doesn't have your stuff in scope.
add into your setup "import thisfile; "
then when you call the setup function myfunc() use "thisfile.myfunc()"
eg "thisfile.py"
def myfunc():
return 5
def testable(par):
pass
t=timeit.timeit(stmt="testable(v)",setup="import thisfile; v=thisfile.myfunc();").repeat(10)
print( t )