Python's locals() documentation says:
Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks.
What are in the current local symbol table exactly?
Is the local symbol table guaranteed to be the same as the argument dict if locals() is called at the very beginning of a function?
For example, if we have the following program:
def foo(x, y):
print(locals() == {'x': 1, 'y': 2})
foo(1, 2)
Will it always output True no matter what platforms and Python implementations we use?
What are in the current local symbol table exactly?
Well, you can take a look yourself. At the very beginging of your Python, program, here is what the symbol table returned by locals() looks like:
print(locals())
Which outputs:
{'__doc__': None, '__spec__': None, '__package__': None, '__builtins__':
<module 'builtins' (built-in)>, '__name__': '__main__', '__loader__':
<class '_frozen_importlib.BuiltinImporter'>,
'__file__': 'C:\\Users\\$Name$\\Desktop\\script.py'}
The symbol table consists of a few "magic variables", and some info about your current Python file. Such as the __file__ key, which contains the name of your current source file. The description of what locals returns pretty much matches the definition for a symbol table:
In computer science, a symbol table is a data structure used by a language translator such as a compiler or interpreter, where each identifier in a program's source code is associated with information relating to its declaration or appearance in the source. - Wikipedia: Symbol table
(emphasis mine)
Is the local symbol table guaranteed to be the same as the argument dict if locals() is called at the very beginning of a function?
To which the answer would 1yes. Functions have their own scope. And, as hinted at by the name, locals() only returns identifiers local to the current scope. So a call to locals() inside of a function, could not be changed by the outer scope of a program. eg.
>>> var = 10 # global variable
>>> locals()['var'] # var is accessible in the current scope
10
>>> def func():
print(locals()['var']) # but not in this scope. Python will raise an error
>>> func()
Traceback (most recent call last):
File "<pyshell#17>", line 1, in <module>
func()
File "<pyshell#16>", line 2, in func
print(locals()['var'])
KeyError: 'var'
>>>
1
Its a little hard to fully understand what your asking in your second question, so I apologize if my answer is not related. But I believe your asking: If I call locals() at the beginning of my function, inside of my definition, is the dict() returned by locals guaranteed to stay the same?. If this isn't the case, update your question and I'll try to re-answer.
I've been looking at dynamic evaluation of Python code, and come across the eval() and compile() functions, and the exec statement.
Can someone please explain the difference between eval and exec, and how the different modes of compile() fit in?
The short answer, or TL;DR
Basically, eval is used to evaluate a single dynamically generated Python expression, and exec is used to execute dynamically generated Python code only for its side effects.
eval and exec have these two differences:
eval accepts only a single expression, exec can take a code block that has Python statements: loops, try: except:, class and function/method definitions and so on.
An expression in Python is whatever you can have as the value in a variable assignment:
a_variable = (anything you can put within these parentheses is an expression)
eval returns the value of the given expression, whereas exec ignores the return value from its code, and always returns None (in Python 2 it is a statement and cannot be used as an expression, so it really does not return anything).
In versions 1.0 - 2.7, exec was a statement, because CPython needed to produce a different kind of code object for functions that used exec for its side effects inside the function.
In Python 3, exec is a function; its use has no effect on the compiled bytecode of the function where it is used.
Thus basically:
>>> a = 5
>>> eval('37 + a') # it is an expression
42
>>> exec('37 + a') # it is an expression statement; value is ignored (None is returned)
>>> exec('a = 47') # modify a global variable as a side effect
>>> a
47
>>> eval('a = 47') # you cannot evaluate a statement
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1
a = 47
^
SyntaxError: invalid syntax
The compile in 'exec' mode compiles any number of statements into a bytecode that implicitly always returns None, whereas in 'eval' mode it compiles a single expression into bytecode that returns the value of that expression.
>>> eval(compile('42', '<string>', 'exec')) # code returns None
>>> eval(compile('42', '<string>', 'eval')) # code returns 42
42
>>> exec(compile('42', '<string>', 'eval')) # code returns 42,
>>> # but ignored by exec
In the 'eval' mode (and thus with the eval function if a string is passed in), the compile raises an exception if the source code contains statements or anything else beyond a single expression:
>>> compile('for i in range(3): print(i)', '<string>', 'eval')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1
for i in range(3): print(i)
^
SyntaxError: invalid syntax
Actually the statement "eval accepts only a single expression" applies only when a string (which contains Python source code) is passed to eval. Then it is internally compiled to bytecode using compile(source, '<string>', 'eval') This is where the difference really comes from.
If a code object (which contains Python bytecode) is passed to exec or eval, they behave identically, excepting for the fact that exec ignores the return value, still returning None always. So it is possible use eval to execute something that has statements, if you just compiled it into bytecode before instead of passing it as a string:
>>> eval(compile('if 1: print("Hello")', '<string>', 'exec'))
Hello
>>>
works without problems, even though the compiled code contains statements. It still returns None, because that is the return value of the code object returned from compile.
In the 'eval' mode (and thus with the eval function if a string is passed in), the compile raises an exception if the source code contains statements or anything else beyond a single expression:
>>> compile('for i in range(3): print(i)', '<string>'. 'eval')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1
for i in range(3): print(i)
^
SyntaxError: invalid syntax
The longer answer, a.k.a the gory details
exec and eval
The exec function (which was a statement in Python 2) is used for executing a dynamically created statement or program:
>>> program = '''
for i in range(3):
print("Python is cool")
'''
>>> exec(program)
Python is cool
Python is cool
Python is cool
>>>
The eval function does the same for a single expression, and returns the value of the expression:
>>> a = 2
>>> my_calculation = '42 * a'
>>> result = eval(my_calculation)
>>> result
84
exec and eval both accept the program/expression to be run either as a str, unicode or bytes object containing source code, or as a code object which contains Python bytecode.
If a str/unicode/bytes containing source code was passed to exec, it behaves equivalently to:
exec(compile(source, '<string>', 'exec'))
and eval similarly behaves equivalent to:
eval(compile(source, '<string>', 'eval'))
Since all expressions can be used as statements in Python (these are called the Expr nodes in the Python abstract grammar; the opposite is not true), you can always use exec if you do not need the return value. That is to say, you can use either eval('my_func(42)') or exec('my_func(42)'), the difference being that eval returns the value returned by my_func, and exec discards it:
>>> def my_func(arg):
... print("Called with %d" % arg)
... return arg * 2
...
>>> exec('my_func(42)')
Called with 42
>>> eval('my_func(42)')
Called with 42
84
>>>
Of the 2, only exec accepts source code that contains statements, like def, for, while, import, or class, the assignment statement (a.k.a a = 42), or entire programs:
>>> exec('for i in range(3): print(i)')
0
1
2
>>> eval('for i in range(3): print(i)')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1
for i in range(3): print(i)
^
SyntaxError: invalid syntax
Both exec and eval accept 2 additional positional arguments - globals and locals - which are the global and local variable scopes that the code sees. These default to the globals() and locals() within the scope that called exec or eval, but any dictionary can be used for globals and any mapping for locals (including dict of course). These can be used not only to restrict/modify the variables that the code sees, but are often also used for capturing the variables that the executed code creates:
>>> g = dict()
>>> l = dict()
>>> exec('global a; a, b = 123, 42', g, l)
>>> g['a']
123
>>> l
{'b': 42}
(If you display the value of the entire g, it would be much longer, because exec and eval add the built-ins module as __builtins__ to the globals automatically if it is missing).
In Python 2, the official syntax for the exec statement is actually exec code in globals, locals, as in
>>> exec 'global a; a, b = 123, 42' in g, l
However the alternate syntax exec(code, globals, locals) has always been accepted too (see below).
compile
The compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1) built-in can be used to speed up repeated invocations of the same code with exec or eval by compiling the source into a code object beforehand. The mode parameter controls the kind of code fragment the compile function accepts and the kind of bytecode it produces. The choices are 'eval', 'exec' and 'single':
'eval' mode expects a single expression, and will produce bytecode that when run will return the value of that expression:
>>> dis.dis(compile('a + b', '<string>', 'eval'))
1 0 LOAD_NAME 0 (a)
3 LOAD_NAME 1 (b)
6 BINARY_ADD
7 RETURN_VALUE
'exec' accepts any kinds of python constructs from single expressions to whole modules of code, and executes them as if they were module top-level statements. The code object returns None:
>>> dis.dis(compile('a + b', '<string>', 'exec'))
1 0 LOAD_NAME 0 (a)
3 LOAD_NAME 1 (b)
6 BINARY_ADD
7 POP_TOP <- discard result
8 LOAD_CONST 0 (None) <- load None on stack
11 RETURN_VALUE <- return top of stack
'single' is a limited form of 'exec' which accepts a source code containing a single statement (or multiple statements separated by ;) if the last statement is an expression statement, the resulting bytecode also prints the repr of the value of that expression to the standard output(!).
An if-elif-else chain, a loop with else, and try with its except, else and finally blocks is considered a single statement.
A source fragment containing 2 top-level statements is an error for the 'single', except in Python 2 there is a bug that sometimes allows multiple toplevel statements in the code; only the first is compiled; the rest are ignored:
In Python 2.7.8:
>>> exec(compile('a = 5\na = 6', '<string>', 'single'))
>>> a
5
And in Python 3.4.2:
>>> exec(compile('a = 5\na = 6', '<string>', 'single'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1
a = 5
^
SyntaxError: multiple statements found while compiling a single statement
This is very useful for making interactive Python shells. However, the value of the expression is not returned, even if you eval the resulting code.
Thus greatest distinction of exec and eval actually comes from the compile function and its modes.
In addition to compiling source code to bytecode, compile supports compiling abstract syntax trees (parse trees of Python code) into code objects; and source code into abstract syntax trees (the ast.parse is written in Python and just calls compile(source, filename, mode, PyCF_ONLY_AST)); these are used for example for modifying source code on the fly, and also for dynamic code creation, as it is often easier to handle the code as a tree of nodes instead of lines of text in complex cases.
While eval only allows you to evaluate a string that contains a single expression, you can eval a whole statement, or even a whole module that has been compiled into bytecode; that is, with Python 2, print is a statement, and cannot be evalled directly:
>>> eval('for i in range(3): print("Python is cool")')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1
for i in range(3): print("Python is cool")
^
SyntaxError: invalid syntax
compile it with 'exec' mode into a code object and you can eval it; the eval function will return None.
>>> code = compile('for i in range(3): print("Python is cool")',
'foo.py', 'exec')
>>> eval(code)
Python is cool
Python is cool
Python is cool
If one looks into eval and exec source code in CPython 3, this is very evident; they both call PyEval_EvalCode with same arguments, the only difference being that exec explicitly returns None.
Syntax differences of exec between Python 2 and Python 3
One of the major differences in Python 2 is that exec is a statement and eval is a built-in function (both are built-in functions in Python 3).
It is a well-known fact that the official syntax of exec in Python 2 is exec code [in globals[, locals]].
Unlike majority of the Python 2-to-3 porting guides seem to suggest, the exec statement in CPython 2 can be also used with syntax that looks exactly like the exec function invocation in Python 3. The reason is that Python 0.9.9 had the exec(code, globals, locals) built-in function! And that built-in function was replaced with exec statement somewhere before Python 1.0 release.
Since it was desirable to not break backwards compatibility with Python 0.9.9, Guido van Rossum added a compatibility hack in 1993: if the code was a tuple of length 2 or 3, and globals and locals were not passed into the exec statement otherwise, the code would be interpreted as if the 2nd and 3rd element of the tuple were the globals and locals respectively. The compatibility hack was not mentioned even in Python 1.4 documentation (the earliest available version online); and thus was not known to many writers of the porting guides and tools, until it was documented again in November 2012:
The first expression may also be a tuple of length 2 or 3. In this case, the optional parts must be omitted. The form exec(expr, globals) is equivalent to exec expr in globals, while the form exec(expr, globals, locals) is equivalent to exec expr in globals, locals. The tuple form of exec provides compatibility with Python 3, where exec is a function rather than a statement.
Yes, in CPython 2.7 that it is handily referred to as being a forward-compatibility option (why confuse people over that there is a backward compatibility option at all),
when it actually had been there for backward-compatibility for two decades.
Thus while exec is a statement in Python 1 and Python 2, and a built-in function in Python 3 and Python 0.9.9,
>>> exec("print(a)", globals(), {'a': 42})
42
has had identical behaviour in possibly every widely released Python version ever; and works in Jython 2.5.2, PyPy 2.3.1 (Python 2.7.6) and IronPython 2.6.1 too (kudos to them following the undocumented behaviour of CPython closely).
What you cannot do in Pythons 1.0 - 2.7 with its compatibility hack, is to store the return value of exec into a variable:
Python 2.7.11+ (default, Apr 17 2016, 14:00:29)
[GCC 5.3.1 20160413] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = exec('print(42)')
File "<stdin>", line 1
a = exec('print(42)')
^
SyntaxError: invalid syntax
(which wouldn't be useful in Python 3 either, as exec always returns None), or pass a reference to exec:
>>> call_later(exec, 'print(42)', delay=1000)
File "<stdin>", line 1
call_later(exec, 'print(42)', delay=1000)
^
SyntaxError: invalid syntax
Which a pattern that someone might actually have used, though unlikely;
Or use it in a list comprehension:
>>> [exec(i) for i in ['print(42)', 'print(foo)']
File "<stdin>", line 1
[exec(i) for i in ['print(42)', 'print(foo)']
^
SyntaxError: invalid syntax
which is abuse of list comprehensions (use a for loop instead!).
exec is not an expression: a statement in Python 2.x, and a function in Python 3.x. It compiles and immediately evaluates a statement or set of statement contained in a string. Example:
exec('print(5)') # prints 5.
# exec 'print 5' if you use Python 2.x, nor the exec neither the print is a function there
exec('print(5)\nprint(6)') # prints 5{newline}6.
exec('if True: print(6)') # prints 6.
exec('5') # does nothing and returns nothing.
eval is a built-in function (not a statement), which evaluates an expression and returns the value that expression produces. Example:
x = eval('5') # x <- 5
x = eval('%d + 6' % x) # x <- 11
x = eval('abs(%d)' % -100) # x <- 100
x = eval('x = 5') # INVALID; assignment is not an expression.
x = eval('if 1: x = 4') # INVALID; if is a statement, not an expression.
compile is a lower level version of exec and eval. It does not execute or evaluate your statements or expressions, but returns a code object that can do it. The modes are as follows:
compile(string, '', 'eval') returns the code object that would have been executed had you done eval(string). Note that you cannot use statements in this mode; only a (single) expression is valid.
compile(string, '', 'exec') returns the code object that would have been executed had you done exec(string). You can use any number of statements here.
compile(string, '', 'single') is like the exec mode but expects exactly one expression/statement, eg compile('a=1 if 1 else 3', 'myf', mode='single')
exec is for statement and does not return anything.
eval is for expression and returns value of expression.
expression means "something" while statement means "do something".
I'm having some trouble importing some stuff using a function, despite being able to do so in the interpreter.
Imagine there is a file, input.py, in a folder A which in turn is in the same directory as my script. In this file, we define variable 'B'.
B = 5
When I go into the interpreter, the following commands give me the correct value of B
>>> import sys
>>> sys.path.append('A')
>>> exec('from inputs import *')
>>> print(B)
Yet if I move that code to a seperate file, say 'test.py':
import sys
def import_stuff(import_dir):
sys.path.append(import_dir)
exec('from inputs import *')
print(B)
Then call it from the interpreter like so:
>>> import test
>>> test.import_stuff('A')
I get a NameError and B is not found. What's going on?
Local variables in functions are treated differently than global variables and object attributes (which both use dictionaries to map names to values).
When a function is defined, the Python compiler examines its code and makes a note of which local variable names are used, and designates a "slot" for each one. When the function is called, the slots refer to part of the memory in the frame object. Local variable assignments and lookups access the slots by number, not by name. This makes local variable lookups notably faster than global variable and attribute lookups (since indexing a slot is much faster than doing a dict lookup).
When you try to use exec to create local variables, it bypasses the slots. The compiler doesn't know what variables will be created in the exec'd code, so there are no slots allocated for them. This is also the reason Python 3 doesn't allow you to use from module import * inside a function: the names to be imported are not known at the function's compile time (only when it is run and the imported module is loaded) so the compiler can't set up slots for the names.
Even if you separately initialized local variables for the names you expect to be assigned to in the exec'd code, it still wouldn't work. The exec'd code doesn't know it's being run from within a function, and always wants to write variables to a dictionary (never to function slots). Function's do have a local namespace dictionary that does catch the assignments (they don't become global variables), but using it via the locals() function is very flaky. The values of all of the local variables stored in slots are copied into the dictionary each time you call locals(), but no copying in the other direction ever happens (modifying the dictionary returned from locals() doesn't effect the values of variables stored in slots and accessed the normal way).
That brings me to what I think is the best way to work around this issue. Rather than having exec modify the current namespace (which occurs in a flaky way if you're in a function), you should explicitly pass in a dictionary for it to use as its namespace.
def import_stuff(import_dir):
sys.path.append(import_dir)
namespace = {} # create a namespace dict
exec('from inputs import *', namespace) # pass it to exec
print(namespace["B"]) # read the results from the namespace
I'm writing Python 3 code and for some reason I want to run everything just in memory and save no files on disk. I managed to solve almost all of my problems so far by reading answers here, but I'm stuck on these lines:
>>> code = compile(source, filename, 'exec')
>>> exec code in module.__dict__
I don't really understand what the second line does, since I have "in" connected with loops and testing whether something is in some set or not which is not this case.
So, what does the second line do? And what is its Python 3 equivalent since in py3 is exec function, not keyword?
exec code in module.__dict__
means execute the commands in the file or string called 'code', taking global and local variables referred to in 'code' from module.__dict__ and storing local and global variables created in 'code' into the dictionary module.__dict__
See http://docs.python.org/reference/simple_stmts.html#exec
Eg:
In [51]: mydict={}
In [52]: exec "val1=100" in mydict
In [53]: mydict['val1']
Out[53]: 100
Eg2:
In [54]: mydict={}
In [55]: mydict['val2']=200
In [56]: exec "val1=val2" in mydict
In [57]: mydict.keys()
Out[57]: ['__builtins__', 'val2', 'val1']
In [58]: mydict['val2']
Out[58]: 200
In [59]: mydict['val1']
Out[59]: 200
The in keyword specifies a dictionary to use for the global and local namespaces. From the python 2 documentation for exec:
In all cases, if the optional parts are omitted, the code is executed in the current scope. If only the first expression after in is specified, it should be a dictionary, which will be used for both the global and the local variables.
The python 3 equivalent is thus exec(code, module.__dict__).
In Python 3 that exec line would translate to:
exec(code, module.__dict__)
Excerpts from the Python 3 help files:
exec(object[, globals[, locals]])
In all cases, if the optional parts are omitted, the code is executed
in the current scope. If only globals is provided, it must be a
dictionary, which will be used for both the global and the local
variables. If globals and locals are given, they are used for the
global and local variables, respectively. If provided, locals can be
any mapping object.
I'm having a bit of trouble understanding what's going wrong with the following function:
def ness():
pie='yum'
vars()[pie]=4
print vars()[pie]
print yum
So When I run that I get this result:
>>> ness()
4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in ness
NameError: global name 'yum' is not defined
If I don't write it as a function and just type it in on the command line one line at a time it works fine, like so:
>>> pie='yum'
>>> vars()[pie]=4
>>> print vars()[pie]
4
>>> print yum
4
>>>
Edit:
Suppose I wanted to make things a bit more complicated than this and instead of setting yum to a value and printing that value, I define some functions, and want to call one of them based on some input:
def ness(choo):
dic={}
dessert=()
dnum=[10,100]
desserts='pie'
dic[dessert]=str(desserts[bisect(dnum,choo)])
vars()[dic[dessert]]()
def p():
print 'ummmm ummm'
def i():
print 'hooo aaaaa'
def e():
print 'woooo'
So when I call ness I get a key error:
>>> ness(3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in ness
KeyError: 'p'
Now I know I can do things like this with some elif statements, but I'm wondering if this would work too, and if using bisect like this would be more efficient (say if i need to check 1000 values of choo) than using elifs.
Thanks much for the assistance.
vars() within a function gives you the local namespace, just like locals() -- see the docs. Outside of a function (e.g. at the prompt) locals() (and vars() of course) gives you the module's global namespace, just like globals(). As the docs say, trying to assign to a function's local variable through locals() (or equivalently, vars() inside a function) is not supported in Python. If you want to assign to a global variable, as you do when you're at the prompt (or otherwise outside of a function), use globals() instead of vars() (maybe not the cleanest approach -- global variables are understandably frowned upon -- but it does work).
There is way to do it with exec
>>> def ness():
... pie='yum'
... exec pie+"=4"
... print vars()[pie]
... print yum
...
>>>
>>> ness()
4
4
But Instead of doing that, using a new dict is better and safe
>>> def ness():
... dic={}
... pie='yum'
... dic[pie]=4
... print dic[pie]
... print dic['yum']
...
>>> ness()
4
4
>>>
It's not safe to modify the dict returned by vars()
vars([object])ΒΆ
Without an argument, act like locals().
With a module, class or class instance object as argument (or
anything else that has a dict
attribute), return that attribute.
Note
The returned dictionary should not be modified: the effects on the
corresponding symbol table are
undefined.
Your second example is a special case. vars() is equivalent to globals() in the global namespace, and the dict returned by globals() behaves as you would expect ( but is frowned upon )
>>> id(vars()),id(globals())
(3085426868L, 3085426868L)
vars() is equivalent to locals(), which in the case of the function is the local variables in its scope and at in the interactive interpreter at the scope you have it, vars() is globals(). locals() is for reading only; the effects of trying to change it are undefined (and in practice, just doesn't work). globals() can be modified, but you still should never directly put anything in the dict it returns.
[Edit: I must be wrong here, since the 'exec' example works.]
As everyone points out, it's a bad idea to modify vars(). You can understand the error, though, by realizing that python in some sense doesn't "see" that "yum" is a local. "print yum" is still resolved as a global reference; this happens before any code is executed.
It's the same reason you get an UnboundLocalError from:
>>> y = 100
>>> def foo(x):
... if x == 1:
... y = 10
... print y
...
>>> foo(1)
10
>>> foo(2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in foo
UnboundLocalError: local variable 'y' referenced before assignment