We Keep Coding

Update locals from inside a function

I would like to write a function which receives a local namespace dictionary and update it. Something like this:
def UpdateLocals(local_dict):
d = {'a':10, 'b':20, 'c':30}
local_dict.update(d)
When I call this function from the interactive python shell it works all right, like this:
a = 1
UpdateLocals(locals())
# prints 20
print a
However, when I call UpdateLocals from inside a function, it doesn't do what I expect:
def TestUpdateLocals():
a = 1
UpdateLocals(locals())
print a
# prints 1
TestUpdateLocals()
How can I make the second case work like the first?
UPDATE:
Aswin's explanation makes sense and is very helpful to me. However I still want a mechanism to update the local variables. Before I figure out a less ugly approach, I'm going to do the following:
def LoadDictionary():
return {'a': 10, 'b': 20, 'c': 30}
def TestUpdateLocals():
a = 1
for name, value in LoadDictionary().iteritems():
exec('%s = value' % name)
Of course the construction of the string statements can be automated, and the details can be hidden from the user.

You have asked a very good question. In fact, the ability to update local variables is very important and crucial in saving and loading datasets for machine learning or in games. However, most developers of Python language have not come to a realization of its importance. They focus too much on conformity and optimization which is nevertheless important too.
Imagine you are developing a game or running a deep neural network (DNN), if all local variables are serializable, saving the entire game or DNN can be simply put into one line as print(locals()), and loading the entire game or DNN can be simply put into one line as locals().update(eval(sys.stdin.read())).
Currently, globals().update(...) takes immediate effect but locals().update(...) does not work because Python documentation says:
The default locals act as described for function locals() below:
modifications to the default locals dictionary should not be
attempted. Pass an explicit locals dictionary if you need to see
effects of the code on locals after function exec() returns.
Why they design Python in such way is because of optimization and conforming the exec statement into a function:
To modify the locals of a function on the fly is not possible without
several consequences: normally, function locals are not stored in a
dictionary, but an array, whose indices are determined at compile time
from the known locales. This collides at least with new locals added
by exec. The old exec statement circumvented this, because the
compiler knew that if an exec without globals/locals args occurred in
a function, that namespace would be "unoptimized", i.e. not using the
locals array. Since exec() is now a normal function, the compiler does
not know what "exec" may be bound to, and therefore can not treat is
specially.
Since global().update(...) works, the following piece of code will work in root namespace (i.e., outside any function) because locals() is the same as globals() in root namespace:
locals().update({'a':3, 'b':4})
print(a, b)
But this will not work inside a function.
However, as hacker-level Python programmers, we can use sys._getframe(1).f_locals instead of locals(). From what I have tested so far, on Python 3, the following piece of code always works:
def f1():
sys._getframe(1).f_locals.update({'a':3, 'b':4})
print(a, b)
f1()
However, sys._getframe(1).f_locals does not work in root namespace.

The locals are not updated here because, in the first case, the variable declared has a global scope. But when declared inside a function, the variable loses scope outside it.
Thus, the original value of the locals() is not changed in the UpdateLocals function.
PS: This might not be related to your question, but using camel case is not a good practice in Python. Try using the other method.
update_locals() instead of UpdateLocals()
Edit To answer the question in your comment:
There is something called a System Stack. The main job of this system stack during the execution of a code is to manage local variables, make sure the control returns to the correct statement after the completion of execution of the called function etc.,
So, everytime a function call is made, a new entry is created in that stack,
which contains the line number (or instruction number) to which the control has to return after the return statement, and a set of fresh local variables.
The local variables when the control is inside the function, will be taken from the stack entry. Thus, the set of locals in both the functions are not the same. The entry in the stack is popped when the control exits from the function. Thus, the changes you made inside the function are erased, unless and until those variables have a global scope.

Reason for globals() in Python?

What is the reason of having globals() function in Python? It only returns dictionary of global variables, which are already global, so they can be used anywhere... I'm asking only out of curiosity, trying to learn python.
def F():
global x
x = 1
def G():
print(globals()["x"]) #will return value of global 'x', which is 1
def H():
print(x) #will also return value of global 'x', which, also, is 1
F()
G()
H()
I can't really see the point here? Only time I would need it, was if I had local and global variables, with same name for both of them
def F():
global x
x = 1
def G():
x = 5
print(x) #5
print(globals()["x"]) #1
F()
G()
But you should never run into a problem of having two variables with same name, and needing to use them both within same scope.

Python gives the programmer a large number of tools for introspecting the running environment. globals() is just one of those, and it can be very useful in a debugging session to see what objects the global scope actually contains.
The rationale behind it, I'm sure, is the same as that of using locals() to see the variables defined in a function, or using dir to see the contents of a module, or the attributes of an object.
Coming from a C++ background, I can understand that these things seem unnecessary. In a statically linked, statically typed environment, they absolutely would be. In that case, it is known at compile time exactly what variables are global, and what members an object will have, and even what names are exported by another compilation unit.
In a dynamic language, however, these things are not fixed; they can change depending on how code is imported, or even during run time. For that reason at least, having access to this sort of information in a debugger can be invaluable.

It's also useful when you need to call a function using function's string name. For example:
def foo():
pass
function_name_as_string = 'foo'
globals()[function_name_as_string]() # foo().

You can pass the result of globals() and locals() to the eval, execfile and __import__ commands. Doing so creates a restricted environment for those commands to work in.
Thus, these functions exist to support other functions that benefit from being given an environment potentially different from the current context. You could, for example, call globals() then remove or add some variables before calling one of those functions.

globals() is useful for eval() -- if you want to evaluate some code that refers to variables in scope, those variables will either be in globals or locals.
To expand a bit, the eval() builtin function will interpret a string of Python code given to it. The signature is: eval(codeString, globals, locals), and you would use it like so:
def foo():
x = 2
y = eval("x + 1", globals(), locals())
print("y=" + y) # should be 3
This works, because the interpreter gets the value of x from the locals() dict of variables. You can of course supply your own dict of variables to eval.

It can be useful in 'declarative python'. For instance, in the below FooDef and BarDef are classes used to define a series of data structures which are then used by some package as its input, or its configuration. This allows you a lot of flexibility in what your input is, and you don't need to write a parser.
# FooDef, BarDef are classes
Foo_one = FooDef("This one", opt1 = False, valence = 3 )
Foo_two = FooDef("The other one", valence = 6, parent = Foo_one )
namelist = []
for i in range(6):
namelist.append("nm%03d"%i)
Foo_other = FooDef("a third one", string_list = namelist )
Bar_thing = BarDef( (Foo_one, Foo_two), method = 'depth-first')
Note that this configuration file uses a loop to build up a list of names which are part of the configuration of Foo_other. So, this configuration language comes with a very powerful 'preprocessor', with an available run-time library. In case you want to, say, find a complex log, or extract things from a zip file and base64 decode them, as part of generating your configuration (this approach is not recommended, of course, for cases where the input may be from an untrusted source...)
The package reads the configuration using something like the following:
conf_globals = {} # make a namespace
# Give the config file the classes it needs
conf_globals['FooDef']= mypkgconfig.FooDef # both of these are based ...
conf_globals['BarDef']= mypkgconfig.BarDef # ... on .DefBase
fname = "user.conf"
try:
exec open(fname) in conf_globals
except Exception:
...as needed...
# now find all the definitions in there
# (I'm assuming the names they are defined with are
# significant to interpreting the data; so they
# are stored under those keys here).
defs = {}
for nm,val in conf_globals.items():
if isinstance(val,mypkgconfig.DefBase):
defs[nm] = val
So, finally getting to the point, globals() is useful, when using such a package, if you want to mint a series of definitions procedurally:
for idx in range(20):
varname = "Foo_%02d" % i
globals()[varname]= FooDef("one of several", id_code = i+1, scale_ratio = 2**i)
This is equivalent to writing out
Foo_00 = FooDef("one of several", id_code = 1, scale_ratio=1)
Foo_01 = FooDef("one of several", id_code = 2, scale_ratio=2)
Foo_02 = FooDef("one of several", id_code = 3, scale_ratio=4)
... 17 more ...
An example of a package which obtains its input by gathering a bunch of definitions from a python module is PLY (Python-lex-yacc) http://www.dabeaz.com/ply/ -- in that case the objects are mostly function objects, but metadata from the function objects (their names, docstrings, and order of definition) also form part of the input. It's not such a good example for use of globals() . Also, it is imported by the 'configuration' - the latter being a normal python script -- rather than the other way around.
I've used 'declarative python' on a few projects I've worked on, and have had occasion to use globals() when writing configurations for those. You could certainly argue that this was due to a weakness in the way the configuration 'language' was designed. Use of globals() in this way doesn't produce very clear results; just results which might be easier to maintain than writing out a dozen nearly-identical statements.
You can also use it to give variables significance within the configuration file, according to their names:
# All variables above here starting with Foo_k_ are collected
# in Bar_klist
#
foo_k = [ v for k,v in globals().items() if k.startswith('Foo_k_')]
Bar_klist = BarDef( foo_k , method = "kset")
This method could be useful for any python module that defines a lot of tables and structures, to make it easier to add items to the data, without having to maintain the references as well.

It can also be used to get an instance of the class 'classname' from a
string:
class C:
def __init__(self, x):
self.x = x
print('Added new instance, x:', self.x)
def call(str):
obj = globals()[str](4)
return obj
c = call('C')
print(c.x)

It might be useful if you like to import module you just have built:
a.py
[...]
def buildModule():
[...code to build module...]
return __import__("somemodule")
[...]
b.py
from a import buildModule
def setup():
globals()["somemodule"] = buildModule()

Not really. Global variables Python really has are module-scoped variables.
# a.py
print(globals())
import b
b.tt()
# b.py
def tt():
print(globals())
run python a.py, at least two output of globals()['__name__'] is different.
Code here in cpython on Github shows it.

I did not notice in answers anything about using globals() to check if you have value set. Maybe you only set value if debugging or have forgotten to set one and want to avoid getting exception. Though locals() might be better solution in some of the cases to avoid accessing global scope and to access local scope only.
# DEBUG = True
if 'DEBUG' in globals():
print(f'We found debug flag and it has value {DEBUG}.')
else:
print(f'Debug flag was not found.')
Also you can use it with combination with get() to set the default value in case variable was not found
# VARIABLE = "Value of var"
my_var = globals().get("VARIABLE", "Value was not found")
print(my_var) # Prints: "Value was not found"
print(VARIABLE) # Raises NameError
VARIABLE = "Value of var"
my_var = globals().get("VARIABLE", "Value was not found")
print(my_var) # prints: "Value of var"
print(VARIABLE) # prints: "Value of var"

Create a new variable as named from input in Python?

This is something that I've been questioning for some time. How would I create a variable at runtime as named by the value of another variable. So, for example, the code would ask the user to input a string. A variable would then be created named after that string with a default value of "default". Is this even possible?

It is possible, but it's certainly not advised. You can access the global namespace as a dict (it's a dict internally) and add entries to it.
If you were doing an interactive interpreter, say, for doing maths, or something. You would actually pass a dict to each eval() or exec that you could then re-use as it's local namespace.
As a quick, bad, example, don't do this at home:
g = globals() # get a reference to the globals dict
g[raw_input("Name Please")] = raw_input("Value Please")
print foo
Run that, it'll traceback unless you provide 'foo' to the first prompt.

exec code in original local scope of frame

I've written a remote Python debugger and one of the features I need is to execute arbitrary code while stopped at a breakpoint. My debugger uses the following to execute code received from the remote debugger:
exec (compile(code, '<string>', 'single') , frame.f_globals, frame.f_locals)
This works fine for the most part, but I've noticed a couple issues.
Assignment statements aren't actually applied to the original locals dictionary. This is probably due to the fact that f_locals is supposed to be read-only.
If stopped within a class method, accessing protected attributes (names beginning with double underscore) does not work. I'm assuming this is due to the name mangling that Python performs on protected attributes.
So my question is, is there a way around these limitations? Can I trick Python into thinking that the code is being executed in the actual local scope of that frame?
I'm using CPython 2.7, and I'm willing to accept a solution/hack specific to this version.

Assignment statements aren't actually
applied to the original locals
dictionary. This is probably due to
the fact that f_locals is supposed to
be read-only.
Not exactly, but the bytecode for the function will not look at locals, using rather a simple but crucial optimization whereby local variables are in a simple array, avoiding runtime lookups. The only way to avoid this (and make the function much, much slower) is compiling different code, e.g. code starting with an exec '' to force the compiler to avoid the optimization (in Python 2; no way, in Python 3). If you need to work with existing bytecode, you're out of luck: there is no way to accomplish what you desire.
If stopped within a class method,
accessing protected attributes (names
beginning with double underscore) does
not work. I'm assuming this is due to
the name mangling that Python performs
on protected attributes.
Yep, so this issue does allow a workaround: prepend _Classname to the name to mimic what the compiler does. Note that double-underscore prefixes means private: protected would be a single underscore (and would give you no trouble). Private names are specifically meant to avoid accidental classes with names bound in subclasses (and work decently for that one purpose, though not perfectly, and not for anything else;-).

I'm not sure I've understood you correctly, but exec does populate the locals parameter with assignments inside the code:
>>> loc = {}
>>> exec(compile('a=3', '<string>', 'single'), {}, loc)
>>> loc
{'a': 3}
Perhaps f_locals doesn't allow writes.

to execute arbitrary code while stopped at a breakpoint ... Can I trick Python into thinking that the code is being executed in the actual local scope of that frame?
The Python debugger, pdb, allows this. For example, let's say you are debugging the file tests/scopeTest.py, and you have the following line in your program, where the variable hasn't been declared in the program itself :
print (NOT_DEFINED_IN_PROGRAM)
so that running the code python tests/scopeTest.py would result in :
NameError: name 'NOT_DEFINED_IN_PROGRAM' is not defined
Now you would like to define that variable when stopped at that line in the debugger, and have the program continue executing, using that variable as if it had been defined in the program all along. In other words, you would like to effect the change within that scope, so that you can continue execution with that change permanent. It is actually possible :
$ python -m pdb tests/scopeTest.py
> /home/user/tests/scopeTest.py(1)<module>()
-> print (NOT_DEFINED_IN_PROGRAM)
(Pdb) 'NOT_DEFINED_IN_PROGRAM' in locals()
False
(Pdb) NOT_DEFINED_IN_PROGRAM = 5
(Pdb) 'NOT_DEFINED_IN_PROGRAM' in locals()
True
(Pdb) step
5
Pdb does this through a compile and exec in its default function, which does the equivalent of :
code = compile(line + '\n', <stdin>, 'single')
exec(code, self.curframe.f_globals, self.curframe_locals)
where self.curframe is a specific frame. Now, self.curframe_locals is not self.curframe.f_locals, because, as the setup function says :
# The f_locals dictionary is updated from the actual frame
# locals whenever the .f_locals accessor is called, so we
# cache it here to ensure that modifications are not overwritten.
self.curframe_locals = self.curframe.f_locals
Hope that helps, and is what you meant!
Take note that, even then, should you want to, for example, replace a function in the context of the program being debugged with a monkey-patched version, such as:
newGlobals['abs'] = myCustomAbsFunction
exec(code, newGlobals, locals)
the scope of the myCustomAbsFunction is not going to be the user program, but is going to be the context of where that function was defined, which is the debugger! There is a way around that too, but as it wasn't specifically asked, it is left as an exercise for the reader, for now. ^__^

How can one create new scopes in python

In many languages (and places) there is a nice practice of creating local scopes by creating a block like this.
void foo()
{
... Do some stuff ...
if(TRUE)
{
char a;
int b;
... Do some more stuff ...
}
... Do even more stuff ...
}
How can I implement this in python without getting the unexpected indent error and without using some sort of if True: tricks

Why do you want to create new scopes in python anyway?
The normal reason for doing it in other languages is variable scoping, but that doesn't happen in python.
if True:
a = 10
print a

In Python, scoping is of three types : global, local and class. You can create specialized 'scope' dictionaries to pass to exec / eval(). In addition you can use nested scopes
(defining a function within another). I found these to be sufficient in all my code.
As Douglas Leeder said already, the main reason to use it in other languages is variable scoping and that doesn't really happen in Python. In addition, Python is the most readable language I have ever used. It would go against the grain of readability to do something like if-true tricks (Which you say you want to avoid). In that case, I think the best bet is to refactor your code into multiple functions, or use a single scope. I think that the available scopes in Python are sufficient to cover every eventuality, so local scoping shouldn't really be necessary.

If you just want to create temp variables and let them be garbage collected right after using them, you can use
del varname
when you don't want them anymore.
If its just for aesthetics, you could use comments or extra newlines, no extra indentation, though.

Python has exactly two scopes, local and global. Variables that are used in a function are in local scope no matter what indentation level they were created at. Calling a nested function will have the effect that you're looking for.
def foo():
a = 1
def bar():
b = 2
print a, b #will print "1 2"
bar()
Still like everyone else, I have to ask you why you want to create a limited scope inside a function.

variables in list comprehension (Python 3+) and generators are local:
>>> i = 0
>>> [i+1 for i in range(10)]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> i
0
but why exactly do you need this?

A scope is a textual region of a
Python program where a namespace is
directly accessible. “Directly
accessible” here means that an
unqualified reference to a name
attempts to find the name in the
namespace...
Please, read the documentation and clarify your question.
btw, you don't need if(TRUE){} in C, a simple {} is sufficient.

As mentioned in the other answers, there is no analogous functionality in Python to creating a new scope with a block, but when writing a script or a Jupyter Notebook, I often (ab)use classes to introduce new namespaces for similar effect. For example, in a notebook where you might have a model "Foo", "Bar" etc. and related variables you might want to create a new scope to avoid having to reuse names like
model = FooModel()
optimizer = FooOptimizer()
...
model = BarModel()
optimizer = BarOptimizer()
or suffix names like
model_foo = ...
optimizer_foo = ...
model_bar = ...
optimizer_bar= ...
Instead you can introduce new namespaces with
class Foo:
model = ...
optimizer = ...
loss = ....
class Bar:
model = ...
optimizer = ...
loss = ...
and then access the variables as
Foo.model
Bar.optimizer
...
I find that using namespaces this way to create new scopes makes code more readable and less error-prone.

While the leaking scope is indeed a feature that is often useful,
I have created a package to simulate block scoping (with selective leaking of your choice, typically to get the results out) anyway.
from scoping import scoping
a = 2
with scoping():
assert(2 == a)
a = 3
b = 4
scoping.keep('b')
assert(3 == a)
assert(2 == a)
assert(4 == b)
https://pypi.org/project/scoping/

I would see this as a clear sign that it's time to create a new function and refactor the code. I can see no reason to create a new scope like that. Any reason in mind?

def a():
def b():
pass
b()
If I just want some extra indentation or am debugging, I'll use if True:

Like so, for arbitrary name t:
### at top of function / script / outer scope (maybe just big jupyter cell)
try: t
except NameError:
class t
pass
else:
raise NameError('please `del t` first')
#### Cut here -- you only need 1x of the above -- example usage below ###
t.tempone = 5 # make new temporary variable that definitely doesn't bother anything else.
# block of calls here...
t.temptwo = 'bar' # another one...
del t.tempone # you can have overlapping scopes this way
# more calls
t.tempthree = t.temptwo; del t.temptwo # done with that now too
print(t.tempthree)
# etc, etc -- any number of variables will fit into t.
### At end of outer scope, to return `t` to being 'unused'
del t
All the above could be in a function def, or just anyplace outside defs along a script.
You can add or del new elements to an arbitrary-named class like that at any point. You really only need one of these -- then manage your 'temporary' namespace as you like.
The del t statement isn't necessary if this is in a function body, but if you include it, then you can copy/paste chunks of code far apart from each other and have them work how you expect (with different uses of 't' being entirely separate, each use starting with the that try: t... block, and ending with del t).
This way if t had been used as a variable already, you'll find out, and it doesn't clobber t so you can find out what it was.
This is less error prone then using a series of random=named functions just to call them once -- since it avoids having to deal with their names, or remembering to call them after their definition, especially if you have to reorder long code.
This basically does exactly what you want: Make a temporary place to put things you know for sure won't collide with anything else, and which you are responsible for cleaning up inside as you go.
Yes, it's ugly, and probably discouraged -- you will be directed to decompose your work into a set of smaller, more reusable functions.

As others have suggested, the python way to execute code without polluting the enclosing namespace is to put it in a class or function. This presents a slight and usually harmless problem: defining the function puts its name in the enclosing namespace. If this causes harm to you, you can name your function using Python's conventional temporary variable "_":
def _():
polluting_variable = foo()
...
_() # Run the code before something overwrites the variable.
This can be done recursively as each local definition masks the definition from the enclosing scope.
This sort of thing should only be needed in very specific circumstances. An example where it is useful is when using Databricks' %run magic, which executes the contents of another notebook in the current notebook's global scope. Wrapping the child notebook's commands in temporary functions prevents them from polluting the global namespace.