This question already has answers here:
Dynamically set local variable [duplicate]
(7 answers)
Any way to modify locals dictionary?
(5 answers)
Closed 2 years ago.
Consider below code:-
dct = {'one':[2,3]}
Now the below works,
for key,val in dct.items():
locals()[key] = val
print(one)
Result:
[2, 3]
But When I use function which I really want doesn't work. Please help
def ff(dct):
for key,val in dct.items():
locals()[key] = val
print(one)
ff(dct)
Result:
NameError: name 'one' is not defined
The result of the locals() call inside a function is not one that you can use to actually update locals using your method. The documentation makes that clear:
locals()
Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks. Note that at the module level, locals() and globals() are the same dictionary.
Note The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter.
The reason why you cannot update locals using this method is that they are heavily optimised within the CPython source code(a). When you call locals(), it actually builds a dictionary (the "update and return" bit) based on these heavily optimised structures and that is what you get.
Writing a new key to that dictionary is not reflected back to the actual locals.
I believe the globals() return value does allow updates using this method because that's already a dictionary and it just gives you a reference to it, not a copy. That's why your code works outside of a function (see the clause above stating locals() and globals() are the same thing in that context).
From memory, Python 2 allowed you to do something like exec "answer = 42" and that would affect the locals. But, as with print, this was changed from a statement to a library call in Python 3, so the execution engine really has no idea what it does under the covers, meaning it cant magically bind locals with exec("answer = 42").
I suppose someone could request this as a feature since it would make dynamic programming a little easier. Whether it would get through the gauntlet, I have no idea, since the fact that you can provide your own locals dictionary to exec() means that you already have a way to detect variables that were bound by arbitrary code. They'll just be in a separate dictionary rather then in the actual locals area.
(a) Access to locals is via known indexes into the stack-frame local variable area, computed at compile time and embedded into the actual bypte code. Being able to add/change variables dynamically would break this optimisation.
This problem was raised as an issue back in early 2009 and the outcome was that it was too difficult without losing a lot of performance.
Related
This question already has answers here:
How do I pass a variable by reference?
(39 answers)
Closed 2 years ago.
I have a code as below,
attractions = [[] for i in range(5)]
def add_attraction(attraction):
attractions_for_destination = attractions[2]
attractions_for_destination.append(attraction)
return
pass by reference occurs in 3rd line "attractions_for_destination = attractions[2]". The general idea of pass by reference that I have is that it happens between actual parameter and formal parameter. But the above line is a simple assignment code. Can pass by reference happen in a simple assignment code?
If so, if I write
attractions_for_destination = attractions[2] + ": best attraction site"
will altering the value force the newly made variable to use a new memory space?
Saying pass by reference or pass by value isn't really applicable to python. For example consider Java. Everything in Java is pass by value, everything. However the value of an object is the reference to the object. So it is, in a way, pass by reference since mutating the object will be reflected to the calling context. Calling it "pass by value" is a bit confusing since for people used to C-family (where this distinction is more straightforward) assume that pass by value means you won't modify the underlying argument passed.
So saying Java is "pass by value" is true but possibly confusing. It is a more relevant statement at much lower levels of the language (like when talking about how the compiler handles passing arguments but not at the level of application programmer).
Python is similar. The confusion about the Java example actually lead Python to not say if it is pass by value/reference. http://stupidpythonideas.blogspot.com/2013/11/does-python-pass-by-value-or-by.html
So for programming, it is save to assume that any object is "passed by reference" (technically no but essentially yes). It is more complicated than that but that link explains it in more depth.
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 5 years ago.
I can't figure out what I'm doing wrong. It seems I'm not able to use dir() as the default argument to a function's input
def print_all_vars(arg=dir()):
for i in arg:
print(i)
foo=1
If I use
print_all_vars()
gives the output
__builtins__
__cached__
__doc__
__file__
__loader__
__name__
__package__
__spec__
but
print_all_vars(dir())
outputs
__builtins__
foo
print_all_vars
So - first, what is going on -
The line containing the def statement is run only once, when the module containing the function is first imported (or on the interactive prompt) - anyway, any expressions used as default parameter values are evaluated a single time, and their resulting object is kept as the default value for that parameter.
So, your dir() call is executed, and the resulting list - a dir on the namespace of the module containing your example function is saved. (The most common related error for Python beginners is to try to put an empty list or dictionary ( [] or {}) as a paramter default: the same list or dictionary will be reused for the lifetime of the function).
That explains why you see in the output some values that won't show up if you call it with a dir from the interactive prompt, like __package__ or __spec__.
Now, your real question is:
... I am actually trying to create an analog of MATLAB's save function
that saves all existing variables to an external file that can be
recalled later –
So, even if your dir() would be lazily executed, when the function is called only, that would still fail: it would still retrieve the variable names from the module where the function is defined, not on the callers.
The coding pattern for doing that would be:
def print_all_vars(arg=None):
if arg is None:
arg = dir()
...
(And this is the most common way to enforce empty lists of dictionaries as default parameters on the function body)
But even this, or if you pass your own list of variable names explicitly to the function, you will still have just a list of strings - not Python objects that need you will want to save.
And - if you want to retrieve the variables from the caller by default, you have a problem in common with having this list of strings: you need to be able to access the namespace of the caller code, in order to either retrieve the variable names existing there, and later to retrieve their contents in order to be saved.
The way to know "who've called your function" in Python is an advanced thing, and I am not even sure it is guarantee to work across all Python implementations - but it will certainly work on the most common (cPython, Pypy, Jython) and even some considered smaller ones (such as Brython):
You call sys._getframe() to retrieve the frame object of the code currently being executed (that is the code in your function itself) - and this frame object will have a f_back attribute, which points to the frame object of the code which called your function. The frame object, on its side, has reference to the globals and locals variables of the code being run as plain dictionaries as f_globals and f_locals respectively.
You can them "save" and "restore" as you want the contents of the global variables on the caller code - but you can't update - using pure Python code - the local variables.
So, just to list by default the global variables on the caller code - you can do:
import sys
def print_all_vars(arg=None):
if arg is None:
arg = list(sys.get_frame.f_back.f_globals.keys())
print (arg)
Being able to properly saving and restoring those variables is another question - as you can see in the comments, you can use pickle to serialize a lot of Python objects - but the caller namespace will have module names (such as mathnp`, etc...) which can't be ordinarily pickled - so you will have to come with an strategy to detect, annotate the used modules, and restoring them, by using further introspection on all objects retrieved.
Anyway, experimenting with the information here should put you on the track for what you want.
Question
We are getting a strange error** and we suspect it is because our script.py*** assigned a variable that already has some built-in meaning. E.g.
str = 2
Is there a way we can check if this has happened?
So far
We're thinking it would involve:
Assign a list at the beggining of the script, containing all built-in objects' names as strings:
builtin_names = get_builtin_var_names() # hypothetical function
Assign a list at the end of the script, containing all user-assigned objects' names as strings:
user_names = get_user_var_names() # hypothetical function
Find the intersection, and check if not empty:
overwritten_names = list(set(user_names) & set(builtin_names))
if overwritten_names:
print("Whoops")
Related
Is there a way to tell if a function in JavaScript is getting over
written?
Is there a common way to check in Python if an object
is any function type?
**Silent error, for those interested in it, it is silent, i.e. it finishes without an error code but the value it spits out differs between two implementations of the same code, call them A and B... both versions require the running of two modules (separate files) that we've made (changes.py and dnds.py), but whereas:
Version A: involves running changes.py -> pickle intermediate data (into a .p file) -> dnds.py,
Version B: involves running changes.py -> return the data (a dict) as arguments to dnds.py -> dnds.py.
And for some reason only version A is the one with the correct final value (benchmarked against MATLAB's dnds function).
***script.py, is actually dnds.py (who has imported changes.py). You can find all the code, but to test the two alternative versions I was talking about in ** you need to specifically look at dnds.py, the line with: CTRL+F: "##TODO:Urgent:debug:2016-11-28:". Once you find that line, you can read the rest of that comment line for instructions how to replicate version B, and its resulting silent error**. For some reason I HAVE to pickle the data to get it to work... when I just return the dicts directly I get the wrong dN/dS values.
You can get the names (and values) of builtins via the dict __builtins__. You can get the names (and values) of global variables with globals() and of locals with locals(). So you could do something like:
import __builtin__
name, val = None, None
for name, val in locals().iteritems():
if hasattr(__builtin__, name) and getattr(__builtin__, name) != val:
print("{} was overwritten!".format(name))
and then the same for globals(). This will check whether there is any object in the local namespace that has a different value in the builtins namespace. (Setting name and val to None is needed so that the variables exist before calling locals, or else you'll get a "dictionary changed sized during iteration" error because the names are added partway through the loop.)
You could also use a tool like pylint which checks for such errors among many others.
I would like to write a function which receives a local namespace dictionary and update it. Something like this:
def UpdateLocals(local_dict):
d = {'a':10, 'b':20, 'c':30}
local_dict.update(d)
When I call this function from the interactive python shell it works all right, like this:
a = 1
UpdateLocals(locals())
# prints 20
print a
However, when I call UpdateLocals from inside a function, it doesn't do what I expect:
def TestUpdateLocals():
a = 1
UpdateLocals(locals())
print a
# prints 1
TestUpdateLocals()
How can I make the second case work like the first?
UPDATE:
Aswin's explanation makes sense and is very helpful to me. However I still want a mechanism to update the local variables. Before I figure out a less ugly approach, I'm going to do the following:
def LoadDictionary():
return {'a': 10, 'b': 20, 'c': 30}
def TestUpdateLocals():
a = 1
for name, value in LoadDictionary().iteritems():
exec('%s = value' % name)
Of course the construction of the string statements can be automated, and the details can be hidden from the user.
You have asked a very good question. In fact, the ability to update local variables is very important and crucial in saving and loading datasets for machine learning or in games. However, most developers of Python language have not come to a realization of its importance. They focus too much on conformity and optimization which is nevertheless important too.
Imagine you are developing a game or running a deep neural network (DNN), if all local variables are serializable, saving the entire game or DNN can be simply put into one line as print(locals()), and loading the entire game or DNN can be simply put into one line as locals().update(eval(sys.stdin.read())).
Currently, globals().update(...) takes immediate effect but locals().update(...) does not work because Python documentation says:
The default locals act as described for function locals() below:
modifications to the default locals dictionary should not be
attempted. Pass an explicit locals dictionary if you need to see
effects of the code on locals after function exec() returns.
Why they design Python in such way is because of optimization and conforming the exec statement into a function:
To modify the locals of a function on the fly is not possible without
several consequences: normally, function locals are not stored in a
dictionary, but an array, whose indices are determined at compile time
from the known locales. This collides at least with new locals added
by exec. The old exec statement circumvented this, because the
compiler knew that if an exec without globals/locals args occurred in
a function, that namespace would be "unoptimized", i.e. not using the
locals array. Since exec() is now a normal function, the compiler does
not know what "exec" may be bound to, and therefore can not treat is
specially.
Since global().update(...) works, the following piece of code will work in root namespace (i.e., outside any function) because locals() is the same as globals() in root namespace:
locals().update({'a':3, 'b':4})
print(a, b)
But this will not work inside a function.
However, as hacker-level Python programmers, we can use sys._getframe(1).f_locals instead of locals(). From what I have tested so far, on Python 3, the following piece of code always works:
def f1():
sys._getframe(1).f_locals.update({'a':3, 'b':4})
print(a, b)
f1()
However, sys._getframe(1).f_locals does not work in root namespace.
The locals are not updated here because, in the first case, the variable declared has a global scope. But when declared inside a function, the variable loses scope outside it.
Thus, the original value of the locals() is not changed in the UpdateLocals function.
PS: This might not be related to your question, but using camel case is not a good practice in Python. Try using the other method.
update_locals() instead of UpdateLocals()
Edit To answer the question in your comment:
There is something called a System Stack. The main job of this system stack during the execution of a code is to manage local variables, make sure the control returns to the correct statement after the completion of execution of the called function etc.,
So, everytime a function call is made, a new entry is created in that stack,
which contains the line number (or instruction number) to which the control has to return after the return statement, and a set of fresh local variables.
The local variables when the control is inside the function, will be taken from the stack entry. Thus, the set of locals in both the functions are not the same. The entry in the stack is popped when the control exits from the function. Thus, the changes you made inside the function are erased, unless and until those variables have a global scope.
This question already has answers here:
How do I check if a variable exists?
(14 answers)
Closed 9 years ago.
While working with many if/elif/else I sometime loose a track if a variable has been already declared. I would like to know a simple way to check if the variable foo has been already declared. What would be the simplest syntax to accomplish that?
P.s. It was interesting to see how globals() and locals() could be used to accomplish this.
EDIT:
I've ended up using:
if not 'myVariableName' in globals().keys(): myVariableName='someValue'
It's hard to track when and where/if app = QtGui.QApplication(sys.argv) had been already declared especially in situation when one gui module calls another and vise versa.
if 'app' in globals() (seems) helps to avoid an accidental variable re-declaration.
try:
foo
except NameError:
# does not exist
else:
# does exist
Rather than worrying about if a variable is declared, declare all variables from the start. You can give variables some value to indicate that it is still in a initial state, such as
foo = None
or, if you want to use None for some other purpose, you can create a unique value to indicate the initial state, such as
INITIAL = object()
foo = INITIAL
Later, if you need to check if the variable has been set, you could just use if:
if foo == INITIAL:
I think the main advantages to doing it this way are
You'll never accidentally raise a NameError.
There will be one place in your code where all the variables are
introduced, so readers of your code do not have to wonder what
variables are in play.
By the way, it is similarly a good idea to define all instance attributes in __init__, so there is no question about what attributes an instance may have.