We Keep Coding

Identify unintentional read/write of global variables inside a python function? For example using static analysis?

One of the things I find frustrating with python is that if I write a function like this:
def UnintentionalValueChangeOfGlobal(a):
SomeDict['SomeKey'] = 100 + a
b = 0.5 * SomeDict['SomeKey']
return b
And then run it like so:
SomeDict = {}
SomeDict['SomeKey'] = 0
b = UnintentionalValueChangeOfGlobal(10)
print(SomeDict['SomeKey'])
Python will: 1) find and use SomeDict during the function call even though I have forgotten to provide it as an input to the function; 2) permanently change the value of SomeDict['SomeKey'] even though it is not included in the return statement of the function.
For me this often leads to variables unintentionally changing values - SomeDict['SomeKey'] in this case becomes 110 after the function is called when the intent was to only manipulate the function output b.
In this case I would have preferred that python: 1) crashes with an error inside the function saying that SomeDict is undefined; 2) under no circumstances permanently changes the value of any variable other than the output b after the function has been called.
I understand that it is not possible to disable the use of globals all together in python, but is there a simple method (a module or an IDE etc.) which can perform static analysis on my python functions and warn me when a function is using and/or changing the value of variables which are not the function's output? I.e., warn me whenever variables are used or manipulated which are not local to the function?

One of the reasons Python doesn't provide any obvious and easy way to prevent accessing (undeclared) global names in a function is that in Python everything (well, everything that can be assigned to a name at least) is an object, including functions, classes and modules, so preventing a function to access undeclared global names would make for quite verbose code... And nested scopes (closures etc) don't help either.
And, of course, despite globals being evils, there ARE still legitimate reasons for mutating a global object sometimes. FWIW, even linters (well pylint and pyflakes at least) don't seem to have any option to detect this AFAICT - but you'll have to double-check by yourself, as I might have overlooked it or it might exist as a pylint extension or in another linter.
OTHO, I very seldom had bugs coming from such an issue in 20+ years (I can't remember a single occurrence actually). Routinely applying basic good practices - short functions avoiding side effects as much as possible, meaningful names and good naming conventions etc, unittesting at least the critical parts etc - seem to be effective enough to prevent such issues.
One of the points here is that I have a rule about non-callable globals being to be considered as (pseudo) constants, which is denoted by naming them ALL_UPPER. This makes it very obvious when you actually either mutate or rebind one...
As a more general rule: Python is by nature a very dynamic language (heck, you can even change the class of an object at runtime...) and with a "we're all consenting adults" philosophy, so it's indeed "lacking" most of the safety guards you'll find in more "B&D" languages like Java and relies instead on conventions, good practices and plain common sense.
Now, Python is not only vey dynamic but also exposes much of it's inners, so you can certainly (if this doesn't already exists) write a pylint extension that would at least detect global names in function codes (hint: you can access the compiled code of a function object with yourfunc.co_code (py2) or yourfunc.__code__ (py3) and then inspect what names are used in the code). But unless you have to deal with a team of sloppy undisciplined devs (in which case you have another issue - there's no technical solutions to stupidity), my very humble opinion is that you're wasting your time.

Ideally I would have wanted the global-checking functionality I’m searching for to be implemented within an IDE and continuously used to assess the use of globals in functions. But since that does not appear to exist I threw together an ad hoc function which takes a python function as input and then looks at the bytecode instructions of the function to see if there are any LOAD_GLOBAL or STORE_GLOBAL instructions present. If it finds any, it tries to assess the type of the global and compare it to a list of user provided types (int, float, etc..). It then prints out the name of all global variables used by the function.
The solution is far from perfect and quite prone to false positives. For instance, if np.unique(x) is used in a function before numpy has been imported (import numpy as np) it will erroneously identify np as a global variable instead of a module. It will also not look into nested functions etc.
But for simple cases such as the example in this post it seems to work fine. I just used it to scan through all the functions in my codebase and it found another global usage that I was unaware of – so at least for me it is useful to have!
Here is the function:
def CheckAgainstGlobals(function, vartypes):
"""
Function for checking if another function reads/writes data from/to global
variables. Only variables of the types contained within 'vartypes' and
unknown types are included in the output.
Inputs:
function - a python function
vartypes - a list of variable types (int, float, dict,...)
Example:
# Define a function
def testfcn(a):
a = 1 + b
return a
# Check if the function read/writes global variables.
CheckAgainstGlobals(testfcn,[int, float, dict, complex, str])
# Should output:
>> Global-check of function: testfcn
>> Loaded global variable: b (of unknown type)
"""
import dis
globalsFound = []
# Disassemble the function's bytecode in a human-readable form.
bytecode = dis.Bytecode(function)
# Step through each instruction in the function.
for instr in bytecode:
# Check if instruction is to either load or store a global.
if instr[0] == 'LOAD_GLOBAL' or instr[0] == 'STORE_GLOBAL':
# Check if its possible to determine the type of the global.
try:
type(eval(instr[3]))
TypeAvailable = True
except:
TypeAvailable = False
"""
Determine if the global variable is being loaded or stored and
check if 'argval' of the global variable matches any of the
vartypes provided as input.
"""
if instr[0] == 'LOAD_GLOBAL':
if TypeAvailable:
for t in vartypes:
if isinstance(eval(instr[3]), t):
s = ('Loaded global variable: %s (of type %s)' %(instr[3], t))
if s not in globalsFound:
globalsFound.append(s)
else:
s = ('Loaded global variable: %s (of unknown type)' %(instr[3]))
if s not in globalsFound:
globalsFound.append(s)
if instr[0] == 'STORE_GLOBAL':
if TypeAvailable:
for t in vartypes:
if isinstance(eval(instr[3]), t):
s = ('Stored global variable: %s (of type %s)' %(instr[3], t))
if s not in globalsFound:
globalsFound.append(s)
else:
s = ('Stored global variable: %s (of unknown type)' %(instr[3]))
if s not in globalsFound:
globalsFound.append(s)
# Print out summary of detected global variable usage.
if len(globalsFound) == 0:
print('\nGlobal-check of fcn: %s. No read/writes of global variables were detected.' %(function.__code__.co_name))
else:
print('\nGlobal-check of fcn: %s' %(function.__code__.co_name))
for s in globalsFound:
print(s)
When used on the function in the example directly after the function has been declared, it will find warn about the usage of the global variable SomeDict but it will not be aware of its type:
def UnintentionalValueChangeOfGlobal(a):
SomeDict['SomeKey'] = 100 + a
b = 0.5 * SomeDict['SomeKey']
return b
# Will find the global, but not know its type.
CheckAgainstGlobals(UnintentionalValueChangeOfGlobal,[int, float, dict, complex, str])
>> Global-check of fcn: UnintentionalValueChangeOfGlobal
>> Loaded global variable: SomeDict (of unknown type)
When used after SomeDict has been defined it also detects that the global is a dict:
SomeDict = {}
SomeDict['SomeKey'] = 0
b = UnintentionalValueChangeOfGlobal(10)
print(SomeDict['SomeKey'])
# Will find the global, and also see its type.
CheckAgainstGlobals(UnintentionalValueChangeOfGlobal,[int, float, dict, complex, str])
>> Global-check of fcn: UnintentionalValueChangeOfGlobal
>> Loaded global variable: SomeDict (of type <class 'dict'>)
Note: in its current state the function fails to detect that SomeDict['SomeKey'] changes value. I.e., it only detects the load instruction, not that the previous value of the global is manipulated. That is because the instruction STORE_SUBSCR seems to be used in this case instead of STORE_GLOBAL. But the use of the global is still detected (since it is being loaded) which is enough for me.

You can check the varible using globals():
def UnintentionalValueChangeOfGlobal(a):
if 'SomeDict' in globals():
raise Exception('Var in globals')
SomeDict['SomeKey'] = 100 + a
b = 0.5 * SomeDict['SomeKey']
return b
SomeDict = {}
SomeDict['SomeKey'] = 0
b = UnintentionalValueChangeOfGlobal(10)
print(SomeDict['SomeKey'])

Limit Python function scope to local variables only

Is there a way to limit function so that it would only have access to local variable and passed arguments?
For example, consider this code
a = 1
def my_fun(x):
print(x)
print(a)
my_fun(2)
Normally the output will be
2
1
However, I want to limit my_fun to local scope so that print(x) would work but throw an error on print(a). Is that possible?

I feel like I should preface this with: Do not actually do this.
You (sort of) can with functions, but you will also disable calls to all other global methods and variables, which I do not imagine you would like to do.
You can use the following decorator to have the function act like there are no variables in the global namespace:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {})
And then call your function:
a = 1
#noglobal
def my_fun(x):
print(x)
print(a)
my_fun(2)
However this actually results in a different error than you want, it results in:
NameError: name 'print' is not defined
By not allowing globals to be used, you cannot use print() either.
Now, you could pass in the functions that you want to use as parameters, which would allow you to use them inside the function, but this is not a good approach and it is much better to just keep your globals clean.
a = 1
#noglobal
def my_fun(x, p):
p(x)
p(a)
my_fun(2, print)
Output:
2
NameError: name 'a' is not defined

Nope. The scoping rules are part of a language's basic definition. To change this, you'd have to alter the compiler to exclude items higher on the context stack, but still within the user space. You obviously don't want to limit all symbols outside the function's context, as you've used one in your example: the external function print. :-)

Is there an equivalent in Python of Fortran's "implicit none"?

In Fortran there is a statement Implicit none that throws a compilation error when a local variable is not declared but used. I understand that Python is a dynamically typed language and the scope of a variable may be determined at runtime.
But I would like to avoid certain unintended errors that happen when I forget to initialize a local variable but use it in the main code. For example, the variable x in the following code is global even though I did not intend that:
def test():
y=x+2 # intended this x to be a local variable but forgot
# x was not initialized
print y
x=3
test()
So my question is that: Is there any way to ensure all variables used in test() are local to it and that there are no side effects. I am using Python 2.7.x. In case there is a local variable, an error is printed.

So my question is that: Is there any way to ensure all variables used
in test() are local to it and that there are no side effects.
There is a technique to validate that globals aren't accessed.
Here's a decorator that scans a function's opcodes for a LOAD_GLOBAL.
import dis, sys, re, StringIO
def check_external(func):
'Validate that a function does not have global lookups'
saved_stdout = sys.stdout
sys.stdout = f = StringIO.StringIO()
try:
dis.dis(func)
result = f.getvalue()
finally:
sys.stdout = saved_stdout
externals = re.findall('^.*LOAD_GLOBAL.*$', result, re.MULTILINE)
if externals:
raise RuntimeError('Found globals: %r', externals)
return func
#check_external
def test():
y=x+2 # intended this x to be a local variable but forgot
# x was not initialized
print y
To make this practical, you will want a stop list of acceptable global references (i.e. modules). The technique can be extended to cover other opcodes such as STORE_GLOBAL and DELETE_GLOBAL.
All that said, I don't see straight-forward way to detect side-effects.

There is no implicit None in the sense you mean. Assignment will create a new variable, thus a typo might introduce a new name into your scope.
One way to get the effect you want is to use the following ugly-ish hack:
def no_globals(func):
if func.func_code.co_names:
raise TypeError(
'Function "%s" uses the following globals: %s' %
(func.__name__, ', '.join(func.func_code.co_names)))
return func
So when you declare your function test–with the no_globals wrapper–you'll get an error, like so:
>>> #no_globals
... def test():
... y = x + 2 # intended this x to be a local variable but forgot
... # x was not initialized
... print y
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in no_globals
TypeError: Function "test" uses the following globals: x
>>>
>>> x = 3
>>> test()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'test' is not defined

Just avoid using globally-scoped variables at all. And if you must, prefix their names with something you'll never use in a local variable name.

If you were really worried about this, you could try the following:
def test():
try:
x
except:
pass
else:
return
y = x+2
print y
But I'd recommend simply being mindful when writing a function that you don't try to reference things before assigning them. If possible, try to test each function separately, with a variety of carefully-defined inputs and intended outputs. There are a variety of testing suites and strategies, not to mention the simple assert keyword.

In Python, this is quite simply entirely legal. In fact, it is a strength of the language! This (lack) of error is the reason why you can do something like this:
def function1():
# stuff here
function2()
def function2():
pass
Whereas in C, you would need to "forward declare" function2.
There are static syntax checkers (like flake8) for Python that do plenty of work to catch errors and bad style, but this is not an error, and it is not caught by such a checker. Otherwise, something like this would be an error:
FILENAME = '/path/to/file'
HOSTNAME = 'example.com'
def main():
with open(FILENAME) as f:
f.write(HOSTNAME)
Or, something even more basic like this would be an error:
import sys
def main():
sys.stdout.write('blah')
The best thing you can do is use a different naming convention (like ALL_CAPS) for module level variable declarations. Also, make it a habit to put all of your code within a function (no module-level logic) in order to prevent variables from leaking into the global namespace.

Is there any way to ensure all variables used in test() are local to it and that there are no side effects.
No. The language offers no such functionality.
There is the built in locals() function. So you could write:
y = locals()['x'] + 2
but I cannot imagine anyone considering that to be an improvement.

To make sure the correct variable is used, you need to limit the scope of the lookup. Inside a function, Python will look to arguments defined in line, then to the args and kwargs. After those, its going to look outside the function. This can cause annoying bugs if the function depends on a global variable that gets changed elsewhere.
To avoid using a global variable by accident, you can define the function with a keyword argument for the variables your going to use:
def test(x=None):
y=x+2 # intended this x to be a local variable but forgot
# x was not initialized
print y
x=3
test()
I'm guessing you don't want to do this for lots of variables. However, it will stop the function from using globals.
Actually, even if you want to use a global variable in the function, I think its best to make it explicit:
x = 2
def test(x=x):
y=x+2 # intended this x to be a local variable but forgot
# x was not initialized
print y
x=3
test()
This example will use x=2 for the function no matter what happens to the global value of x afterwards. Inside the function, x is fixed to the value it had at compile time.
I started passing global variables as keyword arguments after getting burned a couple times. I think this is generally considered good practice?

The offered solutions are interesting, especially the one using dis.dis, but you are really thinking in the wrong direction. You don't want to write such a cumbersome code.
Are you afraid that you will reach a global accidentally? Then don't write globals. The purpose of module globals is mostly to be reached. (in a comment I have read that you have 50 globals in scope, which seems to me that you have some design errors).
If you still DO have to have globals, then either use a naming convention (UPPER_CASE is recommended for constants, which could cover your cases).
If a naming convention is not an option either, just put the functions you don't want to reach any global in a separate module, and do not define globals there. For instance, define pure_funcs and inside of that module, write your "pure" functions there, and then import this module. Since python has lexical scope, functions can only reach variables defined in outer scopes of the module they were written (and locals or built-ins, of course). Something like this:
# Define no globals here, just the functions (which are globals btw)
def pure1(arg1, arg2):
print x # This will raise an error, no way you can mix things up.

static effect on python

how we can construct static effect on python instead of using class and global ?
not like that one :
global a
a = []
#simple ex ;
fonk ( a , b , d)
x = 1
a.append ( x)
EDIT:
I want to create temporary memory , if I exit the function namely fonk , I want to save change as list on temporary memory .
We can do that demand only put static keyword in front of data type but in python , we dont have static, so I want that effect in python . Therefore , how can I do ?
As above code say "a" represents temporary memory

Default values for function arguments are evaluated once, at function definition time, so if you put a mutable object there, it will live across calls. Can be a gotcha, but very useful for caches and similar things static is often used for in other languages. Of course callers can override your cache in this case - but that's not a bad thing, they won't unless they have good reasons and in that case you should allow them to.
Example (this one is usually found in "gotchas" question instead ^^):
def append_and_return_static_list(item, items=[]):
items.append(item)
return items
append_and_return_static_list(0)
append_and_return_static_list(1)
print append_and_return_static_list(2) #=> [0,1,2]
Now, if you absolutely don't want to go that way, you still have other possibilities: You can create a variable outside the function and put the object you want to share there. You should propably prefix the name with a single underscore if you want it to be considered private to that place (not compiler-enforced-private but convention-and-survival-instinct-enforced).
Example (not the best code, the above is better in almost all cases):
_items = []
def append_and_return_static_list(item):
_items.append(item)
return _items
append_and_return_static_list(0)
append_and_return_static_list(1)
print append_and_return_static_list(2) #=> [0,1,2]

Attributes defined in global scope or in class scope are effectively static since modules are treated as singletons and by extension class definitions are singletons within a particular module global scope. (This explanation glosses over the dynamic features of Python that can change this behavior)
The global keyword is used within a local scope to disambiguate assignment. It's a declaration that the attribute name belongs to the enclosing global scope.
In your (broken) example you don't even need to use the global keyword since you are not using assignment, you are calling the append() method of a - no disambiguation is required.
An illustrative example of what global is used for:
>>> a = []
>>> def fonk():
... a = [1]
...
>>> print a
[]
>>> fonk()
>>> print a
[]
>>>
>>> def fonk2():
... global a
... a = [2]
...
>>> print a
[]
>>> fonk2()
>>> print a
[2]
EDIT: I guess I missed the point of the question (do not do it via global and classes), but I don't understand the objection especially when it seems that the example code was broken.

How can one create new scopes in python

In many languages (and places) there is a nice practice of creating local scopes by creating a block like this.
void foo()
{
... Do some stuff ...
if(TRUE)
{
char a;
int b;
... Do some more stuff ...
}
... Do even more stuff ...
}
How can I implement this in python without getting the unexpected indent error and without using some sort of if True: tricks

Why do you want to create new scopes in python anyway?
The normal reason for doing it in other languages is variable scoping, but that doesn't happen in python.
if True:
a = 10
print a

In Python, scoping is of three types : global, local and class. You can create specialized 'scope' dictionaries to pass to exec / eval(). In addition you can use nested scopes
(defining a function within another). I found these to be sufficient in all my code.
As Douglas Leeder said already, the main reason to use it in other languages is variable scoping and that doesn't really happen in Python. In addition, Python is the most readable language I have ever used. It would go against the grain of readability to do something like if-true tricks (Which you say you want to avoid). In that case, I think the best bet is to refactor your code into multiple functions, or use a single scope. I think that the available scopes in Python are sufficient to cover every eventuality, so local scoping shouldn't really be necessary.

If you just want to create temp variables and let them be garbage collected right after using them, you can use
del varname
when you don't want them anymore.
If its just for aesthetics, you could use comments or extra newlines, no extra indentation, though.

Python has exactly two scopes, local and global. Variables that are used in a function are in local scope no matter what indentation level they were created at. Calling a nested function will have the effect that you're looking for.
def foo():
a = 1
def bar():
b = 2
print a, b #will print "1 2"
bar()
Still like everyone else, I have to ask you why you want to create a limited scope inside a function.

variables in list comprehension (Python 3+) and generators are local:
>>> i = 0
>>> [i+1 for i in range(10)]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> i
0
but why exactly do you need this?

A scope is a textual region of a
Python program where a namespace is
directly accessible. “Directly
accessible” here means that an
unqualified reference to a name
attempts to find the name in the
namespace...
Please, read the documentation and clarify your question.
btw, you don't need if(TRUE){} in C, a simple {} is sufficient.

As mentioned in the other answers, there is no analogous functionality in Python to creating a new scope with a block, but when writing a script or a Jupyter Notebook, I often (ab)use classes to introduce new namespaces for similar effect. For example, in a notebook where you might have a model "Foo", "Bar" etc. and related variables you might want to create a new scope to avoid having to reuse names like
model = FooModel()
optimizer = FooOptimizer()
...
model = BarModel()
optimizer = BarOptimizer()
or suffix names like
model_foo = ...
optimizer_foo = ...
model_bar = ...
optimizer_bar= ...
Instead you can introduce new namespaces with
class Foo:
model = ...
optimizer = ...
loss = ....
class Bar:
model = ...
optimizer = ...
loss = ...
and then access the variables as
Foo.model
Bar.optimizer
...
I find that using namespaces this way to create new scopes makes code more readable and less error-prone.

While the leaking scope is indeed a feature that is often useful,
I have created a package to simulate block scoping (with selective leaking of your choice, typically to get the results out) anyway.
from scoping import scoping
a = 2
with scoping():
assert(2 == a)
a = 3
b = 4
scoping.keep('b')
assert(3 == a)
assert(2 == a)
assert(4 == b)
https://pypi.org/project/scoping/

I would see this as a clear sign that it's time to create a new function and refactor the code. I can see no reason to create a new scope like that. Any reason in mind?

def a():
def b():
pass
b()
If I just want some extra indentation or am debugging, I'll use if True:

Like so, for arbitrary name t:
### at top of function / script / outer scope (maybe just big jupyter cell)
try: t
except NameError:
class t
pass
else:
raise NameError('please `del t` first')
#### Cut here -- you only need 1x of the above -- example usage below ###
t.tempone = 5 # make new temporary variable that definitely doesn't bother anything else.
# block of calls here...
t.temptwo = 'bar' # another one...
del t.tempone # you can have overlapping scopes this way
# more calls
t.tempthree = t.temptwo; del t.temptwo # done with that now too
print(t.tempthree)
# etc, etc -- any number of variables will fit into t.
### At end of outer scope, to return `t` to being 'unused'
del t
All the above could be in a function def, or just anyplace outside defs along a script.
You can add or del new elements to an arbitrary-named class like that at any point. You really only need one of these -- then manage your 'temporary' namespace as you like.
The del t statement isn't necessary if this is in a function body, but if you include it, then you can copy/paste chunks of code far apart from each other and have them work how you expect (with different uses of 't' being entirely separate, each use starting with the that try: t... block, and ending with del t).
This way if t had been used as a variable already, you'll find out, and it doesn't clobber t so you can find out what it was.
This is less error prone then using a series of random=named functions just to call them once -- since it avoids having to deal with their names, or remembering to call them after their definition, especially if you have to reorder long code.
This basically does exactly what you want: Make a temporary place to put things you know for sure won't collide with anything else, and which you are responsible for cleaning up inside as you go.
Yes, it's ugly, and probably discouraged -- you will be directed to decompose your work into a set of smaller, more reusable functions.

As others have suggested, the python way to execute code without polluting the enclosing namespace is to put it in a class or function. This presents a slight and usually harmless problem: defining the function puts its name in the enclosing namespace. If this causes harm to you, you can name your function using Python's conventional temporary variable "_":
def _():
polluting_variable = foo()
...
_() # Run the code before something overwrites the variable.
This can be done recursively as each local definition masks the definition from the enclosing scope.
This sort of thing should only be needed in very specific circumstances. An example where it is useful is when using Databricks' %run magic, which executes the contents of another notebook in the current notebook's global scope. Wrapping the child notebook's commands in temporary functions prevents them from polluting the global namespace.