I've gotten myself in trouble a few times now with accidentially (unintentionally) referencing global variables in a function or method definition.
My question is: is there any way to disallow python from letting me reference a global variable? Or at least warn me that I am referencing a global variable?
x = 123
def myfunc() :
print x # throw a warning or something!!!
Let me add that the typical situation where this arrises for my is using IPython as an interactive shell. I use 'execfile' to execute a script that defines a class. In the interpreter, I access the class variable directly to do something useful, then decide I want to add that as a method in my class. When I was in the interpreter, I was referencing the class variable. However, when it becomes a method, it needs to reference 'self'. Here's an example.
class MyClass :
a = 1
b = 2
def add(self) :
return a+b
m = MyClass()
Now in my interpreter I run the script 'execfile('script.py')', I'm inspecting my class and type: 'm.a * m.b' and decide, that would be a useful method to have. So I modify my code to be, with the non-intentional copy/paste error:
class MyClass :
a = 1
b = 2
def add(self) :
return a+b
def mult(self) :
return m.a * m.b # I really meant this to be self.a * self.b
This of course still executes in IPython, but it can really confuse me since it is now referencing the previously defined global variable!
Maybe someone has a suggestion given my typical IPython workflow.
First, you probably don't want to do this. As Martijn Pieters points out, many things, like top-level functions and classes, are globals.
You could filter this for only non-callable globals. Functions, classes, builtin-function-or-methods that you import from a C extension module, etc. are callable. You might also want to filter out modules (anything you import is a global). That still won't catch cases where you, say, assign a function to another name after the def. You could add some kind of whitelisting for that (which would also allow you to create global "constants" that you can use without warnings). Really, anything you come up with will be a very rough guide at best, not something you want to treat as an absolute warning.
Also, no matter how you do it, trying to detect implicit global access, but not explicit access (with a global statement) is going to be very hard, so hopefully that isn't important.
There is no obvious way to detect all implicit uses of global variables at the source level.
However, it's pretty easy to do with reflection from inside the interpreter.
The documentation for the inspect module has a nice chart that shows you the standard members of various types. Note that some of them have different names in Python 2.x and Python 3.x.
This function will get you a list of all the global names accessed by a bound method, unbound method, function, or code object in both versions:
def get_globals(thing):
thing = getattr(thing, 'im_func', thing)
thing = getattr(thing, '__func__', thing)
thing = getattr(thing, 'func_code', thing)
thing = getattr(thing, '__code__', thing)
return thing.co_names
If you want to only handle non-callables, you can filter it:
def get_callable_globals(thing):
thing = getattr(thing, 'im_func', thing)
func_globals = getattr(thing, 'func_globals', {})
thing = getattr(thing, 'func_code', thing)
return [name for name in thing.co_names
if callable(func_globals.get(name))]
This isn't perfect (e.g., if a function's globals have a custom builtins replacement, we won't look it up properly), but it's probably good enough.
A simple example of using it:
>>> def foo(myparam):
... myglobal
... mylocal = 1
>>> print get_globals(foo)
('myglobal',)
And you can pretty easily import a module and recursively walk its callables and call get_globals() on each one, which will work for the major cases (top-level functions, and methods of top-level and nested classes), although it won't work for anything defined dynamically (e.g., functions or classes defined inside functions).
If you only care about CPython, another option is to use the dis module to scan all the bytecode in a module, or .pyc file (or class, or whatever), and log each LOAD_GLOBAL op.
One major advantage of this over the inspect method is that it will find functions that have been compiled, even if they haven't been created yet.
The disadvantage is that there is no way to look up the names (how could there be, if some of them haven't even been created yet?), so you can't easily filter out callables. You can try to do something fancy, like connecting up LOAD_GLOBAL ops to corresponding CALL_FUNCTION (and related) ops, but… that's starting to get pretty complicated.
Finally, if you want to hook things dynamically, you can always replace globals with a wrapper that warns every time you access it. For example:
class GlobalsWrapper(collections.MutableMapping):
def __init__(self, globaldict):
self.globaldict = globaldict
# ... implement at least __setitem__, __delitem__, __iter__, __len__
# in the obvious way, by delegating to self.globaldict
def __getitem__(self, key):
print >>sys.stderr, 'Warning: accessing global "{}"'.format(key)
return self.globaldict[key]
globals_wrapper = GlobalsWrapper(globals())
Again, you can filter on non-callables pretty easily:
def __getitem__(self, key):
value = self.globaldict[key]
if not callable(value):
print >>sys.stderr, 'Warning: accessing global "{}"'.format(key)
return value
Obviously for Python 3 you'd need to change the print statement to a print function call.
You can also raise an exception instead of warning pretty easily. Or you might want to consider using the warnings module.
You can hook this into your code in various different ways. The most obvious one is an import hook that gives each new module a GlobalsWrapper around its normally-built globals. Although I'm not sure how that will interact with C extension modules, but my guess is that it will either work, or be harmlessly ignored, either of which is probably fine. The only problem is that this won't affect your top-level script. If that's important, you can write a wrapper script that execfiles the main script with a GlobalsWrapper, or something like that.
I've been struggling with a similar challenge (especially in Jupyter notebooks) and created a small package to limit the scope of functions.
>>> from localscope import localscope
>>> a = 'hello world'
>>> #localscope
... def print_a():
... print(a)
Traceback (most recent call last):
...
ValueError: `a` is not a permitted global
The #localscope decorator uses python's disassembler to find all instances of the decorated function using a LOAD_GLOBAL (global variable access) or LOAD_DEREF (closure access) statement. If the variable to be loaded is a builtin function, is explicitly listed as an exception, or satisfies a predicate, the variable is permitted. Otherwise, an exception is raised.
Note that the decorator analyses the code statically. Consequently, it does not have access to the values of variables accessed by closure.
Related
I came across weird behavior in Python 3.6.
I was able to call function and access variable defined only in child class from base class method.
I find this useful in my code but I come from C++ and this code looks very weird.
Can someone please explain this behavior?
class a:
def __init__(self):
print(self.var)
self.checker()
class b(a):
def __init__(self):
self.var=5
super().__init__()
def checker(self):
print('inside B checker')
myB = b()
Output:
5
inside B checker
All methods in Python are looked up dynamically. You're calling a method on self, and self is a b instance, and b instances have a checker method, so that method gets called.
Consider this code at the module top level, or in a top-level function:
myB = b()
myB.checker()
Obviously the global module code isn't part of the b class definition, and yet, this is obviously legal. Why should it be any different if you put the code inside the class a definition, and rename myB to welf? Python doesn't care. You're just asking the value—whether you've called it myB or self—"do you have something named checker?", and the answer is yes, so you can call it.
And var is even simpler; self.var just adds var to self.__dict__, so it's there; the fact that it's a b instance isn't even relevant here (except indirectly—being a b instance means it had b.__init___ called n it, and that's where var was created).
If you're wondering how this "asking the value", a slightly oversimplified version is:
Every object has a __dict__. When you do self.var=5, that actually does self.__dict__['var'] = 5. And when you print(self.var), that does print(self.__dict__['var']).
When that raises a KeyError, as it will for self.checker, Python tries type(self).__dict__['checker'], and, if that doesn't work, it loops over type(self).mro() and tries all of those dicts.
When all of those raise a KeyError, as they would with self.spam, Python calls self.__getattr__('spam').
If even that fails, you get an AttributeError.
Notice that if you try to construct an a instance, this will fail with an AttributeError. That's because now self is an a, not a b. It doesn't have a checker method, and it hasn't gone through the __init__ code that adds a var attribute.
The reason you can't do this in C++ is that C++ methods are looked up statically. It's not a matter of what type the value is at runtime, but what type the variable is at compile time. If the statically looked-up method says it's virtual, then the compiler inserts some dynamic-lookup code, but otherwise, it doesn't.1
One way it's often explained is that in Python (and other languages with SmallTalk semantics, like ObjC and Ruby), all methods are automatically virtual. But that's a bit misleading, because in C++, even with virtual methods, the method name and signature still has to be findable on the base class; in Python (and SmallTalk, etc.), that isn't necessary.
If you're thinking this must be horribly slow, that Python must have to do something like search some stack of namespaces for the method by name every time you call a method—well, it does that, but it's not as slow as you've expect. For one thing, a namespace is a dict, so it's a constant-time search. And the strings are interned and have their hash values cached. And the interpreter can even cache the lookup results if it wants to. The result is still slower than dereferencing a pointer through a vtable, but not by a huge margin (and besides, there are plenty of other things in Python that can be 20x slower than C++, like for loops; you don't use pure Python when you need every detail to work as fast as possible).
1. C++ also has another problem: even if you defined a var attribute and a checker virtual method in a, you don't get to choose the order the initializers get called; the compiler automatically calls a's constructor first, then b's. In Python, it calls b.__init__, and you choose when you want to call super().__init__(), whether it's at the start of the method, at the end, in the middle, or even never.
First you are creating an instance of class b which will call the constructor of class b __init__.
Inside the constructor your are setting the attribute self.var as 5.Later super().__init__() will call the constructor of the parent class A.
Inside the constructor of class A both self.var is printed and self.checker() is called.
Note that when calling the super().__init__() will place the child class instance self as the first argument by default.
As you know, when the project's code is very large and there are so many attributes and functions defined in a Class, but some of them never be called by the instance of the Class, and maybe some of them has been discarded. Here is a example:
class Foo(object):
""""""
def __init__(self):
self.a = 1
self.b = 2
self.c = 3
...
self.y = 25
self.z = 26
def func1(self):
pass
def func2(self):
pass
def func3(self):
pass
...
...
def func100(self):
pass
if __name__ == '__main__':
f = Foo()
f.func1()
f.func2()
print f.a, f.b, f.z
In the above code, the instance f of class Foo just called func1() and func2(). And how to find all the attributes and functions of class that never called by the instance of class.
I have tried compiler module but that could not solve my question. And dir(my_instance) is just print all the functions and attributes defined the the class.
Thanks in advance.
You can try coverage.py. It's not static analysis, but actually runs your code and records which statements are executed, outputting annotated html or txt as you wish (quite nicely formatted as well). You can then look for functions and methods whose bodies are not executed at all.
This still doesn't take care of unused attributes. And I don't know the answer to that. Maybe comment them out one at a time and see if tests still pass...
It's pretty hard to prove something is or is not used in the general case. Python is a dynamic language; if even one bit of code calls into code the static analyzer doesn't fully analyze, it could be accessing the variables mentioned.
The pylint and flake8 tools will tell you about local and global names that aren't defined prior to use (unless you break them by using from x import * style imports), and about imports that are never used (an import that is never used is usually wrong, but even then, it could be an intentional part of the interface, where linters would have to be silenced), but I don't believe they can tell you that a given attribute is never accessed; after all, someone else could import your module and access said attributes.
Use the profile module in the standard library.
python -m cProfile -o output_file myscript.py
Then load the stats file and use print_callees() to get all the functions that were called--during that run of the program.
I don't know of any easy way to find out which attributes are used.
What I am trying to do, is creating a module, with a class; and a function, which is an interface of that class; and a variable name on-the-fly in this function, which is pointing to an instance of that class. This function and the class itself should be in a separate module, and their usage should be in a different python file.
I think, it's much easier to understand what I am trying to do, when you are looking at my code:
This is the first.py:
class FirstClass:
def setID(self, _id):
self.id = _id
def func(self):
pass
# An 'interface' for FirstClass
def fst(ID):
globals()['%s' % ID] = FirstClass(ID)
return globals()['%s' % ID]
Now, if I'm calling fst('some_text') right in first.py, the result is pretty much what I dreamed of, because later on, any time I write some_text.func(), it will call the func(), because some_text is pointing to an instance of FirstClass.
But, when the second.py is something like this:
from first import fst
fst('sample_name')
sample_name.func()
Then the answer from python is going to be like this:
NameError: name 'sample_name' is not defined.
Which is somewhat reasonable.. So my question is: is there a "prettier" method or a completely different one to do this? Or do I have to change something small in my code to get this done?
Thank you!
Don't set it as a global in the function. Instead, just return the new instance from the function and set the global to that return value:
def fst(ID):
return FirstClass(ID)
then in second.py:
sample_name = fst('sample_name')
where, if inside a function, you declare sample_name a global.
The globals() method only ever returns the globals of the module in which you call it. It'll never return the globals of whatever is calling the function. If you feel you need to have access to those globals, rethink your code, you rarely, if ever, need to alter the globals of whatever is calling your function.
If you are absolutely certain you need access to the caller globals, you need to start hacking with stack frames:
# retrieve caller globals
import sys
caller_globals = sys._getframe(1).f_globals
But, as the documentation of sys._getframe() states:
CPython implementation detail: This function should be used for internal and specialized purposes only. It is not guaranteed to exist in all implementations of Python.
I'm new to python, and I've been reading that using global to pass variables to other functions is considered noobie, as well as a bad practice. I would like to move away from using global variables, but I'm not sure what to do instead.
Right now I have a UI I've created in wxPython as its own separate class, and I have another class that loads settings from a .ini file. Since the settings in the UI should match those in the .ini, how do I pass around those values? I could using something like: Settings = Settings() and then define the variables as something like self.settings1, but then I would have to make Settings a global variable to pass it to my UI class (which it wouldn't be if I assign in it main()).
So what is the correct and pythonic way to pass around these variables?
Edit: Here is the code that I'm working with, and I'm trying to get it to work like Alex Martelli's example. The following code is saved in Settings.py:
import ConfigParser
class _Settings():
#property
def enableautodownload(self): return self._enableautodownload
def __init__(self):
self.config = ConfigParser.ConfigParser()
self.config.readfp(open('settings.ini'))
self._enableautodownload=self.config.getboolean('DLSettings', 'enableautodownload')
settings = _Settings()
Whenever I try to refer to Settings.settings.enableautodownload from another file I get: AttributeError: 'module' object has no attribute 'settings'. What am I doing wrong?
Edit 2: Never mind about the issue, I retyped the code and it works now, so it must have been a simple spelling or syntax error.
The alternatives to global variables are many -- mostly:
explicit arguments to functions, classes called to create one of their instance, etc (this is usually the clearest, since it makes the dependency most explicit, when feasible and not too repetitious);
instance variables of an object, when the functions that need access to those values are methods on that same object (that's OK too, and a reasonable way to use OOP);
"accessor functions" that provide the values (or an object which has attributes or properties for the values).
Each of these (esp. the first and third ones) is particularly useful for values whose names must not be re-bound by all and sundry, but only accessed. The really big problem with global is that it provides a "covert communication channel" (not in the cryptographic sense, but in the literal one: apparently separate functions can actually be depending on each other, influencing each other, via global values that are not "obvious" from the functions' signatures -- this makes the code hard to test, debug, maintain, and understand).
For your specific problem, if you never use the global statement, but rather access the settings in a "read-only" way from everywhere (and you can ensure that more fully by making said object's attributes be read-only properties!), then having the "read-only" accesses be performed on a single, made-once-then-not-changed, module-level instance, is not too bad. I.e., in some module foo.py:
class _Settings(object):
#property
def one(self): return self._one
#property
def two(self): return self._two
def __init__(self, one, two):
self._one, self._two = one, two
settings = _Settings(23, 45)
and from everywhere else, import foo then just access foo.settings.one and foo.settings.two as needed. Note that I've named the class with a single leading underscore (just like the two instance attributes that underlie the read-only properties) to suggest that it's not meant to be used from "outside" the module -- only the settings object is supposed to be (there's no enforcement -- but any user violating such requested privacy is most obviously the only party responsible for whatever mayhem may ensue;-).
I've got a bunch of functions (outside of any class) where I've set attributes on them, like funcname.fields = 'xxx'. I was hoping I could then access these variables from inside the function with self.fields, but of course it tells me:
global name 'self' is not defined
So... what can I do? Is there some magic variable I can access? Like __this__.fields?
A few people have asked "why?". You will probably disagree with my reasoning, but I have a set of functions that all must share the same signature (accept only one argument). For the most part, this one argument is enough to do the required computation. However, in a few limited cases, some additional information is needed. Rather than forcing every function to accept a long list of mostly unused variables, I've decided to just set them on the function so that they can easily be ignored.
Although, it occurs to me now that you could just use **kwargs as the last argument if you don't care about the additional args. Oh well...
Edit: Actually, some of the functions I didn't write, and would rather not modify to accept the extra args. By "passing in" the additional args as attributes, my code can work both with my custom functions that take advantage of the extra args, and with third party code that don't require the extra args.
Thanks for the speedy answers :)
self isn't a keyword in python, its just a normal variable name. When creating instance methods, you can name the first parameter whatever you want, self is just a convention.
You should almost always prefer passing arguments to functions over setting properties for input, but if you must, you can do so using the actual functions name to access variables within it:
def a:
if a.foo:
#blah
a.foo = false
a()
see python function attributes - uses and abuses for when this comes in handy. :D
def foo():
print(foo.fields)
foo.fields=[1,2,3]
foo()
# [1, 2, 3]
There is nothing wrong with adding attributes to functions. Many memoizers use this to cache results in the function itself.
For example, notice the use of func.cache:
from decorator import decorator
#decorator
def memoize(func, *args, **kw):
# Author: Michele Simoniato
# Source: http://pypi.python.org/pypi/decorator
if not hasattr(func, 'cache'):
func.cache = {}
if kw: # frozenset is used to ensure hashability
key = args, frozenset(kw.iteritems())
else:
key = args
cache = func.cache # attribute added by memoize
if key in cache:
return cache[key]
else:
cache[key] = result = func(*args, **kw)
return result
You can't do that "function accessing its own attributes" correctly for all situations - see for details here how can python function access its own attributes? - but here is a quick demonstration:
>>> def f(): return f.x
...
>>> f.x = 7
>>> f()
7
>>> g = f
>>> g()
7
>>> del f
>>> g()
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "<interactive input>", line 1, in f
NameError: global name 'f' is not defined
Basically most methods directly or indirectly rely on accessing the function object through lookup by name in globals; and if original function name is deleted, this stops working. There are other kludgey ways of accomplishing this, like defining class, or factory - but thanks to your explanation it is clear you don't really need that.
Just do the mentioned keyword catch-all argument, like so:
def fn1(oneArg):
// do the due
def fn2(oneArg, **kw):
if 'option1' in kw:
print 'called with option1=', kw['option1']
//do the rest
fn2(42)
fn2(42, option1='something')
Not sure what you mean in your comment of handling TypeError - that won't arise when using **kw. This approach works very well for some python system functions - check min(), max(), sort(). Recently sorted(dct,key=dct.get,reverse=True) came very handy to me in CodeGolf challenge :)
Example:
>>> def x(): pass
>>> x
<function x at 0x100451050>
>>> x.hello = "World"
>>> x.hello
"World"
You can set attributes on functions, as these are just plain objects, but I actually never saw something like this in real code.
Plus. self is not a keyword, just another variable name, which happens to be the particular instance of the class. self is passed implicitly, but received explicitly.
if you want globally set parameters for a callable 'thing' you could always create a class and implement the __call__ method?
There is no special way, within a function's body, to refer to the function object whose code is executing. Simplest is just to use funcname.field (with funcname being the function's name within the namespace it's in, which you indicate is the case -- it would be harder otherwise).
This isn't something you should do. I can't think of any way to do what you're asking except some walking around on the call stack and some weird introspection -- which isn't something that should happen in production code.
That said, I think this actually does what you asked:
import inspect
_code_to_func = dict()
def enable_function_self(f):
_code_to_func[f.func_code] = f
return f
def get_function_self():
f = inspect.currentframe()
code_obj = f.f_back.f_code
return _code_to_func[code_obj]
#enable_function_self
def foo():
me = get_function_self()
print me
foo()
While I agree with the the rest that this is probably not good design, the question did intrigue me. Here's my first solution, which I may update once I get decorators working. As it stands, it relies pretty heavily on being able to read the stack, which may not be possible in all implementations (something about sys._getframe() not necessarily being present...)
import sys, inspect
def cute():
this = sys.modules[__name__].__dict__.get(inspect.stack()[0][3])
print "My face is..." + this.face
cute.face = "very cute"
cute()
What do you think? :3
You could use the following (hideously ugly) code:
class Generic_Object(object):
pass
def foo(a1, a2, self=Generic_Object()):
self.args=(a1,a2)
print "len(self.args):", len(self.args)
return None
... as you can see it would allow you to use "self" as you described. You can't use an "object()" directly because you can't "monkey patch(*)" values into an object() instance. However, normal subclasses of object (such as the Generic_Object() I've shown here) can be "monkey patched"
If you wanted to always call your function with a reference to some object as the first argument that would be possible. You could put the defaulted argument first, followed by a *args and optional **kwargs parameters (through which any other arguments or dictionaries of options could be passed during calls to this function).
This is, as I said hideously ugly. Please don't ever publish any code like this or share it with anyone in the Python community. I'm only showing it here as a sort of strange educational exercise.
An instance method is like a function in Python. However, it exists within the namespace of a class (thus it must be accessed via an instance ... myobject.foo() for example) and it is called with a reference to "self" (analagous to the "this" pointer in C++) as the first argument. Also there's a method resolution process which causes the interpreter to search the namespace of the instance, then it's class, and then each of the parent classes and so on ... up through the inheritance tree.
An unbound function is called with whatever arguments you pass to it. There can't bee any sort of automatically pre-pended object/instance reference to the argument list. Thus, writing a function with an initial argument named "self" is meaningless. (It's legal because Python doesn't place any special meaning on the name "self." But meaningless because callers to your function would have to manually supply some sort of object reference to the argument list and it's not at all clear what that should be. Just some bizarre "Generic_Object" which then floats around in the global variable space?).
I hope that clarifies things a bit. It sounds like you're suffering from some very fundamental misconceptions about how Python and other object-oriented systems work.
("Monkey patching" is a term used to describe the direct manipulation of an objects attributes -- or "instance variables" by code that is not part of the class hierarchy of which the object is an instance).
As another alternative, you can make the functions into bound class methods like so:
class _FooImpl(object):
a = "Hello "
#classmethod
def foo(cls, param):
return cls.a + param
foo = _FooImpl.foo
# later...
print foo("World") # yes, Hello World
# and if you have to change an attribute:
foo.im_self.a = "Goodbye "
If you want functions to share attribute namespaecs, you just make them part of the same class. If not, give each its own class.
What exactly are you hoping "self" would point to, if the function is defined outside of any class? If your function needs some global information to execute properly, you need to send this information to the function in the form of an argument.
If you want your function to be context aware, you need to declare it within the scope of an object.