How to get hold of the object missing an attribute - python

Suppose we try to access a non-existing attribute:
>>> {'foo': 'bar'}.gte('foo') # well, I meant “get”!
Python’s AttributeError only has the attribute args with a string containing the finished error message: 'dict' object has no attribute 'gte'
Using the inspect and/or traceback modules with sys.last_traceback, is there a way to get hold of the actual dict object?
>>> offending_object = get_attributeerror_obj(sys.last_traceback)
>>> dir(offending_object)
[...
'clear',
'copy',
'fromkeys',
'get', # ah, here it is!
'items',
...]
Edit: since the cat is out of the bag anyway, I’ll share my findings and code (please don’t solve this and submit to PyPI, please ;))
The AttributeError is created here, which shows that there’s clearly no reference to the originating object attached.
Here the code with the same placeholder function:
import sys
import re
import difflib
AE_MSG_RE = re.compile(r"'(\w+)' object has no attribute '(\w+)'")
def get_attributeerror_obj(tb):
???
old_hook = sys.excepthook
def did_you_mean_hook(type, exc, tb):
old_hook(type, exc, tb)
if type is AttributeError:
match = AE_MSG_RE.match(exc.args[0])
sook = match.group(2)
raising_obj = get_attributeerror_obj(tb)
matches = difflib.get_close_matches(sook, dir(raising_obj))
if matches:
print('\n\nDid you mean?', matches[0], file=sys.stderr)
sys.excepthook = did_you_mean_hook

It's not the answer you want, but I'm pretty sure you can't... at least not with sys.excepthook. This is because the reference counts are decremented as the frame is unwound, so it's perfectly valid for the object to be garbage collected before sys.excepthook is called. In fact, this is what happens in CPython:
import sys
class X:
def __del__(self):
print("deleting")
def error():
X().wrong
old_hook = sys.excepthook
def did_you_mean_hook(type, exc, tb):
print("Error!")
sys.excepthook = did_you_mean_hook
error()
#>>> deleting
#>>> Error!
That said, it isn't always the case. Because the exception object points to the frame, if your code looks like:
def error():
x = X()
x.wrong
x cannot yet be collected. x is owned by the frame, and the frame is alive. But since I've already proven that there is no explicit reference made to this object, it's not ever obvious what to do. For example,
def error():
foo().wrong
may or may not have an object that has survived, and the only feasible way to find out is to run foo... but even then you have problems with side effects.
So no, this is not possible. If you don't mind going to any lengths whatsoever, you'll probably end up having to rewrite the AST on load (akin to FuckIt.py). You don't want to do that, though.
My suggestion would be to try using a linter to get the names of all known classes and their methods. You can use this to reverse-engineer the traceback string to get the class and incorrect method, and then run a fuzzy match to find the suggestion.

Adding my 2 cents as I successfully (so far) tried to do something similar for DidYouMean-Python.
The trick here is that it is pretty much the one case where the error message contains enough information to infer what you actually meant. Indeed, what really matters here is that you tried to call gte on a dict object : you need the type, not the object itself.
If you had written {'foo': 'bar'}.get('foob') the situation would be much trickier to handle and I'd be happy to know if anyone had a solution.
Step one
Check that you are handling an AttributeError (using the first argument of the hook).
Step two
Retrieve the relevant information from the message (using the second argument). I did this with regexp. Please note that this exception can take multiple forms depending on the version of Python, the object you are calling the method on, etc.
So far, my regexp is : "^'?(\w+)'? (?:object|instance) has no attribute '(\w+)'$"
Step three
Get the type object corresponding to the type ('dict' in your case) so that you can call dir() on it. A dirty solution would be just use eval(type) but you can do better and cleaner by reusing the information in the trace (third argument of your hook) : the last element of the trace contains the frame in which the exception occured and in that frame, the type was properly defined (either as a local type, a global type or a builtin).
Once you have the type object, you just need to call dir() on it and extract the suggestion you like the most.
Please let me know if you need more details on what I did.

Related

What does 'function' object is not scriptable mean in python? [duplicate]

Which types of objects fall into the domain of "subscriptable"?
It basically means that the object implements the __getitem__() method. In other words, it describes objects that are "containers", meaning they contain other objects. This includes strings, lists, tuples, and dictionaries.
Off the top of my head, the following are the only built-ins that are subscriptable:
string: "foobar"[3] == "b"
tuple: (1,2,3,4)[3] == 4
list: [1,2,3,4][3] == 4
dict: {"a":1, "b":2, "c":3}["c"] == 3
But mipadi's answer is correct - any class that implements __getitem__ is subscriptable
The meaning of subscript in computing is:
"a symbol (notionally written as a subscript but in practice usually not) used in a program, alone or with others, to specify one of the elements of an array."
Now, in the simple example given by #user2194711 we can see that the appending element is not able to be a part of the list because of two reasons:-
1) We are not really calling the method append; because it needs () to call it.
2) The error is indicating that the function or method is not subscriptable; means they are not indexable like a list or sequence.
Now see this:-
>>> var = "myString"
>>> def foo(): return 0
...
>>> var[3]
't'
>>> foo[3]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'function' object is not subscriptable
That means there are no subscripts or say elements in function like they occur in sequences; and we cannot access them like we do, with the help of [].
Also; as mipadi said in his answer; It basically means that the object implements the __getitem__() method. (if it is subscriptable).
Thus the error produced:
arr.append["HI"]
TypeError: 'builtin_function_or_method' object is not subscriptable
A scriptable object is an object that records the operations done to it and it can store them as a "script" which can be replayed.
For example, see: Application Scripting Framework
Now, if Alistair didn't know what he asked and really meant "subscriptable" objects (as edited by others), then (as mipadi also answered) this is the correct one:
A subscriptable object is any object that implements the __getitem__ special method (think lists, dictionaries).
I had this same issue. I was doing
arr = []
arr.append["HI"]
So using [ was causing error. It should be arr.append("HI")
As a corollary to the earlier answers here, very often this is a sign that you think you have a list (or dict, or other subscriptable object) when you do not.
For example, let's say you have a function which should return a list;
def gimme_things():
if something_happens():
return ['all', 'the', 'things']
Now when you call that function, and something_happens() for some reason does not return a True value, what happens? The if fails, and so you fall through; gimme_things doesn't explicitly return anything -- so then in fact, it will implicitly return None. Then this code:
things = gimme_things()
print("My first thing is {0}".format(things[0]))
will fail with "NoneType object is not subscriptable" because, well, things is None and so you are trying to do None[0] which doesn't make sense because ... what the error message says.
There are two ways to fix this bug in your code -- the first is to avoid the error by checking that things is in fact valid before attempting to use it;
things = gimme_things()
if things:
print("My first thing is {0}".format(things[0]))
else:
print("No things") # or raise an error, or do nothing, or ...
or equivalently trap the TypeError exception;
things = gimme_things()
try:
print("My first thing is {0}".format(things[0]))
except TypeError:
print("No things") # or raise an error, or do nothing, or ...
Another is to redesign gimme_things so that you make sure it always returns a list. In this case, that's probably the simpler design because it means if there are many places where you have a similar bug, they can be kept simple and idiomatic.
def gimme_things():
if something_happens():
return ['all', 'the', 'things']
else: # make sure we always return a list, no matter what!
logging.info("Something didn't happen; return empty list")
return []
Of course, what you put in the else: branch depends on your use case. Perhaps you should raise an exception when something_happens() fails, to make it more obvious and explicit where something actually went wrong? Adding exceptions to your own code is an important way to let yourself know exactly what's up when something fails!
(Notice also how this latter fix still doesn't completely fix the bug -- it prevents you from attempting to subscript None but things[0] is still an IndexError when things is an empty list. If you have a try you can do except (TypeError, IndexError) to trap it, too.)
Basically this error will appear in case you are modifying or adding any field after type casting for the mentioned object instead of doing it before.

Differentiate Garbage collector and __del__ in Python [duplicate]

I can't really think of any reason why Python needs the del keyword (and most languages seem to not have a similar keyword). For instance, rather than deleting a variable, one could just assign None to it. And when deleting from a dictionary, a del method could be added.
Is there a reason to keep del in Python, or is it a vestige of Python's pre-garbage collection days?
Firstly, you can del other things besides local variables
del list_item[4]
del dictionary["alpha"]
Both of which should be clearly useful. Secondly, using del on a local variable makes the intent clearer. Compare:
del foo
to
foo = None
I know in the case of del foo that the intent is to remove the variable from scope. It's not clear that foo = None is doing that. If somebody just assigned foo = None I might think it was dead code. But I instantly know what somebody who codes del foo was trying to do.
There's this part of what del does (from the Python Language Reference):
Deletion of a name removes the binding of that name from the local or global namespace
Assigning None to a name does not remove the binding of the name from the namespace.
(I suppose there could be some debate about whether removing a name binding is actually useful, but that's another question.)
One place I've found del useful is cleaning up extraneous variables in for loops:
for x in some_list:
do(x)
del x
Now you can be sure that x will be undefined if you use it outside the for loop.
Deleting a variable is different than setting it to None
Deleting variable names with del is probably something used rarely, but it is something that could not trivially be achieved without a keyword. If you can create a variable name by writing a=1, it is nice that you can theoretically undo this by deleting a.
It can make debugging easier in some cases as trying to access a deleted variable will raise an NameError.
You can delete class instance attributes
Python lets you write something like:
class A(object):
def set_a(self, a):
self.a=a
a=A()
a.set_a(3)
if hasattr(a, "a"):
print("Hallo")
If you choose to dynamically add attributes to a class instance, you certainly want to be able to undo it by writing
del a.a
There is a specific example of when you should use del (there may be others, but I know about this one off hand) when you are using sys.exc_info() to inspect an exception. This function returns a tuple, the type of exception that was raised, the message, and a traceback.
The first two values are usually sufficient to diagnose an error and act on it, but the third contains the entire call stack between where the exception was raised and where the the exception is caught. In particular, if you do something like
try:
do_evil()
except:
exc_type, exc_value, tb = sys.exc_info()
if something(exc_value):
raise
the traceback, tb ends up in the locals of the call stack, creating a circular reference that cannot be garbage collected. Thus, it is important to do:
try:
do_evil()
except:
exc_type, exc_value, tb = sys.exc_info()
del tb
if something(exc_value):
raise
to break the circular reference. In many cases where you would want to call sys.exc_info(), like with metaclass magic, the traceback is useful, so you have to make sure that you clean it up before you can possibly leave the exception handler. If you don't need the traceback, you should delete it immediately, or just do:
exc_type, exc_value = sys.exc_info()[:2]
To avoid it all together.
Just another thinking.
When debugging http applications in framework like Django, the call stack full of useless and messed up variables previously used, especially when it's a very long list, could be very painful for developers. so, at this point, namespace controlling could be useful.
Using "del" explicitly is also better practice than assigning a variable to None. If you attempt to del a variable that doesn't exist, you'll get a runtime error but if you attempt to set a variable that doesn't exist to None, Python will silently set a new variable to None, leaving the variable you wanted deleted where it was. So del will help you catch your mistakes earlier
del is often seen in __init__.py files. Any global variable that is defined in an __init__.py file is automatically "exported" (it will be included in a from module import *). One way to avoid this is to define __all__, but this can get messy and not everyone uses it.
For example, if you had code in __init__.py like
import sys
if sys.version_info < (3,):
print("Python 2 not supported")
Then your module would export the sys name. You should instead write
import sys
if sys.version_info < (3,):
print("Python 2 not supported")
del sys
To add a few points to above answers:
del x
Definition of x indicates r -> o (a reference r pointing to an object o) but del x changes r rather than o. It is an operation on the reference (pointer) to object rather than the object associated with x. Distinguishing between r and o is key here.
It removes it from locals().
Removes it from globals() if x belongs there.
Removes it from the stack frame (removes the reference physically from it, but the object itself resides in object pool and not in the stack frame).
Removes it from the current scope. It is very useful to limit the span of definition of a local variable, which otherwise can cause problems.
It is more about declaration of the name rather than definition of content.
It affects where x belongs to, not where x points to. The only physical change in memory is this. For example if x is in a dictionary or list, it (as a reference) is removed from there(and not necessarily from the object pool). In this example, the dictionary it belongs is the stack frame (locals()), which overlaps with globals().
I've found del to be useful for pseudo-manual memory management when handling large data with Numpy. For example:
for image_name in large_image_set:
large_image = io.imread(image_name)
height, width, depth = large_image.shape
large_mask = np.all(large_image == <some_condition>)
# Clear memory, make space
del large_image; gc.collect()
large_processed_image = np.zeros((height, width, depth))
large_processed_image[large_mask] = (new_value)
io.imsave("processed_image.png", large_processed_image)
# Clear memory, make space
del large_mask, large_processed_image; gc.collect()
This can be the difference between bringing a script to a grinding halt as the system swaps like mad when the Python GC can't keep up, and it running perfectly smooth below a loose memory threshold that leaves plenty of headroom to use the machine to browse and code while it's working.
Force closing a file after using numpy.load:
A niche usage perhaps but I found it useful when using numpy.load to read a file. Every once in a while I would update the file and need to copy a file with the same name to the directory.
I used del to release the file and allow me to copy in the new file.
Note I want to avoid the with context manager as I was playing around with plots on the command line and didn't want to be pressing tab a lot!
See this question.
I would like to elaborate on the accepted answer to highlight the nuance between setting a variable to None versus removing it with del:
Given the variable foo = 'bar', and the following function definition:
def test_var(var):
if var:
print('variable tested true')
else:
print('variable tested false')
Once initially declared, test_var(foo) yields variable tested true as expected.
Now try:
foo = None
test_var(foo)
which yields variable tested false.
Contrast this behavior with:
del foo
test_var(foo)
which now raises NameError: name 'foo' is not defined.
As an example of what del can be used for, I find it useful i situations like this:
def f(a, b, c=3):
return '{} {} {}'.format(a, b, c)
def g(**kwargs):
if 'c' in kwargs and kwargs['c'] is None:
del kwargs['c']
return f(**kwargs)
# g(a=1, b=2, c=None) === '1 2 3'
# g(a=1, b=2) === '1 2 3'
# g(a=1, b=2, c=4) === '1 2 4'
These two functions can be in different packages/modules and the programmer doesn't need to know what default value argument c in f actually have. So by using kwargs in combination with del you can say "I want the default value on c" by setting it to None (or in this case also leave it).
You could do the same thing with something like:
def g(a, b, c=None):
kwargs = {'a': a,
'b': b}
if c is not None:
kwargs['c'] = c
return f(**kwargs)
However I find the previous example more DRY and elegant.
When is del useful in python?
You can use it to remove a single element of an array instead of the slice syntax x[i:i+1]=[]. This may be useful if for example you are in os.walk and wish to delete an element in the directory. I would not consider a keyword useful for this though, since one could just make a [].remove(index) method (the .remove method is actually search-and-remove-first-instance-of-value).
I think one of the reasons that del has its own syntax is that replacing it with a function might be hard in certain cases given it operates on the binding or variable and not the value it references. Thus if a function version of del were to be created a context would need to be passed in. del foo would need to become globals().remove('foo') or locals().remove('foo') which gets messy and less readable. Still I say getting rid of del would be good given its seemingly rare use. But removing language features/flaws can be painful. Maybe python 4 will remove it :)
The "del" command is very useful for controlling data in an array, for example:
elements = ["A", "B", "C", "D"]
# Remove first element.
del elements[:1]
print(elements)
Output:
['B', 'C', 'D']
del deletes the binding of the variable and its object that it points to.
>>> a = ['a', 'b', 'c']
>>> b = a
>>> del a
>>> b
['a', 'b', 'c']
>>> a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
A simple use case I can think of would be in case you have used a built-in function name as a variable, and you want to use that function after it has been already "overridden" by your variable name.
t = ('a', "letter")
value, type = t
print(value, type)
del type
print(type(value))
Output:
a letter
<class 'str'>
Yet another niche usage:
In pyroot with ROOT5 or ROOT6, "del" may be useful to remove a python object that referred to a no-longer existing C++ object. This allows the dynamic lookup of pyroot to find an identically-named C++ object and bind it to the python name. So you can have a scenario such as:
import ROOT as R
input_file = R.TFile('inputs/___my_file_name___.root')
tree = input_file.Get('r')
tree.Draw('hy>>hh(10,0,5)')
R.gPad.Close()
R.hy # shows that hy is still available. It can even be redrawn at this stage.
tree.Draw('hy>>hh(3,0,3)') # overwrites the C++ object in ROOT's namespace
R.hy # shows that R.hy is None, since the C++ object it pointed to is gone
del R.hy
R.hy # now finds the new C++ object
Hopefully, this niche will be closed with ROOT7's saner object management.
del is removing the variable from the current scope unless it is re-initialized. Setting it to None keeps it in the current scope.
a = "python string"
print(a)
del a
print(a)
a = "new python string"
print(a)
Output:
python string
Traceback (most recent call last):
File "testing.py", line 4, in <module>
print(a)
NameError: name 'a' is not defined
As I have not seen a interactive console answer, I'll be showing one.
When foo=None that reference and the object exist, it's not pointing to it.
While del foo destroys the object and reference too.
So if you do something like this if foo is None and it was deleted it will rise NameError as the the reference, it's object and everything in between was deleted with del
Deletion of a target list recursively deletes each target, from left to right.
Meanwhile foo=None is just a reference pointing to None so the reference is still kicking, same for the object.
[...]In Python, variables are references to objects and any variable can reference any object[...]
Link to quote 1
Link to quote 2
Another niche case, but useful.
from getpass import getpass
pass = getpass()
token = get_auth_token(pass)
del pass
# Assume more code here...
After the deletion of the pass variable, you don't run the risk of it being printed out later by mistake, or otherwise ending up in a log or stack trace.
Here goes my 2 cents contribution:
I have a optimization problem where I use a Nlopt library for it.
I initializing the class and some of its methods, I was using in several other parts of the code.
I was having ramdom results even if applying the same numerical problem.
I just realized that by doing it, some spurius data was contained in the object when it should have no issues at all. After using del, I guess the memory is being properly cleared and it might be an internal issue to that class where some variables might not be liking to be reused without proper constructor.
Once I had to use:
del serial
serial = None
because using only:
serial = None
didn't release the serial port fast enough to immediately open it again.
From that lesson I learned that del really meant: "GC this NOW! and wait until it's done" and that is really useful in a lot of situations. Of course, you may have a system.gc.del_this_and_wait_balbalbalba(obj).
del is the equivalent of "unset" in many languages
and as a cross reference point moving from another language to python..
people tend to look for commands that do the same thing that they used to do in their first language...
also
setting a var to "" or none doesn't really remove the var from scope..it just empties its value
the name of the var itself would still be stored in memory...why?!?
in a memory intensive script..keeping trash behind its just a no no
and anyways...every language out there has some form of an "unset/delete" var function..why not python?

How to handle unexpected argument type in python

Background
I am new to python and I am writing a simple function but I am also interested in learning to do things the correct / pythonic way as I progress in my journey.
Lets consider the function below
def test_func(nested_lists,val):
return
I am expecting two arguments. One argument would be a list containing more lists. Something like this [[1,2,3,],[4,5,6,]...]. The second argument could be a value like 1.
If someone say for instance passes in a single value as the first argument and an array as the second argument. My code as it is currently returning the correct output which is 0 , However is there another way that i should be handle this?
For example should I be doing something like this
if(type(value) == list):
return 0
Or do i not need to do anything because my function is returning 0 anyway.
I know this maybe a very basic question so please forgive me but coming from a java background I am new to python so i am not sure how to handle such scenarios in python.
The other answer illustrates the proper way to check in advance for problems you can foresee. I'll provide a different approach.
The idiomatic solution in python is to "ask forgiveness, not permission". There are a lot of things that can go wrong, and where other languages might ask you to foresee all those problems and address them manually, python encourages just handling them as they happen. I would recommend doing:
def test_func(nested_lists, val):
try:
...
except TypeError:
# do whatever error-handling behavior you need to
# either throw a custom exception or return a specific value or whatever
return 0
and then designing your code in such a way that, if nested_lists and values are not compatible types, then they throw a TypeError (e.g. trying to iterate through nested_lists should fail if nested_lists is not a list. You can experiment with this behavior in a python console, but in general trying to do something to a variable that doesn't work because it's not the right type will produce a TypeError).
If your current code is working correctly, there is no pressing need to change anything. However, there are some reasons you might want to code more defensively;
If the code will seem to work correctly when you pass in bogus values, it would be better if it raised an exception instead of return a bogus value. The responsibility to call it correctly lies squarely with the caller, but enforcing it can help make sure the code is correct.
if not isinstance(nested_lists,list):
raise ValueError('Need a list, got {0!r}'.format(nested_lists))
This has the drawback that it hardcodes list as the type for the first argument; properly reusable code should work with any type, as long as it has the required methods and behaviors to remain compatible with your implementation. Perhaps instead check for a behavior:
try:
something involving nested_lists[0][0]
except (IndexError, AttributeError):
raise ValueError('Expected nested list but got {0!r}'.format(nested_lists))
(The try is not strictly necessary here; but see below.)
If you get a traceback when you call the code incorrectly, but it is opaque or misleading, it is more helpful to catch and explicitly point out the error earlier. #or example, the snippet above (without the try wrapper) would produce
Traceback (most recent call last):
module __main__ line 141
traceback.print_exc()
module <module> line 1
test_func(1,1)
module <module> line 2
AttributeError: 'int' object has no attribute '__getitem__'
which is somewhat unobvious to debug.
If the code will be used by third parties, both of the above considerations will be more important from a support point of view, too.
Notice how the code raises an exception when called incorrectly. This is generally better than silently returning some garbage value, and the caller can similarly trap the error with a try/except if this is well-defined (i.e. documented!) behavior.
Finally, since Python 3.5, you have the option to use type annotations:
def test_func(nested_lists: list, val: int) -> int:
...
As noted in the documentation, the core language does not (yet?) enforce these type checks, but they can help static code analysis tools point out possible errors.

When is del useful in Python?

I can't really think of any reason why Python needs the del keyword (and most languages seem to not have a similar keyword). For instance, rather than deleting a variable, one could just assign None to it. And when deleting from a dictionary, a del method could be added.
Is there a reason to keep del in Python, or is it a vestige of Python's pre-garbage collection days?
Firstly, you can del other things besides local variables
del list_item[4]
del dictionary["alpha"]
Both of which should be clearly useful. Secondly, using del on a local variable makes the intent clearer. Compare:
del foo
to
foo = None
I know in the case of del foo that the intent is to remove the variable from scope. It's not clear that foo = None is doing that. If somebody just assigned foo = None I might think it was dead code. But I instantly know what somebody who codes del foo was trying to do.
There's this part of what del does (from the Python Language Reference):
Deletion of a name removes the binding of that name from the local or global namespace
Assigning None to a name does not remove the binding of the name from the namespace.
(I suppose there could be some debate about whether removing a name binding is actually useful, but that's another question.)
One place I've found del useful is cleaning up extraneous variables in for loops:
for x in some_list:
do(x)
del x
Now you can be sure that x will be undefined if you use it outside the for loop.
Deleting a variable is different than setting it to None
Deleting variable names with del is probably something used rarely, but it is something that could not trivially be achieved without a keyword. If you can create a variable name by writing a=1, it is nice that you can theoretically undo this by deleting a.
It can make debugging easier in some cases as trying to access a deleted variable will raise an NameError.
You can delete class instance attributes
Python lets you write something like:
class A(object):
def set_a(self, a):
self.a=a
a=A()
a.set_a(3)
if hasattr(a, "a"):
print("Hallo")
If you choose to dynamically add attributes to a class instance, you certainly want to be able to undo it by writing
del a.a
There is a specific example of when you should use del (there may be others, but I know about this one off hand) when you are using sys.exc_info() to inspect an exception. This function returns a tuple, the type of exception that was raised, the message, and a traceback.
The first two values are usually sufficient to diagnose an error and act on it, but the third contains the entire call stack between where the exception was raised and where the the exception is caught. In particular, if you do something like
try:
do_evil()
except:
exc_type, exc_value, tb = sys.exc_info()
if something(exc_value):
raise
the traceback, tb ends up in the locals of the call stack, creating a circular reference that cannot be garbage collected. Thus, it is important to do:
try:
do_evil()
except:
exc_type, exc_value, tb = sys.exc_info()
del tb
if something(exc_value):
raise
to break the circular reference. In many cases where you would want to call sys.exc_info(), like with metaclass magic, the traceback is useful, so you have to make sure that you clean it up before you can possibly leave the exception handler. If you don't need the traceback, you should delete it immediately, or just do:
exc_type, exc_value = sys.exc_info()[:2]
To avoid it all together.
Just another thinking.
When debugging http applications in framework like Django, the call stack full of useless and messed up variables previously used, especially when it's a very long list, could be very painful for developers. so, at this point, namespace controlling could be useful.
Using "del" explicitly is also better practice than assigning a variable to None. If you attempt to del a variable that doesn't exist, you'll get a runtime error but if you attempt to set a variable that doesn't exist to None, Python will silently set a new variable to None, leaving the variable you wanted deleted where it was. So del will help you catch your mistakes earlier
del is often seen in __init__.py files. Any global variable that is defined in an __init__.py file is automatically "exported" (it will be included in a from module import *). One way to avoid this is to define __all__, but this can get messy and not everyone uses it.
For example, if you had code in __init__.py like
import sys
if sys.version_info < (3,):
print("Python 2 not supported")
Then your module would export the sys name. You should instead write
import sys
if sys.version_info < (3,):
print("Python 2 not supported")
del sys
To add a few points to above answers:
del x
Definition of x indicates r -> o (a reference r pointing to an object o) but del x changes r rather than o. It is an operation on the reference (pointer) to object rather than the object associated with x. Distinguishing between r and o is key here.
It removes it from locals().
Removes it from globals() if x belongs there.
Removes it from the stack frame (removes the reference physically from it, but the object itself resides in object pool and not in the stack frame).
Removes it from the current scope. It is very useful to limit the span of definition of a local variable, which otherwise can cause problems.
It is more about declaration of the name rather than definition of content.
It affects where x belongs to, not where x points to. The only physical change in memory is this. For example if x is in a dictionary or list, it (as a reference) is removed from there(and not necessarily from the object pool). In this example, the dictionary it belongs is the stack frame (locals()), which overlaps with globals().
I've found del to be useful for pseudo-manual memory management when handling large data with Numpy. For example:
for image_name in large_image_set:
large_image = io.imread(image_name)
height, width, depth = large_image.shape
large_mask = np.all(large_image == <some_condition>)
# Clear memory, make space
del large_image; gc.collect()
large_processed_image = np.zeros((height, width, depth))
large_processed_image[large_mask] = (new_value)
io.imsave("processed_image.png", large_processed_image)
# Clear memory, make space
del large_mask, large_processed_image; gc.collect()
This can be the difference between bringing a script to a grinding halt as the system swaps like mad when the Python GC can't keep up, and it running perfectly smooth below a loose memory threshold that leaves plenty of headroom to use the machine to browse and code while it's working.
Force closing a file after using numpy.load:
A niche usage perhaps but I found it useful when using numpy.load to read a file. Every once in a while I would update the file and need to copy a file with the same name to the directory.
I used del to release the file and allow me to copy in the new file.
Note I want to avoid the with context manager as I was playing around with plots on the command line and didn't want to be pressing tab a lot!
See this question.
I would like to elaborate on the accepted answer to highlight the nuance between setting a variable to None versus removing it with del:
Given the variable foo = 'bar', and the following function definition:
def test_var(var):
if var:
print('variable tested true')
else:
print('variable tested false')
Once initially declared, test_var(foo) yields variable tested true as expected.
Now try:
foo = None
test_var(foo)
which yields variable tested false.
Contrast this behavior with:
del foo
test_var(foo)
which now raises NameError: name 'foo' is not defined.
As an example of what del can be used for, I find it useful i situations like this:
def f(a, b, c=3):
return '{} {} {}'.format(a, b, c)
def g(**kwargs):
if 'c' in kwargs and kwargs['c'] is None:
del kwargs['c']
return f(**kwargs)
# g(a=1, b=2, c=None) === '1 2 3'
# g(a=1, b=2) === '1 2 3'
# g(a=1, b=2, c=4) === '1 2 4'
These two functions can be in different packages/modules and the programmer doesn't need to know what default value argument c in f actually have. So by using kwargs in combination with del you can say "I want the default value on c" by setting it to None (or in this case also leave it).
You could do the same thing with something like:
def g(a, b, c=None):
kwargs = {'a': a,
'b': b}
if c is not None:
kwargs['c'] = c
return f(**kwargs)
However I find the previous example more DRY and elegant.
When is del useful in python?
You can use it to remove a single element of an array instead of the slice syntax x[i:i+1]=[]. This may be useful if for example you are in os.walk and wish to delete an element in the directory. I would not consider a keyword useful for this though, since one could just make a [].remove(index) method (the .remove method is actually search-and-remove-first-instance-of-value).
I think one of the reasons that del has its own syntax is that replacing it with a function might be hard in certain cases given it operates on the binding or variable and not the value it references. Thus if a function version of del were to be created a context would need to be passed in. del foo would need to become globals().remove('foo') or locals().remove('foo') which gets messy and less readable. Still I say getting rid of del would be good given its seemingly rare use. But removing language features/flaws can be painful. Maybe python 4 will remove it :)
The "del" command is very useful for controlling data in an array, for example:
elements = ["A", "B", "C", "D"]
# Remove first element.
del elements[:1]
print(elements)
Output:
['B', 'C', 'D']
del deletes the binding of the variable and its object that it points to.
>>> a = ['a', 'b', 'c']
>>> b = a
>>> del a
>>> b
['a', 'b', 'c']
>>> a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
A simple use case I can think of would be in case you have used a built-in function name as a variable, and you want to use that function after it has been already "overridden" by your variable name.
t = ('a', "letter")
value, type = t
print(value, type)
del type
print(type(value))
Output:
a letter
<class 'str'>
Yet another niche usage:
In pyroot with ROOT5 or ROOT6, "del" may be useful to remove a python object that referred to a no-longer existing C++ object. This allows the dynamic lookup of pyroot to find an identically-named C++ object and bind it to the python name. So you can have a scenario such as:
import ROOT as R
input_file = R.TFile('inputs/___my_file_name___.root')
tree = input_file.Get('r')
tree.Draw('hy>>hh(10,0,5)')
R.gPad.Close()
R.hy # shows that hy is still available. It can even be redrawn at this stage.
tree.Draw('hy>>hh(3,0,3)') # overwrites the C++ object in ROOT's namespace
R.hy # shows that R.hy is None, since the C++ object it pointed to is gone
del R.hy
R.hy # now finds the new C++ object
Hopefully, this niche will be closed with ROOT7's saner object management.
del is removing the variable from the current scope unless it is re-initialized. Setting it to None keeps it in the current scope.
a = "python string"
print(a)
del a
print(a)
a = "new python string"
print(a)
Output:
python string
Traceback (most recent call last):
File "testing.py", line 4, in <module>
print(a)
NameError: name 'a' is not defined
As I have not seen a interactive console answer, I'll be showing one.
When foo=None that reference and the object exist, it's not pointing to it.
While del foo destroys the object and reference too.
So if you do something like this if foo is None and it was deleted it will rise NameError as the the reference, it's object and everything in between was deleted with del
Deletion of a target list recursively deletes each target, from left to right.
Meanwhile foo=None is just a reference pointing to None so the reference is still kicking, same for the object.
[...]In Python, variables are references to objects and any variable can reference any object[...]
Link to quote 1
Link to quote 2
Another niche case, but useful.
from getpass import getpass
pass = getpass()
token = get_auth_token(pass)
del pass
# Assume more code here...
After the deletion of the pass variable, you don't run the risk of it being printed out later by mistake, or otherwise ending up in a log or stack trace.
Here goes my 2 cents contribution:
I have a optimization problem where I use a Nlopt library for it.
I initializing the class and some of its methods, I was using in several other parts of the code.
I was having ramdom results even if applying the same numerical problem.
I just realized that by doing it, some spurius data was contained in the object when it should have no issues at all. After using del, I guess the memory is being properly cleared and it might be an internal issue to that class where some variables might not be liking to be reused without proper constructor.
Once I had to use:
del serial
serial = None
because using only:
serial = None
didn't release the serial port fast enough to immediately open it again.
From that lesson I learned that del really meant: "GC this NOW! and wait until it's done" and that is really useful in a lot of situations. Of course, you may have a system.gc.del_this_and_wait_balbalbalba(obj).
del is the equivalent of "unset" in many languages
and as a cross reference point moving from another language to python..
people tend to look for commands that do the same thing that they used to do in their first language...
also
setting a var to "" or none doesn't really remove the var from scope..it just empties its value
the name of the var itself would still be stored in memory...why?!?
in a memory intensive script..keeping trash behind its just a no no
and anyways...every language out there has some form of an "unset/delete" var function..why not python?

What does it mean if a Python object is "subscriptable" or not?

Which types of objects fall into the domain of "subscriptable"?
It basically means that the object implements the __getitem__() method. In other words, it describes objects that are "containers", meaning they contain other objects. This includes strings, lists, tuples, and dictionaries.
Off the top of my head, the following are the only built-ins that are subscriptable:
string: "foobar"[3] == "b"
tuple: (1,2,3,4)[3] == 4
list: [1,2,3,4][3] == 4
dict: {"a":1, "b":2, "c":3}["c"] == 3
But mipadi's answer is correct - any class that implements __getitem__ is subscriptable
The meaning of subscript in computing is:
"a symbol (notionally written as a subscript but in practice usually not) used in a program, alone or with others, to specify one of the elements of an array."
Now, in the simple example given by #user2194711 we can see that the appending element is not able to be a part of the list because of two reasons:-
1) We are not really calling the method append; because it needs () to call it.
2) The error is indicating that the function or method is not subscriptable; means they are not indexable like a list or sequence.
Now see this:-
>>> var = "myString"
>>> def foo(): return 0
...
>>> var[3]
't'
>>> foo[3]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'function' object is not subscriptable
That means there are no subscripts or say elements in function like they occur in sequences; and we cannot access them like we do, with the help of [].
Also; as mipadi said in his answer; It basically means that the object implements the __getitem__() method. (if it is subscriptable).
Thus the error produced:
arr.append["HI"]
TypeError: 'builtin_function_or_method' object is not subscriptable
A scriptable object is an object that records the operations done to it and it can store them as a "script" which can be replayed.
For example, see: Application Scripting Framework
Now, if Alistair didn't know what he asked and really meant "subscriptable" objects (as edited by others), then (as mipadi also answered) this is the correct one:
A subscriptable object is any object that implements the __getitem__ special method (think lists, dictionaries).
I had this same issue. I was doing
arr = []
arr.append["HI"]
So using [ was causing error. It should be arr.append("HI")
As a corollary to the earlier answers here, very often this is a sign that you think you have a list (or dict, or other subscriptable object) when you do not.
For example, let's say you have a function which should return a list;
def gimme_things():
if something_happens():
return ['all', 'the', 'things']
Now when you call that function, and something_happens() for some reason does not return a True value, what happens? The if fails, and so you fall through; gimme_things doesn't explicitly return anything -- so then in fact, it will implicitly return None. Then this code:
things = gimme_things()
print("My first thing is {0}".format(things[0]))
will fail with "NoneType object is not subscriptable" because, well, things is None and so you are trying to do None[0] which doesn't make sense because ... what the error message says.
There are two ways to fix this bug in your code -- the first is to avoid the error by checking that things is in fact valid before attempting to use it;
things = gimme_things()
if things:
print("My first thing is {0}".format(things[0]))
else:
print("No things") # or raise an error, or do nothing, or ...
or equivalently trap the TypeError exception;
things = gimme_things()
try:
print("My first thing is {0}".format(things[0]))
except TypeError:
print("No things") # or raise an error, or do nothing, or ...
Another is to redesign gimme_things so that you make sure it always returns a list. In this case, that's probably the simpler design because it means if there are many places where you have a similar bug, they can be kept simple and idiomatic.
def gimme_things():
if something_happens():
return ['all', 'the', 'things']
else: # make sure we always return a list, no matter what!
logging.info("Something didn't happen; return empty list")
return []
Of course, what you put in the else: branch depends on your use case. Perhaps you should raise an exception when something_happens() fails, to make it more obvious and explicit where something actually went wrong? Adding exceptions to your own code is an important way to let yourself know exactly what's up when something fails!
(Notice also how this latter fix still doesn't completely fix the bug -- it prevents you from attempting to subscript None but things[0] is still an IndexError when things is an empty list. If you have a try you can do except (TypeError, IndexError) to trap it, too.)
Basically this error will appear in case you are modifying or adding any field after type casting for the mentioned object instead of doing it before.

Categories