We Keep Coding

Understanding iterable types in comparisons

Recently I ran into cosmologicon's pywats and now try to understand part about fun with iterators:
>>> a = 2, 1, 3
>>> sorted(a) == sorted(a)
True
>>> reversed(a) == reversed(a)
False
Ok, sorted(a) returns a list and sorted(a) == sorted(a) becomes just a two lists comparision. But reversed(a) returns reversed object. So why these reversed objects are different? And id's comparision makes me even more confused:
>>> id(reversed(a)) == id(reversed(a))
True

The basic reason why id(reversed(a) == id(reversed(a) returns True , whereas reversed(a) == reversed(a) returns False , can be seen from the below example using custom classes -
>>> class CA:
... def __del__(self):
... print('deleted', self)
... def __init__(self):
... print('inited', self)
...
>>> CA() == CA()
inited <__main__.CA object at 0x021B8050>
inited <__main__.CA object at 0x021B8110>
deleted <__main__.CA object at 0x021B8050>
deleted <__main__.CA object at 0x021B8110>
False
>>> id(CA()) == id(CA())
inited <__main__.CA object at 0x021B80F0>
deleted <__main__.CA object at 0x021B80F0>
inited <__main__.CA object at 0x021B80F0>
deleted <__main__.CA object at 0x021B80F0>
True
As you can see when you did customobject == customobject , the object that was created on the fly was not destroyed until after the comparison occurred, this is because that object was required for the comparison.
But in case of id(co) == id(co) , the custom object created was passed to id() function, and then only the result of id function is required for comparison , so the object that was created has no reference left, and hence the object was garbage collected, and then when the Python interpreter recreated a new object for the right side of == operation, it reused the space that was freed previously. Hence, the id for both came as same.
This above behavior is an implementation detail of CPython (it may/may not differ in other implementations of Python) . And you should never rely on the equality of ids . For example in the below case it gives the wrong result -
>>> a = [1,2,3]
>>> b = [4,5,6]
>>> id(reversed(a)) == id(reversed(b))
True
The reason for this is again as explained above (garbage collection of the reversed object created for reversed(a) before creation of reversed object for reversed(b)).
If the lists are large, I think the most memory efficient and most probably the fastest method to compare equality for two iterators would be to use all() built-in function along with zip() function for Python 3.x (or itertools.izip() for Python 2.x).
Example for Python 3.x -
all(x==y for x,y in zip(aiterator,biterator))
Example for Python 2.x -
from itertools import izip
all(x==y for x,y in izip(aiterator,biterator))
This is because all() short circuits at the first False value is encounters, and `zip() in Python 3.x returns an iterator which yields out the corresponding elements from both the different iterators. This does not need to create a separate list in memory.
Demo -
>>> a = [1,2,3]
>>> b = [4,5,6]
>>> all(x==y for x,y in zip(reversed(a),reversed(b)))
False
>>> all(x==y for x,y in zip(reversed(a),reversed(a)))
True

sorted returns a list, whereas reversed returns a reversed object and is a different object. If you were to cast the result of reversed to a list before comparison, they will be equal.
In [8]: reversed(a)
Out[8]: <reversed at 0x2c98d30>
In [9]: reversed(a)
Out[9]: <reversed at 0x2c989b0>

reversed returns an iterable that doesn't implement a specific __eq__ operator and therefore is compared using identity.
The confusion about id(reversed(a)) == id(reversed(a)) is because after evaluating the first id(...) call the iterable can be disposed (nothing references it) and the second iterable may be reallocated at the very same memory address when the second id(...) call is done. This is however just a coincidence.
Try
ra1 = reversed(a)
ra2 = reversed(a)
and compare id(ra1) with id(ra2) and you will see they are different numbers (because in this case the iterable objects cannot be deallocated as they're referenced by ra1/ra2 variables).

You may try list(reversed(a)) ==list(reversed(a)) will return True
list(reversed(a))
[3, 2, 1]
once try
>>> v = id(reversed(a))
>>> n = id(reversed(a))
>>> v == n
False
again
>>> v = id(reversed(a))
>>> n = id(reversed(a))
>>> n1 = id(reversed(a))
>>> v == n1
True

what is the difference between del a[:] and a = [] when I want to empty a list called a in python? [duplicate]

This question already has answers here:
Different ways of deleting lists
(6 answers)
Closed 7 years ago.
Please what is the most efficient way of emptying a list?
I have a list called a = [1,2,3]. To delete the content of the list I usually write a = [ ]. I came across a function in python called del. I want to know if there is a difference between del a [:] and what I use.

There is a difference, and it has to do with whether that list is referenced from multiple places/names.
>>> a = [1, 2, 3]
>>> b = a
>>> del a[:]
>>> print(b)
[]
>>> a = [1, 2, 3]
>>> b = a
>>> a = []
>>> print(b)
[1, 2, 3]
Using del a[:] clears the existing list, which means anywhere it's referenced will become an empty list.
Using a = [] sets a to point to a new empty list, which means that other places the original list is referenced will remain non-empty.
The key to understanding here is to realize that when you assign something to a variable, it just makes that name point to a thing. Things can have multiple names, and changing what a name points to doesn't change the thing itself.

This can probably best be shown:
>>> a = [1, 2, 3]
>>> id(a)
45556280
>>> del a[:]
>>> id(a)
45556280
>>> b = [4, 5, 6]
>>> id(b)
45556680
>>> b = []
>>> id(b)
45556320
When you do a[:] you are referring to all elements within the list "assigned" to a. The del statement removes references to objects. So, doing del a[:] is saying "remove all references to objects from within the list assigned to a". The list itself has not changed. We can see this with the id function, which gives us a number representing an object in memory. The id of the list before using del and after remains the same, indicating the same list object is assigned to a.
On the other hand, when we assign a non-empty list to b and then assign a new empty list to b, the id changes. This is because we have actually moved the b reference from the existing [4, 5, 6] list to the new [] list.
Beyond just the identity of the objects you are dealing with, there are other things to be aware of:
>>> a = [1, 2, 3]
>>> b = a
>>> del a[:]
>>> print a
[]
>>> print b
[]
Both b and a refer to the same list. Removing the elements from the a list without changing the list itself mutates the list in place. As b references the same object, we see the same result there. If you did a = [] instead, then a will refer to a new empty list while b continues to reference the [1, 2, 3] list.

>>> list1 = [1,2,3,4,5]
>>> list2 = list1
To get a better understanding, let us see with the help of pictures what happens internally.
>>> list1 = [1,2,3,4,5]
This creates a list object and assigns it to list1.
>>> list2 = list1
The list object which list1 was referring to is also assigned to list2.
Now, lets look at the methods to empty an list and what actually happens internally.
METHOD-1: Set to empty list [] :
>>> list1 = []
>>> list2
[1,2,3,4,5]
This does not delete the elements of the list but deletes the reference to the list. So, list1 now points to an empty list but all other references will have access to that old list1.
This method just creates a new list object and assigns it to list1. Any other references will remain.
METHOD-2: Delete using slice operator[:] :
>>> del list1[:]
>>> list2
[]
When we use the slice operator to delete all the elements of the list, then all the places where it is referenced, it becomes an empty list. So list2 also becomes an empty list.

Well, del uses just a little less space in the computer as the person above me implied. The computer still accepts the variable as the same code, except with a different value. However, when you variable is assigned something else, the computer assigns a completely different code ID to it in order to account for the change in memory required.

python: list append - each time create a new list?

What's the underlining machanism of append in python's list?
IE. a = [1,2,3]; a.append(3) then a is [1,2,3,3]
Dose python actually create a new list tmp [1,2,3,3], then assign that list to a?

No, it doesn't. List is mutable type in Python which means the append operation would change the original structure in memory.
>>> a = [1,2,3]
>>> id(a)
4325551112
>>> a.append(4)
>>> id(a)
4325551112
this is it.

No it modifies the original list , but does not return a new one, as lists are mutable.
to give you an example:
a = [1,2,3]
print a.append(4)
will return None, as append does not return a new list.
print a.pop(1)
will print 1 as pop removes and returns 1

Why list_a = list_a.append(1) will produce a NoneType instead of through an Error?

If this would pop out an error, as a python newbee, I wouldn't need to find out my mistake for a whole hour!
So I'd like to ask What's the consideration and benefit of this mechanism?

append() is a method on a list which just adds an item to the end of a list and returns nothing. In other words, it modifies a list "in place":
>>> l = [1,2,3]
>>> type(l)
<type 'list'>
>>> l = l.append(4)
>>> print l
None
>>> type(l)
<type 'NoneType'>
And, since NoneType is actually a valid and perfectly normal variable type, l becomes None without any errors - just think about it as l = None.
Hope that helps.

Does a slicing operation give me a deep or shallow copy?

The official Python docs say that using the slicing operator and assigning in Python makes a shallow copy of the sliced list.
But when I write code for example:
o = [1, 2, 4, 5]
p = o[:]
And when I write:
id(o)
id(p)
I get different id's and also appending one one list does not reflect in the other list. Isn't it creating a deep copy or is there somewhere I am going wrong?

You are creating a shallow copy, because nested values are not copied, merely referenced. A deep copy would create copies of the values referenced by the list too.
Demo:
>>> lst = [{}]
>>> lst_copy = lst[:]
>>> lst_copy[0]['foo'] = 'bar'
>>> lst_copy.append(42)
>>> lst
[{'foo': 'bar'}]
>>> id(lst) == id(lst_copy)
False
>>> id(lst[0]) == id(lst_copy[0])
True
Here the nested dictionary is not copied; it is merely referenced by both lists. The new element 42 is not shared.
Remember that everything in Python is an object, and names and list elements are merely references to those objects. A copy of a list creates a new outer list, but the new list merely receives references to the exact same objects.
A proper deep copy creates new copies of each and every object contained in the list, recursively:
>>> from copy import deepcopy
>>> lst_deepcopy = deepcopy(lst)
>>> id(lst_deepcopy[0]) == id(lst[0])
False

You should know that tests using is or id can be misleading of whether a true copy is being made with immutable and interned objects such as strings, integers and tuples that contain immutables.
Consider an easily understood example of interned strings:
>>> l1=['one']
>>> l2=['one']
>>> l1 is l2
False
>>> l1[0] is l2[0]
True
Now make a shallow copy of l1 and test the immutable string:
>>> l3=l1[:]
>>> l3 is l1
False
>>> l3[0] is l1[0]
True
Now make a copy of the string contained by l1[0]:
>>> s1=l1[0][:]
>>> s1
'one'
>>> s1 is l1[0] is l2[0] is l3[0]
True # they are all the same object
Try a deepcopy where every element should be copied:
>>> from copy import deepcopy
>>> l4=deepcopy(l1)
>>> l4[0] is l1[0]
True
In each case, the string 'one' is being interned into Python's internal cache of immutable strings and is will show that they are the same (they have the same id). It is implementation and version dependent of what gets interned and when it does, so you cannot depend on it. It can be a substantial memory and performance enhancement.
You can force an example that does not get interned instantly:
>>> s2=''.join(c for c in 'one')
>>> s2==l1[0]
True
>>> s2 is l1[0]
False
And then you can use the Python intern function to cause that string to refer to the cached object if found:
>>> l1[0] is s2
False
>>> s2=intern(s2)
>>> l1[0] is s2
True
Same applies to tuples of immutables:
>>> t1=('one','two')
>>> t2=t1[:]
>>> t1 is t2
True
>>> t3=deepcopy(t1)
>>> t3 is t2 is t1
True
And mutable lists of immutables (like integers) can have the list members interred:
>>> li1=[1,2,3]
>>> li2=deepcopy(li1)
>>> li2 == li1
True
>>> li2 is li1
False
>>> li1[0] is li2[0]
True
So you may use python operations that you KNOW will copy something but the end result is another reference to an interned immutable object. The is test is only a dispositive test of a copy being made IF the items are mutable.