We Keep Coding

Understanding iterable types in comparisons

Recently I ran into cosmologicon's pywats and now try to understand part about fun with iterators:
>>> a = 2, 1, 3
>>> sorted(a) == sorted(a)
True
>>> reversed(a) == reversed(a)
False
Ok, sorted(a) returns a list and sorted(a) == sorted(a) becomes just a two lists comparision. But reversed(a) returns reversed object. So why these reversed objects are different? And id's comparision makes me even more confused:
>>> id(reversed(a)) == id(reversed(a))
True

The basic reason why id(reversed(a) == id(reversed(a) returns True , whereas reversed(a) == reversed(a) returns False , can be seen from the below example using custom classes -
>>> class CA:
... def __del__(self):
... print('deleted', self)
... def __init__(self):
... print('inited', self)
...
>>> CA() == CA()
inited <__main__.CA object at 0x021B8050>
inited <__main__.CA object at 0x021B8110>
deleted <__main__.CA object at 0x021B8050>
deleted <__main__.CA object at 0x021B8110>
False
>>> id(CA()) == id(CA())
inited <__main__.CA object at 0x021B80F0>
deleted <__main__.CA object at 0x021B80F0>
inited <__main__.CA object at 0x021B80F0>
deleted <__main__.CA object at 0x021B80F0>
True
As you can see when you did customobject == customobject , the object that was created on the fly was not destroyed until after the comparison occurred, this is because that object was required for the comparison.
But in case of id(co) == id(co) , the custom object created was passed to id() function, and then only the result of id function is required for comparison , so the object that was created has no reference left, and hence the object was garbage collected, and then when the Python interpreter recreated a new object for the right side of == operation, it reused the space that was freed previously. Hence, the id for both came as same.
This above behavior is an implementation detail of CPython (it may/may not differ in other implementations of Python) . And you should never rely on the equality of ids . For example in the below case it gives the wrong result -
>>> a = [1,2,3]
>>> b = [4,5,6]
>>> id(reversed(a)) == id(reversed(b))
True
The reason for this is again as explained above (garbage collection of the reversed object created for reversed(a) before creation of reversed object for reversed(b)).
If the lists are large, I think the most memory efficient and most probably the fastest method to compare equality for two iterators would be to use all() built-in function along with zip() function for Python 3.x (or itertools.izip() for Python 2.x).
Example for Python 3.x -
all(x==y for x,y in zip(aiterator,biterator))
Example for Python 2.x -
from itertools import izip
all(x==y for x,y in izip(aiterator,biterator))
This is because all() short circuits at the first False value is encounters, and `zip() in Python 3.x returns an iterator which yields out the corresponding elements from both the different iterators. This does not need to create a separate list in memory.
Demo -
>>> a = [1,2,3]
>>> b = [4,5,6]
>>> all(x==y for x,y in zip(reversed(a),reversed(b)))
False
>>> all(x==y for x,y in zip(reversed(a),reversed(a)))
True

sorted returns a list, whereas reversed returns a reversed object and is a different object. If you were to cast the result of reversed to a list before comparison, they will be equal.
In [8]: reversed(a)
Out[8]: <reversed at 0x2c98d30>
In [9]: reversed(a)
Out[9]: <reversed at 0x2c989b0>

reversed returns an iterable that doesn't implement a specific __eq__ operator and therefore is compared using identity.
The confusion about id(reversed(a)) == id(reversed(a)) is because after evaluating the first id(...) call the iterable can be disposed (nothing references it) and the second iterable may be reallocated at the very same memory address when the second id(...) call is done. This is however just a coincidence.
Try
ra1 = reversed(a)
ra2 = reversed(a)
and compare id(ra1) with id(ra2) and you will see they are different numbers (because in this case the iterable objects cannot be deallocated as they're referenced by ra1/ra2 variables).

You may try list(reversed(a)) ==list(reversed(a)) will return True
list(reversed(a))
[3, 2, 1]
once try
>>> v = id(reversed(a))
>>> n = id(reversed(a))
>>> v == n
False
again
>>> v = id(reversed(a))
>>> n = id(reversed(a))
>>> n1 = id(reversed(a))
>>> v == n1
True

my_set.copy().add( something ) why return None [duplicate]

What am I doing wrong here?
a = set().add(1)
print a # Prints `None`
I'm trying to add the number 1 to the empty set.

It is a convention in Python that methods that mutate sequences return None.
Consider:
>>> a_list = [3, 2, 1]
>>> print a_list.sort()
None
>>> a_list
[1, 2, 3]
>>> a_dict = {}
>>> print a_dict.__setitem__('a', 1)
None
>>> a_dict
{'a': 1}
>>> a_set = set()
>>> print a_set.add(1)
None
>>> a_set
set([1])
Some may consider this convention "a horrible misdesign in Python", but the Design and History FAQ gives the reasoning behind this design decision (with respect to lists):
Why doesn’t list.sort() return the sorted list?
In situations where performance matters, making a copy of the list
just to sort it would be wasteful. Therefore, list.sort() sorts the
list in place. In order to remind you of that fact, it does not return
the sorted list. This way, you won’t be fooled into accidentally
overwriting a list when you need a sorted copy but also need to keep
the unsorted version around.
In Python 2.4 a new built-in function – sorted() – has been added.
This function creates a new list from a provided iterable, sorts it
and returns it.
Your particular problems with this feature come from a misunderstanding of good ways to create a set rather than a language misdesign. As Lattyware points out, in Python versions 2.7 and later you can use a set literal a = {1} or do a = set([1]) as per Sven Marnach's answer.
Parenthetically, I like Ruby's convention of placing an exclamation point after methods that mutate objects, but I find Python's approach acceptable.

The add() method adds an element to the set, but it does not return the set again -- it returns None.
a = set()
a.add(1)
or better
a = set([1])
would work.

Because add() is modifing your set in place returning None:
>>> empty = set()
>>> print(empty.add(1))
None
>>> empty
set([1])

Another way to do it that is relatively simple would be:
a = set()
a = set() | {1}
this creates a union between your set a and a set with 1 as the element
print(a) yields {1} then because a would now have all elements of both a and {1}

You should do this:
a = set()
a.add(1)
print a
Notice that you're assigning to a the result of adding 1, and the add operation, as defined in Python, returns None - and that's what is getting assigned to a in your code.
Alternatively, you can do this for initializing a set:
a = set([1, 2, 3])

The add method updates the set, but returns None.
a = set()
a.add(1)
print a

You are assigning the value returned by set().add(1) to a. This value is None, as add() does not return any value, it instead acts in-place on the list.
What you wanted to do was this:
a = set()
a.add(1)
print(a)
Of course, this example is trivial, but Python does support set literals, so if you really wanted to do this, it's better to do:
a = {1}
print(a)
The curly brackets denote a set (although be warned, {} denotes an empty dict, not an empty set due to the fact that curly brackets are used for both dicts and sets (dicts are separated by the use of the colon to separate keys and values.)

Alternatively to a = set() | {1} consider "in-place" operator:
a = set()
a |= {1}

What do boolean operations on lists mean?

I was going through the source of pyftpdlib and I found this:
if self.rejected_users and self.allowed_users:
raise AuthorizerError("rejected_users and allowed_users options are mutually exclusive")
rejected_users and allowed_users are lists.
What's confusing me is how the and operator operates on two lists. I'd appreciate it if someone helped me out.

All objects in Python have a boolean 'value'; they are either true or false in a boolean context.
Empty lists are false. This applies to all sequences and containers, including tuples, sets, dictionaries and strings.
Numeric 0 is false too, so 0, 0.0, 0j are all false as well, as are None and of course False itself:
>>> bool([])
False
>>> bool([1, 2])
True
>>> bool(0)
False
>>> bool('')
False
Everything else is considered true in a boolean context; so a list that is not empty is true, and two non-empty lists together with and is considered true as well.
You can make custom types look like empty containers by implementing __len__() and returning 0, or look like a number by implementing __nonzero__()* and returning False when the instance is to be the boolean equivalent of numeric zero.
Just remember that and and or shortcircuit; if the first expression locks in the result then that value is returned and the second expression is ignored altogether. For and, that means that in the expression x and y, y is ignored if x is a false value (like an empty list), because the whole expression can never be true in that case. For x or y, y is ignored if x is a true value.
These rules are all covered by the Boolean operations reference documentation.
*In Python 3, use __bool__ instead.

Empty list evaluates to False and non-empty list evaluates to True.
if list1 and list2:
is equivalent to:
if list1 is not empty and list2 is not empty:
List of falsy values in python:
None
False
zero of any numeric type, for example, 0, 0L, 0.0, 0j.
any empty sequence, for example, '', (), [].
any empty mapping, for example, {}.
instances of user-defined classes, if the class defines a
__nonzero__() or __len__() method, when that method returns the integer zero or bool value False.
All other values are considered true — so objects of many types are always true.

If list_a is empty, then list_a and list_b will evaluate to list_a, otherwise it will evaluate to list_b. For example:
>>> [1] and [2]
[2]
>>> [] and [2]
[]

In addition to #Ashwini answer, you can use bool inbuilt function to check what a given object will evaluate to.
>>> bool([])
False
>>> bool([1,2,3])
True
>>> bool('')
False
>>> bool('hello')
True

It looks to me as though it's a logical boolean statement. In Python, when building an If statement, boolean statements have an interesting logical shorthand:
bool = True;
If bool:
return "Blah!"
Is the same as:
bool = True;
If bool==True:
return "Blah!"
The statement If *[boolean]* is a cooler way of saying If *[boolean]* is True. In that same way, the statement If *![boolean]* is a cooler way of saying If *[boolean]* is False, because ! is the logical NOT operator.It takes a little time to acclimate to this logic but once you do, it makes your code a lot nicer looking.
So why do the lists do this like a boolean value should?
Since Python doesn't have explicit type declaration, every data type has a boolean representation. For lists like rejected_users and allowed_users, if the lists are empty {}'s, it returns False and if NOT empty, it returns True. Thus, the english translation of
if self.rejected_users and self.allowed_users:
should read:
if self.rejected_users is NOT empty AND if self.allowed_users is NOT empty:
I hope this helps.

"in" statement behavior in lists vs. strings

In Python, asking if a substring exists in a string is pretty straightforward:
>>> their_string = 'abracadabra'
>>> our_string = 'cad'
>>> our_string in their_string
True
However, checking if these same characters are "in" a list fails:
>>> ours, theirs = map(list, [our_string, their_string])
>>> ours in theirs
False
>>> ours, theirs = map(tuple, [our_string, their_string])
>>> ours in theirs
False
I wasn't able to find any obvious reason why checking for elements "in" an ordered (even immutable) iterable would behave differently than a different type of ordered, immutable iterable.

For container types such as lists and tuples, x in container checks if x is an item in the container. Thus with ours in theirs, Python checks if ours is an item in theirs and finds that it is False.
Remember that a list could contain a list. (e.g [['a','b','c'], ...])
>>> ours = ['a','b','c']
>>> theirs = [['a','b','c'], 1, 2]
>>> ours in theirs
True

Are you looking to see if 'cad' is in any of the strings in a list of strings? That would like something like:
stringsToSearch = ['blah', 'foo', 'bar', 'abracadabra']
if any('cad' in s for s in stringsToSearch):
# 'cad' was in at least one string in the list
else:
# none of the strings in the list contain 'cad'

From the Python documentation, https://docs.python.org/2/library/stdtypes.html for sequences:
x in s True if an item of s is equal to x, else False (1)
x not in s False if an item of s is equal to x, else True (1)
(1) When s is a string or Unicode string object the in and not in operations act like a substring test.
For user defined classes, the __contains__ method implements this in test. list and tuple implement the basic notion. string has the added notion of 'substring'. string is a special case among the basic sequences.

python 2.7 : knowing if it's a list of list of list OR a list of list

a script is feeding me with list of list of list or list of list.
What I plan to do is call this
test = myList[0][0][0]
and if an exception is raised I'll know that it's a list of list.
Is there a better/proper way to do this?
Thanks.

I'm not sure if it's better/proper, but you can also test whether something is a list with isinstance or type functions.
For example
a = [1,2,3]
b = (1,2,3) # Not a list
type(a) == type([]) # True
type(b) == type([]) # False
type(a) is list # True
type(b) is list # False
isinstance(a, list) # True
isinstance(b, list) # False
The first method is probably not ideal, the second would probably be better if you were to use type, but I think the general consensus is that isinstance is generally better.
EDIT: Some discussion about the difference between the two approaches
So, I guess your code would look something like:
if(isinstance(myList[0][0], list)):
# Use myList[0][0][0]
else:
# Use myList[0][0]