I'm teaching myself Python ahead of starting a new job. Its a Django job, so I have to stick to 2.7. As such, I'm reading Beginning Python by Hetland and don't understand his example of using slices to replicate list.extend() functionality.
First, he shows the extend method by
a = [1, 2, 3]
b = [4, 5, 6]
a.extend(b)
produces [1, 2, 3, 4, 5, 6]
Next, he demonstrates extend by slicing via
a = [1, 2, 3]
b = [4, 5, 6]
a[len(a):] = b
which produces the exact same output as the first example.
How does this work? A has a length of 3, and the terminating slice index point is empty, signifying that it runs to the end of the list. How do the b values get added to a?
Python's slice-assignment syntax means "make this slice equal to this value, expanding or shrinking the list if necessary". To fully understand it you may want to try out some other slice values:
a = [1, 2, 3]
b = [4, 5, 6]
First, lets replace part of A with B:
a[1:2] = b
print(a) # prints [1, 4, 5, 6, 3]
Instead of replacing some values, you can add them by assigning to a zero-length slice:
a[1:1] = b
print(a) # prints [1, 4, 5, 6, 2, 3]
Any slice that is "out of bounds" instead simply addresses an empty area at one end of the list or the other (too large positive numbers will address the point just off the end while too large negative numbers will address the point just before the start):
a[200:300] = b
print(a) # prints [1, 2, 3, 4, 5, 6]
Your example code simply uses the most "accurate" out of bounds slice at the end of the list. I don't think that is code you'd use deliberately for extending, but it might be useful as an edge case that you don't need to handle with special logic.
It's simply an extension of normal indexing.
>>> L
[1, 2, 3, 4, 5]
>>> L[2] = 42
>>> L
[1, 2, 42, 4, 5]
The __setitem__() method detects that a slice is being used instead of a normal index and behaves appropriately.
a = [1, 2, 3]
b = [4, 5, 6]
a[len(a):] = b
means element in a from position len(a) are elements in b. Which means extending a with b.
For a demonstration, consider looking at a subclass of list:
from __future__ import print_function # so I can run on Py 3 and Py 2
class EdList(list):
def __setitem__(self,index,value):
print('setitem: index={}, value={}'.format(index,value))
list.__setitem__(self,index,value)
print(self)
def __setslice__(self,i,j,seq):
print('setslice: i:{}, j:{}, seq:{}'.format(i,j,seq))
self.__setitem__(slice(i,j),seq)
Running on Python 3:
>>> a=EdList(range(10))
>>> a[300000:]=[1,2,3]
setitem: index=slice(300000, None, None), value=[1, 2, 3]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3]
>>> a[1:1]=[4,5,6]
setitem: index=slice(1, 1, None), value=[4, 5, 6]
[0, 4, 5, 6, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3]
Running on Python 2:
>>> a=EdList(range(10))
>>> a[300000:]=[1,2,3]
setslice: i:300000, j:9223372036854775807, seq:[1, 2, 3]
setitem: index=slice(300000, 9223372036854775807, None), value=[1, 2, 3]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3]
>>> a[1:1]=[4,5,6]
setslice: i:1, j:1, seq:[4, 5, 6]
setitem: index=slice(1, 1, None), value=[4, 5, 6]
[0, 4, 5, 6, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3]
It is confusing when you are first learning it, but you will learn to love it I think.
Related
I have a list of integers. Each number can appear several times, the list is unordered.
I want to get the list of relative sizes. Meaning, if for example the original list is [2, 5, 7, 7, 3, 10] then the desired output is [0, 2, 3, 3, 1, 4]
Because 2 is the zero'th smallest number in the original list, 3 is one'th, etc.
Any clear easy way to do this?
Try a list comprehension with dictionary and also use set for getting unique values, like below:
>>> lst = [2, 5, 7, 7, 3, 10]
>>> newl = dict(zip(range(len(set(lst))), sorted(set(lst))))
>>> [newl[i] for i in lst]
[0, 2, 3, 3, 1, 4]
>>>
Or use index:
>>> lst = [2, 5, 7, 7, 3, 10]
>>> newl = sorted(set(lst))
>>> [newl.index(i) for i in lst]
[0, 2, 3, 3, 1, 4]
>>>
In other words, I want to accomplish something like the following:
a = [1, 2, 3, 7, 8]
b = [4, 5, 6]
# some magic here to insert list b into list a at index 3 so that
a = [1, 2, 3, 4, 5, 6, 7, 8]
You can assign to a slice of list a like so:
>>> a = [1, 2, 3, 7, 8]
>>> b = [4, 5, 6]
>>> a[3:3] = b
>>> a
[1, 2, 3, 4, 5, 6, 7, 8]
>>>
The [3:3] may look weird, but it is necessary to have the items in list b be inserted into list a correctly. Otherwise, if we were to assign to an index of a, list b itself would be inserted:
>>> a = [1, 2, 3, 7, 8]
>>> b = [4, 5, 6]
>>> a[3] = b
>>> a
[1, 2, 3, [4, 5, 6], 8]
>>>
I tried to find a docs link which explicitly mentions this behavior, but was unsuccessful (please feel free to add one if you can find it). So, I'll just say that doing a[3:3] = b tells Python to take the items in list b and place them in the section of list a represented by [3:3]. Moreover, Python will extend list a as necessary to accommodate this operation.
In short, this is just yet another awesome feature of Python. :)
Try using the sort method as follows:
>>> a = [1,2,3,7,8]
>>> b = [4,5,6]
>>> a = sorted(a+b)
>>> a
[1, 2, 3, 4, 5, 6, 7, 8]
What is the best way to do this:
>>> replace2([1, 2, 3, 4, 5, 6])
[2, 1, 4, 3, 6, 5]
def replace2inplace(lst):
lst[1::2], lst[::2] = lst[::2], lst[1::2]
This uses slice assignment and slice step sizes to swap every pair in the list around, in-place:
>>> somelst = [1, 2, 3, 4, 5, 6]
>>> replace2inplace(somelst)
>>> somelst
[2, 1, 4, 3, 6, 5]
Otherwise you could use some itertools tricks:
from itertools import izip, chain
def replace2copy(lst):
lst1, lst2 = tee(iter(lst), 2)
return list(chain.from_iterable(izip(lst[1::2], lst[::2])))
which gives:
>>> replace2([1, 2, 3, 4, 5, 6])
[2, 1, 4, 3, 6, 5]
with the list() call optional; if you only need to loop over the result the generator is enough:
from itertools import izip, chain, islice, tee
def replace2gen(lst):
lst1, lst2 = tee(iter(lst))
return chain.from_iterable(izip(islice(lst1, 1, None, 2), islice(lst2, None, None, 2)))
for i in replace2gen([1, 2, 3, 4, 5, 6]):
print i
where replace2gen() can take arbitrary iterators too.
Out-of-place version:
def replace2(lst):
return [x for pair in zip(lst[1::2], lst[::2]) for x in pair]
My choice:
x = range(1,7)
res = [e for e in itertools.chain(*zip(x[1::2],x[0::2]))]
>>> a = [1, 2, 3, 4, 5, 6]
>>> sum([[x+1,x] for x in a if x&1 == True],[])
[2, 1, 4, 3, 6, 5]
EDIT: Some further explanation was requested:
The code steps through each element in the list a and, if the element is odd (x&1 == True) it puts that element and the next element into a list in reverse order ([x+1,x]).
With out the sum(...,[]) function we would have
[[2, 1], [4, 3], [6, 5]]
The sum(...,[]) function removes the internal square brackets giving
[2, 1, 4, 3, 6, 5]
This can be done more generally by using the index of the list rather than its value:
>>> a = [1, 2, 3, 4, 5, 6]
>>> sum([[a[x],a[x-1]] for x in range(len(a)) if x&1 == True],[])
[2, 1, 4, 3, 6, 5]
However, this will remove the last element of the list if its length is not even.
Can't figure out how to do this in a pretty way :
I have a list of n elements,
I want to access every m elements of the list.
For example : [1, 2, 3, 4, 5] and m = 2 would give
[2, 4]
I can do it simply with a loop, but ins't there a more "pythonic" way?
Thanks by advance !
EDIT :
Seems like I forgot something.
I want, not only get those values but modify them.
I tried slicing a[::2] = 3, but it doesn't work. . .
I'm searching for something similar
Slicing syntax does this for you:
>>> my_list = range(10)
>>> my_list
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> my_list[::2]
[0, 2, 4, 6, 8]
>>> my_list[1::2]
[1, 3, 5, 7, 9]
Here's a way to wrap a list to get the original assignment behavior you wanted, but I'm not sure I'd recommend it:
class AssignableSlice(list):
def __setitem__(self, i, v):
if isinstance(i, slice):
for ii in xrange(*i.indices(len(self))):
self[ii] = v
else:
super(AssignableSlice, self).__setitem__(i, v)
a = AssignableSlice(range(10))
print a
a[::2] = 3
print a
a[1::3] = 99
print a
produces:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[3, 1, 3, 3, 3, 5, 3, 7, 3, 9]
[3, 99, 3, 3, 99, 5, 3, 99, 3, 9]
Ned's answer shows how to use slices to access a portion of the list. You can also assign to a slice, but you need to assign a list to the slice, for example:
>>> my_list = range(5)
>>> my_list
[0, 1, 2, 3, 4]
>>> my_list[::2]
[0, 2, 4]
>>> my_list[::2] = [0, 0, 0]
>>> my_list
[0, 1, 0, 3, 0]
Note that when the step in your slice is anything besides the default of 1 the list that you assign needs to be the same length, however with a default step you can actually change the size of the list with slice assignment:
>>> my_list = range(5)
>>> my_list
[0, 1, 2, 3, 4]
>>> my_list[:1]
[0]
>>> my_list[:1] = [4, 3, 2] # replace the first item with 3 new items
>>> my_list
[4, 3, 2, 1, 2, 3, 4]
>>> my_list[2:5]
[2, 1, 2]
>>> my_list[2:5] = [] # remove the middle three items from the list
>>> my_list
[4, 3, 3, 4]
I juste found a way to do what I want using slicing.
The following :
candidates[::2] = [1] * len(candidates[::2])
will replace every 2 elements of candidates by 1
I just spent a very long time debugging an issue in python, using the web.py framework, and it has me wondering about a way to check this kind of thing in the future.
In short, one of the methods of web.py's database class was returning a storage object that, I can only surmise, was a reference to a value in memory, instead of a copy. The result was that no matter what I tried to do, I could not append multiple results of a query into a list, because the list just wound up with copies of the same row of data.
To solve it, I had to create another dictionary and convert all values in the web.py storage object explicitly to another datatype (I just used strings to get it working) so that it would be forced to create a new object.
In code, this is what was happening:
>>> result = db.query(query_params)
>>> for row in result:
>>> print row
... result_list.append(row)
{"result" : "from", "row" : "one"}
{"result" : "from", "row" : "two"}
>>> print result_list
[{"result":"from", "row" : "two"}, {"result" : "from", "row" : "two}]
This is what clued me into the fact that this was some sort of reference issue. The list was storing the location of "row", instead of a copy of it.
The first thing I tried was something along the lines of:
copy = {}
for row in result:
for value in row:
copy[value] = row[value]
result_list.append(copy)
But this lead to the same problem.
I only found a solution by tweaking the above to read:
copy[value] = str(row[value])
So, my question is two-fold, really:
Is there an easy way to tell how something is being stored and/or passed around?
Is there a way to explicitly request a copy instead of a reference?
Use copy. It allows simple shallow- and deep-copying of objects in python:
import copy
result_list.append(copy.copy(row))
Welcome to how Python works. It's easy to identify which objects are passed by reference in Python: all of them are. It's just that many (integers, strings, tuples) are immutable, so you don't really notice when more than one name is pointing to them since you can't change them.
If you need a copy, make a copy explicitly. Usually this can be done using the type's constructor:
newlist = list(oldlist)
newdict = dict(olddict)
Lists can also do it with a slice, which you'll often see:
newlist = oldlist[:]
Dictionaries have a copy() method:
newdict = olddict.copy()
These are "shallow" copies: a new container is made, but the items inside are still the original references. This can bite you for, example, with lists of lists. A module called copy exists that contains functions for copying nearly any object, and a deepcopy function that will also copy the contained objects to any depth.
I don't know anything about web.py, but most likely a result_set.append(dict(row)) is what you were looking for.
See also: copy.copy() and copy.deepcopy()
Well, if you're curious about identity you can always check using the is operator:
>>> help('is')
The reason your attempt didn't work is because you were creating the dictionary outside the loop, so of course you're going to have problems:
>>> mydict = {}
>>> for x in xrange(3):
... d = mydict
... print d is mydict
...
...
True
True
True
No matter how many times you add things to d or mydict, mydict will always be the same object.
Others have commented that you should use copy, but they've failed to address your underlying issue - you don't seem to understand reference types.
In Python, everything is an object. There are two basic types - mutable and immutable. Immutable objects are things like strings, numbers, and tuples. You're not allowed to change these objects in memory:
>>> x = 3
>>> y = x
>>> x is y
True
Now x and y refer to the same object (not just have the same value).
>>> x += 4
Because 3 is an integer and immutable, this operation does not modify the value that's stored in x, what it actually does is adds 3 and 4 and finds out that it results in the value of 7, so now x "points" to 7.
>>> y
3
>>> x
7
>>> x is y
False
>>> y = x
>>> x is y
True
With mutable objects, you can modify them in place:
>>> mylist = [1,2,3]
>>> mylist[0] = 3
>>> mylist
[3, 2, 3]
You'll have a lot easier time if you stop thinking about variables in Python as storing values in variables, and instead think of them as name tags - you know the ones that say "Hello, My Name Is"?
So essentially what's happening in your code is that the object being returned through your loop is the same one each time.
Here's an example:
>>> def spam_ref():
... thing = []
... count = 0
... while count < 10:
... count += 1
... thing.append(count)
... yield thing
...
>>> for a in spam_ref():
... print a
...
[1]
[1, 2]
[1, 2, 3]
[1, 2, 3, 4]
[1, 2, 3, 4, 5]
[1, 2, 3, 4, 5, 6]
[1, 2, 3, 4, 5, 6, 7]
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> spam = []
>>> for a in spam_ref():
... spam.append(a)
...
So if you look at the output of the loop, you'd think that spam now contains
[1]
[1, 2]
[1, 2, 3]
[1, 2, 3, 4]
[1, 2, 3, 4, 5]
[1, 2, 3, 4, 5, 6]
[1, 2, 3, 4, 5, 6, 7]
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
But it doesn't:
>>> for a in spam:
... print a
...
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Because a list is mutable so the generator function was able to modify it in place. In your case, you have a dictionary that's being returned, so as others have mentioned, you'll need to use some method of copying the dictionary.
For further reading, check out Python Objects and Call By Object over at effbot.
Either of two should work:
result = db.query(query_params).list()
result = list(db.query(query_params))
db.query and db.select return iterator, you need to convert it into list.