Is there are a way to use Numpy's multidimensional array slicing without using the [slice, slice] syntax?
I need to be able to use it from normal function calls, but I haven't found a way to use the slice object to do it.
I cannot use the syntax [(slice,slice)] for my program because [] is special syntax outside of regular function calls.
The language I am using is Hy, a Lisp for Python, and it does not support this syntax. More importantly, it shouldn't support this syntax. Numpy, however, doesn't seem to support multidimensional slicing without using the [] syntax.
What's tripping me up is that the mix of C and Python in the Numpy source makes it difficult to discern how the [slice,slice] is implemented.
It may not even be possible to circumvent this syntax.
EDIT:
The answer provided below by #Joe Kington allows one to slice Numpy matrices like so:
x = np.array([list(range(5)) for x in list(range(5))])
x.getitem(slice(1,4))
array([[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4]])
x.getitem(tuple([slice(1,4),slice(1,4)]))
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
From your description, it seems like you're asking what function calls are used to implement slicing and slice assignment.
Python uses the "special" methods __getitem__ and __setitem__ to implement and/or allow customization of how slicing works. Any class that implements these can be sliced. There's actually nothing numpy-specific about this.
In other words
x = arr[4:10, 9:15, ::-1]
x[0] = 100
is translated into
x = arr.__getitem__((slice(4, 6), slice(9, 10), slice(None, None, -1)))
x.__setitem__(0, 100)
For example:
class Foo(object):
def __getitem__(self, index):
print 'Getting', index
def __setitem__(self, index, val):
print 'Setting', index, 'to', val
f = Foo()
print 'Getting...'
f[:]
f[4:10, ::-1, ...]
print 'Equivalently:'
f.__getitem__(slice(None))
f.__getitem__((slice(4, 10), slice(None, None, -1), Ellipsis))
print 'Setting...'
f[0] = 1
f[5:10, 100] = 2
f[...] = 100
print 'Equivalently:'
f.__setitem__(0, 1)
f.__setitem__((slice(5,10), 100), 2)
f.__setitem__(Ellipsis, 100)
Also, it can be handy to know about numpy.index_exp (or equivalently, np.s_). It's nothing fancy -- it just translates slicing into the equivalent tuple, etc. It's quite similar to our Foo class above. For example:
In [1]: np.index_exp[10:4, ::-1, ...]
Out[1]: (slice(10, 4, None), slice(None, None, -1), Ellipsis)
I suspect you are trying to pass the slice through as a parameter?
def do_slice(sl, mystring):
return mystring[sl]
sl = slice(0,2)
mystr = "Hello"
print do_slice(sl, mystr)
Related
Say I have the list
a = [[1, 2, 3],
[4, 5, 6]]
and I have an index that I want to use to access an element of this list.
index = [1,2]
I want to use something like
a[*index] = 9
to mean a[index[0]][index[1]] = 9, but this doesn't work and neither does a[**index] = 9. Is there a similar way to do this without having a chain of index calls?
I would like a method to do this without using any libraries that must be imported.
First of all a[c, d, e] is equivalent to a[(c, d, e)] which is equivalent to a.__getitem__((c, d, e)). Note the double parentheses. Any __getitem__ implementation that wants to play nice with the Python data model always expects exactly one (explicit) argument.
That's why unpacking values from index inside the [] does not make much sense. a[*index] will give you a SyntaxError and a.__getitem__(*index) gives you a TypeError (because you are providing too many arguments).
Standard Python lists expect integer arguments to __getitem__, but numpy supports indexing with tuples (a numpy array still only takes exactly one argument for __getitem__, but it's allowed to be a tuple).
Demo:
>>> import numpy as np
>>> a = np.array([[1,2,3], [4,5,6]])
>>> a[(1,2)]
6
Of course, you can omit the parentheses, because
>>> a[1,2]
6
is exactly equivalent.
You can use reduce(), which is part of the standard library:
>>> a = [[1, 2, 3],
... [4, 5, 6]]
>>> index = [1, 2]
>>> import functools, operator
>>> functools.reduce(operator.getitem, index, a)
6
Or, you can write your own class that supports that kind of multi-dimensional indexing:
import functools, operator
class Matrix:
def __init__(self, lst):
self._lst = lst
def __getitem__(self, index):
return functools.reduce(operator.getitem, index, self._lst)
a = Matrix([[1, 2, 3],
[4, 5, 6]])
index = [1, 2]
print(a[index]) # -> 6
Otherwise, this is not possible using just lists and without loops or other functions.
I would like to get slice objects from subscript notation. So far, I have used a hacky trick to exploit Python built-in subscript to slice conversion features:
class Slice:
def __getitem__(self, item):
return item
For example, Slice()[1:2] will return slice(1,2,None).
Does anyone know of a more straightforward way?
If you must use subscript notation, then your current solution is the most compact besides maybe a dynamic class created with type:
>>> Slice = type('', (), {'__getitem__': lambda _, x: x})()
>>> Slice[1:2]
slice(1, 2, None)
>>> Slice[1:2:3]
slice(1, 2, 3)
>>>
But code like this is usually hard to understand/maintain/extend/etc.
Instead, I would recommend that you use slice, which allows you to create slice objects directly:
>>> slice(1, 2)
slice(1, 2, None)
>>> slice(1, 2, 3)
slice(1, 2, 3)
>>>
The built-in was made specifically for this purpose (well, that and a few others such as type-checking with isinstance) and is therefore very portable as well as pythonic.
I want to replace some elements of a list immediately.
Suppose we have these lists:
list = [1, 2, 3, 4, 5, 6]
idx = [1, 3, 4]
new = [100, 200, 300]
I want to replace elements 1, 3, 4 from list with new values. for example :
list[idx] = new
so the final list is => [1, 100, 3, 200, 300, 6]
I know you can use it in this way in Matlab, but want to know what should I do in Python?
Note : I know it's possible to use loops and do this.
Edit : I want to use a pure python solution.
The "pythonic" way would be to use zip:
for i, n in zip(idx, new):
L[i] = n
Python itself doesn't support matlab-style array operations, but you can look into numpy if you're interested in that coding style (see #abarnert's answer).
L = [1, 2, 3, 4, 5, 6]
idx = [1, 3, 4]
new = [100, 200, 300]
for i in range(len(idx)):
L[idx[i]] = new[i]
A slightly slower version without loops:
L = [1, 2, 3, 4, 5, 6]
idx = [1, 3, 4]
new = [100, 200, 300]
L = [num if i not in idx else new[i] for i,num in enumerate(L)]
Since you're looking for a Matlab-like solution, there's a good chance you should really be using NumPy here. In fact, if you do things that way, you can write exactly the code you wanted:
>>> import numpy as np
>>> a = np.array([1, 2, 3, 4, 5, 6])
>>> idx = [1, 3, 4]
>>> new = [100, 200, 300]
>>> a[idx] = new
>>> a
array([ 1, 100, 3, 200, 300, 6])
Besides giving you Matlab-ish element-wise operators and functions, NumPy also gives you convenient multi-dimensional arrays, and access to a huge library of higher-level functions (especially if you include adjunct libraries like SciPy). Plus you typically get performance benefits like, e.g., 6x speed and .25x space.
If you want a pure-Python solution, it's not that hard to implement this much of NumPy (or as much as you need) in Python. You can write your own Array class that emulates a container type in any way you want. In particular, note that a[idx] = new calls a.__setitem__(idx, new). You probably want to handle single numbers and slices the same way as list, but there's nothing at all stopping you from handling other types that list rejects. For example:
def __setitem__(self, idx, value):
if isinstance(idx, collections.abc.Iterable):
for i, v in zip(idx, value):
self.lst[i] = v
else:
self.lst[idx] = value
(You'd probably want to add a bit of error-handling for the case where idx and value have different lengths. You could work out the best rules from first principles, or start by looking at what NumPy does and just decide what you do and don't want to copy…)
Of course it's not an accident that the guts of this implementation will be code very much like alexis's answer, because all we're doing is wrapping up that logic so you only have to write it once, instead of every time you need it.
In Python what is equivalent to Ruby's Array.each method? Does Python have a nice and short closure/lambda syntax for it?
[1,2,3].each do |x|
puts x
end
Does Python have a nice and short closure/lambda syntax for it?
Yes, but you don't want it in this case.
The closest equivalent to that Ruby code is:
new_values = map(print, [1, 2, 3])
That looks pretty nice when you already have a function lying around, like print. When you just have some arbitrary expression and you want to use it in map, you need to create a function out of it with a def or a lambda, like this:
new_values = map(lambda x: print(x), [1, 2, 3])
That's the ugliness you apparently want to avoid. And Python has a nice way to avoid it: comprehensions:
new_values = [print(x) for x in values]
However, in this case, you're just trying to execute some statement for each value, not accumulate the new values for each value. So, while this will work (you'll get back a list of None values), it's definitely not idiomatic.
In this case, the right thing to do is to write it explicitly—no closures, no functions, no comprehensions, just a loop:
for x in values:
print x
The most idiomatic:
for x in [1,2,3]:
print x
You can use numpy for vectorized arithmetic over an array:
>>> import numpy as np
>>> a = np.array([1, 2, 3])
>>> a * 3
array([3, 6, 9])
You can easily define a lambda that can be used over each element of an array:
>>> array_lambda=np.vectorize(lambda x: x * x)
>>> array_lambda([1, 2, 3])
array([1, 4, 9])
But as others have said, if you want to just print each, use a loop.
There are also libraries that wrap objects to expose all the usual functional programming stuff.
PyDash http://pydash.readthedocs.org/en/latest/
underscorepy (Google github underscore.py)
E.g. pydash allows you to do things like this:
>>> from pydash import py_
>>> from __future__ import print_function
>>> x = py_([1,2,3,4]).map(lambda x: x*2).each(print).value()
2
4
6
8
>>> x
[2, 4, 6, 8]
(Just always remember to "trigger" execution and/or to un-wrap the wrapped values with .value() at the end!)
without need of an assignment:
list(print(_) for _ in [1, 2, 3])
or just
[print(_) for _ in [1, 2, 3]]
As I only now noticed after commenting on this answer, slices in Python 3 return shallow copies of whatever they're slicing rather than views. Why is this still the case? Even leaving aside numpy's usage of views rather than copies for slicing, the fact that dict.keys, dict.values, and dict.items all return views in Python 3, and that there are many other aspects of Python 3 geared towards greater use of iterators, makes it seem that there would have been a movement towards slices becoming similar. itertools does have an islice function that makes iterative slices, but that's more limited than normal slicing and does not provide view functionality along the lines of dict.keys or dict.values.
As well, the fact that you can use assignment to slices to modify the original list, but slices are themselves copies and not views, is a contradictory aspect of the language and seems like it violates several of the principles illustrated in the Zen of Python.
That is, the fact you can do
>>> a = [1, 2, 3, 4, 5]
>>> a[::2] = [0, 0, 0]
>>> a
[0, 2, 0, 4, 0]
But not
>>> a = [1, 2, 3, 4, 5]
>>> a[::2][0] = 0
>>> a
[0, 2, 3, 4, 5]
or something like
>>> a = [1, 2, 3, 4, 5]
>>> b = a[::2]
>>> b
view(a[::2] -> [1, 3, 5]) # numpy doesn't explicitly state that its slices are views, but it would probably be a good idea to do it in some way for regular Python
>>> b[0] = 0
>>> b
view(a[::2] -> [0, 3, 5])
>>> a
[0, 2, 3, 4, 5]
Seems somewhat arbitrary/undesirable.
I'm aware of http://www.python.org/dev/peps/pep-3099/ and the part where it says "Slices and extended slices won't go away (even if the __getslice__ and __setslice__ APIs may be replaced) nor will they return views for the standard object types.", but the linked discussion provides no mention of why the decision about slicing with views was made; in fact, the majority of the comments on that specific suggestion out of the suggestions listed in the original post seemed to be positive.
What prevented something like this from being implemented in Python 3.0, which was specifically designed to not be strictly backwards-compatible with Python 2.x and thus would have been the best time to implement such a change in design, and is there anything that may prevent it in future versions of Python?
As well, the fact that you can use assignment to slices to modify the original list, but slices are themselves copies and not views.
Hmm.. that's not quite right; although I can see how you might think that. In other languages, a slice assignment, something like:
a[b:c] = d
is equivalent to
tmp = a.operator[](slice(b, c)) # which returns some sort of reference
tmp.operator=(d) # which has a special meaning for the reference type.
But in python, the first statement is actually converted to this:
a.__setitem__(slice(b, c), d)
Which is to say that an item assignment is actually specially recognized in python to have a special meaning, separate from item lookup and assignment; they may be unrelated. This is consistent with python as a whole, because python doesn't have concepts like the "lvalues" found in C/C++; There's no way to overload the assignment operator itself; only specific cases when the left side of the assignment is not a plain identifier.
Suppose lists did have views; And you tried to use it:
myView = myList[1:10]
yourList = [1, 2, 3, 4]
myView = yourList
In languages besides python, there might be a way to shove yourList into myList, but in python, since the name myView appears as a bare identifier, it can only mean a variable assignemnt; the view is lost.
Well it seems I found a lot of the reasoning behind the views decision, going by the thread starting with http://mail.python.org/pipermail/python-3000/2006-August/003224.html (it's primarily about slicing strings, but at least one e-mail in the thread mentions mutable objects like lists), and also some things from:
http://mail.python.org/pipermail/python-3000/2007-February/005739.html
http://mail.python.org/pipermail/python-dev/2008-May/079692.html and following e-mails in the thread
Looks like the advantages of switching to this style for base Python would be vastly outweighed by the induced complexity and various undesirable edge cases. Oh well.
...And as I then started wondering about the possibility of just replacing the current way slice objects are worked with with an iterable form a la itertools.islice, just as zip, map, etc. all return iterables instead of lists in Python 3, I started realizing all the unexpected behavior and possible problems that could come out of that. Looks like this might be a dead end for now.
On the plus side, numpy's arrays are fairly flexible, so in situations where this sort of thing might be necessary, it wouldn't be too hard to use one-dimensional ndarrays instead of lists. However, it seems ndarrays don't support using slicing to insert additional items within arrays, as happens with Python lists:
>>> a = [0, 0]
>>> a[:1] = [2, 3]
>>> a
[2, 3, 0]
I think the numpy equivalent would instead be something like this:
>>> a = np.array([0, 0]) # or a = np.zeros([2]), but that's not important here
>>> a = np.hstack(([2, 3], a[1:]))
>>> a
array([2, 3, 0])
A slightly more complicated case:
>>> a = [1, 2, 3, 4]
>>> a[1:3] = [0, 0, 0]
>>> a
[1, 0, 0, 0, 4]
versus
>>> a = np.array([1, 2, 3, 4])
>>> a = np.hstack((a[:1], [0, 0, 0], a[3:]))
>>> a
array([1, 0, 0, 0, 4])
And, of course, the above numpy examples don't store the result in the original array as happens with the regular Python list expansion.