Consider the following array manipulations:
import numpy as np
def f(x):
x += 1
x = np.zeros(1)
f(x) # changes `x`
f(x[0]) # doesn't change `x`
x[0] += 1 # changes `x`
Why does x[0] behave differently depending on whether += 1 happens inside or outside the function f?
Can I pass a part of the array to the function, such that the function modifies the original array?
Edit: If we considered = instead of +=, we would probably maintain the core of the question while getting rid of some irrelevant complexity.
You don't even need the function call to see this difference.
x is an array:
In [138]: type(x)
Out[138]: numpy.ndarray
Indexing an element of the array returns a np.float64 object. It in effect "takes" the value out of the array; it is not a reference to the element of the array.
In [140]: y=x[0]
In [141]: type(y)
Out[141]: numpy.float64
This y is a lot like a python float; you can += the same way:
In [142]: y += 1
In [143]: y
Out[143]: 1.0
but this does not change x:
In [144]: x
Out[144]: array([0.])
But this does change x:
In [145]: x[0] += 1
In [146]: x
Out[146]: array([1.])
y=x[0] does a x.__getitem__ call. x[0]=3 does a x.__setitem__ call. += uses __iadd__, but it's similar in effect.
Another example:
Changing x:
In [149]: x[0] = 3
In [150]: x
Out[150]: array([3.])
but attempting to do the same to y fails:
In [151]: y[()] = 3
Traceback (most recent call last):
File "<ipython-input-151-153d89268cbc>", line 1, in <module>
y[()] = 3
TypeError: 'numpy.float64' object does not support item assignment
but y[()] is allowed.
basic indexing of an array with a slice does produce a view that can be modified:
In [154]: x = np.zeros(5)
In [155]: x
Out[155]: array([0., 0., 0., 0., 0.])
In [156]: y= x[0:2]
In [157]: type(y)
Out[157]: numpy.ndarray
In [158]: y += 1
In [159]: y
Out[159]: array([1., 1.])
In [160]: x
Out[160]: array([1., 1., 0., 0., 0.])
===
Python list and dict examples of the x[0]+=1 kind of action:
In [405]: alist = [1,2,3]
In [406]: alist[1]+=12
In [407]: alist
Out[407]: [1, 14, 3]
In [408]: adict = {'a':32}
In [409]: adict['a'] += 12
In [410]: adict
Out[410]: {'a': 44}
__iadd__ can be thought of a __getitem__ followed by a __setitem__ with the same index.
The issue is not scope, since the only thing that depends on scope is the available names. All objects can be accessed in any scope that has a name for them. The issue is one of mutability vs immutability and understanding what operators do.
x is a mutable numpy array. f runs x += 1 directly on it. += is the operator that invokes in-place addition. In other words, it does x = x.__iadd__(1)*. Notice the reassignment to x, which happens in the function. That is a feature of the in-place operators that allows them to operate on immutable objects. In this case, ndarray.__iadd__ is a true in-place operator which just returns x, and everything works as expected.
Now let's analyze f(x[0]) the same way. x[0] calls x.__getitem__(0)*. When you pass in a scalar int index, numpy extracts a one-element array and effectively calls .item() on it. The result is a python int (or float, or even possibly a tuple, depending on what your array's dtype is). Either way, the object is immutable. Once it's been extracted by __getitem__, the += operator in f replaces the name x in f with the new object, but the change is not seen outside the function, much less in the array. In this scenario, f has no reference to x, so no change is to be expected.
The example of x[0] += 1 is not the same as calling f(x[0]). It is equivalent to calling x.__setitem__(0, x.__getitem__(0).__iadd__(1))*. The call to f was only the part with type(x).__getitem__(0).__iadd__(1), which returns a new object, but never reassigns as __setitem__ does. The key is that [] = (__setitem__) in python is an entirely different operator from [] (__getitem__) and = (assingment) separately.
To make the second example (f(x[0]) work, you would have to pass in a mutable object. An integer object extracts a single python object, and an array index makes a copy. However, a slice index returns a view that is mutable and tied to the original array memory. Therefore, you can do
f(x[0:1]) # changes `x`
In this case f does the following: x.__getitem__(slice(0, 1, None)).__iadd__(1). The key is that __getitem__ returns a mutable view into the original array, not an immutable int.
To see why it is important not only that the object is mutable but that it is a view into the original array, try f(x[[0]]). Indexing with a list produces an array, but a copy. In x[[0]].__iadd__ will modify the list you pass in in-place, but the list is not copied back into the original, so the change will not propagate.
* This is an approximation. When invoked by an operator, dunder methods are actually called as type(x).__operator__(x, ...), not x.__operator__(...).
As per this comment and this answer:
The x[0] inside of f(x[0]) performs __getitem__ on x. In this particular case (as opposed to indexing a slice of the array, for example), the value returned by this operation doesn't allow modifying the original array.
x[0] = 1 performs __setitem__ on x.
__getitem__ and __setitem__ can be defined/overloaded to do anything. They don't even have to be consistent with each other.
Related
I want to increment a numpy array using advanced indexing, e.g.
import numpy
x = numpy.array([0,0])
indices = numpy.array([1,1])
x[indices] += [1,2]
print x #prints [0 2]
I would have expected, that the result is [0 3], since both 1 and 2 should be added to the second zero of x, but apparently numpy only adds the last element which matches to a particular index.
Is this the general behaviour and I can rely on that, or is this undefined behaviour and could change with a different version of numpy?
Additionally, is there an (easy) way to get numpy to add all elements which match the index and not just the last one?
From numpy docs:
For advanced assignments, there is in general no guarantee for the iteration order. This means that if an element is set more than once, it is not possible to predict the final result.
You can use np.add.at to get the desired behaviour:
Help on built-in function at in numpy.add:
numpy.add.at = at(...) method of numpy.ufunc instance
at(a, indices, b=None)
Performs unbuffered in place operation on operand 'a' for elements
specified by 'indices'. For addition ufunc, this method is equivalent to
`a[indices] += b`, except that results are accumulated for elements that
are indexed more than once. For example, `a[[0,0]] += 1` will only
increment the first element once because of buffering, whereas
`add.at(a, [0,0], 1)` will increment the first element twice.
.. versionadded:: 1.8.0
< snip >
Example:
>>> b = np.ones(2, int)
>>> a = np.zeros(2, int)
>>> c = np.arange(2,4)
>>> np.add.at(a, b, c)
>>> a
array([0, 5])
I got a very strange problem about the map function, it will increase a dimension automatically.
matrix = range(4)
matrix = numpy.reshape(matrix,(2,2))
vector = numpy.ones((1,2))
newMatrix = map(lambda line: line/vector, matrix)
np.shape(newMatrix) # I got (2,1,2)
I am confused, the matrix has the shape(2,2), but why after the map() function, the newMatrix has such a shape (2,1,2)? How can I fix with this problem?
I think what you are trying to do is simply newMatrix = matrix / vector. Remember that numpy performs element-wise operations. map is doing what it is defined to do, i.e. return a list after applying your function to each item in the iterator. So map operates on each row of your matrix at a time. You have two rows; thus, your new shape is 2 x 1 x 2.
This example may illustrate what is going on (I replaced your 'matrix', and 'vector' names with neutral variable names)
In [13]: x = np.arange(4).reshape(2,2)
In [14]: y=np.ones((1,2))
In [15]: list(map(lambda line:line/y, x))
Out[15]: [array([[ 0., 1.]]), array([[ 2., 3.]])]
Notice the 2 arrays have shape (1,2), which matches that of y. x[0,:]/y shows this as well. Wrap that list in np.array..., and you get a (2,1,2).
Notice what happens when I use a 1d array, z:
In [16]: z=np.ones((2,))
In [17]: list(map(lambda line:line/z, x))
Out[17]: [array([ 0., 1.]), array([ 2., 3.])]
I ran this sample in Python3, where map returns a generator. To get an array from that I have to use
np.array(list(map(...)))
I don't think I've seen the use of map with numpy arrays before. I'm a little surprised that in Python2 it returns an array, not just a list. A more common version of your iteration is to wrap a list comprehension in np.array...
np.array([line/y for line in x])
But as noted in the other answer, you don't need iteration for this simple case. x/y is sufficient. How to avoid iteration is a frequent SO question.
Lets say I have a numpy array like
x = np.arange(10)
is it somehow possible to create a reference to a single element i.e.
y = create_a_reference_to(x[3])
y = 100
print x
[ 0 1 2 100 4 5 6 7 8 9]
You can't create a reference to a single element, but you can get a view over that single element:
>>> x = numpy.arange(10)
>>> y = x[3:4]
>>> y[0] = 100
>>> x
array([0, 1, 2, 100, 4, 5, 6, 7, 8, 9])
The reason you can't do the former is that everything in python is a reference. By doing y = 100, you're modifying what y points to - not it's value.
If you really want to, you can get that behaviour on instance attributes by using properties. Note this is only possible because the python data model specifies additional operations while accessing class attributes - it's not possible to get this behaviour for variables.
No you cannot do that, and that is by design.
Numpy arrays are of type numpy.ndarray. Individual items in it can be accessed with numpy.ndarray.item which does "copy an element of an array to a standard Python scalar and return it".
I'm guessing numpy returns a copy instead of direct reference to the element to prevent mutability of numpy items outside of numpy's own implementation.
Just as a thoughtgame, let's assume this wouldn't be the case and you would be allowed to get reference to individual items. Then what would happen if: numpy was in the midle of calculation and you altered an individual intime in another thread?
#goncalopp gives a correct answer, but there are a few variations that will achieve similar effects.
All of the notations shown below are able to reference a single element while still returning a view:
x = np.arange(10)
two_index_method = [None] * 10
scalar_element_method = [None] * 10
expansion_method = [None] * 10
for i in range(10):
two_index_method[i] = x[i:i+1]
scalar_element_method[i] = x[..., i] # x[i, ...] works, too
expansion_method[i] = x[:, np.newaxis][i] # np.newaxis == None
two_index_method[5] # Returns a length 1 numpy.ndarray, shape=(1,)
# >>> array([5])
scalar_element_method[5] # Returns a numpy scalar, shape = ()
# >>> array(5)
expansion_method[5] # Returns a length 1 numpy.ndarray, shape=(1,)
# >>> array([5])
x[5] = 42 # Change the value in the original `ndarray`
x
# >>> array([0, 1, 2, 3, 4, 42, 6, 7, 8, 9]) # The element has been updated
# All methods presented here are correspondingly updated:
two_index_method[5], scalar_element_method[5], expansion_method[5]
# >>> (array([42]), array(42), array([42]))
Since the object in scalar_element_method is a dimension zero scalar, attempting to reference the element contained within the ndarray via element[0] will return an IndexError. For a scalar ndarray, element[()] can be used to reference the element contained within the numpy scalar. This method can also be used for assignment to a length-1 ndarray, but has the unfortunate side effect that it does not dereference a length-1 ndarray to a python scalar. Fortunately, there is a single method, element.item(), that can be used (for dereferencing only) to obtain the value regardless of whether the element is a length one ndarray or a scalar ndarray:
scalar_element_method[5][0] # This fails
# >>> IndexError: too many indices for array
scalar_element_method[5][()] # This works for scalar `ndarray`s
# >>> 42
scalar_element_method[5][()] = 6
expansion_method[5][0] # This works for length-1 `ndarray`s
# >>> 6
expansion_method[5][()] # Doesn't return a python scalar (or even a numpy scalar)
# >>> array([6])
expansion_method[5][()] = 8 # But can still be used to change the value by reference
scalar_element_method[5].item() # item() works to dereference all methods
# >>> 8
expansion_method[5].item()
# >>> [i]8
TLDR; You can create a single-element view v with v = x[i:i+1], v = x[..., i], or v = x[:, None][i]. While different setters and getters work with each method, you can always assign values with v[()]=new_value, and you can always retrieve a python scalar with v.item().
I just discovered — by chance — that an array in numpy may be indexed by an empty tuple:
In [62]: a = arange(5)
In [63]: a[()]
Out[63]: array([0, 1, 2, 3, 4])
I found some documentation on the numpy wiki ZeroRankArray:
(Sasha) First, whatever choice is made for x[...] and x[()] they should be the same because ... is just syntactic sugar for "as many : as necessary", which in the case of zero rank leads to ... = (:,)*0 = (). Second, rank zero arrays and numpy scalar types are interchangeable within numpy, but numpy scalars can be use in some python constructs where ndarrays can't.
So, for 0-d arrays a[()] and a[...] are supposed to be equivalent. Are they for higher-dimensional arrays, too? They strongly appear to be:
In [65]: a = arange(25).reshape(5, 5)
In [66]: a[()] is a[...]
Out[66]: False
In [67]: (a[()] == a[...]).all()
Out[67]: True
In [68]: a = arange(3**7).reshape((3,)*7)
In [69]: (a[()] == a[...]).all()
Out[69]: True
But, it is not syntactic sugar. Not for a high-dimensional array, and not even for a 0-d array:
In [76]: a[()] is a
Out[76]: False
In [77]: a[...] is a
Out[77]: True
In [79]: b = array(0)
In [80]: b[()] is b
Out[80]: False
In [81]: b[...] is b
Out[81]: True
And then there is the case of indexing by an empty list, which does something else altogether, but appears equivalent to indexing with an empty ndarray:
In [78]: a[[]]
Out[78]: array([], shape=(0, 3, 3, 3, 3, 3, 3), dtype=int64)
In [86]: a[arange(0)]
Out[86]: array([], shape=(0, 3, 3, 3, 3, 3, 3), dtype=int64)
In [82]: b[[]]
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
IndexError: 0-d arrays can't be indexed.
So, it appears that () and ... are similar but not quite identical and indexing with [] means something else altogether. And a[] or b[] are SyntaxErrors. Indexing with lists is documented at index arrays, and there is a short notice about indexing with tuples at the end of the same document.
That leaves the question:
Is the difference between a[()] and a[...] by design? What is the design, then?
(Question somehow reminiscent of: What does the empty `()` do on a Matlab matrix?)
Edit:
In fact, even scalars may be indexed by an empty tuple:
In [36]: numpy.int64(10)[()]
Out[36]: 10
The treatment of A[...] is a special case, optimised to always return A itself:
if (op == Py_Ellipsis) {
Py_INCREF(self);
return (PyObject *)self;
}
Anything else that should be equivalent e.g. A[:], A[(Ellipsis,)], A[()], A[(slice(None),) * A.ndim] will instead return a view of the entirety of A, whose base is A:
>>> A[()] is A
False
>>> A[()].base is A
True
This seems an unnecessary and premature optimisation, as A[(Ellipsis,)] and A[()] will always give the same result (an entire view on A). From looking at https://github.com/numpy/numpy/commit/fa547b80f7035da85f66f9cbabc4ff75969d23cd it seems that it was originally required because indexing with ... didn't work properly on 0d arrays (previously to https://github.com/numpy/numpy/commit/4156b241aa3670f923428d4e72577a9962cdf042 it would return the element as a scalar), then extended to all arrays for consistency; since then, indexing has been fixed on 0d arrays so the optimisation isn't required, but it's managed to stick around vestigially (and there's probably some code that depends on A[...] is A being true).
While in the example you've given, the empty tuple and ellipsis give a similar result, in general they serve different purposes. When indexing an array, A[i, j, k] == A[(i, j, k)] and specifically A[...] == A[(Ellipsis,)]. Here the tuple simply serves as a container for indexing elements. This can be useful when you need to manipulate the index as a variable, for example you can do:
index = (0,) * A.ndim
A[index]
Notice that because the tuple is the container for indexing elements, it cannot be combined with other indices, for example A[(), 0] == A[[], 0] and A[(), 0] != A[..., 0].
Because an array A can be indexed with fewer indices than A.ndim, indexing with an empty tuple is a natural extension of that behavior and it can be useful in some situations, for example the above code snipit will work when A.ndim == 0.
In short, the tuple serves as a container for indexing elements, which is allowed to be empty, while the Ellipsis is one of the possible indexing elements.
According to the official Numpy documentation, the differences is clear:
An empty (tuple) index is a full scalar index into a zero dimensional
array. x[()] returns a scalar if x is zero dimensional and a view
otherwise. On the other hand x[...] always returns a view.
When an ellipsis (...) is present but has no size (i.e. replaces zero
:) the result will still always be an array. A view if no advanced
index is present, otherwise a copy.
>>> import numpy as np
>>> # ---------------------------------- #
>>> # when `x` is at least 1 dimensional #
>>> # ---------------------------------- #
>>> x = np.linspace(0, 10, 100)
>>> x.shape
(100,)
>>> x.ndim
1
>>> a = x[()]
>>> b = x[...]
>>> id(x), id(a), id(b)
(4559933568, 4561560080, 4585410192)
>>> id(x.base), id(a.base), id(b.base)
(4560914432, 4560914432, 4560914432)
>>> # ---------------------------- #
>>> # when `z` is zero dimensional #
>>> # ---------------------------- #
>>> z = np.array(3.14)
>>> z.shape
()
>>> z.ndim
0
>>> a = z[()]
>>> b = z[...]
>>> type(a), type(b)
(<class 'numpy.float64'>, <class 'numpy.ndarray'>)
>>> id(z), id(a), id(b)
(4585422896, 4586829384, 4561560080)
>>> id(z.base), id(a.base), id(b.base)
(4557260904, 4557260904, 4585422896)
>>> b.base is z
True
I came across the fact that numpy arrays are passed by reference at multiple places, but then when I execute the following code, why is there a difference between the behavior of foo and bar
import numpy as np
def foo(arr):
arr = arr - 3
def bar(arr):
arr -= 3
a = np.array([3, 4, 5])
foo(a)
print a # prints [3, 4, 5]
bar(a)
print a # prints [0, 1, 2]
I'm using python 2.7 and numpy version 1.6.1
In Python, all variable names are references to values.
When Python evaluates an assignment, the right-hand side is evaluated before the left-hand side. arr - 3 creates a new array; it does not modify arr in-place.
arr = arr - 3 makes the local variable arr reference this new array. It does not modify the value originally referenced by arr which was passed to foo. The variable name arr simply gets bound to the new array, arr - 3. Moreover, arr is local variable name in the scope of the foo function. Once the foo function completes, there is no more reference to arr and Python is free to garbage collect the value it references. As Reti43 points out, in order for arr's value to affect a, foo must return arr and a must be assigned to that value:
def foo(arr):
arr = arr - 3
return arr
# or simply combine both lines into `return arr - 3`
a = foo(a)
In contrast, arr -= 3, which Python translates into a call to the __iadd__ special method, does modify the array referenced by arr in-place.
The first function calculates (arr - 3), then assigns the local name arr to it, which doesn't affect the array data passed in. My guess is that in the second function, np.array overrides the -= operator, and operates in place on the array data.
Python passes the array by reference:
$:python
...python startup message
>>> import numpy as np
>>> x = np.zeros((2,2))
>>> x
array([[0.,0.],[0.,0.]])
>>> def setx(x):
... x[0,0] = 1
...
>>> setx(x)
>>> x
array([[1.,0.],[0.,0.]])
The top answer is referring to a phenomenon that occurs even in compiled c-code, as any BLAS events will involve a "read-onto" step where either a new array is formed which the user (code writer in this case) is aware of, or a new array is formed "under the hood" in a temporary variable which the user is unaware of (you might see this as a .eval() call).
However, I can clearly access the memory of the array as if it is in a more global scope than the function called (i.e., setx(...)); which is exactly what "passing by reference" is, in terms of writing code.
And let's do a few more tests to check the validity of the accepted answer:
(continuing the session above)
>>> def minus2(x):
... x[:,:] -= 2
...
>>> minus2(x)
>>> x
array([[-1.,-2.],[-2.,-2.]])
Seems to be passed by reference. Let us do a calculation which will definitely compute an intermediate array under the hood, and see if x is modified as if it is passed by reference:
>>> def pow2(x):
... x = x * x
...
>>> pow2(x)
>>> x
array([[-1.,-2.],[-2.,-2.]])
Huh, I thought x was passed by reference, but maybe it is not? -- No, here, we have shadowed the x with a brand new declaration (which is hidden via interpretation in python), and python will not propagate this "shadowing" back to global scope (which would violate the python-use case: namely, to be a beginner level coding language which can still be used effectively by an expert).
However, I can very easily perform this operation in a "pass-by-reference" manner by forcing the memory (which is not copied when I submit x to the function) to be modified instead:
>>> def refpow2(x):
... x *= x
...
>>> refpow2(x)
>>> x
array([[1., 4.],[4., 4.]])
And so you see that python can be finessed a bit to do what you are trying to do.