I came across the fact that numpy arrays are passed by reference at multiple places, but then when I execute the following code, why is there a difference between the behavior of foo and bar
import numpy as np
def foo(arr):
arr = arr - 3
def bar(arr):
arr -= 3
a = np.array([3, 4, 5])
foo(a)
print a # prints [3, 4, 5]
bar(a)
print a # prints [0, 1, 2]
I'm using python 2.7 and numpy version 1.6.1
In Python, all variable names are references to values.
When Python evaluates an assignment, the right-hand side is evaluated before the left-hand side. arr - 3 creates a new array; it does not modify arr in-place.
arr = arr - 3 makes the local variable arr reference this new array. It does not modify the value originally referenced by arr which was passed to foo. The variable name arr simply gets bound to the new array, arr - 3. Moreover, arr is local variable name in the scope of the foo function. Once the foo function completes, there is no more reference to arr and Python is free to garbage collect the value it references. As Reti43 points out, in order for arr's value to affect a, foo must return arr and a must be assigned to that value:
def foo(arr):
arr = arr - 3
return arr
# or simply combine both lines into `return arr - 3`
a = foo(a)
In contrast, arr -= 3, which Python translates into a call to the __iadd__ special method, does modify the array referenced by arr in-place.
The first function calculates (arr - 3), then assigns the local name arr to it, which doesn't affect the array data passed in. My guess is that in the second function, np.array overrides the -= operator, and operates in place on the array data.
Python passes the array by reference:
$:python
...python startup message
>>> import numpy as np
>>> x = np.zeros((2,2))
>>> x
array([[0.,0.],[0.,0.]])
>>> def setx(x):
... x[0,0] = 1
...
>>> setx(x)
>>> x
array([[1.,0.],[0.,0.]])
The top answer is referring to a phenomenon that occurs even in compiled c-code, as any BLAS events will involve a "read-onto" step where either a new array is formed which the user (code writer in this case) is aware of, or a new array is formed "under the hood" in a temporary variable which the user is unaware of (you might see this as a .eval() call).
However, I can clearly access the memory of the array as if it is in a more global scope than the function called (i.e., setx(...)); which is exactly what "passing by reference" is, in terms of writing code.
And let's do a few more tests to check the validity of the accepted answer:
(continuing the session above)
>>> def minus2(x):
... x[:,:] -= 2
...
>>> minus2(x)
>>> x
array([[-1.,-2.],[-2.,-2.]])
Seems to be passed by reference. Let us do a calculation which will definitely compute an intermediate array under the hood, and see if x is modified as if it is passed by reference:
>>> def pow2(x):
... x = x * x
...
>>> pow2(x)
>>> x
array([[-1.,-2.],[-2.,-2.]])
Huh, I thought x was passed by reference, but maybe it is not? -- No, here, we have shadowed the x with a brand new declaration (which is hidden via interpretation in python), and python will not propagate this "shadowing" back to global scope (which would violate the python-use case: namely, to be a beginner level coding language which can still be used effectively by an expert).
However, I can very easily perform this operation in a "pass-by-reference" manner by forcing the memory (which is not copied when I submit x to the function) to be modified instead:
>>> def refpow2(x):
... x *= x
...
>>> refpow2(x)
>>> x
array([[1., 4.],[4., 4.]])
And so you see that python can be finessed a bit to do what you are trying to do.
Related
sorry this question came up before here Setting two arrays equal
But the solution did not work and i dont know why.
import numpy as np
zero_matrix = np.zeros((3,3)) # 3x3 zero matrix
test_matrix = zero_matrix[:] # test_matrix is a view of zero_matrix. Without [:] it would be same object
print (zero_matrix)
print ()
print (test_matrix)
print ()
print(id(test_matrix))
print ()
print(id(zero_matrix))
print ()
test_matrix[1] = 42
print (test_matrix)
print ()
print (zero_matrix)
the 'zero_matrix' is also changed when i set the test_matrix[1] = 42.
And i dont get why since both have different object ids.
This is what is mean by the comment in your code that says test_matrix is a "view". A view does not have its own copy of the data. Rather it shares the underlying data of the original array. Views do not have to be of the entire array, but can be of small sub-sections of the array. These sub sections do not even need to be contiguous if the view is strided. eg.
a = np.arange(10)
b = a[::2] # create a view of every other element starting with the 0-th
assert list(b) == [0, 2, 4, 6, 8]
assert a[4] == 4
b[2] = -1
assert a[4] == -1
Views are powerful as they allow more complex operations without having to copy large amounts of data. Not needing to copy data all the time can mean some operations are faster than they otherwise would be.
Beware, not all index operations create views. eg.
a = np.arange(10, 20)
b = a[[1,2,5]]
assert list(b) == [11, 12, 15]
b[0] == -1
assert a[1] != -1
Use copy to copy your numpy arrays:
zero_matrix = np.zeros((3,3))
test_matrix = zero_matrix.copy()
test_matrix[1] = 42
print(zero_matrix)
print(test_matrix)
Numpy arrays and python lists behave differently in this regard.
They indeed have both different object IDs, but, as you write yourself: test_matrix is a view of zero_matrix.
An object is usually called a "view object" when it provides a way to access another object (be it by reading or by writing). In this case, accesses to this view object are deflected to the other object both by reading and writing.
That's a speciality of numpy objects opposed to "normal" python objects.
But even python has these objects, but doesn't use them unless explicitly requested.
I have a numpy array that is changed by a function.
After calling the function I want to proceed with the initial value of the array (value before calling the modifying function)
# Init of the array
array = np.array([1, 2, 3])
# Function that modifies array
func(array)
# Print the init value [1,2,3]
print(array)
Is there a way to pass the array by value or am I obligated to make a deep copy?
As I mentioned, np.ndarray objects are mutable data structures. This means that any variables that refer to a particular object will all reflect changes when a change is made to the object.
However, keep in mind that most numpy functions that transform arrays return new array objects, leaving the original unchanged.
What you need to do in this scenario depends on exactly what you're doing. If your function modifies the same array in place, then you'll need to pass a copy to the function. You can do this with np.ndarray.copy.
You could use the following library (https://pypi.python.org/pypi/pynverse) that inverts your function and call it, like so :
from pynverse import inversefunc
cube = (lambda x: x**3)
invcube = inversefunc(cube)
arr = func(arr)
In [0]: arr
Out[0]: array([1, 8, 27], dtype=int32)
In [1]: invcube(arr)
Out[1]: array([1, 2, 3])
what is the difference below codes
built-in list code
>>> a = [1,2,3,4]
>>> b = a[1:3]
>>> b[1] = 0
>>> a
[1, 2, 3, 4]
>>> b
[2, 0]
numpy array
>>> c = numpy.array([1,2,3,4])
>>> d = c[1:3]
>>> d[1] = 0
>>> c
array([1, 2, 0, 4])
>>> d
array([2, 0])
as it is seen in numpy array c is effected directly. I think in built-in lists, new memory is allocated for the variable b. Probably in numpy the reference of c[1:3] is assigned d, I am not clear about these.
How this works for numpy and built-in?
The key point to understand is that every assignment in Python associates a name with an object in memory. Python never copies on assignment. It now becomes important to understand when new objects are created and how they behave.
In your first example, the slicing in the list creates a new list object. In this case, both of the lists reference some of the same objects (the int 2 and the int 3). The fact that these references are copied is what is called a "shallow" copy. In other words, the references are copied, but the objects they refer to are still the same. Keep in mind that this will be true regardless of the type of thing that is stored in the list.
Now, we create a new object (the int 0) and assign b[1] = 0. Because a and b are separate lists, it should not surprise us that they now show different elements.
I like the pythontutor visualisation of this situation.
In the array case, "All arrays generated by basic slicing are always views of the original array.".
This new object shares data with the original, and indexed assignment is handled in such a way that any updates to the view will update the shared data.
This has been covered alot, but finding a good duplicate is too much work. :(
Let's see if I can quickly describe things with your examples:
>>> a = [1,2,3,4] # a list contains pointers to numbers elsewhere
>>> b = a[1:3] # a new list, with copies of those pointers
>>> b[1] = 0 # change one pointer in b
>>> a
[1, 2, 3, 4] # does not change any pointers in a
>>> b
[2, 0]
An array has a different structure - it has a data buffer with 'raw' numbers (or other byte values).
numpy array
>>> c = numpy.array([1,2,3,4])
>>> d = c[1:3] # a view; a new array but uses same data buffer
>>> d[1] = 0 # change a value in d;
>>> c
array([1, 2, 0, 4]) # we see the change in the corrsponding slot of c
>>> d
array([2, 0])
The key point with lists is that they contain pointers to objects. You can copy the pointers without copying the objects; and you can change pointers without changing other copies of the pointers.
To save memory and speed numpy as implemented a concept of view. It can make a new array without copying values from the original - because it can share the data buffer. But it is also possible to make a copy, e.g.
e = c[1:3].copy()
e[0] = 10
# no change in c
view v copy is a big topic in numpy and a fundamental one, especially when dealing with different kinds of indexing (slices, basic, advanced). We can help with questions, but you also should read the numpy docs. There's no substitute for understanding the basics of how a numpy array is stored.
http://scipy-cookbook.readthedocs.io/items/ViewsVsCopies.html
http://www.scipy-lectures.org/advanced/advanced_numpy/ (may be more advanced that what you need now)
If I use a list as a parameter for a function, I would expect that the original list would be unmodified. Why is it when I use the code below, x == z is True when it should be x == range(10) is True?
In [83]: def pop_three(collection):
....: new_collection = []
....: new_collection.append(collection.pop())
....: new_collection.append(collection.pop())
....: new_collection.append(collection.pop())
....: return new_collection, collection
....:
In [84]: x = range(10)
In [85]: x
Out[85]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
In [86]: y, z = pop_three(x)
In [87]: y
Out[87]: [9, 8, 7]
In [88]: z
Out[88]: [0, 1, 2, 3, 4, 5, 6]
In [89]: x
Out[89]: [0, 1, 2, 3, 4, 5, 6]
If I use a list as a parameter for a function, I would expect that the original list would be unmodified.
No. I think what you're missing here is that Python never automatically copies anything.
If you're used to a language like, say, C++, Python is very different. Ned Batchelder, as always, explains this brilliantly, but let me summarize:
In C++, a function parameter, or a variable, or an array element, is a location in memory, with a type. When you write a = b, or call f(b), that copies the value in memory location b to memory location a (possibly calling a casting operator or conversion constructor or copy constructor or copy assignment operator—or moving versions of half of those… but we're not here to learn about the intricacies of C++).
In Python, a function parameter, or a variable, or a list element, is just a name. The value has its own typed memory location, somewhere else. You can have as many names as you want for the same object. When you write a = b or call f(b), that just makes another name for the same object.
In particular, in your case, collection and x are names for the same object. If you added print(id(collection)) before the call and print(id(x)) inside the function, they would both print out the same number.
So, in Python, if you want a copy, you have to ask for it explicitly—e.g., pop_three(x[:]) or pop_three(copy.copy(x)).
Or, of course, you could do that inside pop_three.
Or, best of all, you could just avoid using mutating methods like pop in the first place:
def pop_three(collection):
return collection[-3:][::-1], collection[:-3]
I was confused since strings or integers passed into a function can be treated in this fashion.
Well, they can be treated in this fashion exactly as far as lists can. If you never mutate anything, the distinction between separate copies and shared references is irrelevant.* Because Python strings and integers are immutable, the distinction can't come up in any code with strings or integers are arguments. Lists, on the other hand, are mutable, so the distinction can arise—but it's perfectly possible, and in fact often best, to write code that uses them without mutating anything, as in the final example above.
Notice that this assumes the first point above: that, unlike C++ assignment, Python assignment is not a form of mutation. But it can get tricky at the edges. For example, is x[0] = 1 a mutation? Well, it's not a mutation of x[0], but it is a mutation of x. What about x += y? The language leaves that up to the type of x, and for mutable types it usually (but not always) is a mutation, while for immutable types of course it isn't.
* This isn't quite true; you can write code that relies on identity even though it isn't relevant, e.g., by using the id function or the is operator. Such code is almost never Pythonic, or a good idea… but it can be helpful to learning how things work. Try to print(id(x)) suggestion above with x = "a" + "b". However, this can be misleading, because the Python interpreter is allowed to "intern" objects that it knows are guaranteed to be immutable, and it does so with, e.g., small integers, and the Python compiler is allowed to fold constants, and it does so with, e.g., literal strings.
Because when you pass x to the pop_tree() and you called collection.pop(). This effectively did x.pop()
Therefore when you turned from the funciton, x only got 7 elements
Lists are mutable. When you pop from a list, you modify the list.
Any variable that originally pointed to the list will continue to point to the same modified list. Any new variable that you point at the list will point to the same modified list as well.
z = pop_three(x) means you are passing in the list x and then mutating that list x using collection.pop() x so x is being mutated therefore could not be == range(10)
y becomes new_collection and z is collection which is the mutated returned value of x.
In [2]: x = range(10)
In [3]: y, z = pop_three(x)
In [4]: z is x # z is the object x
Out[4]: True
Lets say I have a numpy array like
x = np.arange(10)
is it somehow possible to create a reference to a single element i.e.
y = create_a_reference_to(x[3])
y = 100
print x
[ 0 1 2 100 4 5 6 7 8 9]
You can't create a reference to a single element, but you can get a view over that single element:
>>> x = numpy.arange(10)
>>> y = x[3:4]
>>> y[0] = 100
>>> x
array([0, 1, 2, 100, 4, 5, 6, 7, 8, 9])
The reason you can't do the former is that everything in python is a reference. By doing y = 100, you're modifying what y points to - not it's value.
If you really want to, you can get that behaviour on instance attributes by using properties. Note this is only possible because the python data model specifies additional operations while accessing class attributes - it's not possible to get this behaviour for variables.
No you cannot do that, and that is by design.
Numpy arrays are of type numpy.ndarray. Individual items in it can be accessed with numpy.ndarray.item which does "copy an element of an array to a standard Python scalar and return it".
I'm guessing numpy returns a copy instead of direct reference to the element to prevent mutability of numpy items outside of numpy's own implementation.
Just as a thoughtgame, let's assume this wouldn't be the case and you would be allowed to get reference to individual items. Then what would happen if: numpy was in the midle of calculation and you altered an individual intime in another thread?
#goncalopp gives a correct answer, but there are a few variations that will achieve similar effects.
All of the notations shown below are able to reference a single element while still returning a view:
x = np.arange(10)
two_index_method = [None] * 10
scalar_element_method = [None] * 10
expansion_method = [None] * 10
for i in range(10):
two_index_method[i] = x[i:i+1]
scalar_element_method[i] = x[..., i] # x[i, ...] works, too
expansion_method[i] = x[:, np.newaxis][i] # np.newaxis == None
two_index_method[5] # Returns a length 1 numpy.ndarray, shape=(1,)
# >>> array([5])
scalar_element_method[5] # Returns a numpy scalar, shape = ()
# >>> array(5)
expansion_method[5] # Returns a length 1 numpy.ndarray, shape=(1,)
# >>> array([5])
x[5] = 42 # Change the value in the original `ndarray`
x
# >>> array([0, 1, 2, 3, 4, 42, 6, 7, 8, 9]) # The element has been updated
# All methods presented here are correspondingly updated:
two_index_method[5], scalar_element_method[5], expansion_method[5]
# >>> (array([42]), array(42), array([42]))
Since the object in scalar_element_method is a dimension zero scalar, attempting to reference the element contained within the ndarray via element[0] will return an IndexError. For a scalar ndarray, element[()] can be used to reference the element contained within the numpy scalar. This method can also be used for assignment to a length-1 ndarray, but has the unfortunate side effect that it does not dereference a length-1 ndarray to a python scalar. Fortunately, there is a single method, element.item(), that can be used (for dereferencing only) to obtain the value regardless of whether the element is a length one ndarray or a scalar ndarray:
scalar_element_method[5][0] # This fails
# >>> IndexError: too many indices for array
scalar_element_method[5][()] # This works for scalar `ndarray`s
# >>> 42
scalar_element_method[5][()] = 6
expansion_method[5][0] # This works for length-1 `ndarray`s
# >>> 6
expansion_method[5][()] # Doesn't return a python scalar (or even a numpy scalar)
# >>> array([6])
expansion_method[5][()] = 8 # But can still be used to change the value by reference
scalar_element_method[5].item() # item() works to dereference all methods
# >>> 8
expansion_method[5].item()
# >>> [i]8
TLDR; You can create a single-element view v with v = x[i:i+1], v = x[..., i], or v = x[:, None][i]. While different setters and getters work with each method, you can always assign values with v[()]=new_value, and you can always retrieve a python scalar with v.item().