We Keep Coding

Should I ever return a list that was passed by reference and modified?

I have recently discovered that lists in python are automatically passed by reference (unless the notation array[:] is used). For example, these two functions do the same thing:
def foo(z):
z.append(3)
def bar(z):
z.append(3)
return z
x = [1, 2]
y = [1, 2]
foo(x)
bar(y)
print(x, y)
Before now, I always returned arrays that I manipulated, because I thought I had to. Now, I understand it's superfluous (and perhaps inefficient), but it seems like returning values is generally good practice for code readability. My question is, are there any issues for doing either of these methods/ what are the best practices? Is there a third option that I am missing? I'm sorry if this has been asked before but I couldn't find anything that really answers my question.

This answer works on the assumption that the decision as to whether to modify your input in-place or return a copy has already been made.
As you noted, whether or not to return a modified object is a matter of opinion, since the result is functionally equivalent. In general, it is considered good form to not return a list that is modified in-place. According to the Zen of Python (item #2):
Explicit is better than implicit.
This is borne out in the standard library. List methods are notorious for this on SO: list.append, insert, extend, list.sort, etc.
Numpy also uses this pattern frequently, since it often deals with large data sets that would be impractical to copy and return. A common example is the array method numpy.ndarray.sort, not to be confused with the top-level function numpy.sort, which returns a new copy.
The idea is something that is very much a part of the Python way of thinking. Here is an excerpt from Guido's email that explains the whys and wherefors:
I find the chaining form a threat to readability; it requires that the reader must be intimately familiar with each of the methods. The second [unchained] form makes it clear that each of these calls acts on the same object, and so even if you don't know the class and its methods very well, you can understand that the second and third call are applied to x (and that all calls are made for their side-effects), and not to something else.

Python built-ins, as a rule, will not do both, to avoid confusion over whether the function/method modifies its argument in place or returns a new value. When modifying in place, no return is performed (making it implicitly return None). The exceptions are cases where a mutating function returns something other than the object mutated (e.g. dict.pop, dict.setdefault).
It's generally a good idea to follow the same pattern, to avoid confusion.

The "best practice" is technically to not modify the thing at all:
def baz(z):
return z + [3]
x = [1, 2]
y = baz(x)
print(x, y)
but in general it's clearer if you restrict yourself to either returning a new object or modifying an object in-place, but not both at once.
There are examples in the standard library that both modify an object in-place and return something (the foremost example being list.pop()), but that's a special case because it's not returning the object that was modified.

There's not strict should of course, However, a function should either do something, or return something.. So, you'd better either modify the list in place without returning anything, or return a new one, leaving the original one unchanged.
Note: the list is not exactly passed by reference. It's the value of the reference that is actually passed. Keep that in mind if you re-assign

Pointer addresses of Python Lists and Matrix when reversing

I am trying to understand how Python matrices are implemented as compared to Java/C style 2D arrays.
Specifically the problem I am facing is this:
Given a matrix (list of lists), I am asked to reverse the individual lists in the matrix in-place. I came up with the following code:
CODE 1
------
def flip(matrix):
for list in matrix:
list=list[::-1]
matrix=[[1,0,0],[0,0,1]]
flip(matrix)
print(matrix) # Outputs "[[1,0,0],[0,0,1]]" i.e. does not reverse
If I modify the code a bit,
CODE 2
------
def flip(matrix):
for list in matrix:
list.reverse()
matrix=[[1,0,0],[0,0,1]]
flip(matrix)
print(matrix) # Outputs "[[0,0,1],[1,0,0]]" i.e. works correctly this time
I know that list.reverse() does in-place operation and list[::-1] creates a shallow copy. However in CODE 1, I am assigning the address of the shallow copy to the same variable (list) only. So the variable matrix should effectively get changed. Because the variable matrix[i] is the variable list. So if list gets modified, so should matrix.
To illustrate my previous point, the following code is provided:
CODE 3
------
def test(matrix):
for i, list in enumerate(matrix):
print(matrix[i] is list)
matrix=[[1,0,0],[0,0,1]]
test(matrix) # Outputs "True True"
If matrix[i] is list, then changing list means changing matrix[i] and changing matrix[i] means changing matrix.
If I modify CODE 1 so that instead of list being assigned the address of the newly created reversed list, matrix[i] be assigned that address, then surprisingly it works!
CODE 4
------
def flip(matrix):
for i, list in enumerate(matrix):
# Instead of list=list[::-1]
matrix[i]=list[::-1]
matrix=[[1,0,0],[0,0,1]]
flip(matrix)
print(matrix) # Correctly Outputs [[0,0,1], [1,0,0]]
I would like an explanation why CODE 1 does not work and why CODE 4 works.

The first time through the loop, list is just a name for matrix[0].
Mutating the object that list names, as in CODE 2, obviously mutates the object that matrix[0] names, because they're naming the same object.
But just rebinding list to some different object, as in CODE 1, doesn't change matrix[0] in any way. If you think about it, that makes sense. After all, the next time through the loop, list is going to get rebound to matrix[1], and you certainly wouldn't want that to change what's in matrix[0], right?
In C terms (and this is literally true, if you're using the normal CPython implementation), being names for the same object means being pointers to the same object. If list is a List *, assigning to list doesn't do anything to whatever was in *list.
So, why does CODE 4 work? Well, in code 4, you're still not mutating the list—but you're rebinding matrix[0], instead of list, and of course that rebinds matrix[0].
I'm guessing that, despite talking about "Java/C", you're really thinking in C++ terms. In C++, = is an operator, which can be overloaded. Plus, you don't just have pointers, but references, which sort of magically work without needing to explicitly dereference them. So, if list is a reference to a list object, rather than a pointer, list = isn't changing it into a reference to another list object, it's calling a special method, ListType::operator=. That's actually pretty weird. There's nothing like that in Java. or C. any more than there is in Python.
For more detail on what happens under the covers:
If you want to think of it in C terms, the actual C API used by the main (CPython) implementation may make things clear here.
Your function's locals are just an array of pointers to Python objects. That matrix is locals[0], list is locals[1], etc.
What's in *locals[0] is a PyListObject struct, which contains, among other things, a pointer to an array of Python objects. Each of which is pointing to another PyListObject struct. But inside those inner lists' arrays are pointers to PyLongObject structs, which just hold numbers.
The for loop is a bit more complicated than this, but pretend it's just doing locals[1] = (*locals[0]).elements[0], then locals[1] = (*locals[0]).elements[1], etc.
So, assigning to list is just changing locals[1], not *locals[1], and therefore it's not changing *locals[0].elements[0].
But assigning to *locals[0].elements[0] is a different story.
And so is calling the reverse method. When you do that, self just ends up as yet another pointer to the same object, but its implementation mutates things on *self.

Why are integers immutable in Python?

I understand the differences between mutable and immutable objects in Python. I have read many posts discussing the differences. However, I have not read anything regarding WHY integers are immutable objects.
Does there exist a reason for this? Or is the answer "that's just how it is"?
Edit: I am getting prompted to 'differentiate' this question from other questions as it seems to be a previously asked question. However, I believe what I'm asking is more of a philosophical Python question rather than a technical Python question.
It appears that 'primitive' objects in Python (i.e. strings, booleans, numbers, etc.) are immutable. I've also noticed that derived data types that are made up of primitives (i.e. dicts, lists, classes) are mutable.
Is that where the line is drawn whether or not an object is mutable? Primitive vs derived?

Making integers mutable would be very counter-intuitive to the way we are used to working with them.
Consider this code fragment:
a = 1 # assign 1 to a
b = a+2 # assign 3 to b, leave a at 1
After these assignments are executed we expect a to have the value 1 and b to have the value 3. The addition operation is creating a new integer value from the integer stored in a and an instance of the integer 2.
If the addition operation just took the integer at a and just mutated it then both a and b would have the value 3.
So we expect arithmetic operations to create new values for their results - not to mutate their input parameters.
However, there are cases where mutating a data structure is more convenient and more efficient. Let's suppose for the moment that list.append(x) did not modify list but returned a new copy of list with x appended.
Then a function like this:
def foo():
nums = []
for x in range(0,10):
nums.append(x)
return nums
would just return the empty list. (Remember - here nums.append(x) doesn't alter nums - it returns a new list with x appended. But this new list isn't saved anywhere.)
We would have to write the foo routine like this:
def foo():
nums = []
for x in range(0,10):
nums = nums.append(x)
return nums
(This, in fact, is very similar to the situation with Python strings up until about 2.6 or perhaps 2.5.)
Moreover, every time we assign nums = nums.append(x) we would be copying a list that is increasing in size resulting in quadratic behavior.
For those reasons we make lists mutable objects.
A consequence to making lists mutable is that after these statements:
a = [1,2,3]
b = a
a.append(4)
the list b has changed to [1,2,3,4]. This is something that we live with even though it still trips us up now and then.

What are the design decisions to make numbers immutable in Python?
There are several reasons for immutability, let's see first what are the reasons for immutability?
1- Memory
Saves memory. If it's well known that an object is immutable, it can be easily copied creating a new reference to the same object.
Performance. Python can allocate space for an immutable object at creation time, and the storage requirements are fixed and unchanging.
2- Fast execution.
It doesn't have to copy each part of the object, only a simple reference.
Easy to be compared, comparing equality by reference is faster than comparing values.
3- Security:
In Multi-threading apps Different threads can interact with data contained inside the immutable objects, without to worry about data consistency.
The internal state of your program will be consistent even if you have exceptions.
Classes should be immutable unless there's a very good reason to make them mutable....If a class cannot be made immutable, limit its mutability as much as possible
4- Ease to use
Is easier to read, easier to maintain and less likely to fail in odd and unpredictable ways.
Immutable objects are easier to test, due not only to their easy mockability, but also the code patterns they tend to enforce.
5- Keys must be immutable. Which means you can use strings, numbers or tuples as dictionary key. This is something that you want to use.
The hash table implementation of dictionaries uses a hash value calculated from the key value to find the key. If the key were a mutable object, its value could change, and thus its hash could also change. But since whoever changes the key object can’t tell that it was being used as a dictionary key, it can’t move the entry around in the dictionary. Then, when you try to look up the same object in the dictionary it won’t be found because its hash value is different. If you tried to look up the old value it wouldn’t be found either, because the value of the object found in that hash bin would be different.
Going back to the integers:
Security (3), Easy to use (4) and capacity of using numbers as keys in dictionaries (5) are reasons for taken the decision of making numbers immutable.
Has fixed memory requirements since creation time (1).
All in Python is an object, the numbers (like strings) are "elemental" objects. No amount of activity will change the value 8 to anything else, and no amount of activity will change the string “eight” to anything else. This is because a decision in the design too.

assignment in python

I know that "variable assignment" in python is in fact a binding / re-bindign of a name (the variable) to an object.
This brings the question: is it possible to have proper assignment in python, eg make an object equal to another object?
I guess there is no need for that in python:
Inmutable objects cannot be 'assigned to' since they can't be changed
Mutable objects could potentially be assigned to, since they can change, and this could be useful, since you may want to manipulate a copy of dictionary separately from the original one. However, in these cases the python philosophy is to offer a cloning method on the mutable object, so you can bind a copy rather than the original.
So I guess the answer is that there is no assignment in python, the best way to mimic it would be binding to a cloned object
I simply wanted to share the question in case I'm missing something important here
Thanks
EDIT:
Both Lie Ryan and Sven Marnach answers are good, I guess the overall answer is a mix of both:
For user defined types, use the idiom:
a.dict = dict(b.dict)
(I guess this has problems as well if the assigned class has redefined attribute access methods, but lets not be fussy :))
For mutable built-ins (lists and dicts) use the cloning / copying methods they provide (eg slices, update)
finally inmutable built-ins can't be changed so can't be assigned
I'll choose Lie Ryan because it's an elegant idiom that I hadn't thought of.
Thanks!

I think you are right with your characterization of assignment in Python -- I just would like to add a different method of cloning and ways of assignment in special cases.
"Copy-constructing" a mutable built-in Python object will yield a (shallow) copy of that object:
l = [2, 3]
m = list(l)
l is m
--> False
[Edit: As pointed out by Paul McGuire in the comments, the behaviour of a "copy contructor" (forgive me the C++ terminology) for a immutable built-in Python object is implementation dependent -- you might get a copy or just the same object. But because the object is immutable anyway, you shouldn't care.]
The copy constructor could be called generically by y = type(x)(x), but this seems a bit cryptic. And of course, there is the copy module which allows for shallow and deep copies.
Some Python objects allow assignment. For example, you can assign to a list without creating a new object:
l = [2, 3]
m = l
l[:] = [3, 4, 5]
m
--> [3, 4, 5]
For dictionaries, you could use the clear() method followed by update(otherdict) to assign to a dictionary without creating a new object. For a set s, you can use
s.clear()
s |= otherset

This brings the question: is it
possible to have proper assignment in
python, eg make an object equal to
another object?
Yes you can:
a.__dict__ = dict(b.__dict__)
will do the default assignment semantic in C/C++ (i.e. do a shallow assignment).
The problem with such generalized assignment is that it never works for everybody. In C++, you can override the assignment operator since you always have to pick whether you want a fully shallow assignment, fully deep assignment, or any shade between fully deep copy and fully shallow copy.

I don't think you are missing anything.
I like to picture variables in python as the name written on 'labels' that are attached to boxes but can change its placement by assignment, whereas in other languages, assignment changes the box's contents (and the assignment operator can be overloaded).
Beginners can write quite complex applications without being aware of that, but they are usually messy programs.

Why do we need tuples in Python (or any immutable data type)?

I've read several python tutorials (Dive Into Python, for one), and the language reference on Python.org - I don't see why the language needs tuples.
Tuples have no methods compared to a list or set, and if I must convert a tuple to a set or list to be able to sort them, what's the point of using a tuple in the first place?
Immutability?
Why does anyone care if a variable lives at a different place in memory than when it was originally allocated? This whole business of immutability in Python seems to be over emphasized.
In C/C++ if I allocate a pointer and point to some valid memory, I don't care where the address is located as long as it's not null before I use it.
Whenever I reference that variable, I don't need to know if the pointer is still pointing to the original address or not. I just check for null and use it (or not).
In Python, when I allocate a string (or tuple) assign it to x, then modify the string, why do I care if it's the original object? As long as the variable points to my data, that's all that matters.
>>> x='hello'
>>> id(x)
1234567
>>> x='good bye'
>>> id(x)
5432167
x still references the data I want, why does anyone need to care if its id is the same or different?

immutable objects can allow substantial optimization; this is presumably why strings are also immutable in Java, developed quite separately but about the same time as Python, and just about everything is immutable in truly-functional languages.
in Python in particular, only immutables can be hashable (and, therefore, members of sets, or keys in dictionaries). Again, this afford optimization, but far more than just "substantial" (designing decent hash tables storing completely mutable objects is a nightmare -- either you take copies of everything as soon as you hash it, or the nightmare of checking whether the object's hash has changed since you last took a reference to it rears its ugly head).
Example of optimization issue:
$ python -mtimeit '["fee", "fie", "fo", "fum"]'
1000000 loops, best of 3: 0.432 usec per loop
$ python -mtimeit '("fee", "fie", "fo", "fum")'
10000000 loops, best of 3: 0.0563 usec per loop

None of the answers above point out the real issue of tuples vs lists, which many new to Python seem to not fully understand.
Tuples and lists serve different purposes. Lists store homogenous data. You can and should have a list like this:
["Bob", "Joe", "John", "Sam"]
The reason that is a correct use of lists is because those are all homogenous types of data, specifically, people's names. But take a list like this:
["Billy", "Bob", "Joe", 42]
That list is one person's full name, and their age. That isn't one type of data. The correct way to store that information is either in a tuple, or in an object. Lets say we have a few :
[("Billy", "Bob", "Joe", 42), ("Robert", "", "Smith", 31)]
The immutability and mutability of Tuples and Lists is not the main difference. A list is a list of the same kind of items: files, names, objects. Tuples are a grouping of different types of objects. They have different uses, and many Python coders abuse lists for what tuples are meant for.
Please don't.
Edit:
I think this blog post explains why I think this better than I did:
Understanding tuples vs. lists in Python - E-Scribe

if I must convert a tuple to a set or list to be able to sort them, what's the point of using a tuple in the first place?
In this particular case, there probably isn't a point. This is a non-issue, because this isn't one of the cases where you'd consider using a tuple.
As you point out, tuples are immutable. The reasons for having immutable types apply to tuples:
copy efficiency: rather than copying an immutable object, you can alias it (bind a variable to a reference)
comparison efficiency: when you're using copy-by-reference, you can compare two variables by comparing location, rather than content
interning: you need to store at most one copy of any immutable value
there's no need to synchronize access to immutable objects in concurrent code
const correctness: some values shouldn't be allowed to change. This (to me) is the main reason for immutable types.
Note that a particular Python implementation may not make use of all of the above features.
Dictionary keys must be immutable, otherwise changing the properties of a key-object can invalidate invariants of the underlying data structure. Tuples can thus potentially be used as keys. This is a consequence of const correctness.
See also "Introducing tuples", from Dive Into Python.

Sometimes we like to use objects as dictionary keys
For what it's worth, tuples recently (2.6+) grew index() and count() methods

I've always found having two completely separate types for the same basic data structure (arrays) to be an awkward design, but not a real problem in practice. (Every language has its warts, Python included, but this isn't an important one.)
Why does anyone care if a variable lives at a different place in memory than when it was originally allocated? This whole business of immutability in Python seems to be over emphasized.
These are different things. Mutability isn't related to the place it's stored in memory; it means the stuff it points to can't change.
Python objects can't change location after they're created, mutable or not. (More accurately, the value of id() can't change--same thing, in practice.) The internal storage of mutable objects can change, but that's a hidden implementation detail.
>>> x='hello'
>>> id(x)
1234567
>>> x='good bye'
>>> id(x)
5432167
This isn't modifying ("mutating") the variable; it's creating a new variable with the same name, and discarding the old one. Compare to a mutating operation:
>>> a = [1,2,3]
>>> id(a)
3084599212L
>>> a[1] = 5
>>> a
[1, 5, 3]
>>> id(a)
3084599212L
As others have pointed out, this allows using arrays as keys to dictionaries, and other data structures that need immutability.
Note that keys for dictionaries do not have to be completely immutable. Only the part of it used as a key needs to be immutable; for some uses, this is an important distinction. For example, you could have a class representing a user, which compares equality and a hash by the unique username. You could then hang other mutable data on the class--"user is logged in", etc. Since this doesn't affect equality or the hash, it's possible and perfectly valid to use this as a key in a dictionary. This isn't too commonly needed in Python; I just point it out since several people have claimed that keys need to be "immutable", which is only partially correct. I've used this many times with C++ maps and sets, though.

As gnibbler offered in a comment, Guido had an opinion that is not fully accepted/appreciated: “lists are for homogeneous data, tuples are for heterogeneous data”. Of course, many of the opposers interpreted this as meaning that all elements of a list should be of the same type.
I like to see it differently, not unlike others also have in the past:
blue= 0, 0, 255
alist= ["red", "green", blue]
Note that I consider alist to be homogeneous, even if type(alist[1]) != type(alist[2]).
If I can change the order of the elements and I won't have issues in my code (apart from assumptions, e.g. “it should be sorted”), then a list should be used. If not (like in the tuple blue above), then I should use a tuple.

They are important since they guarantee the caller that the object they pass won't be mutated.
If you do this:
a = [1,1,1]
doWork(a)
The caller has no guarantee of the value of a after the call.
However,
a = (1,1,1)
doWorK(a)
Now you as the caller or as a reader of this code know that a is the same.
You could always for this scenario make a copy of the list and pass that but now you are wasting cycles instead of using a language construct that makes more semantic sense.

you can see here for some discussion on this

Your question (and follow-up comments) focus on whether the id() changes during an assignment. Focusing on this follow-on effect of the difference between immutable object replacement and mutable object modification rather than the difference itself is perhaps not the best approach.
Before we continue, make sure that the behavior demonstrated below is what you expect from Python.
>>> a1 = [1]
>>> a2 = a1
>>> print a2[0]
1
>>> a1[0] = 2
>>> print a2[0]
2
In this case, the contents of a2 was changed, even though only a1 had a new value assigned. Contrast to the following:
>>> a1 = (1,)
>>> a2 = a1
>>> print a2[0]
1
>>> a1 = (2,)
>>> print a2[0]
1
In this latter case, we replaced the entire list, rather than updating its contents. With immutable types such as tuples, this is the only behavior allowed.
Why does this matter? Let's say you have a dict:
>>> t1 = (1,2)
>>> d1 = { t1 : 'three' }
>>> print d1
{(1,2): 'three'}
>>> t1[0] = 0 ## results in a TypeError, as tuples cannot be modified
>>> t1 = (2,3) ## creates a new tuple, does not modify the old one
>>> print d1 ## as seen here, the dict is still intact
{(1,2): 'three'}
Using a tuple, the dictionary is safe from having its keys changed "out from under it" to items which hash to a different value. This is critical to allow efficient implementation.