Related
Sometimes it seems natural to have a default parameter which is an empty list. Yet Python produces unexpected behavior in these situations.
If for example, I have a function:
def my_func(working_list=[]):
working_list.append("a")
print(working_list)
The first time it is called, the default will work, but calls after that will update the existing list (with one "a" each call) and print the updated version.
So, what is the Pythonic way to get the behavior I desire (a fresh list on each call)?
def my_func(working_list=None):
if working_list is None:
working_list = []
# alternative:
# working_list = [] if working_list is None else working_list
working_list.append("a")
print(working_list)
The docs say you should use None as the default and explicitly test for it in the body of the function.
Other answers have already already provided the direct solutions as asked for, however, since this is a very common pitfall for new Python programmers, it's worth adding the explanation of why Python behaves this way, which is nicely summarized in The Hitchhikers Guide to Python under Mutable Default Arguments:
Python's default arguments are evaluated once when the function is defined, not each time the function is called (like it is in say, Ruby). This means that if you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.
Not that it matters in this case, but you can use object identity to test for None:
if working_list is None: working_list = []
You could also take advantage of how the boolean operator or is defined in python:
working_list = working_list or []
Though this will behave unexpectedly if the caller gives you an empty list (which counts as false) as working_list and expects your function to modify the list he gave it.
If the intent of the function is to modify the parameter passed as working_list, see HenryR's answer (=None, check for None inside).
But if you didn't intend to mutate the argument, just use it as starting point for a list, you can simply copy it:
def myFunc(starting_list = []):
starting_list = list(starting_list)
starting_list.append("a")
print starting_list
(or in this simple case just print starting_list + ["a"] but I guess that was just a toy example)
In general, mutating your arguments is bad style in Python. The only functions that are fully expected to mutate an object are methods of the object. It's even rarer to mutate an optional argument — is a side effect that happens only in some calls really the best interface?
If you do it from the C habit of "output arguments", that's completely unnecessary - you can always return multiple values as a tuple.
If you do this to efficiently build a long list of results without building intermediate lists, consider writing it as a generator and using result_list.extend(myFunc()) when you are calling it. This way your calling conventions remains very clean.
One pattern where mutating an optional arg is frequently done is a hidden "memo" arg in recursive functions:
def depth_first_walk_graph(graph, node, _visited=None):
if _visited is None:
_visited = set() # create memo once in top-level call
if node in _visited:
return
_visited.add(node)
for neighbour in graph[node]:
depth_first_walk_graph(graph, neighbour, _visited)
I might be off-topic, but remember that if you just want to pass a variable number of arguments, the pythonic way is to pass a tuple *args or a dictionary **kargs. These are optional and are better than the syntax myFunc([1, 2, 3]).
If you want to pass a tuple:
def myFunc(arg1, *args):
print args
w = []
w += args
print w
>>>myFunc(1, 2, 3, 4, 5, 6, 7)
(2, 3, 4, 5, 6, 7)
[2, 3, 4, 5, 6, 7]
If you want to pass a dictionary:
def myFunc(arg1, **kargs):
print kargs
>>>myFunc(1, option1=2, option2=3)
{'option2' : 2, 'option1' : 3}
Quote from https://docs.python.org/3/reference/compound_stmts.html#function-definitions
Default parameter values are evaluated from left to right when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended. A way around this is to use None as the default, and explicitly test for it in the body of the function, e.g.:
def whats_on_the_telly(penguin=None):
if penguin is None:
penguin = []
penguin.append("property of the zoo")
return penguin
Perhaps the simplest thing of all is to just create a copy of the list or tuple within the script. This avoids the need for checking. For example,
def my_funct(params, lst = []):
liste = lst.copy()
. .
There have already been good and correct answers provided. I just wanted to give another syntax to write what you want to do which I find more beautiful when you for instance want to create a class with default empty lists:
class Node(object):
def __init__(self, _id, val, parents=None, children=None):
self.id = _id
self.val = val
self.parents = parents if parents is not None else []
self.children = children if children is not None else []
This snippet makes use of the if else operator syntax. I like it especially because it's a neat little one-liner without colons, etc. involved and it nearly reads like a normal English sentence. :)
In your case you could write
def myFunc(working_list=None):
working_list = [] if working_list is None else working_list
working_list.append("a")
print working_list
I took the UCSC extension class Python for programmer
Which is true of: def Fn(data = []):
a) is a good idea so that your data lists start empty with every call.
b) is a good idea so that all calls to the function that do not provide any arguments on the call will get the empty list as data.
c) is a reasonable idea as long as your data is a list of strings.
d) is a bad idea because the default [] will accumulate data and the default [] will change with subsequent calls.
Answer:
d) is a bad idea because the default [] will accumulate data and the default [] will change with subsequent calls.
I've been working on a "learning project" which involves several short single-parameter functions, where the parameter might be either a numeric type or a list of numerics. E.g.,
def magnitude_of(v):
try:
return math.sqrt(sum([vi**2 for vi in v]))
except:
return abs(v)
This of course is necessary because if I did this:
def magnitude_of(v):
return math.sqrt(sum([vi**2 for vi in list(v)]))
...the code would fail because list() only accepts an iterable as an argument.
My question is: has there ever been consideration given at PSW to letting list() work with any argument? If so, what prevented the change from being implemented?
I would just create my own list function and use it for this particular purpose.
Ex:
def mlist(v):
try:
return list(v)
except(TypeError):
return [v]
Now you use mlist instead of list
l1 = mlist(1)
l2 = mlist([1])
Both will give [1] as the result
I cannot comment on whether this has come up at PSW, but I generally prefer using a ternary here instead of try... catch
e.g.
def magnitude_of(v):
return math.sqrt(sum([vi*vi for vi in (v if isinstance(v,list) else [v])]))
Granted, as others have mentioned it would probably be best if the function simply requires a list to be passed
This may sound a bit funny, but this is what am trying to do:
I have two lists: myList1 and myList2. I am passing these lists to a function that will perform a specific task depending on which list it receives (i.e., based on the name of the list, not content).
def listProcessor(data):
if data == 'myList1':
perform task1
else: #meaning data is myList2
perform task 2
Is there a way in Python 3 to inquire of the name of a list (or a tuple, dictionary, etc.) and do a comparison like this? Obviously, the way I am doing it here isn't working since it's comparing the content of the list 'data' to the single string 'myList1', which isn't working! :-).
There are a few ways to do this:
You can create separate functions if what needs to be done for each list really is completely separate.
You can update your function to either:
Take two lists as an argument:
def list_processor(list1=None, list2=None):
if list1:
# do stuff for list1
if list2:
# do stuff for list2
You can add an extra flag, identifying what kind of action to be performed, and set a default as well:
def list_processor(some_list=None, type_of_list=1):
if type_of_list == 1:
# do stuff with some_list as if it was list1
if type_of_list == 2:
# do stuff with some_list as if it was list2
You do not want to do what you initially proposed, for various reasons. One key reason is that in Python, what you may call variables in other languages are not "boxes to put stuff in" (as most textbooks refer to them).
In Python variables are actually just names that point to an object. The key thing is that multiple names can point to the same object; which will easily confuse your function if you rely on the lists "name".
Here is an example:
>>> a = [1,2,3]
>>> b = a
>>> b.append('hello')
>>> b is a
True
>>> b == a
True
>>> b
[1, 2, 3, 'hello']
>>> a
[1, 2, 3, 'hello']
In this example, both a and b are pointing to the same list object. What you do with a affects b.
So let's start with this: you really shouldn't do this. Data is just data in Python -- the identifier (e.g. the name you're using to talk about it) means nothing to the program logic itself. It's only meaningful to the programmer.
That said, there's ways to do what you're trying to do, and they're all the wrong thing to do. But they're possible, so let's talk about them.
globals() will give you a dictionary with keys of identifiers and values of, um, values, for all objects currently in the global scope. This means that you can do:
def list_processor(data):
g = globals()
data_name = next((k for k,v in g.items() if v is data))
if data_name == 'myList1':
... # do whatever you want here
Note, however, that you're looking through EVERYTHING in the global scope. First off, that's dumb since it's slow, and secondly, it's dumb because it's buggy. What if myList1 isn't being passed from the global scope? What if it's a local variable inside a function that never hits the global scope? Now your tricky hack fails.
The correct way to do this is to perform some sort of introspection on the argument being passed in as "data". For instance if myList1 always has 8 elements and myList2 always has 10:
def f_if_myList1(lst):
"""Do whatever we do if we pass myList1"""
...
def f_if_myList2(lst):
"""Do whatever we do if we pass myList2"""
...
def list_processor(data):
if len(data) == 8: # myList1!
f_if_myList1(data)
elif len(data) == 10:
f_if_myList2(data)
else:
# not sure. raise ValueError()?
I want to implement a custom list class in Python as a subclass of list. What is the minimal set of methods I need to override from the base list class in order to get full type compatibility for all list operations?
This question suggest that at least __getslice__ needs to be overridden. From further research, also __add__ and __mul__ will be required. So I have this code:
class CustomList(list):
def __getslice__(self,i,j):
return CustomList(list.__getslice__(self, i, j))
def __add__(self,other):
return CustomList(list.__add__(self,other))
def __mul__(self,other):
return CustomList(list.__mul__(self,other))
The following statements work as desired, even without the overriding methods:
l = CustomList((1,2,3))
l.append(4)
l[0] = -1
l[0:2] = CustomList((10,11)) # type(l) is CustomList
These statements work only with the overriding methods in the above class definition:
l3 = l + CustomList((4,5,6)) # type(l3) is CustomList
l4 = 3*l # type(l4) is CustomList
l5 = l[0:2] # type(l5) is CustomList
The only thing I don't know how to achieve is making extended slicing return the right type:
l6 = l[0:2:2] # type(l6) is list
What do I need to add to my class definition in order to get CustomList as type of l6?
Also, are there other list operations other than extended slicing, where the result will be of list type instead of CustomList?
Firstly, I recommend you follow Björn Pollex's advice (+1).
To get past this particular problem (type(l2 + l3) == CustomList), you need to implement a custom __add__():
def __add__(self, rhs):
return CustomList(list.__add__(self, rhs))
And for extended slicing:
def __getitem__(self, item):
result = list.__getitem__(self, item)
try:
return CustomList(result)
except TypeError:
return result
I also recommend...
pydoc list
...at your command prompt. You'll see which methods list exposes and this will give you a good indication as to which ones you need to override.
You should probably read these two sections from the documentation:
Emulating container types
Additional methods for emulating sequence types (Python 2 only)
Edit: In order to handle extended slicing, you should make your __getitem__-method handle slice-objects (see here, a little further down).
Possible cut-the-gordian-knot solution: subclass UserList instead of list. (Worked for me.) That is what UserList is there for.
As a slight modification to Johnsywebs answer. I would only convert to a CustomList if item is a slice. Otherwise CustomList(["ab"])[0] would give you CustomList(["a", "b"]) which is not what you want. Like this:
def __getitem__(self, item):
result = list.__getitem__(self, item)
if type(item) is slice:
return CustomList(result)
else:
return result
Sometimes it seems natural to have a default parameter which is an empty list. Yet Python produces unexpected behavior in these situations.
If for example, I have a function:
def my_func(working_list=[]):
working_list.append("a")
print(working_list)
The first time it is called, the default will work, but calls after that will update the existing list (with one "a" each call) and print the updated version.
So, what is the Pythonic way to get the behavior I desire (a fresh list on each call)?
def my_func(working_list=None):
if working_list is None:
working_list = []
# alternative:
# working_list = [] if working_list is None else working_list
working_list.append("a")
print(working_list)
The docs say you should use None as the default and explicitly test for it in the body of the function.
Other answers have already already provided the direct solutions as asked for, however, since this is a very common pitfall for new Python programmers, it's worth adding the explanation of why Python behaves this way, which is nicely summarized in The Hitchhikers Guide to Python under Mutable Default Arguments:
Python's default arguments are evaluated once when the function is defined, not each time the function is called (like it is in say, Ruby). This means that if you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.
Not that it matters in this case, but you can use object identity to test for None:
if working_list is None: working_list = []
You could also take advantage of how the boolean operator or is defined in python:
working_list = working_list or []
Though this will behave unexpectedly if the caller gives you an empty list (which counts as false) as working_list and expects your function to modify the list he gave it.
If the intent of the function is to modify the parameter passed as working_list, see HenryR's answer (=None, check for None inside).
But if you didn't intend to mutate the argument, just use it as starting point for a list, you can simply copy it:
def myFunc(starting_list = []):
starting_list = list(starting_list)
starting_list.append("a")
print starting_list
(or in this simple case just print starting_list + ["a"] but I guess that was just a toy example)
In general, mutating your arguments is bad style in Python. The only functions that are fully expected to mutate an object are methods of the object. It's even rarer to mutate an optional argument — is a side effect that happens only in some calls really the best interface?
If you do it from the C habit of "output arguments", that's completely unnecessary - you can always return multiple values as a tuple.
If you do this to efficiently build a long list of results without building intermediate lists, consider writing it as a generator and using result_list.extend(myFunc()) when you are calling it. This way your calling conventions remains very clean.
One pattern where mutating an optional arg is frequently done is a hidden "memo" arg in recursive functions:
def depth_first_walk_graph(graph, node, _visited=None):
if _visited is None:
_visited = set() # create memo once in top-level call
if node in _visited:
return
_visited.add(node)
for neighbour in graph[node]:
depth_first_walk_graph(graph, neighbour, _visited)
I might be off-topic, but remember that if you just want to pass a variable number of arguments, the pythonic way is to pass a tuple *args or a dictionary **kargs. These are optional and are better than the syntax myFunc([1, 2, 3]).
If you want to pass a tuple:
def myFunc(arg1, *args):
print args
w = []
w += args
print w
>>>myFunc(1, 2, 3, 4, 5, 6, 7)
(2, 3, 4, 5, 6, 7)
[2, 3, 4, 5, 6, 7]
If you want to pass a dictionary:
def myFunc(arg1, **kargs):
print kargs
>>>myFunc(1, option1=2, option2=3)
{'option2' : 2, 'option1' : 3}
Quote from https://docs.python.org/3/reference/compound_stmts.html#function-definitions
Default parameter values are evaluated from left to right when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended. A way around this is to use None as the default, and explicitly test for it in the body of the function, e.g.:
def whats_on_the_telly(penguin=None):
if penguin is None:
penguin = []
penguin.append("property of the zoo")
return penguin
Perhaps the simplest thing of all is to just create a copy of the list or tuple within the script. This avoids the need for checking. For example,
def my_funct(params, lst = []):
liste = lst.copy()
. .
There have already been good and correct answers provided. I just wanted to give another syntax to write what you want to do which I find more beautiful when you for instance want to create a class with default empty lists:
class Node(object):
def __init__(self, _id, val, parents=None, children=None):
self.id = _id
self.val = val
self.parents = parents if parents is not None else []
self.children = children if children is not None else []
This snippet makes use of the if else operator syntax. I like it especially because it's a neat little one-liner without colons, etc. involved and it nearly reads like a normal English sentence. :)
In your case you could write
def myFunc(working_list=None):
working_list = [] if working_list is None else working_list
working_list.append("a")
print working_list
I took the UCSC extension class Python for programmer
Which is true of: def Fn(data = []):
a) is a good idea so that your data lists start empty with every call.
b) is a good idea so that all calls to the function that do not provide any arguments on the call will get the empty list as data.
c) is a reasonable idea as long as your data is a list of strings.
d) is a bad idea because the default [] will accumulate data and the default [] will change with subsequent calls.
Answer:
d) is a bad idea because the default [] will accumulate data and the default [] will change with subsequent calls.