How to remove duplicates from list in python - python

I have a simple code here:
a=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
b=[]
for i in range (len(a)):
if a[i] not in b:
b.append([a[i]])
print (b)
The output i get is
b=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
i.e. the same as a
The output i need is
b=[['bcn09','113','shift1'],['bps01','132','shift2']]
What am i doing wrong?
Thanks in advance

Very close! You could try this approach, hopefully it makes sense:
a=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
noDups = []
for i in a:
if i not in noDups:
noDups.append(i)
print(noDups)
Output:
>>>[['bcn09', '113', 'shift1'], ['bps01', '132', 'shift2']]

There is no need to create a new list when appending to 'b'.
Modify the below line from -
b.append([a[i]])
to
b.append(a[i])
The new output is (which you want) -
[['bcn09', '113', 'shift1'], ['bps01', '132', 'shift2']]
This explains more about the 'in' membership test operator.

For the most complete answer to getting only unique values from an iterable, itertools gives a recipe (which can also be found in the more-itertools package on PyPI) for this:
def unique_everseen(iterable, key=None):
"List unique elements, preserving order. Remember all elements ever seen."
# unique_everseen('AAAABBBCCDAABBB') --> A B C D
# unique_everseen('ABBCcAD', str.lower) --> A B C D
seen = set()
seen_add = seen.add
if key is None:
for element in filterfalse(seen.__contains__, iterable):
seen_add(element)
yield element
else:
for element in iterable:
k = key(element)
if k not in seen:
seen_add(k)
yield element
This implementation is tested, optimised, maintains order, is a generator so it will work on infinite iterables or cases where you need to be lazy, and allows you to use a key function.
>>> a=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
>>> list(unique_everseen(tuple(item) for item in a))
[('bcn09', '113', 'shift1'), ('bps01', '132', 'shift2')]
Note the change to tuples so the elements are hashable and can be added to the set. You can of course reverse this at the end if needed, although in most cases I can imagine tuples will probably be fine. (In the same way, I am creating a list from the generator to show the output, but most of the time you should be able to just work with the generator directly).

you can achive by using itertools
import itertools
a = [['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
a.sort()
new_num = list(a for a,_ in itertools.groupby(a))
print("New List", new_num)

Related

Check number not a sum of 2 ints on a list

Given a list of integers, I want to check a second list and remove from the first only those which can not be made from the sum of two numbers from the second. So given a = [3,19,20] and b = [1,2,17], I'd want [3,19].
Seems like a a cinch with two nested loops - except that I've gotten stuck with break and continue commands.
Here's what I have:
def myFunction(list_a, list_b):
for i in list_a:
for a in list_b:
for b in list_b:
if a + b == i:
break
else:
continue
break
else:
continue
list_a.remove(i)
return list_a
I know what I need to do, just the syntax seems unnecessarily confusing. Can someone show me an easier way? TIA!
You can do like this,
In [13]: from itertools import combinations
In [15]: [item for item in a if item in [sum(i) for i in combinations(b,2)]]
Out[15]: [3, 19]
combinations will give all possible combinations in b and get the list of sum. And just check the value is present in a
Edit
If you don't want to use the itertools wrote a function for it. Like this,
def comb(s):
for i, v1 in enumerate(s):
for j in range(i+1, len(s)):
yield [v1, s[j]]
result = [item for item in a if item in [sum(i) for i in comb(b)]]
Comments on code:
It's very dangerous to delete elements from a list while iterating over it. Perhaps you could append items you want to keep to a new list, and return that.
Your current algorithm is O(nm^2), where n is the size of list_a, and m is the size of list_b. This is pretty inefficient, but a good start to the problem.
Thee's also a lot of unnecessary continue and break statements, which can lead to complicated code that is hard to debug.
You also put everything into one function. If you split up each task into different functions, such as dedicating one function to finding pairs, and one for checking each item in list_a against list_b. This is a way of splitting problems into smaller problems, and using them to solve the bigger problem.
Overall I think your function is doing too much, and the logic could be condensed into much simpler code by breaking down the problem.
Another approach:
Since I found this task interesting, I decided to try it myself. My outlined approach is illustrated below.
1. You can first check if a list has a pair of a given sum in O(n) time using hashing:
def check_pairs(lst, sums):
lookup = set()
for x in lst:
current = sums - x
if current in lookup:
return True
lookup.add(x)
return False
2. Then you could use this function to check if any any pair in list_b is equal to the sum of numbers iterated in list_a:
def remove_first_sum(list_a, list_b):
new_list_a = []
for x in list_a:
check = check_pairs(list_b, x)
if check:
new_list_a.append(x)
return new_list_a
Which keeps numbers in list_a that contribute to a sum of two numbers in list_b.
3. The above can also be written with a list comprehension:
def remove_first_sum(list_a, list_b):
return [x for x in list_a if check_pairs(list_b, x)]
Both of which works as follows:
>>> remove_first_sum([3,19,20], [1,2,17])
[3, 19]
>>> remove_first_sum([3,19,20,18], [1,2,17])
[3, 19, 18]
>>> remove_first_sum([1,2,5,6],[2,3,4])
[5, 6]
Note: Overall the algorithm above is O(n) time complexity, which doesn't require anything too complicated. However, this also leads to O(n) extra auxiliary space, because a set is kept to record what items have been seen.
You can do it by first creating all possible sum combinations, then filtering out elements which don't belong to that combination list
Define the input lists
>>> a = [3,19,20]
>>> b = [1,2,17]
Next we will define all possible combinations of sum of two elements
>>> y = [i+j for k,j in enumerate(b) for i in b[k+1:]]
Next we will apply a function to every element of list a and check if it is present in above calculated list. map function can be use with an if/else clause. map will yield None in case of else clause is successful. To cater for this we can filter the list to remove None values
>>> list(filter(None, map(lambda x: x if x in y else None,a)))
The above operation will output:
>>> [3,19]
You can also write a one-line by combining all these lines into one, but I don't recommend this.
you can try something like that:
a = [3,19,20]
b= [1,2,17,5]
n_m_s=[]
data=[n_m_s.append(i+j) for i in b for j in b if i+j in a]
print(set(n_m_s))
print("after remove")
final_data=[]
for j,i in enumerate(a):
if i not in n_m_s:
final_data.append(i)
print(final_data)
output:
{19, 3}
after remove
[20]

Why does removing duplicates from a list produce [None, None] output?

I am new to Python and I'm not able to understand why I am getting the results with None values.
#Remove duplicate items from a list
def remove_duplicates(list):
unique_list = []
return [unique_list.append(item) for item in list if item not in unique_list]
print remove_duplicates([1,1,2,2]) -> result [None, None]
When I print the result it shows the following: [None, None]
PS: I've seen other solutions and also aware of the list(set(list)) but I am trying to understand why the above result with integers gives [None, None] output.
Although using a set is the proper way, the problem with your code, as the comments indicated, is that you are not actually returning unique_list from your function, you are returning the result of the list comprehension.
def remove_duplicates(my_list):
unique_list = []
do = [unique_list.append(item) for item in my_list if item not in unique_list]
return unique_list # Actually return the list!
print remove_duplicates([1,1,2,2]) -> result [1, 2]
Here I simply made a throwaway variable do that is useless, it just "runs" the comprehension. Understand?
That comprehension is storing a value each time you call unique_list.append(item) ... and that value is the result of the append method, which is None! So do equals [None, None].
However, your unique_list is in fact being populated correctly, so we can return that and now your function works as expected.
Of course, this is not a normal use for a list comprehension and really weird.
The problem with your code is that the method list.append returns None. You can test this easily with the following code:
myList=[1, 2, 3]
print myList.append(4)
So, a solution for you would issue would be
def remove_duplicates(myList):
alreadyIncluded = []
return [item for item in myList if item not in alreadyIncluded and not alreadyIncluded.append(item)]
print remove_duplicates([1,1,2,2])
The idea is that you will begin with an empty list of aldeady included elements and you will loop over all the elements in list, including them in the alreadyIncluded list. The not is necessary because the append will return None and not None is True, so the if will not be affected by the inclusion.
You were including a list of the result of the appends (always None), but what you need is a list of the elements that passed the if test.
I hope it helps.
As the other answers have explained, the reason you're getting a list of None values is because list.append returns None, and you're calling it in a list comprehension. That means you're building a list full of None values along side your list of unique values.
I would like to suggest that you ditch the list comprehension. Because you need to access outside state (the list of unique values seen so far), a comprehension can't easily do what you want. A regular for loop is much more appropriate:
def remove_duplicates(lst):
unique_list = []
for item in lst:
if item not in unique_list:
unique_list.append(item)
return unique_list
A more Pythonic approach however would be to use a set to handle the unique items, and to make your function a generator:
def remove_duplicates(lst):
uniques = set()
for item in lst:
if item not in unique_list:
yield item
uniques.add(item)
The itertools.ifilterfase function from the standard library can help improve this even further, as shown in the recipe in the docs (you'll have to scroll down a little to find the specific recipe):
def unique_everseen(iterable, key=None):
"List unique elements, preserving order. Remember all elements ever seen."
# unique_everseen('AAAABBBCCDAABBB') --> A B C D
# unique_everseen('ABBCcAD', str.lower) --> A B C D
seen = set()
seen_add = seen.add
if key is None:
for element in filterfalse(seen.__contains__, iterable):
seen_add(element)
yield element
else:
for element in iterable:
k = key(element)
if k not in seen:
seen_add(k)
yield element

Python List Indexing or Appending?

What is the best way to add values to a List in terms of processing time, memory usage and just generally what is the best programming option.
list = []
for i in anotherArray:
list.append(i)
or
list = range(len(anotherArray))
for i in list:
list[i] = anotherArray[i]
Considering that anotherArray is for example an array of Tuples. (This is just a simple example)
It really depends on your use case. There is no generic answer here as it depends on what you are trying to do.
In your example, it looks like you are just trying to create a copy of the array, in which case the best way to do this would be to use copy:
from copy import copy
list = copy(anotherArray)
If you are trying to transform the array into another array you should use list comprehension.
list = [i[0] for i in anotherArray] # get the first item from tuples in anotherArray
If you are trying to use both indexes and objects, you should use enumerate:
for i, j in enumerate(list)
which is much better than your second example.
You can also use generators, lambas, maps, filters, etc. The reason all of these possibilities exist is because they are all "better" for different reasons. The writters of python are pretty big on "one right way", so trust me, if there was one generic way which was always better, that is the only way that would exist in python.
Edit: Ran some results of performance for tuple swap and here are the results:
comprehension: 2.682028295999771
enumerate: 5.359116118001111
for in append: 4.177091988000029
for in indexes: 4.612594166001145
As you can tell, comprehension is usually the best bet. Using enumerate is expensive.
Here is the code for the above test:
from timeit import timeit
some_array = [(i, 'a', True) for i in range(0,100000)]
def use_comprehension():
return [(b, a, i) for i, a, b in some_array]
def use_enumerate():
lst = []
for j, k in enumerate(some_array):
i, a, b = k
lst.append((b, a, i))
return lst
def use_for_in_with_append():
lst = []
for i in some_array:
i, a, b = i
lst.append((b, a, i))
return lst
def use_for_in_with_indexes():
lst = [None] * len(some_array)
for j in range(len(some_array)):
i, a, b = some_array[j]
lst[j] = (b, a, i)
return lst
print('comprehension:', timeit(use_comprehension, number=200))
print('enumerate:', timeit(use_enumerate, number=200))
print('for in append:', timeit(use_for_in_with_append, number=200))
print('for in indexes:', timeit(use_for_in_with_indexes, number=200))
Edit2:
It was pointed out to me the the OP just wanted to know the difference between "indexing" and "appending". Really, those are used for two different use cases as well. Indexing is for replacing objects, whereas appending is for adding. However, in a case where the list starts empty, appending will always be better because the indexing has the overhead of creating the list initially. You can see from the results above that indexing is slightly slower, mostly because you have to create the first list.
Best way is list comprehension :
my_list=[i for i in anotherArray]
But based on your problem you can use a generator expression (is more efficient than list comprehension when you just want to loop over your items and you don't need to use some list methods like indexing or len or ... )
my_list=(i for i in anotherArray)
I would actually say the best is a combination of index loops and value loops with enumeration:
for i, j in enumerate(list): # i is the index, j is the value, can't go wrong

Find how many lists in list have the same element

I am new at Python, so I'm having trouble with something. I have a few string lists in one list.
list=[ [['AA','A0'],['AB','A0']],
[['AA','B0'],['AB','A0']],
[['A0','00'],['00','A0'], [['00','BB'],['AB','A0'],['AA','A0']] ]
]
And I have to find how many lists have the same element. For example, the correct result for the above list is 3 for the element ['AB','A0'] because it is the element that connects the most of them.
I wrote some code...but it's not good...it works for 2 lists in list,but not for more....
Please,help!
This is my code...for the above list...
for t in range(0,len(list)-1):
pattern=[]
flag=True
pattern.append(list[t])
count=1
rest=list[t+1:]
for p in pattern:
for j in p:
if flag==False:
break
pair= j
for i in rest:
for y in i:
if pair==y:
count=count+1
break
if brojac==len(list):
flag=False
break
Since your data structure is rather complex, you might want to build a recursive function, that is a function that calls itself (http://en.wikipedia.org/wiki/Recursion_(computer_science)).
This function is rather simple. You iterate through all items of the original list. If the current item is equal to the value you are searching for, you increment the number of found objects by 1. If the item is itself a list, you will go through that sub-list and find all matches in that sub-list (by calling the same function on the sub-list, instead of the original list). You then increment the total number of found objects by the count in your sub-list. I hope my explanation is somewhat clear.
alist=[[['AA','A0'],['AB','A0']],[['AA','B0'],['AB','A0']],[['A0','00'],['00','A0'],[['00','BB'],['AB','A0'],['AA','A0']]]]
def count_in_list(val, arr):
val_is_list = isinstance(val, list)
ct = 0
for item in arr:
item_is_list = isinstance(item, list)
if item == val or (val_is_list and item_is_list and sorted(item) == sorted(val)):
ct += 1
if item_is_list :
ct += count_in_list(val, item)
return ct
print count_in_list(['AB', 'A0'], alist)
This is an iterative approach that will also work using python3 that will get the count of all sublists:
from collections import defaultdict
d = defaultdict(int)
def counter(lst, d):
it = iter(lst)
nxt = next(it)
while nxt:
if isinstance(nxt, list):
if nxt and isinstance(nxt[0], str):
d[tuple(nxt)] += 1
rev = tuple(reversed(nxt))
if rev in d:
d[rev] += 1
else:
lst += nxt
nxt = next(it,"")
return d
print((counter(lst, d)['AB', 'A0'])
3
It will only work on data like your input, nesting of strings beside lists will break the code.
To get a single sublist count is easier:
def counter(lst, ele):
it = iter(lst)
nxt = next(it)
count = 0
while nxt:
if isinstance(nxt, list):
if ele in (nxt, nxt[::-1]):
count += 1
else:
lst += nxt
nxt = next(it, "")
return count
print(counter(lst, ['AB', 'A0']))
3
Ooookay - this maybe isn't very nice and straightforward code, but that's how i'd try to solve this. Please don't hurt me ;-)
First,
i'd fragment the problem in three smaller ones:
Get rid of your multiple nested lists,
Count the occurence of all value-pairs in the inner lists and
Extract the most occurring value-pair from the counting results.
1.
I'd still use nested lists, but only of two-levels depth. An outer list, to iterate through, and all the two-value-lists inside of it. You can finde an awful lot of information about how to get rid of nested lists right here. As i'm just a beginner, i couldn't make much out of all that very detailed information - but if you scroll down, you'll find an example similar to mine. This is what i understand, this is how i can do.
Note that it's a recursive function. As you mentioned in comments that you think this isn't easy to understand: I think you're right. I'll try to explain it somehow:
I don't know if the nesting depth is consistent in your list. and i don't want to exctract the values themselves, as you want to work with lists. So this function loops through the outer list. For each element, it checks if it's a list. If not, nothing happens. If it is a list, it'll have a look at the first element inside of that list. It'll check again if it's a list or not.
If the first element inside the current list is another list, the function will be called again - recursive - but this time starting with the current inner list. This is repeated until the function finds a list, containing an element on the first position that is NOT a list.
In your example, it'll dig through the complete list-of-lists, until it finds your first string values. Then it gets the list containing this value - and put that in another list, the one that is returned.
Oh boy, that sounds really crazy - tell me if that clarified anything... :-D
"Yo dawg, i herd you like lists, so i put a list in a list..."
def get_inner_lists(some_list):
inner_lists = []
for item in some_list:
if hasattr(item, '__iter__') and not isinstance(item, basestring):
if hasattr(item[0], '__iter__') and not isinstance(item[0], basestring):
inner_lists.extend(get_inner_lists(item))
else:
inner_lists.append(item)
return inner_lists
Whatever - call that function and you'll find your list re-arranged a little bit:
>>> foo = [[['AA','A0'],['AB','A0']],[['AA','B0'],['AB','A0']],[['A0','00'],['00','A0'],[['00','BB'],['AB','A0'],['AA','A0']]]]
>>> print get_inner_lists(foo)
[['AA', 'A0'], ['AB', 'A0'], ['AA', 'B0'], ['AB', 'A0'], ['A0', '00'], ['00', 'A0'], ['00', 'BB'], ['AB', 'A0'], ['AA', 'A0']]
2.
Now i'd iterate through that lists and build a string with their values. This will only work with lists of two values, but as this is what you showed in your example it'll do. While iterating, i'd build up a dictionary with the strings as keys and the occurrence as values. That makes it really easy to add new values and raise the counter of existing ones:
def count_list_values(some_list):
result = {}
for item in some_list:
str = item[0]+'-'+item[1]
if not str in result.keys():
result[str] = 1
else:
result[str] += 1
return result
There you have it, all the counting is done. I don't know if it's needed, but as a side effect there are all values and all occurrences:
>>> print count_list_values(get_inner_lists(foo))
{'00-A0': 1, '00-BB': 1, 'A0-00': 1, 'AB-A0': 3, 'AA-A0': 2, 'AA-B0': 1}
3.
But you want clear results, so let's loop through that dictionary, list all keys and all values, find the maximum value - and return the corresponding key. Having built the string-of-two-values with a seperator (-), it's easy to split it and make a list out of it, again:
def get_max_dict_value(some_dict):
all_keys = []
all_values = []
for key, val in some_dict.items():
all_keys.append(key)
all_values.append(val)
return all_keys[all_values.index(max(all_values))].split('-')
If you define this three little functions and call them combined, this is what you'll get:
>>> print get_max_dict_value(count_list_values(get_inner_lists(foo)))
['AB', 'A0']
Ta-Daa! :-)
If you really have such lists with only nine elements, and you don't need to count values that often - do it manually. By reading values and counting with fingers. It'll be so much easier ;-)
Otherwise, here you go!
Or...
...you wait until some Guru shows up and gives you a super fast, elegant one-line python command that i've never seen before, which will do the same ;-)
This is as simple as I can reasonably make it:
from collections import Counter
lst = [ [['AA','A0'],['AB','A0']],
[['AA','B0'],['AB','A0']],
[['A0','00'],['00','A0'], [['00','BB'],['AB','A0'],['AA','A0']] ]
]
def is_leaf(element):
return (isinstance(element, list) and
len(element) == 2 and
isinstance(element[0], basestring)
and isinstance(element[1], basestring))
def traverse(iterable):
for element in iterable:
if is_leaf(element):
yield tuple(sorted(element))
else:
for value in traverse(element):
yield value
value, count = Counter(traverse(lst)).most_common(1)[0]
print 'Value {!r} is present {} times'.format(value, count)
The traverse() generate yields a series of sorted tuples representing each item in your list. The Counter object counts the number of occurrences of each, and its .most_common(1) method returns the value and count of the most common item.
You've said recursion is too difficult, but I beg to differ: it's the simplest way possible to attack this problem. The sooner you come to love recursion, the happier you'll be. :-)
Hopefully soemthing like this is what you were looking for. It is a bit tenuous and would suggest that recursion is better. But Since you didn't want it that way here is some code that might work. I am not super good at python but hope it will do the job:
def Compare(List):
#Assuming that the list input is a simple list like ["A1","B0"]
myList =[[['AA','A0'],['AB','A0']],[['AA','B0'],['AB','A0']],[['A0','00'],['00','A0'],[['00','BB'],['AB','A0'],['AA','A0']]]]
#Create a counter that will count if the elements are the same
myCounter = 0;
for innerList1 in myList:
for innerList2 in innerList1
for innerList3 in innerList2
for element in innerList3
for myListElements in myList
if (myListElements == element)
myCounter = myCounter + 1;
#I am putting the break here so that it counts how many lists have the
#same elements, not how many elements are the same in the lists
break;
return myCounter;

How can I make my code be a set?

I have a little code that takes a list of objects, and only outputs the items in the list that are unique.
This is my code
def only_once(a):
return [x for x in a if a.count(x) is 1]
My teacher requires us to use sets for this function though.
Can someone show me what I can do?
My code has to take an input such as a=[1,4,6,7,3,2,4,5,7,5,6], and output [1, 3, 2]. Has to retain it's order also.
[I'm assuming that you're also user1744238 and user1744316 -- please pick a username and stick to it, that way it's easier to check to see what variants of a question you've asked and what you've already tried.]
One set-based approach is to use two sets as a counter. You only care about whether you've seen something once or more than once. For example, here's an easy-to-explain approach:
Make an empty set for once and more.
Loop over every element of your list, and:
If you haven't seen it before, add it to once.
If you've seen it once, remove it from once and add it to more.
Now you know what elements you've seen exactly once, in the set once.
Loop over the elements of the list, and if you've seen it once, add it to the output list, and remove it from the once set so you don't output the same element twice.
This gives me:
In [49]: f([1,4,6,7,3,2,4,5,7,5,6])
Out[49]: [1, 3, 2]
To clarify, what you want is a set of items that appear once, and only once.
The best option here is to use collections.Counter(), as it means you only count the items once, rather than once per item, greatly increasing performance:
>>> import collections
>>> {key for key, count in collections.Counter(a).items() if count == 1}
{1, 2, 3}
We simply replace the square brackets with curly braces to signify a set comprehension over a list comprehension, to get a set of results.
If you need to remove any item that is in the list more than once, not just occurences after the first, you can use:
# without using generators / comprehensions
def only_once(iterable):
seen = set()
duplicates = set()
for item in iterable:
if item in seen:
duplicates.add(item)
seen.add(item)
result = []
for item in iterable:
if item not in duplicates:
result.append(item)
return result
For general order-preserving duplicate elimination, see unique_everseen in the itertools recipes:
def unique_everseen(iterable, key=None):
"List unique elements, preserving order. Remember all elements ever seen."
# unique_everseen('AAAABBBCCDAABBB') --> A B C D
# unique_everseen('ABBCcAD', str.lower) --> A B C D
seen = set()
seen_add = seen.add
if key is None:
for element in ifilterfalse(seen.__contains__, iterable):
seen_add(element)
yield element
else:
for element in iterable:
k = key(element)
if k not in seen:
seen_add(k)
yield element

Categories