Related
So I currently have a nested list.
org_network=[[1, 2, 3], [1, 4, 5], [1, 3, 6], [7, 9, 10]]
I need to figure out how to manipulate it to create lists of possible combinations of the nested lists. These combinations cannot have lists that share numbers. Here is an example of what the result should be:
network_1=[[1,2,3],[7,9,10]]
network_2=[[1,4,5],[7,9,10]]
network_3=[[1,3,6],[7,9,10]]
Note:
1. This code is going to be linked to a constantly updated csv file, so the org_network list will have varying amounts of elements within it (which also means that there will be numerous resulting networks.
I have been working on this for about four hours and have yet to figure it out. Any help would be very appreciated. I have primarily been trying to use for loops and any() functions to no avail. Thanks for any help.
You can use itertools.combinations() with set intersection:
>>> from itertools import combinations
>>> org_network=[[1, 2, 3], [1, 4, 5], [1, 3, 6], [7, 9, 10]]
>>> [[x, y] for x, y in combinations(org_network, r=2) if not set(x).intersection(y)]
[[[1, 2, 3], [7, 9, 10]], [[1, 4, 5], [7, 9, 10]], [[1, 3, 6], [7, 9, 10]]]
Here is an approach that will be efficient if the number of unique elements is small relative to the number of sets.
Steps:
For each unique element, store indices of all sets in which the element does not occur.
For each set s in the network, find all other sets that contain every element of s using data from the first step.
Iterate over pairs, discarding duplicates based on ID order.
from functools import reduce
org_network = [[1, 2, 3], [1, 4, 5], [1, 3, 6], [7, 9, 10]]
# convert to sets
sets = [set(lst) for lst in org_network]
# all unique numbers
uniqs = set().union(*sets)
# map each unique number to sets that do not contain it:
other = {x: {i for i, s in enumerate(sets) if x not in s} for x in uniqs}
# iterate over sets:
for i, s in enumerate(sets):
# find all sets not overlapping with i
no_overlap = reduce(lambda l, r: l.intersection(r), (other[x] for x in s))
# iterate over non-overlapping sets
for j in no_overlap:
# discard duplicates
if j <= i:
continue
print([org_network[i], org_network[j]])
# result
# [[1, 2, 3], [7, 9, 10]]
# [[1, 4, 5], [7, 9, 10]]
# [[1, 3, 6], [7, 9, 10]]
Edit: If combinations of size greater than two are required, it is possible to modify the above approach. Here is an extension that uses depth-first search to traverse all pairwise disjoint combinations.
def not_overlapping(set_ids):
candidates = reduce(
lambda l, r: l.intersection(r), (other[x] for sid in set_ids for x in sets[sid])
)
mid = max(set_ids)
return {c for c in candidates if c > mid}
# this will produce "combinations" consisting of a single element
def iter_combinations():
combs = [[i] for i in range(len(sets))]
while combs:
comb = combs.pop()
extension = not_overlapping(comb)
combs.extend(comb + [e] for e in extension)
yield [org_network[i] for i in comb]
def iter_combinations_long():
for comb in iter_combinations():
if len(comb) > 1:
yield comb
all_combs = list(iter_combinations_long())
I learned to create and manage nested lists by building a NxM matrix and filling it with random integers.
I've solved it via a simple one-liner and struggled to write a comprehensive solution.
I call the first solution as a cheating because I got this tip from this website, but didn't completely understand how it works. So, I've tried to write a more detailed piece of code on my own, which was difficult, and I'm not completely sure I got it right.
The original solution:
from random import randint
size_n = int(input("Number of lists N "))
size_m = int(input("Number of elements M "))
matrix = [[randint(1,9) for m in range(size_m)] for n in range(size_n)]
print(matrix)
I had a hard time to create a list of list with the correct levels of loops in loops. I end up with the temporary matrix, but I'm not sure it's good solution.
The final solution:
from random import randint
size_n = int(input("Number of lists N "))
size_m = int(input("Number of elements M "))
matrix = []
for n in range(size_n):
tmp = []
for m in range(size_m):
tmp.append(randint(1,9))
matrix.append(tmp)
print(matrix)
Can you help me to understand what is the correct solution for the task?
P.S. Is it normal for developer to look for another solution if the code just works but you think it might be prettier?
matrix = [[randint(1,9) for m in range(size_m)] for n in range(size_n)]
Let's say our size_n is 5, for an example. Then range(size_n) gives
[0, 1, 2, 3, 4]
E. g. a list of 5 consecutive integers starting with 0. If you do something for each element of the list, that creates another list of the same size, for example:
>>>[88 for n in range(size_n)]
[88, 88, 88, 88, 88]
You are, of course, not limited by simple numbers, you can create, basically, anything. For example, empty lists:
>>>[[] for n in range(size_n)]
[[], [], [], [], []]
Then, of course, the inner lists don't have to be empty, you can populate them with numbers:
>>>[[1, 2, 3] for n in range(size_n)]
[[1, 2, 3], [1, 2, 3], [1, 2, 3], [1, 2, 3], [1, 2, 3]]
Or you can use another range expression inside:
>>> [[range(1, 4)] for n in range(5)]
[[[1, 2, 3]], [[1, 2, 3]], [[1, 2, 3]], [[1, 2, 3]], [[1, 2, 3]]]
And you can use list comprehension on those inner lists just the same way:
>>> [[randint(1, 9) for m in range(1, 4)] for n in range(5)]
[[8, 4, 3], [6, 7, 2], [6, 8, 2], [9, 6, 1], [9, 1, 2]]
You can go deeper and deeper until you're bored or out of memory:
>>> [[[randint(1, 9) for n in range(1, 5)] for n in range(1, 4)] for n in range(5)]
[[[2, 9, 6, 1], [6, 7, 4, 5], [5, 9, 1, 7]], [[5, 2, 9, 3], [1, 8, 9, 7], [8, 4, 4, 8]], [[4, 4, 7, 9], [7, 1, 4, 2], [7, 8, 7, 3]], [[4, 4, 9, 9], [8, 8, 9, 5], [6, 1, 3, 9]], [[5, 9, 3, 2], [7, 5, 4, 7], [7, 7, 4, 3]]]
Nested list comprehension is usually the right solution, they are easy to read once you get used to the notation. Nested loops are OK too.
There's nothing wrong with looking for better ways to re-write your working code (I'd say, rather, it's wrong not to look), unless you develop an unhealthy obsession with it.
For Python 3 range(5) actually doesn't give [0, 1, 2, 3, 4] directly, but for this task it's effectively the same thing.
Saw a question on another site about a piece of Python code that was driving someone nuts. It was a fairly small, straightforward-looking piece of code, so I looked at it, figured out what it was trying to do, then ran it on my local system, and discovered why it was driving the original questioner nuts. Hoping that someone here can help me understand what's going on.
The code seems to be a straightforward "ask the user for three values (x,y,z) and a sum (n); iterate all values to find tuples that sum to n, and add those tuples to a list." solution. But what it outputs is, instead of all tuples that sum to n, a list of tuples the count of which is equal to the count of tuples that sum to n, but the contents of which are all "[x,y,z]". Trying to wrap my head around this, I changed the append call to an extend call (knowing that this would un-list the added tuples), to see if the behavior changed at all. I expected to get the same output, just as "x,y,z,x,y,z..." repeatedly, instead of "[x,y,z],[x,y,z]" repeatedly, because as I read and understand the Python documentation, that's the difference between append and extend on lists. What I got instead when I used extend was the correct values of the tuples that summed to n, just broken out of their tuple form by extend.
Here's the problem code:
my = []
x = 3
y = 5
z = 7
n = 11
part = [0,0,0]
for i in range(x+1):
part[0] = i
for j in range(y+1):
part[1] = j
for k in range(z+1):
part[2] = k
if sum(part) == n:
my.append(part)
print(my)
and the output:
[[3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7]]
And here's the extend output:
[0, 4, 7, 0, 5, 6, 1, 3, 7, 1, 4, 6, 1, 5, 5, 2, 2, 7, 2, 3, 6, 2, 4, 5, 2, 5, 4, 3, 1, 7, 3, 2, 6, 3, 3, 5, 3, 4, 4, 3, 5, 3]
And the extend code:
my = []
x = 3
y = 5
z = 7
n = 11
part = [0,0,0]
for i in range(x+1):
part[0] = i
for j in range(y+1):
part[1] = j
for k in range(z+1):
part[2] = k
if sum(part) == n:
my.extend(part)
print(my)
Any light that could be shed on this would be greatly appreciated. I've dug around for a while on Google and several Q&A sites, and the only things that I found regarding Python append/extend deltas are things that don't seem to have any relevance to this issue.
{edit: environment detail}
Also, ran this in both Python 2.7.10 and Python 3.4.3 (cygwin, under Windows 10 home) with the same results.
extend adds items from the parameter list to the list object making the call. More like, dump objects from one list to another without emptying the former.
append on the other hand, just appends; nothing more. Therefore, appending a list object to another list with an existing reference to the appended list could do some damage - as in this case. After the list has been appended, part still holds a reference to the list (since you're modifying in place), so you're essentially modifying and (re-)appending the same list object every time.
You can prevent this by either building a new list at the start of each parent iteration of the append case.
Or by simply appending a copy of the part list:
my.append(part[:])
my.append(list(part))
my.append(part.copy()) # Python 3 only
This will append a list that has no other existing reference outside its new parent list.
There are a couple of things going on - the difference between append and extend, and the mutability of a list.
Consider a simpler case:
In [320]: part=[0,0,0]
In [321]: alist=[]
In [322]: alist.append(part)
In [323]: alist
Out[323]: [[0, 0, 0]]
The append actually put a pointer to part in the list.
In [324]: alist.extend(part)
In [325]: alist
Out[325]: [[0, 0, 0], 0, 0, 0]
extend put the elements of part in the list, not part itself.
If we change an element in part, we can see the consequences of this difference:
In [326]: part[1]=1
In [327]: alist
Out[327]: [[0, 1, 0], 0, 0, 0]
The append part also changed, but the extended part did not.
That's why your append case consists of sublists, and the sublists all have the final value of part - because they all are part.
The extend puts the current values of part in the list. Not only aren't they sublists, but they don't change as part changes.
Here's a variation on that list pointer issue:
In [333]: alist = [part]*3
In [334]: alist
Out[334]: [[0, 1, 0], [0, 1, 0], [0, 1, 0]]
In [335]: alist[0][0]=2
In [336]: part
Out[336]: [2, 1, 0]
In [337]: alist
Out[337]: [[2, 1, 0], [2, 1, 0], [2, 1, 0]]
alist contains 3 pointers to part (not 3 copies). Change one of those sublists, and we change them all, including part.
I'm relatively new to programming, and I want to sort a 2-D array (lists as they're called in Python) by the value of all the items in each sub-array. For example:
pop = [[1,5,3],[1,1,1],[7,5,8],[2,5,4]]
The sum of the first element of pop would be 9, because 1 + 5 + 3 = 9. The sum of the second would be 3, because 1 + 1 + 1 = 3, and so on.
I want to rearrange this so the new order would be:
newPop = [pop[1], pop[0], pop[3], pop[2]]
How would I do this?
Note: I don't want to sort the elements each sub-array, but sort according to the sum of all the numbers in each sub-array.
You can use sorted():
>>> pop = [[1,5,3],[1,1,1],[7,5,8],[2,5,4]]
>>> newPop = sorted(pop, key=sum)
>>> newPop
[[1, 1, 1], [1, 5, 3], [2, 5, 4], [7, 5, 8]]
You can also sort in-place with pop.sort(key=sum). Unless you definitely want to preserve the original list, you should prefer in-pace sorting.
Try this:
sorted(pop, key=sum)
Explanation:
The sorted() procedure sorts an iterable (a list in this case) in ascending order
Optionally, a key parameter can be passed to determine what property of the elements in the list is going to be used for sorting
In this case, the property is the sum of each of the elements (which are sublists)
So essentially this is what's happening:
[[1,5,3], [1,1,1], [7,5,8], [2,5,4]] # original list
[sum([1,5,3]), sum([1,1,1]), sum([7,5,8]), sum([2,5,4])] # key=sum
[9, 3, 20, 11] # apply key
sorted([9, 3, 20, 11]) # sort
[3, 9, 11, 20] # sorted
[[1,1,1], [1,5,3], [2,5,4], [7,5,8]] # elements coresponding to keys
#arshajii beat me to the punch, and his answer is good. However, if you would prefer an in-place sort:
>>> pop = [[1,5,3],[1,1,1],[7,5,8],[2,5,4]]
>>> pop.sort(key=sum)
>>> pop
[[1, 1, 1], [1, 5, 3], [2, 5, 4], [7, 5, 8]]
I have to look up Python's sorting algorithm -- I think it's called Timsort, bit I'm pretty sure an in-place sort would be less memory intensive and about the same speed.
Edit: As per this answer, I would definitely recommend x.sort()
If you wanted to sort the lists in a less traditional way, you could write your own function (that takes one parameter.) At risk of starting a flame war, I would heavily advise against lambda.
For example, if you wanted the first number to be weighted more heavily than the second number more heavily than the third number, etc:
>>> def weightedSum(listToSum):
... ws = 0
... weight = len(listToSum)
... for i in listToSum:
... ws += i * weight
... weight -= 1
... return ws
...
>>> weightedSum([1, 2, 3])
10
>>> 1 * 3 + 2 * 2 + 3 * 1
10
>>> pop
[[1, 5, 3], [1, 1, 1], [7, 5, 8], [2, 5, 4]]
>>> pop.sort(key=weightedSum)
>>> pop
[[1, 1, 1], [1, 5, 3], [2, 5, 4], [7, 5, 8]]
>>> pop += [[1, 3, 8]]
>>> pop.sort(key=weightedSum)
>>> pop
[[1, 1, 1], [1, 5, 3], [1, 3, 8], [2, 5, 4], [7, 5, 8]]
I want to merge two arrays in python based on the first element in each column of each array.
For example,
A = ([[1, 2, 3],
[4, 5, 6],
[4, 6, 7],
[5, 7, 8],
[5, 9, 1]])
B = ([[1, .002],
[4, .005],
[5, .006]])
So that I get an array
C = ([[1, 2, 3, .002],
[4, 5, 6, .005],
[4, 6, 7, .005],
[5, 7, 8, .006],
[5, 9, 1, .006]])
For more clarity:
First column in A is 1, 4, 4, 5, 5 and
First column of B is 1, 4, 5
So that 1 in A matches up with 1 in B and gets .002
How would I do this in python? Any suggestions would be great.
Is it Ok to modify A in place?:
d = dict((x[0],x[1:]) for x in B)
Now d is a dictionary where the first column are keys and the subsequent columns are values.
for lst in A:
if lst[0] in d: #Is the first value something that we can extend?
lst.extend(d[lst[0]])
print A
To do it out of place (inspired by the answer by Ashwini):
d = dict((x[0],x[1:]) for x in B)
C = [lst + d.get(lst[0],[]) for lst in A]
However, with this approach, you need to have lists in both A and B. If you have some lists and some tuples it'll fail (although it could be worked around if you needed to), but it will complicate the code slightly.
with either of these answers, B can have an arbitrary number of columns
As a side note on style: I would write the lists as:
A = [[1, 2, 3],
[4, 5, 6],
[4, 6, 7],
[5, 7, 8],
[5, 9, 1]]
Where I've dropped the parenthesis ... They make it look too much like you're putting a list in a tuple. Python's automatic line continuation happens with parenthesis (), square brackets [] or braces {}.
(This answer assumes these are just regular lists. If they’re NumPy arrays, you have more options.)
It looks like you want to use B as a lookup table to find values to add to each row of A.
I would start by making a dictionary out of the data in B. As it happens, B is already in just the right form to be passed to the dict() builtin:
B_dict = dict(B)
Then you just need to build C row by row.
For each row in A, row[0] is the first element, so B_dict[row[0]] is the value you want to add to the end of the row. Therefore row + [B_dict[row[0]] is the row you want to add to C.
Here is a list comprehension that builds C from A and B_dict.
C = [row + [B_dict[row[0]]] for row in A]
You can convert B to a dictionary first, with the first element of each sublist as key and second one as value.
Then simply iterate over A and append the related value fetched from the dict.
In [114]: A = ([1, 2, 3],
[4, 5, 6],
[4, 6, 7],
[5, 7, 8],
[6, 9, 1])
In [115]: B = ([1, .002],
[4, .005],
[5, .006])
In [116]: [x + [dic[x[0]]] if x[0] in dic else [] for x in A]
Out[116]:
[[1, 2, 3, 0.002],
[4, 5, 6, 0.005],
[4, 6, 7, 0.005],
[5, 7, 8, 0.006],
[6, 9, 1]]
Here is a solution using itertools.product() that prevents having to create a dictionary for B:
In [1]: from itertools import product
In [2]: [lst_a + lst_b[1:] for (lst_a, lst_b) in product(A, B) if lst_a[0] == lst_b[0]]
Out[2]:
[[1, 2, 3, 0.002],
[4, 5, 6, 0.005],
[4, 6, 7, 0.005],
[5, 7, 8, 0.006],
[5, 9, 1, 0.006]]
The naive, simple way:
for alist in A:
for blist in B:
if blist[0] == alist[0]:
alist.extend(blist[1:])
# alist.append(blist[1]) if B will only ever contain 2-tuples.
break # Remove this if you want to append more than one.
The downside here is that it's O(N^2) complexity. For most small data sets, that should be ok. If you're looking for something more comprehensive, you'll probably want to look at #mgilson's answer. Some comparison:
His response converts everything in B to a dict and performs list slicing on each element. If you have a lot of values in B, that could be expensive. This uses the existing lists (you're only looking at the first value, anyway).
Because he's using dicts, he gets O(1) lookup times (his answer also assumes that you're never going to append multiple values to the end of the values in A). That means overall, his algorithm will achieve O(N). You'll need to weigh whether the overhead of creating a dict is going to outweight the iteration of the values in B.