I'm trying to optimize my critical path. source_and_dest is a list of coordinates, state is a translation from one coordinate to another, and distance_squared is a two-dimensional list describing a relationship between two coordinates. My current approach, which is correct, is the following:
e = 0
for source, dest in self.source_and_dest:
e += self.distance_squared[self.state[source]][self.state[dest]]
However, I run these lines millions of times and I need to improve my runtime (even if only slightly). My cumulative runtime for this is around 6.2s (with other benchmarks, this will grow to be much larger).
My understanding is that using functools.reduce() could improve runtime. My attempt, however, gave me much worse runtime (2x worse).
def do_sum(self,x1, x2):
if type(x1) == int:
return x1 + self.distance_squared[self.state[x2[0]]][ self.state[x2[1]]]
else:
return self.distance_squared[self.state[x1[0]]][ self.state[x1[1]] ] + self.distance_squared[self.state[x2[0]]][ self.state[x2[1]] ]
...
e=functools.reduce(self.do_sum, self.source_and_dest)
I imagine there may be a way to use reduce() more effectively here, rather than having to check for the x1's type and without so many nested array accesses.
Here is some runnable code. Both approaches are included:
import functools
import cProfile
def torus_min_distance_xy( sourcexy, sinkxy, Nx,Ny ):
source_x = sourcexy % Nx
source_y = sourcexy // Nx
sink_x = sinkxy % Nx
sink_y = sinkxy // Nx
if( source_x <= sink_x and source_y <= sink_y ):
return (( sink_x - source_x ), ( sink_y - source_y ))
elif( source_x > sink_x and source_y <= sink_y ):
return (( Nx + sink_x - source_x ) , ( sink_y - source_y ))
elif( source_x <= sink_x and source_y > sink_y ):
return (( sink_x - source_x ) , ( Ny + sink_y - source_y ))
else:
return (( Nx - source_x + sink_x ) , ( Ny - source_y + sink_y ))
class Energy:
def __init__( self ):
self.Nx = 8
self.Ny = 9
self.memoized_source_pe_and_dest_partitions = [(56, 5), (12, 33), (68, 14), (53, 15), (28, 33), (7, 24), (57, 5), (14, 22), (22, 28), (32, 19), (1, 28), (66, 17), (58, 0), (69, 14), (55, 7), (63, 12), (52, 15), (17, 22), (62, 12), (59, 0), (54, 7), (8, 29), (65, 1), (33, 29), (0, 32), (31, 70), (67, 17), (19, 24), (61, 8), (60, 8), (64, 1), (29, 32), (15, 31), (5, 19), (24, 31), (38, 16), (3, 26), (50, 9), (35, 4), (20, 26), (10, 23), (39, 16), (9, 18), (18, 20), (21, 25), (11, 20), (48, 2), (40, 6), (51, 9), (37, 10), (45, 3), (34, 4), (2, 18), (44, 3), (41, 6), (36, 10), (13, 30), (47, 11), (26, 30), (6, 21), (27, 71), (49, 2), (25, 23), (43, 13), (42, 13), (46, 11), (30, 21), (4, 27), (16, 25), (23, 27)]
self.memoized_distance_squared = [[0 for a in range(self.Nx*self.Ny)] for b in range(self.Nx*self.Ny)]
for source in range(self.Nx*self.Ny):
for dest in range(self.Nx*self.Ny):
tmp = torus_min_distance_xy(source,dest,self.Nx,self.Ny)
self.memoized_distance_squared[source][dest] = (tmp[0]+1)*(tmp[1]+1)
self.state = [59, 1, 2, 44, 4, 5, 6, 7, 28, 18, 37, 21, 62, 13, 14, 15, 38, 66, 51, 61, 46, 47, 22, 23, 69, 50, 45, 39, 60, 63, 30, 31, 67, 68, 34, 35, 36, 10, 27, 16, 40, 41, 42, 43, 26, 3, 20, 11, 48, 49, 25, 9, 52, 53, 54, 55, 56, 57, 58, 0, 33, 8, 29, 12, 64, 65, 32, 17, 24, 19, 70, 71]
def do_sum(self, x1, x2):
if type(x1) == int:
return x1 + self.memoized_distance_squared[self.state[x2[0]]][ self.state[x2[1]]]
else:
return self.memoized_distance_squared[self.state[x1[0]]][ self.state[x1[1]] ] + self.memoized_distance_squared[self.state[x2[0]]][ self.state[x2[1]] ]
def loop( self, iterations ):
for i in range( iterations ):
# comment out one approach at a time
# first approach
e0 = 0
for source_partition, dest_partition in self.memoized_source_pe_and_dest_partitions:
e0 += self.memoized_distance_squared[self.state[source_partition]][ self.state[dest_partition] ]
# second approach
# e1 = functools.reduce( self.do_sum, self.memoized_source_pe_and_dest_partitions )
energy = Energy()
cProfile.runctx('energy.loop(100000)',globals(), locals())
Related
As the title illustrated, I want to generate 100 pairs of random integers, such as (1, 0), (1, 31), (5, 7), (3, 19) and so on. And the range of value is from 0 to 31. At the same time, I hope each pair appears only once within the 100 pairs, i.e., the value of each pair is different from others.
So How can I achieve it in Python?
Supplement:
To be precise, what I want is a 2D array, and its shape is (100,2). Each row is required to be unique.
You can use random.sample:
import random
pool = range(32)
random.sample(pool, 2)
# [7, 28]
random.sample(pool, 2)
# [15, 3]
import numpy as np
pool_ = [ (i,j) for i in range(32) for j in range(32) ]
# List of all pairs
pool = np.zeros( len(pool_), dtype = tuple ) # Create an np.array of tuples
pool[:] = pool_ # fill it with pool_
selected = np.random.choice( pool, replace = False, size =100 )
# select 100 random choices from the pool, without replacement
print( selected )
# [(0, 22) (4, 30) (2, 25) (4, 19) (6, 6) (17, 22) (18, 14) (12, 27) (30, 6)
# (22, 18) (13, 5) (23, 22) (27, 17) (17, 26) (26, 22) (7, 15) (15, 27)
# (4, 31) (15, 1) (28, 22) (25, 16) (25, 15) (7, 12) (7, 21) (26, 14)
# (9, 9) (8, 0) (26, 27) (14, 14) (22, 0) (4, 18) (12, 3) (25, 9) (22, 31)
# (11, 6) (23, 7) (18, 19) (19, 25) (23, 19) (25, 5) (5, 19) (3, 24)
# (30, 0) (18, 10) (20, 4) (24, 11) (13, 28) (10, 5) (6, 7) (11, 7)
# (25, 24) (23, 18) (15, 10) (14, 7) (11, 11) (9, 23) (13, 8) (3, 28)
# (28, 3) (21, 3) (24, 31) (29, 27) (24, 28) (17, 6) (30, 19) (25, 28)
# (12, 17) (13, 15) (3, 11) (14, 1) (12, 6) (17, 17) (23, 2) (24, 18)
# (25, 11) (3, 26) (6, 2) (0, 28) (5, 12) (4, 1) (23, 17) (29, 23) (22, 17)
# (24, 15) (2, 5) (28, 11) (19, 27) (9, 20) (1, 11) (30, 5) (30, 21)
# (30, 28) (18, 31) (5, 27) (30, 11) (16, 0) (24, 16) (12, 30) (25, 25)
# (16, 22)]
I haven't thoroughly tested it but with replace = False in random.choice each selection should be unique.
To return a 2D array
import numpy as np
pool_ = [ (i,j) for i in range(32) for j in range(32) ]
# List of all pairs
pool_ix = np.arange( len(pool_) ) # index by row
pool = np.array(pool_) # 2D pool
selected_ix = np.random.choice( pool_ix, replace = False, size =100 )
pool[selected_ix, : ] # Select all of each selected row.
# array([[12, 19],
# [ 6, 23],
# [ 2, 3],
# [ 5, 20],
# :::
# [20, 3],
# [24, 20],
# [ 1, 28]])
Given two lists x, y such that they both have been initialized as shown below:
x = [(0, 3), (5, 8), (16, 19), (21, 24), (28, 30), (40, 42), (46, 47), (50, 54), (58, 63), (69, 71)]
y = [(9, 10), (26, 27), (29, 31), (35, 36), (41, 43), (48, 49), (66, 67), (70, 72), (77, 78), (85, 86)]
I want to form a new list of tuples where each tuple has contiguous tuples from x and an overlapping tuple from y.
For the example above, the output would be:
[((5, 8) (9, 10) (16, 19)), ((21, 24) (26, 27) (28, 30)), ((28, 30) (29, 31) (40, 42)), ((28, 30) (35, 36) (40, 42)), ((40, 42) (41, 43) (46, 47)), ((46, 47) (48, 49) (50, 54)),((58, 63) (66, 67) (69, 71))]
My code:
lst = []
for i in range(len(x)):
if i+1 < len(x):
context = x[i],x[i+1]
for j in y:
if j[0] >= context[0][0] and j[0] <= context[1][0]:
lst.append((context[0],j,context[1]))
I need better and efficient ways to write this code.
You can use two variables to keep track of indices in x and y list. Using the conditions specified in the problem, these indices can be incremented whenever the given condition has been satisfied.
At every iteration, the algorithm checks if x[i][0] < y[j][0] and x[i+1][1] > y[j][1] ( The upper and lower bound provided by the contigous tuples in x. If this condition is true, we increment j (y-index) so that we can check if the next element lies in the given range. Else, we increment i (x-index) and repeat the process.
x = [(0, 3), (5, 8), (16, 19), (21, 24), (28, 30), (40, 42), (46, 47), (50, 54), (58, 63), (69, 71)]
y = [(9, 10), (26, 27), (29, 31), (35, 36), (41, 43), (48, 49), (66, 67), (70, 72), (77, 78), (85, 86)]
i = 0
j = 0
result = list()
while i < len(x) - 1 and j < len(y):
if y[j][0] > x[i][0] and y[j][1] < x[i + 1][1]:
result.append((x[i], y[j], x[i + 1]))
j += 1
else:
i += 1
print(result)
Output -
[((5, 8), (9, 10), (16, 19)),
((21, 24), (26, 27), (28, 30)),
((28, 30), (29, 31), (40, 42)),
((28, 30), (35, 36), (40, 42)),
((40, 42), (41, 43), (46, 47)),
((46, 47), (48, 49), (50, 54)),
((58, 63), (66, 67), (69, 71))]
You can use Python Sorting
from operator import itemgetter, attrgetter
output = sorted((x + y), key=itemgetter(0))
I have two ranges:
range_1 (0,10)
range_2 (11, 40)
I want to create a list of tuples from the two ranges above (range_1 and range_2) if the sum of any of the two elements in the two ranges is an even number.
Thus 0 from range_1 and 12 from range_2 = 12 which is even, the same with 1 from range_1 and 13 from range_2 = 14 which is even.
However I don't want to go through all the elements in range_2. Only 5 successful attempts are needed, then immediately I have to go back to the second iteration in range_1.
Thus for the first iteration:
(0, 12, 12), (0, 14, 14), (0, 16, 16), (0, 18, 18), (0, 20, 20)
then we go to the second iteration:
(1, 11, 12), (1, 13, 14), (1, 15, 16), (1, 17, 18), (1, 19, 20)
and so on till 9 in range_1:
(9, 11, 20), (9, 13, 22), (9, 15, 24), (9, 17, 26), (9, 19, 28)
Here is my code, which goes through all the elements, which is obviously wrong, because it goes through all the elements in range_2!
list_1 = []
for i in range(10):
for j in range(11,40):
if (i+j)%2 == 0:
list_1.append((i, j, (i+j)))
Just store a counter so that if you reach five then you break out of your nested for-loop:
list_1 = []
for i in range(10):
counter = 0
for j in range(11,40):
if (i+j)%2 == 0:
list_1.append((i, j, (i+j)))
counter += 1
if counter == 5:
break
which gives list_1 as:
[(0, 12, 12), (0, 14, 14), (0, 16, 16), (0, 18, 18), (0, 20, 20),
(1, 11, 12), (1, 13, 14), (1, 15, 16), (1, 17, 18), (1, 19, 20),
(2, 12, 14), (2, 14, 16), (2, 16, 18), (2, 18, 20), (2, 20, 22),
(3, 11, 14), (3, 13, 16), (3, 15, 18), (3, 17, 20), (3, 19, 22),
(4, 12, 16), (4, 14, 18), (4, 16, 20), (4, 18, 22), (4, 20, 24),
(5, 11, 16), (5, 13, 18), (5, 15, 20), (5, 17, 22), (5, 19, 24),
(6, 12, 18), (6, 14, 20), (6, 16, 22), (6, 18, 24), (6, 20, 26),
(7, 11, 18), (7, 13, 20), (7, 15, 22), (7, 17, 24), (7, 19, 26),
(8, 12, 20), (8, 14, 22), (8, 16, 24), (8, 18, 26), (8, 20, 28),
(9, 11, 20), (9, 13, 22), (9, 15, 24), (9, 17, 26), (9, 19, 28)]
It should be noted that this is not the most efficient way to go about creating your data structure. Clearly only ever other j-value generated in the inner for-loop will be used, which is wasteful.
Therefore you could specify a step for the j for-loop of 2 so that only other j-value is considered. However, you must be careful with the starting value now. If you were to always start at 11 and step in 2s then you would only get odd j-values and these could never combine with the current i-value to give an even number if i was even. Therefore you would have to change the j for-loop to start at 12 if i is even and 11 if i is odd.
As others have commented, you can simplify the problem quite a bit by constructing ranges that accomodate what you're trying to do. Here it is as a nested comprehension:
[(i, j, i+j) for i in range(0, 10) for j in range((11 if i % 2 else 12), 21, 2)]
If you're smart about creating your range, you can simplify the algorithm. Use a step to make the range skip every other j, adjust the start based on whether i is even, and set the end to 21 since it will only ever get that high.
list_1 = []
for i in range(10):
start = 12 if i % 2 == 0 else 11
for j in range(start, 21, 2):
list_1.append((i, j, i+j))
print(list_1[-5:]) # For testing
First two lines of output:
[(0, 12, 12), (0, 14, 14), (0, 16, 16), (0, 18, 18), (0, 20, 20)]
[(1, 11, 12), (1, 13, 14), (1, 15, 16), (1, 17, 18), (1, 19, 20)]
I'm making a graph from an adj matrix, here is my code, I try to first make the graph, then put a weight with that graph as seen here
for element in elements:
senMatrix[int(element.matrix_row)-1,int(element.matrix_column)-1]=1
G=nx.from_numpy_matrix(senMatrix)
for element in elements:
G[int(element.matrix_row)-1][int(element.matrix_column)-1]['weight']=int(element.value)
print str(G.nodes())
print str(G.edges())
avgClusterNumber=nx.average_clustering(G,weight='weight')
clusterList=nx.clustering(G,weight='weight')
graphDiameter=nx.diameter(G)
The first two functions run without a problem, the last diameter function however throws an issue with infinite path length when which makes me think there are no edges or nodes or something.
Error seen here
networkx.exception.NetworkXError: Graph not connected: infinite path length
When I print them out I get this
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105]
[(0, 1), (0, 3), (0, 5), (0, 6), (0, 7), (0, 8), (0, 9), (0, 11), (0, 12), (0, 13), (0, 14), (0, 17), (0, 18), (0, 19), (0, 20), (0, 23), (0, 25), (0, 26), (0, 27), (0, 28), (0, 32), (0, 33), (0, 35), (0, 36), (0, 37), (0, 38), (0, 39), (0, 43), (0, 46), (0, 47), (0, 48), (0, 50), (0, 52), (0, 55), (0, 56), (0, 57), (0, 59), (0, 60), (0, 61), (0, 63), (0, 65), (0, 66), (0, 69), (0, 70), (0, 71), (0, 72), (0, 77), (0, 79), (0, 80), (0, 82), (0, 85), (0, 86), (0, 88), (0, 90), (0, 92), (0, 96), (0, 97), (0, 98), (0, 99), (0, 100), (0, 101), (0, 104), (0, 105), (1, 2), (1, 3), (1, 5), (1, 6), (1, 7), (1, 8), (1, 9), (1, 10), (1, 11), (1, 12), (1, 13), (1, 14), (1, 15), (1, 16), (1, 17), (1, 18), (1, 19), (1, 20), (1, 21), (1, 22), (1, 23), (1, 24), (1, 25), (1, 26), (1, 27), (1, 28), (1, 29), (1, 30), (1, 31), (1, 33), (1, 34), (1, 35), (1, 36), (1, 37), (1, 38), (1, 39), (1, 40), (1, 41), (1, 42), (1, 44), (1, 46), (1, 47), (1, 49), (1, 51), (1, 52), (1, 53), (1, 54), (1, 55), (1, 56), (1, 57), (1, 58), (1, 59), (1, 60), (1, 61), (1, 62), (1, 63), (1, 64), (1, 65), (1, 66), (1, 67), (1, 68), (1, 69), (1, 70), (1, 71), (1, 72), (1, 74), (1, 75), (1, 76), (1, 77), (1, 78), (1, 79), (1, 80), (1, 81), (1, 83), (1, 85), (1, 86), (1, 87), (1, 88), (1, 90), (1, 91), (1, 95), (1, 96), (1, 97), (1, 98), (1, 99), (1, 101), (1, 102), (1, 103), (1, 104), (1, 105), (2, 66), (2, 6), (2, 7), (2, 40), (2, 78), (2, 79), (2, 18), (2, 83), (2, 87), (2, 56), (2, 25), (2, 27), (2, 92), (2, 31), (3, 6), (3, 7), (3, 8), (3, 9), (3, 11), (3, 12), (3, 15), (3, 16), (3, 21), (3, 22), (3, 25), (3, 26), (3, 27), (3, 28), (3, 29), (3, 30), (3, 32), (3, 40), (3, 41), (3, 42), (3, 47), (3, 51), (3, 52), (3, 53), (3, 55), (3, 59), (3, 60), (3, 62), (3, 63), (3, 66), (3, 67), (3, 68), (3, 69), (3, 70),..
which prints out the nodes and then edges, obviously I've limited it to show the idea.
So I suppose my question is im not sure why this is an issue. Does anyhow know how to fix this?
I'm obviously not thinking about this in the correct light.
try
nx.is_connected(G)
if it returns False then you can separate the graph by components and find the diameter for each component, using
connected_components(G)
This question already has answers here:
How do I split a list into equally-sized chunks?
(66 answers)
Closed 9 years ago.
I have a long list I would like to break into shorter lists. I am using a list comprehension but it seems a bit long and inelegant. Is there a better way?
# z is a list
z = range(99)
## zz should slice z into short lists with three members
## using list comprehension I get this
zz = [ z[i : i+3] for i,x in enumerate(z) if i%3 == 0 ]
# seems a bit verbose. is there a cleaner way?
From itertools (it's one of the common recipes):
import itertools
def grouper(iterable, n, fillvalue=None):
args = [iter(iterable)] * n
return itertools.izip_longest(fillvalue=fillvalue, *args)
Example:
>>> list(grouper(range(100), 3))
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, 10, 11), (12, 13, 14), (15, 16, 17), (18, 19, 20), (21, 22, 23), (24, 25, 26), (27, 28, 29), (30, 31, 32), (33, 34, 35), (36, 37, 38), (39, 40, 41), (42, 43, 44), (45, 46, 47), (48, 49, 50), (51, 52, 53), (54, 55, 56), (57, 58, 59), (60, 61, 62), (63, 64, 65), (66, 67, 68), (69, 70, 71), (72, 73, 74), (75, 76, 77), (78, 79, 80), (81, 82, 83), (84, 85, 86), (87, 88, 89), (90, 91, 92), (93, 94, 95), (96, 97, 98), (99, None, None)]