Dask distributed calculation performance drop in loop - python

I'm new to Dask and still trying to figure out how to get this running smoothly. I've been experimenting with future API and I get some surprising results.
I have a simple while loop in my code that call the function cpi5. When I %timeit the execution of the fucntion I get :
%timeit min(cpci5(x,N,M,n,lciw,uciw))
6.74 s ± 178 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Now I've run the same function but using a distributed sheduler in dask and I get this :
from dask.distributed import Client
client= Client()
%timeit B.append(client.submit(cpci5,x,N,M,n,lciw,uciw)); bb = np.array(client.gather(B))
29 ms ± 6.01 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
So far so good, the expected improvement is there but when I time the execution of the loop itself where the function is called, I barely get any difference and they both run in approximately 19s. (I've profile the initial loop and more than 90% of the computation time is due to the function so the improvement should be there.)
The results are coherent and identical.
What could cause such a difference ?
You'll find below the relevant piece of code.
PS : I've already gone as far as i could in code optimization but it's not enough in my case.
N = 392
n = 326
x = np.arange(0,n+1)
if np.floor(n/2) == n/2:
xvalue = int(n/2 +1)
else :
xvalue = int((n+1)/2)
aa = np.arange(lciw[xvalue-1],np.floor(N/2)).astype(int)
lciw :
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26,
27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 40, 41,
42, 43, 44, 45, 46, 47, 49, 50, 51, 52, 53, 54, 55,
57, 58, 59, 60, 61, 62, 64, 65, 66, 67, 68, 69, 70,
72, 73, 74, 75, 76, 77, 79, 80, 81, 82, 83, 84, 86,
87, 88, 89, 90, 91, 93, 94, 95, 96, 97, 98, 100, 101,
102, 103, 104, 105, 107, 108, 109, 110, 111, 112, 114, 115, 116,
117, 118, 120, 121, 122, 123, 124, 125, 127, 128, 129, 130, 131,
132, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 146, 147,
148, 149, 150, 151, 153, 154, 155, 156, 157, 159, 160, 161, 162,
163, 165, 166, 167, 168, 169, 170, 172, 173, 174, 175, 176, 178,
179, 180, 181, 182, 184, 185, 186, 189, 188, 190, 191, 192, 193,
194, 196, 197, 198, 199, 200, 202, 203, 204, 205, 206, 208, 209,
210, 211, 212, 214, 215, 216, 217, 218, 220, 221, 222, 223, 225,
226, 227, 228, 229, 231, 232, 233, 234, 235, 237, 238, 239, 240,
241, 243, 244, 245, 246, 248, 249, 250, 251, 252, 254, 255, 256,
257, 258, 260, 261, 262, 263, 265, 266, 267, 268, 269, 271, 272,
273, 274, 276, 277, 278, 279, 280, 282, 283, 284, 285, 287, 288,
289, 290, 292, 293, 294, 295, 296, 298, 299, 300, 301, 303, 304,
305, 306, 308, 309, 310, 311, 313, 314, 315, 316, 317, 319, 320,
321, 322, 324, 325, 326, 327, 329, 330, 331, 332, 334, 335, 336,
338, 339, 340, 341, 343, 344, 345, 346, 348, 349, 350, 351, 353,
354, 355, 357, 358, 359, 360, 362, 363, 364, 366, 367, 368, 369,
371, 372, 373, 375, 376, 377, 379, 380, 382, 383, 384, 386, 387,
389, 390])
uciw :
array([2, 3, 5, 6, 8, 9, 10, 12, 13, 15, 16, 17, 19,
20, 21, 23, 24, 25, 26, 28, 29, 30, 32, 33, 34, 35,
37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52,
53, 54, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68,
70, 71, 72, 73, 75, 76, 77, 78, 79, 81, 82, 83, 84,
86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 98, 99, 100,
102, 103, 104, 105, 107, 108, 109, 110, 112, 113, 114, 115, 116,
118, 119, 120, 121, 123, 124, 125, 126, 127, 129, 130, 131, 132,
134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 146, 147, 148,
149, 151, 152, 153, 154, 155, 157, 158, 159, 160, 161, 163, 164,
165, 166, 167, 169, 170, 171, 172, 174, 175, 176, 177, 178, 180,
181, 182, 183, 184, 186, 187, 188, 189, 190, 192, 193, 194, 195,
196, 198, 199, 200, 201, 202, 204, 203, 206, 207, 208, 210, 211,
212, 213, 214, 216, 217, 218, 219, 220, 222, 223, 224, 225, 226,
227, 229, 230, 231, 232, 233, 235, 236, 237, 238, 239, 241, 242,
243, 244, 245, 246, 248, 249, 250, 251, 252, 254, 255, 256, 257,
258, 260, 261, 262, 263, 264, 265, 267, 268, 269, 270, 271, 272,
274, 275, 276, 277, 278, 280, 281, 282, 283, 284, 285, 287, 288,
289, 290, 291, 292, 294, 295, 296, 297, 298, 299, 301, 302, 303,
304, 305, 306, 308, 309, 310, 311, 312, 313, 315, 316, 317, 318,
319, 320, 322, 323, 324, 325, 326, 327, 328, 330, 331, 332, 333,
334, 335, 337, 338, 339, 340, 341, 342, 343, 345, 346, 347, 348,
349, 350, 351, 352, 354, 355, 356, 357, 358, 359, 360, 361, 363,
364, 365, 366, 367, 368, 369, 370, 371, 373, 374, 375, 376, 377,
378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390,
391, 392])
def cpci5(x,N,M,n,lciw,uciw):
f = np.vectorize(hypergeom.pmf)
idd = np.vectorize(ind)
X, m = np.meshgrid(x,M)
kk = idd(m,lciw,uciw) * f(X, N, m, n) #idd just implement a test lciw<= m <=uciw
return min(pd.Series(kk.sum(axis=1)))
M = np.arange(0,N+1) # Initial implementation of the function
ii = 0
while (ii <len(aa)+1):
lciw[xvalue-1] = aa[ii]
uciw[xvalue-1] = N - aa[ii]
bb = min(cpci5(x,N,M,n,lciw,uciw))
if bb >= 1-alpha:
ii1 = ii
ii += 1
else :
ii = len(aa)+1
lciw[xvalue-1] = aa[ii1]
uciw[xvalue-1] = N - lciw[xvalue-1]
M = np.arange(0,N+1) # Distributed version
ii = 0
B = []
while (ii <len(aa)):
lciw[xvalue-1] = aa[ii]
uciw[xvalue-1] = N - aa[ii]
B.append(client.submit(cpci5,x,N,M,n,lciw,uciw))
ii += 1
bb = np.array(client.gather(B))
ii1 = len(bb[bb>1-alpha])-1
lciw[xvalue-1] = aa[ii1]
uciw[xvalue-1] = N - lciw[xvalue-1]

Related

How to save printed console output to pandas dataframe in python?

I have a printed output:
{-1: [2, 10, 11, 13, 16, 19, 24, 28, 30, 32, 34, 35, 36, 40, 42, 49, 54, 56, 59, 64, 66, 78, 94, 99, 101, 102, 103, 106, 107, 109, 110, 114, 117, 123, 126, 127, 129, 131, 132, 133, 136, 144, 146, 147, 150, 155, 156, 164, 166, 177, 179, 181, 182, 188, 190, 192, 194, 201, 202, 204, 209, 214, 217, 220, 221, 225, 231, 232, 234, 235, 236, 240, 244, 246, 248, 253, 254, 257, 259, 260, 261, 262, 263, 264, 265, 266, 268, 271, 275, 277, 279, 280, 281, 285, 286, 287, 288, 297, 302, 309, ...], 0: [3, 6, 8, 25, 27, 33, 38, 57, 62, 63, 67, 69, 70, 72, 74, 83, 89, 91, 92, 98, 111, 112, 122, 124, 135, 158, 175, 187, 197, 198, 199, 200, 205, 206, 207, 215, 216, 242, 243, 258, 267, 272, 283, 299, 300, 303, 305, 306, 307, 310, 311, 312, 313, 314, 315, 316, 319, 326, 329, 348, 353, 355, 376, 377, 378, 380, 385, 386, 387, 389, 399, 402, 406, 418, 424, 425, 426, 427, 431, 432, 433, 434, 435, 447, 486, 487, 503, 511, 512, 514, 515, 524, 525, 535, 536, 539, 547, 549, 550, 554, ...], 1: [0, 5, 21, 44, 46, 48, 51, 82, 115, 118, 274, 293, 330, 331, 332, 361, 401, 413, 507, 520, 522, 523, 558, 560, 643, 650, 681, 700, 734, 747, 753, 782, 784, 836, 839, 893, 905, 934, 951, 976, 999, 1037, 1048, 1052, 1053, 1082, 1109, 1113, 1115, 1121, 1139, 1146, 1219, 1221, 1264, 1355, 1382, 1392, 1432, 1467, 1485, 1490, 1497, 1513, 1526, 1565, 1682, 1728, 1737, 1738, 1806, 1815, 1824, 1828, 1844, 1845, 1885, 1959, 2014, 2017, 2029, 2052, 2072, 2153, 2157, 2168, 2193, 2199, 2214, 2228, 2232, 2240, 2243, 2264, 2300, 2317, 2353, 2376, 2402, 2405, ...], 2: [15, 39, 60, 61, 149, 157, 222, 250, 289, 320, 448, 538, 630, 658, 662, 665, 709, 759, 810, 837, 897, 901, 917, 924, 925, 945, 946, 954, 959, 1049, 1050, 1090, 1131, 1140, 1154, 1172, 1251, 1300, 1313, 1328, 1387, 1393, 1431, 1440, 1448, 1475, 1507, 1535, 1591, 1597, 1603, 1615, 1636, 1705, 1725, 1736, 1771, 1777, 1791, 1796, 1855, 1867, 1903, 1918, 1928, 1930, 1942, 1943, 1989, 2021, 2039, 2095, 2119, 2169, 2195, 2309, 2337, 2418, 2426, 2429, 2522, 2582, 2598, 2678, 2679, 2682], 3: [50, 113, 160, 213, 224, 229, 238, 239, 352, 400, 409, 506, 545, 570, 701, 703, 712, 716, 830, 838, 858, 921, 1008, 1078, 1124, 1130, 1194, 1214, 1305, 1308, 1311, 1360, 1421, 1441, 1473, 1476, 1532, 1533, 1548, 1580, 1616, 1622, 1649, 1679, 1735, 1883, 1897, 1920, 1985, 2015, 2084, 2091, 2097, 2118, 2152, 2181, 2212, 2223, 2237, 2249, 2310, 2313, 2347, 2369, 2381, 2390, 2470, 2496, 2511, 2514, 2529, 2549, 2569, 2601, 2626, 2666, 2688],
Is it possible i can put this to dataframe
, suppose to column: For example:
Number
Value
-1
[2, 10, 11, 13, 16, 19, 24, 28, 30, 32, 34, 35, 36, 40, 42, 49, 54, 56, 59, 64, 66, 78, 94, 99, 101, 102, 103, 106, 107, 109, 110, 114, 117, 123, 126, 127, 129, 131, 132, 133, 136, 144, 146, 147, 150, 155, 156, 164, 166, 177, 179, 181, 182, 188, 190, 192, 194, 201, 202, 204, 209, 214, 217, 220, 221, 225, 231, 232, 234, 235, 236, 240, 244, 246, 248, 253, 254, 257, 259, 260, 261, 262, 263, 264, 265, 266, 268, 271, 275, 277, 279, 280, 281, 285, 286, 287, 288, 297, 302, 309, ...]
0
[3, 6, 8, 25, 27, 33, 38, 57, 62, 63, 67, 69, 70, 72, 74, 83, 89, 91, 92, 98, 111, 112, 122, 124, 135, 158, 175, 187, 197, 198, 199, 200, 205, 206, 207, 215, 216, 242, 243, 258, 267, 272, 283, 299, 300, 303, 305, 306, 307, 310, 311, 312, 313, 314, 315, 316, 319, 326, 329, 348, 353, 355, 376, 377, 378, 380, 385, 386, 387, 389, 399, 402, 406, 418, 424, 425, 426, 427, 431, 432, 433, 434, 435, 447, 486, 487, 503, 511, 512, 514, 515, 524, 525, 535, 536, 539, 547, 549, 550, 554, ...],

Try:
dct = {
-1: [2, 10, 11],
0: [3, 6, 27, 33],
1: [0, 5, 21],
2: [15],
3: [50, 113, 160, 213, 224],
}
df = pd.DataFrame({"Number": dct.keys(), "Value": dct.values()})
print(df)
Prints:
Number Value
0 -1 [2, 10, 11]
1 0 [3, 6, 27, 33]
2 1 [0, 5, 21]
3 2 [15]
4 3 [50, 113, 160, 213, 224]

df = pd.DataFrame()
df["Value"] = list(d.values())
df.index = d.keys()
# OR
df = pd.DataFrame.from_dict({k: [v] for k, v in d.items()},
orient="index",
columns=["Value"])
print(df)
# Value
# -1 [2, 10, 11, 13, 16, 19, 24, 28, 30, 32, 34, 35...
# 0 [3, 6, 8, 25, 27, 33, 38, 57, 62, 63, 67, 69, ...
# 1 [0, 5, 21, 44, 46, 48, 51, 82, 115, 118, 274, ...
# 2 [15, 39, 60, 61, 149, 157, 222, 250, 289, 320,...
# 3 [50, 113, 160, 213, 224, 229, 238, 239, 352, 4...

extract index of element in list python

I have a list named 'partition'
[array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]), array([214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226,
227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239,
240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250]), array([251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263,
264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276]), array([277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289,
290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302,
303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314]), array([30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
64, 65, 66, 67, 68, 69, 70]), array([ 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96,
97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109,
110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121]), array([122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134,
135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147,
148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160,
161, 162, 163, 164]), array([165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203,
204, 205, 206, 207, 208, 209, 210, 211, 212, 213])]
And i try to extract the index of an element (1 for example) with this code:
a= partition.index(1)
But i get this error :
ValueError Traceback (most recent call last)
/tmp/ipykernel_582922/1287398992.py in <module>
----> 1 a= partition.index(1)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Can you help me please to fix this error?
Kind regards

This should solve the problem:
import numpy as np
from numpy import array
partition = array([array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]), array([214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226,
227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239,
240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250]), array([251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263,
264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276]), array([277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289,
290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302,
303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314]), array([30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
64, 65, 66, 67, 68, 69, 70]), array([ 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96,
97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109,
110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121]), array([122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134,
135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147,
148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160,
161, 162, 163, 164]), array([165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203,
204, 205, 206, 207, 208, 209, 210, 211, 212, 213])], dtype="object")
result = [(i, np.where(j == 1)[0]) for i, j in enumerate(partition) if np.any(np.where(j == 1))]
print(result)
The result will be in the format of (This is an example):
[(*the index of the array*, *an array with all indexes of the elements that are 1*), (*the index of the array*, *an array with all indexes of the elements that are 1*)]
I'm not sure if this is the best way to do it though.
Or if you have a list of numpy arrays you could do something like this:
from numpy import array
partition = [array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]), array([214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226,
227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239,
240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250]), array([251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263,
264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276]), array([277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289,
290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302,
303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314]), array([30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
64, 65, 66, 67, 68, 69, 70]), array([ 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96,
97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109,
110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121]), array([122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134,
135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147,
148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160,
161, 162, 163, 164]), array([165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177,
178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203,
204, 205, 206, 207, 208, 209, 210, 211, 212, 213])]
result = []
for i, j in enumerate(partition):
for l in j:
if l == 1:
result.append((i, l))
print(result)

How to deal with large array in numpy/python when memory becomes issue

I have an array all with dimensions (19494500, 376) I need to arrange these 376 columns in a particular sequence I have generated,
l
array([ 0, 94, 188, 282, 1, 95, 189, 283, 2, 96, 190, 284, 3,
97, 191, 285, 4, 98, 192, 286, 5, 99, 193, 287, 6, 100,
194, 288, 7, 101, 195, 289, 8, 102, 196, 290, 9, 103, 197,
291, 10, 104, 198, 292, 11, 105, 199, 293, 12, 106, 200, 294,
13, 107, 201, 295, 14, 108, 202, 296, 15, 109, 203, 297, 16,
110, 204, 298, 17, 111, 205, 299, 18, 112, 206, 300, 19, 113,
207, 301, 20, 114, 208, 302, 21, 115, 209, 303, 22, 116, 210,
304, 23, 117, 211, 305, 24, 118, 212, 306, 25, 119, 213, 307,
26, 120, 214, 308, 27, 121, 215, 309, 28, 122, 216, 310, 29,
123, 217, 311, 30, 124, 218, 312, 31, 125, 219, 313, 32, 126,
220, 314, 33, 127, 221, 315, 34, 128, 222, 316, 35, 129, 223,
317, 36, 130, 224, 318, 37, 131, 225, 319, 38, 132, 226, 320,
39, 133, 227, 321, 40, 134, 228, 322, 41, 135, 229, 323, 42,
136, 230, 324, 43, 137, 231, 325, 44, 138, 232, 326, 45, 139,
233, 327, 46, 140, 234, 328, 47, 141, 235, 329, 48, 142, 236,
330, 49, 143, 237, 331, 50, 144, 238, 332, 51, 145, 239, 333,
52, 146, 240, 334, 53, 147, 241, 335, 54, 148, 242, 336, 55,
149, 243, 337, 56, 150, 244, 338, 57, 151, 245, 339, 58, 152,
246, 340, 59, 153, 247, 341, 60, 154, 248, 342, 61, 155, 249,
343, 62, 156, 250, 344, 63, 157, 251, 345, 64, 158, 252, 346,
65, 159, 253, 347, 66, 160, 254, 348, 67, 161, 255, 349, 68,
162, 256, 350, 69, 163, 257, 351, 70, 164, 258, 352, 71, 165,
259, 353, 72, 166, 260, 354, 73, 167, 261, 355, 74, 168, 262,
356, 75, 169, 263, 357, 76, 170, 264, 358, 77, 171, 265, 359,
78, 172, 266, 360, 79, 173, 267, 361, 80, 174, 268, 362, 81,
175, 269, 363, 82, 176, 270, 364, 83, 177, 271, 365, 84, 178,
272, 366, 85, 179, 273, 367, 86, 180, 274, 368, 87, 181, 275,
369, 88, 182, 276, 370, 89, 183, 277, 371, 90, 184, 278, 372,
91, 185, 279, 373, 92, 186, 280, 374, 93, 187, 281, 375])
So I am doing following
all_c = all[:,l]
but I am getting
"memory error"
Can you suggest what could be the most memory-efficient way?

Rather than permute the whole array at once you can do it row by row in place. Try
for r in range(all.shape[0]):
all[r] = all[r, l]

image analysis : separating intersecting spaghettis

I've got an image which to the human eye clearly contains several (2 here) overlapping 'laces'.
This stems from a pretty lengthy image analysis of an experiment.
The red dots are all the points contained in two vectors, px and py, one with the x_position of the red pixels, the other one with the y positions.
px= array([ 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1,
2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3,
3, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5,
5, 5, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7,
7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9,
9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10,
11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12,
13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14,
15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16,
16, 16, 16, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18,
18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20,
20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 22, 22,
22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23,
24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25,
25, 25, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27,
27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29,
29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 31,
31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32,
32, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34,
34, 34, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36,
36, 36, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38,
38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40,
40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 42,
42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43,
44, 44, 44, 44, 44, 44, 44, 44, 45, 45, 45, 45, 45,
45, 45, 45, 45, 45, 45, 46, 46, 46, 46, 46, 46, 46,
46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 47, 47,
47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 48, 48,
48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48,
48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49,
49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50,
50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50,
50, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51,
51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52,
52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52,
52, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53,
53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54,
54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54,
54, 54, 54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 55,
55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55,
55, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56,
56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56,
57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57,
57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58,
58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58,
58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59,
59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59,
59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60,
61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61,
61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 62, 62, 62,
62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62,
62, 62, 62, 62, 62, 62, 62, 63, 63, 63, 63, 63, 63,
63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63,
63, 63, 63, 63, 63, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65,
65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 66, 66,
66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66,
66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67, 67,
67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67,
67, 67, 67, 67, 67, 67, 68, 68, 68, 68, 68, 68, 68,
68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68,
68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69,
69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69,
69, 69, 69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 70,
70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70,
71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71,
71, 71, 71, 71, 71, 72, 72, 72, 72, 72, 72, 72, 72,
72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73,
73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73,
73, 73, 73, 73, 74, 74, 74, 74, 74, 74, 74, 74, 74,
74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 75,
75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75,
75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76,
76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76,
76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76,
77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77,
77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78,
78, 78, 78, 78, 79, 79, 79, 79, 79, 79, 79, 79, 79,
79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79,
79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79,
79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79,
79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 80,
80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80,
80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80,
80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80,
80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81,
81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81,
81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81,
81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81,
81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81,
81, 81, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82,
82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82,
82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82,
82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82,
82, 82, 82, 82, 82, 82, 82, 82, 83, 83, 83, 83, 83,
83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83,
83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83,
83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83,
83, 83, 83, 83, 83, 83, 83, 83, 83, 84, 84, 84, 84,
84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84,
84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84,
84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84,
84, 84, 84, 84, 85, 85, 85, 85, 85, 85, 85, 85, 85,
85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85,
85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 86,
86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 87, 87,
87, 87, 87, 87, 88, 88, 88, 88, 88, 88, 89, 89, 89,
89, 89, 89, 89, 90, 90, 90, 90, 90, 90, 91, 91, 91,
91, 91, 91, 91, 92, 92, 92, 92, 92, 92, 92, 93, 93,
93, 93, 93, 93, 93, 94, 94, 94, 94, 94, 94, 95, 95,
95, 95, 95, 95, 96, 96, 96, 96, 96, 96, 97, 97, 97,
97, 97, 97, 97, 98, 98, 98, 98, 98, 98, 99, 99, 99,
99, 99, 99, 100, 100, 100, 100, 100, 100, 101, 101, 101, 101,
101, 101, 102, 102, 102, 102, 102, 102, 103, 103, 103, 103, 103,
103, 104, 104, 104, 104, 104, 104, 105, 105, 105, 105, 105, 105,
106, 106, 106, 106, 106, 106, 106, 107, 107, 107, 107, 107, 107,
108, 108, 108, 108, 108, 108, 108, 109, 109, 109, 109, 109, 109,
110, 110, 110, 110, 110, 110, 110, 111, 111, 111, 111, 111, 111,
112, 112, 112, 112, 112, 112, 113, 113, 113, 113, 113, 113, 114,
114, 114, 114, 114, 114, 115, 115, 115, 115, 115, 115, 116, 116,
116, 116, 116, 116, 117, 117, 117, 117, 117, 117, 117, 118, 118,
118, 118, 118, 118, 119, 119, 119, 119, 119, 119, 119, 120, 120,
120, 120, 120, 120, 121, 121, 121, 121, 121, 121, 121, 122, 122,
122, 122, 122, 122, 123, 123, 123, 123, 123, 123, 123, 124, 124,
124, 124, 124, 124, 125, 125, 125, 125, 125, 125, 125, 126, 126,
126, 126, 126, 126, 127, 127, 127, 127, 127, 127, 128, 128, 128,
128, 128, 128, 129, 129, 129, 129, 129, 129, 129, 130, 130, 130,
130, 130, 130, 131, 131, 131, 131, 131, 131, 131, 132, 132, 132,
132, 132, 132, 133, 133, 133, 133, 133, 133, 133, 134, 134, 134,
134, 134, 134, 135, 135, 135, 135, 135, 135, 135, 136, 136, 136,
136, 136, 136, 137, 137, 137, 137, 137, 137, 138, 138, 138, 138,
138, 138, 139, 139, 139, 139, 139, 139, 140, 140, 140, 140, 140,
140, 141, 141, 141, 141, 141, 141, 141, 142, 142, 142, 142, 142,
142, 143, 143, 143, 143, 143, 143, 143, 144, 144, 144, 144, 144,
144, 145, 145, 145, 145, 145, 145, 145, 146, 146, 146, 146, 146,
146, 147, 147, 147, 147, 147, 147, 148, 148, 148, 148, 148, 148,
149, 149, 149, 149, 149, 149, 150, 150, 150, 150, 150, 150, 150,
151, 151, 151, 151, 151, 151, 152, 152, 152, 152, 152, 152, 153,
153, 153, 153, 153, 153, 153, 154, 154, 154, 154, 154, 154, 155,
155, 155, 155, 155, 155, 155, 156, 156, 156, 156, 156, 156, 157,
157, 157, 157, 157, 157, 158, 158, 158, 158, 158, 158, 158, 159,
159, 159, 159, 159, 159, 159, 160, 160, 160, 160, 160, 160, 160,
161, 161, 161, 161, 161, 161, 162, 162, 162, 162, 162, 162, 162,
163, 163, 163, 163, 163, 163, 163, 164, 164, 164, 164, 164, 164,
164, 165, 165, 165, 165, 165, 165, 165, 165, 166, 166, 166, 166,
166, 166, 166, 166, 167, 167, 167, 167, 167, 167, 167, 168, 168,
168, 168, 168, 168, 168, 169, 169, 169, 169, 169, 169, 169, 170,
170, 170, 170, 170, 170, 170, 171, 171, 171, 171, 171, 171, 171,
172, 172, 172, 172, 172, 172, 172, 173, 173, 173, 173])
and
py=array([189, 190, 191, 192, 193, 194, 189, 190, 191, 192, 193, 194, 195,
190, 191, 192, 193, 194, 195, 196, 190, 191, 192, 193, 194, 195,
196, 191, 192, 193, 194, 195, 196, 197, 192, 193, 194, 195, 196,
197, 198, 193, 194, 195, 196, 197, 198, 199, 194, 195, 196, 197,
198, 199, 200, 194, 195, 196, 197, 198, 199, 200, 201, 195, 196,
197, 198, 199, 200, 201, 202, 196, 197, 198, 199, 200, 201, 202,
197, 198, 199, 200, 201, 202, 203, 198, 199, 200, 201, 202, 203,
199, 200, 201, 202, 203, 204, 199, 200, 201, 202, 203, 204, 205,
199, 200, 201, 202, 203, 204, 205, 206, 200, 201, 202, 203, 204,
205, 206, 207, 202, 203, 204, 205, 206, 207, 202, 203, 204, 205,
206, 207, 208, 203, 204, 205, 206, 207, 208, 209, 204, 205, 206,
207, 208, 209, 210, 205, 206, 207, 208, 209, 210, 211, 206, 207,
208, 209, 210, 211, 212, 206, 207, 208, 209, 210, 211, 212, 213,
207, 208, 209, 210, 211, 212, 213, 214, 208, 209, 210, 211, 212,
213, 214, 209, 210, 211, 212, 213, 214, 215, 210, 211, 212, 213,
214, 215, 216, 211, 212, 213, 214, 215, 216, 217, 211, 212, 213,
214, 215, 216, 217, 212, 213, 214, 215, 216, 217, 218, 219, 213,
214, 215, 216, 217, 218, 219, 220, 214, 215, 216, 217, 218, 219,
220, 215, 216, 217, 218, 219, 220, 221, 216, 217, 218, 219, 220,
221, 222, 217, 218, 219, 220, 221, 222, 223, 218, 219, 220, 221,
222, 223, 218, 219, 220, 221, 222, 223, 224, 219, 220, 221, 222,
223, 224, 225, 220, 221, 222, 223, 224, 225, 226, 221, 222, 223,
224, 225, 226, 227, 221, 222, 223, 224, 225, 226, 227, 228, 222,
223, 224, 225, 226, 227, 228, 223, 224, 225, 226, 227, 228, 229,
224, 225, 226, 227, 228, 229, 230, 311, 225, 226, 227, 228, 229,
230, 309, 310, 311, 312, 313, 225, 226, 227, 228, 229, 230, 231,
307, 308, 309, 310, 311, 312, 313, 314, 226, 227, 228, 229, 230,
231, 232, 306, 307, 308, 309, 310, 311, 312, 313, 314, 227, 228,
229, 230, 231, 232, 304, 305, 306, 307, 308, 309, 310, 311, 312,
313, 314, 228, 229, 230, 231, 232, 233, 302, 303, 304, 305, 306,
307, 308, 309, 310, 311, 312, 313, 228, 229, 230, 231, 232, 233,
234, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311,
312, 229, 230, 231, 232, 233, 234, 298, 299, 300, 301, 302, 303,
304, 305, 306, 307, 308, 309, 310, 230, 231, 232, 233, 234, 235,
296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308,
309, 230, 231, 232, 233, 234, 235, 236, 294, 295, 296, 297, 298,
299, 300, 301, 302, 303, 304, 305, 306, 231, 232, 233, 234, 235,
236, 237, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301,
302, 303, 304, 305, 232, 233, 234, 235, 236, 237, 238, 239, 288,
289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301,
302, 233, 234, 235, 236, 237, 238, 239, 240, 285, 286, 287, 288,
289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301,
234, 235, 236, 237, 238, 239, 240, 283, 284, 285, 286, 287, 288,
289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 235, 236, 237,
238, 239, 240, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290,
291, 292, 293, 294, 295, 296, 235, 236, 237, 238, 239, 240, 241,
278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290,
291, 292, 293, 236, 237, 238, 239, 240, 241, 242, 276, 277, 278,
279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291,
237, 238, 239, 240, 241, 242, 243, 273, 274, 275, 276, 277, 278,
279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 238, 239, 240,
241, 242, 243, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278,
279, 280, 281, 282, 283, 284, 285, 238, 239, 240, 241, 242, 243,
244, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278,
279, 280, 281, 282, 283, 239, 240, 241, 242, 243, 244, 265, 266,
267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279,
280, 240, 241, 242, 243, 244, 245, 261, 262, 263, 264, 265, 266,
267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 240, 241,
242, 243, 244, 245, 258, 259, 260, 261, 262, 263, 264, 265, 266,
267, 268, 269, 270, 271, 272, 273, 274, 241, 242, 243, 244, 245,
246, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266,
267, 268, 269, 270, 271, 272, 241, 242, 243, 244, 245, 246, 247,
252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264,
265, 266, 267, 268, 269, 242, 243, 244, 245, 246, 247, 249, 250,
251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263,
264, 265, 266, 267, 242, 243, 244, 245, 246, 247, 248, 249, 250,
251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263,
243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255,
256, 257, 258, 259, 260, 240, 241, 242, 243, 244, 245, 246, 247,
248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 235, 236, 237,
238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250,
251, 252, 253, 254, 231, 232, 233, 234, 235, 236, 237, 238, 239,
240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 227,
228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240,
241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 223, 224, 225,
226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238,
239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251,
215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227,
228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240,
241, 242, 246, 247, 248, 249, 250, 251, 210, 211, 212, 213, 214,
215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227,
228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 246, 247,
248, 249, 250, 251, 189, 190, 191, 192, 193, 194, 195, 196, 197,
198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210,
211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223,
224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 247, 248,
249, 250, 251, 252, 183, 184, 185, 186, 187, 188, 189, 190, 191,
192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204,
205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217,
218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230,
247, 248, 249, 250, 251, 252, 172, 173, 174, 175, 176, 177, 178,
179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204,
205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217,
218, 219, 220, 221, 222, 223, 224, 225, 226, 248, 249, 250, 251,
252, 253, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178,
179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204,
205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217,
218, 219, 249, 250, 251, 252, 253, 254, 167, 168, 169, 170, 171,
172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184,
185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197,
198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210,
211, 212, 213, 249, 250, 251, 252, 253, 254, 167, 168, 169, 170,
171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183,
184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196,
197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 250, 251,
252, 253, 254, 255, 167, 168, 169, 170, 171, 172, 173, 174, 175,
176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 250, 251,
252, 253, 254, 255, 168, 169, 170, 171, 172, 173, 174, 175, 176,
177, 178, 179, 180, 250, 251, 252, 253, 254, 255, 256, 251, 252,
253, 254, 255, 256, 252, 253, 254, 255, 256, 257, 252, 253, 254,
255, 256, 257, 258, 253, 254, 255, 256, 257, 258, 253, 254, 255,
256, 257, 258, 259, 254, 255, 256, 257, 258, 259, 260, 254, 255,
256, 257, 258, 259, 260, 255, 256, 257, 258, 259, 260, 256, 257,
258, 259, 260, 261, 256, 257, 258, 259, 260, 261, 256, 257, 258,
259, 260, 261, 262, 257, 258, 259, 260, 261, 262, 258, 259, 260,
261, 262, 263, 258, 259, 260, 261, 262, 263, 259, 260, 261, 262,
263, 264, 259, 260, 261, 262, 263, 264, 260, 261, 262, 263, 264,
265, 260, 261, 262, 263, 264, 265, 261, 262, 263, 264, 265, 266,
261, 262, 263, 264, 265, 266, 267, 262, 263, 264, 265, 266, 267,
262, 263, 264, 265, 266, 267, 268, 263, 264, 265, 266, 267, 268,
263, 264, 265, 266, 267, 268, 269, 264, 265, 266, 267, 268, 269,
265, 266, 267, 268, 269, 270, 265, 266, 267, 268, 269, 270, 266,
267, 268, 269, 270, 271, 266, 267, 268, 269, 270, 271, 267, 268,
269, 270, 271, 272, 267, 268, 269, 270, 271, 272, 273, 268, 269,
270, 271, 272, 273, 268, 269, 270, 271, 272, 273, 274, 269, 270,
271, 272, 273, 274, 269, 270, 271, 272, 273, 274, 275, 270, 271,
272, 273, 274, 275, 270, 271, 272, 273, 274, 275, 276, 271, 272,
273, 274, 275, 276, 271, 272, 273, 274, 275, 276, 277, 272, 273,
274, 275, 276, 277, 273, 274, 275, 276, 277, 278, 273, 274, 275,
276, 277, 278, 273, 274, 275, 276, 277, 278, 279, 274, 275, 276,
277, 278, 279, 274, 275, 276, 277, 278, 279, 280, 275, 276, 277,
278, 279, 280, 275, 276, 277, 278, 279, 280, 281, 276, 277, 278,
279, 280, 281, 276, 277, 278, 279, 280, 281, 282, 277, 278, 279,
280, 281, 282, 278, 279, 280, 281, 282, 283, 278, 279, 280, 281,
282, 283, 279, 280, 281, 282, 283, 284, 279, 280, 281, 282, 283,
284, 279, 280, 281, 282, 283, 284, 285, 280, 281, 282, 283, 284,
285, 280, 281, 282, 283, 284, 285, 286, 281, 282, 283, 284, 285,
286, 281, 282, 283, 284, 285, 286, 287, 282, 283, 284, 285, 286,
287, 283, 284, 285, 286, 287, 288, 283, 284, 285, 286, 287, 288,
284, 285, 286, 287, 288, 289, 284, 285, 286, 287, 288, 289, 290,
285, 286, 287, 288, 289, 290, 286, 287, 288, 289, 290, 291, 286,
287, 288, 289, 290, 291, 292, 287, 288, 289, 290, 291, 292, 287,
288, 289, 290, 291, 292, 293, 288, 289, 290, 291, 292, 293, 289,
290, 291, 292, 293, 294, 289, 290, 291, 292, 293, 294, 295, 290,
291, 292, 293, 294, 295, 296, 291, 292, 293, 294, 295, 296, 297,
292, 293, 294, 295, 296, 297, 292, 293, 294, 295, 296, 297, 298,
293, 294, 295, 296, 297, 298, 299, 294, 295, 296, 297, 298, 299,
300, 294, 295, 296, 297, 298, 299, 300, 301, 295, 296, 297, 298,
299, 300, 301, 302, 296, 297, 298, 299, 300, 301, 302, 297, 298,
299, 300, 301, 302, 303, 298, 299, 300, 301, 302, 303, 304, 299,
300, 301, 302, 303, 304, 305, 300, 301, 302, 303, 304, 305, 306,
301, 302, 303, 304, 305, 306, 307, 302, 303, 304, 305])
To reproduce the image, this MWE should be enough, once you copy-paste the arrays as px and py into your editor
import numpy as np
plot_mtx = np.zeros( (5120,5120) )
px=...
py=...
window_size = max(px[-1]-px[0],[py[-1]-py[0]])
imshow(plot_mtx[min(px):min(px)+window_size,min(py):min(py)+window_size])
Essentially, these are two lines that each have a width of about 6 pixels. I would like to separate them into two distinct lines, so that I would have px_1,py_1 for the one running from top to down, and px_2,py_2 for the one running from left to right.
I've failed so far to find the right keywords to find such algorithms. What I would have in mind is a for loop, that starts at one extremity, to complete a list with pixel positions in such a way, that the direction in which the loop is progressing never exceeds a certain angle with respect to several previous steps. This way, the algorithm would always take the set of pixels that cross smoothly through the intersection. Then it's just a matter of removing the new found line from the data and starting again.
Would you have any better input ?
Any feedback appreciated

Skeletonize the image.
Split by the crossing point(it has more than 2 neighbours) - that gives you 4 segments.
Join similarly directed pairs together - you get 2 skeleton lines.
Assign original pixels to each of those lines by proximity threshold.

Python, calculating lag of the center of the data

My two data sets are:
fnamerp1=([ 93, 87, 96, 93, 90, 123, 111, 82, 87, 115, 103,
101, 93, 92, 111, 107, 114, 106, 116, 106, 128, 115,
141, 134, 120, 149, 140, 166, 152, 171, 192, 207, 227,
266, 270, 286, 355, 385, 397, 488, 462, 531, 579, 622,
711, 720, 801, 858, 906, 915, 915, 956, 1004, 1012, 1045,
1076, 1063, 1013, 985, 924, 959, 838, 766, 763, 742, 642,
587, 557, 484, 393, 353, 341, 284, 240, 221, 209, 147,
109, 113, 102, 71, 63, 63, 50, 29, 39, 36, 25,
30, 23, 27, 23, 19, 19, 24, 15, 23, 21, 26,
15])
fnamerp2=([ 105, 89, 120, 121, 103, 105, 113, 94, 104, 115, 122, 116, 121,
129, 118, 126, 138, 146, 161, 163, 178, 192, 194, 222, 268, 272,
285, 342, 380, 378, 373, 448, 493, 511, 571, 603, 691, 772, 738,
796, 839, 832, 883, 930, 963, 975, 972, 931, 947, 941, 934, 964,
871, 869, 826, 793, 733, 708, 606, 610, 515, 483, 409, 352, 358,
264, 266, 205, 191, 167, 136, 138, 99, 102, 82, 57, 65, 53,
51, 32, 26, 27, 39, 21, 29, 23, 25, 24, 16, 17, 27,
33, 19, 13, 24, 26, 18, 22, 18, 20])
I want to find the lag between the center of the two peaks (not just their max). And my plan is to use np.argmax(signal.correlate(fnamerp1,fnamerp2)).
What is the right way to do this both from a mathematical perspective and also elegant in Python?

Categories