We Keep Coding

finding the size of an array in python

for i in set(data['MovementNumber'].values):
data2 = data.loc[data['MovementNumber'] == i]
x = len(data2)
print(x)
This for loop basically makes a cell array for the 83 trials I have. The issue im having is that when I use len for the cell array. i get the array with the length of the individual arrays. and i get this array:
'''
[957
280
305
217
204
321
228
291
341
245
381
177
284
410
226
279
270
241
302
235
260
218
207
202
261
370
288
210
243
199
282
184
223
191
205
228
244
213
164
230
226
245
197
303
187
239
242
267
250
221
271
218
208
225
248
334
215
462
321
319
346
231
223
293
261
274
181
304
329
295
311
298
303
227
218
245
235
182
215
242
174
261
460]
I want the length of this array but when i try to use len(x) i get the following error
object of type 'int' has no len()
can someone help me with this? Thanks!

Reshaping a 3D array to a 2D array to produce a DataFrame: keep track of indices to produce column names

The following code generates a pandas.DataFrame from a 3D array over the first axis. I manually create the columns names (defining cols): is there a more built-in way to do this (to avoid potential errors e.g. regarding C-order)?
--> I am looking for a way to guarantee the respect of the order of the indices after the reshape operation (here it relies on the correct order of the iterations over range(nrow) and range(ncol)).
import numpy as np
import pandas as pd
nt = 6 ; nrow = 4 ; ncol = 3 ; shp = (nt, nrow, ncol)
np.random.seed(0)
a = np.array(np.random.randint(0, 1000, nt*nrow*ncol)).reshape(shp)
# This is the line I think should be improved --> any numpy function or so?
cols = [str(i) + '-' + str(j) for i in range(nrow) for j in range(ncol)]
adf = pd.DataFrame(a.reshape(nt, -1), columns = cols)
print(adf)
0-0 0-1 0-2 1-0 1-1 1-2 2-0 2-1 2-2 3-0 3-1 3-2
0 684 559 629 192 835 763 707 359 9 723 277 754
1 804 599 70 472 600 396 314 705 486 551 87 174
2 600 849 677 537 845 72 777 916 115 976 755 709
3 847 431 448 850 99 984 177 755 797 659 147 910
4 423 288 961 265 697 639 544 543 714 244 151 675
5 510 459 882 183 28 802 128 128 932 53 901 550
EDIT
Illustrating why I don't like my solution - it is just too easy to make a code which technically works but produce a wrong result (inverting i and j or nrow and ncol):
wrongcols1 = [str(i) + '-' + str(j) for i in range(ncol) for j in range(nrow)]
adf2 = pd.DataFrame(a.reshape(nt, -1), columns=wrongcols1)
print(adf2)
0-0 0-1 0-2 0-3 1-0 1-1 1-2 1-3 2-0 2-1 2-2 2-3
0 684 559 629 192 835 763 707 359 9 723 277 754
1 804 599 70 472 600 396 314 705 486 551 87 174
2 600 849 677 537 845 72 777 916 115 976 755 709
3 847 431 448 850 99 984 177 755 797 659 147 910
4 423 288 961 265 697 639 544 543 714 244 151 675
5 510 459 882 183 28 802 128 128 932 53 901 550
wrongcols2 = [str(j) + '-' + str(i) for i in range(nrow) for j in range(ncol)]
adf3 = pd.DataFrame(a.reshape(nt, -1), columns=wrongcols2)
print(adf3)
0-0 1-0 2-0 0-1 1-1 2-1 0-2 1-2 2-2 0-3 1-3 2-3
0 684 559 629 192 835 763 707 359 9 723 277 754
1 804 599 70 472 600 396 314 705 486 551 87 174
2 600 849 677 537 845 72 777 916 115 976 755 709
3 847 431 448 850 99 984 177 755 797 659 147 910
4 423 288 961 265 697 639 544 543 714 244 151 675
5 510 459 882 183 28 802 128 128 932 53 901 550

Try this and see if it fits your use case:
Generate columns via a combination of np.indices, np.dstack and np.vstack :
columns = np.vstack(np.dstack(np.indices((nrow, ncol))))
array([[0, 0],
[0, 1],
[0, 2],
[1, 0],
[1, 1],
[1, 2],
[2, 0],
[2, 1],
[2, 2],
[3, 0],
[3, 1],
[3, 2]])
Now convert to string via a combination of map, join and list comprehension:
columns = ["-".join(map(str, entry)) for entry in columns]
['0-0',
'0-1',
'0-2',
'1-0',
'1-1',
'1-2',
'2-0',
'2-1',
'2-2',
'3-0',
'3-1',
'3-2']
Let's know how it goes.

You could try to use pd.MultiIndex to construct your hierarchy.
First redefine your cols to a list of tuples:
cols = [(i, j) for i in range(nrow) for j in range(ncol)]
Then construct the multi index with cols:
multi_cols = pd.MultiIndex.from_tuples(cols)
And build the dataframe:
adf = pd.DataFrame(a.reshape(nt, -1), columns=multi_cols)
Result:
0 1 2 3
0 1 2 0 1 2 0 1 2 0 1 2
0 684 559 629 192 835 763 707 359 9 723 277 754
1 804 599 70 472 600 396 314 705 486 551 87 174
2 600 849 677 537 845 72 777 916 115 976 755 709
3 847 431 448 850 99 984 177 755 797 659 147 910
4 423 288 961 265 697 639 544 543 714 244 151 675
5 510 459 882 183 28 802 128 128 932 53 901 550
Access of elements:
print(adf[1][2][0])
>>> 763

Generate missing multi-indexes

having a dataframe such as:
myFrame = pd.DataFrame(np.random.randint(1000, size=[7,4]),
index=[['GER','GER','GER','GER','FRA','FRA','FRA'],
['Phone','Email','Chat','Other','Phone','Chat','Email']])
0 1 2 3
GER Phone 765 876 588 933
Email 819 364 42 73
Chat 954 665 317 918
Other 692 531 312 206
FRA Phone 272 261 426 270
Chat 158 172 347 902
Email 453 721 67 6
How could I easily add the missing index(es) of the inner level? E.g. you can see that GER has an "Other" index label. I'd like to add that "Other" to all countries and fill it's values with e.g. 0. There might be a third outer index (e.g. ITA), for which yet another inner-index could be found (e.g. SMS).
At the end, all countries should have exactly the same amount of indexes.
Thanks!

Use reindex with MultiIndex.from_product created by unique values of each levels generated by MultiIndex.get_level_values:
mux = pd.MultiIndex.from_product([myFrame.index.get_level_values(0).unique(),
myFrame.index.get_level_values(1).unique()])
print (myFrame.reindex(mux, fill_value=0))
0 1 2 3
GER Phone 250 614 226 777
Email 917 156 148 902
Chat 537 665 87 75
Other 431 203 921 572
FRA Phone 159 790 646 810
Email 294 205 949 726
Chat 209 895 128 282
Other 0 0 0 0
Another solution with unstack and stack - MultiIndex is sorted:
print (myFrame.unstack(fill_value=0).stack(dropna=False))
0 1 2 3
FRA Chat 209 895 128 282
Email 294 205 949 726
Other 0 0 0 0
Phone 159 790 646 810
GER Chat 537 665 87 75
Email 917 156 148 902
Other 431 203 921 572
Phone 250 614 226 777

How to get integers instead of floats from DataFrame's rank method

To substitute the numbers with their corresponding "ranks":
import pandas as pd
import numpy as np
numbers = np.random.random_integers(low=0.0, high=10000.0, size=(1000,))
df = pd.DataFrame({'a': numbers})
df['a_rank'] = df['a'].rank()
I am getting the float values as the default output type of rankmethod:
987 82.0
988 36.5
989 526.0
990 219.0
991 957.0
992 819.5
993 787.5
994 513.0
Instead of floats I would rather have the integers. Rounding the resulted float values using asType(int) would be risky since converting to int would probably introduce the duplicated values from the float values that are too close to each other such as 3.5 and 4.0. Those when converted to the integers both would result to the integer value of 4.
Is there any way to guide rank method to output the integers?

The above solution did not work for me. The following did work though. The critical line with edits is:
df['a_rank'] = df['a'].rank(method='dense').astype(int);
This could be a version issue.

Pass param method='dense', this will increase the ranks by 1 between groups, see the docs:
In [2]:
numbers = np.random.random_integers(low=0.0, high=10000.0, size=(1000,))
df = pd.DataFrame({'a': numbers})
df['a_rank'] = df['a'].rank(method='dense')
df
Out[2]:
a a_rank
0 1095 114
1 2514 248
2 500 53
3 6112 592
4 5582 533
5 851 91
6 2887 287
7 3798 366
8 4698 458
9 1699 170
10 4739 462
11 7199 693
12 817 88
13 3801 367
14 5584 534
15 4939 481
16 2569 258
17 6806 656
18 93 8
19 8574 816
20 4107 396
21 7086 684
22 6819 657
23 8844 847
24 170 15
25 6629 634
26 9905 950
27 5312 512
28 3794 365
29 9476 911
.. ... ...
970 4607 447
971 8430 801
972 6527 625
973 2794 280
974 4414 425
975 1069 111
976 2849 285
977 7955 759
978 5767 547
979 7767 742
980 2956 294
981 5847 554
982 1029 107
983 4967 485
984 256 25
985 5577 532
986 6866 662
987 5903 563
988 1785 181
989 749 78
990 2164 212
991 1074 112
992 8752 837
993 2737 272
994 2761 277
995 7355 705
996 8956 857
997 4831 473
998 222 21
999 9531 917
[1000 rows x 2 columns]

No need to use method='dense', just convert to an integer.
df['a_rank'] = df['a'].rank().astype(int)

Python : Gann square of 9

I want to simulate the Gann's Square of 9 chart in Python.
For people who are not aware of what the chart looks like, click here for a basic idea and enter a value(say, 100) in the current market price blank.
The chart essentially expands as a spiral, starting from the center position. ( A picture of the Gann chart spiral )
My questions regarding this are as follows :
What is the best data structure I should choose to build an expanding spiral? Was thinking of constructing a matrix for it, but how do I regulate the matrix?
How do I search (quickly, preferably without traversing through the entire matrix) for a particular value in the chart? (assuming it is a big chart with about 100 levels in the spiral)
I am stuck at the approach to start coding the chart, so any insight into this would be wonderful.

Gann Square of Any Size
Hopefully this solves your problem
class GannSquare():
"""An container object for Gann Square"""
def __init__(self, size):
""" attributes and method of Gann Square
input parameters: size
output returned: an object along with its attributes
"""
self.size = size
self.even_size = size % 2 == 0
self.odd_size = size % 2 == 1
self.num_elements = size ** 2
self.matrix = self.generate()
self.horizontal_axis = self.get_horizontal_axis()
self.vertical_axis = self.get_vertical_axis()
self.diagonal_1 = self.get_diagonal_1()
self.diagonal_2 = self.get_diagonal_2()
self.diagonal_axis = self.get_diagonal_axis()
def generate(self):
"""
generate a Gann Square based on the input parameter size
"""
from numpy import array
NORTH, SOUTH, EAST, WEST = (0, 1), (0, -1), (1, 0), (-1, 0) # directional vectors
clockwise = {
WEST:NORTH,
NORTH: EAST,
EAST: SOUTH,
SOUTH: WEST
} # clockwise transformation
RIGHT, LEFT = 1, -1
# forward or backward increment
if self.size < 1:
raise ValueError
x, y = self.size // 2, self.size // 2
# the middle of the box
dx, dy = WEST # initial direction
inc = LEFT # backward increment
G = [[None] * self.size for _ in range(self.size)]
count = 0
while True:
count += 1
G[x][y] = count # visit
# follow predefined direction
_dx, _dy = clockwise[dx,dy]
_x, _y = x +inc* _dx, y +inc* _dy
if (0 <= _x < self.size and 0 <= _y < self.size and
G[_x][_y] is None):
# in the box
x, y = _x, _y
dx, dy = _dx, _dy
else: # fill in the box
x, y = x +inc* dx, y +inc* dy
if not (0 <= x < self.size and 0 <= y < self.size):
return array(G) # out of the box
def display_matrix(self):
width = len(str(max(e for row in self.matrix for e in row if e is not None)))
fmt = "{:0%dd}" % width
for row in self.matrix:
print(" ".join("_"*width if e is None else fmt.format(e) for e in row))
def get_horizontal_axis(self):
return self.matrix[:,self.size//2]
def get_vertical_axis(self):
return self.matrix[self.size//2, :]
def get_diagonal_1(self):
diagonal_1 = []
for i in range(self.size):
diagonal_1.append(self.matrix[i,self.size-i-1])
return diagonal_1
def get_diagonal_2(self):
diagonal_2 = []
for i in range(self.size):
diagonal_2.append(self.matrix[i,i])
return diagonal_2
def get_diagonal_axis(self):
from numpy import concatenate
return concatenate((self.diagonal_1, self.diagonal_2))
Example usage:
gs = GannSquare(21)
gs.display_matrix()
Result:
381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401
380 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 402
379 306 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 326 403
378 305 240 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 258 327 404
377 304 239 182 133 134 135 136 137 138 139 140 141 142 143 144 145 198 259 328 405
376 303 238 181 132 091 092 093 094 095 096 097 098 099 100 101 146 199 260 329 406
375 302 237 180 131 090 057 058 059 060 061 062 063 064 065 102 147 200 261 330 407
374 301 236 179 130 089 056 031 032 033 034 035 036 037 066 103 148 201 262 331 408
373 300 235 178 129 088 055 030 013 014 015 016 017 038 067 104 149 202 263 332 409
372 299 234 177 128 087 054 029 012 003 004 005 018 039 068 105 150 203 264 333 410
371 298 233 176 127 086 053 028 011 002 001 006 019 040 069 106 151 204 265 334 411
370 297 232 175 126 085 052 027 010 009 008 007 020 041 070 107 152 205 266 335 412
369 296 231 174 125 084 051 026 025 024 023 022 021 042 071 108 153 206 267 336 413
368 295 230 173 124 083 050 049 048 047 046 045 044 043 072 109 154 207 268 337 414
367 294 229 172 123 082 081 080 079 078 077 076 075 074 073 110 155 208 269 338 415
366 293 228 171 122 121 120 119 118 117 116 115 114 113 112 111 156 209 270 339 416
365 292 227 170 169 168 167 166 165 164 163 162 161 160 159 158 157 210 271 340 417
364 291 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 272 341 418
363 290 289 288 287 286 285 284 283 282 281 280 279 278 277 276 275 274 273 342 419
362 361 360 359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 420
441 440 439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421
and several other methods
gs.size, gs.num_elements, gs.even_size, gs.odd_size,
gs.horizontal_axis, gs.vertical_axis, gs.diagonal_axis
are self explanatory.
Numpy array is used to store the Gann matrix so that we can use numpy.where whenever we need to search a particular value in it.

Suppose 1 is located (0, 0), the exact position of any given positive integer can be obtained by following function:
import math
def f(num):
n = math.floor(math.sqrt(num)) // 2
if (num <= 4*n*n):
k = num - (2*n-1)**2
if k <= 2*n:
return (- n, k - n + 1)
else:
return (k - 3 * n, n)
else:
k = num - (2*n)**2
if k <= 2*n + 1:
return (n, n - k + 1)
else:
return (3 * n - k + 1, - n)
This is derived from two facts:
number (2*n+1)^2 is always located (-n, -n)
number (2*n)^2 is always located (n-1, n)