Python - Reverse every other slice of an array - python
I wrote some code to reverse every other slice of rows.
import numpy as np
test_arr = np.arange(120).reshape(12,10)
test_arr = test_arr.tolist()
def rev_rows(matrix):
for I in range(len(matrix)): #did this to get the index of each row
if((int(I / 4) % 2) == True): #selct rows whose index divided by 4 truncate to an odd number
print("flip")
matrix[I].reverse() #flip said row
print(matrix[I])
rev_rows(test_arr)
There has to by an easier and more efficient way of doing this. I was thinking another way would be to use list operators like slices, but I can't think of one which is faster than this. Is there an easier way with numpy?
Note: the length of the matrix would be divisible by 4. i.e. (4x10), (8x10), ...
EDIT:
Sorry about the ambiguous usage of slice. What I meant by a slice is a set of rows (like test_arr[3] -> test_arr[7]). So, reversing every other slice would be reversing every row between indexes 3 and 7. I was in my little blurb about the slicing operator I was talking about this operator -> [3:7]. I don't have experience with them, and I read somewhere that they are called slicing, my bad.
Update
The question wasn't very clear, so my original answer didn't solve it. Here is a working example.
With a loop
The loop version's performance is more predictable, because it's not always clear when a reshape will trigger a copy.
>>> test_arr = np.arange(120).reshape(12, 10)
>>> for i in range(4, 8):
... test_arr[i::8] = test_arr[i::8,::-1]
>>> test_arr
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[ 30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[ 49, 48, 47, 46, 45, 44, 43, 42, 41, 40],
[ 59, 58, 57, 56, 55, 54, 53, 52, 51, 50],
[ 69, 68, 67, 66, 65, 64, 63, 62, 61, 60],
[ 79, 78, 77, 76, 75, 74, 73, 72, 71, 70],
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[ 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119]])
>>>
Without a loop
A loopless version, as asked by #KellyBundy.
>>> test_arr = np.arange(120).reshape(12, 10)
>>> temp_arr = test_arr.reshape(test_arr.shape[0]//4, 4, test_arr.shape[1])
>>> temp_arr[1::2] = temp_arr[1::2,:,::-1]
>>> test_arr = temp_arr.reshape(*test_arr.shape)
>>> test_arr
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[ 30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[ 49, 48, 47, 46, 45, 44, 43, 42, 41, 40],
[ 59, 58, 57, 56, 55, 54, 53, 52, 51, 50],
[ 69, 68, 67, 66, 65, 64, 63, 62, 61, 60],
[ 79, 78, 77, 76, 75, 74, 73, 72, 71, 70],
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[ 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119]])
>>>
Original answer
You can do this with slicing:
test_arr[::2] = test_arr[::2,::-1] or test_arr[1::2] = test_arr[1::2,::-1].
See the examples:
>>> import numpy as np
>>> test_arr = np.arange(120).reshape(12, 10)
>>> test_arr
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[ 30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[ 40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[ 50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
[ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[ 70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[ 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119]])
>>> test_arr[::2] = test_arr[::2,::-1]
>>> test_arr
array([[ 9, 8, 7, 6, 5, 4, 3, 2, 1, 0],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[ 29, 28, 27, 26, 25, 24, 23, 22, 21, 20],
[ 30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[ 49, 48, 47, 46, 45, 44, 43, 42, 41, 40],
[ 50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
[ 69, 68, 67, 66, 65, 64, 63, 62, 61, 60],
[ 70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
[ 89, 88, 87, 86, 85, 84, 83, 82, 81, 80],
[ 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
[109, 108, 107, 106, 105, 104, 103, 102, 101, 100],
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119]])
>>>
If, instead, you wanted to reverse rows with odd indices, you'd do
>>> test_arr = np.arange(120).reshape(12, 10)
>>> test_arr[1::2] = test_arr[1::2,::-1]
>>> test_arr
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 19, 18, 17, 16, 15, 14, 13, 12, 11, 10],
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[ 39, 38, 37, 36, 35, 34, 33, 32, 31, 30],
[ 40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[ 59, 58, 57, 56, 55, 54, 53, 52, 51, 50],
[ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[ 79, 78, 77, 76, 75, 74, 73, 72, 71, 70],
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[ 99, 98, 97, 96, 95, 94, 93, 92, 91, 90],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
[119, 118, 117, 116, 115, 114, 113, 112, 111, 110]])
>>>
Slice after reshape:
>>> test_arr = np.arange(120).reshape(12, 10)
>>> test_arr
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[ 30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[ 40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[ 50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
[ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[ 70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[ 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119]])
>>> reshaped = test_arr.reshape(-1, 4, 10)
>>> reshaped[1::2] = reshaped[1::2, ..., ::-1]
>>> test_arr
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[ 30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[ 49, 48, 47, 46, 45, 44, 43, 42, 41, 40],
[ 59, 58, 57, 56, 55, 54, 53, 52, 51, 50],
[ 69, 68, 67, 66, 65, 64, 63, 62, 61, 60],
[ 79, 78, 77, 76, 75, 74, 73, 72, 71, 70],
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[ 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119]])
Note:
This mainly depends on the fact that ndarray.reshape generally does not copy array. However, according to this problem, we can know some details of reshape:
After reshaping the values in the data buffer need to be in a contiguous order, either 'C' or 'F'.
And condition for returning copies:
It will do a copy if the initial order is so 'messed up' that it can't return values like this.
If a copy is returned, this method will obviously fail. But here, I attach a conjecture of my own: if we simply raise the dimension of the array, rather than trying to do things like flattening a transposed array that affect data continuity, it may not cause data discontinuity, so this method may always be effective.
Related
Sending random data to API using Python Flask
I am trying to send random data to API using python flask with intervals of 1 second. But it only shows the last array of data. I am using the following code:
import time
import random
import datetime
from flask import Flask
mylist = []
ct = datetime.datetime.now()
app = Flask(__name__)
#app.route('/')
def index():
mylist = []
ct = datetime.datetime.now()
for i in range(0, 61):
x = random.randint(1, 100)
mylist.append(x)
if len(mylist) == 11:
right_in_left_out = mylist.pop(0)
else:
right_in_left_out = None
time.sleep(1)
print(mylist)
return mylist
if __name__ == "__main__":
app.run(debug=True)
OUTPUT:
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://127.0.0.1:5000
Press CTRL+C to quit
* Restarting with stat
* Debugger is active!
* Debugger PIN: 516-689-025
[50]
[50, 61]
[50, 61, 47]
[50, 61, 47, 63]
[50, 61, 47, 63, 24]
[50, 61, 47, 63, 24, 92]
[50, 61, 47, 63, 24, 92, 18]
[50, 61, 47, 63, 24, 92, 18, 75]
[50, 61, 47, 63, 24, 92, 18, 75, 95]
[50, 61, 47, 63, 24, 92, 18, 75, 95, 4]
[61, 47, 63, 24, 92, 18, 75, 95, 4, 40]
[47, 63, 24, 92, 18, 75, 95, 4, 40, 88]
[63, 24, 92, 18, 75, 95, 4, 40, 88, 39]
[24, 92, 18, 75, 95, 4, 40, 88, 39, 47]
[92, 18, 75, 95, 4, 40, 88, 39, 47, 58]
[18, 75, 95, 4, 40, 88, 39, 47, 58, 82]
[75, 95, 4, 40, 88, 39, 47, 58, 82, 88]
[95, 4, 40, 88, 39, 47, 58, 82, 88, 7]
[4, 40, 88, 39, 47, 58, 82, 88, 7, 90]
[40, 88, 39, 47, 58, 82, 88, 7, 90, 65]
[88, 39, 47, 58, 82, 88, 7, 90, 65, 93]
[39, 47, 58, 82, 88, 7, 90, 65, 93, 9]
[47, 58, 82, 88, 7, 90, 65, 93, 9, 55]
[58, 82, 88, 7, 90, 65, 93, 9, 55, 48]
[82, 88, 7, 90, 65, 93, 9, 55, 48, 83]
[88, 7, 90, 65, 93, 9, 55, 48, 83, 96]
[7, 90, 65, 93, 9, 55, 48, 83, 96, 63]
[90, 65, 93, 9, 55, 48, 83, 96, 63, 8]
[65, 93, 9, 55, 48, 83, 96, 63, 8, 43]
[93, 9, 55, 48, 83, 96, 63, 8, 43, 49]
[9, 55, 48, 83, 96, 63, 8, 43, 49, 95]
[55, 48, 83, 96, 63, 8, 43, 49, 95, 92]
[48, 83, 96, 63, 8, 43, 49, 95, 92, 43]
[83, 96, 63, 8, 43, 49, 95, 92, 43, 57]
[96, 63, 8, 43, 49, 95, 92, 43, 57, 91]
[63, 8, 43, 49, 95, 92, 43, 57, 91, 61]
[8, 43, 49, 95, 92, 43, 57, 91, 61, 27]
[43, 49, 95, 92, 43, 57, 91, 61, 27, 66]
[49, 95, 92, 43, 57, 91, 61, 27, 66, 70]
[95, 92, 43, 57, 91, 61, 27, 66, 70, 4]
[92, 43, 57, 91, 61, 27, 66, 70, 4, 34]
[43, 57, 91, 61, 27, 66, 70, 4, 34, 11]
[57, 91, 61, 27, 66, 70, 4, 34, 11, 95]
[91, 61, 27, 66, 70, 4, 34, 11, 95, 71]
[61, 27, 66, 70, 4, 34, 11, 95, 71, 35]
[27, 66, 70, 4, 34, 11, 95, 71, 35, 4]
[66, 70, 4, 34, 11, 95, 71, 35, 4, 98]
[70, 4, 34, 11, 95, 71, 35, 4, 98, 18]
[4, 34, 11, 95, 71, 35, 4, 98, 18, 81]
[34, 11, 95, 71, 35, 4, 98, 18, 81, 87]
[11, 95, 71, 35, 4, 98, 18, 81, 87, 84]
[95, 71, 35, 4, 98, 18, 81, 87, 84, 37]
[71, 35, 4, 98, 18, 81, 87, 84, 37, 63]
[35, 4, 98, 18, 81, 87, 84, 37, 63, 42]
[4, 98, 18, 81, 87, 84, 37, 63, 42, 18]
[98, 18, 81, 87, 84, 37, 63, 42, 18, 79]
[18, 81, 87, 84, 37, 63, 42, 18, 79, 28]
[81, 87, 84, 37, 63, 42, 18, 79, 28, 12]
[87, 84, 37, 63, 42, 18, 79, 28, 12, 36]
[84, 37, 63, 42, 18, 79, 28, 12, 36, 23]
[37, 63, 42, 18, 79, 28, 12, 36, 23, 49]
127.0.0.1 - - [21/Sep/2022 12:55:46] "GET / HTTP/1.1" 200 -
OUTPUT AT API:
I am looking to send data the same way as it is being displayed in the IDE with 1 sec intervals.
The problem lies here if len(mylist) == 11: right_in_left_out = mylist.pop(0) Once this code executes for the first time, your list size is back to 10 , further iterations everytime it becomes 11 and then back to 10!
Your code is returning a list, and that is logical. If you want to return all the lists like the ones you displayed, you have to store them in a list of lists. I mean by that: all_lists = [] mylist = [] ct = datetime.datetime.now() for i in range(0, 61): x = random.randint(1, 100) mylist.append(x) if len(mylist) == 11: mylist.pop(0) time.sleep(1) all_lists.append(mylist) print(mylist) return all_lists There is also no need to use right_in_left_out variable in your code.
Add list as list element Python weird behaviour
I am trying to store the file indexes from a directory in a list variable using Python. This list will be used for a recursive function that will change the content of files, but the list is storing incorrect values.
My directory has more than 5k files with the format: ccc-ppp.htm where ccc is the chapter number (from 0 to 24) and ppp is page number, that varies from chapter to chapter. For example:
0-1.htm
0-2.htm
0-3.htm
1-1.htm
1-2.htm
...
1-10.htm
2-1.htm
...
2.158.htm
...
24-100.htm
Some chapters have more than 500 pages.
My intention is to store in a list of lists:
Dir = [[1, 2, 3], [1, 2, 3, ..., 10], [1, 2, ..., 158], ..., [0, 1, 2, ..., 100]]
(Note that some chapters start with ...-0.htm
My code:
import os.path
Folder = "C:\MyFoler\\"
Ch = 0
Pg = 0
Chapter = []
Dir = []
while Ch <= 24:
File = Folder + str(Ch) + "-" + str(Pg) + ".htm"
if os.path.exists(File):
Chapter.append(Pg)
else:
print(f"\nChapter {Ch}: {Chapter}") # OBS(1)
Dir.append(Chapter)
print(f"\nDir: {Dir}") # OBS(2)
Chapter.clear()
Ch += 1
Pg = -1
print(f"\nDir: {Dir}") # OBS(3)
Pg += 1
OBS(1): It prints the list of pages in a chapter (working OK). For example to Chapter 24:
Chapter 24: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
OBS(2): It prints a list, with Ch elements, and each element is the Chapter, but it repeats the chapter in all list elements. For example to Ch = 24:
Dir: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]]
The problem is that it is storing Chapter in all Dir elements, which is unwanted. The wanted result is:
Dir[0] = [1, 2, 3]
Dir[1] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
...
Dir[24] =[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
OBS(3): I don't know why the result of this line is:
Dir: [[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []]
It seems that when I clear the Chapter with Chapter.clear(), it clear the contents of Dir. Why?
Then, can someone help me to fix the code, please? Thanks in advance for your attention.
You are not properly appending the chapters to the list Dir.
I would recommend to do this, first create a dictionary where the number of the chapter is the key and then set he value of each key to be and empty list. Then use a loop to parse the dictionary and append the the page correctly to the corresponding list.
dir = {}
for chapter in range(0,25):
dir[chapter]=[]
You are continuing to append the same list (Chapter) to Dir each time through. Consider the below very simple example, and what happens when we go from using the same list, vs. creating a new one on each loop iteration. >>> a = [] >>> b = [] >>> for i in range(3): ... for j in range(i): ... a.append(j) ... b.append(a) ... >>> b [[0, 0, 1], [0, 0, 1], [0, 0, 1]] >>> >>> b = [] >>> for i in range(3): ... a = [] ... for j in range(i): ... a.append(j) ... b.append(a) ... >>> b [[], [0], [0, 1]] >>>
How to use tf dataset sharding AFTER a shuffle operation, but not repeat entries
I am trying to solve the following problem:
I have 128 files that I want to break into 4 subsets. Each time around, I want the division to be different.
If I do tf.data.Dataset.list_files('glob_pattern', shuffle=False), the dataset has the right number of files. Sharding this works as expected, but each shard only ever has the same files.
I want to shard and end up with a different division of the files each go-through the data. However, if I turn shuffle=True, then each shard seems to have its own copy of the original dataset, meaning that I can see the same file multiple times before seeing all the files once.
Is there an idiomatic way of splitting these files?
Basically, I'm wondering why the original list_files dataset is able to have some files show up multiple times before all the files have been seen.
Here is some TF2.0 code to see the problem:
ds = tf.data.Dataset.from_tensor_slices([f'train_{str(i).zfill(5)}' for i in range(128)])
ds = ds.shuffle(128)
n_splits = 4
sub_datasets = [ds.shard(n_splits, i) for i in range(n_splits)]
output = []
# go through each of the subsets
for i in range(n_splits):
results = [x.numpy().decode() for x in sub_datasets[i]]
output.extend(results)
print(len(set(output)), 'is the number of unique files seen (128 desired)')
Here's an answer from what I can understand of your question. To generate a new subset of 4 datasets (shared) each time randomly shuffled, you can use the following code.
import numpy as np
import tensorflow as tf
# ####################
# I used numbers for visualization ... feel free to replace with your demo code
# ####################
# ds = tf.data.Dataset.from_tensor_slices([f'train_{str(i).zfill(5)}' for i in range(128)])
# ####################
arr = np.arange(128)
ds = tf.data.Dataset.from_tensor_slices(arr)
def get_four_datasets(original_ds, window_size=32, shuffle_size=128):
""" Every time you call this function you will get a new four datasets """
return original_ds.shuffle(shuffle_size).window(window_size)
remake_ds_1 = list()
remake_ds_2 = list()
for i, (dataset_1, dataset_2) in enumerate(zip(get_four_datasets(ds), get_four_datasets(ds))):
print(f"\n\nDATASET #1-{i+1}")
ds_subset = [value for value in dataset_1.as_numpy_iterator()]
print("\t", ds_subset)
remake_ds_1.extend(ds_subset)
print(f"\nDATASET #2-{i+1}")
ds_subset_2 = [value for value in dataset_2.as_numpy_iterator()]
print("\t", ds_subset_2)
remake_ds_2.extend(ds_subset_2)
print("\n\nCounts\n")
print("DS 1 ALL: ", len(remake_ds_1))
print("DS 1 UNIQUE: ", len(set(remake_ds_1)))
print("DS 2 ALL: ", len(remake_ds_2))
print("DS 2 UNIQUE: ", len(set(remake_ds_2)))
OUTPUT
DATASET #1-1
[96, 4, 66, 120, 42, 54, 110, 57, 67, 7, 13, 9, 69, 86, 122, 88, 10, 55, 27, 106, 77, 107, 114, 87, 59, 81, 1, 49, 118, 17, 36, 11]
DATASET #2-1
[47, 26, 122, 10, 110, 31, 86, 34, 52, 121, 36, 112, 55, 48, 50, 108, 100, 103, 113, 68, 58, 29, 32, 84, 124, 15, 38, 51, 6, 66, 24, 41]
DATASET #1-2
[56, 80, 94, 124, 52, 109, 83, 90, 112, 35, 6, 101, 20, 84, 73, 74, 100, 99, 108, 15, 14, 12, 89, 24, 8, 29, 68, 85, 125, 3, 33, 58]
DATASET #2-2
[125, 127, 74, 97, 12, 39, 109, 126, 98, 40, 99, 93, 35, 107, 91, 88, 45, 13, 106, 120, 19, 73, 83, 11, 105, 61, 16, 114, 79, 95, 94, 44]
DATASET #1-3
[105, 38, 43, 60, 0, 26, 127, 65, 22, 18, 123, 82, 121, 71, 51, 23, 113, 30, 63, 40, 2, 61, 16, 98, 64, 25, 41, 28, 45, 19, 117, 39]
DATASET #2-3
[75, 64, 1, 17, 7, 42, 80, 92, 3, 9, 54, 33, 82, 56, 118, 102, 115, 43, 28, 90, 60, 119, 0, 57, 123, 62, 22, 72, 65, 23, 30, 87]
DATASET #1-4
[48, 62, 31, 102, 111, 46, 103, 44, 116, 79, 21, 50, 53, 78, 93, 32, 95, 34, 92, 126, 104, 47, 119, 37, 5, 70, 97, 91, 76, 75, 72, 115]
DATASET #2-4
[4, 85, 21, 116, 78, 27, 117, 2, 59, 111, 69, 46, 63, 20, 49, 5, 81, 53, 18, 37, 8, 76, 71, 89, 14, 104, 25, 96, 67, 101, 77, 70]
Counts
DS 1 ALL: 128
DS 1 UNIQUE: 128
DS 2 ALL: 128
DS 2 UNIQUE: 128
If you just want to generate a dataset where every 32 examples pulled from the dataset is shuffled and you want to iterate over the dataset multiple times getting new 32-set samples every time, you can do the following.
import numpy as np
import tensorflow as tf
arr = np.arange(128)
N_REPEATS = 10
ds = tf.data.Dataset.from_tensor_slices(arr)
ds = ds.shuffle(128).batch(32).repeat(N_REPEATS)
OUTPUT
BATCH 1: [92, 94, 76, 38, 58, 9, 44, 16, 86, 28, 64, 7, 60, 42, 31, 0, 46, 1, 83, 57, 18, 102, 67, 110, 113, 101, 93, 61, 96, 17, 105, 6]
BATCH 2: [59, 15, 121, 3, 72, 100, 50, 52, 45, 23, 87, 43, 33, 29, 62, 25, 74, 65, 75, 68, 4, 56, 117, 47, 73, 109, 106, 35, 88, 91, 119, 66]
BATCH 3: [98, 78, 125, 24, 99, 51, 14, 114, 26, 22, 54, 89, 79, 63, 30, 124, 20, 13, 2, 34, 95, 41, 85, 39, 37, 77, 90, 107, 104, 118, 27, 97]
BATCH 4: [49, 5, 53, 115, 126, 40, 108, 48, 8, 84, 120, 32, 82, 11, 112, 55, 80, 69, 12, 70, 111, 123, 81, 116, 71, 122, 36, 21, 103, 19, 127, 10]
BATCH 5: [74, 61, 97, 6, 127, 119, 65, 15, 78, 72, 99, 18, 41, 76, 79, 33, 0, 105, 103, 46, 14, 50, 113, 26, 43, 45, 100, 90, 28, 48, 19, 9]
BATCH 6: [35, 20, 3, 64, 5, 96, 114, 34, 126, 85, 124, 69, 110, 54, 109, 24, 104, 32, 73, 92, 11, 13, 58, 107, 84, 88, 59, 75, 95, 40, 16, 101]
BATCH 7: [93, 66, 106, 44, 102, 125, 7, 30, 12, 116, 87, 111, 81, 56, 83, 37, 31, 77, 67, 21, 118, 1, 120, 36, 86, 62, 71, 98, 82, 52, 25, 27]
BATCH 8: [112, 68, 60, 70, 115, 117, 29, 91, 57, 10, 121, 89, 4, 2, 122, 39, 51, 22, 53, 63, 108, 94, 42, 17, 8, 23, 80, 38, 55, 49, 47, 123]
BATCH 9: [67, 20, 101, 123, 109, 4, 39, 65, 34, 71, 22, 62, 73, 81, 114, 112, 66, 35, 43, 49, 92, 68, 1, 54, 27, 103, 46, 12, 82, 6, 119, 99]
BATCH 10: [86, 69, 13, 44, 16, 50, 75, 61, 58, 104, 64, 47, 95, 10, 79, 70, 97, 63, 45, 17, 56, 74, 87, 53, 91, 21, 48, 76, 9, 51, 28, 126]
...
...
...
BATCH 40: [10, 41, 29, 39, 57, 127, 101, 106, 55, 62, 72, 76, 124, 81, 66, 126, 53, 24, 33, 49, 102, 75, 34, 61, 47, 15, 21, 121, 8, 94, 52, 13]
Please let me know if I misunderstood and I can update accordingly.
You need to set reshuffle_each_iteration=False:
ds = ds.shuffle(128, reshuffle_each_iteration=False)
Full code:
import tensorflow as tf
ds = tf.data.Dataset.from_tensor_slices([f'train_{str(i).zfill(5)}' for i in range(128)])
ds = ds.shuffle(128, reshuffle_each_iteration=False)
n_splits = 4
sub_datasets = [ds.shard(n_splits, i) for i in range(n_splits)]
output = []
# go through each of the subsets
for i in range(n_splits):
results = [x.numpy().decode() for x in sub_datasets[i]]
output.extend(results)
print(len(set(output)), 'is the number of unique files seen (128 desired)')
128 is the number of unique files seen (128 desired)
Numpy blocks reshaping
I am looking for a way to reshape the following 1d-numpy array: # dimensions n = 2 # int : 1 ... N h = 2 # int : 1 ... N m = n*(2*h+1) input_data = np.arange(0,(n*(2*h+1))**2) The expected output should be reshaped into (2*h+1)**2 blocks of shape (n,n) such as: input_data.reshape(((2*h+1)**2,n,n)) >>> array([[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] ... [[92 93] [94 95]] [[96 97] [98 99]]] These blocks finally need to be reshaped into a (m,m) matrix so that they are stacked in rows of 2*h+1 blocks: >>> array([[ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17], [ 2, 3, 6, 7, 10, 11, 14, 15, 18, 19], ... [80, 81, 84, 85, 88, 89, 92, 93, 96, 97], [82, 83, 86, 87, 90, 91, 94, 95, 98, 99]]) My problem is that I can't seem to find proper axis permutations after the first reshape into (n,n) blocks. I have looked at several answers such as this one but in vain. As the real dimensions n and h are quite bigger and this operation takes place in an iterative process, I am looking for an efficient reshaping operation.
I don't think you can do this with reshape and transpose alone (although I'd love to be proven wrong). Using np.block works, but it's a bit messy: np.block([list(i) for i in input_data.reshape( (2*h+1), (2*h+1), n, n )]) array([[ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17], [ 2, 3, 6, 7, 10, 11, 14, 15, 18, 19], [20, 21, 24, 25, 28, 29, 32, 33, 36, 37], [22, 23, 26, 27, 30, 31, 34, 35, 38, 39], [40, 41, 44, 45, 48, 49, 52, 53, 56, 57], [42, 43, 46, 47, 50, 51, 54, 55, 58, 59], [60, 61, 64, 65, 68, 69, 72, 73, 76, 77], [62, 63, 66, 67, 70, 71, 74, 75, 78, 79], [80, 81, 84, 85, 88, 89, 92, 93, 96, 97], [82, 83, 86, 87, 90, 91, 94, 95, 98, 99]]) EDIT: Never mind, you can do without np.block: input_data.reshape( (2*h+1), (2*h+1), n, n).transpose(0, 2, 1, 3).reshape(10, 10) array([[ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17], [ 2, 3, 6, 7, 10, 11, 14, 15, 18, 19], [20, 21, 24, 25, 28, 29, 32, 33, 36, 37], [22, 23, 26, 27, 30, 31, 34, 35, 38, 39], [40, 41, 44, 45, 48, 49, 52, 53, 56, 57], [42, 43, 46, 47, 50, 51, 54, 55, 58, 59], [60, 61, 64, 65, 68, 69, 72, 73, 76, 77], [62, 63, 66, 67, 70, 71, 74, 75, 78, 79], [80, 81, 84, 85, 88, 89, 92, 93, 96, 97], [82, 83, 86, 87, 90, 91, 94, 95, 98, 99]])
numpy.correlate vs numpy documentation - is there a contradiction here ? Why is the resulting list reversed ?
I get the following result using numpy's correlate function: In [153]: np.correlate([1],np.arange(100)) Out[153]: array([99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]) In [154]: This result seems to be in contradiction with the 90. page of the numpy book : Based on the formula above I would have expected an increasing array 0..99, however, the result is a decreasing array 99..0. Can someone explain what is going on here ? Why does the implementation contradicts the specification ? Why does it make sense to reverse the list ?
Looks like you are expecting the old_behaviour of numpy.correlate. The book you link to is very old (2006), so it looks like numpy.correlate has changed since it was written (it actually changed in numpy v1.4). From the docs for numpy v1.9: old_behavior : bool If True, uses the old behavior from Numeric, (correlate(a,v) == correlate(v,a), and the conjugate is not taken for complex arrays). If False, uses the conventional signal processing definition. In [2]: np.correlate([1],np.arange(100)) Out[2]: array([99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]) In [3]: np.correlate([1],np.arange(100),old_behavior=True) Out[3]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]) In [4]: np.correlate(np.arange(100),[1]) Out[4]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]) EDIT On further inspection, I think the difference is due to this line in the old definition: K=len(x)-1 and M=len(y)-1, and we assume K ≥ M (without loss of generality because we can interchange the roles of x and y without effect). So, I believe for your case, in the old definition, it is making y=[1] and x=np.arange(100), because len(x) must be greater than len(y). The new definition does not do that, instead "input arrays are never swapped", so x=[1] and y=np.arange(100). Thus, the differences.