We Keep Coding

Add list as list element Python weird behaviour

I am trying to store the file indexes from a directory in a list variable using Python. This list will be used for a recursive function that will change the content of files, but the list is storing incorrect values.
My directory has more than 5k files with the format: ccc-ppp.htm where ccc is the chapter number (from 0 to 24) and ppp is page number, that varies from chapter to chapter. For example:
0-1.htm
0-2.htm
0-3.htm
1-1.htm
1-2.htm
...
1-10.htm
2-1.htm
...
2.158.htm
...
24-100.htm
Some chapters have more than 500 pages.
My intention is to store in a list of lists:
Dir = [[1, 2, 3], [1, 2, 3, ..., 10], [1, 2, ..., 158], ..., [0, 1, 2, ..., 100]]
(Note that some chapters start with ...-0.htm
My code:
import os.path
Folder = "C:\MyFoler\\"
Ch = 0
Pg = 0
Chapter = []
Dir = []
while Ch <= 24:
File = Folder + str(Ch) + "-" + str(Pg) + ".htm"
if os.path.exists(File):
Chapter.append(Pg)
else:
print(f"\nChapter {Ch}: {Chapter}") # OBS(1)
Dir.append(Chapter)
print(f"\nDir: {Dir}") # OBS(2)
Chapter.clear()
Ch += 1
Pg = -1
print(f"\nDir: {Dir}") # OBS(3)
Pg += 1
OBS(1): It prints the list of pages in a chapter (working OK). For example to Chapter 24:
Chapter 24: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
OBS(2): It prints a list, with Ch elements, and each element is the Chapter, but it repeats the chapter in all list elements. For example to Ch = 24:
Dir: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]]
The problem is that it is storing Chapter in all Dir elements, which is unwanted. The wanted result is:
Dir[0] = [1, 2, 3]
Dir[1] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
...
Dir[24] =[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
OBS(3): I don't know why the result of this line is:
Dir: [[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []]
It seems that when I clear the Chapter with Chapter.clear(), it clear the contents of Dir. Why?
Then, can someone help me to fix the code, please? Thanks in advance for your attention.

You are not properly appending the chapters to the list Dir.
I would recommend to do this, first create a dictionary where the number of the chapter is the key and then set he value of each key to be and empty list. Then use a loop to parse the dictionary and append the the page correctly to the corresponding list.
dir = {}
for chapter in range(0,25):
dir[chapter]=[]

You are continuing to append the same list (Chapter) to Dir each time through. Consider the below very simple example, and what happens when we go from using the same list, vs. creating a new one on each loop iteration.
>>> a = []
>>> b = []
>>> for i in range(3):
... for j in range(i):
... a.append(j)
... b.append(a)
...
>>> b
[[0, 0, 1], [0, 0, 1], [0, 0, 1]]
>>>
>>> b = []
>>> for i in range(3):
... a = []
... for j in range(i):
... a.append(j)
... b.append(a)
...
>>> b
[[], [0], [0, 1]]
>>>

Python, delete subinterval from an interval

In python, I have an interval
a = 1-100
And I have a list with intervals
b = [11-20,41-50,91-110]
Now I would like my output to be like this in python:
a-b = 1-10,21-40,51-90

You have this :
a=range(1,100)
b = [range(11,20),range(41,50),range(91,110)] #list of ranges
So you can try :
d=[]
i=a.start
for sub_range in b: #for each sub range
#get the start and stop
start=sub_range.start
end=sub_range.stop
#append range to list
d.append(range(i,start-1))
i=end+1
print(d)
[range(1, 10), range(21, 40), range(51, 90)]

You can try this
a = range(1,101)
set(a)
b = [11,12,13,14,15,16,17,18,19,20,41,42,43,44,45,46,47,48,49,50,91,92,93,94,95,96,97,98,99,100]
set(a)-set(b)
{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90}
You can use more_itertoools to divide the list
import more_itertools as mit
iterable = list(d)
[list(group) for group in mit.consecutive_groups(iterable)]
Result
[[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40], [51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90]]

Why does my non-normal sample pass the normality tests? [duplicate]

This question already has answers here:
What does "e" in "1e-5" in Python language mean and what is the name of this notation? [duplicate]
(3 answers)
What is the meaning of number 1e5?
(5 answers)
Closed 4 years ago.
I am running non-normal samples against normality tests, expecting it to fail. However, the p-value {normaltest: 2.64, shapiro: 6.23} is much higher than 0.05 so I cannnot reject the null hypotheses that this was drawn from a normal distribution. Could someone please enlighten me?
import random
from scipy.stats import shapiro
from scipy import stats
x = list()
for y in range(1000):
x.append(random.randrange(1,100))
print(stats.normaltest(x))
print(shapiro(x))
print(x)
NormaltestResult(statistic=500.02063130202464, pvalue=2.641797311523516e-109)
(0.9599024057388306, 6.233162463518298e-16)
[25, 94, 79, 12, 67, 27, 89, 94, 89, 34, 99, 58, 53, 29, 81, 94, 4, 52, 14, 19, 60, 26, 6, 85, 71, 57, 23, 13, 58, 75, 75, 10, 51, 12, 80, 3, 82, 64, 74, 57, 83, 15, 23, 45, 52, 51, 36, 61, 78, 50, 26, 72, 40, 19, 59, 90, 23, 71, 52, 25, 3, 16, 20, 62, 50, 56, 60, 73, 28, 96, 69, 80, 36, 11, 11, 7, 18, 1, 73, 17, 29, 57, 72, 87, 43, 18, 22, 54, 32, 35, 79, 27, 45, 81, 80, 79, 82, 49, 77, 73, 21, 17, 90, 96, 2, 72, 7, 43, 37, 72, 64, 53, 63, 5, 36, 74, 36, 59, 53, 55, 54, 15, 83, 65, 2, 64, 46, 51, 31, 1, 77, 28, 47, 75, 46, 56, 3, 16, 24, 27, 31, 66, 4, 61, 46, 2, 56, 59, 98, 86, 83, 86, 97, 59, 45, 80, 55, 23, 21, 61, 6, 20, 13, 54, 20, 58, 86, 38, 18, 47, 68, 52, 74, 19, 34, 56, 17, 91, 15, 54, 82, 95, 23, 54, 42, 81, 82, 6, 70, 1, 78, 49, 12, 25, 33, 38, 47, 41, 68, 75, 73, 76, 46, 7, 90, 89, 63, 43, 41, 46, 88, 14, 97, 37, 92, 76, 60, 7, 5, 56, 77, 98, 61, 60, 59, 64, 4, 76, 34, 84, 78, 39, 66, 24, 49, 60, 57, 13, 57, 18, 37, 52, 26, 36, 97, 47, 95, 26, 82, 82, 10, 76, 54, 67, 98, 22, 56, 20, 34, 76, 28, 50, 70, 87, 83, 13, 76, 87, 98, 29, 99, 29, 23, 74, 5, 54, 73, 31, 89, 10, 24, 15, 9, 34, 85, 23, 6, 25, 64, 94, 37, 30, 11, 9, 58, 43, 2, 1, 73, 49, 48, 41, 99, 30, 91, 17, 31, 58, 70, 46, 20, 33, 94, 35, 41, 19, 22, 2, 37, 8, 54, 41, 21, 16, 20, 65, 27, 68, 24, 19, 36, 63, 80, 12, 82, 74, 74, 46, 7, 36, 42, 72, 16, 26, 96, 48, 75, 86, 62, 20, 79, 66, 71, 43, 43, 96, 67, 97, 76, 40, 80, 97, 75, 2, 3, 97, 37, 78, 77, 25, 84, 82, 25, 87, 44, 80, 92, 95, 99, 48, 67, 12, 82, 3, 15, 40, 45, 94, 32, 87, 92, 24, 42, 73, 66, 20, 62, 9, 75, 51, 31, 3, 13, 76, 21, 32, 14, 58, 28, 14, 99, 14, 50, 45, 13, 83, 45, 59, 63, 39, 65, 78, 46, 96, 27, 16, 69, 42, 65, 68, 68, 90, 39, 50, 86, 7, 75, 93, 84, 23, 53, 31, 23, 63, 32, 60, 85, 67, 42, 5, 72, 44, 43, 98, 75, 55, 28, 99, 71, 72, 66, 46, 61, 52, 50, 16, 44, 63, 64, 32, 59, 73, 33, 36, 32, 45, 75, 44, 36, 74, 97, 2, 38, 30, 74, 12, 57, 11, 37, 83, 64, 3, 63, 3, 35, 61, 55, 59, 99, 51, 58, 63, 70, 84, 4, 18, 13, 51, 27, 75, 43, 63, 35, 76, 67, 32, 15, 54, 51, 31, 77, 97, 83, 50, 76, 87, 26, 55, 93, 31, 70, 5, 11, 54, 48, 55, 48, 76, 90, 75, 43, 34, 6, 22, 35, 29, 4, 47, 83, 44, 7, 2, 97, 74, 90, 91, 17, 12, 33, 52, 28, 95, 57, 22, 53, 83, 56, 71, 28, 76, 55, 12, 89, 27, 20, 20, 93, 43, 65, 34, 83, 92, 11, 22, 38, 90, 83, 77, 11, 5, 22, 73, 29, 67, 49, 16, 47, 60, 26, 20, 76, 57, 46, 70, 35, 9, 28, 33, 8, 33, 21, 65, 3, 67, 52, 45, 82, 32, 94, 89, 15, 27, 63, 53, 96, 4, 74, 29, 59, 67, 22, 24, 45, 63, 76, 66, 51, 28, 42, 83, 37, 56, 83, 14, 35, 99, 48, 93, 83, 76, 2, 20, 99, 41, 43, 61, 3, 13, 7, 74, 60, 17, 84, 16, 44, 76, 63, 85, 44, 27, 38, 29, 61, 34, 55, 91, 13, 31, 42, 35, 38, 6, 46, 31, 99, 85, 23, 67, 11, 4, 52, 57, 11, 9, 21, 64, 17, 46, 78, 83, 45, 68, 98, 88, 47, 1, 94, 24, 79, 47, 33, 7, 81, 12, 26, 99, 80, 78, 53, 88, 1, 49, 17, 91, 27, 44, 31, 20, 6, 46, 59, 40, 57, 80, 3, 72, 83, 81, 2, 27, 36, 94, 31, 30, 22, 26, 31, 14, 93, 11, 32, 14, 75, 17, 49, 54, 42, 56, 76, 42, 51, 69, 22, 86, 46, 97, 70, 24, 81, 3, 75, 81, 63, 48, 51, 72, 19, 16, 16, 1, 61, 95, 53, 36, 82, 93, 53, 65, 9, 40, 91, 41, 85, 65, 38, 59, 4, 92, 50, 51, 7, 87, 80, 45, 84, 57, 21, 44, 3, 52, 7, 53, 97, 46, 65, 37, 76, 7, 81, 49, 21, 25, 18, 84, 53, 84, 89, 5, 95, 69, 70, 56, 31, 69, 12, 72, 36, 12, 44, 94, 39, 97, 91, 92, 15, 17, 57, 17, 49, 47, 1, 8, 2, 93, 91, 29, 41, 12, 46, 75, 98, 14, 34, 6, 26, 6, 81, 75, 49, 61, 70, 83, 26, 83, 38, 40, 81, 27, 14, 40, 54, 35, 10, 22, 30, 38, 2, 95, 31, 32, 25, 88, 70, 33, 85, 52, 7, 47, 4, 87, 70, 90, 15, 53, 74, 45, 76, 50, 23, 54, 33, 90, 53, 70, 4, 6, 47, 77, 87, 60, 92, 94, 79, 36, 26, 20, 94, 59, 50, 72, 31, 25, 60, 24, 82, 93, 13, 57, 21, 72, 78, 27, 62, 67, 67, 62, 56, 40, 40, 49, 31, 52, 16, 67, 51, 87, 77, 1, 47, 49, 64, 48, 62, 90, 29, 68, 2, 39, 92, 89, 92, 34, 78, 48, 36, 32, 43, 26, 81, 22, 76, 95, 69, 79, 13, 77, 26, 70, 32, 21, 92, 97, 90, 98, 16, 81, 47, 83, 93, 50]

Calculate three vectors Pearson Correlation

I want to calculate the Pearson Correlation of three vectors. Like:
np.corrcoef(X,Y,Z)
Anyone knows how to do that in Python? Any library available?
Big thanks!

Create a 2-D array:
import numpy as np
X = np.random.randint(0,100,100)
Y = np.random.randint(0,100,100)
Z = np.random.randint(0,100,100)
spam = np.array([X,Y,Z])
Then run np.corrcoeff on that:
eggs = np.corrcoef(spam)
Which gives
spam
array([[12, 1, 23, 95, 58, 37, 38, 76, 99, 24, 50, 47, 29, 67, 38, 62,
28, 39, 26, 32, 55, 11, 67, 71, 71, 50, 9, 54, 59, 22, 70, 63,
11, 43, 30, 54, 14, 11, 89, 68, 98, 27, 18, 66, 14, 23, 83, 81,
24, 90, 56, 40, 3, 94, 86, 54, 66, 68, 96, 74, 46, 19, 58, 74,
7, 73, 93, 91, 10, 75, 32, 91, 45, 57, 81, 96, 20, 25, 18, 86,
59, 58, 94, 97, 10, 11, 6, 30, 28, 76, 56, 86, 64, 38, 26, 25,
26, 77, 76, 0],
[76, 54, 66, 16, 97, 22, 66, 59, 90, 67, 85, 58, 49, 13, 36, 44,
91, 20, 63, 2, 1, 51, 66, 12, 32, 10, 48, 88, 6, 96, 68, 60,
22, 24, 51, 2, 94, 95, 98, 79, 24, 22, 94, 69, 36, 37, 50, 94,
8, 2, 15, 80, 99, 77, 15, 95, 76, 25, 0, 69, 46, 21, 47, 87,
88, 6, 34, 63, 46, 12, 62, 4, 36, 89, 21, 44, 96, 22, 11, 14,
63, 55, 70, 58, 33, 93, 88, 39, 0, 18, 9, 92, 37, 13, 12, 67,
98, 34, 9, 6],
[56, 21, 61, 55, 82, 60, 13, 36, 17, 52, 93, 88, 65, 29, 67, 81,
65, 59, 33, 89, 58, 99, 78, 1, 75, 39, 10, 75, 94, 57, 42, 32,
19, 7, 77, 82, 96, 61, 94, 50, 61, 13, 21, 51, 70, 14, 60, 79,
32, 29, 22, 93, 31, 19, 74, 37, 85, 74, 50, 50, 37, 5, 33, 39,
75, 32, 43, 10, 74, 39, 74, 26, 57, 38, 79, 90, 63, 13, 53, 57,
44, 4, 13, 39, 14, 97, 21, 81, 2, 49, 72, 5, 72, 32, 63, 37,
53, 31, 8, 12]])
eggs
array([[ 1. , -0.14396307, -0.01107133],
[-0.14396307, 1. , 0.17098877],
[-0.01107133, 0.17098877, 1. ]])

numpy.correlate vs numpy documentation - is there a contradiction here ? Why is the resulting list reversed ?

I get the following result using numpy's correlate function:
In [153]: np.correlate([1],np.arange(100))
Out[153]:
array([99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83,
82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66,
65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49,
48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36, 35, 34, 33, 32,
31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15,
14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0])
In [154]:
This result seems to be in contradiction with the 90. page of the numpy book :
Based on the formula above I would have expected an increasing array 0..99, however, the result is a decreasing array 99..0.
Can someone explain what is going on here ?
Why does the implementation contradicts the specification ?
Why does it make sense to reverse the list ?

Looks like you are expecting the old_behaviour of numpy.correlate. The book you link to is very old (2006), so it looks like numpy.correlate has changed since it was written (it actually changed in numpy v1.4). From the docs for numpy v1.9:
old_behavior : bool
If True, uses the old behavior from Numeric, (correlate(a,v) == correlate(v,a), and the conjugate is not taken for complex arrays). If False, uses the conventional signal processing definition.
In [2]: np.correlate([1],np.arange(100))
Out[2]:
array([99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83,
82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66,
65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49,
48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36, 35, 34, 33, 32,
31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15,
14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0])
In [3]: np.correlate([1],np.arange(100),old_behavior=True)
Out[3]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,
85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99])
In [4]: np.correlate(np.arange(100),[1])
Out[4]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,
85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99])
EDIT
On further inspection, I think the difference is due to this line in the old definition:
K=len(x)-1 and M=len(y)-1, and we assume K ≥ M (without loss of generality because we can interchange the roles of x and y without effect).
So, I believe for your case, in the old definition, it is making y=[1] and x=np.arange(100), because len(x) must be greater than len(y). The new definition does not do that, instead "input arrays are never swapped", so x=[1] and y=np.arange(100). Thus, the differences.