prints elements from a list with 10 digits per line
from this
n = [85, 13, 99, 34, 71, 15, 82, 24, 64, 61,
67, 99, 50, 68, 25, 37, 32, 27, 14, 91,
79, 15, 47, 48, 74, 88, 64, 53, 77, 50,
24, 91, 87, 55, 60, 75, 91, 22, 47, 63,
81, 88, 26, 48, 69, 59, 84, 77, 28, 36,
59, 74, 89, 73, 91, 64, 55, 88, 90, 48,
73, 97, 98, 40, 93, 50, 78, 60, 44, 77,
82, 51, 53, 65, 98, 59, 94, 91, 52, 44,
65, 85, 72, 92, 49, 67, 58, 48, 62, 54,
89, 67, 58, 48, 85, 45, 77, 76, 81, 77]
to this (without parenthesis)
value n :
85, 13, 99, 34, 71, 15, 82, 24, 64, 61,
67, 99, 50, 68, 25, 37, 32, 27, 14, 91,
79, 15, 47, 48, 74, 88, 64, 53, 77, 50,
24, 91, 87, 55, 60, 75, 91, 22, 47, 63,
81, 88, 26, 48, 69, 59, 84, 77, 28, 36,
59, 74, 89, 73, 91, 64, 55, 88, 90, 48,
73, 97, 98, 40, 93, 50, 78, 60, 44, 77,
82, 51, 53, 65, 98, 59, 94, 91, 52, 44,
65, 85, 72, 92, 49, 67, 58, 48, 62, 54,
89, 67, 58, 48, 85, 45, 77, 76, 81, 77
You can try this:
n = [85, 13, 99, 34, 71, 15, 82, 24, 64, 61,
67, 99, 50, 68, 25, 37, 32, 27, 14, 91,
79, 15, 47, 48, 74, 88, 64, 53, 77, 50,
24, 91, 87, 55, 60, 75, 91, 22, 47, 63,
81, 88, 26, 48, 69, 59, 84, 77, 28, 36,
59, 74, 89, 73, 91, 64, 55, 88, 90, 48,
73, 97, 98, 40, 93, 50, 78, 60, 44, 77,
82, 51, 53, 65, 98, 59, 94, 91, 52, 44,
65, 85, 72, 92, 49, 67, 58, 48, 62, 54,
89, 67, 58, 48, 85, 45, 77, 76, 81, 77]
print("value n :")
for x in range(10):
print(', '.join([str(num) for num in (n[x*10:x*10+10])]))
Output:
value n :
85, 13, 99, 34, 71, 15, 82, 24, 64, 61
67, 99, 50, 68, 25, 37, 32, 27, 14, 91
79, 15, 47, 48, 74, 88, 64, 53, 77, 50
24, 91, 87, 55, 60, 75, 91, 22, 47, 63
81, 88, 26, 48, 69, 59, 84, 77, 28, 36
59, 74, 89, 73, 91, 64, 55, 88, 90, 48
73, 97, 98, 40, 93, 50, 78, 60, 44, 77
82, 51, 53, 65, 98, 59, 94, 91, 52, 44
65, 85, 72, 92, 49, 67, 58, 48, 62, 54
89, 67, 58, 48, 85, 45, 77, 76, 81, 77
y = [", ".join([str(x) for x in n[10*(i-1):10*i]]) for i in range(1, 11)]
print("\n".join(y))
I first splitter the original list into 10 sublists, turned each one to a string, and then concat those strings.
You can just you simply to get desired output using separator in print
print('value n :')
for i in range(0, len(n), 10):
print(*n[i:i+10], sep = ', ')
Explanation
range(0, len(n), 10) creates starting index on each row
n[i:i+1] is the array of 10 values for each row
*n[i:i+1] is the unpacking operator that turns list n[i:i+1 into positional arguments for print (i.e. equivalent to print(n[i], n[i+1], ...n[i+10])
sep = ', ' causes each positional argument to be printed with a comma separator
Input:
n = [85, 13, 99, 34, 71, 15, 82, 24, 64, 61,
67, 99, 50, 68, 25, 37, 32, 27, 14, 91,
79, 15, 47, 48, 74, 88, 64, 53, 77, 50,
24, 91, 87, 55, 60, 75, 91, 22, 47, 63,
81, 88, 26, 48, 69, 59, 84, 77, 28, 36,
59, 74, 89, 73, 91, 64, 55, 88, 90, 48,
73, 97, 98, 40, 93, 50, 78, 60, 44, 77,
82, 51, 53, 65, 98, 59, 94, 91, 52, 44,
65, 85, 72, 92, 49, 67, 58, 48, 62, 54,
89, 67, 58, 48, 85, 45, 77, 76, 81, 77]
Code:
print('value n:')
for i in range(0, n.__len__(), 10):
print(*n[i:i+10], sep=', ',end='\n')
Output:
value n:
85, 13, 99, 34, 71, 15, 82, 24, 64, 61
67, 99, 50, 68, 25, 37, 32, 27, 14, 91
79, 15, 47, 48, 74, 88, 64, 53, 77, 50
24, 91, 87, 55, 60, 75, 91, 22, 47, 63
81, 88, 26, 48, 69, 59, 84, 77, 28, 36
59, 74, 89, 73, 91, 64, 55, 88, 90, 48
73, 97, 98, 40, 93, 50, 78, 60, 44, 77
82, 51, 53, 65, 98, 59, 94, 91, 52, 44
65, 85, 72, 92, 49, 67, 58, 48, 62, 54
89, 67, 58, 48, 85, 45, 77, 76, 81, 77
Related
I am trying to send random data to API using python flask with intervals of 1 second. But it only shows the last array of data. I am using the following code:
import time
import random
import datetime
from flask import Flask
mylist = []
ct = datetime.datetime.now()
app = Flask(__name__)
#app.route('/')
def index():
mylist = []
ct = datetime.datetime.now()
for i in range(0, 61):
x = random.randint(1, 100)
mylist.append(x)
if len(mylist) == 11:
right_in_left_out = mylist.pop(0)
else:
right_in_left_out = None
time.sleep(1)
print(mylist)
return mylist
if __name__ == "__main__":
app.run(debug=True)
OUTPUT:
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://127.0.0.1:5000
Press CTRL+C to quit
* Restarting with stat
* Debugger is active!
* Debugger PIN: 516-689-025
[50]
[50, 61]
[50, 61, 47]
[50, 61, 47, 63]
[50, 61, 47, 63, 24]
[50, 61, 47, 63, 24, 92]
[50, 61, 47, 63, 24, 92, 18]
[50, 61, 47, 63, 24, 92, 18, 75]
[50, 61, 47, 63, 24, 92, 18, 75, 95]
[50, 61, 47, 63, 24, 92, 18, 75, 95, 4]
[61, 47, 63, 24, 92, 18, 75, 95, 4, 40]
[47, 63, 24, 92, 18, 75, 95, 4, 40, 88]
[63, 24, 92, 18, 75, 95, 4, 40, 88, 39]
[24, 92, 18, 75, 95, 4, 40, 88, 39, 47]
[92, 18, 75, 95, 4, 40, 88, 39, 47, 58]
[18, 75, 95, 4, 40, 88, 39, 47, 58, 82]
[75, 95, 4, 40, 88, 39, 47, 58, 82, 88]
[95, 4, 40, 88, 39, 47, 58, 82, 88, 7]
[4, 40, 88, 39, 47, 58, 82, 88, 7, 90]
[40, 88, 39, 47, 58, 82, 88, 7, 90, 65]
[88, 39, 47, 58, 82, 88, 7, 90, 65, 93]
[39, 47, 58, 82, 88, 7, 90, 65, 93, 9]
[47, 58, 82, 88, 7, 90, 65, 93, 9, 55]
[58, 82, 88, 7, 90, 65, 93, 9, 55, 48]
[82, 88, 7, 90, 65, 93, 9, 55, 48, 83]
[88, 7, 90, 65, 93, 9, 55, 48, 83, 96]
[7, 90, 65, 93, 9, 55, 48, 83, 96, 63]
[90, 65, 93, 9, 55, 48, 83, 96, 63, 8]
[65, 93, 9, 55, 48, 83, 96, 63, 8, 43]
[93, 9, 55, 48, 83, 96, 63, 8, 43, 49]
[9, 55, 48, 83, 96, 63, 8, 43, 49, 95]
[55, 48, 83, 96, 63, 8, 43, 49, 95, 92]
[48, 83, 96, 63, 8, 43, 49, 95, 92, 43]
[83, 96, 63, 8, 43, 49, 95, 92, 43, 57]
[96, 63, 8, 43, 49, 95, 92, 43, 57, 91]
[63, 8, 43, 49, 95, 92, 43, 57, 91, 61]
[8, 43, 49, 95, 92, 43, 57, 91, 61, 27]
[43, 49, 95, 92, 43, 57, 91, 61, 27, 66]
[49, 95, 92, 43, 57, 91, 61, 27, 66, 70]
[95, 92, 43, 57, 91, 61, 27, 66, 70, 4]
[92, 43, 57, 91, 61, 27, 66, 70, 4, 34]
[43, 57, 91, 61, 27, 66, 70, 4, 34, 11]
[57, 91, 61, 27, 66, 70, 4, 34, 11, 95]
[91, 61, 27, 66, 70, 4, 34, 11, 95, 71]
[61, 27, 66, 70, 4, 34, 11, 95, 71, 35]
[27, 66, 70, 4, 34, 11, 95, 71, 35, 4]
[66, 70, 4, 34, 11, 95, 71, 35, 4, 98]
[70, 4, 34, 11, 95, 71, 35, 4, 98, 18]
[4, 34, 11, 95, 71, 35, 4, 98, 18, 81]
[34, 11, 95, 71, 35, 4, 98, 18, 81, 87]
[11, 95, 71, 35, 4, 98, 18, 81, 87, 84]
[95, 71, 35, 4, 98, 18, 81, 87, 84, 37]
[71, 35, 4, 98, 18, 81, 87, 84, 37, 63]
[35, 4, 98, 18, 81, 87, 84, 37, 63, 42]
[4, 98, 18, 81, 87, 84, 37, 63, 42, 18]
[98, 18, 81, 87, 84, 37, 63, 42, 18, 79]
[18, 81, 87, 84, 37, 63, 42, 18, 79, 28]
[81, 87, 84, 37, 63, 42, 18, 79, 28, 12]
[87, 84, 37, 63, 42, 18, 79, 28, 12, 36]
[84, 37, 63, 42, 18, 79, 28, 12, 36, 23]
[37, 63, 42, 18, 79, 28, 12, 36, 23, 49]
127.0.0.1 - - [21/Sep/2022 12:55:46] "GET / HTTP/1.1" 200 -
OUTPUT AT API:
I am looking to send data the same way as it is being displayed in the IDE with 1 sec intervals.
The problem lies here
if len(mylist) == 11:
right_in_left_out = mylist.pop(0)
Once this code executes for the first time, your list size is back to 10 , further iterations everytime it becomes 11 and then back to 10!
Your code is returning a list, and that is logical. If you want to return all the lists like the ones you displayed, you have to store them in a list of lists.
I mean by that:
all_lists = []
mylist = []
ct = datetime.datetime.now()
for i in range(0, 61):
x = random.randint(1, 100)
mylist.append(x)
if len(mylist) == 11:
mylist.pop(0)
time.sleep(1)
all_lists.append(mylist)
print(mylist)
return all_lists
There is also no need to use right_in_left_out variable in your code.
I am trying to automate full-width half max in python. I have a data frame with 6 lines (signals). I can do it for one signal at a time but I want to make it for all signals in the data frame. Here is how my code looks like so far:
First I import and plot the data frame just to get an idea of how everything looks, as such:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
df = pd.read_csv("Values_S250_3_level_YS.csv")
plt.plot(df)
Then I make a list to print all the values from the data frame into it as follows:
listb = [] # to print all the values of dataframe in a list form
for name, values in df.iteritems():
print('{name}: {value}'.format(name=name, value=values))
listb.append(values)
So far everything is good. Next, I want to find the values in listb that is greater than half of the maximum,
xss = [] # to get a list of
for x in range(len(listb)):
for a in range(len(listb[x])):
if listb[x][a] > max(listb[x])/2.0:
xss.append(a)
print(xss)
Now this works ok, but the problem that I have is that all the results from all 6 lines are appended in one list (xss), as such
[32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92]
How can I make 6 different lists in my for loop?
Once I have that, I will be able to find min(xss) and max(xss) to find the difference, which corresponds to full-width half max of each line.
Here is the link to the csv file if you want to try:
https://drive.google.com/file/d/1Doxzr4I7-baQptbBBct-FFQOMsMmy9In/view?usp=sharing
Here is how plot looks like
If I understood it correctly, with a minimum change to your code:
xss = []
for x in range(len(listb)):
xss_current = []
for a in range(len(listb[x])):
if listb[x][a] > max(listb[x])/2.0:
xss_current.append(a)
xss.append(xss_current)
Now xss contains the 6 lists.
I'm a beginner in python , and I am doing the binary search task, I'm trying a different code than what it is common. My issue is that I can't return a new binary list. In the first time the function is working as it suppose to be, but for the second time the function isn't return the new list.
My Code:
import random, math
user_choice=random.randint(0,100)
print (user_choice)
max=100
number_elements=50
number_list=random.sample(range(max), number_elements)
sort_list=sorted(number_list)
print (sort_list)
count=0
limit=int(math.sqrt(number_elements))
# divide by 2 the length of the number_list
def divide_list(sort_list):
global count,number_elements
number_elements=int(number_elements//2)
count += 1
half=len(sort_list)//2
if user_choice <= sort_list[number_elements] :
sort_list=sort_list[:half]
print(sort_list)
else :
sort_list=sort_list[half:]
print(sort_list)
return sort_list
while len(sort_list)==0 or count <=limit:
max /= 2
divide_list(sort_list)
Output :
99
[5, 6, 8, 9, 14, 15, 17, 18, 19, 22, 23, 24, 26, 27, 28, 34, 35, 36, 38, 39, 40, 41, 44, 46, 47, 48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99],
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
[48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 63, 64, 65, 67, 68, 69, 75, 76, 80, 81, 86, 90, 96, 97, 99]
Thanks for your help.
Your function is returning the list but you are not storing/updating it.
Solution:
Change the line divide_list(sort_list) to sort_list = divide_list(sort_list).
Explanation:
When you are updating the variable sort_list inside the function before printing (the line --> sort_list=sort_list[half:]), it is updating the variable sort_list associated to that function call and not the global variable sort_list. So by storing the return value in the global variable (as given in my solution), your list gets updated and passes the updated list next time the function is called.
I am trying to solve the following problem:
I have 128 files that I want to break into 4 subsets. Each time around, I want the division to be different.
If I do tf.data.Dataset.list_files('glob_pattern', shuffle=False), the dataset has the right number of files. Sharding this works as expected, but each shard only ever has the same files.
I want to shard and end up with a different division of the files each go-through the data. However, if I turn shuffle=True, then each shard seems to have its own copy of the original dataset, meaning that I can see the same file multiple times before seeing all the files once.
Is there an idiomatic way of splitting these files?
Basically, I'm wondering why the original list_files dataset is able to have some files show up multiple times before all the files have been seen.
Here is some TF2.0 code to see the problem:
ds = tf.data.Dataset.from_tensor_slices([f'train_{str(i).zfill(5)}' for i in range(128)])
ds = ds.shuffle(128)
n_splits = 4
sub_datasets = [ds.shard(n_splits, i) for i in range(n_splits)]
output = []
# go through each of the subsets
for i in range(n_splits):
results = [x.numpy().decode() for x in sub_datasets[i]]
output.extend(results)
print(len(set(output)), 'is the number of unique files seen (128 desired)')
Here's an answer from what I can understand of your question. To generate a new subset of 4 datasets (shared) each time randomly shuffled, you can use the following code.
import numpy as np
import tensorflow as tf
# ####################
# I used numbers for visualization ... feel free to replace with your demo code
# ####################
# ds = tf.data.Dataset.from_tensor_slices([f'train_{str(i).zfill(5)}' for i in range(128)])
# ####################
arr = np.arange(128)
ds = tf.data.Dataset.from_tensor_slices(arr)
def get_four_datasets(original_ds, window_size=32, shuffle_size=128):
""" Every time you call this function you will get a new four datasets """
return original_ds.shuffle(shuffle_size).window(window_size)
remake_ds_1 = list()
remake_ds_2 = list()
for i, (dataset_1, dataset_2) in enumerate(zip(get_four_datasets(ds), get_four_datasets(ds))):
print(f"\n\nDATASET #1-{i+1}")
ds_subset = [value for value in dataset_1.as_numpy_iterator()]
print("\t", ds_subset)
remake_ds_1.extend(ds_subset)
print(f"\nDATASET #2-{i+1}")
ds_subset_2 = [value for value in dataset_2.as_numpy_iterator()]
print("\t", ds_subset_2)
remake_ds_2.extend(ds_subset_2)
print("\n\nCounts\n")
print("DS 1 ALL: ", len(remake_ds_1))
print("DS 1 UNIQUE: ", len(set(remake_ds_1)))
print("DS 2 ALL: ", len(remake_ds_2))
print("DS 2 UNIQUE: ", len(set(remake_ds_2)))
OUTPUT
DATASET #1-1
[96, 4, 66, 120, 42, 54, 110, 57, 67, 7, 13, 9, 69, 86, 122, 88, 10, 55, 27, 106, 77, 107, 114, 87, 59, 81, 1, 49, 118, 17, 36, 11]
DATASET #2-1
[47, 26, 122, 10, 110, 31, 86, 34, 52, 121, 36, 112, 55, 48, 50, 108, 100, 103, 113, 68, 58, 29, 32, 84, 124, 15, 38, 51, 6, 66, 24, 41]
DATASET #1-2
[56, 80, 94, 124, 52, 109, 83, 90, 112, 35, 6, 101, 20, 84, 73, 74, 100, 99, 108, 15, 14, 12, 89, 24, 8, 29, 68, 85, 125, 3, 33, 58]
DATASET #2-2
[125, 127, 74, 97, 12, 39, 109, 126, 98, 40, 99, 93, 35, 107, 91, 88, 45, 13, 106, 120, 19, 73, 83, 11, 105, 61, 16, 114, 79, 95, 94, 44]
DATASET #1-3
[105, 38, 43, 60, 0, 26, 127, 65, 22, 18, 123, 82, 121, 71, 51, 23, 113, 30, 63, 40, 2, 61, 16, 98, 64, 25, 41, 28, 45, 19, 117, 39]
DATASET #2-3
[75, 64, 1, 17, 7, 42, 80, 92, 3, 9, 54, 33, 82, 56, 118, 102, 115, 43, 28, 90, 60, 119, 0, 57, 123, 62, 22, 72, 65, 23, 30, 87]
DATASET #1-4
[48, 62, 31, 102, 111, 46, 103, 44, 116, 79, 21, 50, 53, 78, 93, 32, 95, 34, 92, 126, 104, 47, 119, 37, 5, 70, 97, 91, 76, 75, 72, 115]
DATASET #2-4
[4, 85, 21, 116, 78, 27, 117, 2, 59, 111, 69, 46, 63, 20, 49, 5, 81, 53, 18, 37, 8, 76, 71, 89, 14, 104, 25, 96, 67, 101, 77, 70]
Counts
DS 1 ALL: 128
DS 1 UNIQUE: 128
DS 2 ALL: 128
DS 2 UNIQUE: 128
If you just want to generate a dataset where every 32 examples pulled from the dataset is shuffled and you want to iterate over the dataset multiple times getting new 32-set samples every time, you can do the following.
import numpy as np
import tensorflow as tf
arr = np.arange(128)
N_REPEATS = 10
ds = tf.data.Dataset.from_tensor_slices(arr)
ds = ds.shuffle(128).batch(32).repeat(N_REPEATS)
OUTPUT
BATCH 1: [92, 94, 76, 38, 58, 9, 44, 16, 86, 28, 64, 7, 60, 42, 31, 0, 46, 1, 83, 57, 18, 102, 67, 110, 113, 101, 93, 61, 96, 17, 105, 6]
BATCH 2: [59, 15, 121, 3, 72, 100, 50, 52, 45, 23, 87, 43, 33, 29, 62, 25, 74, 65, 75, 68, 4, 56, 117, 47, 73, 109, 106, 35, 88, 91, 119, 66]
BATCH 3: [98, 78, 125, 24, 99, 51, 14, 114, 26, 22, 54, 89, 79, 63, 30, 124, 20, 13, 2, 34, 95, 41, 85, 39, 37, 77, 90, 107, 104, 118, 27, 97]
BATCH 4: [49, 5, 53, 115, 126, 40, 108, 48, 8, 84, 120, 32, 82, 11, 112, 55, 80, 69, 12, 70, 111, 123, 81, 116, 71, 122, 36, 21, 103, 19, 127, 10]
BATCH 5: [74, 61, 97, 6, 127, 119, 65, 15, 78, 72, 99, 18, 41, 76, 79, 33, 0, 105, 103, 46, 14, 50, 113, 26, 43, 45, 100, 90, 28, 48, 19, 9]
BATCH 6: [35, 20, 3, 64, 5, 96, 114, 34, 126, 85, 124, 69, 110, 54, 109, 24, 104, 32, 73, 92, 11, 13, 58, 107, 84, 88, 59, 75, 95, 40, 16, 101]
BATCH 7: [93, 66, 106, 44, 102, 125, 7, 30, 12, 116, 87, 111, 81, 56, 83, 37, 31, 77, 67, 21, 118, 1, 120, 36, 86, 62, 71, 98, 82, 52, 25, 27]
BATCH 8: [112, 68, 60, 70, 115, 117, 29, 91, 57, 10, 121, 89, 4, 2, 122, 39, 51, 22, 53, 63, 108, 94, 42, 17, 8, 23, 80, 38, 55, 49, 47, 123]
BATCH 9: [67, 20, 101, 123, 109, 4, 39, 65, 34, 71, 22, 62, 73, 81, 114, 112, 66, 35, 43, 49, 92, 68, 1, 54, 27, 103, 46, 12, 82, 6, 119, 99]
BATCH 10: [86, 69, 13, 44, 16, 50, 75, 61, 58, 104, 64, 47, 95, 10, 79, 70, 97, 63, 45, 17, 56, 74, 87, 53, 91, 21, 48, 76, 9, 51, 28, 126]
...
...
...
BATCH 40: [10, 41, 29, 39, 57, 127, 101, 106, 55, 62, 72, 76, 124, 81, 66, 126, 53, 24, 33, 49, 102, 75, 34, 61, 47, 15, 21, 121, 8, 94, 52, 13]
Please let me know if I misunderstood and I can update accordingly.
You need to set reshuffle_each_iteration=False:
ds = ds.shuffle(128, reshuffle_each_iteration=False)
Full code:
import tensorflow as tf
ds = tf.data.Dataset.from_tensor_slices([f'train_{str(i).zfill(5)}' for i in range(128)])
ds = ds.shuffle(128, reshuffle_each_iteration=False)
n_splits = 4
sub_datasets = [ds.shard(n_splits, i) for i in range(n_splits)]
output = []
# go through each of the subsets
for i in range(n_splits):
results = [x.numpy().decode() for x in sub_datasets[i]]
output.extend(results)
print(len(set(output)), 'is the number of unique files seen (128 desired)')
128 is the number of unique files seen (128 desired)
This question already has answers here:
What does "e" in "1e-5" in Python language mean and what is the name of this notation? [duplicate]
(3 answers)
What is the meaning of number 1e5?
(5 answers)
Closed 4 years ago.
I am running non-normal samples against normality tests, expecting it to fail. However, the p-value {normaltest: 2.64, shapiro: 6.23} is much higher than 0.05 so I cannnot reject the null hypotheses that this was drawn from a normal distribution. Could someone please enlighten me?
import random
from scipy.stats import shapiro
from scipy import stats
x = list()
for y in range(1000):
x.append(random.randrange(1,100))
print(stats.normaltest(x))
print(shapiro(x))
print(x)
NormaltestResult(statistic=500.02063130202464, pvalue=2.641797311523516e-109)
(0.9599024057388306, 6.233162463518298e-16)
[25, 94, 79, 12, 67, 27, 89, 94, 89, 34, 99, 58, 53, 29, 81, 94, 4, 52, 14, 19, 60, 26, 6, 85, 71, 57, 23, 13, 58, 75, 75, 10, 51, 12, 80, 3, 82, 64, 74, 57, 83, 15, 23, 45, 52, 51, 36, 61, 78, 50, 26, 72, 40, 19, 59, 90, 23, 71, 52, 25, 3, 16, 20, 62, 50, 56, 60, 73, 28, 96, 69, 80, 36, 11, 11, 7, 18, 1, 73, 17, 29, 57, 72, 87, 43, 18, 22, 54, 32, 35, 79, 27, 45, 81, 80, 79, 82, 49, 77, 73, 21, 17, 90, 96, 2, 72, 7, 43, 37, 72, 64, 53, 63, 5, 36, 74, 36, 59, 53, 55, 54, 15, 83, 65, 2, 64, 46, 51, 31, 1, 77, 28, 47, 75, 46, 56, 3, 16, 24, 27, 31, 66, 4, 61, 46, 2, 56, 59, 98, 86, 83, 86, 97, 59, 45, 80, 55, 23, 21, 61, 6, 20, 13, 54, 20, 58, 86, 38, 18, 47, 68, 52, 74, 19, 34, 56, 17, 91, 15, 54, 82, 95, 23, 54, 42, 81, 82, 6, 70, 1, 78, 49, 12, 25, 33, 38, 47, 41, 68, 75, 73, 76, 46, 7, 90, 89, 63, 43, 41, 46, 88, 14, 97, 37, 92, 76, 60, 7, 5, 56, 77, 98, 61, 60, 59, 64, 4, 76, 34, 84, 78, 39, 66, 24, 49, 60, 57, 13, 57, 18, 37, 52, 26, 36, 97, 47, 95, 26, 82, 82, 10, 76, 54, 67, 98, 22, 56, 20, 34, 76, 28, 50, 70, 87, 83, 13, 76, 87, 98, 29, 99, 29, 23, 74, 5, 54, 73, 31, 89, 10, 24, 15, 9, 34, 85, 23, 6, 25, 64, 94, 37, 30, 11, 9, 58, 43, 2, 1, 73, 49, 48, 41, 99, 30, 91, 17, 31, 58, 70, 46, 20, 33, 94, 35, 41, 19, 22, 2, 37, 8, 54, 41, 21, 16, 20, 65, 27, 68, 24, 19, 36, 63, 80, 12, 82, 74, 74, 46, 7, 36, 42, 72, 16, 26, 96, 48, 75, 86, 62, 20, 79, 66, 71, 43, 43, 96, 67, 97, 76, 40, 80, 97, 75, 2, 3, 97, 37, 78, 77, 25, 84, 82, 25, 87, 44, 80, 92, 95, 99, 48, 67, 12, 82, 3, 15, 40, 45, 94, 32, 87, 92, 24, 42, 73, 66, 20, 62, 9, 75, 51, 31, 3, 13, 76, 21, 32, 14, 58, 28, 14, 99, 14, 50, 45, 13, 83, 45, 59, 63, 39, 65, 78, 46, 96, 27, 16, 69, 42, 65, 68, 68, 90, 39, 50, 86, 7, 75, 93, 84, 23, 53, 31, 23, 63, 32, 60, 85, 67, 42, 5, 72, 44, 43, 98, 75, 55, 28, 99, 71, 72, 66, 46, 61, 52, 50, 16, 44, 63, 64, 32, 59, 73, 33, 36, 32, 45, 75, 44, 36, 74, 97, 2, 38, 30, 74, 12, 57, 11, 37, 83, 64, 3, 63, 3, 35, 61, 55, 59, 99, 51, 58, 63, 70, 84, 4, 18, 13, 51, 27, 75, 43, 63, 35, 76, 67, 32, 15, 54, 51, 31, 77, 97, 83, 50, 76, 87, 26, 55, 93, 31, 70, 5, 11, 54, 48, 55, 48, 76, 90, 75, 43, 34, 6, 22, 35, 29, 4, 47, 83, 44, 7, 2, 97, 74, 90, 91, 17, 12, 33, 52, 28, 95, 57, 22, 53, 83, 56, 71, 28, 76, 55, 12, 89, 27, 20, 20, 93, 43, 65, 34, 83, 92, 11, 22, 38, 90, 83, 77, 11, 5, 22, 73, 29, 67, 49, 16, 47, 60, 26, 20, 76, 57, 46, 70, 35, 9, 28, 33, 8, 33, 21, 65, 3, 67, 52, 45, 82, 32, 94, 89, 15, 27, 63, 53, 96, 4, 74, 29, 59, 67, 22, 24, 45, 63, 76, 66, 51, 28, 42, 83, 37, 56, 83, 14, 35, 99, 48, 93, 83, 76, 2, 20, 99, 41, 43, 61, 3, 13, 7, 74, 60, 17, 84, 16, 44, 76, 63, 85, 44, 27, 38, 29, 61, 34, 55, 91, 13, 31, 42, 35, 38, 6, 46, 31, 99, 85, 23, 67, 11, 4, 52, 57, 11, 9, 21, 64, 17, 46, 78, 83, 45, 68, 98, 88, 47, 1, 94, 24, 79, 47, 33, 7, 81, 12, 26, 99, 80, 78, 53, 88, 1, 49, 17, 91, 27, 44, 31, 20, 6, 46, 59, 40, 57, 80, 3, 72, 83, 81, 2, 27, 36, 94, 31, 30, 22, 26, 31, 14, 93, 11, 32, 14, 75, 17, 49, 54, 42, 56, 76, 42, 51, 69, 22, 86, 46, 97, 70, 24, 81, 3, 75, 81, 63, 48, 51, 72, 19, 16, 16, 1, 61, 95, 53, 36, 82, 93, 53, 65, 9, 40, 91, 41, 85, 65, 38, 59, 4, 92, 50, 51, 7, 87, 80, 45, 84, 57, 21, 44, 3, 52, 7, 53, 97, 46, 65, 37, 76, 7, 81, 49, 21, 25, 18, 84, 53, 84, 89, 5, 95, 69, 70, 56, 31, 69, 12, 72, 36, 12, 44, 94, 39, 97, 91, 92, 15, 17, 57, 17, 49, 47, 1, 8, 2, 93, 91, 29, 41, 12, 46, 75, 98, 14, 34, 6, 26, 6, 81, 75, 49, 61, 70, 83, 26, 83, 38, 40, 81, 27, 14, 40, 54, 35, 10, 22, 30, 38, 2, 95, 31, 32, 25, 88, 70, 33, 85, 52, 7, 47, 4, 87, 70, 90, 15, 53, 74, 45, 76, 50, 23, 54, 33, 90, 53, 70, 4, 6, 47, 77, 87, 60, 92, 94, 79, 36, 26, 20, 94, 59, 50, 72, 31, 25, 60, 24, 82, 93, 13, 57, 21, 72, 78, 27, 62, 67, 67, 62, 56, 40, 40, 49, 31, 52, 16, 67, 51, 87, 77, 1, 47, 49, 64, 48, 62, 90, 29, 68, 2, 39, 92, 89, 92, 34, 78, 48, 36, 32, 43, 26, 81, 22, 76, 95, 69, 79, 13, 77, 26, 70, 32, 21, 92, 97, 90, 98, 16, 81, 47, 83, 93, 50]