Python matplotlib histogram does not plot all bins

Python matplotlib histogram does not plot all bins - python

When I put the following data in a histogram, the result is not plotted correctly. For example, there is a wheight equal to 5 (near the end) but it is not plotted.
How can I solve this?
import matplotlib.pyplot as plt
a=[]
for i in range(1000):
a.append(i*0.001)
b=[2.0, 1.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 5.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 2.0, 0.0, 1.0, 0.0, 1.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 2.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 2.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 2.0, 0.0, 1.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 1.0, 1.0, 2.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 2.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 3.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 3.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 2.0, 2.0, 0.0, 0.0, 0.0, 1.0, 3.0, 0.0, 0.0, 1.0, 0.0, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 3.0, 0.0, 3.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 2.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 3.0, 0.0, 2.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 2.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 2.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 3.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 2.0, 0.0, 1.0, 2.0, 2.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 2.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 2.0, 1.0, 0.0, 2.0, 0.0, 2.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 2.0, 3.0, 2.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 3.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 2.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 2.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 3.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 2.0, 0.0, 1.0, 1.0, 2.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 4.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 2.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 0.0, 0.0, 0.0, 2.0, 1.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 2.0, 1.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 2.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 2.0, 3.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 3.0, 0.0, 0.0, 1.0, 2.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 1.0, 2.0, 1.0, 1.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 2.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 2.0, 1.0, 2.0, 0.0, 0.0, 2.0, 1.0, 0.0, 0.0, 3.0, 0.0, 0.0, 0.0, 0.0, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 2.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 2.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 2.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 3.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 2.0, 0.0, 5.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0]
plt.hist(a,1000, weights=b)
plt.show()

Check out the documentation of the matplotlib.pyplot.hist function here.
The first parameter, in your case a, specifies the values you plot.
The weights parameter says how much every value from a contributes to the accumulated weight.
Not sure why you are using a and b the way you do, but if you run this:
plt.hist(b,1000)
You will see the value 5.
You can also use the third parameter - the range of the histogram.
Use (0, 5) to show values in this range now matter what you have in a:
plt.hist(a, 1000, (0,5), weights=b)

Related

Improve computational speed for chunk-wise distance calculation

I have an issue with the performance of a code I have to calculate the distance between vectors, but I think a little context is in order before exposing the problem.
I have two sets of vectors stored in two dataframes. What I want to do is to compute the distance between the every vector in set of vectors in one dataframe to every vector in the other dataframe. Here are examples of how these dataframes looks like (I post these at the end of the question in the form of dictionaries) here only the first 5 lines:
df_sample =
CalVec
1272 [0.0, 4.0, 8.0, 15.0, 10.0, 8.0, 2.54, 2.0, 4.91, 0.0, 0.0, 0.0, 0.0, 3.59, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0]
657 [1.44, 12.0, 10.0, 5.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.23, 4.36, 15.0]
806 [4.58, 13.09, 15.46, 3.59, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 6.31]
771 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 4.0, 0.0, 2.0, 0.0, 5.59, 11.67, 3.91, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
1370 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 15.0, 2.89, 0.0, 0.0, 0.0, 0.0]
df_sample.to_dict()
and
DF =
id \
4538 A4060462000516278
5043 A4050494272716275
11663 A4070271111316245
2701 A4060462848716270
825 A4060454573516274
MeasVec
4538 [0.0, 0.0, 0.0, 0.0, 6.0, 15.0, 16.0, 0.0, 0.0, 5.0, 0.0, 15.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.5, 0.0, 3.0]
5043 [0.0, 0.0, 0.0, 0.0, 0.0, 16.0, 12.0, 0.0, 13.0, 15.0, 0.0, 15.0, 0.0, 0.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 3.0, 0.0]
11663 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 15.0, 0.0, 0.0, 0.0, 6.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
2701 [0.0, 0.0, 0.0, 8.0, 13.0, 16.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.0, 0.0, 7.0]
825 [0.0, 0.0, 0.0, 0.0, 0.0, 11.0, 15.0, 0.0, 13.0, 16.0, 0.0, 9.0, 3.0, 0.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
df_sample
M = len(DF)
In reality df_sample has 1700 rows while DF has 12000 rows. I provide a sample of 10 and 50 respectively.
Now, to compute the distances (in my full size data) I am forced to chunk the larger dataframe into smaller pieces and in my actual distance computation I need to make sure that the chunks have the same amount of rows as df_sample, hence I create empty vectors for every chunk until it matches the length of df_sample.
M = len(DF)
N = len(df_sample)
P = int(round(M/N,0))-1
Number_of_id = int(round(M/P,0)) #There are only unique id:s in DF
Number_AP = 26
def zerolistmaker(n):
listofzeros = [0.0] * n
return listofzeros
def split_dataframe(df, chunk_size):
chunks = list()
num_chunks = len(df) // chunk_size + 1
for i in range(num_chunks):
chunks.append(df[i*chunk_size:(i+1)*chunk_size])
return chunks
DF_chunked = split_dataframe(DF,Number_of_id)
and here I compute the distances (actually, weighted distances, so there is no commutativity, i.e. d(v1,v2) != d(v2,v1)).
import time
t = time.process_time()
DIST = []
for i in range(P):
vec = DF_chunked[i]
number_zero_vectors = len(vec)-len(df_sample)
df =pd.DataFrame(columns = ['CalVec'])
for k in range(number_zero_vectors):
a = zerolistmaker(Number_AP)
df = df.append({'CalVec':a},ignore_index=True)
df_sample_ = pd.concat([df_sample, df])
m = np.repeat(np.vstack(df_sample_['CalVec']), df_sample_.shape[0], axis=0)
n = np.tile(np.vstack(vec['MeasVec']), (vec.shape[0], 1))
d = np.count_nonzero(m, axis=1, keepdims=True)
dist = np.sqrt(np.sum((m - n)**2/d, axis=-1))
mi = pd.MultiIndex.from_product([vec['id']] * 2, names=['id2','id'])
out = pd.DataFrame({'CalVec': m.tolist(),
'MeasVec': n.tolist(),
'distance': dist}, index=mi).reset_index()
DIST.append(out)
elapsed_time = time.process_time() - t
distances = pd.concat(DIST)
distances = distances.drop(['id2'], axis = 1)
distances = distances.dropna()
print(elapsed_time)
which gives the time 0.0625 and the distance df:
id \
0 A4060462000516278
1 A4050494272716275
2 A4070271111316245
3 A4060462848716270
CalVec \
0 [0.0, 4.0, 8.0, 15.0, 10.0, 8.0, 2.54, 2.0, 4.91, 0.0, 0.0, 0.0, 0.0, 3.59, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0]
1 [0.0, 4.0, 8.0, 15.0, 10.0, 8.0, 2.54, 2.0, 4.91, 0.0, 0.0, 0.0, 0.0, 3.59, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0]
2 [0.0, 4.0, 8.0, 15.0, 10.0, 8.0, 2.54, 2.0, 4.91, 0.0, 0.0, 0.0, 0.0, 3.59, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0]
3 [0.0, 4.0, 8.0, 15.0, 10.0, 8.0, 2.54, 2.0, 4.91, 0.0, 0.0, 0.0, 0.0, 3.59, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0]
MeasVec \
0 [0.0, 0.0, 0.0, 0.0, 6.0, 15.0, 16.0, 0.0, 0.0, 5.0, 0.0, 15.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.5, 0.0, 3.0]
1 [0.0, 0.0, 0.0, 0.0, 0.0, 16.0, 12.0, 0.0, 13.0, 15.0, 0.0, 15.0, 0.0, 0.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 3.0, 0.0]
2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 15.0, 0.0, 0.0, 0.0, 6.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
3 [0.0, 0.0, 0.0, 8.0, 13.0, 16.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.0, 0.0, 7.0]
distance
0 8.98
1 10.45
2 8.92
3 5.19
Now, this seems to be fast but it isn't. In fact the time grows exponentially and when considering the entire sets, it takes almost 20 minutes, if the kernel doesn't crash. It is so memory consuming that I cannot do anything else on my computer.
I would appreciate any insight.
DATA
df_sample = {'CalVec': {1272: [0.0,
4.0,
8.0,
15.0,
10.0,
8.0,
2.54,
2.0,
4.91,
0.0,
0.0,
0.0,
0.0,
3.59,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
8.0],
657: [1.44,
12.0,
10.0,
5.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
8.23,
4.36,
15.0],
806: [4.58,
13.09,
15.46,
3.59,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
0.0,
6.31],
771: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
4.0,
0.0,
2.0,
0.0,
5.59,
11.67,
3.91,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
1370: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
15.0,
2.89,
0.0,
0.0,
0.0,
0.0],
991: [0.0,
0.0,
0.0,
0.0,
9.0,
1.75,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
6.5,
14.71,
13.0,
9.0],
194: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
4.0,
15.54,
13.0,
2.12,
0.0],
1128: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.77,
1.8,
7.0,
6.0,
0.0,
1.8,
0.0,
9.0,
7.0,
0.0,
2.5,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
158: [0.0,
0.0,
0.0,
0.0,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
8.0,
15.44,
13.0,
2.0],
580: [0.0,
2.0,
6.0,
15.64,
2.0,
2.0,
9.23,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
4.23]}}
and
DF = {'id': {4538: 'A4060462000516278',
5043: 'A4050494272716275',
11663: 'A4070271111316245',
2701: 'A4060462848716270',
825: 'A4060454573516274',
8679: 'A4060462010016274',
11700: 'A4060462080916270',
8594: 'A4060461067716272',
8707: 'A4060454363916275',
1071: 'A4060463723916275',
7128: 'A4050494407616274',
8828: 'A4060464006116272',
8505: 'A4050500855716270',
9958: 'A4060462054116273',
2048: 'A4060461032216279',
8522: 'A4050494268116274',
10934: 'A4070270449716242',
10128: 'A4050500604416279',
9453: 'A4050500735216272',
11820: 'A4060462873316274',
7617: 'A4060461991516276',
6930: 'A4050500905516274',
11376: 'A4060454760216279',
5619: 'A4139300114013544',
35: 'A4050470904716271',
7957: 'A4090281675416244',
4216: 'A4050494309816277',
6244: 'A4050494283216272',
11922: 'A4070271196316248',
8914: 'A4060461041916276',
6054: 'A4060462056416278',
12014: 'A4060464023316273',
1362: 'A4050494275316274',
749: 'A4620451876116275',
4405: 'A4620451903216277',
2021: 'A4060454386016271',
7175: 'A4060462829816270',
351: 'A4060454654316272',
5853: 'A4050494877016279',
7980: 'A4050500932116270',
17: 'A4620451899116270',
8234: 'A4050494361416271',
10271: 'A4050500470516271',
1325: 'A4050500771516275',
2391: 'A4050500683216274',
372: 'A4050494830916277',
5527: 'A4050490253316276',
5431: 'A4050500884316278',
717: 'A4060461998716275',
10015: 'A4050500032916279'},
'MeasVec': {4538: [0.0,
0.0,
0.0,
0.0,
6.0,
15.0,
16.0,
0.0,
0.0,
5.0,
0.0,
15.0,
0.0,
0.0,
0.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.5,
0.0,
3.0],
5043: [0.0,
0.0,
0.0,
0.0,
0.0,
16.0,
12.0,
0.0,
13.0,
15.0,
0.0,
15.0,
0.0,
0.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.0,
3.0,
0.0],
11663: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
15.0,
0.0,
0.0,
0.0,
6.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
2701: [0.0,
0.0,
0.0,
8.0,
13.0,
16.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
6.0,
0.0,
7.0],
825: [0.0,
0.0,
0.0,
0.0,
0.0,
11.0,
15.0,
0.0,
13.0,
16.0,
0.0,
9.0,
3.0,
0.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
8679: [0.0,
4.0,
9.0,
15.0,
10.0,
3.0,
2.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
9.0],
11700: [0.0,
0.0,
6.0,
0.0,
15.0,
8.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
16.0,
0.0,
6.0],
8594: [12.0,
16.0,
16.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
8.0,
0.0,
5.0],
8707: [7.0,
5.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
8.0,
15.0],
1071: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
12.0,
15.5,
6.0,
3.5,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
7128: [0.0,
0.0,
0.0,
0.0,
10.0,
15.0,
16.0,
0.0,
8.0,
12.0,
0.0,
12.0,
0.0,
0.0,
4.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.0,
0.0,
11.0],
8828: [0.0,
0.0,
0.0,
0.0,
11.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
12.0,
15.0,
15.0,
7.0],
8505: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
15.0,
16.0,
4.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
9958: [0.0,
0.0,
0.0,
0.0,
14.0,
9.0,
6.0,
0.0,
0.0,
0.0,
0.0,
13.0,
0.0,
0.0,
7.0,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
7.0,
0.0,
6.0],
2048: [0.0,
0.0,
0.0,
11.0,
0.0,
16.0,
14.0,
0.0,
7.0,
5.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
8522: [0.0,
0.0,
0.0,
4.0,
4.0,
16.0,
9.0,
0.0,
0.0,
3.0,
0.0,
14.0,
0.0,
0.0,
5.5,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
11.5,
0.0,
0.0],
10934: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
8.0,
4.5,
0.0,
2.0,
0.0,
15.0,
5.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
10128: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
7.0,
12.0,
0.0,
12.0,
5.0,
3.0,
6.0,
0.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
9453: [0.0,
0.0,
5.0,
16.0,
0.0,
2.0,
6.0,
0.0,
4.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
11820: [0.0,
0.0,
0.0,
10.0,
9.0,
15.0,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
7617: [0.0,
3.0,
10.0,
9.0,
15.0,
11.0,
8.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
2.0,
15.0],
6930: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
0.0,
0.0,
0.0,
10.0,
15.5,
14.0,
15.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
11376: [0.0,
0.0,
10.0,
7.0,
7.0,
11.0,
7.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
16.0],
5619: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
12.0,
14.0,
2.5,
2.0,
8.0],
35: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
13.0,
16.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
7957: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
4.5,
0.0,
7.0,
7.0,
0.0,
2.0,
0.0,
15.0,
8.0,
4.5,
4.5,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
4216: [16.0,
6.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.0,
5.0],
6244: [11.0,
7.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
10.0],
11922: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
15.0,
0.0,
0.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
8914: [2.0,
0.0,
4.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
15.0],
6054: [0.0,
0.0,
0.0,
0.0,
15.0,
9.0,
5.0,
0.0,
0.0,
0.0,
0.0,
13.0,
0.0,
0.0,
6.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
8.0,
0.0,
6.0],
12014: [3.0,
7.0,
6.0,
0.0,
14.0,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
16.0],
1362: [15.0,
16.0,
5.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
0.0],
749: [14.0,
15.0,
16.0,
3.0,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
6.0,
2.0,
12.0],
4405: [11.0,
16.0,
16.0,
3.0,
4.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
4.0],
2021: [0.0,
0.0,
0.0,
0.0,
0.0,
8.0,
16.0,
0.0,
0.0,
4.0,
0.0,
6.5,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
7175: [0.0,
0.0,
0.0,
2.0,
9.0,
16.0,
15.0,
0.0,
0.0,
3.0,
0.0,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
0.0,
5.0],
351: [0.0,
0.0,
0.0,
0.0,
12.0,
16.0,
5.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
7.0,
0.0,
5.0],
5853: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
16.0,
8.0,
1.5,
0.0,
0.0,
0.0,
4.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
7980: [0.0,
0.0,
13.0,
8.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
4.0],
17: [11.0,
16.0,
16.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.0,
11.0],
8234: [0.0,
0.0,
0.0,
0.0,
0.0,
6.0,
7.0,
5.0,
11.0,
13.0,
0.0,
13.0,
3.0,
11.0,
15.0,
12.0,
12.0,
5.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
10271: [0.0,
0.0,
0.0,
0.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
9.0,
0.0,
15.0,
9.0,
5.0,
5.0],
1325: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.0,
0.0,
0.0,
5.0,
0.0,
16.0,
0.0,
0.0,
9.0,
0.0,
5.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
2391: [0.0,
0.0,
3.0,
16.0,
0.0,
0.0,
0.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
4.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0],
372: [0.0,
0.0,
0.0,
0.0,
4.0,
16.0,
10.0,
0.0,
0.0,
3.0,
0.0,
12.0,
0.0,
0.0,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
7.0,
6.0,
0.0],
5527: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
0.0,
2.0,
0.0,
0.0,
14.0,
16.0,
7.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
5431: [0.0,
0.0,
0.0,
0.0,
2.0,
3.0,
8.0,
0.0,
4.0,
7.0,
0.0,
16.0,
0.0,
0.0,
8.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
717: [0.0,
0.0,
0.0,
11.0,
2.0,
14.0,
9.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
10015: [0.0,
0.0,
0.0,
7.0,
14.0,
16.0,
15.0,
0.0,
4.0,
9.0,
0.0,
11.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
6.0,
3.0,
12.0]}}

Distance calculation is a common problem, so it can be a good idea to use the available functions for that, specifically sklearn. The data you provided is not convenient to manage, but the example below might give ideas on how to adapt this workflow to the specifics of your data:
import numpy as np
import pandas as pd
from sklearn.metrics import pairwise_distances
X = pd.DataFrame(np.random.rand(10, 30))
Y = pd.DataFrame(np.random.rand(20, 30))
def custom_distance(x, y):
"""Sample asymmetric function."""
return max(x) + min(y)
# use n_jobs=-1 to run calculations with all cores
result = pairwise_distances(X, Y, metric=custom_distance, n_jobs=-1)
To complete #SultanOrazbayev:
from sklearn.metrics import pairwise_distances
Ax = df_sample['CalVec'] = df_sample['CalVec'].apply(lambda x: np.array(x))
Bx = DF['MeasVec'] = DF['MeasVec'].apply(lambda x: np.array(x))
A = Ax.to_numpy()
B = Bx.to_numpy()
AA = np.stack(A)
BB = np.stack(B)
result = pairwise_distances(AA, BB, metric=custom_distance, n_jobs=-1)
which is performed in under 3 minutes.

'ValueError: setting an array element with a sequence' while using scipy.spacial.distance function cdist

I am trying to compute the distance between vectors in two pandas dataframes using cdist from scipy.spatial.distance, but the output is all wrong and I can't pinpoint where is fails. This is actually a follow up on the question Problems computing cdist of two columns in two different dataframes
So, my original dataframes are of the type:
df_sample =
Fingerprint
1272 [0.0, 4.0, 8.0, 15.0, 10.0, 8.0, 2.54, 2.0, 4.91, 0.0, 0.0, 0.0, 0.0, 3.59, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0]
657 [1.44, 12.0, 10.0, 5.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.23, 4.36, 15.0]
806 [4.58, 13.09, 15.46, 3.59, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 6.31]
and
DF =
barcode \
4538 A4060462000516278
5043 A4050494272716275
11663 A4070271111316245
2701 A4060462848716270
825 A4060454573516274
8679 A4060462010016274
11700 A4060462080916270
8594 A4060461067716272
8707 A4060454363916275
1071 A4060463723916275
Geopos Ack
4538 [0.0, 0.0, 0.0, 0.0, 6.0, 15.0, 16.0, 0.0, 0.0, 5.0, 0.0, 15.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.5, 0.0, 3.0]
5043 [0.0, 0.0, 0.0, 0.0, 0.0, 16.0, 12.0, 0.0, 13.0, 15.0, 0.0, 15.0, 0.0, 0.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 3.0, 0.0]
11663 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 15.0, 0.0, 0.0, 0.0, 6.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
2701 [0.0, 0.0, 0.0, 8.0, 13.0, 16.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.0, 0.0, 7.0]
825 [0.0, 0.0, 0.0, 0.0, 0.0, 11.0, 15.0, 0.0, 13.0, 16.0, 0.0, 9.0, 3.0, 0.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
8679 [0.0, 4.0, 9.0, 15.0, 10.0, 3.0, 2.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 9.0]
11700 [0.0, 0.0, 6.0, 0.0, 15.0, 8.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 16.0, 0.0, 6.0]
8594 [12.0, 16.0, 16.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 0.0, 5.0]
8707 [7.0, 5.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0, 15.0]
1071 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 12.0, 15.5, 6.0, 3.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
(I provide dictionaries for both at the end of the question).
As you can see, they are of different dimension (although the vectors belong to the same space). So, to remedy this I create zero vectors in df_sample by doing this:
Number_AP = 26
number_zero_vectors = len(DF)-len(df_sample)
df =pd.DataFrame(columns = ['Fingerprint'])
for k in range(number_zero_vectors):
a = zerolistmaker(Number_AP)
df = df.append({'Fingerprint':a},ignore_index=True)
df_sample_ = pd.concat([df_sample, df])
Hence, DF and df_sample_ have the same shape. My first approach (described in detail in the question mentioned above) was to deal with both columns and all their rows simulataneously, but it did not work. So, my approach now is to deal with both column incrementally (e.g. take the first row of df_sample_ and compute the distance to every single vectors in DF, all this in a loop:
distances = []
for i in range(len(df_sample_)):
a = df_sample_['Fingerprint'][i:i+1]
print(a)
for j in range(len(DF)):
b = DF['Geopos Ack'][j:j+1]
print(b)
ax = a.to_numpy()
bx = b.to_numpy()
aa = ax.reshape(-1,1)
bb = bx.reshape(-1,1)
print(aa.shape)
print(bb.shape)
d = sp.cdist(aa,bb,'euclidean')
print(d)
### I leave out the part were distances.append(d) since what come before it fails
This returns:
1272 [0.0, 4.0, 8.0, 15.0, 10.0, 8.0, 2.54, 2.0, 4.91, 0.0, 0.0, 0.0, 0.0, 3.59, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 8.0]
Name: Fingerprint, dtype: object
4538 [0.0, 0.0, 0.0, 0.0, 6.0, 15.0, 16.0, 0.0, 0.0, 5.0, 0.0, 15.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.5, 0.0, 3.0]
Name: Geopos Ack, dtype: object
(1, 1)
(1, 1)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
TypeError: only size-1 arrays can be converted to Python scalars
The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
<ipython-input-368-e29e9515d73a> in <module>
14 print(bb.shape)
15
---> 16 d = sp.cdist(aa,bb,'euclidean')
17 print(d)
18
~\Anaconda3\envs\conda-qgis\lib\site-packages\scipy\spatial\distance.py in cdist(XA, XB, metric, *args, **kwargs)
2802 metric_name = _METRIC_ALIAS.get(mstr, None)
2803 if metric_name is not None:
-> 2804 XA, XB, typ, kwargs = _validate_cdist_input(XA, XB, mA, mB, n,
2805 metric_name, **kwargs)
2806
~\Anaconda3\envs\conda-qgis\lib\site-packages\scipy\spatial\distance.py in _validate_cdist_input(XA, XB, mA, mB, n, metric_name, **kwargs)
245 typ = types[types.index(XA.dtype)] if XA.dtype in types else types[0]
246 # validate data
--> 247 XA = _convert_to_type(XA, out_type=typ)
248 XB = _convert_to_type(XB, out_type=typ)
249
~\Anaconda3\envs\conda-qgis\lib\site-packages\scipy\spatial\distance.py in _convert_to_type(X, out_type)
182
183 def _convert_to_type(X, out_type):
--> 184 return np.ascontiguousarray(X, dtype=out_type)
185
186
~\Anaconda3\envs\conda-qgis\lib\site-packages\numpy\core\_asarray.py in ascontiguousarray(a, dtype)
175
176 """
--> 177 return array(a, dtype, copy=False, order='C', ndmin=1)
178
179
ValueError: setting an array element with a sequence.
Where did I go wrong? All arrays consist of floats and the shapes look right.
I know another approach would be to use pairwise_distance from sklearn but I did not manage to apply it to my dataframes.
Any help would be appreciated.
Data:
df_sample =
{'Fingerprint': {1272: [0.0,
4.0,
8.0,
15.0,
10.0,
8.0,
2.54,
2.0,
4.91,
0.0,
0.0,
0.0,
0.0,
3.59,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
8.0],
657: [1.44,
12.0,
10.0,
5.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
8.23,
4.36,
15.0],
806: [4.58,
13.09,
15.46,
3.59,
3.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
0.0,
6.31]}}
and
DF =
{'barcode': {4538: 'A4060462000516278',
5043: 'A4050494272716275',
11663: 'A4070271111316245',
2701: 'A4060462848716270',
825: 'A4060454573516274',
8679: 'A4060462010016274',
11700: 'A4060462080916270',
8594: 'A4060461067716272',
8707: 'A4060454363916275',
1071: 'A4060463723916275'},
'Geopos Ack': {4538: [0.0,
0.0,
0.0,
0.0,
6.0,
15.0,
16.0,
0.0,
0.0,
5.0,
0.0,
15.0,
0.0,
0.0,
0.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.5,
0.0,
3.0],
5043: [0.0,
0.0,
0.0,
0.0,
0.0,
16.0,
12.0,
0.0,
13.0,
15.0,
0.0,
15.0,
0.0,
0.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
3.0,
3.0,
0.0],
11663: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
5.0,
15.0,
0.0,
0.0,
0.0,
6.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
2701: [0.0,
0.0,
0.0,
8.0,
13.0,
16.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
6.0,
0.0,
7.0],
825: [0.0,
0.0,
0.0,
0.0,
0.0,
11.0,
15.0,
0.0,
13.0,
16.0,
0.0,
9.0,
3.0,
0.0,
6.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0],
8679: [0.0,
4.0,
9.0,
15.0,
10.0,
3.0,
2.0,
0.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
9.0],
11700: [0.0,
0.0,
6.0,
0.0,
15.0,
8.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
16.0,
0.0,
6.0],
8594: [12.0,
16.0,
16.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
8.0,
0.0,
5.0],
8707: [7.0,
5.0,
2.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
2.0,
8.0,
15.0],
1071: [0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
12.0,
15.5,
6.0,
3.5,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0]}}

append list with for loop and get multiple values in each cell in python

I'm trying to run 3 optimization with for loop and store the results in one dataframe.
After each optimization (element of the for loop), I append lists of results and being able to get all the reults in one list. However, when I try to convert the list to dataframe, I get one row for each of the optimization and multiple values in each cell corresponding to the variable name and the optimization number like this:
Date = []
results = []
for idx, df in enumerate([df0,df1,df2]):
model = ConcreteModel()
model.T = Set(initialize=df.hour.tolist(), ordered=True)
...
# Solve model
solver = SolverFactory('glpk')
solver.solve(model)
Date = list(df['Date'])
results.append([Date, model.Ein.get_values().values(), model.Eout.get_values().values(),
model.Z.get_values().values(), model.NES.get_values().values(),
model.L.get_values().values()])
df_results = pd.DataFrame(results)
df_results.rename(columns = {0: 'Date', 1: 'Ein', 2:'Eout', 3:'Z', 4:'NES', 5:'L'}, inplace = True)
df_results
## The output of the df is:
Date Ein
0 [2019-01-01, 2019-01-01, 2019-01-01, 2019-01-0... (0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, ... (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... (0.0, 0.0, 1.0, 2.0, 3.0, 3.0, 4.0, 5.0, 5.0, ... (0.0, 0.0, -100.0, -100.0, -100.0, 0.0, -100.0... (16231.0, 16051.0, 15806.0, 15581.0, 15610.0, ...
1 [2019-01-16, 2019-01-16, 2019-01-16, 2019-01-1... (0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, ... (0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.0, 0.0, 0.0, ... (0.0, 1.0, 1.0, 1.0, 1.0, 0.5, 1.5, 2.5, 3.5, ... (0.0, -100.0, 0.0, 0.0, 0.0, 50.0, -100.0, -10... (17643.0, 18654.0, 20462.0, 20448.0, 20305.0, ...
2 [2019-01-31, 2019-01-31, 2019-01-31, 2019-01-3... (0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, ... (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, ... (0.0, 0.0, 1.0, 1.0, 2.0, 3.0, 4.0, 3.0, 3.0, ... (0.0, 0.0, -100.0, 0.0, -100.0, -100.0, -100.0... (22155.0, 22184.0, 21510.0, 21193.0, 20884.0, ...
#The output of the list named results is:
[[['2019-01-01',
'2019-01-01',
'2019-01-01',
...
'2019-01-15',
'2019-01-15',
'2019-01-15',
'2019-01-15',
'2019-01-15',
'2019-01-15',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16'],
dict_values([0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
...
-1.11022302462516e-16, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.11022302462516e-16, 0.0, 1.0, 0.5, 0.0, 0.0, 0.0, 0.166666666666667, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.333333333333333, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.666666666666667, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]),
dict_values([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.5, 0.25, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.25,
...
0.333333333333333, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.166666666666667, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.833333333333333, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.166666666666667, 0.0, 0.0, 0.0, 0.666666666666667, 0.333333333333333, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625, 0.001953125, 0.0009765625, 0.00048828125, 0.0]),
dict_values([0.0, 0.0, 1.0, 2.0, 3.0, 3.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 0.5, 0.5, 0.5, 1.5, 2.5, 3.5, 4.5, 4.5, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 3.0, 4.0, 5.0, 5.0, 5.0, 4.0, 3.0, 3.0, 3.0,
...
0.142857142857143, 0.142857142857143, 1.0, 2.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 3.0, 4.0, 5.0, 5.0, 5.0, 4.0, 3.0, 3.0, 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.33333333333333, 1.33333333333333, 0.666666666666667, 0.666666666666667, 0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625, 0.001953125, 0.0009765625, 0.00048828125, 0.00048828125]),
[['2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
'2019-01-16',
...
Is it because each result in the for loop has de different dictionary? How could my results in this form:
Date Ein Eout Z NES L
0 2019-01-01 1.0 0.0 1.0 -100.0 16231.0
1 2019-01-01 1.0 1.0 0.0 100.0 16051.0,
...

You're constantly appending to results creating a list of lists of the wrong dimension. I hope this solution works for you -
df_results = pd.DataFrame(zip(Date, model.Ein.get_values().values(), model.Eout.get_values().values(),
model.Z.get_values().values(), model.NES.get_values().values(),
model.L.get_values().values()))
Let me know if it doesn't.

Plotting a histogram using a range of values and their frequency as a dictionary

Assume that I have the following dictionary:
scenario_summary = {'Day1': {'22459-22585': 0.0, '22585-22711': 0.0, '22711-22837': 0.0, '22837-22963': 0.0, '22963-23089': 0.0, '23089-23215': 0.0, '23215-23341': 0.0, '23341-23467': 0.0, '23467-23593': 0.0, '23593-23719': 0.0, '23719-23845': 0.0, '23845-23971': 0.0, '23971-24097': 0.0, '24097-24223': 0.0, '24223-24349': 0.0, '24349-24475': 0.0, '24475-24601': 0.0, '24601-24727': 0.0, '24727-24853': 0.0, '24853-24979': 0.0, '24979-25105': 0.0, '25105-25231': 0.0, '25231-25357': 0.0, '25357-25483': 0.0, '25483-25609': 0.0, '25609-25735': 0.0, '25735-25861': 0.0, '25861-25987': 0.0, '25987-26113': 1.0, '26113-26239': 1.0, '26239-26365': 0.0, '26365-26491': 2.0, '26491-26617': 5.0, '26617-26743': 5.0, '26743-26869': 5.0, '26869-26995': 12.0, '26995-27121': 19.0, '27121-27247': 7.000000000000001, '27247-27373': 11.0, '27373-27499': 15.0, '27499-27625': 7.000000000000001, '27625-27751': 4.0, '27751-27877': 4.0, '27877-28003': 2.0, '28003-28129': 0.0, '28129-28255': 0.0, '28255-28381': 0.0, '28381-28507': 0.0, '28507-28633': 0.0, '28633-28759': 0.0, '28759-28885': 0.0, '28885-29011': 0.0, '29011-29137': 0.0, '29137-29263': 0.0, '29263-29389': 0.0, '29389-29515': 0.0, '29515-29641': 0.0, '29641-29767': 0.0, '29767-29893': 0.0, '29893-30019': 0.0, '30019-30145': 0.0, '30145-30271': 0.0, '30271-30397': 0.0, '30397-30523': 0.0, '30523-30649': 0.0, '30649-30775': 0.0, '30775-30901': 0.0, '30901-31027': 0.0, '31027-31153': 0.0, '31153-31279': 0.0, '31279-31405': 0.0, '31405-31531': 0.0, '31531-31657': 0.0, '31657-31783': 0.0, '31783-31909': 0.0, '31909-32035': 0.0, '32035-32161': 0.0, '32161-32287': 0.0, '32287-32413': 0.0, '32413-32539': 0.0, '32539-32665': 0.0, '32665-32791': 0.0, '32791-32917': 0.0, '32917-33043': 0.0, '33043-33169': 0.0, '33169-33295': 0.0, '33295-33421': 0.0, '33421-33547': 0.0, '33547-33673': 0.0, '33673-33799': 0.0, '33799-33925': 0.0, '33925-34051': 0.0, '34051-34177': 0.0, '34177-34303': 0.0, '34303-34429': 0.0, '34429-34555': 0.0, '34555-34681': 0.0, '34681-34807': 0.0}}
As you can see, the dictionary consists of a range of values in string and their frequency. I would like to plot this as a histogram, but I don't know how I would be able to transform the string into a form that pandas or plotly would understand. What would your approach be? Or is there an easier way to do it, instead of hardcoding things? Or, would another module be easier option in doing so?
Thanks!

Since the bins (ranges) are already defined and their counts are already aggregated at an initial level, maybe it can help if you build something that overlays a histogram (distribution) on the top of the existing bin ranges:
import matplotlib
%matplotlib inline
def plot_hist(bins,input_dict):
df1 = pd.DataFrame(input_dict).reset_index()
df1['min'] = df1['index'].apply(lambda x:x.split('-')[0]).astype(int)
df1['max'] = df1['index'].apply(lambda x:x.split('-')[1]).astype(int)
df1['group'] = pd.cut(df1['max'],bins,labels=False)
df2 = df1.groupby('group' [['Day1','min','max']].agg({'min':'min','max':'max','Day1':'sum'}).reset_index()
df2['range_new'] = df2['min'].astype(str) + str('-') + df2['max'].astype(str)
df2.plot(x='range_new',y='Day1',kind='bar')
...and call the function by choosing bins lesser than the length of the dictionary - or the first level of 98 bins that are already there, like, say if you want a distribution of 20 groups aggregate:
plot_hist(20,scenario_summary)
Result Image :
hope it helps...

A histogram is basically a simple bar chart, where each bar represents a bin (usually in the form of a range) and a frequency of the elements that fall into that bin.
This is exactly the data that you already have. So instead of computing values for a histogram (as it would be done with plt.hist), you can simply pass your data to plt.bar, as it is. The result would then be this:
The code with your data, as a MCVE :
import matplotlib.pyplot as plt
scenario_summary = { 'Day1': {
'22459-22585': 0.0, '22585-22711': 0.0, '22711-22837': 0.0,
'22837-22963': 0.0, '22963-23089': 0.0, '23089-23215': 0.0,
'23215-23341': 0.0, '23341-23467': 0.0, '23467-23593': 0.0,
'23593-23719': 0.0, '23719-23845': 0.0, '23845-23971': 0.0,
'23971-24097': 0.0, '24097-24223': 0.0, '24223-24349': 0.0,
'24349-24475': 0.0, '24475-24601': 0.0, '24601-24727': 0.0,
'24727-24853': 0.0, '24853-24979': 0.0, '24979-25105': 0.0,
'25105-25231': 0.0, '25231-25357': 0.0, '25357-25483': 0.0,
'25483-25609': 0.0, '25609-25735': 0.0, '25735-25861': 0.0,
'25861-25987': 0.0, '25987-26113': 1.0, '26113-26239': 1.0,
'26239-26365': 0.0, '26365-26491': 2.0, '26491-26617': 5.0,
'26617-26743': 5.0, '26743-26869': 5.0, '26869-26995': 12.0,
'26995-27121': 19.0, '27121-27247': 7.0, '27247-27373': 11.0,
'27373-27499': 15.0, '27499-27625': 7.0, '27625-27751': 4.0,
'27751-27877': 4.0, '27877-28003': 2.0, '28003-28129': 0.0,
'28129-28255': 0.0, '28255-28381': 0.0, '28381-28507': 0.0,
'28507-28633': 0.0, '28633-28759': 0.0, '28759-28885': 0.0,
'28885-29011': 0.0, '29011-29137': 0.0, '29137-29263': 0.0,
'29263-29389': 0.0, '29389-29515': 0.0, '29515-29641': 0.0,
'29641-29767': 0.0, '29767-29893': 0.0, '29893-30019': 0.0,
'30019-30145': 0.0, '30145-30271': 0.0, '30271-30397': 0.0,
'30397-30523': 0.0, '30523-30649': 0.0, '30649-30775': 0.0,
'30775-30901': 0.0, '30901-31027': 0.0, '31027-31153': 0.0,
'31153-31279': 0.0, '31279-31405': 0.0, '31405-31531': 0.0,
'31531-31657': 0.0, '31657-31783': 0.0, '31783-31909': 0.0,
'31909-32035': 0.0, '32035-32161': 0.0, '32161-32287': 0.0,
'32287-32413': 0.0, '32413-32539': 0.0, '32539-32665': 0.0,
'32665-32791': 0.0, '32791-32917': 0.0, '32917-33043': 0.0,
'33043-33169': 0.0, '33169-33295': 0.0, '33295-33421': 0.0,
'33421-33547': 0.0, '33547-33673': 0.0, '33673-33799': 0.0,
'33799-33925': 0.0, '33925-34051': 0.0, '34051-34177': 0.0,
'34177-34303': 0.0, '34303-34429': 0.0, '34429-34555': 0.0,
'34555-34681': 0.0, '34681-34807': 0.0}}
data = scenario_summary['Day1']
x = range(len(data))
y = list(data.values())
plt.figure(figsize=(16, 9))
plt.bar(x, y)
plt.subplots_adjust(bottom=0.2)
plt.xticks(x, data.keys(), rotation='vertical')
plt.show()

You can use pandas module to convert dictionary data into data frame:
import pandas as pd
import matplotlib.pyplot as plt
scenario_summary = {'Day1': {'22459-22585': 0.0, '22585-22711': 0.0, '22711-22837': 0.0,
'22837-22963': 0.0, '22963-23089': 0.0, '23089-23215': 0.0,
'23215-23341': 0.0, '23341-23467': 0.0, '23467-23593': 0.0,
'23593-23719': 0.0, '23719-23845': 0.0, '23845-23971': 0.0,
'23971-24097': 0.0, '24097-24223': 0.0, '24223-24349': 0.0,
'24349-24475': 0.0, '24475-24601': 0.0, '24601-24727': 0.0,
'24727-24853': 0.0, '24853-24979': 0.0, '24979-25105': 0.0,
'25105-25231': 0.0, '25231-25357': 0.0, '25357-25483': 0.0,
'25483-25609': 0.0, '25609-25735': 0.0, '25735-25861': 0.0,
'25861-25987': 0.0, '25987-26113': 1.0, '26113-26239': 1.0,
'26239-26365': 0.0, '26365-26491': 2.0, '26491-26617': 5.0,
'26617-26743': 5.0, '26743-26869': 5.0, '26869-26995': 12.0,
'26995-27121': 19.0, '27121-27247': 7.000000000000001, '27247-27373': 11.0,
'27373-27499': 15.0, '27499-27625': 7.000000000000001, '27625-27751': 4.0,
'27751-27877': 4.0, '27877-28003': 2.0, '28003-28129': 0.0,
'28129-28255': 0.0, '28255-28381': 0.0, '28381-28507': 0.0,
'28507-28633': 0.0, '28633-28759': 0.0, '28759-28885': 0.0,
'28885-29011': 0.0, '29011-29137': 0.0, '29137-29263': 0.0,
'29263-29389': 0.0, '29389-29515': 0.0, '29515-29641': 0.0,
'29641-29767': 0.0, '29767-29893': 0.0, '29893-30019': 0.0,
'30019-30145': 0.0, '30145-30271': 0.0, '30271-30397': 0.0,
'30397-30523': 0.0, '30523-30649': 0.0, '30649-30775': 0.0,
'30775-30901': 0.0, '30901-31027': 0.0, '31027-31153': 0.0,
'31153-31279': 0.0, '31279-31405': 0.0, '31405-31531': 0.0,
'31531-31657': 0.0, '31657-31783': 0.0, '31783-31909': 0.0,
'31909-32035': 0.0, '32035-32161': 0.0, '32161-32287': 0.0,
'32287-32413': 0.0, '32413-32539': 0.0, '32539-32665': 0.0,
'32665-32791': 0.0, '32791-32917': 0.0, '32917-33043': 0.0,
'33043-33169': 0.0, '33169-33295': 0.0, '33295-33421': 0.0,
'33421-33547': 0.0, '33547-33673': 0.0, '33673-33799': 0.0,
'33799-33925': 0.0, '33925-34051': 0.0, '34051-34177': 0.0,
'34177-34303': 0.0, '34303-34429': 0.0, '34429-34555': 0.0,
'34555-34681': 0.0, '34681-34807': 0.0}}
# convert to data frame
data_frame = pd.DataFrame.from_dict(scenario_summary)
# plot data
plt.hist(data_frame['Day1'], density=1, bins=20)
plt.show()

Could not convert string to float -Using Pandas and Numpy for a SVM Classifier

I'm trying to use pandas to create a SVM classifier. I already generated my feature and save it using to_csv from pandas lib. This feature(Color) consists in a whole histogram. So, I have a list of 0 to 255 float values per line. There are 362 lines.
Here is a piece of my code:
if __name__ == '__main__':
train = pd.read_csv('Train.csv',index_col='Object')
XTrain = train['Color']
ColorLabel = train['ColorLabel']
leTrain = LabelEncoder()
leTrain.fit(ColorLabel)
ColorLabel = leTrain.transform(ColorLabel)
svm = SVC()
parameters = {'kernel': ('linear', 'rbf'), 'C': (1, 0.25, 0.5, 0.75,0.05), 'gamma': (0.5,1, 2, 3, 'auto'),
'decision_function_shape': ('ovo', 'ovr'),'class_weight': [{0: 1,1: w2} for w2 in [2, 4, 6, 10,12]]}
clf = GridSearchCV(svm, parameters,verbose = 2)
clf.fit(XTrain, ColorLabel)
Im just trying to fit the feature column Color in SVC.fit, however I receive an error message that says:
return array(a, dtype, copy=False, order=order)
ValueError: could not convert string to float: '[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 2.0, 10.0, 0.0, 2.0, 0.0, 7.0, 0.0, 12.0, 2.0, 18.0, 36.0, 0.0, 87.0, 34.0, 13.0, 41.0, 30.0, 118.0, 137.0, 169.0, 530.0, 4684.0, 5746.0, 1975.0, 1815.0, 4079.0, 4725.0, 2411.0, 131.0, 434.0, 3799.0, 1435.0, 4380.0, 5.0, 0.0, 546.0, 0.0, 1695.0, 15.0, 0.0, 116.0, 82.0, 4.0, 52.0, 54.0, 4.0, 2.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]'
Here is the head of my Train.csv
Object,Kurtosis,Skewness,Color,TextureLabel,ColorLabel
0122_LSG.jpg,-0.19026044432874611,-0.9694201939544961,"[0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 10.0, 6.0, 16.0, 90.0, 47.0, 114.0, 126.0, 1918.0, 733.0, 5404.0, 3956.0, 12750.0, 13551.0, 3222.0, 3927.0, 5776.0, 4896.0, 3807.0, 9007.0, 8835.0, 1029.0, 684.0, 495.0, 172.0, 121.0, 125.0, 37.0, 93.0, 31.0, 96.0, 73.0, 7.0, 15.0, 0.0, 22.0, 0.0, 0.0, 7.0, 5.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]",L,S 0075_LSG.jpg,-0.25089779696431913,-0.5106815852572715,"[0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 22.0, 16.0, 461.0, 6.0, 37.0, 216.0, 5.0, 348.0, 45.0, 264.0, 294.0, 316.0, 999.0, 3057.0, 3625.0, 5399.0, 2420.0, 6031.0, 6636.0, 7442.0, 801.0, 5958.0, 7289.0, 11785.0, 6150.0, 8537.0, 4414.0, 398.0, 489.0, 449.0, 155.0, 270.0, 64.0, 230.0, 51.0, 101.0, 121.0, 73.0, 76.0, 36.0, 46.0, 123.0, 45.0, 51.0, 1.0, 78.0, 28.0, 0.0, 4.0, 70.0, 53.0, 0.0, 41.0, 75.0, 4.0, 39.0, 1.0, 94.0, 0.0, 18.0, 198.0, 0.0, 4.0, 225.0, 16.0, 158.0, 147.0, 8.0, 0.0, 6.0, 22.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]",L,S 0157_LSP.jpg,-0.604961472275447,-0.8074495729146061,"[0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 5.0, 0.0, 48.0, 0.0, 0.0, 0.0, 0.0, 28.0, 0.0,
I TRIED ALL THE TYPES OF TYPE CASTING THAT I KNOW astype,dtype,converters... PLEASE HELP ME

XTrain =[list(map(float, hist)) for hist in train['Color']]
Plus using ; as sep when reading and writing file
SOLVE IT .

We Keep Coding

Python is a programming language that lets you work quickly and integrate systems more effectively.

Python matplotlib histogram does not plot all bins - python

Related

Improve computational speed for chunk-wise distance calculation

'ValueError: setting an array element with a sequence' while using scipy.spacial.distance function cdist

append list with for loop and get multiple values in each cell in python

Plotting a histogram using a range of values and their frequency as a dictionary

Could not convert string to float -Using Pandas and Numpy for a SVM Classifier

Categories

Resources