Take the probability distribution of a XOR gate in which every configuration is equally probable (configurations are given by outcomes_sub; the probability mass function by pmf_xor_sub):
import numpy as np
import itertools as it
outcomes_sub = [list(item) for item in list(it.product([0,1], repeat=3))]
pmf_xor_sub = np.array([1/4, 0, 0, 1/4, 0, 1/4, 1/4, 0])
Now take the probability distribution corresponding to two uncorrelated such XORs:
outcomes = [outcome1 + outcome2 for (outcome1, outcome2)
in it.product(outcomes_sub, outcomes_sub)]
pmf_xor = [pmf1 * pmf2 for (pmf1, pmf2)
in it.product(pmf_xor_sub, pmf_xor_sub)]
And create some data based on it:
indices = np.random.choice(len(outcomes), 10000, p=pmf_xor)
data_xor = np.array([outcomes[index] for index in indices])
data_xor looks like this:
array([[1, 1, 0, 0, 0, 0],
[1, 0, 1, 0, 0, 0],
[0, 1, 1, 1, 1, 0],
...,
[0, 1, 1, 1, 1, 0],
[1, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 0]])
I.e., two independent XORs back to back. What's the right way to perform dimensionality reduction on it? PCA won't work (because the dependence is non-linear, right?):
from sklearn import decomposition
pca_xor = decomposition.PCA()
pca_xor.fit(data_xor)
Now, pca_xor.explained_variance_ratio_ gives:
array([ 0.17145045, 0.17018817, 0.16758773, 0.16575979, 0.16410862,
0.16090524], dtype=float32)
No two components stand out. I understand that a non-linear method such as kernel PCA should work here, but I am struggling to find pointers to ways of applying it to my problem.
To give a bit more context: what I am actually after is ways to bring out the structure in data_xor: two big XOR blobs, each of which is composed of some finer-grained stuff. If I am going about it all wrong, feel free to point that out too.
Related
can anyone fix undersampling confusion matrix error from the line:
undersample_cm = confusion_matrix(original_ytest, undersample_fraud_predictions)
I think the problem is from the import or original_ytest and undersample_fraud_predictions
undersample_cm = confusion_matrix(original_ytest, undersample_fraud_predictions)
actual_cm = confusion_matrix(original_ytest, original_ytest)
labels = ['No Fraud', 'Fraud']
The picture of the error here
confusion_matrix works with two arrays of same size, same type of data.
y_true = [1, 0, 0, 1, 0, 1]
y_pred = [0, 0, 0, 0, 0, 1]
confusion_matrix(y_true, y_pred)
I guess there is a continuous variable in any of your arrays. For this, if you run the actual_cm line first, you can see which array is the problem. One of your arrays, namely original_ytest, contains integer values as it should, but you're probably getting an error because it contains continuous values in the undersample_fraud_predictions array.
original_ytest = [1, 0, 0, 1, 0, 1]
undersample_fraud_predictions = [1, 0, 0, 0, 0, 1.5]
confusion_matrix(original_ytest , undersample_fraud_predictions )
When you run the code I want to explain above, the error you get will be the same (Classification metrics can't handle a mix of binary and continuous targets).
I am trying to implement a simple mapping to a set of values from an array created with numpy of 2-D.
For each row in the array I need to choose the correct value corresponding with the set of values and add it to a array.
For example:
[0, 1, 0, 0] -> 3
...
[1, 0, 1, 0] -> 2
But, my first implementation made me wonder if I'm doing something really wrong or not efficient at all because of the size of my dataset, so I did this workaround without using for loops and optimize speed execution using dictionary lookup.
import numpy as np
# function to perform the search and return the index accordingly (it is supposed to be fast because of data structure)
def get_val(n):
map_list = {0: [0, 1, 0], 1: [0, 1, 0], 2: [1, 0, 0], 3: [0, 0, 1]}
map_vals = list(map_list.values())
index = map_vals.index(list(n))
return(index)
# set of arbitrary arrays
li = np.array([[0, 1, 0], [0, 0, 1]])
# here is the performance improvement attempt with the help of the function above
arr = [get_val(n) for n in li]
print(arr)
I'm not completely sure if this is the correct way to do it for getting the needed value for a set like this. If there is a better way, please let me know.
Otherwise, I refer to my main question:
what is the best way possible to optimize the code?
Thanks so much for your help.
You can try use matrix multiplication (dot product):
a=np.array([[0, 0, 0],[0, 1, 0], [1, 0, 0], [0, 0, 1]]) # dict values
c=np.array([0,1,2,3]) # dict keys
li = np.array([[0, 1, 0], [0, 0, 1]])
b=np.linalg.pinv(a)#c # decoding table
result=li#b
print(result)
In Pytorch, you can hardcode your filters to be whatever you like.
At the moment, I'm doing text detection and I need to identify the location of a certain information. This information always starts with the letter 'X'. Could this radically improve detection performance if I hardcode the 'X' filter?
Here's what I have so far:
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
kernel = (torch.zeros((9, 9)) + \
torch.eye(9) + \
torch.rot90(torch.eye(9))).type(torch.bool)*1
print(kernel)
tensor([[1, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 1, 0],
[0, 0, 1, 0, 0, 0, 1, 0, 0],
[0, 0, 0, 1, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 1, 0, 0, 0],
[0, 0, 1, 0, 0, 0, 1, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 1, 0],
[1, 0, 0, 0, 0, 0, 0, 0, 1]])
We can visualize it like this:
plt.imshow(kernel)
plt.show()
Then, we can set the filter weights as such:
conv = nn.Conv2d(in_channels=1,
out_channels=1,
kernel_size=3,
stride=3,
bias=None)
conv.weight.data = kernel
No, I do not think this will improve detection performance.
Detection performance is usually known as "inference," that is, it is the process of running your network on new data where the training labels are unknown. Hard-coding the weights will make absolutely no difference on the test performance of the network, as you still need to compute the convolutions.
We could also ask if it will improve the training performance. Here, too, I expect that the answer is no. One of the reasons that neural networks achieve the high accuracy they do is that they pick up on subtle patterns in the training data. A real x on a real page is very unlikely to align with the pixels you set to 1 in your example. Slight rotations or sub-pixel shifts or even different aspect ratios of the letter will change what the optimal filter will look like.
Indeed, one of the major changes in Machine Learning as we move into the Deep Learning era is that neural networks do a better job of picking the low-level features than a human engineer could do.
But thank you for the question -- just the code snippet of how to hard-code the value of a layer was useful to me!
I need to calculate the eigenvalues of an 8x8-matrix and plot each of the eigenvalues for a symbolic variable occuring in the matrix. For the matrix I'm using I get 8 different eigenvalues where each is representing a function in "W", which is my symbolic variable.
Using python I tried calculating the eigenvalues with Scipy and Sympy which worked kind of, but the results are stored in a weird way (at least for me as a newbie not understanding much of programming so far) and I didn't find a way to extract just one eigenvalue in order to plot it.
import numpy as np
import sympy as sp
W = sp.Symbol('W')
w0=1/780
wl=1/1064
# This is my 8x8-matrix
A= sp.Matrix([[w0+3*wl, 2*W, 0, 0, 0, np.sqrt(3)*W, 0, 0],
[2*W, 4*wl, 0, 0, 0, 0, 0, 0],
[0, 0, 2*wl+w0, np.sqrt(3)*W, 0, 0, 0, np.sqrt(2)*W],
[0, 0, np.sqrt(3)*W, 3*wl, 0, 0, 0, 0],
[0, 0, 0, 0, wl+w0, np.sqrt(2)*W, 0, 0],
[np.sqrt(3)*W, 0, 0, 0, np.sqrt(2)*W, 2*wl, 0, 0],
[0, 0, 0, 0, 0, 0, w0, W],
[0, 0, np.sqrt(2)*W, 0, 0, 0, W, wl]])
# Calculating eigenvalues
eva = A.eigenvals()
evaRR = np.array(list(eva.keys()))
eva1p = evaRR[0] # <- this is my try to refer to the first eigenvalue
In the end I hope to get a plot over "W" where the interesting range is [-0.002 0.002]. For the ones interested it's about atomic physics and W refers to the rabi frequency and I'm looking at so called dressed states.
You're not doing anything incorrectly -- I think you're just caught up since your eigenvalues look so jambled and complicated.
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
W = sp.Symbol('W')
w0=1/780
wl=1/1064
# This is my 8x8-matrix
A= sp.Matrix([[w0+3*wl, 2*W, 0, 0, 0, np.sqrt(3)*W, 0, 0],
[2*W, 4*wl, 0, 0, 0, 0, 0, 0],
[0, 0, 2*wl+w0, np.sqrt(3)*W, 0, 0, 0, np.sqrt(2)*W],
[0, 0, np.sqrt(3)*W, 3*wl, 0, 0, 0, 0],
[0, 0, 0, 0, wl+w0, np.sqrt(2)*W, 0, 0],
[np.sqrt(3)*W, 0, 0, 0, np.sqrt(2)*W, 2*wl, 0, 0],
[0, 0, 0, 0, 0, 0, w0, W],
[0, 0, np.sqrt(2)*W, 0, 0, 0, W, wl]])
# Calculating eigenvalues
eva = A.eigenvals()
evaRR = np.array(list(eva.keys()))
# The above is copied from your question
# We have to answer what exactly the eigenvalue is in this case
print(type(evaRR[0])) # >>> Piecewise
# Okay, so it's a piecewise function (link to documentation below).
# In the documentation we see that we can use the .subs method to evaluate
# the piecewise function by substituting a symbol for a value. For instance,
print(evaRR[0].subs(W, 0)) # Will substitute 0 for W
# This prints out something really nasty with tons of fractions..
# We can evaluate this mess with sympy's numerical evaluation method, N
print(sp.N(evaRR[0].subs(W, 0)))
# >>> 0.00222190090611143 - 6.49672880062804e-34*I
# That's looking more like it! Notice the e-34 exponent on the imaginary part...
# I think it's safe to assume we can just trim that off.
# This is done by setting the chop keyword to True when using N:
print(sp.N(evaRR[0].subs(W, 0), chop=True)) # >>> 0.00222190090611143
# Now let's try to plot each of the eigenvalues over your specified range
fig, ax = plt.subplots(3, 3) # 3x3 grid of plots (for our 8 e.vals)
ax = ax.flatten() # This is so we can index the axes easier
plot_range = np.linspace(-0.002, 0.002, 10) # Range from -0.002 to 0.002 with 10 steps
for n in range(8):
current_eigenval = evaRR[n]
# There may be a way to vectorize this computation, but I'm not familiar enough with sympy.
evaluated_array = np.zeros(np.size(plot_range))
# This will be our Y-axis (or W-value). It is set to be the same shape as
# plot_range and is initally filled with all zeros.
for i in range(np.size(plot_range)):
evaluated_array[i] = sp.N(current_eigenval.subs(W, plot_range[i]),
chop=True)
# The above line is evaluating your eigenvalue at a specific point,
# approximating it numerically, and then chopping off the imaginary.
ax[n].plot(plot_range, evaluated_array, "c-")
ax[n].set_title("Eigenvalue #{}".format(n))
ax[n].grid()
plt.tight_layout()
plt.show()
And as promised, the Piecewise documentation.
I am converting some R code (not mine) for estimating the parameters of a choice model to Python. My Python version does not converge onto same parameters as the R version for some test data and I am not sure why.
The R code defines a log-likelihood function (L) and then uses the nlm() function to estimate the parameters:
L <- function(p, y1, m, i1, i0)
-sum(dbinom(y1, m, 1/(1 + i0 %*% p/i1 %*% p), log=TRUE))
out <- nlm(L, s, y1=y1, m=n, i1=idx1, i0=idx0)
For a set of test data this produces parameter estimates:
[1] 0.014302792 0.001703516 0.002347832 0.035365775 0.517465153 0.063503823 0.005776879
In python I have written (what I believe to be) an equivalent log-likelihood function (it returns the same values as R version for test parameters) and tried using scipy.optimize.minimize() in place of nlm():
def LL(p, *args):
y1=args[0]
m=args[1]
i1=args[2]
i0=args[3]
i0p=np.dot(i0,p)
i1p=np.dot(i1,p)
P=1/(1 + np.divide(i0p,i1p))
# y1 are observed successes in pairwise comparison experiment
# m the number of trials, P the probability of success in one trial.
# I'm fairly sure these inputs are the same in python and R versions
return -np.sum(stats.binom.logpmf(y1, m, P))
out = scipy.optimize.minimize(LL, s, args=(y1,n,idx1,idx0))
However, on running, minimize() seems to be unsuccessful:
out:
status: 2
success: False
njev: 21
nfev: 201
hess_inv: array([[1, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0],
[0, 0, 1, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0],
[0, 0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 0, 1]])
fun: -273.75549396685
x: array([ 0.14285714, 0.14285714, 0.14285714, 0.14285714, 0.14285714,
0.14285714, 0.14285714])
message: 'Desired error not necessarily achieved due to precision loss.'
jac: array([ 27.99998093, -552.99998856, -500.49999237, 111.99997711,
671.99995422, 255.49996948, -14.00000381])
Other methods (e.g. 'Powell') report success but parameters are way off those from the example in R.
My questions are:
Elsewhere I've seen that 'Desired error not necessarily achieved due to precision loss.' is a result of badly behaved likelihood function - Can anyone tell is this is the case here? How might I fix it?
Should I try some of the other optimisation methods? They require derivatives to be passed to the minimise() method - How do I define the gradient (and if necessary hessian) for my LL function? I saw an example using statsmodel GenericLikelihoodModel but became confused about exog/endog...