I am trying to understand the FTT and convolution (cross-correlation) theory and for that reason I have created the following code to understand it. The code is Matlab/Octave, however I could also do it in Python.
In 1D:
x = [5 6 8 2 5];
y = [6 -1 3 5 1];
x1 = [x zeros(1,4)];
y1 = [y zeros(1,4)];
c1 = ifft(fft(x1).*fft(y1));
c2 = conv(x,y);
c1 = 30 31 57 47 87 47 33 27 5
c2 = 30 31 57 47 87 47 33 27 5
In 2D:
X=[1 2 3;4 5 6; 7 8 9]
y=[-1 1];
conv1 = conv2(x,y)
conv1 =
24 53 89 29 21
96 140 197 65 42
168 227 305 101 63
Here is where I find the problem, padding a matrix and a vector? How should I do it? I could pad x with zeros around? or just on one side? and what about y? I know that the length of the convolution should be M+L-1 when x and y are vectors, but what about when they are matrices?
How could I continue my example here?
You need to zero-pad one variable with:
As many zero-columns as the number of columns of other variable minus
one.
As many zero-rows as the number of rows of the other variable minus one.
In Matlab, it would look in the following way:
% 1D
x = [5 6 8 2 5];
y = [6 -1 3 5 1];
x1 = [x zeros(1,size(x,2))];
y1 = [y zeros(1,size(y,2))];
c1 = ifft(fft(x1).*fft(y1));
c2 = conv(x,y,'full');
% 2D
X = [1 2 3;4 5 6; 7 8 9];
Y = [-1 1];
X1 = [X zeros(size(X,1),size(Y,2)-1);zeros(size(Y,1)-1,size(X,2)+size(Y,2)-1)];
Y1 = zeros(size(X1)); Y1(1:size(Y,1),1:size(Y,2)) = Y;
c1 = ifft2(fft2(X1).*fft2(Y1));
c2 = conv2(X,Y,'full');
In order to clarify the convolution, look also at this picture:
Related
Lets say we have a set of codes as given below. Currently, we have two parameters whose value are initialized by user input. The output here is a dataframe.
What we want?
Use a function, to create a dataframe with all combinations of X and Y. Lets say X and Y has 4 input values each. Then
Join the output dataframe, df for each combination to get the desired output dataframe.
X= float(input("Enter the value of X: "))
Y = float(input("Enter the value of Y: "))
A= X*Y
B=X*(Y^2)
df = pd.DataFrame({"X": X, "Y": Y, "A": A, "B": B})
Desired output
X Y A B
1 2 2 4
1 4 4 16
1 6 6 36
1 8 8 64
2 2 4 8
2 4 8 32
2 6 12 72
2 8 16 128
3 2 6 12
3 4 12 48
3 6 18 108
3 8 24 192
4 2 8 16
4 4 16 64
4 6 24 144
4 8 32 256
Is this what you were looking for?
def so_help():
x = input('Please enter all X values separated by a comma(,)')
y = input('Please enter all Y values separated by a comma(,)')
#In case anyone gets comma happy
x = x.strip(',')
y = y.strip(',')
x_list = x.split(',')
y_list = y.split(',')
df_x = pd.DataFrame({'X' : x_list})
df_y = pd.DataFrame({'Y' : y_list})
df_cross = pd.merge(df_x, df_y, how = 'cross')
df_cross['X'] = df_cross['X'].astype(int)
df_cross['Y'] = df_cross['Y'].astype(int)
df_cross['A'] = df_cross['X'].mul(df_cross['Y'])
df_cross['B'] = df_cross['X'].mul(df_cross['Y'].pow(2))
return df_cross
so_help()
I have matrix (3 x 15) dummies with sequences of tokens as rows:
[[ 1 66 67 68 0 0 0 0 0 0 0 0 0 0 0]
[ 1 66 67 66 68 66 67 66 0 0 0 0 0 0 0]
[ 1 66 67 68 18 19 20 21 22 23 24 25 26 17 0]]
Also, there's a tensor probs of shape (3 x 15 x n_tokens) with token probabilities.
From probs I need to select only probabilities of tokens in dummies.
I think, it may be possible to use the matrix as indices for the tensor, but I haven't found how to do that.
You can do that like this:
import tensorflow as tf
dummies = ...
probs = ...
s = tf.shape(dummies)
i = tf.range(s[0])
j = tf.range(s[1])
ii, jj = tf.meshgrid(i, j, indexing='ij')
idx = tf.stack([ii, jj, dummies], axis=-1)
result = tf.gather_nd(probs, idx)
I'm very new to python and just coding in general, I'm trying to complete this assignment and there's just one more thing I need help with to finish it.
The task is to generate a square matrix with user input dimensions, and from that matrix create a new one, by removing rows and columns on intersections of which there is an element which is an absolute maximum of the elements of the matrix.
Here's what I got so far:
import numpy as np
print ("input number of rows in rmatr:")
n = int(input())
print ("input number of columns rmatr:")
m = int(input())
def form_rmatr():
rmatr = np.ndarray(shape=(n,m),dtype=int)
for i in range(n):
for j in range(m):
rmatr[i,j] = np.random.randint(-50,50)
return rmatr
a = form_rmatr()
print (a)
b=np.abs(np.max(a))
print ("absolute max value of rmatr = ", b)
max = (0,0)
for i in range(n):
for j in range(m):
if np.abs(a[i,j]) == np.abs(b):
max = i, j
new_a = np.delete(np.delete(a,[i],0),[j],1)
print(new_a)
Now, it does work, but it removes only one intersection, the first one it finds an absolute max value. I need it to remove all intersections. I tried making a while statement instead of if, but obviously, the loop just goes forever since it's searching for absolute max values in the original a matrix. The solution I need is probably to input conditions inside the np.delete function. Something along the lines np.delete(np.where...) , but I have no idea how to actually write it down.
Edit: an example of what it does would be
input number of rows in rmatr rmatr:
8
input number of columns rmatr:
8
[[ 29 -24 -42 14 12 18 -23 44]
[-50 9 -41 -3 -14 30 11 -33]
[ 14 -22 -43 -12 35 42 3 48]
[-26 34 23 -9 47 -5 -33 6]
[-33 29 0 -32 -26 24 -31 1]
[ 15 -31 -40 1 47 30 33 -41]
[ 48 -41 9 44 -4 0 17 -3]
[-32 -23 31 5 -35 3 8 -31]]
absolute max value of rmatr = 48
[[-24 -42 14 12 18 -23 44]
[ 9 -41 -3 -14 30 11 -33]
[-22 -43 -12 35 42 3 48]
[ 34 23 -9 47 -5 -33 6]
[ 29 0 -32 -26 24 -31 1]
[-31 -40 1 47 30 33 -41]
[-23 31 5 -35 3 8 -31]]
It deletes a row and column at intersections of which the number 48 is.
What I need is for it to delete all intersections of rows and columns where a number 48 or -48 are. So seeing as there is one more intersection like that, I need it to look like:
[[-24 -42 14 12 18 -23 ]
[ 9 -41 -3 -14 30 11 ]
[ 34 23 -9 47 -5 -33 ]
[ 29 0 -32 -26 24 -31 ]
[-31 -40 1 47 30 33 ]
[-23 31 5 -35 3 8 ]]
NumPy is designed to allow you to vectorize your computations. This generally means you should rarely if ever need native Python for-loops. Here is a short definition of vectorization.
You can do this in 5 lines of code:
a = np.random.randint(-50, 50, size=(n, m), dtype=int)
ne = np.abs(a) != np.abs(a).max()
cols = np.nonzero(ne.all(axis=0))[0]
rows = np.nonzero(ne.all(axis=1))[0]
new_a = a[rows[:, None], cols]
print(a)
[[ -2 20 10 10 -25]
[-15 -24 22 -43 -37]
[-48 29 23 -16 23]
[-26 -25 1 -48 -32]
[ 22 15 -24 -24 -40]]
print(new_a)
[[ 20 10 -25]
[-24 22 -37]
[ 15 -24 -40]]
Here's a walkthrough of the above:
Instead of creating a with a nested for-loop, we can specify its size (really, its shape) directly as a tuple. It is an (n x m) array.
ne is an array that is the same shape as a. It is False only in places that meet your condition that a given cell's maximum value is equal to the maximum absolute value of the entire array. (If I'm not interpreting that right, you should be able to revise easily.)
[[ True True True True True]
[ True True True True True]
[False True True True True]
[ True True True False True]
[ True True True True True]]
Now we need the indexes of rows and columns, respectively, that contain all Trues.
print(rows)
[0 1 4]
print(cols)
[1 2 4]
Finally, you can use a bit of advanced indexing to slice a on both of these 1-dimensional arrays at once.
Hope this works-
import numpy as np
print ("input number of rowsm/columns in rmatr:")
n = int(input())
m = n
def form_rmatr():
rmatr = np.ndarray(shape=(n,m),dtype=int)
for i in range(n):
for j in range(m):
rmatr[i,j] = np.random.randint(-50,50)
return rmatr
a = form_rmatr()
print (a)
b=np.abs(a).max()
print(b)
print ("absolute max value of rmatr = ", b)
max_rows=[]
max_cols=[]
for i in range(0,n):
for j in range(0,m):
if abs(a[i][j])==b:
max_rows.append(i)
max_cols.append(j)
a=np.delete(a,max_rows, axis=0)
a=np.delete(a,max_cols, axis=1)
print(a)
I have data of the form in a text file.
Text file entry
#x y z
1 1 1
2 4
3 9
4 16
5 25
6 36
7 49
8 64 512
9 81 729
10 100 1000
11 121
12 144 1728
13 169
14 196
15 225
16 256 4096
17 289
18 324
19 361 6859
20 400
21 441 9261
22 484
23 529 12167
24 576
25 625
Some of the entries in the third column are empty. I am trying to create an array of x (column 1) and z (column 3) ignoring nan. Let the array be B. The contents of B should be:
1 1
8 512
9 729
10 1000
12 1728
16 4096
19 6859
21 9261
23 12167
I tried doing this using the code:
import numpy as np
A = np.genfromtxt('data.dat', comments='#', delimiter='\t')
B = []
for i in range(len(A)):
if ~ np.isnan(A[i, 2]):
B = np.append(B, np.column_stack((A[i, 0], A[i, 2])))
print B.shape
This does not work. It creates a column vector. How can this be done in Python?
Using pandas would make your life quite easier (note the regular expression to define delimiter):
from pandas import read_csv
data = read_csv('data.dat', delimiter='\s+').values
print(data[~np.isnan(data[:, 2])][:, [0, 2]])
Which results in:
array([[ 8.00000000e+00, 5.12000000e+02],
[ 9.00000000e+00, 7.29000000e+02],
[ 1.00000000e+01, 1.00000000e+03],
[ 1.20000000e+01, 1.72800000e+03],
[ 1.60000000e+01, 4.09600000e+03],
[ 1.90000000e+01, 6.85900000e+03],
[ 2.10000000e+01, 9.26100000e+03],
[ 2.30000000e+01, 1.21670000e+04]])
If you read your data.dat file and assign the content to a variable, say data:
You can iterate over the lines and split them and process only the ones that have 3 elements:
B=[]
for line in data.split('\n'):
if len(line.split()) == 3:
x,y,z = line.split()
B.append((x,z)) # or B.append(str(x)+'\t'+str(z)+'\n')
# or any othr format you need
Not always the functions provided by the libraries are easy to use, as you found out. The following program does it manually, and creates an array with the values from the datafile.
import numpy as np
def main():
B = np.empty([0, 2], dtype = int)
with open("data.dat") as inf:
for line in inf:
if line[0] == "#": continue
l = line.split()
if len(l) == 3:
l = [int(d) for d in l[1:]]
B = np.vstack((B, l))
print B.shape
print B
return 0
if __name__ == '__main__':
main()
Note that:
1) The append() function works on lists, not on arrays - at least not in the syntax you used. The easiest way to extend arrays is 'piling' rows, using vstack (or hstack for columns)
2) Specifying a delimiter in genfromtxt() can come to bite you. By default the delimiter is any white space, which is normally what you want.
From your input dataframe:
In [33]: df.head()
Out[33]:
x y z
0 1 1 1
1 2 4 NaN
2 3 9 NaN
3 4 16 NaN
4 5 25 NaN
.. you can get to the output dataframe B by doing this :
In [34]: df.dropna().head().drop('y', axis=1)
Out[34]:
x z
0 1 1
7 8 512
8 9 729
9 10 1000
11 12 1728
I have a numpy array like this:
foo_array = [38,26,14,55,31,0,15,8,0,0,0,18,40,27,3,19,0,49,29,21,5,38,29,17,16]
I want to replace all the zeros with the median value of the whole array (where the zero values are not to be included in the calculation of the median)
So far I have this going on:
foo_array = [38,26,14,55,31,0,15,8,0,0,0,18,40,27,3,19,0,49,29,21,5,38,29,17,16]
foo = np.array(foo_array)
foo = np.sort(foo)
print "foo sorted:",foo
#foo sorted: [ 0 0 0 0 0 3 5 8 14 15 16 17 18 19 21 26 27 29 29 31 38 38 40 49 55]
nonzero_values = foo[0::] > 0
nz_values = foo[nonzero_values]
print "nonzero_values?:",nz_values
#nonzero_values?: [ 3 5 8 14 15 16 17 18 19 21 26 27 29 29 31 38 38 40 49 55]
size = np.size(nz_values)
middle = size / 2
print "median is:",nz_values[middle]
#median is: 26
Is there a clever way to achieve this with numpy syntax?
Thank you
This solution takes advantage of numpy.median:
import numpy as np
foo_array = [38,26,14,55,31,0,15,8,0,0,0,18,40,27,3,19,0,49,29,21,5,38,29,17,16]
foo = np.array(foo_array)
# Compute the median of the non-zero elements
m = np.median(foo[foo > 0])
# Assign the median to the zero elements
foo[foo == 0] = m
Just a note of caution, the median for your array (with no zeroes) is 23.5 but as written this sticks in 23.
foo2 = foo[:]
foo2[foo2 == 0] = nz_values[middle]
Instead of foo2, you could just update foo if you want. Numpy's smart array syntax can combine a few lines of the code you made. For example, instead of,
nonzero_values = foo[0::] > 0
nz_values = foo[nonzero_values]
You can just do
nz_values = foo[foo > 0]
You can find out more about "fancy indexing" in the documentation.