I receive arrays of 9 doubles via tcp, split them, convert with np.array() method, store them in list, and finally again convert this list to numpy array, so i could save it, and later load to train keras model.
Each time I re-run code, shape of output is (500, 6) or (500,) randomly,
I don't change anything, just keep running same code and I get different results.
How is that possible?
How to convert (n,) to (n, 6)? I Tried with .reshape()
Edit: my full code:
def tcp_server_get_training_data():
host = "localhost"
port = 5367
msg = ""
mySocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mySocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
mySocket.bind((host, port))
# print(socket.getfqdn())
# print(socket.gethostbyname(socket.getfqdn()))
mySocket.listen(1)
print('waiting for connection...')
conn, addr = mySocket.accept()
print("Connection from: " + str(addr))
x = []
y = []
i = 0
n = 500
while i < n:
data = conn.recv(128)
doubles_sequence = array.array('d', data)
doubles_sequence2 = np.array(doubles_sequence)
x.append(doubles_sequence2[:6])
y.append(doubles_sequence2[-3:])
i += 1
print(str(round((i/n)*100, 2))+"%")
#print(doubles_sequence[:6])
xp = np.array(x)
yp = np.array(y)
print("X shape: "+str(xp.shape))
print("y shape: "+str(yp.shape))
np.save(file='x', arr=xp)
np.save(file='y', arr=yp)
conn.close()
when I print x with shape (500,6) i get:
>>> x.shape
(500, 6)
>>> x
array([[ 0. , 0. , 0. , -0.29219246, 0. , 0. ],
[ 0. , 0. , 0. , 0.34277358, 0. , 0. ],
[ 0. , 0. , 0. , 0.34277358, 0. , 0. ],
when I print x with shape (500,) i get:
>>> x.shape
(500,)
>>> x
array([array([ 0., 0. , 0. , -0.29219246, 0., 0. ]),
array([ 0. , 0. , 0. , 0.34277358, 0., 0.]),
array([ 0. , 0., 0., -0.10241638, , dtype=object)
I'd really appreciate some help, I tried to figure it out on my own, but after few hours just get frustrated.
I'm relatively new in programming and spend more time on C#, in python I'm very confused without type declaration.
Related
I'm currently trying to develop a function that performs matrix multiplication while expanding a differential equation with odeint in Python and am seeing strange results.
I converted the function:
def f(x, t):
return [
-0.1 * x[0] + 2 * x[1],
-2 * x[0] - 0.1 * x[1]
]
to the below so that I can incorporate different matrices.
I have the below matrix of values and function that takes specific values of that matrix:
from scipy.integrate import odeint
x0_train = [2,0]
dt = 0.01
t = np.arange(0, 1000, dt)
matrix_a = np.array([-0.09999975, 1.999999, -1.999999, -0.09999974])
# Function to run odeint with
def f(x, t, a):
return [
a[0] * x[0] + a[1] * x[1],
a[2] * x[0] - a[3] * x[1]
]
odeint(f, x0_train, t, args=(matrix_a,))
>>> array([[ 2. , 0. ],
[ 1.99760115, -0.03999731],
[ 1.99440529, -0.07997867],
...,
[ 1.69090227, 1.15608741],
[ 1.71199436, 1.12319701],
[ 1.73240339, 1.08985846]])
This seems right, but when I create my own function to perform multiplication/regression, I see the results at the bottom of the array are completely different. I have two sparse arrays that provide the same conditions as matrix_a but with zeros around them.
from sklearn.preprocessing import PolynomialFeatures
new_matrix_a = array([[ 0. , -0.09999975, 1.999999 , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. ],
[ 0. , -1.999999 , -0.09999974, 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. ]])
# New function
def f_new(x, t, parameters):
polynomials = PolynomialFeatures(degree=5)
x = np.array(x).reshape(-1,2)
#x0_train_array_reshape = x0_train_array.reshape(1,2)
polynomial_transform = polynomials.fit(x)
polynomial_features = polynomial_transform.fit_transform(x).T
x_ode = np.matmul(parameters[0],polynomial_features)
y_ode = np.matmul(parameters[1],polynomial_features)
return np.concatenate((x_ode, y_ode), axis=None).tolist()
odeint(f_new, x0_train, t, args=(new_matrix_a,))
>>> array([[ 2.00000000e+00, 0.00000000e+00],
[ 1.99760142e+00, -3.99573216e-02],
[ 1.99440742e+00, -7.98188169e-02],
...,
[-3.50784051e-21, -9.99729456e-22],
[-3.50782881e-21, -9.99726119e-22],
[-3.50781711e-21, -9.99722781e-22]])
As you can see, I'm getting completely different values at the end of the array. I've been running through my code and can't seem to find a reason why they would be different. Does anybody have a clear reason why or if I'm doing something wrong with my f_new? Ideally, I'd like to develop a function that can take any values in that matrix_a, which is why I'm trying to create this new function.
Thanks in advance.
You should perhaps use numpy even more in the first version, to avoid sign errors in routine algorithms.
def f(x, t, a):
return a.reshape([2,2]) # x # or use matmul, or a.reshape([2,2]).dot(x)
or, for efficiency, pass the already reshaped a.
So I have an array XsN of shape (590,) and I am trying to standardise the data.
This is an example of one of the 590 elements in my array:
print(XsN[:1])
[array([[ 0. , 0.27229556, -1.8033657 , ..., 0. ,
0. , 0. ],
[ 0. , 0.20665401, -1.9340569 , ..., 0. ,
0. , 0. ],
[ 4. , 0. , 0.04352444, ..., 0. ,
0. , 0. ],
...,
[10. , 0. , -0.5655 , ..., 0. ,
0. , 0. ],
[10. , 0. , 0.9150001 , ..., 0. ,
0. , 0. ],
[10. , 0. , 1.0005 , ..., 0. ,
0. , 0. ]], dtype=float32)]
I'm then reshaping it so that it has shape (590,1):
XsN_2 = XsN.reshape(-1,1)
Now when I use StandardScaler:
from sklearn.preprocessing import StandardScaler
standardized_data = StandardScaler().fit_transform(XsN_2)
I get the error that
TypeError: only size-1 arrays can be converted to Python scalars
and
ValueError: setting an array element with a sequence.
I understand it tries to find a number but instead it finds an ndarray but I'm not quite sure how to standardise data of shape (590,) where each element is its own ndarray.
Edit 1:
Referring to this csv file: https://gofile.io/?c=YGxCWQ
Here is some code with a sample data:
import pandas as pd
from sklearn.preprocessing import StandardScaler
imp = pd.read_csv('foo.csv', sep=',', header=None)
data = imp.values
print(data)
standardized_data = StandardScaler().fit_transform(data)
The error I get now is:
ValueError: could not convert string to float
Is there any way I can standardise this data?
Without access to your original data in the form of a valid .csv file it is a little difficult to debug this. From the look of what you printed it seems like XsN is a list of arrays, so you may want to loop through each in turn or convert it into an array with expanded dimensions.
Here is an example of standardizing some dummy data which I think resembles the structure of your data. Hope that helps.
n = 100
# Create feature 1
mean1 = 10
standard_dev1 = 2
col1 = np.random.normal(loc=mean1,scale=standard_dev1,size=[n,1])
# Create feature 2
mean2 = 20
standard_dev2 = 4
col2 = np.random.normal(loc=mean2,scale=standard_dev2,size=[n,1])
data = np.concatenate([col1,col2],axis=1)
print(f"means of raw data: {data.mean(axis=0)}")
>>>
means of raw data: [10.15783287 19.82541124]
print(f"standard devations of raw data: {data.std(axis=0)}")
>>>
standard devations of raw data: [2.00049111 3.87277793]
from sklearn.preprocessing import StandardScaler
standardized_data = StandardScaler().fit_transform(data)
print(f"means of standardized data: {standardized_data.mean(axis=0)}")
>>>
means of standardized data: [-6.92779167e-16 -1.78745907e-15]
print(f"standard devations of standardized data: {standardized_data.std(axis=0)}")
>>>
standard devations of standardized data: [1. 1.]
I have been given a .mat file which is 1024*1024*360 i.e., a 3D object. I have divided the data in to three .mat files A,B and C. All three of them are 1024*1024*120 . I am loading them to a matrix 'mat' which is 1024*360 . I am loading each one of them one by one and then deleting them to make space. Basically it's just a 2D slice of the 3D object at the point 240. Later I am trying to plot the image. Following is my code :
import scipy.io
import numpy as np
mat = np.zeros((1024,360))
x = scipy.io.loadmat('/home/imaging/Desktop/PRAKRITI/Project/A.mat')
x = x.values()
mat[:,0:120]= x[240,:,:]
del x
y = scipy.io.loadmat('/home/imaging/Desktop/PRAKRITI/Project/B.mat')
y = y.values()
mat[:,120:240]= y[240,:,:]
del y
z = scipy.io.loadmat('/home/imaging/Desktop/PRAKRITI/Project/C.mat')
z = z.values()
mat[:,240:360]= z[240,:,:]
del z
import matplotlib.py as plt
imageplot = plt.imshow(matrix)
I am getting this error :
mat[:,0:120]= x[240,:,:]
TypeError: List indices must be integers, not tuple
Can anyone suggest what I am doing wrong here?
You have to create a numpy array from the original x matrix.
This is why the normal python array doesn't accept the numpy type fancy indexing, like matrix[x,y,z] only like matrix[x][y][z].
import scipy.io
import numpy as np
mat = np.zeros((1024,360))
x = scipy.io.loadmat('/home/imaging/Desktop/PRAKRITI/Project/A.mat')
x = np.array((x.values()))
mat[:,0:120]= x[240,:,:]
del x
y = scipy.io.loadmat('/home/imaging/Desktop/PRAKRITI/Project/B.mat')
y = np.array((y.values()))
mat[:,120:240]= y[240,:,:]
del y
z = scipy.io.loadmat('/home/imaging/Desktop/PRAKRITI/Project/C.mat')
z = np.array((z.values()))
mat[:,240:360]= z[240,:,:]
del z
import matplotlib.py as plt
imageplot = plt.imshow(matrix)
Alternately you can use x[240][:][:] instead of x[240,:,:]
Glad to have been of help! Feel free to accept my answer if you feel it was useful to you. :-)
continuing:
Because the following code worked fine, i guess the problem is somewhere at the loaded matrixs' dimensions i.e. x.values() etc. So please check it first, with print x.shape().
import numpy as np
mat = np.zeros((1024,360))
x = np.zeros((1024,1024,120))
mat[:,0:120] = x[240,:,:]
print mat
[[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
...,
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]]
I am tying to get the physical sky coordinates of a given pixel from within a python script. I would like to use astropy's WCS, but I'll do anything from within python.
I have tried these two snippets of code.
from astropy.io import fits
from astropy.wcs import WCS
def astropymethod1(img):
# from http://astropy.readthedocs.org/en/latest/wcs/
w = WCS(img)
lon, lat = w.all_pix2world( 100., 100., 1)
print lon, lat
def astropymethod2(img):
# from http://astropy.readthedocs.org/en/latest/wcs/
hdu = fits.open(img)
w = WCS(hdu[0].header)
lon, lat = w.wcs_pix2world(100., 100., 1)
print lon, lat
The issues are I get an error the first time I try to use WCS and the result is only ever the pixel values I put in.
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
The problem is that you have a multi-extension FITS file. Here's an example session showing how you can get access to the appropriate WCS:
In [1]: from astropy.io import fits
In [2]: h = fits.getheader('SN1415_F625W_1_drz.fits')
In [3]: f = fits.open('SN1415_F625W_1_drz.fits')
In [4]: f
Out[4]:
[<astropy.io.fits.hdu.image.PrimaryHDU at 0x106735490>,
<astropy.io.fits.hdu.image.ImageHDU at 0x106749750>,
<astropy.io.fits.hdu.image.ImageHDU at 0x106751310>,
<astropy.io.fits.hdu.image.ImageHDU at 0x106751d10>,
<astropy.io.fits.hdu.table.BinTableHDU at 0x1067dfdd0>]
In [5]: from astropy import wcs
In [6]: w = wcs.WCS(f[0].header)
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
In [7]: w.wcs.naxis
Out[7]: 2
In [8]: f[0].data
In [9]: w = wcs.WCS(f[1].header)
In [10]: w.wcs.naxis
Out[10]: 2
In [11]: f[1].data
Out[11]:
array([[ 0.01986978, -0.04018363, 0.03330525, ..., 0. ,
0. , 0. ],
[ 0.0695872 , -0.00979143, 0.00147662, ..., 0. ,
0. , 0. ],
[-0.09292094, 0.02481506, -0.01057338, ..., 0. ,
0. , 0. ],
...,
[ 0. , 0. , 0. , ..., 0.02375774,
0.0389731 , 0.03825707],
[ 0. , 0. , 0. , ..., -0.01570918,
-0.01053802, 0.00461219],
[ 0. , 0. , 0. , ..., -0.0638448 ,
-0.0240754 , 0.02679451]], dtype=float32)
In [12]: w.wcs_pix2world(100., 100., 1)
Out[12]: [array(6.113076380801787), array(0.616758775753701)]
So you probably want to redefine your method:
def astropymethod2(img, hduid=1):
# from http://astropy.readthedocs.org/en/latest/wcs/
hdu = fits.open(img)
w = WCS(hdu[hduid].header)
lon, lat = w.wcs_pix2world(100., 100., 1)
print lon, lat
I am solving simply linear problem A*x=b by using conjugate gradient method. I want to find x unknown.
Note that conjGrad calls the function Av that returns the product Av
The code is given below:
Inputs:
A - sparse matrix. 2D array;
b - right hand-side vector. 1D array;
x - initial guess. Here, it is just 1D array with zero values.
Code:
import numpy as np
import math
A = np.array([[ 0.56244579, 0. , 0. , 0. , 0.52936075,
0.59553084, 0. , 0. , 0. , 1.1248915 ,
0. , 0. , 0. , 0.46319065, 0.43672262,
0. ],
[ 0.5 , 1. , 1. , 0.5 , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. ],
[ 0. , 0. , 0. , 0.58009067, 0. ,
0. , 0.75411788, 0.40606347, 0. , 0. ,
0.23203627, 0. , 0. , 0. , 0. ,
0. ]])
x = np.array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0.])
b = np.array([ 3.99464617, 1.81663614, 1.86413003])
def Av(v):
return np.dot(A,v)
def conjGrad(Av, x, b, tol=1.0e-9):
n = len(b)
r = b - Av(x)
s = r.copy()
for i in range(n):
u = Av(s)
alpha = np.dot(s,r)/np.dot(s,u)
x = x + aplha*s
r = b - Av(x)
if(math.sqrt(np.dot(r,r))) < tol:
break
else:
beta = - np.dot(r,u)/np.dot(s,u)
s = r + beta * s
return x,i
if __name__ == '__main__':
x, iter_number = conjGrad(Av, x, b)
Traceback (most recent call last):
File "C:\Python27\Conjugate_Gradient.py", line 59, in <module>
x, iter_number = conjGrad(Av, x, b)
File "C:\Python27\Conjugate_Gradient.py", line 47, in conjGrad
u = Av(s)
File "C:\Python27\Conjugate_Gradient.py", line 40, in Av
return np.dot(A,v)
ValueError: matrices are not aligned
Is there any simple solution to avoid this message? Any answers will be appreciated
You have implemented the CG method wrong. The error message shows you one of the lines where there is a problem.
In particular, your matrix is not square.
The conjugate gradients method solves for Ax=b when A is SPD.
If A is not SPD, like in your case, then you can still use conjugate gradients to find the least squares solution for your problem:
A^t A x = A^t b
The matrix A^t A is SPD and well suited for your method.