We Keep Coding

Python matrix column cyclic assignment

I have a matrix $X=(X_{ij})\in R^{n\times d}$ and a vector $y=(y_i)\in R^{n}$ and I want to centralize $y$ and centralize and normalize $X$ in columns , i.e., $\sum_{i=1}^{n} y_i=0$, $\sum_{i=1}^{n} X_{ij}=0$ 和 $\sum_{i=1}^{n} X_{ij}^2=1$, $j=1,2,\ldots,d$.
I have implemented it in R successfully, with a toy example as follows (PS: My toy example is very simple that can be manually computed and checked! )
### wirte the function
normalize = function(X,y){
n = dim(X)[1]; d = dim(X)[2]
D = matrix(0,d,1)
y = y - mean(y) #center y
for (k in 1:d){
Xk = X[,k]
Xk = Xk - mean(Xk) #column-center X
ck = 1/norm(Xk,"2")
X[,k] = Xk * ck #column-normalization X
D[k,1] = ck
}
out = list(X=X,D=D,y=y)
return(out)
}
### test the function
X = matrix(1:4,2,byrow=T)
y = matrix(1:2)
out = normalize(X,y)
( ny = out$y ) #show centered y
mean(ny) #check y center
( nX = out$X ) #show centered and normalized X (each column)
apply(nX,2,mean) #check X column-center
apply(nX,2,function(x)norm(x,"2")) #check X column-normalization
Then I want to reproduce the R procedure in Python as strictly as possible and try it as
### wirte the function
import numpy as np
def normalize(X,y):
n,d = X.shape
D = np.zeros((d,1))
y = y-np.mean(y) #center y
for k in range(d):
Xk = X[:,k]
Xk = Xk - np.mean(Xk) #column-center X
ck = 1/np.linalg.norm(Xk)
X[:,k] = Xk * ck #column-normalization X
D[k,:] = ck
return X,D,y
### test the function
X = np.arange(1,5,1).reshape(2,2)
y = np.array([1,2]).reshape(-1,1)
nX,D,ny = normalize(X,y)
ny #show centered y
np.mean(ny) #check y center
nX #show centered and normalized X (each column)
np.mean(nX,axis=0) #check X column-center
np.linalg.norm(nX, axis=0) #check X column-normalization
The results of ny and D in Python matches that in R. However, the nX output in Python gives obviously the wrong result, which shows the zero matrix! Did I missing something?

This happens because when you are creating the X matrix with np.arange, numpy infers the dtype of the matrix to be int. Hence, when you are setting the values with 1/sqrt(2) they get truncated to 0.
Declare X as follows:
X = np.arange(1, 5, 1, dtype=np.float64).reshape(2, 2)
Resulting X:
array([[-0.70710678, -0.70710678],
[ 0.70710678, 0.70710678]])
As a side note, you can create a function that returns the same result and does not use a loop by using vectorised numpy functions:
def normalize(X, y):
Xk = X - np.mean(X, axis=0)
D = 1/np.linalg.norm(Xk, axis=0)
return Xk * D, D.reshape(-1, 1), y - np.mean(y)
Anothet difference is in side-effects. This function does not modify the original X matrix, while yours does.

Generating random points in a box

I want to generate random points in a box (a=0.2m, b=0.2m, c=1m). This points should have random distance between each other but minimum distance between two points is should be 0.03m, for this I used random.choice. When I run my code it generates random points but distance management is so wrong. Also my float converting approximation is terrible because I don't want to change random values which I generate before but I couldn't find any other solution. I'm open to suggestions.
Images
graph1
graph2
import random
import matplotlib.pyplot as plt
# BOX a = 0.2m b=0.2m h=1m
save = 0 #for saving 3 different plot.
for k in range(3):
pointsX = [] #information of x coordinates of points
pointsY = [] #information of y coordinates of points
pointsZ = [] #information of z coordinates of points
for i in range(100): #number of the points
a = random.uniform(0.0,0.00001) #for the numbers generated below are float.
x = random.choice(range(3, 21,3)) #random coordinates for x
x1 = x/100 + a
pointsX.append(x1)
y = random.choice(range(3, 21,3)) #random coordinates for y
y1 = y/100 + a
pointsY.append(y1)
z = random.choice(range(3, 98,3)) #random coordinates for z
z1 = z/100 + a
pointsZ.append(z1)
new_pointsX = list(set(pointsX)) # deleting if there is a duplicates
new_pointsY = list(set(pointsY))
new_pointsZ = list(set(pointsZ))
# i wonder max and min values it is or not between borders.
print("X-Min", min(new_pointsX))
print("X-Max", max(new_pointsX))
print("Y-Min", min(new_pointsY))
print("Y-Max", max(new_pointsY))
print("Z-Min", min(new_pointsZ))
print("Z-Max", max(new_pointsZ))
if max(new_pointsX) >= 0.2 or max(new_pointsY) >= 0.2:
print("MAX VALUE GREATER THAN 0.2")
if max(new_pointsZ) >= 0.97:
print("MAX VALUE GREATER THAN 0.97")
#3D graph
fig = plt.figure(figsize=(18,9))
ax = plt.axes(projection='3d')
ax.set_xlim([0, 0.2])
ax.set_ylim([0, 0.2])
ax.set_zlim([0, 1])
ax.set_title('title',fontsize=18)
ax.set_xlabel('X',fontsize=14)
ax.set_ylabel('Y',fontsize=14)
ax.set_zlabel('Z',fontsize=14)
ax.scatter3D(new_pointsX, new_pointsY, new_pointsZ);
save += 1
plt.savefig("graph" + str(save) + ".png", dpi=900)

As mentioned in the comments by #user3431635, you can check each point with all previous points before appending that new point to the list. I would do that something like this:
import random
import numpy as np
import matplotlib.pyplot as plt
plt.close("all")
a = 0.2 # x bound
b = 0.2 # y bound
c = 1.0 # z bound
N = 1000 # number of points
def distance(p, points, min_distance):
"""
Determines if any points in the list are less than the minimum specified
distance apart.
Parameters
----------
p : tuple
`(x,y,z)` point.
points : ndarray
Array of points to check against. `x, y, z` points are columnwise.
min_distance : float
Minimum allowable distance between any two points.
Returns
-------
bool
True if point `p` is at least `min_distance` from all points in `points`.
"""
distances = np.sqrt(np.sum((p+points)**2, axis=1))
distances = np.where(distances < min_distance)
return distances[0].size < 1
points = np.array([]) # x, y, z columnwise
while points.shape[0] < 1000:
x = random.choice(np.linspace(0, a, 100000))
y = random.choice(np.linspace(0, b, 100000))
z = random.choice(np.linspace(0, c, 100000))
p = (x,y,z)
if len(points) == 0: # add first point blindly
points = np.array([p])
elif distance(p, points, 0.03): # ensure the minimum distance is met
points = np.vstack((points, p))
fig = plt.figure(figsize=(18,9))
ax = plt.axes(projection='3d')
ax.set_xlim([0, a])
ax.set_ylim([0, b])
ax.set_zlim([0, c])
ax.set_title('title',fontsize=18)
ax.set_xlabel('X',fontsize=14)
ax.set_ylabel('Y',fontsize=14)
ax.set_zlabel('Z',fontsize=14)
ax.scatter(points[:,0], points[:,1], points[:,2])
Note, this might not be the randomness you're looking for. I have written it to take the range of x, y, and z values and split it into 100000 increments; a new x, y, or z point is then chosen from those values.

Calculating random sample points using polar coordinates on cartesian map

I'm trying to generate random sample points on a cartesian plane using polar coordinates. I have a cartesian map with polar sectors, I'd like to put a random sample point within each of the sectors.
Problem Visual Description
I've added a sample point in the first sector. The problem is I don't know how to set the min and max limits for each sector as it's a cartesian plane (using cartesian min and max of the sector corners will give you boxes instead of the entire polar sector).
Code is commented for clarity. Final output posted below.
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [10, 10]
import math
import pylab as pl
from matplotlib import collections as mc
import pprint
from IPython.utils import io
from random import randrange, uniform
#convertes cartesian x,y coordinates to polar r, theta coordinates
def cart2pol(x, y):
rho = np.sqrt(x**2 + y**2)
phi = np.arctan2(y, x)
return np.array([rho, phi])
#convertes polar r,theta coordinates to cartesian x,y coordinates
def pol2cart(r, theta): #r is distance
x = r * np.cos(theta)
y = r * np.sin(theta)
return np.array([x, y])
#cooks delicious pie
pi = np.pi
#no idea what this does
theta = np.linspace(0,2*pi,100)
#x theta
def x_size(r):
return r*np.cos(theta)
#y theta
def y_size(r):
return r*np.sin(theta)
#calculates distribution of sectors on a circle in radians
#eg. sub_liner(3) = array([0. , 2.0943951, 4.1887902])
def sub_liner(k):
sub_lines = []
for c,b in enumerate(range(0,k)):
sub_lines = np.append(sub_lines,((12*pi/6)/k)*c)
return sub_lines
#calculates all distribution sectors for every ring and puts them in a list
def mlp(i):
master_lines = []
k = 3
for a in range(0,i):
master_lines.append(sub_liner(k))
k += 3
return master_lines
#calculates all four corners of each sector for a ring
#(ring,ring points,number of rings)
def cg(r,rp,n):
return [[[pol2cart(r-1,mlp(n)[r-1][i])[0],pol2cart(r-1,mlp(n)[r-1][i])[1]]\
,[pol2cart(r,mlp(n)[r-1][i])[0],pol2cart(r,mlp(n)[r-1][i])[1]]] for i in range(0,rp)]
#generates all corners for the ring sectors
def rg(n):
cgl = []
k = 3
for r in range(1,11):
cgl.append(cg(r,k,n))
k += 3
output = cgl[0]
for q in range(1,10):
output = np.concatenate((output,cgl[q]))
return output
#print(cg(1,3,10)[0][0][0])
#print(cg(1,3,10))
# randrange gives you an integral value
irand = randrange(0, 10)
# uniform gives you a floating-point value
frand = uniform(0, 10)
#define ring sectors
ring_sectors = rg(10)
#define node points
nx = 0.5
ny = 0.5
#define ring distance
ymin = [0]
ymax = [1]
#generate rings
ring_r = np.sqrt(1.0)
master_array = np.array([[x_size(i),y_size(i)] for i in range(0,11)])
#plot rings
fig, ax = plt.subplots(1)
[ax.plot(master_array[i][0],master_array[i][1]) for i in range(0,11)]
ax.set_aspect(1)
size = 10
plt.xlim(-size,size)
plt.ylim(-size,size)
#generate nodes
ax.plot(nx, ny, 'o', color='black');
#ring lines
lc = mc.LineCollection(ring_sectors, color='black', linewidths=2)
ax.add_collection(lc)
plt.grid(linestyle='--')
plt.title('System Generator', fontsize=8)
plt.show()
Sample output can be viewed at.
Edit:
What I've tried:
Based on feedback, I implemented a system which gets random uniform values between the polar coordinates, and it works, but the points aren't neatly distributed within their sectors as they should be, and I'm not sure why. Maybe my math is off or I made a mistake in the generator functions. If anyone has any insight, I'm all ears.
Output with points
def ngx(n):
rmin = 0
rmax = 1
nxl = []
s1 = 0
s2 = 1
k = 0
for i in range(0,n):
for a in range(0,rmax*3):
nxl.append(pol2cart(np.random.uniform(rmin,rmax),\
np.random.uniform(sub_liner(rmax*3)[(s1+k)%(rmax*3)],sub_liner(rmax*3)[(s2+k)%(rmax*3)]))[0])
k += 1
rmin += 1
rmax += 1
return nxl
def ngy(n):
rmin = 0
rmax = 1
nyl = []
s1 = 0
s2 = 1
k = 0
for i in range(0,n):
for a in range(0,rmax*3):
nyl.append(pol2cart(np.random.uniform(rmin,rmax),\
np.random.uniform(sub_liner(rmax*3)[(s1+k)%(rmax*3)],sub_liner(rmax*3)[(s2+k)%(rmax*3)]))[1])
k += 1
rmin += 1
rmax += 1
return nyl
#define node points
nx = ngx(10)
ny = ngy(10)

How do I only plot the values I want?

I currently have the code and I having some trouble trying to plot it, I know that trying to plot both ymax and y won't work in this case, but how would I go about plotting just the value for y? I have plotted the function before by removing the ymax from the return, but I need to print the values and plot the solution for y.
import numpy as np
import matplotlib.pyplot as plt
def GaussElimination(A):
'''
Description: Use Gauss elimination to solve a set of simultaneous equations
Parameters: A a matrix of coefficient and constant value for the system
Return: a matrix holding the solution to the equation. This corresponds to the last n
'''
nr,nc=A.shape
B= A.copy()
# start the gauss elimination
for r in range(nr):
#pivoting
max=abs(B[r][r])
maxr = r
for rr in range(r,nr):
if max < abs(B[rr][r]):
max = abs(B[rr][r])
maxr = rr
if max == 0:
print("Singular Matrix")
return []
# swap if needed
if (maxr != r):
for c in range(nc):
temp = B[r][c]
B[r][c]=B[maxr][c]
B[maxr][c] = temp
# scale the row
scale = B[r][r]
for c in range(r,nc):
B[r][c] = B[r][c]/scale
# eliminate values in the columns
for rr in range(nr):
if rr != r:
scale = B[rr][r]
for c in range(r,nc):
B[rr][c]=B[rr][c] - scale*B[r][c]
if (nc == nr+1):
return B[:,nc-1]
else:
return B[:,(nr):nc]
def SimplySupportedBeam(n):
M = np.zeros([n+1,n+1])
C = np.array([[0],[150],[0],[0],[0],[0]])
for r in range(n-3):
M[r][r] = 1
M[r][r+1] = -4
M[r][r+2] = 6
M[r][r+3] = -4
M[r][r+4] = 1
M[n-3][1] = 1
M[n-2][n-1] = 1
M[n-1][n-5] = 1
M[n-1][n-4] = -2
M[n-1][n-3] = 1
M[n][n-2] = 1
M[n][n-1] = -2
M[n][n] = 1
A = np.concatenate((M,C), axis=1)
y0 = GaussElimination(A)
y = y0[1:n]
ymax = np.amax(abs(y))
return y, ymax
n = int(input("Index of the last node: "))
print (SimplySupportedBeam(n))
plt.figure(1)
plt.plot(SimplySupportedBeam(n))
plt.show()
How would I plot just the value I get for y from my code?

It seems like y is 1D numpy array.
If you just want to plot its values against their indices you should be able to do so using either
plt.plot(SimplySupportedBeam(n)[0])
or
y, ymax = SimplySupportedBeam(n)
plt.plot(y)
The problem was that your function returns two values, i.e. y and ymax.
(I did not

How can I vectorize this numpy rotation code?

The Goal:
I would like to vectorize (or otherwise speed up) this code. It rotates a 3d numpy model around its center point (let x,y,z denote the dimensions; then we want to rotate around the z-axis). The np model is binary voxels that are either "on" or "off"
I bet some basic matrix operation could do it, like take a layer and apply the rotation matrix to each element. The only issue with that is decimals; where should I have the new value land since cos(pi / 6) == sqrt(3) / 2?
The Code:
def rotate_model(m, theta):
'''
theta in degrees
'''
n =np.zeros(m.shape)
for i,layer in enumerate(m):
rotated = rotate(layer,theta)
n[i] = rotated
return n
where rotate() is:
def rotate(arr, theta):
'''
Rotates theta clockwise
rotated.shape == arr.shape, unlike scipy.ndimage.rotate(), which inflates size and also does some strange mixing
'''
if theta == int(theta):
theta *= pi / 180
theta = -theta
# theta=-theta b/c clockwise. Otherwise would default to counterclockwise
rotated =np.zeros(arr.shape)
#print rotated.shape[0], rotated.shape[1]
y_mid = arr.shape[0]//2
x_mid = arr.shape[1]//2
val = 0
for x_new in range(rotated.shape[1]):
for y_new in range(rotated.shape[0]):
x_centered = x_new - x_mid
y_centered = y_new - y_mid
x = x_centered*cos(theta) - y_centered*sin(theta)
y = x_centered*sin(theta) + y_centered*cos(theta)
x += x_mid
y += y_mid
x = int(round(x)); y = int(round(y)) # cast so range() picks it up
# lossy rotation
if x in range(arr.shape[1]) and y in range(arr.shape[0]):
val = arr[y,x]
rotated[y_new,x_new] = val
#print val
#print x,y
return rotated

You have a couple of problems in your code. First, if you want to fit the original image onto a rotated grid then you need a larger grid (usually). Alternatively, imagine a regular grid but the shape of your object - a rectangle - is rotated, thus becoming a "rhomb". It is obvious if you want to fit the entire rhomb - you need a larger output grid (array). On the other hand, you say in the code "rotated.shape == arr.shape, unlike scipy.ndimage.rotate(), which inflates size". If that is the case, maybe you do not want to fit the entire object? So, maybe it is OK to do this: rotated=np.zeros(arr.shape). But in general, yeah, one has to have a larger grid in order to fit the entire input image after it is rotated.
Another issue is angle conversion that you are doing:
if theta == int(theta):
theta *= pi / 180
theta = -theta
Why??? What will happen when I want to rotate the image by 1 radian? Or 2 radians? Am I forbidden to use integer number of radians? I think you are trying to do too much in this function and therefore it will be very confusing to do use it. Just require the caller to convert angles to radians. Or, you can do it inside this function if input theta is always in degrees. Or, you can add another parameter called, e.g., units and caller could set it to radians or degrees. Don't try to guess it based on "integer-ness" of input!
Now, let's rewrite your code a little bit:
rotated = np.zeros_like(arr) # instead of np.zero(arr.shape)
y_mid = arr.shape[0] // 2
x_mid = arr.shape[1] // 2
# val = 0 <- this is unnecessary
# pre-compute cos(theta) and sin(theta):
cs = cos(theta)
sn = sin(theta)
for x_new in range(rotated.shape[1]):
for y_new in range(rotated.shape[0]):
x = int(round((x_new - x_mid) * cs - (y_new - y_mid) * sn + x_mid)
y = int(round((x_new - x_mid) * sn - (y_new - y_mid) * cs + y_mid)
# just use comparisons, don't search through many values!
if 0 <= x < arr.shape[1] and 0 <= y < arr.shape[0]:
rotated[y_new, x_new] = arr[y, x]
So, now I can see (more easily) that for each pixel from the output array is mapped to a location in the input array. Yes, you can vectorize this.
import numpy as np
def rotate(arr, theta, unit='rad'):
# deal with theta units:
if unit.startswith('deg'):
theta = np.deg2rad(theta)
# for convenience, store array size:
ny, nx = arr.shape
# generate arrays of indices and flatten them:
y_new, x_new = np.indices(arr.shape)
x_new = x_new.ravel()
y_new = y_new.ravel()
# compute center of the array:
x0 = nx // 2
y0 = ny // 2
# compute old coordinates
xc = x_new - x0
yc = y_new - y0
x = np.round(np.cos(theta) * xc - np.sin(theta) * yc + x0).astype(np.int)
y = np.round(np.sin(theta) * xc - np.cos(theta) * yc + y0).astype(np.int)
# main idea to deal with indices is to create a mask:
mask = (x >= 0) & (x < nx) & (y >= 0) & (y < ny)
# ... and then select only those coordinates (both in
# input and "new" coordinates) that satisfy the above condition:
x = x[mask]
y = y[mask]
x_new = x_new[mask]
y_new = y_new[mask]
# map input values to output pixels *only* for selected "good" pixels:
rotated = np.zeros_like(arr)
rotated[y_new, x_new] = arr[y, x]
return rotated

Here is some code for anyone also doing 3d modeling. It solved my specific use-case pretty well. Still figuring out how to rotate in the proper plane. Hope it's helpful to you as well:
def rotate_model(m, theta):
'''
Redefines the prev 'rotate_model()' method
theta has to be in degrees
'''
rotated = scipy.ndimage.rotate(m, theta, axes=(1,2))
# have tried (1,0), (2,0), and now (1,2)
# ^ z is "up" and "2"
# scipy.ndimage.rotate() shrinks the model
# TODO: regrow it back
x_r = rotated.shape[1]
y_r = rotated.shape[0]
x_m = m.shape[1]
y_m = m.shape[0]
x_diff = abs(x_r - x_m)
y_diff = abs(y_r - y_m)
if x_diff%2==0 and y_diff%2==0:
return rotated[
x_diff//2 : x_r-x_diff//2,
y_diff//2 : y_r-y_diff//2,
:
]
elif x_diff%2==0 and y_diff%2==1:
# if this shift ends up turning the model to shit in a few iterations,
# change the following lines to include a flag that alternates cutting off the top and bottom bits of the array
return rotated[
x_diff//2 : x_r-x_diff//2,
y_diff//2+1 : y_r-y_diff//2,
:
]
elif x_diff%2==1 and y_diff%2==0:
return rotated[
x_diff//2+1 : x_r-x_diff//2,
y_diff//2 : y_r-y_diff//2,
:
]
else:
# x_diff%2==1 and y_diff%2==1:
return rotated[
x_diff//2+1 : x_r-x_diff//2,
y_diff//2+1 : y_r-y_diff//2,
:
]