We Keep Coding

Apply rotation matrix determined by separate fixed point - python

I'm applying a rotation matrix to a group of points with the aim to align the points along the horizontal axis. Using below, the xy points I want to adjust are recorded in x and y.
I'm hoping to transform the points using the angle between X_Ref and Y_Ref and X_Fixed and Y_Fixed. I'm also hoping to transform the points so X_Ref and Y_Ref is at 0,0 once the rotation is completed.
The rotated points currently don't adjust for this. I'm not sure if I should account for the reference point prior to rotating or afterwards.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
import pandas as pd
df = pd.DataFrame({
'Period' : ['1','1','1','1','2','2','2','2'],
'Label' : ['A','B','C','D','A','B','C','D'],
'x' : [2.0,3.0,3.0,2.0,2.0,3.0,3.0,1.0],
'y' : [2.0,3.0,-1.0,0.0,2.0,3.0,-1.0,1.0],
'X_Ref' : [1,1,1,1,2,2,2,2],
'Y_Ref' : [1,1,1,1,0,0,0,0],
'X_Fixed' : [0,0,0,0,0,0,0,0],
'Y_Fixed' : [0,0,0,0,2,2,2,2],
})
np.random.seed(1)
xy = df[['x','y']].values
Ref = df[['X_Ref','Y_Ref']].values
Fix = df[['X_Fixed','Y_Fixed']].values
fig, ax = plt.subplots()
plot_kws = {'alpha': 0.75,
'edgecolor': 'white',
'linewidths': 0.75}
ax.scatter(xy[:, 0], xy[:, 1], **plot_kws)
ax.scatter(Ref[:, 0], Ref[:, 1], marker = 'x')
ax.scatter(Fix[:, 0], Fix[:, 1], marker = '+')
pca = PCA(2)
# Fit the PCA object, but do not transform the data
pca.fit(xy)
# pca.components_ : array, shape (n_components, n_features)
# cos theta
ct = pca.components_[0, 0]
# sin theta
st = pca.components_[0, 1]
# One possible value of theta that lies in [0, pi]
t = np.arccos(ct)
# If t is in quadrant 1, rotate CLOCKwise by t
if ct > 0 and st > 0:
t *= -1
# If t is in Q2, rotate COUNTERclockwise by the complement of theta
elif ct < 0 and st > 0:
t = np.pi - t
# If t is in Q3, rotate CLOCKwise by the complement of theta
elif ct < 0 and st < 0:
t = -(np.pi - t)
# If t is in Q4, rotate COUNTERclockwise by theta, i.e., do nothing
elif ct > 0 and st < 0:
pass
# Manually build the ccw rotation matrix
rotmat = np.array([[np.cos(t), -np.sin(t)],
[np.sin(t), np.cos(t)]])
# Apply rotation to each row of 'm'. The output (m2)
# will be the rotated FIFA input coordinates.
m2 = (rotmat # xy.T).T
# Center the rotated point cloud at (0, 0)
m2 -= m2.mean(axis=0)
Initial distribution period 1:
Intended distribution period 1:
Initial distribution period 2:
Intended distribution period 2:

Your question in unclear, since the "intended rotation" mentioned in the question can be already achieved if you plot m2 which has been already calculated:
fig, ax = plt.subplots()
ax.scatter(m2[:, 0], m2[:, 1], **plot_kws)
Output:
But you have also mentioned the following in the question:
The rotation angle is determined by the angle between X_Ref,Y_Ref and X_Fixed,Y_Fixed.
This is a totally different scenario. You can calculate the angle between two points by calculating the arctan between them, without having to use PCA at all. This can be done using numpy.arctan as follows:
t = np.arctan((Y_Fixed - Y_Ref/ X_Fixed - X_Ref))
Here (X_Fixed, Y_Fixed) and (X_Ref, Y_Ref) are being assumed as two points.
For each row in your dataframe, you can then calculate the x and y values after rotation with respect to the angle between (X_Fixed, Y_Fixed) and (X_Ref, Y_Ref) in that particular row. This can be done using the following code snippet;
def rotate_points(row):
t = np.arctan((row['Y_Fixed'] - row['Y_Ref']/ row['X_Fixed'] - row['X_Ref']))
rotmat = np.array([[np.cos(t), -np.sin(t)],
[np.sin(t), np.cos(t)]])
xy = row[['x','y']].values
rotated = rotmat # xy
return rotated
df['rotated_x'] = df.apply(lambda row: rotate_points(row)[0], axis = 1)
df['rotated_y'] = df.apply(lambda row: rotate_points(row)[1], axis = 1)
Your dataframe would now look like this with the two new columns added to the right:
+----+----------+---------+-----+-----+---------+---------+-----------+-----------+-------------+-------------+-------------+
| | Period | Label | x | y | X_Ref | Y_Ref | X_Fixed | Y_Fixed | Direction | rotated_x | rotated_y |
|----+----------+---------+-----+-----+---------+---------+-----------+-----------+-------------+-------------+-------------|
| 0 | 1 | A | -1 | 1 | 1 | 3 | -2 | 0 | Left | -1.34164 | 0.447214 |
| 1 | 1 | B | 0 | 4 | 1 | 3 | -2 | 0 | Left | -1.78885 | 3.57771 |
| 2 | 1 | C | 2 | 2 | 1 | 3 | -2 | 0 | Left | 0.894427 | 2.68328 |
| 3 | 1 | D | 2 | 3 | 1 | 3 | -2 | 0 | Left | 0.447214 | 3.57771 |
| 4 | 2 | E | 2 | 4 | 1 | 3 | -2 | 0 | Right | 0 | 4.47214 |
| 5 | 2 | F | 1 | 4 | 1 | 3 | -2 | 0 | Right | -0.894427 | 4.02492 |
| 6 | 2 | G | 3 | 5 | 1 | 3 | -2 | 0 | Right | 0.447214 | 5.81378 |
| 7 | 2 | H | 0 | 2 | 1 | 3 | -2 | 0 | Right | -0.894427 | 1.78885 |
+----+----------+---------+-----+-----+---------+---------+-----------+-----------+-------------+-------------+-------------+
Now you have your rotated x and y points as desired.
UPDATE:
As per the amended question, you can add the reference point at (0,0) in your plot as follows:
fig, ax = plt.subplots()
ax.scatter(m2[:, 0], m2[:, 1], **plot_kws)
ax.scatter(list(np.repeat(0, len(Ref))), list(np.repeat(0, len(Ref))) , **plot_kws)
plt.show()
Output:

There is no need for any PCA if I understood what you try to achieve. I'd use complex number and that seems more straightforward :
EDIT
There was a small mistake in the order of steps for translation previously. This edit will correct it as well as use your new dataset including changing ref/fixed points at different periods.
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
df = pd.DataFrame({
'Period' : ['1','1','1','1','2','2','2','2'],
'Label' : ['A','B','C','D','A','B','C','D'],
'x' : [2.0,3.0,3.0,2.0,2.0,3.0,3.0,1.0],
'y' : [2.0,3.0,-1.0,0.0,2.0,3.0,-1.0,1.0],
'X_Ref' : [1,1,1,1,2,2,2,2],
'Y_Ref' : [1,1,1,1,0,0,0,0],
'X_Fixed' : [0,0,0,0,0,0,0,0],
'Y_Fixed' : [0,0,0,0,2,2,2,2],
})
First, transform fixed/ref points to complex numbers :
for f in ['Ref', 'Fixed']:
df[f] = df['X_'+f] + 1j*df['Y_'+f]
df.drop(['X_'+f, 'Y_'+f], axis=1, inplace=True)
Compute the rotation (note that it is the opposite angle of what you stated in your question to match your expected results) :
df['angle'] = - np.angle(df['Ref'] - df['Fixed'])
Compute the rotation for every point (ref/fixed included) :
df['rotated'] = (df['x'] + 1j*df["y"]) * np.exp(1j*df['angle'])
for f in ['Ref', 'Fixed']:
df[f+'_Rotated'] = df[f] * np.exp(1j*df['angle'])
Center your dataset around the "ref" point :
df['translation'] = - df['Ref_Rotated']
df['NewPoint'] = df['rotated'] + df['translation']
for f in ['Ref', 'Fixed']:
df[f+'_Transformed'] = df[f+'_Rotated'] + df['translation']
Revert to cartesian coordinates :
df['x2'] = np.real(df['NewPoint'])
df['y2'] = np.imag(df['NewPoint'])
for f in ['Ref', 'Fixed']:
df['NewX_'+f] = np.real(df[f+'_Transformed'])
df['NewY_'+f] = np.imag(df[f+'_Transformed'])
And then plot the output for any period you like :
output = df[['Period', 'Label', 'x2', 'y2', 'NewX_Ref', 'NewY_Ref', 'NewX_Fixed', 'NewY_Fixed']]
output.set_index('Period', inplace=True)
fig, ax = plt.subplots()
plot_kws = {'alpha': 0.75,
'edgecolor': 'white',
'linewidths': 0.75}
plt.xlim(-5,5)
plt.ylim(-5,5)
period = '1'
ax.scatter(output.loc[period, 'NewX_Ref'], output.loc[period, 'NewY_Ref'])
ax.scatter(output.loc[period, 'NewX_Fixed'], output.loc[period, 'NewY_Fixed'])
ax.scatter(output.loc[period, 'x2'], output.loc[period, 'y2'], **plot_kws, marker = '+')
plt.gca().set_aspect('equal', adjustable='box')
plt.show()
Result for period 1 :
Result for period 2 :

Clustering some graph form matrix

A few questions on stackoverflow mention similar problem, but...
I have a square matrix, for example like this:
| A | B | C | D
A | 1 | 1 | 0 | 0
B | 1 | 1 | 1 | 0
C | 0 | 1 | 1 | 0
D | 0 | 0 | 0 | 1
The square matrix can be of any size (can be 1000x1000 or larger). I want to get clusters (I don't know how many...). For the above example I should get two clusters:
A, B, C (because A-B and B-C)
D

The example you gave is a classic clique detection, well studied in graph theory with several algorithms. It gets more interesting when you allow some edges to be missing. This will lead to modularity etc.
What you have is an unweighted adjacency matrix. There are clustering algorithms that work on this - affinity propagation needs an affinity, and hierarchical agglomerative clustering can be implemented with similarities (beware, most implementations use distances)
In your case, you can trivially use 1-X as distance matrix for many many clustering algorithm except k-means and GMM and a few other exceptions that require a coordinate matrix.

Matplotlib dot plot with two categorical variables

I would like to produce a specific type of visualization, consisting of a rather simple dot plot but with a twist: both of the axes are categorical variables (i.e. ordinal or non-numerical values). And this complicates matters instead of making it easier.
To illustrate this question, I will be using a small example dataset that is a modification from seaborn.load_dataset("tips") and defined as such:
import pandas
from six import StringIO
df = """total_bill | tip | sex | smoker | day | time | size
16.99 | 1.01 | Male | No | Mon | Dinner | 2
10.34 | 1.66 | Male | No | Sun | Dinner | 3
21.01 | 3.50 | Male | No | Sun | Dinner | 3
23.68 | 3.31 | Male | No | Sun | Dinner | 2
24.59 | 3.61 | Female | No | Sun | Dinner | 4
25.29 | 4.71 | Female | No | Mon | Lunch | 4
8.77 | 2.00 | Female | No | Tue | Lunch | 2
26.88 | 3.12 | Male | No | Wed | Lunch | 4
15.04 | 3.96 | Male | No | Sat | Lunch | 2
14.78 | 3.23 | Male | No | Sun | Lunch | 2"""
df = pandas.read_csv(StringIO(df.replace(' ','')), sep="|", header=0)
My first approach to produce my graph was to try a call to seaborn as such:
import seaborn
axes = seaborn.pointplot(x="time", y="sex", data=df)
This fails with:
ValueError: Neither the `x` nor `y` variable appears to be numeric.
So does the equivalent seaborn.stripplot and seaborn.swarmplot calls. It does work however if one of the variables is categorical and the other one is numerical. Indeed seaborn.pointplot(x="total_bill", y="sex", data=df) works, but is not what I want.
I also attempted a scatterplot like such:
axes = seaborn.scatterplot(x="time", y="sex", size="day", data=df,
x_jitter=True, y_jitter=True)
This produces the following graph which does not contain any jitter and has all the dots overlapping, making it useless:
Do you know of any elegant approach or library that could solve my problem ?
I started writing something myself, which I will include below, but this implementation is suboptimal and limited by the number of points that can overlap at the same spot (currently it fails if more than 4 points overlap).
# Modules #
import seaborn, pandas, matplotlib
from six import StringIO
################################################################################
def amount_to_offets(amount):
"""A function that takes an amount of overlapping points (e.g. 3)
and returns a list of offsets (jittered) coordinates for each of the
points.
It follows the logic that two points are displayed side by side:
2 -> * *
Three points are organized in a triangle
3 -> *
* *
Four points are sorted into a square, and so on.
4 -> * *
* *
"""
assert isinstance(amount, int)
solutions = {
1: [( 0.0, 0.0)],
2: [(-0.5, 0.0), ( 0.5, 0.0)],
3: [(-0.5, -0.5), ( 0.0, 0.5), ( 0.5, -0.5)],
4: [(-0.5, -0.5), ( 0.5, 0.5), ( 0.5, -0.5), (-0.5, 0.5)],
}
return solutions[amount]
################################################################################
class JitterDotplot(object):
def __init__(self, data, x_col='time', y_col='sex', z_col='tip'):
self.data = data
self.x_col = x_col
self.y_col = y_col
self.z_col = z_col
def plot(self, **kwargs):
# Load data #
self.df = self.data.copy()
# Assign numerical values to the categorical data #
# So that ['Dinner', 'Lunch'] becomes [0, 1] etc. #
self.x_values = self.df[self.x_col].unique()
self.y_values = self.df[self.y_col].unique()
self.x_mapping = dict(zip(self.x_values, range(len(self.x_values))))
self.y_mapping = dict(zip(self.y_values, range(len(self.y_values))))
self.df = self.df.replace({self.x_col: self.x_mapping, self.y_col: self.y_mapping})
# Offset points that are overlapping in the same location #
# So that (2.0, 3.0) becomes (2.05, 2.95) for instance #
cols = [self.x_col, self.y_col]
scaling_factor = 0.05
for values, df_view in self.df.groupby(cols):
offsets = amount_to_offets(len(df_view))
offsets = pandas.DataFrame(offsets, index=df_view.index, columns=cols)
offsets *= scaling_factor
self.df.loc[offsets.index, cols] += offsets
# Plot a standard scatter plot #
g = seaborn.scatterplot(x=self.x_col, y=self.y_col, size=self.z_col, data=self.df, **kwargs)
# Force integer ticks on the x and y axes #
locator = matplotlib.ticker.MaxNLocator(integer=True)
g.xaxis.set_major_locator(locator)
g.yaxis.set_major_locator(locator)
g.grid(False)
# Expand the axis limits for x and y #
margin = 0.4
xmin, xmax, ymin, ymax = g.get_xlim() + g.get_ylim()
g.set_xlim(xmin-margin, xmax+margin)
g.set_ylim(ymin-margin, ymax+margin)
# Replace ticks with the original categorical names #
g.set_xticklabels([''] + list(self.x_mapping.keys()))
g.set_yticklabels([''] + list(self.y_mapping.keys()))
# Return for display in notebooks for instance #
return g
################################################################################
# Graph #
graph = JitterDotplot(data=df)
axes = graph.plot()
axes.figure.savefig('jitter_dotplot.png')

you could first convert time and sex to categorical type and tweak it a little bit:
df.sex = pd.Categorical(df.sex)
df.time = pd.Categorical(df.time)
axes = sns.scatterplot(x=df.time.cat.codes+np.random.uniform(-0.1,0.1, len(df)),
y=df.sex.cat.codes+np.random.uniform(-0.1,0.1, len(df)),
size=df.tip)
Output:
With that idea, you can modify the offsets (np.random) in the above code to the respective distance. For example:
# grouping
groups = df.groupby(['time', 'sex'])
# compute the number of samples per group
num_samples = groups.tip.transform('size')
# enumerate the samples within a group
sample_ranks = df.groupby(['time']).cumcount() * (2*np.pi) / num_samples
# compute the offset
x_offsets = np.where(num_samples.eq(1), 0, np.cos(df.sample_rank) * 0.03)
y_offsets = np.where(num_samples.eq(1), 0, np.sin(df.sample_rank) * 0.03)
# plot
axes = sns.scatterplot(x=df.time.cat.codes + x_offsets,
y=df.sex.cat.codes + y_offsets,
size=df.tip)
Output:

Nearest neighbors in a given range

I faced the problem of quickly finding the nearest neighbors in a given range.
Example of dataset:
id | string | float
0 | AA | 0.1
12 | BB | 0.5
2 | CC | 0.3
102| AA | 1.1
33 | AA | 2.8
17 | AA | 0.5
For each line, print the number of lines satisfying the following conditions:
string field is equal to current
float field <= current float - del
For this example with del = 1.5:
id | count
0 | 0
12 | 0
2 | 0
102| 2 (string is equal row with id=0,33,17 but only in row id=0,17 float value: 1.1-1.5<=0.1, 1.1-1.5<=0.5)
33 | 0 (string is equal row with id=0,102,17 but 2.8-1.5>=0.1/1.1/1.5)
17 | 1
To solve this problem, I used a class BallTree with custom metric, but it works for a very long time due to a reverse tree walk (on a large dataset).
Can someone suggest other solutions or how you can increase the speed of custom metrics to the speed of the metrics from the sklearn.neighbors.DistanceMetric?
My code:
from sklearn.neighbors import BallTree
def distance(x, y):
if(x[0]==y[0] and x[1]>y[1]):
return (x[1] - y[1])
else:
return (x[1] + y[1])
tree2 = BallTree(X, leaf_size=X.shape[0], metric=distance)
mas=tree2.query_radius(X, r=del, count_only = True)

Gurobi: How can I sum just a part of a variable?

I have the following model:
from gurobipy import *
n_units = 1
n_periods = 3
n_ageclasses = 4
units = range(1,n_units+1)
periods = range(1,n_periods+1)
periods_plus1 = periods[:]
periods_plus1.append(max(periods_plus1)+1)
ageclasses = range(1,n_ageclasses+1)
nothickets = ageclasses[1:]
model = Model('MPPM')
HARVEST = model.addVars(units, periods, nothickets, vtype=GRB.INTEGER, name="HARVEST")
FOREST = model.addVars(units, periods_plus1, ageclasses, vtype=GRB.INTEGER, name="FOREST")
model.addConstrs((quicksum(HARVEST[(k+1), (t+1), nothicket] for k in range(n_units) for t in range(n_periods) for nothicket in nothickets) == FOREST[unit, period+1, 1] for unit in units for period in periods if period < max(periods_plus1)), name="A_Thicket")
I have a problem with formulating the constraint. I want for every unit and every period to sum the nothickets part of the variable HARVEST. Concretely I want xk=1,t=1,2 + xk=1,t=1,3 + xk=1,t=1,4
and so on. This should result in only three ones per row of the constraint matrix. But with the formulation above I get 9 ones.
I tried to use a for loop outside of the sum, but this results in another problem:
for k in range(n_units):
for t in range(n_periods):
model.addConstrs((quicksum(HARVEST[(k+1), (t+1), nothicket] for nothicket in nothickets) == FOREST[unit,period+1, 1] for unit in units for period in periods if period < max(periods_plus1)), name="A_Thicket")
With this formulation I get this matrix:
constraint matrix
But what I want is:
row_idx | col_idx | coeff
0 | 0 | 1
0 | 1 | 1
0 | 2 | 1
0 | 13 | -1
1 | 3 | 1
1 | 4 | 1
1 | 5 | 1
1 | 17 | -1
2 | 6 | 1
2 | 7 | 1
2 | 8 | 1
2 | 21 | -1
Can anybody please help me to reformulate this constraint?

This worked for me:
model.addConstrs((HARVEST.sum(unit, period, '*') == ...