We Keep Coding

Viewing 3d Field Lines of an unstructured dataset

I have tried to view field lines of an uncomplete regular grid vector field with first pyVista Streamlines and then with plotly without success... I have yet good results with other 2d streamplots :
2d streamplot of the data
Could someone help me with this ? I found no answer... Here is my data : https://wetransfer.com/downloads/7f3c4ae01e5922e753ea708134f956e720230214141330/bf11ab
import pandas as pd
import numpy as np
import pyvista as pv
import plotly.graph_objects as go
df = pd.read_csv("mix_griddata.csv")
X = df['X']
Y = df['Y']
Z = df['Z']
Vx = df['Vx']
Vy = df['Vy']
Vz = df['Vz']
fig = go.Figure(data=go.Streamtube(
x = X,
y = Y,
z = Z,
u = Vx,
v = Vy,
w = Vz,
starts = dict(
x = X.sample(frac=0.01,replace=False),
y = Y.sample(frac=0.01,replace=False),
z = Z.sample(frac=0.01,replace=False)
),
sizeref =1,
colorscale = 'Portland',
showscale = False,
maxdisplayed = 30000000
))
fig.update_layout(
scene = dict(
aspectratio = dict(
x = 1,
y = 1,
z = 1
)
),
margin = dict(
t = 10,
b = 10,
l = 10,
r = 10
)
)
fig.show(renderer="browser")
#Streamlines
mix_FD_grid = np.load("C:/Users/hd377/OneDrive - ensam.eu/0-Thesis/Fibres_Direction_in_allvolume/mix/mix_FD_grid.npy")
origin = (0,0,0)
mesh = pv.UniformGrid(dimensions=mix_FD_grid[:,:,:,0].shape, spacing=(1, 1, 1), origin=origin)
vectors = np.empty((mesh.n_points, 3))
vectors[:, 0] = mix_FD_grid[:,:,:,0].flatten()
vectors[:, 1] = mix_FD_grid[:,:,:,1].flatten()
vectors[:, 2] = mix_FD_grid[:,:,:,2].flatten()
mesh['vectors'] = vectors
stream, src = mesh.streamlines(
'vectors', return_source=True, max_steps = 20000, n_points=200, source_radius=25, source_center=(15, 0, 30)
)
p = pv.Plotter()
p.add_mesh(mesh.outline(), color="k")
p.add_mesh(stream.tube(radius=0.1))
p.camera_position = [(182.0, 177.0, 50), (139, 105, 19), (-0.2, -0.2, 1)]
p.show()
The plotly window does appear in my browser but no tube are visible at all, and the axes values are false.
The pyVista does show something, but in the wrong direction, and clearly not what expected (longitudinal flux circumventing a central cone).

I'll only be tackling PyVista. It's hard to say for sure and I'm only guessing, but your data is probably laid out in the wrong order.
For starters, your data is inconsistent to begin with: your CSV has 1274117 rows whereas your multidimensional array has shape (37, 364, 100, 3), for a total of 1346800 vectors. And your question title says "unstructured", but your PyVista attempt uses a uniform grid with.
Secondly, your CSV doesn't correspond to a regular grid in the first place, e.g. at the end of the file you have 15 rows starting with 368.693,36.971999999999994, then 8 rows starting with 369.71999999999997,36.971999999999994, then a single row starting with 370.74699999999996,36.971999999999994. In a regular grid you'd get the same number of items in each block.
Thirdly, your CSV has an unusual (MATLAB-smelling) layout that the order of axes is z-x-y (rather than either x-y-z or z-y-x). This is a strong clue that your data is mangled due to memory layout issues when flattened. But the previous two point mean that I can't verify how your 4d array was created, I have to take it for granted that it's correct.
Just plotting your raw data makes it obvious that the data is mangled in your original version (with some style cleanup):
import numpy as np
import pyvista as pv
mix_FD_grid = np.load("mix_FD_grid.npy")
origin = (0, 0, 0)
mesh = pv.UniformGrid(dimensions=mix_FD_grid.shape[:-1], spacing=(1, 1, 1), origin=origin)
vectors = np.empty_like(mesh.points)
vectors[:, 0] = mix_FD_grid[..., 0].ravel()
vectors[:, 1] = mix_FD_grid[..., 1].ravel()
vectors[:, 2] = mix_FD_grid[..., 2].ravel()
mesh.point_data['vectors'] = vectors
mesh.plot()
The fragmented pattern you can see is a hallmark of data mangling due to mistaken memory layout.
If we assume the layout is more or less sane, trying column-major layout ("F" for "Fortran", also used by MATLAB) seems to make a lot more sense:
vectors[:, 0] = mix_FD_grid[..., 0].ravel('F')
vectors[:, 1] = mix_FD_grid[..., 1].ravel('F')
vectors[:, 2] = mix_FD_grid[..., 2].ravel('F')
mesh.point_data['vectors'] = vectors
mesh.plot()
So we can try using streamlines using that:
stream, src = mesh.streamlines(
'vectors', return_source=True, max_steps=20000, n_points=200, source_radius=25, source_center=(15, 0, 30)
)
p = pv.Plotter()
p.add_mesh(mesh.outline(), color="k")
p.add_mesh(stream.tube(radius=0.1))
p.show()
It doesn't look great:
So, you said that the streamlines should be longitudinal, but here they are clearly transversal. Can it be that the x and y field components are swapped? I can't tell, so let's try!
import numpy as np
import pyvista as pv
mix_FD_grid = np.load("mix_FD_grid.npy")
origin = (0, 0, 0)
mesh = pv.UniformGrid(dimensions=mix_FD_grid.shape[:-1], spacing=(1, 1, 1), origin=origin)
vectors = np.empty_like(mesh.points)
vectors[:, 0] = mix_FD_grid[..., 1].ravel('F') # swap 0 <-> 1
vectors[:, 1] = mix_FD_grid[..., 0].ravel('F') # swap 0 <-> 1
vectors[:, 2] = mix_FD_grid[..., 2].ravel('F')
mesh.point_data['vectors'] = vectors
stream, src = mesh.streamlines(
'vectors', return_source=True, max_steps=20000, n_points=200, source_radius=25, source_center=(15, 0, 30)
)
p = pv.Plotter()
p.add_mesh(mesh.outline(), color="k")
p.add_mesh(stream.tube(radius=0.1))
p.show()
Now we're talking!
Bonus: y field component on a volumetric plot:
mesh.plot(volume=True, scalars=vectors[:, 1], show_scalar_bar=False)

Display issue of fitted curve: cannot solve coarseness

Despite having a working script for curve fitting using the lmfit library, I am not able to solve a display issue. Indeed, having only 5 dependent values, the resulting graph is rather coarse.
Before switching to lmfit, I was using curve_fit and could solve the display issue by simply using np.linspace and plot the optimized values resulting from the fit procedure. Then, I was displaying the "real" values through plt.errorbar. With lmfit, the above solution yields a mismatch error, since it recognizes the "fake" independent variables and launches a mismatch type error.
My full script is the following:
import lmfit as lf
from lmfit import Model, Parameters
import numpy as np
import matplotlib.pyplot as plt
from math import atan
def on_res(omega_eff, thetas, R2avg=5, k_ex=0.1, phi_ex=500):
return R2avg*(np.sin(thetas))**2 + ((np.sin(thetas))**2)*(phi_ex*k_ex/(k_ex**2 + omega_eff**2))
model = Model(on_res,independent_vars=['omega_eff','thetas'])
params = model.make_params(R2avg=5, k_ex=0.01, phi_ex=1500)
carrier = 6146.53
O_1 = 5846
spin_locks = (1000, 2000, 3000, 4000, 5000)
delta_omega = (O_1 - carrier)
omega_eff1 = ((delta_omega**2) + (spin_locks[0]**2))**0.5
omega_eff2 = ((delta_omega**2) + (spin_locks[1]**2))**0.5
omega_eff3 = ((delta_omega**2) + (spin_locks[2]**2))**0.5
omega_eff4 = ((delta_omega**2) + (spin_locks[3]**2))**0.5
omega_eff5 = ((delta_omega**2) + (spin_locks[4]**2))**0.5
theta_rad1 = atan(spin_locks[0]/delta_omega)
theta_rad2 = atan(spin_locks[1]/delta_omega)
theta_rad3 = atan(spin_locks[2]/delta_omega)
theta_rad4 = atan(spin_locks[3]/delta_omega)
theta_rad5 = atan(spin_locks[4]/delta_omega)
x = (omega_eff1/1000, omega_eff2/1000, omega_eff3/1000, omega_eff4/1000, omega_eff5/1000)# , omega_eff6/1000)# , omega_eff7/1000)
theta = (theta_rad1, theta_rad2, theta_rad3, theta_rad4, theta_rad5)
R1rho_vals = (7.9328, 6.2642, 6.0005, 5.9972, 5.988)
e = (0.2, 0.2, 0.2, 0.2, 0.2)
new_x = np.linspace(0, 6, 1000)
omega_eff = np.array(x, dtype=float)
thetas = np.array(theta, dtype=float)
R1rho_vals = np.array(R1rho_vals, dtype=float)
error = np.array(e, dtype=float)
R2avg = []
k_ex = []
phi_ex = []
result = model.fit(R1rho_vals, params, weights=1/error, thetas=thetas, omega_eff=omega_eff, method = "emcee", steps = 1000)
print(result.fit_report())
plt.errorbar(x, R1rho_vals, yerr = error, fmt = ".k", markersize = 8, capsize = 3)
plt.plot(new_x, result.best_fit)
plt.show()
As you can see running it, it launches the mismatch shape error message. Changing the plt.plot line to plt.plot(x, result.best_fit) yields the graph correctly, but displaying a very coarse profile (as one would expect, having only 5 points on the x-axis).
Are you aware of any way to solve this? Checking the documentation, I noticed the examples provided all plot the results via the actual independent variables values, since they have enough experimental values.

You need to re-evaluate the ModelResult with your new values for the independent variables:
plt.plot(new_x, result.eval(omega_eff=new_x/1000., thetas=thetas))

How can I interpolate data in python?

I have a 4D dataset (time, z, y, x) and I would like to interpolate the data to get a higher resolution, this is a simple example code:
import numpy as np
from scipy.interpolate import griddata
x_0 = 10
cut_index = 10
res = 200j
x_index = x_0
y_index = np.linspace(0, 100, 50).astype(int)
z_index = np.linspace(0, 50, 25).astype(int)
#Time, zyx-coordinate
u = np.random.randn(20, 110, 110, 110)
z_index, y_index = np.meshgrid(z_index, y_index)
data = u[cut_index, z_index, y_index, x_index]
res = 200j
y_f = np.mgrid[0:100:res]
z_f = np.mgrid[0:50:res]
z_f, y_f = np.meshgrid(z_f, y_f)
data = griddata((z_index, y_index), data, (z_f, y_f))
I am getting the ValueError: invalid shape for input data points error. What kind of input is expected by the griddata function?

Your data parameter has to be a 1D array. Try flattening the arrays:
data = griddata((z_index.flatten(), y_index.flatten()), data.flatten(), (z_f, y_f))

IndexError: too many indices for array for an array that is definitely as big

I'm trying to make a movie by taking png images of an updating plot and stitching them together. There are three variables: degrees, ksB, and mp. Only mp changes each frame; the other two are constant. The data for mp for all times is stored in X. This is the relevant part of the code:
def plot(fname, haveMLPY=False):
# Load data from .npz file.
data = np.load(fname)
X = data["X"]
T = data["T"]
N = X.shape[1]
A = data["vipWeights"]
degrees = A.sum(1)
ksB = data["ksB"]
# Initialize a figure.
figure = plt.figure()
# Generate a plottable axis as the first subplot in 1 rows and 1 columns.
axis = figure.add_subplot(1,1,1)
# MP is the first (0th) variable. Plot one trajectory for each cell over time.
axis.plot(T, X[:,:,0], color="black")
# Decorate the plot.
axis.set_xlabel("time [hours]")
axis.set_ylabel("MP [nM]")
axis.set_title("PER mRNA concentration across all %d cells" % N)
firstInd = int(T.size / 2)
if haveMLPY:
import circadian.analysis
# Generate a and plot Signal object, which encapsulates wavelet analysis.
signal = circadian.analysis.Signal(X[firstInd:, 0, 0], T[firstInd:])
signal.showSpectrum(show=False)
files=[]
# filename for the name of the resulting movie
filename = 'animation'
mp = X[10**4-1,:,0]
from mpl_toolkits.mplot3d import Axes3D
for i in range(10**4):
print i
mp = X[i,:,0]
data2 = np.c_[degrees, ksB, mp]
# Find best fit surface for data2
# regular grid covering the domain of the data
mn = np.min(data2, axis=0)
mx = np.max(data2, axis=0)
X,Y = np.meshgrid(np.linspace(mn[0], mx[0], 20), np.linspace(mn[1], mx[1], 20))
XX = X.flatten()
YY = Y.flatten()
order = 2 # 1: linear, 2: quadratic
if order == 1:
# best-fit linear plane
A = np.c_[data2[:,0], data2[:,1], np.ones(data2.shape[0])]
C,_,_,_ = scipy.linalg.lstsq(A, data2[:,2]) # coefficients
# evaluate it on grid
Z = C[0]*X + C[1]*Y + C[2]
# or expressed using matrix/vector product
#Z = np.dot(np.c_[XX, YY, np.ones(XX.shape)], C).reshape(X.shape)
elif order == 2:
# best-fit quadratic curve
A = np.c_[np.ones(data2.shape[0]), data2[:,:2], np.prod(data2[:,:2], axis=1), data2[:,:2]**2]
C,_,_,_ = scipy.linalg.lstsq(A, data2[:,2])
# evaluate it on a grid
Z = np.dot(np.c_[np.ones(XX.shape), XX, YY, XX*YY, XX**2, YY**2], C).reshape(X.shape)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X, Y, Z, rstride=1, cstride=1, alpha=0.2)
ax.scatter(degrees, ksB, mp)
ax.set_xlabel('degrees')
ax.set_ylabel('ksB')
ax.set_zlabel('mp')
# form a filename
fname2 = '_tmp%03d.png'%i
# save the frame
savefig(fname2)
# append the filename to the list
files.append(fname2)
# call mencoder
os.system("mencoder 'mf://_tmp*.png' -mf type=png:fps=10 -ovc lavc -lavcopts vcodec=wmv2 -oac copy -o " + filename + ".mpg")
# cleanup
for fname2 in files: os.remove(fname2)
Basically, all the data is stored in X. The format X[i, i, i] means X[time, neuron, data type]. Each time through the loop, I want to update the time, but still plot mp (the 0th variable) for all the neurons.
When I run this code, I get "IndexError: too many indices for array". I asked it to print i to see when the code was going wrong. I get an error when i = 1, meaning that the code loops through once but then has the error the second time.
However, I have data for 10^4 time steps. You can see in the first line of the provided code, I access X[10**4-1, :, 0] successfully. That's why it's confusing to me why X[1,:,0] would be out of range. If anybody could explain why/help me get around this, that would be great.
The traceback error is
Traceback (most recent call last):
File"/Users/angadanand/Documents/LiClipseWorkspace/Circadian/scripts /runMeNets.py", line 196, in module
plot(fname)
File"/Users/angadanand/Documents/LiClipseWorkspace/Circadian/scripts /runMeNets.py", line 142, in plot
mp = X[i,:,0]
IndexError: too many indices for array
Thanks!

Your problem is that you overwrite your X inside your loop:
X,Y = np.meshgrid(np.linspace(mn[0], mx[0], 20), np.linspace(mn[1], mx[1], 20))
So afterwards it will have another shape and contain different data. I would suggest changing this second X to x_grid and check where you need this "other" X and where the original.
for example:
X_grid, Y_grid = np.meshgrid(np.linspace(mn[0], mx[0], 20), np.linspace(mn[1], mx[1], 20))

Find the position of a lowest difference between numpy arrays

I've got two musical files: one lossless with little sound gap (at this time it's just silence but it could be anything: sinusoid or just some noise) at the beginning and one mp3:
In [1]: plt.plot(y[:100000])
Out[1]:
In [2]: plt.plot(y2[:100000])
Out[2]:
This lists are similar but not identical so I need to cut this gap, to find the first occurrence of one list in another with lowest delta error.
And here's my solution (5.7065 sec.):
error = []
for i in range(25000):
y_n = y[i:100000]
y2_n = y2[:100000-i]
error.append(abs(y_n - y2_n).mean())
start = np.array(error).argmin()
print(start, error[start]) #23057 0.0100046
Is there any pythonic way to solve this?
Edit:
After calculating the mean distance between special points (e.g. where data == 0.5) I reduce the area of search from 25000 to 2000. This gives me reasonable time of 0.3871s:
a = np.where(y[:100000].round(1) == 0.5)[0]
b = np.where(y2[:100000].round(1) == 0.5)[0]
mean = int((a - b[:len(a)]).mean())
delta = 1000
error = []
for i in range(mean - delta, mean + delta):
...

What you are trying to do is a cross-correlation of the two signals.
This can be done easily using signal.correlate from the scipy library:
import scipy.signal
import numpy as np
# limit your signal length to speed things up
lim = 25000
# do the actual correlation
corr = scipy.signal.correlate(y[:lim], y2[:lim], mode='full')
# The offset is the maximum of your correlation array,
# itself being offset by (lim - 1):
offset = np.argmax(corr) - (lim - 1)
You might want to take a look at this answer to a similar problem.

Let's generate some data first
N = 1000
y1 = np.random.randn(N)
y2 = y1 + np.random.randn(N) * 0.05
y2[0:int(N / 10)] = 0
In these data, y1 and y2 are almost the same (note the small added noise), but the first 10% of y2 are empty (similarly to your example)
We can now calculate the absolute difference between the two vectors and find the first element for which the absolute difference is below a sensitivity threshold:
abs_delta = np.abs(y1 - y2)
THRESHOLD = 1e-2
sel = abs_delta < THRESHOLD
ix_start = np.where(sel)[0][0]
fig, axes = plt.subplots(3, 1)
ax = axes[0]
ax.plot(y1, '-')
ax.set_title('y1')
ax.axvline(ix_start, color='red')
ax = axes[1]
ax.plot(y2, '-')
ax.axvline(ix_start, color='red')
ax.set_title('y2')
ax = axes[2]
ax.plot(abs_delta)
ax.axvline(ix_start, color='red')
ax.set_title('abs diff')
This method works if the overlapping parts are indeed "almost identical". You will have to think of smarter alignment ways if the similarity is low.

I think what you are looking for is correlation. Here is a small example.
import numpy as np
equal_part = [0, 1, 2, 3, -2, -4, 5, 0]
y1 = equal_part + [0, 1, 2, 3, -2, -4, 5, 0]
y2 = [1, 2, 4, -3, -2, -1, 3, 2]+y1
np.argmax(np.correlate(y1, y2, 'same'))
Out:
7
So this returns the time-difference, where the correlation between both signals is at its maximum. As you can see, in the example the time difference should be 8, but this depends on your data...
Also note that both signals have the same length.