We Keep Coding

What is the best way/method to digitize the data of a 3D surface into a grid of pixels with smaller resolution in Python?

I want to digitize (= average out over cells) photon count data into pixels given by a grid that tells how they are aligned. The photon count data is stored in a 2D array. I want to split that data into cells, each of which would correspond to a pixel. The idea is basically the same as changing an HD image to a smaller resolution. I'd like to achieve this in Python.
The digitizing function I've written:
import numpy as np
def digitize(function_data, grid_shape):
"""
function_data = 2D array of function values of some 3D shape,
eg.: exp(-(x^2 + y^2 -> want to digitize this
grid_shape: an array of length 2 which contains the dimensions of the smaller resolution
"""
l = len(function_data)
pixel_len_x = int(l/grid_shape[0])
pixel_len_y = int(l/grid_shape[1])
digitized_data = np.empty((grid_shape[0], grid_shape[1]))
for i in range(grid_shape[0]): #row-index of pixel in smaller-resolution grid
for j in range(grid_shape[1]): #column-index of pixel in smaller-resolution grid
hd_pixel = []
for k in range(pixel_len_y):
hd_pixel.append(z_data[k][j:j*pixel_len_x])
hd_pixel = np.ravel(hd_pixel) #turns 2D array into 1D to be able to compute average
pixel_avg = np.average(hd_pixel)
digitized_data[i][j] = pixel_avg
return digitized_data
In theory, this function should do what I want to achieve, but when tested it doesn't yield the expected results. Either a completed version of my function or any other method that achieves my goal would be extremely helpful.

You could also use a interpolation function, if you can use SciPy. Here we use one of the gridded data interpolating functions, RectBivariateSpline to upsample your function, but you can find numerous examples on this and other sites.
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import RectBivariateSpline as rbs
# Sampling coordinates
x = np.linspace(-2,2,20)
y = np.linspace(-2,2,30)
# Your function
f = np.exp(-(x[:,None]**2 + y**2))
# Interpolator
interp = rbs(x, y, f)
# Higher resolution coordinates
x_hd = np.linspace(x.min(), x.max(), x.size * 5)
y_hd = np.linspace(y.min(), y.max(), y.size * 5)
# New higher res function
f_hd = interp(x_hd, y_hd, grid = True)
# Some plots
fig, ax = plt.subplots(ncols = 2)
ax[0].imshow(f)
ax[1].imshow(f_hd)

How can I cut a piece away from a plot and set the point I need to zero?

In my work I have the task to read in a CSV file and do calculations with it. The CSV file consists of 9 different columns and about 150 lines with different values acquired from sensors. First the horizontal acceleration was determined, from which the distance was derived by double integration. This represents the lower plot of the two plots in the picture. The upper plot represents the so-called force data. The orange graph shows the plot over the 9th column of the CSV file and the blue graph shows the plot over the 7th column of the CSV file.
As you can see I have drawn two vertical lines in the lower plot in the picture. These lines represent the x-value, which in the upper plot is the global minimum of the orange function and the intersection with the blue function. Now I want to do the following, but I need some help: While I want the intersection point between the first vertical line and the graph to be (0,0), i.e. the function has to be moved down. How do I achieve this? Furthermore, the piece of the function before this first intersection point (shown in purple) should be omitted, so that the function really only starts at this point. How can I do this?
In the following picture I try to demonstrate how I would like to do that:
If you need my code, here you can see it:
import numpy as np
import matplotlib.pyplot as plt
import math as m
import loaddataa as ld
import scipy.integrate as inte
from scipy.signal import find_peaks
import pandas as pd
import os
# Loading of the values
print(os.path.realpath(__file__))
a,b = os.path.split(os.path.realpath(__file__))
print(os.chdir(a))
print(os.chdir('..'))
print(os.chdir('..'))
path=os.getcwd()
path=path+"\\Data\\1 Fabienne\\Test1\\left foot\\50cm"
print(path)
dataListStride = ld.loadData(path)
indexStrideData = 0
strideData = dataListStride[indexStrideData]
#%%Calculation of the horizontal acceleration
def horizontal(yAngle, yAcceleration, xAcceleration):
a = ((m.cos(m.radians(yAngle)))*yAcceleration)-((m.sin(m.radians(yAngle)))*xAcceleration)
return a
resultsHorizontal = list()
for i in range (len(strideData)):
strideData_yAngle = strideData.to_numpy()[i, 2]
strideData_xAcceleration = strideData.to_numpy()[i, 4]
strideData_yAcceleration = strideData.to_numpy()[i, 5]
resultsHorizontal.append(horizontal(strideData_yAngle, strideData_yAcceleration, strideData_xAcceleration))
resultsHorizontal.insert(0, 0)
#plt.plot(x_values, resultsHorizontal)
#%%
#x-axis "convert" into time: 100 Hertz makes 0.01 seconds
scale_factor = 0.01
x_values = np.arange(len(resultsHorizontal)) * scale_factor
#Calculation of the global high and low points
heel_one=pd.Series(strideData.iloc[:,7])
plt.scatter(heel_one.idxmax()*scale_factor,heel_one.max(), color='red')
plt.scatter(heel_one.idxmin()*scale_factor,heel_one.min(), color='blue')
heel_two=pd.Series(strideData.iloc[:,9])
plt.scatter(heel_two.idxmax()*scale_factor,heel_two.max(), color='orange')
plt.scatter(heel_two.idxmin()*scale_factor,heel_two.min(), color='green')#!
#Plot of force data
plt.plot(x_values[:-1],strideData.iloc[:,7]) #force heel
plt.plot(x_values[:-1],strideData.iloc[:,9]) #force toe
# while - loop to calculate the point of intersection with the blue function
i = heel_one.idxmax()
while strideData.iloc[i,7] > strideData.iloc[i,9]:
i = i-1
# Length calculation between global minimum orange function and intersection with blue function
laenge=(i-heel_two.idxmin())*scale_factor
print(laenge)
#%% Integration of horizontal acceleration
velocity = inte.cumtrapz(resultsHorizontal,x_values)
plt.plot(x_values[:-1], velocity)
#%% Integration of the velocity
s = inte.cumtrapz(velocity, x_values[:-1])
plt.plot(x_values[:-2],s)
I hope it's clear what I want to do. Thanks for helping me!

I didn't dig all the way through your code, but the following tricks may be useful.
Say you have x and y values:
x = np.linspace(0,3,100)
y = x**2
Now, you only want the values corresponding to, say, .5 < x < 1.5. First, create a boolean mask for the arrays as follows:
mask = np.logical_and(.5 < x, x < 1.5)
(If this seems magical, then run x < 1.5 in your interpreter and observe the results).
Then use this mask to select your desired x and y values:
x_masked = x[mask]
y_masked = y[mask]
Then, you can translate all these values so that the first x,y pair is at the origin:
x_translated = x_masked - x_masked[0]
y_translated = y_masked - y_masked[0]
Is this the type of thing you were looking for?

Fourier Transformation of 2D Matrix in Python

So, I have a matrix with 72x72 values, each corresponding to some energy on a triangular lattice with 72x72 sites. I'm trying to Fourier transform the values, but I'm not understanding how to do that with np.fft.fftn .
To illustrate my problem I have written the following basic code with some random values. The triangular gives the lattice x,y coordinates.
import numpy as np
import matplotlib.pyplot as plt
def triangular(nsize):
x=0
y=0
X=np.zeros((nsize,nsize))
Y=np.zeros((nsize,nsize))
for i in range(nsize):
for j in range(nsize):
X[i,j]+=1/2*j+i
Y[i,j]+=np.sqrt(3)/2*j
return(X,Y)
xx = triangular(72)[0]
yy = triangular(72)[1]
plt.figure()
plt.pcolormesh(xx, yy, np.reshape(np.random.rand(72**2),(72,72)))
I'm not using random data, but I wanted not to make the example that complicated. In fact I see everytime the same plot, when I now use the following FFT:
matrix = []
matrix.append(triangular(72)[0])
matrix.append(triangular(72)[1])
matrix.append(np.reshape(np.random.rand(72**2),(72,72)))
spectrum_3d = np.fft.fftn(matrix) # Fourrier transform along x, y, energy
kx = np.linspace(-4*np.pi/3,4*np.pi/3,72) #this is the range I want to plot
ky = np.linspace(-2*np.pi/np.sqrt(3),2*np.pi/np.sqrt(3),72)
Ky, Kx = np.meshgrid(ky, kx, indexing='ij') #making a grid
plt.figure(figsize=(11,9))
psd = plt.pcolormesh(Kx, Ky, abs(spectrum_3d[2])**2)
cbar = plt.colorbar(psd)
plt.xlabel('kx')
plt.ylabel('ky')
My result looks always the same and I don't know what went wrong. Also for my correlated values, which have a large symmetry the plot looks the same.

You can't 'see' the spectrum because of the DC dominance.
import numpy as np
import matplotlib.pyplot as p
%matplotlib inline
n=72
x=np.arange(n)
y=np.arange(n)
X,Y= np.meshgrid(x,y)
data=np.reshape(np.random.rand(n**2),(n,n))
data_wo_DC= data- np.mean(data)
spectrum = np.fft.fftshift(np.fft.fft2(data))
spectrum_wo_DC = np.fft.fftshift(np.fft.fft2(data_wo_DC))
freqx=np.fft.fftshift(np.fft.fftfreq(72,1)) #q(n, d=1.0)
freqy=np.fft.fftshift(np.fft.fftfreq(72,1))
fX,fY= np.meshgrid(freqx,freqy)
p.figure(figsize=(20,6))
p.subplot(131)
p.pcolormesh(X,Y, data)
p.colorbar()
p.subplot(132)
p.pcolormesh(fX,fY,np.abs(spectrum))
p.colorbar()
p.title('most data is in the DC')
p.subplot(133)
p.pcolormesh(fX,fY,np.abs(spectrum_wo_DC))
p.colorbar()
p.title('wo DC we can see the structure');

Matplotlib graphing distribution with two colors

The goal here is to color value above a certain threshold into one color and values below this threshold into another color. The code below tries to just separate it into two histographs but it only looks balanced if the threshold is at 50%. I'm assuming I must play around with the discreetlevel variable.
finalutilityrange is some vector with a bunch of values(you must generate it to test the code), which I am trying to graph. The value deter is the value that determines whether they will be blue or red. discreetlevel is just the amount of bins I would want.
import random
import numpy as np
import matplotlib.pyplot as plt
discreetlevel = 10
deter = 2
for x in range(0,len(finalutilityrange)):
if finalutilityrange[x-1]>=deter:
piraterange.append(finalutilityrange[x-1])
else:
nonpiraterange.append(finalutilityrange[x-1])
plt.hist(piraterange,bins=discreetlevel,normed=False,cumulative=False,color = 'b')
plt.hist(nonpiraterange,bins=discreetlevel),normed=False,cumulative=False,color = 'r')
plt.title("Histogram")
plt.xlabel("Utlity")
plt.ylabel("Probability")
plt.show()

This solution is a bit more complex than #user2699's. I am just presenting it for completeness. You have full control over the patch objects that hist returns, so if you can ensure that the threshold you are using is exactly on a bin edge, it is easy to change to color of selected patches. You can do this because hist can accept a sequence of bin edges as the bins parameter.
import numpy as np
from matplotlib import pyplot as plt
# Make sample data
finalutilityrange = np.random.randn(100)
discreetlevel = 10
deter = 0.2
# Manually create `discreetlevel` bins anchored to `deter`
binsAbove = round(discreetlevel * np.count_nonzero(finalutilityrange > deter) / finalutilityrange.size)
binsBelow = discreetlevel - binsAbove
binwidth = max((finalutilityrange.max() - deter) / binsAbove,
(deter - finalutilityrange.min()) / binsBelow)
bins = np.concatenate([
np.arange(deter - binsBelow * binwidth, deter, binwidth),
np.arange(deter, deter + (binsAbove + 0.5) * binwidth, binwidth)
])
# Use the bins to make a single histogram
h, bins, patches = plt.hist(finalutilityrange, bins, color='b')
# Change the appropriate patches to red
plt.setp([p for p, b in zip(patches, bins) if b >= deter], color='r')
The result is a homogenous histogram with bins of different colors:
The bins may be a tad wider than if you did not anchor to deter. Either the first or last bin will generally go a little past the edge of the data.

This answer doesn't address your code since it isn't self-contained, but for what you're trying to do the default histogram should work (assuming numpy/pyplot is loaded)
x = randn(100)
idx = x < 0.2 # Threshold to separate values
hist([x[idx], x[~idx]], color=['b', 'r'])
Explanation:
first line just generates some random data to test,
creates an index for where the data is below some threshold, this can be negated with ~ to find where it's above the threshold
Last line plots the histogram. The command takes a list of separate groups to plot, which doesn't make a big difference here but if normed=True it will
There's more the hist plot can do, so look over the documentation before you accidentally implement it yourself.

Just as above do:
x = np.random.randn(100)
threshold_x = 0.2 # Threshold to separate values
x_lower, x_upper = (
[_ for _ in x if _ < threshold_x],
[_ for _ in x if _ >= threshold_x]
)
hist([x_lower, x_upper], color=['b', 'r'])

Interpolation of curve

I have a code where a curve is generated using random values. and a Horizontal line which runs through it. The code is as follows.
import numpy as np
import matplotlib.pylab as pl
data = np.random.uniform(low=-1600, high=-550, size=(288,))
line = [-1290] * 288
pl.figure(figsize = (10,5))
pl.plot(data)
pl.plot(line)
Now I need to find the the coordinates for the all the points of intersections of the curve (data) and the line. The curve is made of linear segments that join neighboring points . And there are a lot of intersection points where the curve meets the line. any help would be appreciated. thank you!

I like the Shapely answer because Shapely is awesome, but you might not want that dependency. Here's a version of some code I use in signal processing adapted from this Gist by #endolith. It basically implements kazemakase's suggestion.
from matplotlib import mlab
def find_crossings(a, value):
# Normalize the 'signal' to zero.
sig = a - value
# Find all indices right before any crossing.
indices = mlab.find((sig[1:] >= 0) & (sig[:-1] < 0) | (sig[1:] < 0) & (sig[:-1] >= 0))
# Use linear interpolation to find intersample crossings.
return [i - sig[i] / (sig[i+1] - sig[i]) for i in indices]
This returns the indices (your x values) of where the curve crosses the value (-1290 in your case). You would call it like this:
find_crossings(data, -1290)
Here's what I get for 100 points:
x = find_crossings(data, -1290)
plt.figure(figsize=(10,5))
plt.plot(data)
plt.plot(line)
plt.scatter(x, [-1290 for p in x], color='red')
plt.show()

I think the curve, as you interpret it, does in fact follow an equation. In particular, it is made of linear segments that join neighboring points.
Here is what you can do:
find all pairs of neighbors where one lies above and the other below the line
for each pair find the intersection of the horizontal line with the line joining the points

Here is a solution that use shapely:
import numpy as np
import matplotlib.pylab as pl
np.random.seed(0)
data = np.random.uniform(low=-1600, high=-550, size=(50,))
line = [-1290] * len(data)
pl.figure(figsize = (10,5))
pl.plot(data)
pl.plot(line)
from shapely import geometry
line = geometry.LineString(np.c_[np.arange(len(data)), data])
hline = geometry.LineString([[-100, -1290], [1000, -1290]])
points = line.intersection(hline)
x = [p.x for p in points]
y = [p.y for p in points]
pl.plot(x, y, "o")
the output: