When I'm trying to run a neighborhood analysis dask_image.ndfilters.maximum_filter() on a quite big geotiff file (~7GB / 30000x50000 px) but also small ones (800x800 px) using a circle pattern for the footprint attribute I get a MemoryError. I get the memory error when computing .compute() the dask array back to a numpy array.
This occurs when using big radius for the circle pattern (> 100). When I'm using smaller circle patterns or size instead of footprint the system is not running out of memory so I think handling the circle pattern floods the RAM.
I have tried to chunk both the input array aswell as the circle array (i converted the circle array to a dask array) but still getting MemoryErrors.
import numpy as np
import dask.array as da
from dask.distributed import Client, LocalCluster
import dask_image.imread
import dask_image.ndfilters
cluster = LocalCluster(n_workers=1,
threads_per_worker=1,
memory_target_fraction=0.98,
memory_limit='32GB')
client = Client(cluster)
client
# Import raster file as dask array
raster_path = "data_file.tif"
raster_data = dask_image.imread.imread(raster_path)
raster_data = np.squeeze(raster_data, axis=0)
radius = 300
# Create a circle filter for the neighbor analysis
circle_filter = np.zeros((2*radius+1, 2*radius+1), dtype=int)
y, x = np.ogrid[-radius:radius+1, -radius:radius+1]
mask = x**2 + y**2 <= radius**2
circle_filter[mask] = 1
circle_filter = da.from_array(circle_filter)
# Calculate max value in neighorhood
max_value = dask_image.ndfilters.maximum_filter(raster_data, footprint=circle_filter, mode='nearest')
# Convert dask array to numpy array
output = max_value.compute()
My overall goal is to export the array back to a GeoTIFF.
Any suggestions how to handle neighborhood analysis with large circled patterns?
I'm having trouble getting my shapes to work for a Dirichlet Process Gaussian Mixture Model. My data observations has shape (number of samples, number of dimensions). Each Gaussian's mean should be drawn from an isotropic prior, and each Gaussian's covariance should be the identity matrix. I thought I set this up correctly, but I'm getting the following error:
Input dimension mis-match. (input[0].shape[1] = 13, input[1].shape[1] = 2)
My code is:
import numpy as np
import pymc3 as pm
import theano.tensor as tt
num_obs, obs_dim = observations.shape
max_num_clusters = 13
def stick_breaking(beta):
portion_remaining = tt.concatenate([[1], tt.extra_ops.cumprod(1 - beta)[:-1]])
return beta * portion_remaining
with pm.Model() as model:
w = pm.Deterministic("w", stick_breaking(beta))
cluster_means = pm.MvNormal(f'cluster_means',
mu=pm.floatX(np.zeros(obs_dim)),
cov=pm.floatX(gaussian_mean_prior_cov_scaling * np.eye(obs_dim)),
shape=(max_num_clusters, obs_dim))
comp_dists = pm.MvNormal.dist(mu=cluster_means,
cov=gaussian_cov_scaling * np.eye(obs_dim),
shape=(max_num_clusters, obs_dim))
obs = pm.Mixture(
"obs",
w=w,
comp_dists=comp_dists,
observed=observations,
shape=obs_dim)
Can someone clarify how to get the shapes to work?
I have two pictures, one that was the original and another one that I have modified so that it's translated up and left a bit and then rotated 90 degrees (so the shape of the picture is transposed as well).
Now I'd like to determine how many pixels (or any distance unit) the modified picture is translated from the original, as well as the degrees of rotation relative to the original. Phase correlation is supposed to solve this problem by first converting the coordinates to logpolar coordinates, then doing a number of things so that in the end you get a correlation matrix. From that matrix I'm supposed to find the peak and the (x,y) combination will reveal the translation and rotation somehow. This link explains it much better:
Phase correlation
This is the following code I have:
import scipy as sp
from scipy import ndimage
from PIL import Image
from math import *
import numpy as np
def logpolar(input,silent=False):
# This takes a numpy array and returns it in Log-Polar coordinates.
if not silent: print("Creating log-polar coordinates...")
# Create a cartesian array which will be used to compute log-polar coordinates.
coordinates = sp.mgrid[0:max(input.shape)*2,0:360]
# Compute a normalized logarithmic gradient
log_r = 10**(coordinates[0,:]/(input.shape[0]*2.)*log10(input.shape[1]))
# Create a linear gradient going from 0 to 2*Pi
angle = 2.*pi*(coordinates[1,:]/360.)
# Using scipy's map_coordinates(), we map the input array on the log-polar
# coordinate. Do not forget to center the coordinates!
if not silent: print("Interpolation...")
lpinput = ndimage.interpolation.map_coordinates(input,
(log_r*sp.cos(angle)+input.shape[0]/2.,
log_r*sp.sin(angle)+input.shape[1]/2.),
order=3,mode='constant')
# Returning log-normal...
return lpinput
def load_image( infilename ) :
img = Image.open( infilename )
img.load()
data = np.asarray( img, dtype="int32" )
return data
def save_image( npdata, outfilename ) :
img = Image.fromarray( np.asarray( np.clip(npdata,0,255), dtype="uint8"), "L" )
img.save( outfilename )
image = load_image("C:/images/testing_image1.jpg")
target = load_image("C:/images/testing_otherimage.jpg")
# Conversion to log-polar coordinates
lpimage = logpolar(image)
lptarget = logpolar(target)
# Correlation through FFTs
Fcorr = np.fft.fft(lpimage)*np.fft.fft(lptarget)
correlation = np.fft.ifft(Fcorr)
The problem I have now is that this code will give as output:
Traceback (most recent call last):
File "./phase.py", line 44, in <module>
lpimage = logpolar(image)
File "./phase.py", line 24, in logpolar
order=3,mode='constant')
File "C:\Python27\lib\site-packages\scipy\ndimage\interpolation.py", line 295, in map_coordinates
raise RuntimeError('invalid shape for coordinate array')
RuntimeError: invalid shape for coordinate array
As I just have a very superficial understanding of what exactly is happening in the whole phase correlation process, I'm unclear on what the problem is about. I have tried to see if something's wrong with the input so I added save_image(image,"C:/testing.jpg") right after loading the image to see if there's something wrong with the numpy array from my images. And sure enough, the images I convert to np array, cannot be converted back to an image. This is the error I get:
Traceback (most recent call last):
File "./phase.py", line 41, in <module>
save_image(image,"C:/testing.jpg")
File "./phase.py", line 36, in save_image
img = Image.fromarray( np.asarray( np.clip(npdata,0,255), dtype="uint8"), "L" )
File "C:\Python27\lib\site-packages\PIL\Image.py", line 1917, in fromarray
raise ValueError("Too many dimensions.")
ValueError: Too many dimensions.
Taking a peek at the original documentation didn't give me much inspiration on what the problem could be. I don't think the code to convert images to numpy arrays are wrong as I've tested for the type with print type(image) and the results looked legit. Yet I can't convert it back to an image. Any help I can get would be greatly appreciated.
I think the problem is that you are trying to input a 3D image array (R,G,B,A?), into your function. Whereas the input only takes a 2D arrays. Try using a single channel to determine the transformation. E.g.
image = load_image("/path/to/image")[:,:,0]
I am a beginner to python and I am implementing Principal component analysis (PCA) using python, but I am having a problem computing the mean.
Here is my code:
import Image
import os
from PIL import Image
from numpy import *
import numpy as np
#import images
dirname = "C:\\Users\\Karim\\Downloads\\att_faces\\New folder"
X = [np.asarray(Image.open(os.path.join(dirname, fn))) for fn in os.listdir(dirname)]
#get number of images and dimentions
path, dirs, files = os.walk(dirname).next()
num_images = len(files)
image_file = "C:\\Users\\Karim\\Downloads\\att_faces\\New folder\\2.pgm"
img = Image.open(image_file)
width, height = img.size
print width
print height
print num_images
M = (X-mean(X.T,axis=1)).T # subtract the mean (along columns)
I get the error:
AttributeError: 'list' object has no attribute 'T'
The problem is X.T in your last line because X is a python list, not a numpy.ndarray. It isn't clear what you're trying to do here but if you wanted to combine all the image arrays into a single numpy array, you could convert X = np.array(X) before the last line.
Also, unless you specifically want to roll your own PCA implementation, you can do this much more easily with numpy by using np.cov (for covariance calculation) and np.linalg.eig (to compute the eigenvalues and eigenvectors of the covariance matrix).
images -= np.mean(images, axis=0)
I am considering to use OpenCV's Kmeans implementation since it says to be faster...
Now I am using package cv2 and function kmeans,
I can not understand the parameters' description in their reference:
Python: cv2.kmeans(data, K, criteria, attempts, flags[, bestLabels[, centers]]) → retval, bestLabels, centers
samples – Floating-point matrix of input samples, one row per sample.
clusterCount – Number of clusters to split the set by.
labels – Input/output integer array that stores the cluster indices for every sample.
criteria – The algorithm termination criteria, that is, the maximum number of iterations and/or the desired accuracy. The accuracy is specified as criteria.epsilon. As soon as each of the cluster centers moves by less than criteria.epsilon on some iteration, the algorithm stops.
attempts – Flag to specify the number of times the algorithm is executed using different initial labelings. The algorithm returns the labels that yield the best compactness (see the last function parameter).
flags –
Flag that can take the following values:
KMEANS_RANDOM_CENTERS Select random initial centers in each attempt.
KMEANS_PP_CENTERS Use kmeans++ center initialization by Arthur and Vassilvitskii [Arthur2007].
KMEANS_USE_INITIAL_LABELS During the first (and possibly the only) attempt, use the user-supplied labels instead of computing them from the initial centers. For the second and further attempts, use the random or semi-random centers. Use one of KMEANS_*_CENTERS flag to specify the exact method.
centers – Output matrix of the cluster centers, one row per each cluster center.
what is the argument flags[, bestLabels[, centers]]) mean? and what about his one: → retval, bestLabels, centers ?
Here's my code:
import cv, cv2
import scipy.io
import numpy
# read data from .mat file
mat = scipy.io.loadmat('...')
keys = mat.keys()
values = mat.viewvalues()
data_1 = mat[keys[0]]
nRows = data_1.shape[1]
nCols = data_1.shape[0]
samples = cv.CreateMat(nRows, nCols, cv.CV_32FC1)
labels = cv.CreateMat(nRows, 1, cv.CV_32SC1)
centers = cv.CreateMat(nRows, 100, cv.CV_32FC1)
#centers = numpy.
for i in range(0, nCols):
for j in range(0, nRows):
samples[j, i] = data_1[i, j]
cv2.kmeans(data_1.transpose,
100,
criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_MAX_ITER, 0.1, 10),
attempts=cv2.KMEANS_PP_CENTERS,
flags=cv2.KMEANS_PP_CENTERS,
)
And I encounter such error:
flags=cv2.KMEANS_PP_CENTERS,
TypeError: <unknown> is not a numpy array
How should I understand the parameter list and the usage of cv2.kmeans? Thanks
the documentation on this function is almost impossible to find. I wrote the following Python code in a bit of a hurry, but it works on my machine. It generates two multi-variate Gaussian Distributions with different means and then classifies them using cv2.kmeans(). You may refer to this blog post to get some idea of the parameters.
Handle imports:
import cv
import cv2
import numpy as np
import numpy.random as r
Generate some random points and shape them appropriately:
samples = cv.CreateMat(50, 2, cv.CV_32FC1)
random_points = r.multivariate_normal((100,100), np.array([[150,400],[150,150]]), size=(25))
random_points_2 = r.multivariate_normal((300,300), np.array([[150,400],[150,150]]), size=(25))
samples_list = np.append(random_points, random_points_2).reshape(50,2)
random_points_list = np.array(samples_list, np.float32)
samples = cv.fromarray(random_points_list)
Plot the points before and after classification:
blank_image = np.zeros((400,400,3))
blank_image_classified = np.zeros((400,400,3))
for point in random_points_list:
cv2.circle(blank_image, (int(point[0]),int(point[1])), 1, (0,255,0),-1)
temp, classified_points, means = cv2.kmeans(data=np.asarray(samples), K=2, bestLabels=None,
criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_MAX_ITER, 1, 10), attempts=1,
flags=cv2.KMEANS_RANDOM_CENTERS) #Let OpenCV choose random centers for the clusters
for point, allocation in zip(random_points_list, classified_points):
if allocation == 0:
color = (255,0,0)
elif allocation == 1:
color = (0,0,255)
cv2.circle(blank_image_classified, (int(point[0]),int(point[1])), 1, color,-1)
cv2.imshow("Points", blank_image)
cv2.imshow("Points Classified", blank_image_classified)
cv2.waitKey()
Here you can see the original points:
Here are the points after they have been classified:
I hope that this answer may help you, it is not a complete guide to k-means, but it will at least show you how to pass the parameters to OpenCV.
The problem here is your data_1.transpose is not a numpy array.
OpenCV 2.3.1 and higher python bindings do not take anything except numpy array as image/array parameters. so, data_1.transpose has to be a numpy array.
Generally, all the points in OpenCV are of type numpy.ndarray
eg.
array([[[100., 433.]],
[[157., 377.]],
.
.
[[147., 247.]], dtype=float32)
where each element of array is
array([[100., 433.]], dtype=float32)
and the element of that array is
array([100., 433.], dtype=float32)