We Keep Coding

Build Shapely point objects from .TIF

I would like to convert an image (.tiff) into Shapely points. There are 45 million pixels, I need a way to accomplish this without a loop (currently taking 15+ hours)
For example, I have a .tiff file which when opened is a 5000x9000 array. The values are pixel values (colors) that range from 1 to 215.
I open tif with rasterio.open(xxxx.tif).
Desired epsg is 32615
I need to preserve the pixel value but also attach geospatial positioning. This is to be able to sjoin over a polygon to see if the points are inside. I can handle the transform after processing, but I cannot figure a way to accomplish this without a loop. Any help would be greatly appreciated!

If you just want a boolean array indicating whether the points are within any of the geometries, I'd dissolve the shapes into a single MultiPolygon then use shapely.vectorized.contains. The shapely.vectorized module is currently not covered in the documentation, but it's really good to know about!
Something along the lines of
# for a gridded dataset with 2-D arrays lats, lons
# and a list of shapely polygons/multipolygons all_shapes
XX = lons.ravel()
YY = lats.ravel()
single_multipolygon = shapely.ops.unary_union(all_shapes)
in_any_shape = shapely.vectorized.contains(single_multipolygon, XX, YY)
If you're looking to identify which shape the points are in, use geopandas.points_from_xy to convert your x, y point coordinates into a GeometryArray, then use geopandas.sjoin to find the index of the shape corresponding to each (x, y) point:
geoarray = geopandas.points_from_xy(XX, YY)
points_gdf = geopandas.GeoDataFrame(geometry=geoarray)
shapes_gdf = geopandas.GeoDataFrame(geometry=all_shapes)
shape_index_by_point = geopandas.sjoin(
shapes_gdf, points_gdf, how='right', predicate='contains',
)
This is still a large operation, but it's vectorized and will be significantly faster than a looped solution. The geopandas route is also a good option if you'd like to convert the projection of your data or use other geopandas functionality.

Power spectrum plot from numpy data set by using healpy

I am just starting to use healpy in python. In shortly, I have a data of position let
x = np.array([7.47098722,  7.47805867,  7.47098779....12.57804461, 12.5809734])
y = np.array([58.32516929, 58.33223959, 58.33516828, ..., 63.41415155, 63.41707922])
where phi(x, y) is the potential of the matter at these location. Now, I want to calculate the power spectrum for this. But I don't understand how to set my data for healpy to make a map by using mollview and calculating the power spectrum by using anafast.

You just need to convert the thetas and phis to pixel indices using one of
healpy's pixel functions, healpy.pixelfunc.ang2pix. You can read more about it from https://healpy.readthedocs.io/en/latest/generated/healpy.pixelfunc.ang2pix.html
After you have indices, you also have potential values for those indices, in other words, you have the corresponding HEALPix map(use mollview to get a
mollview projection of this map). Finally, use healpy's anafast, https://healpy.readthedocs.io/en/latest/healpy_spht.htm and you have you power spectrum.
For more information about healpy, kindly refer-
https://buildmedia.readthedocs.org/media/pdf/healpy/1.8.6/healpy.pdf
https://healpix.sourceforge.io/
also, visit https://healpix.jpl.nasa.gov

Apply rotation to HEALPix map in healpy

I have a HEALPix map that I have read in using healpy, however it is in galactic coordinates and I need it in celestial/equatorial coordinates. Does anybody know of a simple way to convert the map?
I have tried using healpy.Rotator to convert from (l,b) to (phi,theta) and then using healpy.ang2pix to reorder the pixels, but the map still looks strange.
It would be great if there was a function similar to Rotator that you could call like: map = AnotherRotator(map,coord=['G','C']). Anybody know of any such function??
Thanks,
Alex

I realise that this was asked ages ago, but I was having the same problem myself this week and found your post. I've found a couple of potential solutions, so I'll share incase someone else comes upon this and finds it useful.
Solution 1: This sort of depends on the format your data is coming in. Mine came in a (theta, phi) grid.
import numpy as np
import healpy as H
map = <your original map>
nside = <your map resolution, mine=256>
npix = H.nside2npix(nside)
pix = N.arange(npix)
t,p = H.pix2ang(nside,pix) #theta, phi
r = H.Rotator(deg=True, rot=[<THETA ROTATION>, <PHI ROTATION>])
map_rot = np.zeros(npix)
for i in pix:
trot, prot = r(t[i],p[i])
tpix = int(trot*180./np.pi) #my data came in a theta, phi grid -- this finds its location there
ppix = int(prot*180./np.pi)
map_rot[i] = map[ppix,tpix] #this being the rright way round may need double-checking
Solution 2: Haven't quite finished testing this, but just came across it AFTER doing the annoying work above...
map_rot = H.mollview(map,deg=True,rot=[<THETA>,<PHI>], return_projected_map=True)
which gives a 2D numpy array. I'm interested to know how to convert this back into a healpix map...

I found another potential solution, after searching off and on for a few months. I haven't tested it very much yet, so please be careful!
Saul's Solution 2, above, is the key (great suggestion!)
Basically, you combine the functionality of healpy.mollview (gnomview, cartview, and orthview work as well) with that of the reproject_to_healpix function in the reproject package (http://reproject.readthedocs.org/en/stable/).
The resulting map is suitable for my angular scales, but I can't say how accurate the transformation is compared to other methods.
-----Basic Outline----------
Step 1: Read-in the map and make the rectangular array via cartview. As Saul indicated above, this is also one way to do the rotation. If you're just doing a standard rotation/coordinate transformation, then all you need is the coord keyword. From Celestial to Galactic coordinates, set coord = ['C','G']
map_Gal = hp.cartview(map_Cel, coord=['C','G'], return_projected_map=True, xsize=desired_xsize, norm='hist',nest=False)
Step 2: Write a template all-sky FITS header (as in the example below). I wrote mine to have the same average pixel-scale as my desired HEALPix map.
Step 3: Use reproject.transform_to_healpix
reproject includes a function for mapping a "normal" array (or FITS file) into the HEALPix projection. Combine that with the ability to return the array created by healpy.mollview/cartview/orthview/gnomview, and you can rotate a HEALPix map of one coordinate system (Celestial) into another coordinate system (Galactic).
map_Gal_HP, footprint_Gal_HP = rp.reproject_to_healpix((map_Gal, target_header), coord_system_out= 'GALACTIC', nside=nside, nested=False)
It comes down, essentially, to those two commands. However you'll have to make a template header, giving the pixel scale and size corresponding to the intermediary all-sky map you want to make.
-----Full Working Example (iPython notebook format + FITS sample data)------
https://github.com/aaroncnb/healpix_coordtrans_example/tree/master
The code there should run very quickly, but that's because the maps are heavily degraded. I did the same for my NSIDE 1024 and 2048 maps, and it took
about an hour.
------Before and After Images------

This function seems to do the trick (reasonably slow, but should be better than the for loop):
def rotate_map(hmap, rot_theta, rot_phi):
"""
Take hmap (a healpix map array) and return another healpix map array
which is ordered such that it has been rotated in (theta, phi) by the
amounts given.
"""
nside = hp.npix2nside(len(hmap))
# Get theta, phi for non-rotated map
t,p = hp.pix2ang(nside, np.arange(hp.nside2npix(nside))) #theta, phi
# Define a rotator
r = hp.Rotator(deg=False, rot=[rot_phi,rot_theta])
# Get theta, phi under rotated co-ordinates
trot, prot = r(t,p)
# Interpolate map onto these co-ordinates
rot_map = hp.get_interp_val(hmap, trot, prot)
return rot_map
Using this on data from PyGSM gives the following:
hp.mollview(np.log(rotate_map(gsm.generated_map_data, 0,0)))
Upon rotation of phi:
hp.mollview(np.log(rotate_map(gsm.generated_map_data, 0,np.pi)))
Or rotating theta:
hp.mollview(np.log(rotate_map(gsm.generated_map_data, np.pi/4,0)))

Scipy normalization-- localize values to set discrete points

I am currently displaying two separate 2D images (x,y plane and z,y plane) that are derived from 96x512 arrays of 0-255 values. I would like to be able to filter the data so that anything under a certain value is done away with (the highest values are indicative of targets). What I would like to be able to do is from these images, separate discrete points that may be then mapped three-dimensionally as points, rather than mapping two intersecting planes. I'm not entirely sure how to do this or where to start (I'm very new to python). I am producing the images using scipy and have done some normalization and noise reduction, but I'm not sure how to then separate out anything over the threshold as it's own individual point. Is this possible?

If I understand correctly what you want, filtering points can be done like this:
A=numpy.random.rand(5,5)
B=A>0.5
Now B is a binary mask, and you can use it in a number of ways:
A[B]
will return an array with all values of A that are true in B.
A[B]=0
will assign 0 to all values in A that are true in B.
numpy.nonzero(B)
will give you the x,y coordinates of each point that is true in B.

Resizing image algorithm in python

So, I'm learning my self python by this tutorial and I'm stuck with exercise number 13 which says:
Write a function to uniformly shrink or enlarge an image. Your function should take an image along with a scaling factor. To shrink the image the scale factor should be between 0 and 1 to enlarge the image the scaling factor should be greater than 1.
This is not meant as a question about PIL, but to ask which algorithm to use so I can code it myself.
I've found some similar questions like this, but I dunno how to translate this into python.
Any help would be appreciated.
I've come to this:
import image
win = image.ImageWin()
img = image.Image("cy.png")
factor = 2
W = img.getWidth()
H = img.getHeight()
newW = int(W*factor)
newH = int(H*factor)
newImage = image.EmptyImage(newW, newH)
for col in range(newW):
for row in range(newH):
p = img.getPixel(col,row)
newImage.setPixel(col*factor,row*factor,p)
newImage.draw(win)
win.exitonclick()
I should do this in a function, but this doesn't matter right now. Arguments for function would be (image, factor). You can try it on OP tutorial in ActiveCode. It makes a stretched image with empty columns :.

Your code as shown is simple and effective for what's known as a Nearest Neighbor resize, except for one little bug:
p = img.getPixel(col/factor,row/factor)
newImage.setPixel(col,row,p)
Edit: since you're sending a floating point coordinate into getPixel you're not limited to Nearest Neighbor - you can implement any interpolation algorithm you want inside. The simplest thing to do is simply truncate the coordinates to int which will cause pixels to be replicated when factor is greater than 1, or skipped when factor is less than 1.

Mark has the correct approach. To get a smoother result, you replace:
p = img.getPixel(col/factor,row/factor)
with a function that takes floating point coordinates and returns a pixel interpolated from several neighboring points in the source image. For linear interpolation it takes the four nearest neigbors; for higher-order interpolation it takes a larger number of surrounding pixels.
For example, if col/factor = 3.75 and row/factor = 1.9, a linear interpolation would take the source pixels at (3,1), (3,2), (4,1), and (4,2) and give a result between those 4 rgb values, weighted most heavily to the pixel at (4,2).

You can do that using the Python Imaging Library.
Image.resize() should do what you want.
See http://effbot.org/imagingbook/image.htm
EDIT
Since you want to program this yourself without using a module, I have added an extra solution.
You will have to use the following algorithm.
load your image
extract it's size
calculate the desired size (height * factor, width * factor)
create a new EmptyImage with the desired size
Using a nested loop through the pixels (row by column) in your image.
Then (for shrinking) you remove some pixels every once in while, or for (enlarging) you duplicate some pixels in your image.
If you want you want to get fancy, you could smooth the added, or removed pixels, by averaging the rgb values with their neighbours.