We Keep Coding

numpy.fromfile seems to be unable to read large files

I wanted to write some very simple python helper tool for my project which is reading binary data from an ECG record. I have found somewhere that numpy.fromfile is the most appropriate tool to approach it, so I wrote:
#!/usr/bin/env python3
import sys
import numpy as np
arrayOfNums = np.fromfile(sys.argv[1], 'short')
print("Converting " + sys.argv[1] + "...")
conversionOutput = open("output", "x")
conversionOutput.write(np.array2string(arrayOfNums, separator=' '))
conversionOutput.close()
print("Conversion done.")
I did that to write the data which is 2 byte records unseparated. The input file is somewhat large for a simple text file (over 7MB), however not large enough I think to cause numpy troubles.
The output I got in the file: [-32243 -32141 -32666 ... -32580 -32635 -32690]
Why the dots between? It seems to convert it okay, but omits almost everything it is supposed to save. Any help would be appreciated.

Numpy reads correctly your file. To avoid a long display, numpy uses the dots:
import numpy as np
a = np.random.random(10000)
Output:
>>> a
array([0.20902653, 0.80097215, 0.06909818, ..., 0.5963183 , 0.94024005,
0.31870234])
>>> a.shape
(10000,)
a contains 10000 values and not only the 6 displayed values.
Update
To display the full output:
import sys
np.set_printoptions(threshold=sys.maxsize)
print(a)

Preserving the WCS information of a FITS file when rebinned

Aim : Rebin an existing image (FITS file) and write the new entries into a new rebinned image (also a FITS file).
Issue : Rebinned FITS file and the original FITS file seem to have mismatched co-ordinates (figure shown later in the question).
Process : I will briefly describe my process to shed more light. The first step is to read the existing fits file and define numpy arrays
from math import *
import numpy as np
import matplotlib.pyplot as plt
from astropy.visualization import astropy_mpl_style
from astropy.io import fits
import matplotlib.pyplot as plt
%matplotlib notebook
import aplpy
from aplpy import FITSFigure
file = 'F0621_HA_POL_0700471_HAWDHWPD_PMP_070-199Augy20.fits'
hawc = fits.open(file)
stokes_i = np.array(hawc[0].data)
stokes_i_rebinned = congrid(stokes_i,newdim,method="neighbour", centre=False, minusone=False)
Here "congrid" is a function I have used for near-neigbhour rebinning that rebins the original array to a new dimension given by "newdim". Now the goal is to write this rebinned array back into the FITS file format as a new file. I have several more such arrays but for brevity, I just include one array as an example. To keep the header information same, I read the header information of that array from the existing FITS file and use that to write the new array into a new FITS file. After writing, the rebinned file can be read just like the original :-
header_0= hawc[0].header
fits.writeto("CasA_HAWC+_rebinned_congrid.fits", rebinned_stokes_i, header_0, overwrite=True)
rebinned_file = 'CasA_HAWC+_rebinned_congrid.fits'
hawc_rebinned= fits.open(rebinned_file)
To check how the rebinned image looks now I plot them
cmap = 'rainbow'
stokes_i = hawc[0]
stokes_i_rebinned = hawc_rebinned[0]
axi = FITSFigure(stokes_i, subplot=(1,2,1)) # generate FITSFigure as subplot to have two axes together
axi.show_colorscale(cmap=cmap) # show I
axi_rebinned = FITSFigure(stokes_i_rebinned, subplot=(1,2,2),figure=plt.gcf())
axi_rebinned.show_colorscale(cmap=cmap) # show I rebinned
# FORMATTING
axi.set_title('Stokes I (146 x 146)')
axi_rebinned.set_title('Rebinned Stokes I (50 x 50)')
axi_rebinned.axis_labels.set_yposition('right')
axi_rebinned.tick_labels.set_yposition('right')
axi.tick_labels.set_font(size='small')
axi.axis_labels.set_font(size='small')
axi_rebinned.tick_labels.set_font(size='small')
axi_rebinned.axis_labels.set_font(size='small')
As you see for the original and rebinned image, the X,Y co-ordinates seem mismatched and my best guess was that WCS (world co-ordinate system) for the original FITS file wasn't properly copied for the new FITS file, thus causing any mismatch. So how do I align these co-ordinates ?
Any help will be deeply appreciated ! Thanks

I'm posting my answer in an astropy slack channel here should this be useful for others.
congrid will not work because it doesn't include information about the WCS. For example, your CD matrix is tied to the original image, not the re-binned set. There are a number of way to re-bin data with proper WCS. You might consider reproject although this often requires another WCS header to re-bin to.
Montage (though not a Python tool but has Python wrappers) is potentially another way.

As #astrochun already said, your re-binning function does not adjust the WCS of the re-binned image. In addition to reproject and Montage, astropy.wcs.WCSobject has slice() method. You could try using it to "re-bin" the WCS like this:
from astropy.wcs import WCS
import numpy as np
wcs = WCS(hawc[0].header, hawc)
wcs_rebinned = wcs.slice((np.s_[::2], np.s_[::2]))
wcs_hdr = wcs_rebinned.to_header()
header_0.update(wcs_hdr) # but watch out for CD->PC conversion
You should also make a "real" copy of hawc[0].header in header_0= hawc[0].header, for example as header_0= hawc[0].header.copy() or else header_0.update(wcs_hdr) will modify hawc[0].header as well.

Plotting a large number of stacked spheres generated from python-- mayavi? paraview and pyevtk? How to transmute .npy to .vtk?

I want to produce plots like this, except with many more particles. Matplotlib is woefully inadequate.
Right now I am using mayavi in python 3.5 running through a jupyter notebook. As I need to plot 5x10^5 spheres it will not be practical, since time is a limiting factor already at 2x10^4 spheres.
Here is my python code to produce the mayavi plot. I have a numpy array of values [a,r,x,y,z]. It's not relevant what the first quantity is for this problem.
"""VISUALIZATION WITH MAYAVI"""
#I think this is too slow to be practical.
#view particles with mayavi
import mayavi
from mayavi import mlab
%gui qt
def plot_sphere(p): #feed it p and get back a sphere \n",
t1,R,a,b,c = p
[phi,theta] = np.mgrid[0:2*np.pi:12j,0:np.pi:12j] #increase the numbers before j for better resolution but more time
x = R*np.cos(phi)*np.sin(theta) + a
y = R*np.sin(phi)*np.sin(theta) + b
z = R*np.cos(theta) + c
return mlab.mesh(x, y, z)
#run s over all particles and plot it
def view(particles):
for p in particles:
plot_sphere(p)
view(spheres)
This code produces plots like this:
I have been told I should look into writing my numpy arrays to .vtk files using evtk, then visualizing these in paraview. I downloaded paraview and read this, but perhaps my version of python is limiting me? First, install pyevtk-- okay:
I tried conda install -c opengeostat pyevtk=1.0.0, but it fails due to incompatibility with my python version. I looked for details but could not find any.
Next I downloaded the repository [here][https://pypi.python.org/pypi/PyEVTK/1.0.0], then used pip to install it successfully.
Next I put evtk.py, vtk.py, hl.py, and xml.py, and tried some examples in the repository-- none of them work. Seemingly there is some problem with
from .vtk import *
type commands. I tried replacing all of these in the four .py files with
from evtk import vtk
from vtk import *
and such, but no luck. Long story short, I can't get pyevtk working to export my numpy arrays as .vtk files. I could use some help in this regard, or better yet I would love a different option to get my numpy arrays rendered by paraview. Any help is appreciated !

Ok, I solved my own problem. This image is made using paraview, after converting numpy arrays to a .vtu object using pyevtk.
Out of the box, the repository did not work, there was some problem with importing the modules inside the four .py files, so I modified them all. Instead of from .vtk import *, I changed it to from vtk import *, and so on, in every module in the library. evtk.py was not able to import a class from xml.py, so I just copied it and pasted, then deleted xml.py. After some tinkering and clueless modifying to make the errors go away, eventually it worked.

Python error : index out of bounds

I was curious about image processing with python, so I found this great library imageio,
I tried to manipulate the pixels of a picture and save them in a new file,
but i had some problems with the loops
this is what the code looks like
enter image description here
and this the error that i Got !
IndexError: index 3507 is out of bounds for axis 0 with size 3507
the code :
# -*- coding: iso-8859-1 -*-
import imageio
import numpy as np
im = imageio.imread("JAFFRE009a.png")
taille=im.shape #taille is a tuple (Width,Height)
print taille # (4961,3507)
matrice_pixels=open("matrice.txt",'w')
for i in range(taille[1]):
line=""
for j in range(taille[0]):
line+=repr(im[i][j])
matrice_pixels.write(line+'\n')
matrice_pixels.close()

Because your image doesn't have squarred shape, reshape it before you go through your loop

EDIT
We can iterate through each row/column position and save to a file as below.It will take very long time depending upon file size.
Instead of writing your own function, you may want to take advantage of inbuilt binary save (which is more efficient) as
np.save('matrix.py', np_array)
You can load this file as np array and manipulate
Or as a text file using np.save [ will take longer ]
np.save('matrix.txt', np_array)
Working Code:
import imageio
import numpy as np
im = imageio.imread("9v9zU.png")
matrice_pixels=open("matric.txt","wb")
nx,ny = im.shape
for i in range(nx):
line=""
for j in range(ny):
line+=repr(im[i][j])
matrice_pixels.write(line+'\n')
matrice_pixels.close()
#Save as Binary data
np.save('matrix1.npy', im)
#Save as Human readable data
np.savetxt('matrix1.txt', im)
Alternately, you may want to look into off the shelf libraries that will do what you are intending to do.
For e.g. This SO link discusses how to remove section of the picture based upon its color using PIL library.
Also , in future, please DO NOT post a picture of your code. Copy/pase to SO window so that we can copy and modify. In this case I had write everything down line by line to test(thankfully code was not that long).

Pure python way to compress a numpy array to a text or image file?

I'm looking for a way in pure python to export a numpy array to either a text file or a compressed image file. My array is 500x700 so exporting it produces a file with a size of ~3mb. I need it to be at least under 1mb.
I've tried the tifffile package but the compression modes are only from 1 to 9 so it is not enough.
My file could be in a "matrix" format, a X Y Z format, a tiff image or other format type that does not lose data when compressing and that are compatible with GIS softwares.
I also need it to be in pure python and without external requirements (like calling a subprocess) if possible to avoid compatibility problems when running on a distant machine.¸
Any ideas?

If you need it for GIS software, then use either GDAL or rasterio. Use, for example, the GTiff driver to make a GeoTiff.
Assuming you have floats, here is GDAL:
import numpy as np
from osgeo import gdal
gdal.UseExceptions()
driver = gdal.GetDriverByName('GTiff')
ds = driver.Create('file.tif', 500, 700, 1, gdal.GDT_Float32, ['COMPRESS=LZW'])
ly = ds.GetRasterBand(1)
ly.WriteArray(np.arange(500 * 700).reshape(700, 500))
ly = ds = None # save, close
Or rasterio:
import rasterio
with rasterio.open('file2.tif', 'w', 'GTiff', width=500, height=700, count=1, dtype='f', COMPRESS='LZW'):
ds.write(np.arange(500 * 700, dtype='f').reshape(1, 700, 500))
These files are <1 MB. You can get smaller if you use Byte or Int16 types.
(Note: projection or georeferencing were not added)

Have a look at np.savez_compressed
numpy.savez_compressed(file, *args, **kwds)
Save several arrays into a single file in compressed .npz format.
Example
from tempfile import TemporaryFile
outfile = TemporaryFile()
x = np.arange(10)
y = np.sin(x)
np.savez_compressed(outfile, x, y)
outfile.seek(0) # Only needed here to simulate closing & reopening file
npzfile = np.load(outfile)
npzfile.files
npzfile['arr_0']