I want to create a memmap in MATLAB.
In python I could do this by:
ut = np.memmap('my_array.mmap', dtype=np.float64, mode='w+', shape=(140000,3504))
Then I use it as a normal array, the OS ensured my memory never overflowed. How to do this in MATLAB?
From the docs it seems it wants me to create some array in MATLAB first then write it to a file and read using memmap!
Matlab docs are not clear enough:
Please provide an example of creating an random array of size (140000,15000) and multiply it some other similar matrix.
You have to create an empty file first, then use memmapfile:
size=[140000,3504];
filesize=0;
datatype='float64';
filename='my_array.dat';
fid=fopen(filename,'w+');
max_chunk_size=1000000;
%fills an empty file
while filesize<prod(size)
to_write=min(prod(size)-filesize,max_chunk_size);
filesize=filesize+fwrite(f, zeros(to_write,1), datatype);
end
fclose(fid);
m = memmapfile(filename,'Format','double', 'Writable',true);
I think what you are looking for is the function memmapfile
Example:
m = memmapfile('my_array.dat','Format','double', 'Writable',true)
Related
I have an image sequence as a numpy array;
Mov (15916, 480, 768)
dtype = int16
i've tried using Mov.tofile(filename)
this saves the array and I can load it again in python and view the images.
In matlab the images are corrupted after about 3000 frames.
Using the following also works but has the same problem when I retrieve the images in matlab;
fp = np.memmap(sbxpath, dtype='int16', mode='w+', shape=Mov.shape)
fp[:,:,:] = Mov[:,:,:]
If I use:
mv['mov'] = Mov
sio.savemat(sbxpath, mv)
I get the following error;
OverflowError: Python int too large to convert to C long
what am I doing wrong?
I'm sorry for this, because it is a beginners problem. Python saves variables as integers or floats depending on how they are initialized. Matlab defaults to 8 byte doubles. My matlab script expects doubles, my python script was outputting all kinds of variable types, so naturally things got messed up.
I am trying to read in a large structure from a .mat file into a jupyter notebook. I am a little new to python so I'm not sure why my solution isn't working.
The structure from MATLAB(2020) I am reading in is structured like this
pose.frames.ind
Where there are 44 frames and in each frame 63 ind. I am reading it into a jupyter notebook like this with mat4py. I am trying to adapt someone else's code so after I read it in I need to convert it to a tensor so it can go into another function.
from mat4py import loadmat
val = loadmat('pose.mat')
pose_body = val['pose']['frames']['ind'][0]
pose_body = np.asarray(pose_body)
pose_body = torch.FloatTensor(pose_body).to(comp_device)
When I feed pose_body = np.zeros([1,63]) into the line that changes it to a torch tensor, the code works fine. However, when I try to feed it this array I imported something goes wrong and I get the error of
RuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 2 and 1 at C:/w/1/s/tmp_conda_3.7_055306/conda/conda-bld/pytorch_1556690124416/work/aten/src\THC/generic/THCTensorMath.cu:62
Is there an easier/better way to convert the data from matlab in the format I need? I am a little unfamiliar with python, and it seems like they're the same type of array. From doing type(pose_body), and tf.size(pose_body) I know that both numpy arrays have a shape of 63, a dtype of int32, and are of class "numpy.ndarray".
I try to use h5py to open a file which was created by another program. Unfortunately I don't know the inner structure of the file. All I know is that it should contain a 20x20 matrix which I would like to process with numpy.
Here is what I have done so far:
import numpy
import h5py
f = h5py.File('example.hdf5')
print(f.keys())
The result is as follows:
KeysViewWithLock(<HDF5 file "example.hdf5" (mode r+)>)
How do I go from here? I want to access the matrix as a single numpy.ndarray. The h5py documentation always talks about creating hdf5 files, not reading unknown files.
Thanks a lot.
SOLUTION (thanks to akash karothiya)
use print(list(f.keys())) instead. That gives the names of groups/datasets which can then be accessed as a=f['dataset'].
Ok, as mentioned before akash karothiya helped me find the solution.
Instead of print(f.keys()) use print(list(f.keys())). This returns ['dataset'].
Using this information I can get an h5py dataset object which I then converted into a numpy array as follows:
a = f['dataset']
b = numpy.zeros(np.shape(a), dtype=complex)
for i in range(numpy.size(a,0)):
b[i,:] = np.asarray(a[i]['real'] + 1j*a[i]['imag'], dtype=complex)
UPDATE:
New version without for loop, potentially faster and very versatile (works for both complex and real data and cubes with dimensions NxMxO as well):
a = f['dataset']
if len(a.dtype) == 0:
b = np.squeeze(a[()])
elif len(a.dtype) == 2:
b = np.squeeze(a[()]['real'] + 1.0j*a[()]['imag'])
I want to write some random numbers into an ascii output file.
I generate the numbers with numpy, so the numbers are stored in numpy.array
import numpy as np
random1=np.random.uniform(-1.2,1.2,7e6)
random2=...
random3=...
All three array are of the same size.
I used standard file output, but this is really slow. Just about 8000 lines per 30 min. This may because I loop over three large arrays though.
fout1 = open("output.dat","w")
for i in range(len(random1)):
fout1.write(str(random1[i])+"\t"+ str(random2[i])+"\t"+ str(random3[i])+"\n")
fout1.close()
I also just used print str(random1[i])+"\t"+ str(random2[i])+"\t"+ str(random3[i]) and dumped everything in a file usind shell ./myprog.py > output.dat which seems a bit faster but still I am not satisfied with the output speed.
Any recommendations are really welcome.
Have you tried
random = np.vstack((random1, random2, random3)).T
random.savetxt("output.dat", delimiter="\t")
Im guessing the disk io is the most expensive operation you are doing.. You could try to create your own buffer to deal with this, instead of writing every line every loop buffer up say 100 lines and write them in one big block. Then experiment with this and see what the most benficial buffer size is
I'm working with Python and MATLAB right now and I have a 2D array in Python that I need to write to a file and then be able to read it into MATLAB as a matrix. Any ideas on how to do this?
Thanks!
If you use numpy/scipy, you can use the scipy.io.savemat function:
import numpy, scipy.io
arr = numpy.arange(9) # 1d array of 9 numbers
arr = arr.reshape((3, 3)) # 2d array of 3x3
scipy.io.savemat('c:/tmp/arrdata.mat', mdict={'arr': arr})
Now, you can load this data into MATLAB using File -> Load Data. Select the file and the arr variable (a 3x3 matrix) will be available in your environment.
Note: I did this on scipy 0.7.0. (scipy 0.6 has savemat in the scipy.io.mio module.) See the latest documentation for more detail
EDIT: updated link thanks to #gnovice.
I think ars has the most straight-forward answer for saving the data to a .mat file from Python (using savemat). To add just a little to their answer, you can also load the .mat file into MATLAB programmatically using the LOAD function instead of doing it by hand using the MATLAB command window menu...
You can use either the command syntax form of LOAD:
load c:/tmp/arrdata.mat
or the function syntax form (if you have the file path stored in a string):
filePath = 'c:/tmp/arrdata.mat';
data = load(filePath);
I wrote a small function to do this same thing, without need for numpy. It takes a list of lists and returns a string with a MATLAB-formatted matrix.
def arrayOfArrayToMatlabString(array):
return '[' + "\n ".join(" ".join("%6g" % val for val in line) for line in array) + ']'
Write "myMatrix = " + arrayOfArrayToMatlabString(array) to a .m file, open it in matlab, and execute it.
I would probably use numpy.savetxt('yourfile.mat',yourarray) in Python
and then yourarray = load('yourfile.mat') in MATLAB.
You could write the matrix in Python to a CSV file and read it in MATLAB using csvread.
You can also call matlab directly from python:
from mlabwrap import mlab
import numpy
a = numpy.array([1,2,3])
mlab.plot(a)
The toolbox npy-matlab can read *.npy binary files into MATLAB. *.npy files can be directly exported with the NumPy module. From the documentation:
>> a = rand(5,4,3);
>> writeNPY(a, 'a.npy');
>> b = readNPY('a.npy');
>> sum(a(:)==b(:))
ans =
60
npy-matlab is a simple collection of M-files available from GitHub, with a 2-clause BSD licence.