I'm trying to use FortranFile to get an output that I can use in my simulation code written in F95. I'm having troubles getting fortranfile to work properly. Maybe it's because I do not understand how it works. Here's my problem:
If I want to write a 1D array using FortranFile, it works fine:
nx = 128
bxo = np.zeros(nx, dtype=float)
bxo = something
import fortranfile as fofi
bxf=fofi.FortranFile('Fbx.dat',mode='w')
bxf.writeReals(bxo,prec='d')
bxf.close()
The above 1D version works like a charm. As soon as I try to do it for a 2D array, I get problems
nx = 128; ny = 128
bxo = np.zeros((nx,ny), dtype=float)
bxo = something
import fortranfile as fofi
bxf=fofi.FortranFile('Fbx.dat',mode='w')
bxf.writeReals(bxo,prec='d')
bxf.close()
When I try to do this, I get the following error:
---------------------------------------------------------------------------
error Traceback (most recent call last)
/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/IPython/utils/py3compat.py in execfile(fname, *where)
173 else:
174 filename = fname
--> 175 __builtin__.execfile(filename, *where)
/Users/parashar/Dropbox/sandbox/test2d.py in <module>()
130 vyf=fofi.FortranFile('Fvy.dat',mode='w')
131 vzf=fofi.FortranFile('Fvz.dat',mode='w')
--> 132 bxf.writeReals(bxo,prec='d')
133 byf.writeReals(byo,prec='d')
134 bzf.writeReals(bzo,prec='d')
/Users/parashar/Dropbox/sandbox/fortranfile.py in writeReals(self, reals, prec)
215 _fmt = self.ENDIAN + prec
216 for r in reals:
--> 217 self.write(struct.pack(_fmt,r))
218 self._write_check(length_bytes)
219
error: required argument is not a float
Any ideas what could be going on?
Thanks!
I like Emmet's suggestion but given that my knowledge of python is very basic, it would be a lot of effort for a short goal. I just realized that I could handle the situation in a slightly different manner.
Direct access files in Fortran do not have the unnecessary leading/trailing information that the usual unformatted fortran files do. So the simplest way to deal with unformatted data exchanged between Fortran and Python is to treat the files as direct access in Fortran. Here is an example of how we can use data to/from python/fortran.
Python code:
import numpy as np
nx=128; ny=128;
bxo=np.zeros((nx,ny),dtype=float)
bxo=something
bxf=open('Fbx.dat',mode='wb')
np.transpose(bxo).tofile(bxf) # We transpose the array to map indices
# from python to fortran properly
bxo.close()
Fortran code:
program test
implicit none
double precision, dimension(128,128) :: bx
integer :: NNN, i, j
inquire(iolength=NNN) bx
open(unit=23,file='Fbx.dat',form='unformatted',status='old',&
access='direct',recl=NNN)
read(23,rec=1) bx
close(23)
! Write it out to a text file to test it
! by plotting in gnuplot
do i=1,128; do j=1,128
write(23,*) i,j,bx(i,j)
enddo; enddo
end
Because we are using the standard binary formate to read/write data, this method will work with arrays of any size unlike the FortranFile method.
I have realized that by sticking to direct access files in Fortran, we can have wider compatibility with other languages e.g. Python, IDL etc. This way we do not have to worry about the weird leading trailing markers, endian-ness etc.
I hope this will help someone else in my situation too.
I've never had to do anything with Fortran and Python at the same time, and know absolutely nothing about fortranfile.py, but my best guess is that fortranfile is not numpy-aware.
When you use 1D numpy arrays, or Python arrays, lists, etc. the iteration in the last part of the stack trace suggests that it's expecting an iterable of numbers ("for r in reals"), whereas when you try to serialize a 2D numpy array, I'm not sure what it gets (i.e. what "r" ends up being), maybe an iterable of iterators, or an iterable of 1D arrays. In short, 'r' is not just the number that's expected ("required argument is not a float"), but something else (like a 1D array, list, etc.).
I would try have a look and see if there's an alternative to writeReals() in fortranfile and, if not, hack one in that can handle 2D arrays with a bit of copypasta.
I would start by putting in a diagnostic print before that "self.write()" line (217) that tells you what 'r' actually is, since it isn't the expected float.
Related
I have an image sequence as a numpy array;
Mov (15916, 480, 768)
dtype = int16
i've tried using Mov.tofile(filename)
this saves the array and I can load it again in python and view the images.
In matlab the images are corrupted after about 3000 frames.
Using the following also works but has the same problem when I retrieve the images in matlab;
fp = np.memmap(sbxpath, dtype='int16', mode='w+', shape=Mov.shape)
fp[:,:,:] = Mov[:,:,:]
If I use:
mv['mov'] = Mov
sio.savemat(sbxpath, mv)
I get the following error;
OverflowError: Python int too large to convert to C long
what am I doing wrong?
I'm sorry for this, because it is a beginners problem. Python saves variables as integers or floats depending on how they are initialized. Matlab defaults to 8 byte doubles. My matlab script expects doubles, my python script was outputting all kinds of variable types, so naturally things got messed up.
I am trying to use np.fromfile in order to read a binary file that I have written with Fortran using direct access. However if I set count=-1, instead of max_items, np.fromfile returns a larger array than expected; adding zeros to the vector I've written in binary.
Fortran test code:
program testing
implicit none
integer*4::i
open(1,access='DIRECT', recl=20, file='mytest', form='unformatted',convert='big_endian')
write(1,rec=1) (i,i=1,20)
close(1)
end program
How I am using np.fromfile:
import numpy as np
f=open('mytest','rb')
f.seek(0)
x=np.fromfile(f,'>i4',count=20)
print len(x),x
so if I use it like this it returns exactly my [1,...,20] np array, but setting count=-1 returns [1,...,20,0,0,0,0,0] with a size of 1600.
I am using a little endian machine (shouldn't affect anything) and I am compiling the Fortran code with ifort.
I am just curious about the reason this happens, to avoid any surprises in the future.
I want to create a memmap in MATLAB.
In python I could do this by:
ut = np.memmap('my_array.mmap', dtype=np.float64, mode='w+', shape=(140000,3504))
Then I use it as a normal array, the OS ensured my memory never overflowed. How to do this in MATLAB?
From the docs it seems it wants me to create some array in MATLAB first then write it to a file and read using memmap!
Matlab docs are not clear enough:
Please provide an example of creating an random array of size (140000,15000) and multiply it some other similar matrix.
You have to create an empty file first, then use memmapfile:
size=[140000,3504];
filesize=0;
datatype='float64';
filename='my_array.dat';
fid=fopen(filename,'w+');
max_chunk_size=1000000;
%fills an empty file
while filesize<prod(size)
to_write=min(prod(size)-filesize,max_chunk_size);
filesize=filesize+fwrite(f, zeros(to_write,1), datatype);
end
fclose(fid);
m = memmapfile(filename,'Format','double', 'Writable',true);
I think what you are looking for is the function memmapfile
Example:
m = memmapfile('my_array.dat','Format','double', 'Writable',true)
I am a very beginner in programming and I use Ubuntu.
But now I am trying to perform sound analysis with Python.
In the following code I used wav package to open the wav file and the struct to convert the information:
from wav import *
from struct import *
fp = wave.open(sound.wav, "rb")
total_num_samps = fp.getnframes()
num_fft = (total_num_samps / 512) - 2 #for a fft lenght of 512
for i in range(num_fft):
tempb = fp.readframes(512);
tempb2 = struct.unpack('f', tempb)
print (tempb2)
So in terminal the message that appears is:
struct.error: unpack requires a string argument of length 4
Please, can someone help me to solve this? Someone have a suggestion of other strategy to interpret the sound file?
The format string provided to struct has to tell it exactly the format of the second argument. For example, "there are one hundred and three unsigned shorts". The way you've written it, the format string says "there is exactly one float". But then you provide it a string with way more data than that, and it barfs.
So issue one is that you need to specify the exact number of packed c types in your byte string. In this case, 512 (the number of frames) times the number of channels (likely 2, but your code doesn't take this into account).
The second issue is that your .wav file simply doesn't contain floats. If it's 8-bit, it contains unsigned chars, if it's 16 bit it contains signed shorts, etc. You can check the actual sample width for your .wav by doing fp.getsampwidth().
So then: let's assume you have 512 frames of two-channel 16 bit audio; you would write the call to struct as something like:
channels = fp.getnchannels()
...
tempb = fp.readframes(512);
tempb2 = struct.unpack('{}h'.format(512*channels), tempb)
Using SciPy, you could load the .wav file into a NumPy array using:
import scipy.io.wavfile as wavfile
sample_rate, data = wavfile.read(FILENAME)
NumPy/SciPy will also be useful for computing the FFT.
Tips:
On Ubuntu, you can install NumPy/SciPy with
sudo apt-get install python-scipy
This will install NumPy as well, since NumPy is a dependency of SciPy.
Avoid using * imports such as from struct import *. This copies
names from the struct namespace into the current module's global
namespace. Although it saves you a bit of typing, you pay an awful
price later when the script becomes more complex and you lose
track of where variables are coming from (or worse, the imported variables
mask the value of other variables with the same name).
I want to write some random numbers into an ascii output file.
I generate the numbers with numpy, so the numbers are stored in numpy.array
import numpy as np
random1=np.random.uniform(-1.2,1.2,7e6)
random2=...
random3=...
All three array are of the same size.
I used standard file output, but this is really slow. Just about 8000 lines per 30 min. This may because I loop over three large arrays though.
fout1 = open("output.dat","w")
for i in range(len(random1)):
fout1.write(str(random1[i])+"\t"+ str(random2[i])+"\t"+ str(random3[i])+"\n")
fout1.close()
I also just used print str(random1[i])+"\t"+ str(random2[i])+"\t"+ str(random3[i]) and dumped everything in a file usind shell ./myprog.py > output.dat which seems a bit faster but still I am not satisfied with the output speed.
Any recommendations are really welcome.
Have you tried
random = np.vstack((random1, random2, random3)).T
random.savetxt("output.dat", delimiter="\t")
Im guessing the disk io is the most expensive operation you are doing.. You could try to create your own buffer to deal with this, instead of writing every line every loop buffer up say 100 lines and write them in one big block. Then experiment with this and see what the most benficial buffer size is