I have a large numpy array and I'd like to dump it into a file using ASCII format. I would like to specify the format. This works:
import numpy
a = numpy.random.rand(5)
fmt = "{:.11e}\n"
with open("out.dat", "w") as f:
for item in a:
f.write(fmt.format(item))
but is slow because I manually loop over all entries of a. Is there a way to handle this in only one write operation?
Provided RAM is not an issue, you can try formatting the array to a string and then exporting it:
a_str = np.array2string(a, formatter={'float_kind':lambda x: "%.11f" % x}, separator='\n', threshold=np.inf)[1:-1]
with open("out.dat", "w") as f:
f.write(a_str)
Related
I have 500 arrays named A0 to A499 (All arrays are of different sizes). I want to save these arrays in a text file. Is there any way to get my job done? Also, if possible, I would like to keep their names (A0,A1 etc) so that it is easier to recall later.
I am able to save a single array using np.savetxt but i have no idea how to do it for these 500 arrays.
Thank you very much.
for i in range(500):
exec("A%s=SMtoM(outputS(115,15,0.62))"%(i))
this is how I made my 500 Arrays!
Using pickle :
import pickle
def load(filename):
with open(filename, 'rb') as f:
my_lists = pickle.load(f)
return my_lists
def save(filename, my_lists):
with open(filename, 'wb') as f:
pickle.dump(my_lists)
# Where my_lists = [A0, A1, ..., A499]
Try:
outpt= open("file.txt", "w")
for array in arraylist:
# write each array as a line
outpt.write(arraylist)
outpt.write("\n")
outpt.close()
Pickle works even better as mentioned above.
I'm a python user for scientific computation. Now, I have some numpy arrays, and the size of each of them is huge. Thus, I can not short all of them in the memory at the same time. I want to save the arrays in the disk and read them one by one at each time to do some calculation. How to perform this process pythonicly?
I know if all the data are stored in the memory, I can create a list named array_list like this:
array_list = []
for i0 in range(n_array):
t_ayyay = do_some_calculate()
array_list.append(t_ayyay)
and when I want to use them:
for i0 in range(n_array):
t_ayyay = array_list[i0]
# do something.
How to save array_list in the disk, and I can read each object using the index without load all of them in the memory?
Thanks.
Pickle is your friend for serialization.
import pickle
some_list = [....]
pickle_out = open("some_list.pickle", "w")
pickle.dump(some_list, pickle_out)
pickle_out.close()
to open up your saved array
pickle_in = open("some_list.pickle", "r")
some_list = pickle.open(pickle_in)
How can I import an array to python (numpy.arry) from a file and that way the file must be written if it doesn't already exist.
For example, save out a matrix to a file then load it back.
Checkout the entry on the numpy example list. Here is the entry on .loadtxt()
>>> from numpy import *
>>>
>>> data = loadtxt("myfile.txt") # myfile.txt contains 4 columns of numbers
>>> t,z = data[:,0], data[:,3] # data is 2D numpy array
>>>
>>> t,x,y,z = loadtxt("myfile.txt", unpack=True) # to unpack all columns
>>> t,z = loadtxt("myfile.txt", usecols = (0,3), unpack=True) # to select just a few columns
>>> data = loadtxt("myfile.txt", skiprows = 7) # to skip 7 rows from top of file
>>> data = loadtxt("myfile.txt", comments = '!') # use '!' as comment char instead of '#'
>>> data = loadtxt("myfile.txt", delimiter=';') # use ';' as column separator instead of whitespace
>>> data = loadtxt("myfile.txt", dtype = int) # file contains integers instead of floats
Another option is numpy.genfromtxt, e.g:
import numpy as np
data = np.genfromtxt("myfile.dat",delimiter=",")
This will make data a numpy array with as many rows and columns as are in your file
(I know the question is old, but I think this might be good as a reference for people with similar questions)
If you want to load data from an ASCII/text file (which has the benefit or being more or less human-readable and easy to parse in other software), numpy.loadtxt is probably what you want:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html
If you just want to quickly save and load numpy arrays/matrices to and from a file, take a look at numpy.save and numpy.load:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.save.html
http://docs.scipy.org/doc/numpy/reference/generated/numpy.load.html
In Python, Storing a bare python list as a numpy.array and then saving it out to file, then loading it back, and converting it back to a list takes some conversion tricks. The confusion is because python lists are not at all the same thing as numpy.arrays:
import numpy as np
foods = ['grape', 'cherry', 'mango']
filename = "./outfile.dat.npy"
np.save(filename, np.array(foods))
z = np.load(filename).tolist()
print("z is: " + str(z))
This prints:
z is: ['grape', 'cherry', 'mango']
Which is stored on disk as the filename: outfile.dat.npy
The important methods here are the tolist() and np.array(...) conversion functions.
Have a look at SciPy cookbook. It should give you an idea of some basic methods to import /export data.
If you save/load the files from your own Python programs, you may also want to consider the Pickle module, or cPickle.
I need to write a long list of ints and floats with Python the same way fwrite would do in C - in a binary form.
This is necessary to create input files for another piece of code I am working with.
What is the best way to do this?
You can do this quite simply with the struct module.
For example, to write a list of 32-bit integers in binary:
import struct
ints = [10,50,100,2500,256]
with open('output', 'w') as fh:
data = struct.pack('i' * len(ints), *ints)
fh.write(data)
Will write '\n\x00\x00\x002\x00\x00\x00d\x00\x00\x00\xc4\t\x00\x00\x00\x01\x00\x00'
Have a look at numpy: numpy tofile:
With the array-method 'tofile' you can write binary-data:
# define output-format
numdtype = num.dtype('2f')
# write data
myarray.tofile('filename', numdtype)
Another way is to use memmaps: numpy memmaps
# create memmap
data = num.memmap('filename', mode='w+', dtype=num.float, offset=myoffset, shape=(my_shape), order='C')
# put some data into in:
data[1:10] = num.random.rand(9)
# flush to disk:
data.flush()
del data
How can I import an array to python (numpy.arry) from a file and that way the file must be written if it doesn't already exist.
For example, save out a matrix to a file then load it back.
Checkout the entry on the numpy example list. Here is the entry on .loadtxt()
>>> from numpy import *
>>>
>>> data = loadtxt("myfile.txt") # myfile.txt contains 4 columns of numbers
>>> t,z = data[:,0], data[:,3] # data is 2D numpy array
>>>
>>> t,x,y,z = loadtxt("myfile.txt", unpack=True) # to unpack all columns
>>> t,z = loadtxt("myfile.txt", usecols = (0,3), unpack=True) # to select just a few columns
>>> data = loadtxt("myfile.txt", skiprows = 7) # to skip 7 rows from top of file
>>> data = loadtxt("myfile.txt", comments = '!') # use '!' as comment char instead of '#'
>>> data = loadtxt("myfile.txt", delimiter=';') # use ';' as column separator instead of whitespace
>>> data = loadtxt("myfile.txt", dtype = int) # file contains integers instead of floats
Another option is numpy.genfromtxt, e.g:
import numpy as np
data = np.genfromtxt("myfile.dat",delimiter=",")
This will make data a numpy array with as many rows and columns as are in your file
(I know the question is old, but I think this might be good as a reference for people with similar questions)
If you want to load data from an ASCII/text file (which has the benefit or being more or less human-readable and easy to parse in other software), numpy.loadtxt is probably what you want:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html
If you just want to quickly save and load numpy arrays/matrices to and from a file, take a look at numpy.save and numpy.load:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.save.html
http://docs.scipy.org/doc/numpy/reference/generated/numpy.load.html
In Python, Storing a bare python list as a numpy.array and then saving it out to file, then loading it back, and converting it back to a list takes some conversion tricks. The confusion is because python lists are not at all the same thing as numpy.arrays:
import numpy as np
foods = ['grape', 'cherry', 'mango']
filename = "./outfile.dat.npy"
np.save(filename, np.array(foods))
z = np.load(filename).tolist()
print("z is: " + str(z))
This prints:
z is: ['grape', 'cherry', 'mango']
Which is stored on disk as the filename: outfile.dat.npy
The important methods here are the tolist() and np.array(...) conversion functions.
Have a look at SciPy cookbook. It should give you an idea of some basic methods to import /export data.
If you save/load the files from your own Python programs, you may also want to consider the Pickle module, or cPickle.