I defined a function(procedure) to read a file. I want it returns arrays with data I want to read from the file, as it follows:
import csv
import numpy as np
import matplotlib.pyplot as plt
# Subroutine to read the day, Ta,Tp from a file and convert them into arrays
def readstr(fname,day,Ta,Tp):
van = open(fname,'r')
van_csv = van.readlines()[7:] # Skip seven lines
van.close() # close the file
van_csv = csv.reader(van_csv) # now the file is separated by colunms
for row in van_csv: # Passing the values of the each column to arrays
day.append(row[1])
Ta.append(row[8])
Tp.append(row[7])
day = np.array(day,dtype=np.integer)
Ta = np.array(Ta,dtype=np.float)
Tp = np.array(Tp,dtype=np.float)
van = "file"
# Defining the lists
dayVan = []
Tav = []
Tpv = []
readstr(van,dayVan,Tav,Tpv)
print Tav
I thought it would work, but dayVan, Tpv, Tav keep being lists.
The line
Ta = np.array(Ta,dtype=np.float)
Creates a new array object from the contents of the list Ta, it then assigns this array to the local identifier Ta. It does not change the global that references the list.
Python doesn't have "variables". It has identifiers. When doing a = b you simply say "bind the name a to the object bound to b". The a is simply a label that can be used to retrieve an object. If you then do a = 0 you are re-binding the label a but this does not affect the object bound to b. The identifiers are not memory locations.
To pass the resulting arrays out of the function you can:
Return them and re-assign the global Ta.
Assign directly to the global variable. However in order to do this the local Ta should be given a new name and you'd have to use the global statement(Note: avoid this solution.)
The transformation is correctly done, but only inside your function.
Try to return day, Ta and Tp at the end of your function, and get them from the caller, it will work better.
def readstr(fname):
van = open(fname,'r')
van_csv = van.readlines()[7:] # Skip seven lines
van.close() # close the file
van_csv = csv.reader(van_csv) # now the file is separated by colunms
day, Ta, Tp = [], [], []
for row in van_csv: # Passing the values of the each column to arrays
day.append(row[1])
Ta.append(row[8])
Tp.append(row[7])
day = np.array(day,dtype=np.integer)
Ta = np.array(Ta,dtype=np.float)
Tp = np.array(Tp,dtype=np.float)
return day, Ta, Tp
dayVan, Tav, Tpv = readstr(van)
Perhaps you can simply do:
dayVan, Tpv, Tav = np.loadtxt(fname, usecols=(1,7,8), skiprows=7, delimiter=',', unpack=True)
Related
This MATLAB code is from Main_MOHHO.m from https://www.mathworks.com/matlabcentral/fileexchange/80776-multi-objective-harris-hawks-optimization-mohho. I want to make the same code using python, but I can't make the Rabbits variabel.
clc;
clear;
close all;
%% Problem Definition
nVar=3; % Number of Decision Variables
VarSize=[1 nVar]; % Size of Decision Variables Matrix
VarMin=0; % Lower Bound of Variables
VarMax=1; % Upper Bound of Variables
nPop=5; % Population Size
%% Initialization
empty_Rabbit.Location=[];
empty_Rabbit.Cost=[];
empty_Rabbit.Sol=[];
empty_Rabbit.IsDominated=[];
empty_Rabbit.GridIndex=[];
empty_Rabbit.GridSubIndex=[];
Rabbits=repmat(empty_Rabbit,nPop,1);
for i=1:nPop
Rabbits(i).Location = rand(VarSize).*(VarMax-VarMin)+VarMin;
X(i,:) = rand(VarSize).*(VarMax-VarMin)+VarMin;
end
I try to make it on google colab like this.
import numpy as np
nVar = 3 # Number of Decision Variables
VarSize = np.array((1, nVar)) # Size of Decision Variables Matrix
VarMin = 0 # Lower Bound of Variables
VarMax = 1 # Upper Bound of Variables
nPop = 5 # Population Size
class empty_Rabbit:
Location = []
Cost = []
IsDominated = []
GridIndex = []
GridSubIndex = []
Sol = []
Rabbits = np.tile(empty_Rabbit, (nPop, 1))
X = np.zeros((nPop, nVar))
Rabbit_Location = np.zeros((VarSize))
Rabbit_Energy = math.inf
for i in range(nPop):
Rabbits[i, 0].Location = np.multiply(np.random.rand(VarSize[0], VarSize[1]),
(VarMax-VarMin) + VarMin)
print(Rabbits[i,0].Location)
But, the Rabbits_Location same for each row.
Output Google Colab
What is the correct way to create Rabbits variable in python so the output like the output with number 1 in the pic? Thank you.
Two issues exist in your code. First, np.tile repeats the same object (nPop, 1) times. So, when you change one of the objects, you actually change the same memory location. Second, you want to initialize a different object each time instead of referring to the same object, so you want to write empty_Rabbit() to create a new instance of that object. Both suggestions can be achieved using a comprehension like [empty_Rabbit() for i in range(nPop)] and reshape to any new dimensions if required.
import numpy as np
nVar = 3 # Number of Decision Variables
VarSize = np.array((1, nVar)) # Size of Decision Variables Matrix
VarMin = 0 # Lower Bound of Variables
VarMax = 1 # Upper Bound of Variables
nPop = 5 # Population Size
class empty_Rabbit:
Location = []
Cost = []
IsDominated = []
GridIndex = []
GridSubIndex = []
Sol = []
Rabbits = np.array([empty_Rabbit() for i in range(nPop)]).reshape(nPop,1)
X = np.zeros((nPop, nVar))
Rabbit_Location = np.zeros((VarSize))
Rabbit_Energy = np.inf
for i in range(nPop):
Rabbits[i, 0].Location = np.multiply(np.random.rand(VarSize[0], VarSize[1]),
(VarMax-VarMin) + VarMin)
print(Rabbits[i,0].Location)
for i in range(nPop):
print(Rabbits[i,0].Location)
Now, the output of both print statements will be identical with distinct rows:
[[0.5392264 0.39375339 0.59483626]]
[[0.53959355 0.91049574 0.58115175]]
[[0.46152304 0.43111977 0.06882631]]
[[0.13693784 0.82075653 0.49488394]]
[[0.06901317 0.34133836 0.91453956]]
[[0.5392264 0.39375339 0.59483626]]
[[0.53959355 0.91049574 0.58115175]]
[[0.46152304 0.43111977 0.06882631]]
[[0.13693784 0.82075653 0.49488394]]
[[0.06901317 0.34133836 0.91453956]]
scipy.io.loadmat uses structured arrays when loading struct from MATLAB .mat files. But I think that's too advanced for you.
I think you need to create a set of numpy arrays, rather than try for some sort of class or more complicated structure.
empty_Rabbit.Location=[];
empty_Rabbit.Cost=[];
empty_Rabbit.Sol=[];
empty_Rabbit.IsDominated=[];
empty_Rabbit.GridIndex=[];
empty_Rabbit.GridSubIndex=[];
becomes instead
location = np.zeros(nPop)
cost = np.zeros(nPop)
sol = np.zeros(nPop)
isDominated = np.zeros(nPop) # or bool dtype?
gridIndex = np.zeros(nPop)
gridSubIndex = np.zeros(nPop)
np.zeros makes a float array; for some of those you might want np.zeros(nPop, dtype=int) (if used as index).
rabbit= np.zeros(nPop, dtype=[('location',float), ('cost',float),('sol',float), ....])
could be used to make structured array, but you'll need to read more about those.
MATLAB lets you use iteration freely as in
for i=1:nPop
Rabbits(i).Location = rand(VarSize).*(VarMax-VarMin)+VarMin;
X(i,:) = rand(VarSize).*(VarMax-VarMin)+VarMin;
end
but that's slow (as it used to be MATLAB before jit compilation). It's better to use whole array calculations
location = np.random.rand(nPop,VarSize) * (VarMax-VarMin)+VarMin
will make a (nPop,VarSize) 2d array, not the 1d that np.zeros(nPop) created.
Looks like X could be created in the same way (without iteration).
I have a function that returns 4 values after doing some calculations. I give as input 5 parameters.
I run the above function 6 times using 6 different input parameters to obtain 6 different outputs.
def id_match(zcosmo,zphot,zmin,zmax,mlim):
data_zcosmo_lastz = zcosmo[(data_m200>mlim)*(zcosmo>zmin)*(zcosmo<zmax)]
data_zphot_lastz = zphot[(data_m200>mlim)*(zphot>zmin)*(zphot<zmax)]
halo_id_zcosmo = data_halo_id[(data_m200>mlim)*(zcosmo>zmin)*(zcosmo<zmax)]
halo_id_zphot = data_halo_id[(data_m200>mlim)*(zphot>zmin)*(zphot<zmax)]
idrep_zcosmo = data_idrep[(data_m200>mlim)*(zcosmo>zmin)*(zcosmo<zmax)]
idrep_zphot = data_idrep[(data_m200>mlim)*(zphot>zmin)*(zphot<zmax)]
file2freq1 = Counter(zip(halo_id_zcosmo,idrep_zcosmo))
file2freq2 = Counter(zip(halo_id_zphot,idrep_zphot))
set_a = len(set(file2freq1) & set(file2freq2)) # this has the number of common objects
difference = 100.0 - (set_a*100.0)/len(data_zcosmo_lastz)
print difference
return (len(data_zcosmo_lastz),len(data_zphot_lastz),set_a,difference)
zmin_limits = [0.1,0.4,0.7,1.0,1.3,1.6]
zmax_limits = [0.4,0.7,1.0,1.3,1.6,2.1]
mlim_limits = [5e13,5e13,5e13,5e13,5e13,5e13]
for a,b,c in zip(zmin_limits,zmax_limits,mlim_limits):
id_match(data_zcosmo_lastz,data_zphot_lastz,a,b,c)
The above code prints the difference for each of the 6 different input parameters.
But I would like to know how I can save the output from the function into an array so that I can save it as a csv file???
I know that by doing
a,b,c,d = id_match(input params)
will give a,b,c,d to have one of the outputs of id_match. But I want to store all the return values inside a single array.
id_match() already returns a tuple. You don't need to convert it to anything because csv.DictWriter.writerow() can handle a tuple. All you need to do is assign a variable to what id_match() returns and write to a csv file:
with open(myfilename, 'w') as csvfile:
writer = csv.DictWriter(csvfile)
for a,b,c in zip(zmin_limits,zmax_limits,mlim_limits):
info = id_match(data_zcosmo_lastz,data_zphot_lastz,a,b,c)
writer.writerow(info)
I'm trying to load a large number of files saved in the Ensight gold format into a numpy array. In order to conduct this read I've written my own class libvec which reads the geometry file and then preallocates the arrays which python will use to save the data as shown in the code below.
N = len(file_list)
# Create the class object and read geometry file
gvec = vec.libvec(os.path.join(current_dir,casefile))
x,y,z = gvec.xyz()
# Preallocate arrays
U_temp = np.zeros((len(y),len(x),N),dtype=np.dtype('f4'))
V_temp = np.zeros((len(y),len(x),N),dtype=np.dtype('f4'))
u_temp = np.zeros((len(x),len(x),N),dtype=np.dtype('f4'))
v_temp = np.zeros((len(x),len(y),N),dtype=np.dtype('f4'))
# Read the individual files into the previously allocated arrays
for idx,current_file in enumerate(file_list):
U,V =gvec.readvec(os.path.join(current_dir,current_file))
U_temp[:,:,idx] = U
V_temp[:,:,idx] = V
del U,V
However this takes seemingly forever so I was wondering if you have any idea how to speed up this process? The code reading the individual files into the array structure can be seen below:
def readvec(self,filename):
# we are supposing for the moment that the naming scheme PIV__vxy.case PIV__vxy.geo not changes should that
# not be the case appropriate changes have to be made to the corresponding file
data_temp = np.loadtxt(filename, dtype=np.dtype('f4'), delimiter=None, converters=None, skiprows=4)
# U value
for i in range(len(self.__y)):
# x value counter
for j in range(len(self.__x)):
# y value counter
self.__U[i,j]=data_temp[i*len(self.__x)+j]
# V value
for i in range(len(self.__y)):
# x value counter
for j in range(len(self.__x)):
# y value counter
self.__V[i,j]=data_temp[len(self.__x)*len(self.__y)+i*len(self.__x)+j]
# W value
if len(self.__z)>1:
for i in range(len(self.__y)):
# x value counter
for j in range(len(self.__xd)):
# y value counter
self.__W[i,j]=data_temp[2*len(self.__x)*len(self.__y)+i*len(self.__x)+j]
return self.__U,self.__V,self.__W
else:
return self.__U,self.__V
Thanks a lot in advance and best regards,
J
It'a bit hard to say without any test input\output to compare against. But i think this would give you the same U\V arrays as your nested for loops in readvec. This method should be considerably faster then the for loops.
U = data[:size_x*size_y].reshape(size_x, size_y)
V = data[size_x*size_y:].reshape(size_x, size_y)
Returning these directly into U_temp and V_temp should also help. Right now you're doing 3(?) copies of your data to get them into U_temp and V_temp
From file to temp_data
From temp_data to self.__U\V
From U\V into U\V_temp
Although my guess is that the two nested for loop, and accessing one element at a time is causing the slowness
I am running the following: output.to_csv("hi.csv") where output is a pandas dataframe.
My variables all have values but when I run this in iPython, no file is created. What should I do?
Better give the complete path for your output csv file. May be that you are checking in a wrong folder.
You have to make sure that your 'to_csv' method of 'output' object has a write-file function implemented.
And there is a lib for csv manipulation in python, so you dont need to handle all the work:
https://docs.python.org/2/library/csv.html
I'm not sure if this will be useful to you, but I write to CSV files frequenly in python. Here is an example generating random vectors (X, V, Z) values and writing them to a CSV, using the CSV module. (The paths are os paths are for OSX but you should get the idea even on a different os.
Working Writing Python to CSV example
import os, csv, random
# Generates random vectors and writes them to a CSV file
WriteFile = True # Write CSV file if true - useful for testing
CSVFileName = "DataOutput.csv"
CSVfile = open(os.path.join('/Users/Si/Desktop/', CSVFileName), 'w')
def genlist():
# Generates a list of random vectors
global v, ListLength
ListLength = 25 #Amount of vectors to be produced
Max = 100 #Maximum range value
x = [] #Empty x vector list
y = [] #Empty y vector list
z = [] #Empty x vector list
v = [] #Empty xyz vector list
for i in xrange (ListLength):
rnd = random.randrange(0,(Max)) #Generate random number
x.append(rnd) #Add it to x list
for i in xrange (ListLength):
rnd = random.randrange(0,(Max))
y.append(rnd) #Add it to y list
for i in xrange (ListLength):
rnd = random.randrange(0,(Max)) #Generate random number
z.append(rnd) #Add it to z list
for i in xrange (ListLength):
merge = x[i], y[i],z[i] # Merge x[i], y[i], x[i]
v.append(merge) #Add merged list into v list
def writeCSV():
# Write Vectors to CSV file
wr = csv.writer(CSVfile, quoting = csv.QUOTE_MINIMAL, dialect='excel')
wr.writerow(('Point Number', 'X Vector', 'Y Vector', 'Z Vector'))
for i in xrange (ListLength):
wr.writerow((i+1, v[i][0], v[i][1], v[i][2]))
print "Data written to", CSVfile
genlist()
if WriteFile is True:
writeCSV()
Hopefully there is something useful in here for you!
I am just starting out with Python. I have some fortran and some Matlab skills, but I am by no means a coder. I need to post-process some output files.
I can't figure out how to read each value into the respective variable. The data looks something like this:
h5097600N1 2348.13 2348.35 -0.2219 20.0 -4.438
h5443200N1 2348.12 2348.36 -0.2326 20.0 -4.651
h8467200N2 2348.11 2348.39 -0.2813 20.0 -5.627
...
In my limited Matlab notation, I would like to assign the following variables of the form tN1(i,j) something like this:
tN1(1,1)=5097600; tN1(1,2)=5443200; tN2(1,3)=8467200; #time between 'h' and 'N#'
hmN1(1,1)=2348.13; hmN1(1,2)=2348.12; hmN2(1,3)=2348.11; #value in 2nd column
hsN1(1,1)=2348.35; hsN1(1,2)=2348.36; hsN2(1,3)=2348.39; #value in 3rd column
I will have about 30 sets, or tN1(1:30,1:j); hmN1(1:30,1:j);hsN1(1:30,1:j)
I know it may not seem like it, but I have been trying to figure this out for 2 days now. I am trying to learn this on my own and it seems I am missing something fundamental in my understanding of python.
I wrote a simple script which does what you asks. It creates three dictionaries, t, hm and hs. These will have keys as the N values.
import csv
import re
path = 'vector_data.txt'
# Using the <with func as obj> syntax handles the closing of the file for you.
with open(path) as in_file:
# Use the csv package to read csv files
csv_reader = csv.reader(in_file, delimiter=' ')
# Create empty dictionaries to store the values
t = dict()
hm = dict()
hs = dict()
# Iterate over all rows
for row in csv_reader:
# Get the <n> and <t_i> values by using regular expressions, only
# save the integer part (hence [1:] and [1:-1])
n = int(re.findall('N[0-9]+', row[0])[0][1:])
t_i = int(re.findall('h.+N', row[0])[0][1:-1])
# Cast the other values to float
hm_i = float(row[1])
hs_i = float(row[2])
# Try to append the values to an existing list in the dictionaries.
# If that fails, new lists is added to the dictionaries.
try:
t[n].append(t_i)
hm[n].append(hm_i)
hs[n].append(hs_i)
except KeyError:
t[n] = [t_i]
hm[n] = [hm_i]
hs[n] = [hs_i]
Output:
>> t
{1: [5097600, 5443200], 2: [8467200]}
>> hm
{1: [2348.13, 2348.12], 2: [2348.11]}
>> hn
{1: [2348.35, 2348.36], 2: [2348.39]}
(remember that Python uses zero-indexing)
Thanks for all your comments. Suggested readings led to other things which helped. Here is what I came up with:
if len(line) >= 45:
if line[0:45] == " FIT OF SIMULATED EQUIVALENTS TO OBSERVATIONS": #! indicates data to follow, after 4 lines of junk text
for i in range (0,4):
junk = file.readline()
for i in range (0,int(nobs)):
line = file.readline()
sline = line.split()
obsname.append(sline[0])
hm.append(sline[1])
hs.append(sline[2])