I have yearly data files in different folders. each file contains daily data ranging from Jan 1 to Dec 31. Data files name is looks like AS060419.67 where last four digit represent year i.e. 1967 and 0604 is folder name.
I tried to read these multiple files by using the code (below), but it reads only for last year data in last folder
def date_parser(doy, year):
return dt.datetime.strptime(doy.zfill(3)+year, '%j%Y')
files = glob.glob('????/AS*')
files.sort()
files
STNS = {}
for f in files:
stn_id, info = f.split('/')
year = "".join(info[-5:].split('.'))
#print (f,stn_id)
with open(f) as fo:
data = fo.readlines()[:-1]
data = [d.strip() for d in data]
data = '\n'.join(data)
with open('data.dump', 'w') as dump:
dump.write(data)
parser = lambda date: date_parser(date, year=year)
df = pd.read_table('data.dump', delim_whitespace=True,names=['date','prec'],
na_values='DNA', parse_dates=[0], date_parser=parser, index_col='date' )
df.replace({'T': 0})
df = df.apply(pd.to_numeric, args=('coerce',))
df.name = stn_name
df.sid = stn_id
if stn_id not in STNS.keys():
STNS[stn_name] = df
else:
STNS[stn_id] = STNS[stn_id].append(df)
STNS[stn_id].name = df.name
STNS[stn_id].sid = df.sid
#outfile.write(line)
For making plot
for stn in STNS:
STNS[stn_id].plot()
plt.title('Precipitation for {0}'.format(STNS[stn].name))
The problem is it reads only last year data in last folder. Can anyone help to figure out this problem.Your help will be highly appreciated.
You can do it like this:
import os
import glob
import pandas as pd
import matplotlib.pyplot as plt
# file mask
fmask = r'./data/????/AS*.??'
# all RegEx replacements
replacements = {
r'T': 0
}
# list of data files
flist = glob.glob(fmask)
def read_data(flist, date_col='date', **kwargs):
dfs = []
for f in flist:
# parse year from the file name
y = os.path.basename(f).replace('.', '')[-4:]
df = pd.read_table(f, **kwargs)
# replace day of year with a date
df[date_col] = pd.to_datetime(y + df[date_col].astype(str).str.zfill(3), format='%Y%j')
dfs.append(df)
return pd.concat(dfs, ignore_index=True)
df = read_data(flist,
date_col='date',
sep=r'\s+',
header=None,
names=['date','prec'],
engine='python',
skipfooter=1,
) \
.replace(replacements, regex=True) \
.set_index('date') \
.apply(pd.to_numeric, args=('coerce',))
df.plot()
plt.show()
I've downloaded only four files, so the corresponding data you can see on the plot...
You overwrite the same file over and over again.
Derive your target file name from your source file name.
Or use the append mode if you want it all in the same file.
How do you append to a file?
Related
I want to build a pandas dataframe with columns = ["periods","opens","highs","lows","closes","volumes","consolidating", "5D Perf"] by looping through all excel files in a folder.
Periods ... volumes data values are extracted from files containing "features.xlsx" in the filename and "consolidating", "5D Perf" are extracted from files containing "labels.csv" in the filename but I am not sure how to construct the loop. I am using the code below but this way I am getting a broken df.
base_df = pd.DataFrame([[1,2,3,4,5,6,7,8]], columns = ["periods","opens","highs","lows","closes","volumes","consolidating", "5D Perf"])
for filename in os.listdir(dir):
if filename.endswith(".xlsx"):
df = pd.read_excel(filename)
dimensions = df.shape
period = dimensions[0]-1
open = df['Open'].tolist()
high = df['High'].tolist()
low = df['Low'].tolist()
close = df['Close'].tolist()
volume = df['Volume'].tolist()
base_df[['consolidating', "consolidating", ]] = df[['consolidating', '5D Perf']]
#to_append = [period, open, high, low, close, volume, consolidating, fivedperf]
elif filename.endswith(".csv"):
df = pd.read_csv(filename)
consolidating = df['Consolidating'].iloc[0]
fivedperf = df['5D Performance'].iloc[1]
base_df[['consolidating', "consolidating", ]] = df[['consolidating', '5D Perf']]
If I had show graphically what I want to achieve it would be something like this :
How can I fix this so that part of the new row is taken from one file and the other part from another file ?
Here are also some sample data files : https://wetransfer.com/downloads/f37f68146274beba41d3fd36fec5bd5120220603075831/55edb1
You can just do the following:
base_df = pd.DataFrame(columns = ["Period","Time","Date","Label1","Label2","Label3"])
for filename in os.listdir(dir):
if filename.endswith(".xlsx"):
df = pd.read_excel(filename)
base_df[['Period', 'Time', 'Date']] = df[['Period', 'Time', 'Date']]
elif filename.endswith(".csv"):
df = pd.read_csv(filename)
base_df[['Label1', 'Label2', 'Label3']] = df[['Label1', 'Label2', 'Label3']]
I am trying to add data from several files in a folder to a data frame. Each .csv file has varying lengths but has the same number of columns. I am trying to add all of them to one data frame with ignoring the index so that the new data frame is just vertically combined. For some reason every time I try to concatenate the data I am left with ~ 363 columns when there should only be 9. Each csv file has the same number of columns so I am confused.
import os
import pandas as pd
import glob
cwd = os.getcwd()
folder = cwd +'\\downloads\\prepared_csv_files\\prepared_csv_files\\'
all_files = glob.glob(folder + "/*.csv")
li = []
for filename in all_files:
df = pd.read_csv(filename, index_col=None, header=0)
li.append(df)
frame = pd.concat(li, axis=0, ignore_index=True)
I have also tried
final_df = pd.DataFrame(li, columns = ['tool','pressure'])
# and I name all columns not doing it now
here final is the name of the final dataset.
I am assuming tool and pressure are the columns name in your all .csv files
final = pd.DataFrame(columns = ['tool','pressure'])
for filename in all_files:
df = pd.read_csv(filename)
df = pd.DataFrame(df)
final = pd.concat([final,df],ignore_index= True,join="inner")
I have a code that merges all txt files from a directory into a dataframe
follow the code below
import pandas as pd
import os
import glob
diretorio = "F:\PROJETOS\LOTE45\ARQUIVOS\RISK\RISK_CUSTOM_FUND_N1"
files = []
files = [pd.read_csv(file, delimiter='\t')
for file in glob.glob(os.path.join(diretorio ,"*.txt"))]
df = pd.concat(files, ignore_index=True)
df
that gives result to this table
I needed to add a date column to this table, but I only have the date available at the end of the filename.
How can I get the date at the end of the filename and put it inside the dataframe.
I have no idea how to do this
Assuming the file naming pattern is constant, you can parse the end of the filename for every iteration of the loop this way :-
from datetime import datetime
files = []
for file in glob.glob(os.path.join(diretorio ,"*.txt")):
df_f = pd.read_csv(file, delimiter='\t')
df_f['date'] = datetime.strptime(file[-11:-4], "%d%m%Y")
files.append(df_f)
df = pd.concat(files, ignore_index=True)
import pandas as pd
import os
diretorio = "F:/PROJETOS/LOTE45/ARQUIVOS/RISK/RISK_CUSTOM_FUND_N1/"
files = []
for filename in os.listdir(diretorio):
if filename.endswith(".csv"):
df = pd.read_csv(diretorio + filename, sep=";")
df['Date'] = filename.split('.')[0].split("_")[-1]
files.append(df)
df = pd.concat(files, ignore_index=True)
print(df)
I have the following code:
import glob
import pandas as pd
import os
import csv
myList = []
path = "/home/reallymemorable/Documents/git/COVID-19/csse_covid_19_data/csse_covid_19_daily_reports_us/*.csv"
for fname in glob.glob(path):
df = pd.read_csv(fname)
row = df.loc[df['Province_State'] == 'Pennsylvania']
dateFromFilename = os.path.basename(fname).replace('.csv','')
fileDate = pd.DataFrame({'Date': [dateFromFilename]})
myList.append(row.join(fileDate))
concatList = pd.concat(myList, sort=True)
print(concatList)
concatList.to_csv('/home/reallymemorable/Documents/test.csv', index=False, header=True
It goes through a folder of CSVs and grabs a specific row and puts it all in a CSV. The files themselves have names like 10-10-2020.csv. I have some code in there that gets the filename and removes the file extension, so I am left with the date alone.
I am trying to add another column called "Date" that contains the filename for each file.
The script almost works: it gives me a CSV of all the rows I pulled out of the various CSVs, but the Date column itself is empty.
If I do print(dateFromFilename), the date/filename prints as expected (e.g. 10-10-2020).
What am I doing wrong?
I believe join has how=left by default. And your fileDate dataframe has different index than row, so you wouldn't get the date. Instead, do an assignment:
for fname in glob.glob(path):
df = pd.read_csv(fname)
row = df.loc[df['Province_State'] == 'Pennsylvania']
dateFromFilename = os.path.basename(fname).replace('.csv','')
myList.append(row.assign(Date=dateFromFilename))
concatList = pd.concat(myList, sort=True)
Another way is to store the dataframes as a dictionary, then concat:
myList = dict()
for fname in glob.glob(path):
df = pd.read_csv(fname)
row = df.loc[df['Province_State'] == 'Pennsylvania']
dateFromFilename = os.path.basename(fname).replace('.csv','')
myList[dateFromFilename] = row
concatList = pd.concat(myList, sort=True)
I have the same file with quarterly data saved in different folders corresponding to the quarter. In other words, a quarter 1 folder, quarter 2 , quarter 3, quarter 4. This is the only difference in the file path. I am looking to read all four files in and concatenate them into one database. I can do this manually using a version of the simplified code below and changing the period each time.
period = ‘Q1’
filepath = ‘filepath/’ + period
file = filepath + ‘/file.xls’
df = pd.read_excel(file)
I would like to automate it with some form of for loop (I assume). That loops through the 4 periods, reads the file into a database and then concatenates. I have read other answers as to how this can be done with files in the same folder. But am struggling to do it where the file path changes. Manually putting the files into the same folder is not a desirable solution.
I tried making period a tuple and a list containing all 4 periods then a simple for loop but this didn’t work. I got the following error message.
TypeError: Can't convert 'list' object to str implicitly
Greatly appreciate any advice.
How about you first use list comprehension to get a list of all files:
periods= ["Q1", "Q2", "Q3", "Q4"]
files = ["filepath/"+ p + "/file.xls" for p in periods]
and then load them all into a list of data frames with
dfs = []
for f in files:
df = pd.read_excel(f)
dfs.append(df)
You can use these loops to create full file paths and to iterate over them to create one DataFrame containing all the files.
filepath = 'path/'
file = 'file.xlsx'
periods=['Q1','Q2','Q3','Q4']
files = []
for p in periods:
files.append(filepath+p+'/'+file)
files
data = []
for f in files:
data.append(pd.read_excel(f))
df = pd.concat(data)
You probably want something like this:
periods = ['Q1', 'Q2', 'Q3', 'Q4']
df = None
for period in periods:
filepath = 'filepath/' + period
file = filepath + '/file.xls'
if df is None:
df = pd.read_excel(file)
else:
df.append(pd.read_excel(file))
You could try something like this:
complete_df = pd.DataFrame()
for i in range(1,5):
quarter = 'Q'+str(i)
filepath = 'filepath/' + quarter
file = filepath + '/file.xls'
df = pd.read_excel(file)
complete_df = complete_df.append(df)