I have a code that merges all txt files from a directory into a dataframe
follow the code below
import pandas as pd
import os
import glob
diretorio = "F:\PROJETOS\LOTE45\ARQUIVOS\RISK\RISK_CUSTOM_FUND_N1"
files = []
files = [pd.read_csv(file, delimiter='\t')
for file in glob.glob(os.path.join(diretorio ,"*.txt"))]
df = pd.concat(files, ignore_index=True)
df
that gives result to this table
I needed to add a date column to this table, but I only have the date available at the end of the filename.
How can I get the date at the end of the filename and put it inside the dataframe.
I have no idea how to do this
Assuming the file naming pattern is constant, you can parse the end of the filename for every iteration of the loop this way :-
from datetime import datetime
files = []
for file in glob.glob(os.path.join(diretorio ,"*.txt")):
df_f = pd.read_csv(file, delimiter='\t')
df_f['date'] = datetime.strptime(file[-11:-4], "%d%m%Y")
files.append(df_f)
df = pd.concat(files, ignore_index=True)
import pandas as pd
import os
diretorio = "F:/PROJETOS/LOTE45/ARQUIVOS/RISK/RISK_CUSTOM_FUND_N1/"
files = []
for filename in os.listdir(diretorio):
if filename.endswith(".csv"):
df = pd.read_csv(diretorio + filename, sep=";")
df['Date'] = filename.split('.')[0].split("_")[-1]
files.append(df)
df = pd.concat(files, ignore_index=True)
print(df)
Related
I would like to read just csv last 7 days createds csv files from a directory into pandas and concatenate them into one big DataFrame. I have not been able to figure it out though. Here is what I have so far:
Edit: I'm trying to filter by the creation date of csv file, not by any column in csv.
from datetime import datetime, timedelta
import pandas as pd
import glob
fileday = datetime.now() - timedelta(7)
fileday = datetime.strftime(fileday, '%Y%m%d')
path = r'C:\DRO\DCL_rawdata_files' # use your path
all_files = glob.glob(path + "/*.csv")
li = []
for filename in all_files:
df = pd.read_csv(filename, index_col=None, header=0)
li.append(df)
frame = pd.concat(li, axis=0, ignore_index=True)
Since you're using pandas, let's use a combination of pathlib and pandas.
from pathlib import Path
import pandas as pd
p = Path(r'C:\DRO\DCL_rawdata_files')
all_files = p.glob('*.csv')
df = pd.DataFrame({'files' : all_files})
df['date'] = pd.to_datetime(df['files'].apply(lambda x : x.stat().st_mtime),unit='s')
# filter your files.
trg_files = df[df['date'] >= pd.Timestamp('now') - pd.DateOffset(days=7)]['files'].tolist()
dfs = [pd.read_csv(f) for f in trg_files]
You could do something like this.
df = pd.DataFrame()
for filename in all_files:
df = df.append(pd.read_csv(filename))
I have the following code:
import glob
import pandas as pd
import os
import csv
myList = []
path = "/home/reallymemorable/Documents/git/COVID-19/csse_covid_19_data/csse_covid_19_daily_reports_us/*.csv"
for fname in glob.glob(path):
df = pd.read_csv(fname)
row = df.loc[df['Province_State'] == 'Pennsylvania']
dateFromFilename = os.path.basename(fname).replace('.csv','')
fileDate = pd.DataFrame({'Date': [dateFromFilename]})
myList.append(row.join(fileDate))
concatList = pd.concat(myList, sort=True)
print(concatList)
concatList.to_csv('/home/reallymemorable/Documents/test.csv', index=False, header=True
It goes through a folder of CSVs and grabs a specific row and puts it all in a CSV. The files themselves have names like 10-10-2020.csv. I have some code in there that gets the filename and removes the file extension, so I am left with the date alone.
I am trying to add another column called "Date" that contains the filename for each file.
The script almost works: it gives me a CSV of all the rows I pulled out of the various CSVs, but the Date column itself is empty.
If I do print(dateFromFilename), the date/filename prints as expected (e.g. 10-10-2020).
What am I doing wrong?
I believe join has how=left by default. And your fileDate dataframe has different index than row, so you wouldn't get the date. Instead, do an assignment:
for fname in glob.glob(path):
df = pd.read_csv(fname)
row = df.loc[df['Province_State'] == 'Pennsylvania']
dateFromFilename = os.path.basename(fname).replace('.csv','')
myList.append(row.assign(Date=dateFromFilename))
concatList = pd.concat(myList, sort=True)
Another way is to store the dataframes as a dictionary, then concat:
myList = dict()
for fname in glob.glob(path):
df = pd.read_csv(fname)
row = df.loc[df['Province_State'] == 'Pennsylvania']
dateFromFilename = os.path.basename(fname).replace('.csv','')
myList[dateFromFilename] = row
concatList = pd.concat(myList, sort=True)
I'm trying to contact all excel files and worksheets in them into one using the below script. It kinda works but then the excel file c.xlsx is overwritten per file, so only the last excel file is concated not sure why?
import pandas as pd
import os
import ntpath
import glob
dir_path = os.path.dirname(os.path.realpath(__file__))
os.chdir(dir_path)
cdf = None
for excel_names in glob.glob('*.xlsx'):
print(excel_names)
df = pd.read_excel(excel_names, sheet_name=None, ignore_index=True)
cdf = pd.concat(df.values())
cdf.to_excel("c.xlsx", header=False, index=False)
Idea is create list of DataFrames in list comprehension, but because working with orderdict is necessary concat in loop and then again concat for one big final DataFrame:
cdf = [pd.read_excel(excel_names, sheet_name=None, ignore_index=True).values()
for excel_names in glob.glob('files/*.xlsx')]
df = pd.concat([pd.concat(x) for x in cdf], ignore_index=True)
#print (df)
df.to_excel("c.xlsx", index=False)
I just tested the code below. It merges data from all Excel files in a folder into one, single, Excel file.
import pandas as pd
import numpy as np
import glob
glob.glob("C:\\your_path\\*.xlsx")
all_data = pd.DataFrame()
for f in glob.glob("C:\\your_path\\*.xlsx"):
df = pd.read_excel(f)
all_data = all_data.append(df,ignore_index=True)
print(all_data)
df = pd.DataFrame(all_data)
df.shape
df.to_excel("C:\\your_path\\final.xlsx", sheet_name='Sheet1')
I got it working using the below script which uses #ryguy72's answer but works on all worksheets as well as the header row.
import pandas as pd
import numpy as np
import glob
all_data = pd.DataFrame()
for f in glob.glob("my_path/*.xlsx"):
df = pd.read_excel(f, sheet_name=None, ignore_index=True)
cdf = pd.concat(df.values())
all_data = all_data.append(cdf,ignore_index=True)
print(all_data)
df = pd.DataFrame(all_data)
df.shape
df.to_excel("my_path/final.xlsx", sheet_name='Sheet1')
I have multiple csv files in the same folder with all the same data columns,
20100104 080100;5369;5378.5;5365;5378;2368
20100104 080200;5378;5385;5377;5384.5;652
20100104 080300;5384.5;5391.5;5383;5390;457
20100104 080400;5390.5;5391;5387;5389.5;392
I want to merge the csv files into pandas and add a column with the file name to each line so I can track where it came from later. There seems to be similar threads but I haven't been able to adapt any of the solutions. This is what I have so far. The merge data into one data frame works but I'm stuck on the adding file name column,
import os
import glob
import pandas as pd
path = r'/filepath/'
all_files = glob.glob(os.path.join(path, "*.csv"))
names = [os.path.basename(x) for x in glob.glob(path+'\*.csv')]
list_ = []
for file_ in all_files:
list_.append(pd.read_csv(file_,sep=';', parse_dates=[0], infer_datetime_format=True,header=None ))
df = pd.concat(list_)
Instead of using a list just use DataFrame's append.
df = pd.DataFrame()
for file_ in all_files:
file_df = pd.read_csv(file_,sep=';', parse_dates=[0], infer_datetime_format=True,header=None )
file_df['file_name'] = file_
df = df.append(file_df)
I'm trying to read a bunch of CSV-files into a single pandas dataframe. Some of the CSVs have data for multiple dates. I want only the data from each CSV that has a date equal to the modification date of each file.
Here is my current attempt:
import os
import datetime
import pandas as pd
from pandas import Series, DataFrame
import glob as glob
path =r'C:xxx'
allFiles = glob.glob(path + "/*.csv")
frame = pd.DataFrame()
def modification_date(filename):
t = os.path.getmtime(filename)
return datetime.datetime.fromtimestamp(t).strftime('%Y-%m-%d')
list_ = []
for file_ in allFiles:
df = pd.read_csv(file_,index_col=None, header=0)
df["DATE"] = pd.to_datetime(df["DATE"], format='%Y-%m-%d')
filedate = modification_date(allFiles)
df = df[(df["DATE"] == filedate)]
list_.append(df)
frame = pd.concat(list_)
frame.reset_index(inplace=True, drop=True)
This fails because the loop here creates a list of modification dates (since the folder contains many CSV's) that the function modification_date can't handle. Error is: "TypeError: coercing to Unicode: need string or buffer, list found"
I'm trying to wrap my head around how to modify this so each CSV is evaluated separately but can't seem to get far.
I would do it this way:
import os
import glob
import pandas as pd
fmask = 'C:/Temp/.data/aaa/*.csv'
all_files = glob.glob(fmask)
# returns file's modification date (the time part will be truncated)
def get_mdate(filename):
return (pd.to_datetime(os.path.getmtime(filename), unit='s')
.replace(hour=0, minute=0, second=0, microsecond=0))
df = pd.concat([pd.read_csv(f, parse_dates=['DATE'])
.query('DATE == #get_mdate(#f)')
for f in all_files
],
ignore_index=True)
Test:
1.csv: # modification date: 2016-07-07
DATE,VAL
2016-07-06,10
2016-07-06,10
2016-07-05,10
2016-07-07,110
2016-07-07,110
2.csv: # modification date: 2016-07-05
DATE,VAL
2016-07-06,1
2016-07-06,1
2016-07-05,1
2016-07-07,11
2016-07-07,11
Result:
In [208]: %paste
df = pd.concat([pd.read_csv(f, parse_dates=['DATE'])
.query('DATE == #get_mdate(#f)')
for f in all_files
],
ignore_index=True)
## -- End pasted text --
In [209]: df
Out[209]:
DATE VAL
0 2016-07-07 110
1 2016-07-07 110
2 2016-07-05 1