The folder contains at least 20 excels. Each excel contains nine sheets. These excels have same type of sheets (same header but different data). I need to concat these 20 excels sheet by sheet into one excel. And the first two sheets in each excel are instruction. They are skippable. How can I achieve this? Thanks!
Example: File A Sheet 3, File B sheet 3, File A sheet 4, File B sheet 4
So eventually the combination file will be like:
I had to do something similair a while back:
This code should do the trick for you:
import pandas as pd
import os
collection = {}
for file in os.listdir():
if file.endswith(".xlsx"):
mysheets = pd.ExcelFile(file)
mysheetnames = mysheets.sheet_names
for i in mysheetnames[2:]: #change the 2 in [2:] to change how many sheets you delete
mydata = pd.read_excel(file, i)
combi = collection.get(i, [])
collection[i] = combi + [mydata]
writer = pd.ExcelWriter('output.xlsx', engine='xlsxwriter')
for key in collection:
myresult = pd.concat(collection.get(key), sort=False)
myresult.to_excel(writer, sheet_name=key)
writer.save()
Related
I'm trying to read in an Excel file with multiple sheets (s.t that all columns are strings). The below code works for that but it doen't get the correct sheet names. So my dic_excel which is a dictionary with all sheet names and the corresponding data has the following keys: 'Sheet1', 'Sheet2', 'Sheet3', etc. But the actual names of the sheets are different. How do I get the actual names of the sheets?
dic_excel={}
excel = pd.ExcelFile(excel_path)
for sheet in excel.sheet_names:
print(sheet)
columns = excel.parse(sheet).columns
converters = {col: str for col in columns}
dic_excel[sheet] = excel.parse(sheet, converters=converters)
Here is two ways to get the real names of your Excel sheets:
By using pandas.DataFrame.keys with pandas
import pandas as pd
excel = pd.read_excel(excel_path, sheet_name=None)
dic_excel = df.keys()
This will return a dictionnary of the sheetnames
By using Workbook.sheetname with openpyxl
import openpyxl
wb = openpyxl.load_workbook(excel_path)
list_excel = wb.sheetnames
This will return a list of the sheetnames
I have a code that read multiple files inside the directory and every excel file have more than 10 sheet's. After that I need to exclude some sheet's every file's and the others extracted.
I got all data needed, but the problem is every sheet's from the excel created new Dataframe even I used concat so when I save it to json only the last dataframe per file saved instead of whole data.
Here's my code:
excluded_sheet = ['Sheet 2','Sheet 6']
for index, xls_path in enumerate(file_paths):
data_file = pd.ExcelFile(xls_path)
sheets = [ sheet for sheet in data_file.sheet_names if sheet not in excluded_sheet ]
for sheet_name in sheets:
file = xls_path.rfind(".")
head, tail = os.path.split(xls_path[1:file])
df =pd.concat([pd.read_excel(xls_path, sheet_name=sheet_name, header=None)],ignore_index=True)
df.insert(loc=0, column='sheet name', value=sheet_name)
pd.DataFrame(df.to_json(f"{json_folder_path}{tail}.json", orient='records',indent=4))
I didn't used sheet_name=None because I need to read sheet name and add to column values.
Data status of my dataFrame:
I got many DF because every sheet create new DF, instead of 2 DF only since I have 2 files inside the directory. Thanks guys for your help.
You can use list comprehension for join all sheetnames to one DataFrame:
...
...
sheets = [ sheet for sheet in data_file.sheet_names if sheet not in excluded_sheet ]
file = xls_path.rfind(".")
head, tail = os.path.split(xls_path[1:file])
dfs = [pd.read_excel(xls_path,sheet_name=sheet_name,header=None) for sheet_name in sheets]
df =pd.concat(dfs,keys=sheets)
df = df.reset_index(level=0, drop=True).rename_axis('sheet name').reset_index()
pd.DataFrame(df.to_json(f"{json_folder_path}{tail}.json", orient='records',indent=4))
Or create helper list dfs with append DataFrames per loop, outside loop use concat:
...
...
sheets = [ sheet for sheet in data_file.sheet_names if sheet not in excluded_sheet ]
dfs = []
for sheet_name in sheets:
file = xls_path.rfind(".")
head, tail = os.path.split(xls_path[1:file])
df = pd.read_excel(xls_path, sheet_name=sheet_name, header=None)
df.insert(loc=0, column='sheet name', value=sheet_name)
dfs.append(df)
df1 = pd.concat(dfs,ignore_index=True)
pd.DataFrame(df1.to_json(f"{json_folder_path}{tail}.json", orient='records',indent=4))
It's my first time to use pandas, I have multiple excel files, that i want to combine all into one Excel file using python pandas.
I managed to merge the content of the first sheets in each excel file into one sheet in a new excel file like this shown in the figure below:
combined sheets in one sheet
I wrote this code to implement this:
import glob
import pandas as pd
path = "C:/folder"
file_identifier = "*.xls"
all_data = pd.DataFrame()
for f in glob.glob(path + "/*" + file_identifier):
df = pd.read_excel(f)
all_data = all_data.append(df,ignore_index=True)
writer = pd.ExcelWriter('combined.xls', engine='xlsxwriter')
all_data.to_excel(writer, sheet_name='Summary Sheet')
writer.save()
file_df = pd.read_excel("C:/folder/combined.xls")
# Keep only FIRST record from set of duplicates
file_df_first_record = file_df.drop_duplicates(subset=["Test summary", "Unnamed: 1", "Unnamed: 2",
"Unnamed: 3"], keep="first")
file_df_first_record.to_excel("filtered.xls", index=False, sheet_name='Summary Sheet')
But I have two issues:
How to remove cells that has "Unnamed" as shown in the previous figure
How to copy other worksheets (the second worksheet in each Excel file, not the first worksheet) from all other Excel files and put it in one Excel file with multiple worksheets and with different students names like shown in the picture.
all worksheets in one excel file
So i managed to combine worksheet1 from all Excel files in one sheet, but now I want to copy A, B, C, D, E worksheets into one Excel file that has all other remaining worksheets in other Excel files.
Each Excel file of the ones I have looks like this
single excel file
If you want to have all data gathered together in one worksheet you can use the following script:
Put all excel workbooks (i.e. excel files) to be processed into a
folder (see variable paths).
Get the paths of all workbooks in that folder using
glob.glob.
Return all worksheets of each workbook with read_excel(path, sheet_name=None) and prepare them for merging.
Merge all worksheets with pd.concat.
Export the final output to_excel.
import pandas as pd
import glob
paths = glob.glob(r"C:\excelfiles\*.xlsx")
path_save = r"finished.xlsx"
df_lst = [pd.read_excel(path, sheet_name=None).values() for path in paths]
df_lst = [y.transpose().reset_index().transpose() for x in df_lst for y in x]
df_result = pd.concat(df_lst, ignore_index=True)
df_result.to_excel(path_save, index=False, header=False)
i want merge multi excel file(1.xlsm, 2.xlsm....) to [A.xlsm] file with macro, 3sheets
so i try to merge
# input_file = (./*.xlsx)
all_data = pd.DataFrame()
for f in (input_file):
df = pd.read_excel(f)
all_data = all_data.append(df,ignore_index=True, sort=False)
writer = pd.ExcelWriter(A.xlsm, engine='openpyxl')
all_data.to_excel(writer,'Sheet1')
writer.save()
the code dose not error,
but result file[A.xlsm] is error to open,
so i change extension to A.xlsx and open.
it opening is OK but disappear all Sheets and macro.
how can i merge multi xlsx file to xlsm file with macro?
I believe that if you want to use macro-enabled workbooks you need to load them with keep_vba=True:
from openpyxl import load_workbook
XlMacroFile = load_workbook('A.xlsm',keep_vba=True)
To preserve separate sheets, you can do something like
df_list = #list of your dataframes
filename = #name of your output file
with pd.ExcelWriter(filename) as writer:
for df in df_list:
df.to_excel(writer, sheet_name='sheet_name_goes_here')
This will write each dataframe in a separate sheet in your output excel file.
I am able to combine multiple excel files having one sheet currently.
I want to combine multiple sheets having two different sheets in each excel file with giving name to each sheets How can I achieve this?
Here below is my current code for combining single sheet in multiple excel files without giving sheet name to Combined excel file
import pandas as pd
# filenames
excel_names = ["xlsx1.xlsx", "xlsx2.xlsx", "xlsx3.xlsx"]
# read them in
excels = [pd.ExcelFile(name) for name in excel_names]
# turn them into dataframes
frames = [x.parse(x.sheet_names[0], header=None,index_col=None) for x in excels]
# delete the first row for all frames except the first
# i.e. remove the header row -- assumes it's the first
frames[1:] = [df[1:] for df in frames[1:]]
# concatenate them..
combined = pd.concat(frames)
# write it out
combined.to_excel("c.xlsx", header=False, index=False)
First combine the first and the second sheet separately
import pandas as pd
# filenames
excel_names = ["xlsx1.xlsx", "xlsx2.xlsx", "xlsx3.xlsx"]
def combine_excel_to_dfs(excel_names, sheet_name):
sheet_frames = [pd.read_excel(x, sheet_name=sheet_name) for x in excel_names]
combined_df = pd.concat(sheet_frames).reset_index(drop=True)
return combined_df
df_first = combine_excel_to_dfs(excel_names, 0)
df_second = combine_excel_to_dfs(excel_names, 1)
Use pd.ExcelWriter
And write these sheets to the same excel file:
# Create a Pandas Excel writer using XlsxWriter as the engine.
writer = pd.ExcelWriter('two_sheets_combined.xlsx', engine='xlsxwriter')
# Write each dataframe to a different worksheet.
df_first.to_excel(writer, sheet_name='Sheet1')
df_second.to_excel(writer, sheet_name='Sheet2')
# Close the Pandas Excel writer and output the Excel file.
writer.save()
You can use:
#number of sheets
N = 2
#get all sheets to nested lists
frames = [[x.parse(y, index_col=None) for y in x.sheet_names] for x in excels]
#print (frames)
#combine firt dataframe from first list with first df with second list...
combined = [pd.concat([x[i] for x in frames], ignore_index=True) for i in range(N)]
#print (combined)
#write to file
writer = pd.ExcelWriter('c.xlsx', engine='xlsxwriter')
for i, x in enumerate(combined):
x.to_excel(writer, sheet_name='Sheet{}'.format(i + 1))
writer.save()