I'm using xlwt to create tables in excel. In excel there is a feature format as table which makes the table have an automatic filters for each column. Is there a way to do it using python?
You can do it with Pandas also. Here's an example:
import pandas as pd
df = pd.DataFrame({
'city': ['New York', 'London', 'Prague'],
'population': [19.5, 7.4, 1.3],
'date_of_birth': ['1625', '43', 'early 8th century'],
'status_of_magnetism': ['nice to visit', 'nice to visit', 'definetely MUST visit']
})
# initialize ExcelWriter and set df as output
writer = pd.ExcelWriter(r'D:\temp\sample.xlsx', engine='xlsxwriter')
df.to_excel(writer, sheet_name='Cities', index=False)
# worksheet is an instance of Excel sheet "Cities" - used for inserting the table
worksheet = writer.sheets['Cities']
# workbook is an instance of the whole book - used i.e. for cell format assignment
workbook = writer.book
Then define format of a cell (i.e. rotate text, set vertical and horizontal align) via workbook.add_format
header_cell_format = workbook.add_format()
header_cell_format.set_rotation(90)
header_cell_format.set_align('center')
header_cell_format.set_align('vcenter')
Then...
# create list of dicts for header names
# (columns property accepts {'header': value} as header name)
col_names = [{'header': col_name} for col_name in df.columns]
# add table with coordinates: first row, first col, last row, last col;
# header names or formatting can be inserted into dict
worksheet.add_table(0, 0, df.shape[0], df.shape[1]-1, {
'columns': col_names,
# 'style' = option Format as table value and is case sensitive
# (look at the exact name into Excel)
'style': 'Table Style Medium 10'
})
Alternatively worksheet.add_table('A1:D{}'.format(shape[0]), {...}) can be used, but for df with more columns or shifted start position the AA, AB,... combinations would have to be calculated (instead of "D")
And finally - the following loop rewrites headers and applies header_cell_format. Which we already did in worksheet.add_table(...) and so it looks redundant, but this is a way to use Excel's AutoFit option - without this all header cells would have default width (or cell height if you use the 90degs rotation) and so either not the whole content would be visble, or set_shrink() would have to be applied...and then the content wouldn't be readable :).
(tested in Office 365)
# skip the loop completly if AutoFit for header is not needed
for i, col in enumerate(col_names):
# apply header_cell_format to cell on [row:0, column:i] and write text value from col_names in
worksheet.write(0, i, col['header'], header_cell_format)
# save writer object and created Excel file with data from DataFrame
writer.save()
OK, after searching the web, I realized that with xlwt it's not possible to do it, but with XlsxWriter it's possible and very easy and convenient.
If you want to apply table formatting to a dataframe that you output to excel using XlsxWriter use the docs at https://xlsxwriter.readthedocs.io/example_pandas_table.html
Per the comment recommendation.
The following was my original less elegant solution format_tbl:
import pandas as pd
def format_tbl(writer, sheet_name, df):
outcols = df.columns
if len(outcols) > 25:
raise ValueError('table width out of range for current logic')
tbl_hdr = [{'header':c} for c in outcols]
bottom_num = len(df)+1
right_letter = chr(65-1+len(outcols))
tbl_corner = right_letter + str(bottom_num)
worksheet = writer.sheets[sheet_name]
worksheet.add_table('A1:' + tbl_corner, {'columns':tbl_hdr})
df = pd.DataFrame({
'city': ['New York', 'London', 'Prague'],
'population': [19.5, 7.4, 1.3],
'date_of_birth': ['1625', '43', 'early 8th century'],
'status_of_magnetism': ['nice to visit', 'nice to visit', 'definetely MUST visit']
})
fn_out='blah.xlsx'
with pd.ExcelWriter(fn_out, mode='w', engine='xlsxwriter') as writer:
sheet_name='xxx'
df.to_excel(writer, sheet_name=sheet_name, index=False)
format_tbl(writer, sheet_name, df)
Related
I am being asked to generate some Excel reports. I am currently using pandas quite heavily for my data, so naturally I would like to use the pandas.ExcelWriter method to generate these reports. However the fixed column widths are a problem.
The code I have so far is simple enough. Say I have a dataframe called df:
writer = pd.ExcelWriter(excel_file_path, engine='openpyxl')
df.to_excel(writer, sheet_name="Summary")
I was looking over the pandas docs, and I don't really see any options to set column widths. Is there a trick to make it such that the columns auto-adjust to the data? Or is there something I can do after the fact to the xlsx file to adjust the column widths?
(I am using the OpenPyXL library, and generating .xlsx files - if that makes any difference.)
Inspired by user6178746's answer, I have the following:
# Given a dict of dataframes, for example:
# dfs = {'gadgets': df_gadgets, 'widgets': df_widgets}
writer = pd.ExcelWriter(filename, engine='xlsxwriter')
for sheetname, df in dfs.items(): # loop through `dict` of dataframes
df.to_excel(writer, sheet_name=sheetname) # send df to writer
worksheet = writer.sheets[sheetname] # pull worksheet object
for idx, col in enumerate(df): # loop through all columns
series = df[col]
max_len = max((
series.astype(str).map(len).max(), # len of largest item
len(str(series.name)) # len of column name/header
)) + 1 # adding a little extra space
worksheet.set_column(idx, idx, max_len) # set column width
writer.save()
Dynamically adjust all the column lengths
writer = pd.ExcelWriter('/path/to/output/file.xlsx')
df.to_excel(writer, sheet_name='sheetName', index=False, na_rep='NaN')
for column in df:
column_length = max(df[column].astype(str).map(len).max(), len(column))
col_idx = df.columns.get_loc(column)
writer.sheets['sheetName'].set_column(col_idx, col_idx, column_length)
writer.save()
Manually adjust a column using Column Name
col_idx = df.columns.get_loc('columnName')
writer.sheets['sheetName'].set_column(col_idx, col_idx, 15)
Manually adjust a column using Column Index
writer.sheets['sheetName'].set_column(col_idx, col_idx, 15)
In case any of the above is failing with
AttributeError: 'Worksheet' object has no attribute 'set_column'
make sure to install xlsxwriter:
pip install xlsxwriter
For a more comprehensive explanation you can read the article How to Auto-Adjust the Width of Excel Columns with Pandas ExcelWriter on TDS.
I'm posting this because I just ran into the same issue and found that the official documentation for Xlsxwriter and pandas still have this functionality listed as unsupported. I hacked together a solution that solved the issue i was having. I basically just iterate through each column and use worksheet.set_column to set the column width == the max length of the contents of that column.
One important note, however. This solution does not fit the column headers, simply the column values. That should be an easy change though if you need to fit the headers instead. Hope this helps someone :)
import pandas as pd
import sqlalchemy as sa
import urllib
read_server = 'serverName'
read_database = 'databaseName'
read_params = urllib.quote_plus("DRIVER={SQL Server};SERVER="+read_server+";DATABASE="+read_database+";TRUSTED_CONNECTION=Yes")
read_engine = sa.create_engine("mssql+pyodbc:///?odbc_connect=%s" % read_params)
#Output some SQL Server data into a dataframe
my_sql_query = """ SELECT * FROM dbo.my_table """
my_dataframe = pd.read_sql_query(my_sql_query,con=read_engine)
#Set destination directory to save excel.
xlsFilepath = r'H:\my_project' + "\\" + 'my_file_name.xlsx'
writer = pd.ExcelWriter(xlsFilepath, engine='xlsxwriter')
#Write excel to file using pandas to_excel
my_dataframe.to_excel(writer, startrow = 1, sheet_name='Sheet1', index=False)
#Indicate workbook and worksheet for formatting
workbook = writer.book
worksheet = writer.sheets['Sheet1']
#Iterate through each column and set the width == the max length in that column. A padding length of 2 is also added.
for i, col in enumerate(my_dataframe.columns):
# find length of column i
column_len = my_dataframe[col].astype(str).str.len().max()
# Setting the length if the column header is larger
# than the max column value length
column_len = max(column_len, len(col)) + 2
# set the column length
worksheet.set_column(i, i, column_len)
writer.save()
There is a nice package that I started to use recently called StyleFrame.
it gets DataFrame and lets you to style it very easily...
by default the columns width is auto-adjusting.
for example:
from StyleFrame import StyleFrame
import pandas as pd
df = pd.DataFrame({'aaaaaaaaaaa': [1, 2, 3],
'bbbbbbbbb': [1, 1, 1],
'ccccccccccc': [2, 3, 4]})
excel_writer = StyleFrame.ExcelWriter('example.xlsx')
sf = StyleFrame(df)
sf.to_excel(excel_writer=excel_writer, row_to_add_filters=0,
columns_and_rows_to_freeze='B2')
excel_writer.save()
you can also change the columns width:
sf.set_column_width(columns=['aaaaaaaaaaa', 'bbbbbbbbb'],
width=35.3)
UPDATE 1
In version 1.4 best_fit argument was added to StyleFrame.to_excel.
See the documentation.
UPDATE 2
Here's a sample of code that works for StyleFrame 3.x.x
from styleframe import StyleFrame
import pandas as pd
columns = ['aaaaaaaaaaa', 'bbbbbbbbb', 'ccccccccccc', ]
df = pd.DataFrame(data={
'aaaaaaaaaaa': [1, 2, 3, ],
'bbbbbbbbb': [1, 1, 1, ],
'ccccccccccc': [2, 3, 4, ],
}, columns=columns,
)
excel_writer = StyleFrame.ExcelWriter('example.xlsx')
sf = StyleFrame(df)
sf.to_excel(
excel_writer=excel_writer,
best_fit=columns,
columns_and_rows_to_freeze='B2',
row_to_add_filters=0,
)
excel_writer.save()
There is probably no automatic way to do it right now, but as you use openpyxl, the following line (adapted from another answer by user Bufke on how to do in manually) allows you to specify a sane value (in character widths):
writer.sheets['Summary'].column_dimensions['A'].width = 15
By using pandas and xlsxwriter you can do your task, below code will perfectly work in Python 3.x. For more details on working with XlsxWriter with pandas this link might be useful https://xlsxwriter.readthedocs.io/working_with_pandas.html
import pandas as pd
writer = pd.ExcelWriter(excel_file_path, engine='xlsxwriter')
df.to_excel(writer, sheet_name="Summary")
workbook = writer.book
worksheet = writer.sheets["Summary"]
#set the column width as per your requirement
worksheet.set_column('A:A', 25)
writer.save()
I found that it was more useful to adjust the column with based on the column header rather than column content.
Using df.columns.values.tolist() I generate a list of the column headers and use the lengths of these headers to determine the width of the columns.
See full code below:
import pandas as pd
import xlsxwriter
writer = pd.ExcelWriter(filename, engine='xlsxwriter')
df.to_excel(writer, index=False, sheet_name=sheetname)
workbook = writer.book # Access the workbook
worksheet= writer.sheets[sheetname] # Access the Worksheet
header_list = df.columns.values.tolist() # Generate list of headers
for i in range(0, len(header_list)):
worksheet.set_column(i, i, len(header_list[i])) # Set column widths based on len(header)
writer.save() # Save the excel file
At work, I am always writing the dataframes to excel files. So instead of writing the same code over and over, I have created a modulus. Now I just import it and use it to write and formate the excel files. There is one downside though, it takes a long time if the dataframe is extra large.
So here is the code:
def result_to_excel(output_name, dataframes_list, sheet_names_list, output_dir):
out_path = os.path.join(output_dir, output_name)
writerReport = pd.ExcelWriter(out_path, engine='xlsxwriter',
datetime_format='yyyymmdd', date_format='yyyymmdd')
workbook = writerReport.book
# loop through the list of dataframes to save every dataframe into a new sheet in the excel file
for i, dataframe in enumerate(dataframes_list):
sheet_name = sheet_names_list[i] # choose the sheet name from sheet_names_list
dataframe.to_excel(writerReport, sheet_name=sheet_name, index=False, startrow=0)
# Add a header format.
format = workbook.add_format({
'bold': True,
'border': 1,
'fg_color': '#0000FF',
'font_color': 'white'})
# Write the column headers with the defined format.
worksheet = writerReport.sheets[sheet_name]
for col_num, col_name in enumerate(dataframe.columns.values):
worksheet.write(0, col_num, col_name, format)
worksheet.autofilter(0, 0, 0, len(dataframe.columns) - 1)
worksheet.freeze_panes(1, 0)
# loop through the columns in the dataframe to get the width of the column
for j, col in enumerate(dataframe.columns):
max_width = max([len(str(s)) for s in dataframe[col].values] + [len(col) + 2])
# define a max width to not get to wide column
if max_width > 50:
max_width = 50
worksheet.set_column(j, j, max_width)
writerReport.save()
return output_dir + output_name
Combining the other answers and comments and also supporting multi-indices:
def autosize_excel_columns(worksheet, df):
autosize_excel_columns_df(worksheet, df.index.to_frame())
autosize_excel_columns_df(worksheet, df, offset=df.index.nlevels)
def autosize_excel_columns_df(worksheet, df, offset=0):
for idx, col in enumerate(df):
series = df[col]
max_len = max((
series.astype(str).map(len).max(),
len(str(series.name))
)) + 1
worksheet.set_column(idx+offset, idx+offset, max_len)
sheetname=...
df.to_excel(writer, sheet_name=sheetname, freeze_panes=(df.columns.nlevels, df.index.nlevels))
worksheet = writer.sheets[sheetname]
autosize_excel_columns(worksheet, df)
writer.save()
you can solve the problem by calling the following function, where df is the dataframe you want to get the sizes and the sheetname is the sheet in excel where you want the modifications to take place
def auto_width_columns(df, sheetname):
workbook = writer.book
worksheet= writer.sheets[sheetname]
for i, col in enumerate(df.columns):
column_len = max(df[col].astype(str).str.len().max(), len(col) + 2)
worksheet.set_column(i, i, column_len)
import re
import openpyxl
..
for col in _ws.columns:
max_lenght = 0
print(col[0])
col_name = re.findall('\w\d', str(col[0]))
col_name = col_name[0]
col_name = re.findall('\w', str(col_name))[0]
print(col_name)
for cell in col:
try:
if len(str(cell.value)) > max_lenght:
max_lenght = len(cell.value)
except:
pass
adjusted_width = (max_lenght+2)
_ws.column_dimensions[col_name].width = adjusted_width
Yes, there is there is something you can do subsequently to the xlsx file to adjust the column widths.
Use xlwings to autofit columns. It's a pretty simple solution, see the 6 last lines of the example code. The advantage of this procedure is that you don't have to worry about font size, font type or anything else.
Requirement: Excel installation.
import pandas as pd
import xlwings as xw
path = r"test.xlsx"
# Export your dataframe in question.
df = pd._testing.makeDataFrame()
df.to_excel(path)
# Autofit all columns with xlwings.
with xw.App(visible=False) as app:
wb = xw.Book(path)
for ws in wb.sheets:
ws.autofit(axis="columns")
wb.save(path)
wb.close()
Easiest solution is to specify width of column in set_column method.
for worksheet in writer.sheets.values():
worksheet.set_column(0,last_column_value, required_width_constant)
This function works for me, also fixes the index width
def write_to_excel(writer, X, sheet_name, sep_only=False):
#writer=writer object
#X=dataframe
#sheet_name=name of sheet
#sep_only=True:write only as separate excel file, False: write as sheet to the writer object
if sheet_name=="":
print("specify sheet_name!")
else:
X.to_excel(f"{output_folder}{prefix_excel_save}_{sheet_name}.xlsx")
if not sep_only:
X.to_excel(writer, sheet_name=sheet_name)
#fix column widths
worksheet = writer.sheets[sheet_name] # pull worksheet object
for idx, col in enumerate(X.columns): # loop through all columns
series = X[col]
max_len = max((
series.astype(str).map(len).max(), # len of largest item
len(str(series.name)) # len of column name/header
)) + 1 # adding a little extra space
worksheet.set_column(idx+1, idx+1, max_len) # set column width (=1 because index = 1)
#fix index width
max_len=pd.Series(X.index.values).astype(str).map(len).max()+1
worksheet.set_column(0, 0, max_len)
if sep_only:
print(f'{sheet_name} is written as seperate file')
else:
print(f'{sheet_name} is written as seperate file')
print(f'{sheet_name} is written as sheet')
return writer
call example:
writer = write_to_excel(writer, dataframe, "Statistical_Analysis")
I may be a bit late to the party but this code works when using 'openpyxl' as your engine, sometimes pip install xlsxwriter wont solve the issue. This code below works like a charm. Edit any part as you wish.
def text_length(text):
"""
Get the effective text length in characters, taking into account newlines
"""
if not text:
return 0
lines = text.split("\n")
return max(len(line) for line in lines)
def _to_str_for_length(v, decimals=3):
"""
Like str() but rounds decimals to predefined length
"""
if isinstance(v, float):
# Round to [decimal] places
return str(Decimal(v).quantize(Decimal('1.' + '0' * decimals)).normalize())
else:
return str(v)
def auto_adjust_xlsx_column_width(df, writer, sheet_name, margin=3, length_factor=1.0, decimals=3, index=False):
sheet = writer.sheets[sheet_name]
_to_str = functools.partial(_to_str_for_length, decimals=decimals)
# Compute & set column width for each column
for column_name in df.columns:
# Convert the value of the columns to string and select the
column_length = max(df[column_name].apply(_to_str).map(text_length).max(), text_length(column_name)) + 5
# Get index of column in XLSX
# Column index is +1 if we also export the index column
col_idx = df.columns.get_loc(column_name)
if index:
col_idx += 1
# Set width of column to (column_length + margin)
sheet.column_dimensions[openpyxl.utils.cell.get_column_letter(col_idx + 1)].width = column_length * length_factor + margin
# Compute column width of index column (if enabled)
if index: # If the index column is being exported
index_length = max(df.index.map(_to_str).map(text_length).max(), text_length(df.index.name))
sheet.column_dimensions["A"].width = index_length * length_factor + margin
An openpyxl version based on #alichaudry's code.
The code 1) loads an excel file, 2) adjusts column widths and 3) saves it.
def auto_adjust_column_widths(excel_file : "Excel File Path", extra_space = 1) -> None:
"""
Adjusts column widths of the excel file and replaces it with the adjusted one.
Adjusting columns is based on the lengths of columns values (including column names).
Parameters
----------
excel_file :
excel_file to adjust column widths.
extra_space :
extra column width in addition to the value-based-widths
"""
from openpyxl import load_workbook
from openpyxl.utils import get_column_letter
wb = load_workbook(excel_file)
for ws in wb:
df = pd.DataFrame(ws.values,)
for i,r in (df.astype(str).applymap(len).max(axis=0) + extra_space).iteritems():
ws.column_dimensions[get_column_letter(i+1)].width = r
wb.save(excel_file)
Hello everyone hope you all are doing well.
Currently, I have a project I am working on that deals with a lot of data and I'm creating numerous pandas DataFrames with all the data I have and trying to compile it all into an excel file with each DataFrame having its own excel sheet. What I want to do is create a function that automatically adds each sheet to the excel file, expands the columns in each sheet, and colors cells in each sheet accordingly.
For example...
sheet14 would look something like what is attached...
Each sheet looks just like this but could have various amounts of rows but always the same amount of columns.
What I want to do is color the cells of Col1 that have a length of 1 green, length of 3 yellow, length of 5 purple, and so on.
How am I able to do this? I am able to do this with one sheet easily but to automate it is tedious because the multiple sheets part is making it difficult for me since I never had to deal with that.
Just so you know, cycled_data_aggregate looks like,
[DataFrame, 'A', 'A']
It is a <class 'list'> which contains,
[<class 'pandas.core.frame.DataFrame'>, <class 'str'>, <class 'str'>]
Thank you all so much if you help! Hope I explained everything well enough. If not just a general explanation would help as the code I made is pretty weird likely haha! :)
import pandas as pd
import openpyxl
from openpyxl.styles import Color, PatternFill, Font, Border, Side
import xlsxwriter
from xlsxwriter.utility import xl_rowcol_to_cell
out_path = "C:\\....\\....xlsx"
writer1 = pd.ExcelWriter(out_path)
def MultipleSheetAdder(cycled_data_aggregate, overwrite_sheet_name, true_false):
# If the function for cycled_data_aggregate returns None...
if cycled_data_aggregate == None:
return None
# The sheet's data
cycled_data = cycled_data_aggregate[0]
# If you want to overwrite what the sheet name is called and not use the
# cycled_data_aggregate's returned data
if true_false:
sheet_name = overwrite_sheet_name
else:
sheet_name = cycled_data_aggregate[1]
cycled_data.to_excel(writer, sheet_name=sheet_name)
for column in cycled_data:
column_length = max(cycled_data[column].astype(str).map(len).max(), len(column)) + 3
col_idx = cycled_data.columns.get_loc(column)
writer.sheets[sheet_name].set_column(col_idx, col_idx, column_length)
# Add section here to change colors of specific rows in the first two columns depending on what
# values they are.
{INSERT CODE HERE}
return None # Does this function need to even return anything?
MultipleSheetAdder(Function(raw_data), '', False)
writer1.save()
One way to add the colours is with conditional formatting. Here is an example based on your data:
import pandas as pd
# Create a Pandas dataframe from some data.
df = pd.DataFrame({'Col1': ['1.2.4', '2.2', '1.2.2', '2', '1.7.4'],
'Col2': [200, 100, 130, 140, 300],
'Col3': ['Text 1', 'Text 2', 'Text 3', 'Text 4', 'Text 5']})
# Create a Pandas Excel writer using XlsxWriter as the engine.
writer = pd.ExcelWriter('pandas_conditional.xlsx', engine='xlsxwriter')
# Convert the dataframe to an XlsxWriter Excel object.
df.to_excel(writer, sheet_name='Sheet1', index=False)
# Get the xlsxwriter workbook and worksheet objects.
workbook = writer.book
worksheet = writer.sheets['Sheet1']
format1 = workbook.add_format({'bg_color': 'green'})
format2 = workbook.add_format({'bg_color': 'yellow'})
format3 = workbook.add_format({'bg_color': 'purple'})
# Apply a conditional format to the cell range.
max_row = df.shape[0]
worksheet.conditional_format(1, 0, max_row, 0, {'type': 'formula',
'criteria': '=LEN($A2)=1',
'format': format1})
worksheet.conditional_format(1, 0, max_row, 0, {'type': 'formula',
'criteria': '=LEN($A2)=3',
'format': format2})
worksheet.conditional_format(1, 0, max_row, 0, {'type': 'formula',
'criteria': '=LEN($A2)=5',
'format': format3})
# Close the Pandas Excel writer and output the Excel file.
writer.save()
Output:
Python newbie here, I will like to flag sporadic numbers that are obviously off from the rest of the row.
In simple terms, flag numbers that seem not to belong to each row. Numbers in 100s and 100000s are considered 'off the rest'
import pandas as pd
# intialise data of lists.
data = {'A':['R1', 'R2', 'R3', 'R4', 'R5'],
'B':[12005, 18190, 1021, 13301, 31119,],
'C':[11021, 19112, 19021,15, 24509 ],
'D':[10022,19910, 19113,449999, 25519],
'E':[14029, 29100, 39022, 24509, 412271],
'F':[52119,32991,52883,69359,57835],
'G':[41218, 52991,1021,69152,79355],
'H': [43211,7672991,56881,211,77342],
'J': [31211,42901,53818,62158,69325],
}
# Create DataFrame
df = pd.DataFrame(data)
# Print the output.
df.describe()
I am trying to do something exactly like this
# I need help with step 1
#my code/pseudocode
# step 1: identify the values in each row that are don't belong to the group
# step 2: flag the identified values and export to excel
style_df = .applymap(lambda x: "background-color: yellow" if x else "") # flags the values that meets the criteria
with pd.ExcelWriter("flagged_data.xlsx", engine="openpyxl") as writer:
df.style.apply(lambda x: style_df, axis=None).to_excel(writer,index=False)
I used two conditions here one to check less than 1000 and another one for greater than 99999. Based on this condition, the code will highlight outliers in red color.
# Create a Pandas Excel writer using XlsxWriter as the engine.
writer = pd.ExcelWriter('pandas_conditional.xlsx', engine='xlsxwriter')
# Convert the dataframe to an XlsxWriter Excel object.
df.to_excel(writer, sheet_name='Sheet1')
# Get the xlsxwriter workbook and worksheet objects.
workbook = writer.book
worksheet = writer.sheets['Sheet1']
# Add a format. Light red fill with dark red text.
format1 = workbook.add_format({'bg_color': '#FFC7CE',
'font_color': '#9C0006'})
first_row = 1
first_col = 2
last_row = len(df)
last_col = 9
worksheet.conditional_format(first_row, first_col, last_row, last_col,
{'type': 'cell',
'criteria': '<',
'value': 1000,
'format': format1})
worksheet.conditional_format(first_row, first_col, last_row, last_col,
{'type': 'cell',
'criteria': '>',
'value': 99999,
'format': format1})
# Close the Pandas Excel writer and output the Excel file.
writer.save()
If you don't need to use machine learning outliers detection or Hampel filter and you already know the limits of your filter, you can simply do
def higlight_outliers(s):
# force to numeric and coerce string to NaN
s = pd.to_numeric(s, errors='coerce')
indexes = (s<1500)|(s>1000000)
return ['background-color: yellow' if v else '' for v in indexes]
styled = df.style.apply(higlight_outliers, axis=1)
styled.to_excel("flagged_data.xlsx", index=False)
I guess you could define a little better what you consider "off from the rest". This is very important when working with data.
Do you want to flag the outliers of your column B distribution for example? You could simply do a calculation of quartiles for your distributions and append those to a dict of some kind, those which are either below the lowest quartile or over the highest quartile. But you obviously would need more than those 5 rows you showed.
There are whole fields dedicated to identification of outliers using machine learning as well. The assumptions you are taking to define what should be considered "off from the rest" are very important.
Read this if you'd like more info on specifics of outlier detection:
https://towardsdatascience.com/a-brief-overview-of-outlier-detection-techniques-1e0b2c19e561
I have a dataframe with 3 columns.
I like to highlight column a as orange, column b as green, column c as yellow but controlled by end of row.
using xlsxwriter I found examples for highlighting the entire column with ".add_format" but I didn't want the entire column to be highlighted.
How can I use xlsxwriter to highlight specific cells without using ".conditional_format"?
df = {'a': ['','',''],
'b':[1,2,2]
'c':[1,2,2]}
With xlsxwriter i am applying format using 2 different ways. Mainly with the function set_column (if you don't mind the format expanding until the end of the file) and using for loops if i do not want the format to expand until the end of the file (for example borderlines and background colors).
So this is how you can apply format to your dataframe:
import pandas as pd
# Create a test df
data = {'a': ['','',''], 'b': [1,2,2], 'c': [1,2,2]}
df = pd.DataFrame(data)
# Import the file through xlsxwriter
writer = pd.ExcelWriter('test.xlsx', engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1', index=False)
workbook = writer.book
worksheet = writer.sheets['Sheet1']
# Define the formats
format_orange = workbook.add_format({'bg_color': 'orange'})
format_green = workbook.add_format({'bg_color': 'green'})
format_bold = workbook.add_format({'bold': True, 'align': 'center'})
# Start iterating through the columns and the rows to apply the format
for row in range(df.shape[0]):
worksheet.write(row+1, 0, df.iloc[row,0], format_orange)
# Alternative syntax
#for row in range(df.shape[0]):
# worksheet.write(f'A{row+2}', df.iloc[row,0], format_orange)
for row in range(df.shape[0]):
worksheet.write(row+1, 1, df.iloc[row,1], format_green)
# Here you can use the faster set_column function as you do not apply color
worksheet.set_column('C:C', 15, format_bold)
# Finally write the file
writer.save()
Output:
I am creating an excel report that should give me a result of automatic tests. It should say if they failed/ passed.
I have created the excel report from csv using this code:
import pandas as pd
import string
writer = pd.ExcelWriter("file.xlsx", engine="xlsxwriter")
df = pd.read_csv("K:\\results.csv")
df.to_excel(writer, sheet_name=os.path.basename("K:\\results.csv"))
# skip 2 rows
df.to_excel(writer, sheet_name='Sheet1', startrow=2, header=False, index=False)
workbook = writer.book
worksheet = writer.sheets['Sheet1']
# Add a header format.
header_format = workbook.add_format({
'bold': True,
'fg_color': '#ffcccc',
'border': 1})
# create dictionary for map length of columns
d = dict(zip(range(25), string.ascii_uppercase))
print (d)
max_len = d[len(df.columns) - 1]
print(max_len)
# C
# dynamically set merged columns in first row
worksheet.merge_range('A1:' + max_len + '1', 'This Sheet is for Personal Details')
for col_num, value in enumerate(df.columns.values):
# write to second row
worksheet.write(1, col_num, value, header_format)
column_len = df[value].astype(str).str.len().max()
column_len = max(column_len, len(value)) + 3
worksheet.set_column(col_num, col_num, column_len)
writer.save()
Now, if i have a cell that has the word" success" in it, i want to color it green, and if i have a cell in the excel which says "fail" in it i want to color it red. How can i access a specific cell in the excel file with the condition of whats written in it?
Thanks.
You could use a conditional format for this:
import pandas as pd
# Create a Pandas dataframe from some data.
df = pd.DataFrame({'Data': ['success', 'bar', 'fail', 'foo', 'success']})
# Create a Pandas Excel writer using XlsxWriter as the engine.
writer = pd.ExcelWriter('pandas_conditional.xlsx', engine='xlsxwriter')
# Convert the dataframe to an XlsxWriter Excel object.
df.to_excel(writer, sheet_name='Sheet1')
# Get the xlsxwriter workbook and worksheet objects.
workbook = writer.book
worksheet = writer.sheets['Sheet1']
# Add a format for fail. Light red fill with dark red text.
fail_format = workbook.add_format({'bg_color': '#FFC7CE',
'font_color': '#9C0006'})
# Add a format for pass. Green fill with dark green text.
pass_format = workbook.add_format({'bg_color': '#C6EFCE',
'font_color': '#006100'})
# Apply conditional formats to the cell range.
worksheet.conditional_format('B2:B6', {'type': 'text',
'criteria': 'containing',
'value': 'fail',
'format': fail_format})
worksheet.conditional_format('B2:B6', {'type': 'text',
'criteria': 'containing',
'value': 'success',
'format': pass_format})
# Close the Pandas Excel writer and output the Excel file.
writer.save()
Output:
See the XlsxWriter docs on Working with Conditional Formatting. Note, you can also use a numerical (row, col) range instead of the A1:D4 range, see the conditional_format().