We Keep Coding

Python Pandas xlsxwriter failing to close

I am building automation for Excel a multi-tabbed excel document. When I try to close the document I get the error below (full traceback, minus the personal details at the top), which then is corrupted and I cannot open the xlsx document. Unfortunately I haven't found any clues to go off of. I am using xlsxwriter functions to set row and column formatting, from what I've found this could be an issue but I haven't been able to track it down. Any thoughts on possible solutions?
writer.close()
File "/opt/homebrew/lib/python3.10/site-packages/pandas/io/excel/_base.py", line 1480, in close
self._save()
File "/opt/homebrew/lib/python3.10/site-packages/pandas/io/excel/_xlsxwriter.py", line 244, in _save
self.book.close()
File "/opt/homebrew/lib/python3.10/site-packages/xlsxwriter/workbook.py", line 324, in close
self._store_workbook()
File "/opt/homebrew/lib/python3.10/site-packages/xlsxwriter/workbook.py", line 709, in _store_workbook
xml_files = packager._create_package()
File "/opt/homebrew/lib/python3.10/site-packages/xlsxwriter/packager.py", line 137, in _create_package
self._write_worksheet_files()
File "/opt/homebrew/lib/python3.10/site-packages/xlsxwriter/packager.py", line 193, in _write_worksheet_files
worksheet._assemble_xml_file()
File "/opt/homebrew/lib/python3.10/site-packages/xlsxwriter/worksheet.py", line 4221, in _assemble_xml_file
self._write_cols()
File "/opt/homebrew/lib/python3.10/site-packages/xlsxwriter/worksheet.py", line 5807, in _write_cols
self._write_col_info(self.colinfo[col])
File "/opt/homebrew/lib/python3.10/site-packages/xlsxwriter/worksheet.py", line 5836, in _write_col_info
if width > 0:
TypeError: '>' not supported between instances of 'Format' and 'int'

Why does pandas.read_csv read all of the blank values in one code but not the other?

I am writing code that requires use of the pandas.read_csv function and I like to use a test python file before I implement my code into the main python file. The part of the CSV file I am trying to read in just has data displayed in a seemingly random fashion with no real columns or headers. I just want to read the information into a dataframe so I can parse the data exactly where I know it is going to be for every file. For the test code and the main code I am using the same list of CSV files. The only difference is the test code runs in a different folder and does not sit inside a function. In my test code I have no issue extracting data from the CSV file using the read_csv function but on my main program it is giving me errors. In my test code I can easily use pd.read_csv this way:
for x in range(len(filelist)):
df = pd.read_csv(filelist[x], index_col=False, nrows=15, header=None, usecols=(0,1,2,3),
dtype={0:"string",1:"string",2:"string",3:"string"})
print(df)
The output is shown below:
Output from test code execution
However, when I try to port this over into my main code it won't function the same way. If I copy the code exactly it says there is no column 1,2, or 3. My next step was to erase the usecols and dtype variables and then it gave me the error:
pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 2, saw 2
I tried adding the comma delimiter and I tried changing the engine to python. Neither worked. Eventually, I determined from the tokenizing error that the program was expecting less information on certain lines so I broke up the code so that there were two dataframes and each would hold a respective amount of columns. This finally worked. The dataframes I created were structured as shown below:
df1 = pd.read_csv(filelist[x],skiprows=range(1), index_col=False, nrows=11, header=None)
df2 = pd.read_csv(filelist[x],skiprows=range(0,13), index_col=False, nrows=2,
usecols=(0,1,2), header=None)
print(df1)
print(df2)
The output for this is shown below:
Output from main code execution
This gives me something I can work with to accomplish my task, but it was extremely frustrating working through this and I have no idea why I even needed to go through all of this. I still will have to go back through and make some final adjustments including all the calls to the variables I need from these so if I can figure out why it is not working the same in the main code it would make my life a little easier. Does anyone have any clue why I had to make these adjustments? It seems the code for my main program is not reading in empty cells or just takes the amount of spaces used for the first row it looks at and just assumes the rest should be the same. Any information would be greatly appreciated. Thank you.
I am adding the full error messages below. I made it so it only calls the first file in the list for debugging purposes. This first one is when I copy the read_csv command over exactly:
Traceback (most recent call last):
File "c:\Users\jacob.hollidge\Desktop\DCPR Threshold\DCPRthresholdV2.0.py", line 484, in <module>
checkfilevariables(filelist)
File "c:\Users\jacob.hollidge\Desktop\DCPR Threshold\DCPRthresholdV2.0.py", line 221, in checkfilevariables
df = pd.read_csv(filelist[0], index_col=False, nrows=15, header=None, usecols=(0,1,2,3),
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\util\_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 680, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 575, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 933, in __init__
self._engine = self._make_engine(f, self.engine)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 1231, in _make_engine
return mapping[engine](f, **self.options)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 146, in __init__
self._validate_usecols_names(
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\base_parser.py", line 913, in _validate_usecols_names
raise ValueError(
ValueError: Usecols do not match columns, columns expected but not found: [1, 2, 3]
This next error occurs after I remove usecols and dtype from the parameters.
Traceback (most recent call last):
File "c:\Users\jacob.hollidge\Desktop\DCPR Threshold\DCPRthresholdV2.0.py", line 483, in <module>
checkfilevariables(filelist)
File "c:\Users\jacob.hollidge\Desktop\DCPR Threshold\DCPRthresholdV2.0.py", line 221, in checkfilevariables
df = pd.read_csv(filelist[0], index_col=False, nrows=15, header=None)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\util\_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 680, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 581, in _read
return parser.read(nrows)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 1250, in read
index, columns, col_dict = self._engine.read(nrows)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 225, in read
chunks = self._reader.read_low_memory(nrows)
File "pandas\_libs\parsers.pyx", line 817, in pandas._libs.parsers.TextReader.read_low_memory
File "pandas\_libs\parsers.pyx", line 861, in pandas._libs.parsers.TextReader._read_rows
File "pandas\_libs\parsers.pyx", line 847, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas\_libs\parsers.pyx", line 1960, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 2, saw 2
This final set of errors is given after I add the delimiter=',' and engine='python' parameters while usecols and dtypes have still been removed.
Traceback (most recent call last):
File "c:\Users\jacob.hollidge\Desktop\DCPR Threshold\DCPRthresholdV2.0.py", line 483, in <module>
checkfilevariables(filelist)
File "c:\Users\jacob.hollidge\Desktop\DCPR Threshold\DCPRthresholdV2.0.py", line 221, in checkfilevariables
df = pd.read_csv(filelist[0], index_col=False, nrows=15, header=None, delimiter=',', engine='python')
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\util\_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 680, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 581, in _read
return parser.read(nrows)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py", line 1250, in read
index, columns, col_dict = self._engine.read(nrows)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\python_parser.py", line 270, in read
alldata = self._rows_to_cols(content)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\python_parser.py", line 1013, in _rows_to_cols
self._alert_malformed(msg, row_num + 1)
File "C:\Users\jacob.hollidge\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\python_parser.py", line 739, in _alert_malformed
raise ParserError(msg)
pandas.errors.ParserError: Expected 2 fields in line 13, saw 4

Pandas and glob: convert all xlsx files in folder to csv – TypeError: __init__() got an unexpected keyword argument 'xfid'

I have a folder with many xlsx files that I'd like to convert to csv files.
During my research, if found several threads about this topic, such as this or that one. Based on this, I formulated the following code using glob and pandas:
import glob
import pandas as pd
path = r'/Users/.../xlsx files'
excel_files = glob.glob(path + '/*.xlsx')
for excel in excel_files:
out = excel.split('.')[0]+'.csv'
df = pd.read_excel(excel) # error occurs here
df.to_csv(out)
But unfortunately, I got the following error message that I could not interpret in this context and I could not figure out how to solve this problem:
Traceback (most recent call last):
File "<input>", line 11, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/util/_decorators.py", line 299, in wrapper
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 336, in read_excel
io = ExcelFile(io, storage_options=storage_options, engine=engine)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 1131, in __init__
self._reader = self._engines[engine](self._io, storage_options=storage_options)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 475, in __init__
super().__init__(filepath_or_buffer, storage_options=storage_options)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 391, in __init__
self.book = self.load_workbook(self.handles.handle)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 486, in load_workbook
return load_workbook(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 317, in load_workbook
reader.read()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 281, in read
apply_stylesheet(self.archive, self.wb)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/styles/stylesheet.py", line 198, in apply_stylesheet
stylesheet = Stylesheet.from_tree(node)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/styles/stylesheet.py", line 103, in from_tree
return super(Stylesheet, cls).from_tree(node)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/descriptors/serialisable.py", line 87, in from_tree
obj = desc.expected_type.from_tree(el)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/descriptors/serialisable.py", line 87, in from_tree
obj = desc.expected_type.from_tree(el)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/descriptors/serialisable.py", line 103, in from_tree
return cls(**attrib)
TypeError: __init__() got an unexpected keyword argument 'xfid'
Does anyone know how to fix this? Thanks a lot for your help!

I had the same problem here. After some hours thinking and searching I realized the problem is, actually, the file. I opened it using MS Excel, and save. Alakazan, problem solved.
The file was downloaded, so i think it's a "security" error or just an error from how the file was created. xD
EDIT:
It's not a security problem, but actually an error from the generation of file. The correct has the double of kb the wrong file.
An solution is: if using xlrd==1.2.0 the file can be opened, you can, after doing this, call read_excel to the Book(file opened by xlrd).
import xlrd
# df = pd.read_excel('TabelaPrecos.xlsx')
# The line above is the same result
a = xlrd.open_workbook('TabelaPrecos.xlsx')
b = pd.read_excel(a)

Pyexcel doesn't merge .xls files

I'm trying to merge multiple .xls files into a single workbook, where each file is inserted into a sheet, named with the .xls filename.
While surfing on web, I've seen the documentation of Pyexcel and a specific module which, as written here, could do the job easly.
Here's the code.
from pyexcel.cookbook import merge_all_to_a_book
import glob
merge_all_to_a_book(glob.glob("Dir\*.xls"),"output.xls")
As expected, it doesn't work. Here's the console output.
File "..\Desktop\scripts\provaimport.py", line 48, in <module>
merge_all_to_a_book(glob.glob("C:\Users\Tesisti\Desktop\forpythonscript\*.xls"),"output.xls")
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel\cookbook.py", line 148, in merge_all_to_a_book
merged.save_as(outfilename)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel\internal\meta.py", line 339, in save_as
return save_book(self, file_name=filename, **keywords)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel\internal\core.py", line 51, in save_book
return _save_any(a_source, book)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel\internal\core.py", line 55, in _save_any
a_source.write_data(instance)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel\plugins\sources\file_output.py", line 38, in write_data
**self._keywords)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel\plugins\renderers\excel.py", line 30, in render_book_to_file
save_data(file_name, book.to_dict(), **keywords)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel_io\io.py", line 119, in save_data
**keywords)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel_io\io.py", line 141, in store_data
writer.write(data)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel_io\book.py", line 58, in __exit__
self.close()
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\pyexcel_xls\xlsw.py", line 86, in close
self.work_book.save(self._file_alike_object)
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\xlwt\Workbook.py", line 710, in save
doc.save(filename_or_stream, self.get_biff_data())
File "C:\Users\Tesisti\Anaconda2\lib\site-packages\xlwt\Workbook.py", line 680, in get_biff_data
self.__worksheets[self.__active_sheet].selected = True
Any idea on how to fix?

It seems to me that glob.glob("Dir*.xls") returned an empty list of files. Hence pyexcel's plugin pyexcel-xls fails to create an empty file.
The current solution, I would recommend is to take the latest pyexcel-xls and use try-except statement around merge_all_to_a_book, catching empty file case.

Writing data to an existing excel sheet using openpyxl

I'm quit new to coding in general.
What i want to achieve is to make an script that runs to a list of employers in excel and weekly generate a new hour-sheet. And by generating i mean copy for every employer an empty hour-sheet and rename it, and also change the week-number and employer-name in the newly made copy.
I didn't start with a loop, because i first wanted to made the part that change the employers-name and week-number. I've already search the internet for some answers, but i can't get the code to work, keep getting error messages.
So here is my code so far:
import os
import shutil
import time
from openpyxl import load_workbook
#calculate the year and week number
from time import strftime
year = (time.strftime("%Y"))
week = str(int(time.strftime("%W"))+1)
year_week = year + "_" + week
#create weekly houresheets per employer
employer = "Adam"
hsheets_dir = "C:\\test\\"
old_file_name = "blanco.xlsx"
new_file_name = employer + "_" + year_week + ".xlsx"
dest_filename = (hsheets_dir + new_file_name)
shutil.copy2((hsheets_dir + old_file_name), dest_filename)
#change employer name and weeknumber
def insert_xlsx(dest, empl, wk):
#Open an xlsx for reading
print (dest)
wb = load_workbook(filename = dest)
#Get the current Active Sheet
ws = wb.get_sheet_by_name("Auto")
ws.cell(row=1,column=2).value = empl
ws.cell(row=2,column=2).value = wk
wb.save(dest)
insert_xlsx(dest_filename, employer, week_str)
And here is the error message i keep getting:
Traceback (most recent call last):
File "G:\ALL\Urenverantwoording\Wekelijks\Genereer_weekstaten.py", line 46, in <module>
insert_xlsx(dest_filename, employer, week)
File "G:\ALL\Urenverantwoording\Wekelijks\Genereer_weekstaten.py", line 44, in insert_xlsx
wb.save(dest)
File "C:\Python34\lib\site-packages\openpyxl\workbook\workbook.py", line 298, in save
save_workbook(self, filename)
File "C:\Python34\lib\site-packages\openpyxl\writer\excel.py", line 198, in save_workbook
writer.save(filename, as_template=as_template)
File "C:\Python34\lib\site-packages\openpyxl\writer\excel.py", line 181, in save
self.write_data(archive, as_template=as_template)
File "C:\Python34\lib\site-packages\openpyxl\writer\excel.py", line 87, in write_data
self._write_worksheets(archive)
File "C:\Python34\lib\site-packages\openpyxl\writer\excel.py", line 114, in _write_worksheets
write_worksheet(sheet, self.workbook.shared_strings,
File "C:\Python34\lib\site-packages\openpyxl\writer\worksheet.py", line 302, in write_worksheet
xf.write(comments)
File "C:\Python34\lib\contextlib.py", line 66, in __exit__
next(self.gen)
File "C:\Python34\lib\site-packages\openpyxl\xml\xmlfile.py", line 51, in element
self._write_element(el)
File "C:\Python34\lib\site-packages\openpyxl\xml\xmlfile.py", line 78, in _write_element
xml = tostring(element)
File "C:\Python34\lib\xml\etree\ElementTree.py", line 1126, in tostring
short_empty_elements=short_empty_elements)
File "C:\Python34\lib\xml\etree\ElementTree.py", line 778, in write
short_empty_elements=short_empty_elements)
File "C:\Python34\lib\xml\etree\ElementTree.py", line 943, in _serialize_xml
short_empty_elements=short_empty_elements)
File "C:\Python34\lib\xml\etree\ElementTree.py", line 943, in _serialize_xml
short_empty_elements=short_empty_elements)
File "C:\Python34\lib\xml\etree\ElementTree.py", line 935, in _serialize_xml
v = _escape_attrib(v)
File "C:\Python34\lib\xml\etree\ElementTree.py", line 1093, in _escape_attrib
_raise_serialization_error(text)
File "C:\Python34\lib\xml\etree\ElementTree.py", line 1059, in _raise_serialization_error
"cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize 3 (type int)
Can somewone put me in the right directions?
Many thanks

I think based on your responses then that the problem lies with your existing hour-sheet Excel spreadsheet:
Try starting with a copy of your existing spreadsheet and removing all of the entries. Hopefully this too will work.
If this fails, start with a new blank spreadsheet.
Bit by bit copy the existing data and repeat your script.
By doing this you will might be able to isolate the feature which is not compatible with openpyxl.
Alternatively, you might be able to write the whole thing from your Python script, and skip trying to modify a semi-filled in one. This would then be 100% compatible.