We Keep Coding

Python assertion error - reading xls from URL

I am trying to read an xls file from a url into python dataframe. However I am getting below assertion error.
Traceback (most recent call last):
File "c:\Sample_project\venv\excel_read_v1.py", line 17, in
df = pd.read_excel("file.xls",
File "C:\Sample_project\venv\lib\site-packages\pandas\util_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "C:\Sample_project\venv\lib\site-packages\pandas\io\excel_base.py", line 457, in read_excel
io = ExcelFile(io, storage_options=storage_options, engine=engine)
File "C:\Sample_project\venv\lib\site-packages\pandas\io\excel_base.py", line 1419, in init
self._reader = self._engines[engine](self.io, storage_options=storage_options)
File "C:\Sample_project\venv\lib\site-packages\pandas\io\excel_xlrd.py", line 25, in init
super().init(filepath_or_buffer, storage_options=storage_options)
File "C:\Sample_project\venv\lib\site-packages\pandas\io\excel_base.py", line 518, in init
self.book = self.load_workbook(self.handles.handle)
File "C:\Sample_project\venv\lib\site-packages\pandas\io\excel_xlrd.py", line 38, in load_workbook
return open_workbook(file_contents=data)
File "C:\Sample_project\venv\lib\site-packages\xlrd_init.py", line 172, in open_workbook
bk = open_workbook_xls(
File "C:\Sample_project\venv\lib\site-packages\xlrd\book.py", line 104, in open_workbook_xls
bk.parse_globals()
File "C:\Sample_project\venv\lib\site-packages\xlrd\book.py", line 1211, in parse_globals
self.handle_sst(data)
File "C:\Sample_project\venv\lib\site-packages\xlrd\book.py", line 1178, in handle_sst
self._sharedstrings, rt_runlist = unpack_SST_table(strlist, uniquestrings)
File "C:\Sample_project\venv\lib\site-packages\xlrd\book.py", line 1472, in unpack_SST_table
assert _unused_i == nstrings - 1
AssertionError
I read some other suggestions on stackoverflow that if I remove the last few empty lines from the excel then it would work. So I tried that out by downloading the file in a local folder , removing the last 2 empty rows and then reading the file from the local folder, this works. But i need the code to somehow able to handle it while reading from the url so that we can automate the process
I have tried using openpyxl and xlrd to read the file.
---Code snapshot below--------
import openpyxl
import xlrd
from xlrd import open_workbook
import requests
import pandas as pd
url = url
r = requests.get(url)
with open('maskefile.xls', 'wb') as output:
output.write(r.content)
df = pd.read_excel("maskedfile.xls",sheet_name = "maskedsheetname")
df.to_csv("C:\Sample_project\maskedfile.csv" ,index = False)

Pandas and glob: convert all xlsx files in folder to csv – TypeError: __init__() got an unexpected keyword argument 'xfid'

I have a folder with many xlsx files that I'd like to convert to csv files.
During my research, if found several threads about this topic, such as this or that one. Based on this, I formulated the following code using glob and pandas:
import glob
import pandas as pd
path = r'/Users/.../xlsx files'
excel_files = glob.glob(path + '/*.xlsx')
for excel in excel_files:
out = excel.split('.')[0]+'.csv'
df = pd.read_excel(excel) # error occurs here
df.to_csv(out)
But unfortunately, I got the following error message that I could not interpret in this context and I could not figure out how to solve this problem:
Traceback (most recent call last):
File "<input>", line 11, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/util/_decorators.py", line 299, in wrapper
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 336, in read_excel
io = ExcelFile(io, storage_options=storage_options, engine=engine)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 1131, in __init__
self._reader = self._engines[engine](self._io, storage_options=storage_options)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 475, in __init__
super().__init__(filepath_or_buffer, storage_options=storage_options)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 391, in __init__
self.book = self.load_workbook(self.handles.handle)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 486, in load_workbook
return load_workbook(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 317, in load_workbook
reader.read()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 281, in read
apply_stylesheet(self.archive, self.wb)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/styles/stylesheet.py", line 198, in apply_stylesheet
stylesheet = Stylesheet.from_tree(node)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/styles/stylesheet.py", line 103, in from_tree
return super(Stylesheet, cls).from_tree(node)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/descriptors/serialisable.py", line 87, in from_tree
obj = desc.expected_type.from_tree(el)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/descriptors/serialisable.py", line 87, in from_tree
obj = desc.expected_type.from_tree(el)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openpyxl/descriptors/serialisable.py", line 103, in from_tree
return cls(**attrib)
TypeError: __init__() got an unexpected keyword argument 'xfid'
Does anyone know how to fix this? Thanks a lot for your help!

I had the same problem here. After some hours thinking and searching I realized the problem is, actually, the file. I opened it using MS Excel, and save. Alakazan, problem solved.
The file was downloaded, so i think it's a "security" error or just an error from how the file was created. xD
EDIT:
It's not a security problem, but actually an error from the generation of file. The correct has the double of kb the wrong file.
An solution is: if using xlrd==1.2.0 the file can be opened, you can, after doing this, call read_excel to the Book(file opened by xlrd).
import xlrd
# df = pd.read_excel('TabelaPrecos.xlsx')
# The line above is the same result
a = xlrd.open_workbook('TabelaPrecos.xlsx')
b = pd.read_excel(a)

Using Pandas to read excel from url

I am working on a personal project to analyze COVID19 data. Presently, I am download the excel sheet provided by ourworldindata.org, available at this url -> https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-data.xlsx
However, when i try to execute the command in pandas (below), I get a list of errors. What could be the root cause ?
url = 'https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-data.xlsx'
df = pd.read_excel(url, sheet_name='Sheet1')
Error
Traceback (most recent call last): File "<input>", line 1, in <module> File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\pandas\io\excel\_base.py", line 304, in read_excel
io = ExcelFile(io, engine=engine) File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\pandas\io\excel\_base.py", line 824, in __init__
self._reader = self._engines[engine](self._io) File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\pandas\io\excel\_xlrd.py", line 21, in __init__
super().__init__(filepath_or_buffer) File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\pandas\io\excel\_base.py", line 351, in __init__
self.book = self.load_workbook(filepath_or_buffer) File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\pandas\io\excel\_xlrd.py", line 34, in load_workbook
return open_workbook(file_contents=data) File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\xlrd\__init__.py", line 157, in open_workbook
ragged_rows=ragged_rows, File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\xlrd\book.py", line 92, in open_workbook_xls
biff_version = bk.getbof(XL_WORKBOOK_GLOBALS) File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\xlrd\book.py", line 1278, in getbof
bof_error('Expected BOF record; found %r' % self.mem[savpos:savpos+8]) File "C:\Users\masoom.kumar\PycharmProjects\ReadingINCA_Data\venv\lib\site-packages\xlrd\book.py", line 1272, in bof_error
raise XLRDError('Unsupported format, or corrupt file: ' + msg) xlrd.biffh.XLRDError: Unsupported format, or corrupt file: Expected BOF record; found b'\n\n\n\n\n<!D'
Please not that pandas can read the excel if I download it on my computer

Try the link to raw excel file:
import pandas as pd
url='https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-data.xlsx?raw=true'
df=pd.read_excel(url, sheet_name='Sheet1')

You can do it with requests
import pandas as pd
import io
import requests
url = 'https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-data.xlsx'
get_content = requests.get(url).content
df = pd.read_csv(io.StringIO(get_content .decode('utf-8')))
I do this to avoid using local drive or google drive , and saves time of connection.

Why can't I edit my .xlsx file with openpyxl?

I am encountering a problem when running with openpyxl the code below
import openpyxl
import os
wb = openpyxl.load_workbook('example.xlsx')
sheet = wb.get_sheet_by_name('Sheet1')
sheet["A1"].value
sheet["A1"].value == None
sheet["A1"].value = 42
sheet["A3"].value = 'Hello'
os.chdir("/Users/mac/Desktop")
wb.save('exceeeel.xlsx')
The error is
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openpyxl/reader/excel.py", line 312, in load_wo
rkbook
reader = ExcelReader(filename, read_only, keep_vba,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openpyxl/reader/excel.py", line 124, in __init_
_
self.archive = _validate_archive(fn)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/openpyxl/reader/excel.py", line 96, in _validat
e_archive
What am I doing wrong? I am using the current version of the openpyxl library.

I can't provide a confident answer because the question only includes a partial traceback. That being said, it looks like the traceback one would get for a FileNotFoundError:
C:\Python37\python.exe C:/Users/user/PycharmProjects/scratch/scratch2.py
Traceback (most recent call last):
File "C:/Users/user/PycharmProjects/scratch/scratch2.py", line 3, in <module>
wb = openpyxl.load_workbook('example.xlsx')
File "C:\Python37\lib\site-packages\openpyxl\reader\excel.py", line 313, in load_workbook
data_only, keep_links)
File "C:\Python37\lib\site-packages\openpyxl\reader\excel.py", line 124, in __init__
self.archive = _validate_archive(fn)
File "C:\Python37\lib\site-packages\openpyxl\reader\excel.py", line 96, in _validate_archive
archive = ZipFile(filename, 'r')
File "C:\Python37\lib\zipfile.py", line 1207, in __init__
self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: 'example.xlsx'
Process finished with exit code 1
This error will be raised when the path your provided to the file you want to load with openpyxl.load_workbook does not contain the specified file. Since the only argument you provided in your call of that function is 'example.xlsx' that probably means there is no file in the folder you are running this script from.
If this 'example.xlsx' file is in a different folder then you'll want to either specify the relative path to that file as your argument or move the file into the same folder as your script.
If this isn't what's going on then you'll need to provide the full traceback that you are seeing on your end in order to get a better answer.

Pandas sometimes writes empty or damaged files

I've been using pandas for a while and I think it is a great tool. I made a program to generate some excel files from some data collected by the user. The final user have been testing and using it for 6 months; it never failled till yesterday, when it generated a dagamaged excel file. When I opened it with a text editor, it was totally blank. The code to generate this file is this:
escritor = pandas.ExcelWriter(direccion, engine='xlsxwriter')
listaTotal.to_excel(escritor, index = False)
escritor.save()
and:
escritor = pandas.ExcelWriter(direccion + '.xlsx', engine='xlsxwriter')
self.listaFact.to_excel(escritor, index = False, startrow = 1, startcol = 0, sheet_name = 'Hoja1')
escritor.save()
The second code fragment also uses some format options for the 'xlsxwriter', an example here:
format = workbook.add_format()
format.set_font_size(9)
format.set_font_name('Sans Serif 12cpi')
format.set_border()
format.set_text_wrap()
This error happened twice; about 1 month ago and yesterday. I can't duplicate the error, I don't know what happened. And also the traceback is here, it shows the problem when the program reads the file, but this file was generated by the code posted before:
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Python27\lib\lib-tk\Tkinter.py", line 1532, in __call__
return self.func(*args)
File "C:\Users\WINNER\Documents\Visual Studio 2013\Projects\PythonApplication4\PythonApplication4\PythonApplication4.py", line 792, in botonGenerarPedido
self.generarPedido()
File "C:\Users\WINNER\Documents\Visual Studio 2013\Projects\PythonApplication4\PythonApplication4\PythonApplication4.py", line 904, in generarPedido
self.generarVentasDia()
File "C:\Users\WINNER\Documents\Visual Studio 2013\Projects\PythonApplication4\PythonApplication4\PythonApplication4.py", line 927, in generarVentasDia
listaTotal = pandas.io.excel.read_excel(direccion)
File "C:\Python27\lib\site-packages\pandas\io\excel.py", line 151, in read_excel
return ExcelFile(io, engine=engine).parse(sheetname=sheetname, **kwds)
File "C:\Python27\lib\site-packages\pandas\io\excel.py", line 188, in __init__
self.book = xlrd.open_workbook(io)
File "C:\Python27\lib\site-packages\xlrd\__init__.py", line 435, in open_workbook
ragged_rows=ragged_rows,
File "C:\Python27\lib\site-packages\xlrd\book.py", line 91, in open_workbook_xls
biff_version = bk.getbof(XL_WORKBOOK_GLOBALS)
File "C:\Python27\lib\site-packages\xlrd\book.py", line 1230, in getbof
bof_error('Expected BOF record; found %r' % self.mem[savpos:savpos+8])
File "C:\Python27\lib\site-packages\xlrd\book.py", line 1224, in bof_error
raise XLRDError('Unsupported format, or corrupt file: ' + msg)
XLRDError: Unsupported format, or corrupt file: Expected BOF record; found '\x00\x00\x00\x00\x00\x00\x00\x00'