I have to read an online csv-file into a postgres database, and in that context I have some problems reading the online csv-file properly.
If I just import the file it reads as bytes, so I have to decode it. During the decoding it, however, seems that the entire file is turned into one long string.
# Libraries
import csv
import urllib.request
# Function for importing csv from url
def csv_import(url):
url_open = urllib.request.urlopen(url)
csvfile = csv.reader(url_open.decode('utf-8'), delimiter=',')
return csvfile;
# Reading file
p_pladser = csv_import("http://wfs-kbhkort.kk.dk/k101/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=k101:p_pladser&outputFormat=csv&SRSNAME=EPSG:4326")
When I try to read the imported file line by line it only reads one character at the time.
for row in p_pladser:
print(row)
break
['F']
Can you help me identify where it goes wrong? I am using Python 3.6.
EDIT: Per request my solution in R
# Loading library
library(RPostgreSQL)
# Reading dataframe
p_pladser = read.csv("http://wfs-kbhkort.kk.dk/k101/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=k101:p_pladser&outputFormat=csv&SRSNAME=EPSG:4326", encoding = "UTF-8", stringsAsFactors = FALSE)
# Creating database connection
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, dbname = "secretdatabase", host = "secrethost", user = "secretuser", password = "secretpassword")
# Uploading dataframe to postgres database
dbWriteTable(con, "p_pladser", p_pladser , append = TRUE, row.names = FALSE, encoding = "UTF-8")
I have to upload several tables for 10,000 to 100,000 rows, and it total in R it takes 1-2 seconds to upload them all.
csv.reader expect as argument a file like object and not a string. You have 2 options here:
either you read the data into a string (as you currently do) and then use a io.StringIO to build a file like object around that string:
def csv_import(url):
url_open = urllib.request.urlopen(url)
csvfile = csv.reader(io.StringIO(url_open.read().decode('utf-8')), delimiter=',')
return csvfile;
or you use a io.TextIOWrapper around the binary stream provided by urllib.request:
def csv_import(url):
url_open = urllib.request.urlopen(url)
csvfile = csv.reader(io.TextIOWrapper(url_open, encoding = 'utf-8'), delimiter=',')
return csvfile;
How about loading the CSV with pandas!
import pandas as pd
csv = pd.read_csv("http://wfs-kbhkort.kk.dk/k101/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=k101:p_pladser&outputFormat=csv&SRSNAME=EPSG:4326")
print csv.columns
OR if you have the CSV downloaded in your machine, then directly
csv = pd.read_csv("<path_to_csv>")
Ok! You may consider passing delimiter and quotechar arguments to csv.reader, because the CSV contains quotes as well! Something like this,
with open('p_pladser.csv') as f:
rows = csv.reader(f, delimiter=',', quotechar='"')
for row in rows:
print(row)
I'm automating a long task that involves vulnerabilities within a spreadsheet. However, I'm noticing that the "recommendation" for these vulnerabilities are sometimes pretty long.
The CSV module for python seems to be truncating some of this text when writing new rows. Is there any way to prevent this from happening? I simply see "NOTE: THIS FIELD WAS TRUNCATED" in places where the recommendation (which is a lot of text) would be.
The whole objective is to do this:
Import a master spreadsheet which has confirmation statuses and everything up-to-date
Take a new spreadsheet containing vulnerabilities which doesn't have conf status/severity up-to-date.
Compare the second spreadsheet to the first. It'll update the severity levels from the second spreadsheet, and then write to a new file.
Newly created csv file can be copied and pasted into master spreadsheet. All vulnerabilities which match the first spreadsheet now have the same severity level/confirmation status.
What I'm noticing though, even in Ruby for some reason, is that some of the recommendations in these vulnerabilities have long text; therefore, it's being truncated when the CSV file is created for some reason. Here's a sample piece of the code that I've quickly written for demonstration:
#!/usr/bin/python
from sys import argv
import getopt, csv
master_vulns = {}
criticality = {}
############################ Extracting unique vulnerabilities from master file
contents = csv.reader(open(argv[1], 'rb'), delimiter=',')
for row in contents:
if "Confirmation_Status" in row:
continue
try:
if row[7] in master_vulns:
continue
if row[7] in master_vulns:
continue
master_vulns[row[7]] = row[3]
criticality[rows[7]] = row[2]
except Exception:
pass
############################ Updating confirmation status of newly created file
new_contents = csv.reader(open(argv[1], 'rb'), delimiter=',')
new_data = []
results = open('results.csv','wb')
writer = csv.writer(results, delimiter=',')
for nrow in new_contents:
if "Confirmation_Status" in nrow:
continue
try:
if nrow[1] == "DELETE":
continue
vuln_name = nrow[7]
vuln_status = nrow[3]
criticality = criticality[vuln_name]
vuln_status = master_vulns[vuln_name]
nrow[3] = vuln_status
nrow[2] = criticality
writer.writerow(nrow)
except Exception:
writer.writerow(nrow)
pass
results.close()
views.py
def export_to_excel(request):
lists = MyModel.objects.all()
# your excel html format
template_name = "sample_excel_format.html"
response = render_to_response(template_name, {'lists': lists})
# this is the output file
filename = "model.csv"
response['Content-Disposition'] = 'attachment; filename='+filename
response['Content-Type'] = 'application/vnd.ms-excel; charset=utf-16'
return response
urls.py
from django.conf.urls.defaults import *
urlpatterns = patterns('app_name.views',
url(r'^export/$', 'export_to_excel', name='export_to_excel'),
)
Last, in your page create a button or link that will point in exporting.
page.html
Export
Nothing getting file option for download and not giving any error but i can see all result in log its working fine.
I think that the solution to export excel files is :
if 'excel' in request.POST:
response = HttpResponse(content_type='application/vnd.ms-excel')
response['Content-Disposition'] = 'attachment; filename=Report.xlsx'
xlsx_data = WriteToExcel(weather_period, town)
response.write(xlsx_data)
return response
In this example the library used for exporting is xlsxWriter.
Here is a very complete and practical solution for this, and many others: http://assist-software.net/blog/how-export-excel-files-python-django-application .
In addition to the options shown in the other answers you can also use XlsxWriter to create Excel files.
See this example.
It seems that you are trying to generate an excel workbook with HTML content. I don't know if Excel (or LibreOffice) is able to open such file but I think it is not the right approach.
You should fist generate a excel file : you can use csv, xlwt for xls and openpyxl for xlsx
The content of the file can be passed to the HttpResponse
for example, if you work with xlwt:
import xlwt
wb = xlwt.Workbook()
#use xlwt to fill the workbook
#
#ws = wb.add_sheet("sheet")
#ws.write(0, 0, "something")
response = HttpResponse(mimetype='application/vnd.ms-excel')
response['Content-Disposition'] = 'attachment; filename=the-file.xls'
wb.save(response)
return response
You can also look at
django-excel-response which does all the work for you. (I think it doesn't support xlsx format)
django-excel-export
I hope it helps
This seems to be based on the practice of tricking excel into opening an HTML table by changing the file name and MIME type. In order to make this work, the HTML file has to assemble an HTML table, and this is likely to trigger a warning that the real content of the file is different from the declared content.
IMHO it is a crude hack and should be avoided. Instead you can create a real excel file using the xlwt module, or you can create a real CSV file using the csv module.
[update]
After looking the blog post you refered, I see it is recommending another bad practice: using anything but the csv module to produce CSV files is dangerous because if the data contains the delimiter character, quotes or line breaks, you may end up with a bad CSV.
The csv module will take care of all corner cases and produce a proper formatted output.
I've seen people use a Django template naming the file "something.xls" and using HTML tables instead of the CSV format, but this has some corner cases as well.
Export Data to XLS File
Use it if you really need to export to a .xls file. You will be able to add formating as bold font, font size, define column size, etc.
First of all, install the xlwt module. The easiest way is to use pip.
pip install xlwt
views.py
import xlwt
from django.http import HttpResponse
from django.contrib.auth.models import User
def export_users_xls(request):
response = HttpResponse(content_type='application/ms-excel')
response['Content-Disposition'] = 'attachment; filename="users.xls"'
wb = xlwt.Workbook(encoding='utf-8')
ws = wb.add_sheet('Users')
# Sheet header, first row
row_num = 0
font_style = xlwt.XFStyle()
font_style.font.bold = True
columns = ['Username', 'First name', 'Last name', 'Email address', ]
for col_num in range(len(columns)):
ws.write(row_num, col_num, columns[col_num], font_style)
# Sheet body, remaining rows
font_style = xlwt.XFStyle()
rows = User.objects.all().values_list('username', 'first_name', 'last_name', 'email')
for row in rows:
row_num += 1
for col_num in range(len(row)):
ws.write(row_num, col_num, row[col_num], font_style)
wb.save(response)
return response
urls.py
import views
urlpatterns = [
...
url(r'^export/xls/$', views.export_users_xls, name='export_users_xls'),
]
template.html
Export all users
Learn more about the xlwt module reading its official documentation. https://simpleisbetterthancomplex.com/tutorial/2016/07/29/how-to-export-to-excel.html
I am using win32.client in python for converting my .xlsx and .xls file into a .csv. When I execute this code it's giving an error. My code is:
def convertXLS2CSV(aFile):
'''converts a MS Excel file to csv w/ the same name in the same directory'''
print "------ beginning to convert XLS to CSV ------"
try:
import win32com.client, os
from win32com.client import constants as c
excel = win32com.client.Dispatch('Excel.Application')
fileDir, fileName = os.path.split(aFile)
nameOnly = os.path.splitext(fileName)
newName = nameOnly[0] + ".csv"
outCSV = os.path.join(fileDir, newName)
workbook = excel.Workbooks.Open(aFile)
workbook.SaveAs(outCSV, c.xlCSVMSDOS) # 24 represents xlCSVMSDOS
workbook.Close(False)
excel.Quit()
del excel
print "...Converted " + nameOnly + " to CSV"
except:
print ">>>>>>> FAILED to convert " + aFile + " to CSV!"
convertXLS2CSV("G:\\hello.xlsx")
I am not able to find the error in this code. Please help.
I would use xlrd - it's faster, cross platform and works directly with the file.
As of version 0.8.0, xlrd reads both XLS and XLSX files.
But as of version 2.0.0, support was reduced back to only XLS.
import xlrd
import csv
def csv_from_excel():
wb = xlrd.open_workbook('your_workbook.xls')
sh = wb.sheet_by_name('Sheet1')
your_csv_file = open('your_csv_file.csv', 'wb')
wr = csv.writer(your_csv_file, quoting=csv.QUOTE_ALL)
for rownum in xrange(sh.nrows):
wr.writerow(sh.row_values(rownum))
your_csv_file.close()
I would use pandas. The computationally heavy parts are written in cython or c-extensions to speed up the process and the syntax is very clean. For example, if you want to turn "Sheet1" from the file "your_workbook.xls" into the file "your_csv.csv", you just use the top-level function read_excel and the method to_csv from the DataFrame class as follows:
import pandas as pd
data_xls = pd.read_excel('your_workbook.xls', 'Sheet1', index_col=None)
data_xls.to_csv('your_csv.csv', encoding='utf-8')
Setting encoding='utf-8' alleviates the UnicodeEncodeError mentioned in other answers.
Maybe someone find this ready-to-use piece of code useful. It allows to create CSVs from all spreadsheets in Excel's workbook.
Python 2:
# -*- coding: utf-8 -*-
import xlrd
import csv
from os import sys
def csv_from_excel(excel_file):
workbook = xlrd.open_workbook(excel_file)
all_worksheets = workbook.sheet_names()
for worksheet_name in all_worksheets:
worksheet = workbook.sheet_by_name(worksheet_name)
with open(u'{}.csv'.format(worksheet_name), 'wb') as your_csv_file:
wr = csv.writer(your_csv_file, quoting=csv.QUOTE_ALL)
for rownum in xrange(worksheet.nrows):
wr.writerow([unicode(entry).encode("utf-8") for entry in worksheet.row_values(rownum)])
if __name__ == "__main__":
csv_from_excel(sys.argv[1])
Python 3:
import xlrd
import csv
from os import sys
def csv_from_excel(excel_file):
workbook = xlrd.open_workbook(excel_file)
all_worksheets = workbook.sheet_names()
for worksheet_name in all_worksheets:
worksheet = workbook.sheet_by_name(worksheet_name)
with open(u'{}.csv'.format(worksheet_name), 'w', encoding="utf-8") as your_csv_file:
wr = csv.writer(your_csv_file, quoting=csv.QUOTE_ALL)
for rownum in range(worksheet.nrows):
wr.writerow(worksheet.row_values(rownum))
if __name__ == "__main__":
csv_from_excel(sys.argv[1])
I'd use csvkit, which uses xlrd (for xls) and openpyxl (for xlsx) to convert just about any tabular data to csv.
Once installed, with its dependencies, it's a matter of:
python in2csv myfile > myoutput.csv
It takes care of all the format detection issues, so you can pass it just about any tabular data source. It's cross-platform too (no win32 dependency).
First read your excel spreadsheet into pandas, below code will import your excel spreadsheet into pandas as a OrderedDict type which contain all of your worksheet as dataframes. Then simply use worksheet_name as a key to access specific worksheet as a dataframe and save only required worksheet as csv file by using df.to_csv(). Hope this will workout in your case.
import pandas as pd
df = pd.read_excel('YourExcel.xlsx', sheet_name=None)
df['worksheet_name'].to_csv('YourCsv.csv')
If your Excel file contain only one worksheet then simply use below code:
import pandas as pd
df = pd.read_excel('YourExcel.xlsx')
df.to_csv('YourCsv.csv')
If someone want to convert all the excel worksheets from single excel workbook to the different csv files, try below code:
import pandas as pd
def excelTOcsv(filename):
df = pd.read_excel(filename, sheet_name=None)
for key, value in df.items():
return df[key].to_csv('%s.csv' %key)
This function is working as a multiple Excel sheet of same excel workbook to multiple csv file converter. Where key is the sheet name and value is the content inside sheet.
#andi I tested your code, it works great, BUT
In my sheets there's a column like this
2013-03-06T04:00:00
date and time in the same cell
It gets garbled during exportation, it's like this in the exported file
41275.0416667
other columns are ok.
csvkit, on the other side, does ok with that column but only exports ONE sheet, and my files have many.
xlsx2csv is faster than pandas and xlrd.
xlsx2csv -s 0 crunchbase_monthly_.xlsx cruchbase
excel file usually comes with n sheetname.
-s is sheetname index.
then, cruchbase folder will be created, each sheet belongs to xlsx will be converted to a single csv.
p.s. csvkit is awesome too.
Quoting an answer from Scott Ming, which works with workbook containing multiple sheets:
Here is a python script getsheets.py (mirror), you should install pandas and xlrd before you use it.
Run this:
pip3 install pandas xlrd # or `pip install pandas xlrd`
How does it works?
$ python3 getsheets.py -h
Usage: getsheets.py [OPTIONS] INPUTFILE
Convert a Excel file with multiple sheets to several file with one sheet.
Examples:
getsheets filename
getsheets filename -f csv
Options:
-f, --format [xlsx|csv] Default xlsx.
-h, --help Show this message and exit.
Convert to several xlsx:
$ python3 getsheets.py goods_temp.xlsx
Sheet.xlsx Done!
Sheet1.xlsx Done!
All Done!
Convert to several csv:
$ python3 getsheets.py goods_temp.xlsx -f csv
Sheet.csv Done!
Sheet1.csv Done!
All Done!
getsheets.py:
# -*- coding: utf-8 -*-
import click
import os
import pandas as pd
def file_split(file):
s = file.split('.')
name = '.'.join(s[:-1]) # get directory name
return name
def getsheets(inputfile, fileformat):
name = file_split(inputfile)
try:
os.makedirs(name)
except:
pass
df1 = pd.ExcelFile(inputfile)
for x in df1.sheet_names:
print(x + '.' + fileformat, 'Done!')
df2 = pd.read_excel(inputfile, sheetname=x)
filename = os.path.join(name, x + '.' + fileformat)
if fileformat == 'csv':
df2.to_csv(filename, index=False)
else:
df2.to_excel(filename, index=False)
print('\nAll Done!')
CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'])
#click.command(context_settings=CONTEXT_SETTINGS)
#click.argument('inputfile')
#click.option('-f', '--format', type=click.Choice([
'xlsx', 'csv']), default='xlsx', help='Default xlsx.')
def cli(inputfile, format):
'''Convert a Excel file with multiple sheets to several file with one sheet.
Examples:
\b
getsheets filename
\b
getsheets filename -f csv
'''
if format == 'csv':
getsheets(inputfile, 'csv')
else:
getsheets(inputfile, 'xlsx')
cli()
We can use Pandas lib of Python to conevert xls file to csv file
Below code will convert xls file to csv file .
import pandas as pd
Read Excel File from Local Path :
df = pd.read_excel("C:/Users/IBM_ADMIN/BU GPA Scorecard.xlsx",sheetname=1)
Trim Spaces present on columns :
df.columns = df.columns.str.strip()
Send Data frame to CSV file which will be pipe symbol delimted and without Index :
df.to_csv("C:/Users/IBM_ADMIN/BU GPA Scorecard csv.csv",sep="|",index=False)
Python is not the best tool for this task. I tried several approaches in Python but none of them work 100% (e.g. 10% converts to 0.1, or column types are messed up, etc). The right tool here is PowerShell, because it is an MS product (as is Excel) and has the best integration.
Simply download this PowerShell script, edit line 47 to enter the path for the folder containing the Excel files and run the script using PowerShell.
Using xlrd is a flawed way to do this, because you lose the Date Formats in Excel.
My use case is the following.
Take an Excel File with more than one sheet and convert each one into a file of its own.
I have done this using the xlsx2csv library and calling this using a subprocess.
import csv
import sys, os, json, re, time
import subprocess
def csv_from_excel(fname):
subprocess.Popen(["xlsx2csv " + fname + " --all -d '|' -i -p "
"'<New Sheet>' > " + 'test.csv'], shell=True)
return
lstSheets = csv_from_excel(sys.argv[1])
time.sleep(3) # system needs to wait a second to recognize the file was written
with open('[YOUR PATH]/test.csv') as f:
lines = f.readlines()
firstSheet = True
for line in lines:
if line.startswith('<New Sheet>'):
if firstSheet:
sh_2_fname = line.replace('<New Sheet>', '').strip().replace(' - ', '_').replace(' ','_')
print(sh_2_fname)
sh2f = open(sh_2_fname+".csv", "w")
firstSheet = False
else:
sh2f.close()
sh_2_fname = line.replace('<New Sheet>', '').strip().replace(' - ', '_').replace(' ','_')
print(sh_2_fname)
sh2f = open(sh_2_fname+".csv", "w")
else:
sh2f.write(line)
sh2f.close()
I've tested all anwers, but they were all too slow for me. If you have Excel installed you can use the COM.
I thought initially it would be slower since it will load everything for the actual Excel application, but it isn't for huge files. Maybe because the algorithm for opening and saving files runs a heavily optimized compiled code, Microsoft guys make a lot of money for it after all.
import sys
import os
import glob
from win32com.client import Dispatch
def main(path):
excel = Dispatch("Excel.Application")
if is_full_path(path):
process_file(excel, path)
else:
files = glob.glob(path)
for file_path in files:
process_file(excel, file_path)
excel.Quit()
def process_file(excel, path):
fullpath = os.path.abspath(path)
full_csv_path = os.path.splitext(fullpath)[0] + '.csv'
workbook = excel.Workbooks.Open(fullpath)
workbook.Worksheets(1).SaveAs(full_csv_path, 6)
workbook.Saved = 1
workbook.Close()
def is_full_path(path):
return path.find(":") > -1
if __name__ == '__main__':
main(sys.argv[1])
This is very raw code and won't check for errors, print help or anything, it will just create a csv file for each file that matches the pattern you entered in the function so you can batch process a lot of files only launching excel application once.
As much as I hate to rely on Windows Excel proprietary software, which is not cross-platform, my testing of csvkit for .xls, which uses xlrd under the hood, failed to correctly parse dates (even when using the commandline parameters to specify strptime format).
For example, this xls file, when parsed with csvkit, will convert cell G1 of 12/31/2002 to 37621, whereas when converted to csv via excel -> save_as (using below) cell G1 will be "December 31, 2002".
import re
import os
from win32com.client import Dispatch
xlCSVMSDOS = 24
class CsvConverter(object):
def __init__(self, *, input_dir, output_dir):
self._excel = None
self.input_dir = input_dir
self.output_dir = output_dir
if not os.path.isdir(self.output_dir):
os.makedirs(self.output_dir)
def isSheetEmpty(self, sheet):
# https://archive.is/RuxR7
# WorksheetFunction.CountA(ActiveSheet.UsedRange) = 0 And ActiveSheet.Shapes.Count = 0
return \
(not self._excel.WorksheetFunction.CountA(sheet.UsedRange)) \
and \
(not sheet.Shapes.Count)
def getNonEmptySheets(self, wb, as_name=False):
return [ \
(sheet.Name if as_name else sheet) \
for sheet in wb.Sheets \
if not self.isSheetEmpty(sheet) \
]
def saveWorkbookAsCsv(self, wb, csv_path):
non_empty_sheet_names = self.getNonEmptySheets(wb, as_name=True)
assert (len(non_empty_sheet_names) == 1), \
"Expected exactly 1 sheet but found %i non-empty sheets: '%s'" \
%(
len(non_empty_sheet_names),
"', '".join(name.replace("'", r"\'") for name in non_empty_sheet_names)
)
wb.Worksheets(non_empty_sheet_names[0]).SaveAs(csv_path, xlCSVMSDOS)
wb.Saved = 1
def isXlsFilename(self, filename):
return bool(re.search(r'(?i)\.xls$', filename))
def batchConvertXlsToCsv(self):
xls_names = tuple( filename for filename in next(os.walk(self.input_dir))[2] if self.isXlsFilename(filename) )
self._excel = Dispatch('Excel.Application')
try:
for xls_name in xls_names:
csv_path = os.path.join(self.output_dir, '%s.csv' %os.path.splitext(xls_name)[0])
if not os.path.isfile(csv_path):
workbook = self._excel.Workbooks.Open(os.path.join(self.input_dir, xls_name))
try:
self.saveWorkbookAsCsv(workbook, csv_path)
finally:
workbook.Close()
finally:
if not len(self._excel.Workbooks):
self._excel.Quit()
self._excel = None
if __name__ == '__main__':
self = CsvConverter(
input_dir='C:\\data\\xls\\',
output_dir='C:\\data\\csv\\'
)
self.batchConvertXlsToCsv()
The above will take an input_dir containing .xls and output them to output_dir as .csv -- it will assert that there is exactly 1 non-empty sheet in the .xls; if you need to handle multiple sheets into multiple csv then you'll need to edit saveWorkbookAsCsv.
I was trying to use xlrd library in order to convert the format xlsx into csv, but I was getting error: xlrd.biffh.XLRDError: Excel xlsx file; not supported. That was happening because this package is no longer reading any other format unless xls, according to xlrd documentation.
Following the answer from Chris Withers I was able to change the engine for the function read_excel() from pandas, then I was able to a create a function that is converting any sheet from your Excel spreadsheet you want to successfully.
In order to work the function below, don't forget to install the openpyxl library from here.
Function:
import os
import pathlib
import pandas as pd
# Function to convert excel spreadsheet into csv format
def Excel_to_csv():
# Excel file full path
excel_file = os.path.join(os.path.sep, pathlib.Path(__file__).parent.resolve(), "Excel_Spreadsheet.xlsx")
# Excel sheets
excel_sheets = ['Sheet1', 'Sheet2', 'Sheet3']
for sheet in excel_sheets:
# Create dataframe for each sheet
df = pd.DataFrame(pd.read_excel(excel_file, sheet, index_col=None, engine='openpyxl'))
# Export to csv. i.e: sheet_name.csv
df.to_csv(os.path.join(os.path.sep, pathlib.Path(__file__).parent.resolve(), sheet + '.csv'), sep=",", encoding='utf-8', index=False, header=True)
# Runs the excel_to_csv function:
Excel_to_csv()