How to implement csvwriter in python on Google App Engine - python

I have an app that works to write a csv as a response to a specific URL call. But, I don't know how to escape characters.
Right now, my code looks like this
import csv
class ReturnCSV(BaseHandler):
def get(self, group_id):
self.response.headers['Content-Type'] = 'text/csv'
self.response.headers['Content-Disposition'] = "attachment; filename=scenarios.csv"
self.write(','.join(header)) #header is a list defined elsewhere
for scenario in list_of_scenarios:
#do stuff
self.write('\r\n' + ','.join(output)))
This gets me the expected output, except that it doesn't escape special characters. I figured I need to implement csv.writer with QUOTE_ALL, but when trying that, I get an IOError(errno.EROFS, 'Read-only file system', filename)
self.response.headers['Content-Type'] = 'text/csv'
self.response.headers['Content-Disposition'] = "attachment; filename=scenarios.csv"
with open('scenarios.csv', 'w') as output:
row = ','.join(header)
wr = csv.writer(output, quoting=csv.QUOTE_ALL)
wr.writerow(row)
for scenario in list_of_scenarios:
#do stuff
wr.writerow(output)

So, got it to work if I don't try to open the file and then I take out all the ','.join that I was doing to manually create CSV.
output = StringIO.StringIO()
row = header #header created elsewhere
wr = csv.writer(output, quoting=csv.QUOTE_MINIMAL)
wr.writerow(row)
for scenario in list_of_scenarios:
# do stuff
wr.writerow(datarow)
self.response.headers['Content-Type'] = 'text/csv'
self.response.headers['Content-Disposition'] = "attachment; filename=scenarios.csv"
self.write(output.getvalue())

Related

Django how to create a tmp excel file and return it to the browser within the response

I have a process to build a tmp file and then return it to the browser in csv. Now i want to do the same but return a excel file.
So what i have for the csv is a view in django that does:
def export_wallet_view(request):
tmp = tempfile.NamedTemporaryFile(delete=False)
with open(tmp.name, 'w', encoding="utf-8-sig") as fi:
csv_headers = [
'Id',
'Name'
]
fi.write(';'.join(csv_headers))
fi.write('\n')
//here also i save the rows into the file
response = FileResponse(open(tmp.name, 'rb'))
response['Content-Disposition'] = 'attachment; filename="wallet.csv"'
return response
So to convert it to excel i try to do something like this using pandas:
df = pd.read_csv(tmp.name)
df.to_excel('pandas_to_excel.xlsx', sheet_name='new_sheet_name')
The problem is that this creates the excel in the server, and i would like to do something like:
df = pd.read_csv(tmp.name)
df.to_excel('pandas_to_excel.xlsx', sheet_name='new_sheet_name') //this being a tmp file
response = FileResponse(open(tmp.name, 'rb')) //this should be the new excel tmp file
response['Content-Disposition'] = 'attachment; filename="wallet.csv"'
return response
Thanks
I don't understand your problem.
You should use the same 'pandas_to_excel.xlsx' in both
df.to_excel('pandas_to_excel.xlsx', ...)
... open('pandas_to_excel.xlsx', 'rb')
or the same tmp.name in both
df.to_excel(tmp.name, ...)
... open(tmp.name, 'rb')
You can even use again NamedTemporaryFile() to create new temporary name.
tmp = tempfile.NamedTemporaryFile(delete=False)
df.to_excel(tmp.name, ...)
... open(tmp.name, 'rb')
But popular method is to use io.String() or io.Bytes() to create file-like object in memory - without creating file on disk.
def export_wallet_view(request):
csv_headers = ['Id', 'Name']
file_like_object = io.Bytes()
file_like_object.write(';'.join(csv_headers).encode('utf-8-sig'))
file_like_object.write('\n'.encode('utf-8-sig'))
file_like_object.write('other rows'.encode('utf-8-sig'))
file_like_object.seek(0) # move to the beginning of file
response = FileResponse(file_like_object)
response['Content-Disposition'] = 'attachment; filename="wallet.csv"'
return response
For excel it could be something like this. I use io.String() to read csv directly to pandas, and later I use io.Bytes() to create file-like object with excel data.
def export_wallet_view(request):
csv_headers = ['Id', 'Name']
text = ';'.join(csv_headers)
text += '\n'
text += 'other rows'
df = pd.read_csv(io.String(text))
file_like_object = io.Bytes()
df.to_excel(file_like_object)
file_like_object.seek(0) # move to the beginning of file
response = FileResponse(file_like_object)
response['Content-Disposition'] = 'attachment; filename="pandas_to_excel.xlsx"'
return response

How can I edit my code to print out the content of my created json file?

My program takes a csv file as input and writes it as an output file in json format. On the final line, I use the print command to output the contents of the json format file to the screen. However, it does not print out the json file contents and I don't understand why.
Here is my code that I have so far:
import csv
import json
def jsonformat(infile,outfile):
contents = {}
csvfile = open(infile, 'r')
reader = csvfile.read()
for m in reader:
key = m['No']
contents[key] = m
jsonfile = open(outfile, 'w')
jsonfile.write(json.dumps(contents))
csvfile.close()
jsonfile.close()
return jsonfile
infile = 'orders.csv'
outfile = 'orders.json'
output = jsonformat(infile,outfile)
print(output)
Your function returns the jsonfile variable, which is a file.
Try adding this:
jsonfile.close()
with open(outfile, 'r') as file:
return file.read()
Your function returns a file handle to the file jsonfile that you then print. Instead, return the contents that you wrote to that file. Since you opened the file in w mode, any previous contents are removed before writing the new contents, so the contents of your file are going to be whatever you just wrote to it.
In your function, do:
def jsonformat(infile,outfile):
...
# Instead of this:
# jsonfile.write(json.dumps(contents))
# do this:
json_contents = json.dumps(contents, indent=4) # indent=4 to pretty-print
jsonfile.write(json_contents)
...
return json_contents
Aside from that, you aren't reading the CSV file the correct way. If your file has a header, you can use csv.DictReader to read each row as a dictionary. Then, you'll be able to use for m in reader: key = m['No']. Change reader = csvfile.read() to reader = csv.DictReader(csvfile)
As of now, reader is a string that contains all the contents of your file. for m in reader makes m each character in this string, and you cannot access the "No" key on a character.
a_file = open("sample.json", "r")
a_json = json.load(a_file)
pretty_json = json.dumps(a_json, indent=4)
a_file.close()
print(pretty_json)
Using this sample to print the contents of your json file. Have a good day.

Within a file to add data

I am trying to attempt something that I have not before within python.
The code below collects data from my test database and put it into a text under my headers of 'Test1','Test2','Test3'. This is working fine.
What I am trying to attempt now is to add a header (on top of the current header) and footer to the file.
python code:
file = 'file.txt'
header_names = {'t1':'Test1', 't2': 'Test2','t3':'Test3'}
with open(file, 'w', newline='') as f:
w = csv.DictWriter(f, fieldnames=header_names.keys(), restval='', extrasaction='ignore')
w.writerow(header_names)
for doc in res['test']['test']:
my_dict = doc['test']
w.writerow(my_dict)
current file output using the above code.
file.txt
Test1,Test2,Test3
Bob,john,Male
Cat,Long,female
Dog,Short,Male
Case,Fast,Male
Nice,who,Male
ideal txt output.
{header}
Filename:file.txt
date:
{data}
Test1,Test2,Test3
Bob,john,Male
Cat,Long,female
Dog,Short,Male
Case,Fast,Male
Nice,who,Male
{Footer}
this file was generated by using python.
the {header}, {data} and {footer} is not needed within the file that is just to make clear what is needed. i hope this makes sense.
Something like this
import csv
from datetime import date
# prepare some sample data
data = [['Bob', 'John', 'Male'],
['Cat', 'Long', 'Female']]
fieldnames = ['test1', 'test2', 'test3']
data = [dict(zip(fieldnames, row)) for row in data]
# actual part that writes to a file
with open('spam.txt', 'w', newline='') as f:
f.write('filename:spam.txt\n')
f.write(f'date:{date.today().strftime("%Y%m%d")}\n\n')
wrtr = csv.DictWriter(f, fieldnames = fieldnames)
wrtr.writeheader()
wrtr.writerows(data)
f.write('\nwritten with python\n')
Output in the file:
filename:spam.txt
date:20190321
test1,test2,test3
Bob,John,Male
Cat,Long,Female
written with python
Now, all that said, do you really need to write header and footer. It will just break a nicely formatted csv file and would require extra effort later on when reading it.
Or if you prefer - is the csv format what best suits your needs? Maybe using json would be better...
vardate= datetime.datetime.now().strftime("%x")
file = 'file.txt'
header_names = {'t1':'Test1', 't2': 'Test2','t3':'Test3'}
with open(file, 'w', newline='') as f:
f.seek(0,0) //This will move cursor to start position of file
f.writelines("File Name: ", file)
f.writelines("date: ", vardate)
f.writelines(".Try out next..")
w = csv.DictWriter(f, fieldnames=header_names.keys(), restval='',
extrasaction='ignore')
w.writerow(header_names)
for doc in res['test']['test']:
my_dict = doc['test']
w.writerow(my_dict)
f.seek(0,2)
f.writelines("This is generated using Python")

Write to specific columns in a csv file

I've a script that gets data in a json format through an API. I need to write headers to the file, then append with some of the json data.
import requests, csv, json
def test(limit="", api_token=""):
url_rack = "url"
api_token = "Token " + api_token
headers = {'Authorization': api_token}
params = {'limit': limit}
session = requests.Session()
json_data = session.get(url_rack, headers=headers, params=params)
with open('my_file.csv', 'wb') as csvfile:
filewriter = csv.writer(csvfile, delimiter=',',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
filewriter.writerow(['Header1', 'Header2'])
json_data = json_data.json()['results']
for i in json_data:
bunit = i['tenant']
open(csvfile)
if not bunit:
print 'Placeholder'
else:
bunit = i['tenant']['name']
print bunit
for i in json_data:
owner = i['tenant']
if not owner:
print 'Placeholder Owner'
else:
owner = i['tenant']['name']
print owner
def main():
api_token = "mytoken"
test(api_token=api_token)
if __name__ == "__main__":
main()
I tried to write the data taken from for i in tenants: in the column under Header1, and the data from for i in owners under Header2. I have two trivial problems. When I tried to open the csv file inside the for loop I got the error TypeError: coercing to Unicode: need string or buffer, file found. I got that the file is open so I can't reopen it, but don't want to create it in the loop as that would rewrite the file each iteration?
Secondly filewriter.writerow obviously writes in rows, but I need to write the data in each column under the header. I printed it out to test that I can actually call the api and get the data but can't work out how to append it under Header1 and Header2.
EDIT
My data that prints:
Placeholder tenant
Placeholder tenant
Tenant1
Placeholder tenant
Placeholder tenant
Placeholder tenant
Placeholder tenant
Tenant2
Placeholder tenant
Tenant3
Tenant4
and
Placeholder Owner
Placeholder Owner
Owner1
Placeholder Owner
Placeholder Owner
Placeholder Owner
Placeholder Owner
Owner2
Placeholder Owner
Owner3
Owner4
Why not to rewrite your code like this:
...
with open('my_file.csv', 'wb') as csvfile:
filewriter = csv.writer(csvfile, delimiter=',',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
filewriter.writerow(['Header1', 'Header2'])
json_data = json_data.json()['results']
for i in json_data:
bunit = i['tenant']
if not bunit:
print 'Placeholder'
else:
bunit = i['tenant']['name']
owner = i['tenant']
if not owner:
print 'Placeholder Owner'
else:
owner = i['tenant']['name']
filewriter.writerow([bunit, owner])
...
To open() function, you need to provide a filename as parameter.
Here you are providing an already open file as a parameter. Hence, first error.
The csvfile is already open and you do not need to open it again to write to it.
To write to the file, you can store the data in a list and then write to file at end of function.
Suppose I want to write r1c1,r1c2,r2c1,r2c2,r3c1,r3c2 in cells-A1,B1,A2,B2,A3,B3 respectively, then I would do something like the following:-
listtowrite = [['r1c1','r1c2'],['r2c1','r2c2'],['r3c1','r3c2']]
csvfile = open('my_file.csv', 'w')
for arow in listtowrite:
for acolumn in arow:
csvfile.write(acolumn+',')
csvfile.write('\n')
csvfile.close()
Figured it out. Just needed to write each to a list
json_data = prefixes_response.json()['results']
tenant = []
for i in json_data:
bunit = i['tenant']
if not bunit:
tenant.append('Placeholder tenant')
else:
bunit = i['tenant']['name']
tenant.append(bunit)
and zip both lists and write
rows = zip(Header1, Header2)
for row in rows:
filewriter.writerow(row)

How to skip a column when writing to a csv file in python

Sorry if this has been asked, but is it possible to skip a column when writing to a csv file?
Here is the code I have:
with open("list.csv","r") as f:
reader2 = csv.reader(f)
for row in reader2:
url = 'http://peopleus.intelius.com/results.php?ReportType=33&qi=0&qk=10&qp='+row
req = urllib.request.Request(url)
response = urllib.request.urlopen(req)
html = response.read()
retrieved_name = b'class="singleName">(.*?)<\/h1'
retrieved_number = b'<div\sclass="phone">(.*?)<\/div'
retrieved_nothing = b"(Sorry\swe\scouldn\\'t\sfind\sany\sresults)"
if re.search(retrieved_nothing,html):
noth = re.search(retrieved_nothing.decode('utf-8'),html.decode('utf-8')).group(1)
add_list(phone_data, noth)
else:
if re.search(retrieved_name,html):
name_found = re.search(retrieved_name.decode('utf-8'),html.decode('utf-8')).group(1)
else:
name_found = "No name found on peopleus.intelius.com"
if re.search(retrieved_number,html):
number_found = re.search(retrieved_number.decode('utf-8'),html.decode('utf-8')).group(1)
else:
number_found = "No number found on peopleus.intelius.com"
add_list(phone_data, name_found, number_found)
with open('column_skip.csv','a+', newline='') as mess:
writ = csv.writer(mess, dialect='excel')
writ.writerow(phone_data[-1])
time.sleep(10)
Assuming that there is data in the first three rows of column_skip.csv, can I have my program start writing its info in column 4?
Yeah, don't use csv.writer method and write it as an simple file write operation:
`file_path ='your_csv_file.csv'
with open(file_path, 'w') as fp:
#following are the data you want to write to csv
fp.write("%s, %s, %s" % ('Name of col1', 'col2', 'col4'))
fp.write("\n")`
I hope this helps...

Categories