Suppose the Flask-Admin view below (note I'm using flask_wtf not wtforms). I'd like to upload a csv file, and then on the model_change, parse the csv and do some stuff to it before returning the result which will then be stored into the model. However, I get the error: TypeError: coercing to Unicode: need string or buffer, FileField found
from flask_wtf.file import FileField
class myView(ModelView):
[...]
def scaffold_form(self):
form_class = super(myView, self).scaffold_form()
form_class.csv = FileField('Upload CSV')
return form_class
def on_model_change(self, form, model):
csv = form.csv
csv_data = self.parse_file(csv)
model.csv_data = csv_data
def parse_file(self, csv):
with open(csv, 'rb') as csvfile:
data = csv.reader(csvfile, delimiter=',')
for row in data:
doSomething()
When accessing csv.data, I'll get <FileStorage: u'samplefile.csv' ('text/csv')> but this object doesn't actually contain the csv's data.
Okay, after digging further into the flask_wtf module I was able to find enough to go on and get a workaround. The FileField object has a data attribute that wraps the werkzeug.datastructures.FileStorage class which exposes the stream attribute. The docs say this typically points to the open file resource, but since I'm doing this in memory in this case it's a stream buffer io.BytesIO object.
Attempting to open():
with open(field.data.stream, 'rU') as csv_data:
[...]
Will result in an TypeError: coercing to Unicode: need string or buffer, _io.BytesIO found.
BUT, csv.reader can also take a string or buffer directly, so we pass in the straight shootin' buffer to the csv.reader:
buffer = csv_field.data.stream # csv_field is the FileField obj
csv_data = csv.reader(buffer, delimiter=',')
for row in csv_data:
print row
I found it interesting that if you need additional coercion to/from Unicode UTF-8, the csv examples in docs provides a snippet on wrapping an encoder/decoder.
For me, this did the trick:
def on_model_change(self, form, model):
tweet_file = form.tweet_keywords_file
buffer = tweet_file.data.stream
file_data = buffer.getvalue()
Related
I'm currently building an application in python where I have a class Corpus. I would like to convert this class to a json format and save it to a json file. Then load the file and finally turn back the json to its Class Corpus.
In order to do that I use the library jsonpickle. The problem is when I load the json, the type is a dictionary and jsonpickle.decode wants a string. I tried to convert the dictionary to a string but its not working.
I hope someone will be able to help me.
Here is my code of my class "Json" (to save and load my Corpus)"
import json, jsonpickle
class Json:
def __init__(self):
self.corpus = {}
def saveCorpus(self,corpus):
jsonCorpus = jsonpickle.encode(corpus,indent=4,make_refs=False)
with open('json_data.json', 'w') as outfile:
outfile.write(jsonCorpus)
def loadCorpus(self):
with open('json_data.json', 'r') as f:
self.corpus = json.load(f)
def getCorpus(self):
return self.corpus
error :
TypeError: the JSON object must be str, bytes or bytearray, not dict
I found the problem.
The issue was the way I was saving my json to a file.
Here is the solution:
def saveCorpus(self,corpus):
jsonCorpus = jsonpickle.encode(corpus,indent=4,make_refs=False)
with open('json_data.json', 'w') as outfile:
json.dump(jsonCorpus, outfile)
def convertToBinaryData(filename):
# Convert digital data to binary format
with open(filename, 'rb') as file:
binaryData = file.read()
return binaryData
This is my function for converting an image to binary...
uploaded_file = request.files['file']
if uploaded_file.filename != '':
uploaded_file.save(uploaded_file.filename)
empPicture = convertToBinaryData(uploaded_file)
and this is the block of code where the uploaded file is received and saved, however, when it runs, I get this error...
with open(filename, 'rb') as file:
TypeError: expected str, bytes or os.PathLike object, not FileStorage
I'm pretty new to python and I've been stuck on this for a while, any help would be appreciated. Thanks in advance
while calling 'convertToBinaryData' you are passing 'uploaded_file' which is not a filename but and object.
You need to pass the filename (with correct path if saved in custom location) to your 'convertToBinaryData' funciton.
Something like this:
convertToBinaryData(uploaded_file.filename)
uploaded_file is not a filename, it's a Flask FileStorage object. You can read from this directly, you don't need to call open().
So just do:
empPicture = uploaded_file.read()
See Read file data without saving it in Flask
I am working on a scrapy spider, trying to extract text multiple pdfs in a directory, using slate (https://pypi.python.org/pypi/slate). I have no interest in saving the actual PDF to disk , and so I've been advised to look into the io.bytesIO subclass at https://docs.python.org/2/library/io.html#buffered-streams. Based on Creating bytesIO object , I have initialized the bytesIO class with the pdf body, but now I need to pass the data to the slate module. So far I have:
def save_pdf(self, response):
in_memory_pdf = BytesIO(response.body)
with open(in_memory_pdf, 'rb') as f:
doc = slate.PDF(f)
print(doc[0])
I'm getting:
in_memory_pdf.read(response.body)
TypeError: integer argument expected, got 'str'
How can I get this working?
edit:
with open(in_memory_pdf, 'rb') as f:
TypeError: coercing to Unicode: need string or buffer, _io.BytesIO found
edit 2:
def save_pdf(self, response):
in_memory_pdf = BytesIO(bytes(response.body))
in_memory_pdf.seek(0)
doc = slate.PDF(in_memory_pdf)
print(doc)
You already know the answer. It is clearly mentioned in the Python TypeError message and clear from the documentation:
class io.BytesIO([initial_bytes])
BytesIO accepts bytes. And you are passing it contents. i.e: response.body which is a string.
I'm developing an application in python with django. User can upload a CSV file. I use file upload to get the file. But, it's not stored any where. I try to take it from request to process the file. While I'm trying to open the file, it gives an error. I use the CSV library exists in python to process. Form elements and attributes used as per django. Request object which I try to take the uploaded file is also django designed object.
import csv
from rootFolder.UploadFileForm
def uploadFile(request):
if request.method == 'POST':
form = UploadFileForm(request.POST, request.FILES)
if form.is_valid():
paramFile = open(request.FILES['uploadFile'], 'rb')
portfolio = csv.DictReader(paramFile)
users = []
for row in portfolio:
users.append(row)
This below line gives the error.
paramFile = open(request.FILES['uploadFile'], 'rb')
The given error is :
TypeError: coercing to Unicode: need string or buffer, InMemoryUploadedFile found
Please kindly give your suggestion if you got any idea on this.
Thanks in advance.
This works for Python 3
import csv
import io
...
csv_file = request.FILES['uploadFile']
decoded_file = csv_file.read().decode('utf-8')
io_string = io.StringIO(decoded_file)
for line in csv.reader(io_string, delimiter=';', quotechar='|'):
print(line)
No need to call open on the file, it's already open. You should be able to pass it straight into the DictReader.
open() takes the name of the file as the argument and not the file object itself.
Can you try something like this:
paramFile = request.FILES['uploadFile'].read()
portfolio = csv.DictReader(paramFile)
Django 2, Python 3.6, same problem, half a day of googling, this is the solution that worked for me.
form.is_valid:
csv_file = request.FILES['file']
csv_file.seek(0)
reader = csv.DictReader(io.StringIO(csv_file.read().decode('utf-8')))
for row in reader
.....
.....
detailed writeup here -> source
You get a TypeError, because the built in function open expects a string that is a path to a file.
Does this work?
if form.is_valid():
request.FILES['uploadFile'].open("rb")
portfolio = csv.DictReader(request.FILES['uploadFile'].file)
I have a design question. I have a function loadImage() for loading an image file. Now it accepts a string which is a file path. But I also want to be able to load files which are not on physical disk, eg. generated procedurally. I could have it accept a string, but then how could it know the string is not a file path but file data? I could add an extra boolean argument to specify that, but that doesn't sound very clean. Any ideas?
It's something like this now:
def loadImage(filepath):
file = open(filepath, 'rb')
data = file.read()
# do stuff with data
The other version would be
def loadImage(data):
# do stuff with data
How to have this function accept both 'filepath' or 'data' and guess what it is?
You can change your loadImage function to expect an opened file-like object, such as:
def load_image(f):
data = file.read()
... and then have that called from two functions, one of which expects a path and the other a string that contains the data:
from StringIO import StringIO
def load_image_from_path(path):
with open(path, 'rb') as f:
load_image(f)
def load_image_from_string(s):
sio = StringIO(s)
try:
load_image(sio)
finally:
sio.close()
How about just creating two functions, loadImageFromString and loadImageFromFile?
This being Python, you can easily distinguish between a filename and a data string. I would do something like this:
import os.path as P
from StringIO import StringIO
def load_image(im):
fin = None
if P.isfile(im):
fin = open(im, 'rb')
else:
fin = StringIO(im)
# Read from fin like you would from any open file object
Other ways to do it would be a try block instead of using os.path, but the essence of the approach remains the same.