So for the past few days I've been trying to learn Python in App Engine. However, I've been encountering a number of problems with ASCII and UTF encoding. The freshest issue is as follows:
I have the following piece of code of a simplistic chatroom from the book 'Code in the Cloud'
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
import datetime
# START: MainPage
class ChatMessage(object):
def __init__(self, user, msg):
self.user = user
self.message = msg
self.time = datetime.datetime.now()
def __str__(self):
return "%s (%s): %s" % (self.user, self.time, self.message)
Messages = []
class ChatRoomPage(webapp.RequestHandler):
def get(self):
self.response.headers["Content-Type"] = "text/html"
self.response.out.write("""
<html>
<head>
<title>MarkCC's AppEngine Chat Room</title>
</head>
<body>
<h1>Welcome to MarkCC's AppEngine Chat Room</h1>
<p>(Current time is %s)</p>
""" % (datetime.datetime.now()))
# Output the set of chat messages
global Messages
for msg in Messages:
self.response.out.write("<p>%s</p>" % msg)
self.response.out.write("""
<form action="" method="post">
<div><b>Name:</b>
<textarea name="name" rows="1" cols="20"></textarea></div>
<p><b>Message</b></p>
<div><textarea name="message" rows="5" cols="60"></textarea></div>
<div><input type="submit" value="Send ChatMessage"></input></div>
</form>
</body>
</html>
""")
# END: MainPage
# START: PostHandler
def post(self):
chatter = self.request.get("name")
msg = self.request.get("message")
global Messages
Messages.append(ChatMessage(chatter, msg))
# Now that we've added the message to the chat, we'll redirect
# to the root page, which will make the user's browser refresh to
# show the chat including their new message.
self.redirect('/')
# END: PostHandler
# START: Frame
chatapp = webapp.WSGIApplication([('/', ChatRoomPage)])
def main():
run_wsgi_app(chatapp)
if __name__ == "__main__":
main()
# END: Frame
It works ok in English. However, the moment I add some non-standard characters all sorts of problems start
First of all, in order for the thing to be actually able to display characters in HTML I add meta tag - charset=UTF-8" etc
Curiously, if you enter non-standard letters, the program processes them nicely, and displays them with no issues. However, it fails to load if I enter any non-ascii letters to the web layout iteself withing the script. I figured out that adding utf-8 encoding line would work. So I added (# -- coding: utf-8 --). This was not enough. Of course I forgot to save the file in UTF-8 format. Upon that the program started running.
That would be the good end to the story, alas....
It doesn't work
Long story short this code:
# -*- coding: utf-8 -*-
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
import datetime
# START: MainPage
class ChatMessage(object):
def __init__(self, user, msg):
self.user = user
self.message = msg
self.time = datetime.datetime.now()
def __str__(self):
return "%s (%s): %s" % (self.user, self.time, self.message)
Messages = []
class ChatRoomPage(webapp.RequestHandler):
def get(self):
self.response.headers["Content-Type"] = "text/html"
self.response.out.write("""
<html>
<head>
<title>Witaj w pokoju czatu MarkCC w App Engine</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<h1>Witaj w pokoju czatu MarkCC w App Engine</h1>
<p>(Dokladny czas Twojego logowania to: %s)</p>
""" % (datetime.datetime.now()))
# Output the set of chat messages
global Messages
for msg in Messages:
self.response.out.write("<p>%s</p>" % msg)
self.response.out.write("""
<form action="" method="post">
<div><b>Twój Nick:</b>
<textarea name="name" rows="1" cols="20"></textarea></div>
<p><b>Twoja Wiadomość</b></p>
<div><textarea name="message" rows="5" cols="60"></textarea></div>
<div><input type="submit" value="Send ChatMessage"></input></div>
</form>
</body>
</html>
""")
# END: MainPage
# START: PostHandler
def post(self):
chatter = self.request.get(u"name")
msg = self.request.get(u"message")
global Messages
Messages.append(ChatMessage(chatter, msg))
# Now that we've added the message to the chat, we'll redirect
# to the root page, which will make the user's browser refresh to
# show the chat including their new message.
self.redirect('/')
# END: PostHandler
# START: Frame
chatapp = webapp.WSGIApplication([('/', ChatRoomPage)])
def main():
run_wsgi_app(chatapp)
if __name__ == "__main__":
main()
# END: Frame
Fails to process anything I write in the chat application when it's running. It loads but the moment I enter my message (even using only standard characters) I receive
File "D:\Python25\lib\StringIO.py", line 270, in getvalue
self.buf += ''.join(self.buflist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 64: ordinal not in range(128)
error message. In other words, if I want to be able to use any characters within the application I cannot put non-English ones in my interface. Or the other way round, I can use non-English characters within the app only if I don't encode the file in utf-8. How to make it all work together?
Your strings contain unicode characters, but they're not unicode strings, they're byte strings. You need to prefix each one with u (as in u"foo") in order to make them into unicode strings. If you ensure all your strings are Unicode strings, you should eliminate that error.
You should also specify the encoding in the Content-Type header rather than a meta tag, like this:
self.response.headers['Content-Type'] = 'text/html; charset=UTF-8'
Note your life would be a lot easier if you used a templating system instead of writing HTML inline with your Python code.
#Thomas K.
Thank you for your guidance here. Thanks to you I was able to come up with, maybe - as you said - a little roudabout solution - so the credit for the answer should go to you. The following line of code:
Messages.append(ChatMessage(chatter, msg))
Should look like this:
Messages.append(ChatMessage(chatter.encode( "utf-8" ), msg.encode( "utf-8" )))
Basically I have to encode all the utf-8 string to ascii.
Related
This question already has answers here:
Sending data from HTML form to a Python script in Flask
(2 answers)
Closed 5 years ago.
Essentially what I am trying to do :
I have a simple HTML page with a Textarea to input a bunch of text, my use case is a single code on each line like below:
1234
5678
1456
etc.
Ideally I want to take that into Python and be able to work with the data and return the results. So lets start simple and say take each line as a separate entry and run it against a function to add the word "Hi" in front of it So the results are:
Hi 1234
Hi 5678
etc.
So far have this working example I found but I tend to break it anytime I try something.
Html:
<!DOCTYPE html>
<html lang="en">
<body>
<h1>Enter some text</h1>
<form action="submit" id="textform" method="post">
<textarea name="text">Hello World!</textarea>
<input type="submit" value="Submit">
</form>
</body>
</html>
Python:
From flask import Flask, request
app = Flask(__name__)
#app.route('/')
def main_form():
return '<form action="submit" id="textform" method="post"><textarea name="text">Hello World!</textarea><input type="submit" value="Submit"></form>'
#app.route('/submit', methods=['POST'])
def submit_textarea():
return "You entered: {}".format(request.form["text"])
if __name__ == '__main__':
app.run()
Example :
i try to extract the textarea to a string and then return that back to the page with :
x = format(request.form["text"])
return x
Any help or guidance would be appreciated!
You can access and store the text from textarea with the following lines :
#app.route('/submit', methods=['POST'])
def submit_textarea():
# store the given text in a variable
text = request.form.get("text")
# split the text to get each line in a list
text2 = text.split('\n')
# change the text (add 'Hi' to each new line)
text_changed = ''.join(['<br>Hi ' + line for line in text2])
# (i used <br> to materialize a newline in the returned value)
return "You entered: {}".format(text_changed)
I'm new in StackOverflow and Python. I'm trying to make a basic application where I have my < form > tag, my flask app, and my .txt. I'm trying to change values but it's not working or I don't know why it's not working. Could any of you give me a hand?
Python Flask:
from flask import Flask,render_template,flash,request,redirect
import os
app = Flask(__name__)
from lines import get_line
#app.route('/', methods=['POST'])
def change_line():
search_line= "this"
try:
for line in this.input(os.path.join(APP_STATIC, u'line.txt'),inplace=1):
if search_line in line:
x = line.replace(search_line,search_line + "\n" + request.form.get(u'this'))
print (x)
else:
print (x)
except BaseException as e:
print e
return render_template('line.html')
#app.route('/')
def showLine():
line = get_line()
return render_template('line.html', line=line)
if __name__ == '__main__':
app.run()
HTML:
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Change the value of this line</title>
</head>
<body>
<form method="post" name="test">
<h4>Chaging Values with POST Method</h4>
<div class="col-sm-9">
<label class="col-sm-3 col-sm-3 control-label">I want to change : </label>
<input type="text" class="form-control" name="address" value="{{ line }}">
</div>
<input type="submit" value="Save Changes!">
</form>
My .txt file:
I want to change this.
It's returning the value of "this".
Basically when I run my app it's displaying "this" when I try to edit for "Hello" it's returning me an error:
Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) Line
Found! : this.
127.0.0.1 - - [21/Oct/2016 13:09:07] "GET / HTTP/1.1" 200 - global name 'this' is not defined
127.0.0.1 - - [21/Oct/2016 13:09:30] "POST / HTTP/1.1" 200 -
My output:
Please click here to see my output
I'm sorry if this is a silly question, or if this was answered before, but I've been browsing for an answer for time and I have found nothing about this, I've tried different codes but not working, been watching youtube videos and everything.
If someone know the solution would be really helpful for me.
It's for learning and Python School. Thanks!
Working on Python 2.7
EDIT
I updated code with the suggestions below but it's not working yet.
it's not overwriting my "this" with replace.
from flask import Flask,render_template,flash,request,redirect
import os
app = Flask(__name__)
from lines import get_line
import fileinput
#app.route('/', methods=['POST'])
def change_line():
search_line= "this"
try:
for line in fileinput.input(os.path.join(APP_STATIC, u'line.txt'),inplace=1):
if search_line in line:
x = line.replace(search_line,search_line + "\n" + request.form.get(u'asd'))
print (x)
else:
print (x)
except BaseException as e:
print e
return render_template('line.html')
#app.route('/')
def showLine():
line = get_line()
return render_template('line.html', line=line)
if __name__ == '__main__':
app.run(debug=True)
this.input(os.path.join(APP_STATIC, u'line.txt'),inplace=1):
should be
fileinput.input(os.path.join(APP_STATIC, u'line.txt'),inplace=1):
(you will also need import fileinput)
you need to change this because this.input doesnt mean anyhthing to python ... it has no idea what this is ...
global name 'this' is not defined
as an aside you should really run with app.run(debug=True) while developing
In my flask-app, i'm want to see a preview of pdf that will be generated, before definitely printing and saving it in my application_base_folder.
I could save previews on a tmp dir, but that's not what I really looking for.
I'm creating a pdf with report lab :
def gen_pdf(text):
output = cStringIO.StringIO()
c = canvas.Canvas(output)
c.drawString(100,100, text)
c.showPage()
c.save()
pdf_output=output.getvalue()
output.close()
return pdf_output
The pdf would be then send to html template with my form to update part
class Form(wtforms.Form):
text = TextField('text')
#app.route('/finalize/pdf/')
def finalize_pdf():
form = Form(request.form)
pdf_output = gen_pdf(form.text.data)
return render_template('preview_pdf.html', form=form, pdf_output=pdf_output)
and in the html page, I have my form from which i can update the text, a button to POST value (it doesn't appear in the view finalize_pdf() ), and the pdf preview :
<form method='post' action={{ url_for('finalize_pdf') }}
{{ form.text }}
<input name='update' value='update'>
</form>
<br />
<embed src="{{ pdf_output }}" type="application/pdf" width='30%'>
The problem here is that "src" is probably looking for a path on the FS, I guess ? Anyway, it "dies" with this error :
UnicodeDecodeError: 'ascii' codec can't decode byte 0x93 in position 11: ordinal not in range(128)
Am I so missing something, or is it impossible to embed / object in html a StringIO and I must write it on a tmp_path on my FS ?
This is a bit late but you could use base64 encoding for this problem:
from base64 import b64encode
# in your view:
return render_template(..., pdf_output=b64encode(pdf_output))
And your template:
<embed src="data:application/pdf;base64,{{ pdf_output }}" type="application/pdf" width='30%'>
I am trying to use Bottle.py to get input information from users in a web page.
Everything works fine except when I have latin characters (accents mostly). I have try using utf-8 and latin-1 coding on the first two lines of the code, but it won't work.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import bottle
#bottle.post('/newpost')
def post_newpost():
subject = bottle.request.forms.get("subject")
body = bottle.request.forms.get("body")
tags = bottle.request.forms.get("tags")
and the html code from the page is:
<html>
<head>
<meta charset="utf-8" />
<title>New Posts</title>
</head>
<body>
<form action="/newpost" method="POST">
<h2>Post title</h2>
<input type="text" name="subject" size="120" value="{{subject}}" ><br>
<h2>Post<h2>
<textarea name="body" cols="120" rows="20">{{body}}</textarea><br>
<h2>Tags</h2>
<input type="text" name="tags" size="120" value="{{tags}}"><br>
<p>
<input type="submit" value="Submit">
</body>
</html>
I read in Bottle page that:
In Python 3 all strings are unicode, but HTTP is a byte-based wire
protocol. The server has to decode the byte strings somehow before
they are passed to the application. To be on the safe side, WSGI
suggests ISO-8859-1 (aka latin1), a reversible single-byte codec that
can be re-encoded with a different encoding later. Bottle does that
for FormsDict.getunicode() and attribute access, but not for the
dict-access methods. These return the unchanged values as provided by
the server implementation, which is probably not what you want.
request.query['city']
'Göttingen' # An utf8 string provisionally decoded as ISO-8859-1 by the server
request.query.city
'Göttingen' # The same string correctly re-encoded as utf8 by bottle
If you need the whole dictionary with correctly decoded values (e.g. for WTForms), you can call FormsDict.decode() to get a re-encoded copy.
After reading that I tried using that function but don't know how.
Right now Bottle form returns strings, so I can not use encode('utf-8') or decode('utf-8').
Please help me!
Thanks!
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import bottle
#bottle.post('/newpost')
def post_newpost():
subject = bottle.request.forms.subject
body = bottle.request.forms.body
tags = bottle.request.forms.tags
That will do it.... Thanks!
In mod_python Publisher, I could write a file loader, (uploader.py in this example) using code like this:
def index():
html = '''
<html>
<body>
<form id="fileUpload" action="uploader.py/upload" enctype="multipart/form-data" method="post">
<input type="file" id="file" name="file_name"/>
<input type="submit" value="Upload"/>
</form>
</body>
</html>
'''
return html
def upload(req, file_name):
fd = open("/mydir/myfile", "w")
for line in file_name.file.readlines():
fd.write(line)
fd.close()
This example, excludes any error checking but illustrates what I could do with mod_python and Publisher. The key things are that Publisher was able to call my function, upload, in the file uploader.py and pass it an argument, file_name with the attribute file with the file that was selected in the form.
What I want to do is to do the same thing in mod_wsgi. I'm looking for a way to call a function like this from a form and invoke my function in a similar fashion. I've looked at webob but cannot figure out how to do the same thing with this package. I would appreciate any help in coding a similar solution without having to use mod_python and Publisher.
Answer migrated from question text.
After some digging, I figured out a solution to this problem. This is the solution I came up with. I implemented this with two files: uploader1.py and uploader2.py. This is code for uploader1.py:
def application(environ, start_response):
from webob import Response
html = '''
<html>
<body>
<h1>Upload</h1>
<form id="fileUpload" action="uploader2.py" enctype="multipart/form-data" method="post">
<input type="file" name="filename"/>
<input type="submit" value="Upload"/>
</form>
</body>
</html>
'''
response = Response(body = html,
content_type = "text/html",
charset = "utf8",
status = "200 OK")
return response(environ, start_response)
And this is the code for uploader2.py:
import os
import cgi
def application(environ, start_response):
from webob import Response
form = cgi.FieldStorage(fp=environ['wsgi.input'], environ=environ, keep_blank_values=True)
try:
fileitem = form['filename']
except KeyError:
fileitem = None
if fileitem is not None and fileitem.filename and fileitem.file:
# remove any relative paths to prevent directory traversal attacks
fn = os.path.basename(fileitem.filename)
# directory to upload to
fn = "/tmp/"+fn
with open(fn, 'w') as f:
# this upload is for text files
lines = fileitem.file.readlines()
for line in lines:
f.write(line)
f.close()
message = "Uploaded file: %s" % fn
else:
message = "Error: no file selected for upload"
html = '''
<html>
<body>
<p>%s
</body>
</html>
''' % message
response = Response(body = html,
content_type = "text/html",
charset = "utf8",
status = "200 OK")
return response(environ, start_response)
The only drawbacks to this implementation over the mod_python one is that I couldn't figure out a way to implement this solution in a single file. mod_python allowed me to call a function within a module and I can't figure out a way to do this with mod_wsgi. Also, during debugging it would have been nice to be able to log output to the same custom error_log file I had set up for mod_wsgi and I couldn't find a way of doing this, so had to log to the main Apache log file by using print statements. If someone has a way of doing this, I would be pleased to know a solution.