I'm trying to build a simple web socket server that loads a file with some tweets in it (as CSV) and then just sends the string of the tweet to a web browser through a websocket. Here is a gist with the sample that I'm using for testing. Here's the Autobahn server component (server.py):
import random
import time
from twisted.internet import reactor
from autobahn.websocket import WebSocketServerFactory, \
WebSocketServerProtocol, \
listenWS
f = open("C:/mypath/parsed_tweets_sample.csv")
class TweetStreamProtocol(WebSocketServerProtocol):
def sendTweet(self):
tweet = f.readline().split(",")[2]
self.sendMessage(tweet, binary=False)
def onMessage(self, msg, binary):
self.sendTweet()
if __name__ == '__main__':
factory = WebSocketServerFactory("ws://localhost:9000", debug = False)
factory.protocol = TweetStreamProtocol
listenWS(factory)
reactor.run()
And here is the web component (index.html):
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<script type="text/javascript">
var ws = new WebSocket("ws://localhost:9000");
ws.onmessage = function(e) {
document.getElementById('msg').textContent = e.data; //unescape(encodeURIComponent(e.data));
console.log("Got echo: " + e.data);
}
</script>
</head>
<body>
<h3>Twitter Stream Visualization</h3>
<div id="msg"></div>
<button onclick='ws.send("tweetme");'>
Get Tweet
</button>
</body>
</html>
When the tweet arrives in the browser, the UTF-8 characters aren't properly displayed. How can I modify these simple scripts to display the proper UTF-8 characters in the browser?
This works for me:
from autobahn.twisted.websocket import WebSocketServerProtocol, \
WebSocketServerFactory
class TweetStreamProtocol(WebSocketServerProtocol):
def sendTweets(self):
for line in open('gistfile1.txt').readlines():
## decode UTF8 encoded file
data = line.decode('utf8').split(',')
## now operate on data using Python string functions ..
## encode and send payload
payload = data[2].encode('utf8')
self.sendMessage(payload)
self.sendMessage((u"\u03C0"*10).encode("utf8"))
def onMessage(self, payload, isBinary):
if payload == "tweetme":
self.sendTweets()
if __name__ == '__main__':
import sys
from twisted.python import log
from twisted.internet import reactor
log.startLogging(sys.stdout)
factory = WebSocketServerFactory("ws://localhost:9000", debug = False)
factory.protocol = TweetStreamProtocol
reactor.listenTCP(9000, factory)
reactor.run()
Notes:
above code is for Autobahn|Python 0.7 and above
I'm not sure if you sample Gist is properly UTF8 encoded file
However, the "last" pseudo Tweet is 10x "pi", and that properly shows in the browser, so
it works in principle ..
Also note: for reasons too long to explain here, Autobahn's sendMessage function expects payload to be already UTF8 encoded if isBinary == False. A "normal" Python string is Unicode, which needs to be encoded like above to UTF8 for sending.
instead of <meta http-equiv="content-type" content="text/html; charset=UTF-8">< try <meta charset = utf-8>.
if you're using XHTML then write <meta charset = utf-8 />
Related
It appears that Flask assumes that the server is returning html to the client (browser).
Here's a simple example;
import json
from flask import Flask
app = Flask(__name__)
#app.route("/")
def home():
msg = ['Hello, world!']
return json.dumps(msg) + '\n'
This code works as expected and returns the desired json;
$ curl -s http://localhost:5000/
["Hello, world!"]
But if I introduce an error;
import json
from flask import Flask
app = Flask(__name__)
#app.route("/")
def home():
msg = ['Hello, world!']
return json.dumps(XXmsg) + '\n'
Then Flask emits the error wrapped in several pages worth of html, starting like;
$ curl -s http://localhost:5000/
<!DOCTYPE html>
<html>
<head>
<title>NameError: name 'XXmsg' is not defined
// Werkzeug Debugger</title>
<link rel="stylesheet" href="?__debugger__=yes&cmd=resource&f=style.css">
<link rel="shortcut icon"
href="?__debugger__=yes&cmd=resource&f=console.png">
<script src="?__debugger__=yes&cmd=resource&f=debugger.js"></script>
<script>
var CONSOLE_MODE = false,
EVALEX = true,
EVALEX_TRUSTED = false,
SECRET = "Mq5TSy6QE4OuOHUfvk8b";
</script>
</head>
<body style="background-color: #fff">
<div class="debugger">
Emitting html makes sense if you're creating a page load app. But I'm creating an api that only returns json.
Is there anyway to prevent Flask from emitting html at all?
Thanks
Mike
Have a look at the section Returning API Errors as JSON of the Flask docs.
Basically, you have to replace the default error handler with a function that returns the error as json. A very basic example:
#app.errorhandler(HTTPException)
def handle_exception(exception):
response = exception.get_response()
response.content_type = "application/json"
response.data = json.dumps({"code": exception.code})
return response
The accepted response gives a good hint for handling HTTPException but it won't work for all exceptions unless you create a handler for the mother of all exceptions:Exception. And you might not want to do this for security reasons, if you have some custom defined exceptions with sensible data it'll get handled by this handler.
I suspect the true reason you have those lengthy html responses is because you started your flask app with the --debug option.
Let's say I have a simple HTML webpage (served using apache) as
<!DOCTYPE HTML>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width">
<meta name="description" content="CGI script test">
<meta name="keywords" content="test">
<meta name="author" content="cgi test">
<title> CGI Script Test </title>
</head>
<body>
<form action="/cgi-bin/submit.py" method="POST">
<label for="entry">Entry name: </label>
<input type="text" id="entry" name="entryname" placeholder="placeholder" maxlength="10">
</form>
</body>
</html>
where data submitted in the form is processed using submit.py, a python script (placed in my cgi-bin directory) as
#!/usr/bin/python
import cgi,re
form = cgi.FieldStorage()
print("Content-Type: text/html\n\n")
print("<title>Hello World</title>")
print("<h1>HELLO</h1>")
text=str(form.getvalue("entryname"))
print("<p> Parsing result...</p>")
result = re.sub('[^a-zA-Z0-9:##/-_,]', ' ', text)
print("<h3> Resulting Info: </h3>")
print("<p>" + str(result) + "</p>")`
I want to avoid my server getting stuffed with POSTs that are excessively long. If I load the HTML webpage above, I can use the "inspect element" tool in firefox to delete the "maxlength" requirement and stuff in as much information as I want. The python script then receives the full input text. This is my first website and I want to make sure I do this right. Is there a limit to the size of the POST sent to the server, and if not, how do I limit it to prevent abuse?
You can examine Content-Length header and compare it with a limit. cgitb.enable should be helpful with displaying errors when the limit is reached.
import cgitb
import os
MAX_POST_BODY_SIZE = 1024
cgitb.enable(display=0)
content_length = int(os.getenv('CONTENT_LENGTH', 0))
if content_length > MAX_POST_BODY_SIZE:
raise RuntimeError('POST body too long')
Update 1
I've looked into cgi module's code and it seems POST body size limiting is actually implemented in FieldStorage but it's not documented. There's cgi.maxlen attribute:
# Maximum input we will accept when REQUEST_METHOD is POST
# 0 ==> unlimited input
maxlen = 0
Hence, it should be just:
import cgi
import cgitb
cgi.maxlen = 1024
cgitb.enable(display=0)
form = cgi.FieldStorage()
[...] would the server still have to allocate memory to receive the full post?
As far as I can see in the initialiser of FieldStorage the steps are as follows:
fp is assigned, self.fp = sys.stdin.buffer
Content-Length is validated
when then Content-Type is application/x-www-form-urlencoded read_urlencoded is called which reads Content-Length bytes from the instance's fp attribute.
To test it with your CGI server, send a big request and look at htop or other process monitor for the CGI process' memory usage.
from urllib.request import urlopen
urlopen(
'http://localhost:8000/cgi-bin/submit.py',
data=b'entryname=%s' % (b'abcd' * 2 ** 24), # 64 MiB
)
I'm building a little server that sends a page with a script in javascript, it works when I try to open it in my browser, but if I request the page from the server, the page is recived, but not the script, so I get these errors:
Script.js is missing between the files:
Looks strange because from the network session i can see a request for the script with the status 200:
The js file i'm tryng to add is Chart.js, so I can't add it internally, it would become impossible to work with it, but for now the server is just a few lines of code in python that use the SimpleHTTPRequestHandler, and I'm probably going to replace it, may it be because the SimpleHTTPRequestHandler can't handle multiple file requests?
Here's the code, tried to make a snippet but it does't work there too (probably that's just me never wrote a snippet before):
HTML:
<!doctype html>
<html>
<head>
<script src="script.js"></script>
</head>
<body>
<p id = "paragraph"></p>
<script>
document.getElementById('paragraph').innerHTML = sayHello();
</script>
</body>
</html>
JS:
function sayHello(){
return "HelloWorld!"
}
Here is the python server script:
from http.server import HTTPServer, BaseHTTPRequestHandler
class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.end_headers()
with open("index.html", "r") as page:
self.wfile.write(page.read())
httpd = HTTPServer(("192.168.1.100", 8000), SimpleHTTPRequestHandler)
httpd.serve_forever()
I think you get an element with id, tag name or class Name and add file
var head = document.getElementsByTagName('head')[0];
var script = document.createElement('script');
script.type = 'text/javascript';
script.onload = function() {
callFunctionFromScript();
}
script.src = 'path/to/your-script.js';
head.appendChild(script);
also check this link
Program description: I already have a functioning program that runs on console window, but I'd like to present its output on a locally hosted web page. The program consists on getting lyrics for currently playing songs by making requests to Spotify's API. I store the current lyrics in a "lyrics.txt" file.
What I want:
Change the web page from the running lyrics program when it detects the song has changed.
[EDIT:]
Is there a way to make the flask page display a variable, that is updated by a python request.post of the lyrics app to the flask url with the updated variable as the data?
What I have:
I'm using Flask as the framework since its a one local web page.
import os, io
from flask import Flask, render_template
app = Flask(__name__)
#app.route("/")
def test():
'''reads the current lyrics file and passes it to the web page
manually reload the page to update lyrics'''
with io.open('lyrics.txt', 'r') as f:
HEAD = f.readline().strip("\n")
BODY = f.read().split('\n')
lyrics = {"HEAD": HEAD, "BODY": BODY}
return render_template("home.html", lyrics=lyrics)
if __name__ == "__main__":
app.run(debug=1)
link to lyrics app github
You would need JavaScript/AJAX on page which periodically sends request for new content and Flask should send current content from file.
In this example I use jQuery.get() to send request to server, and setTimeout() to repeat it periodically.
Flask sends current time to show different content.
import datetime
from flask import Flask, render_template_string
app = Flask(__name__)
#app.route("/")
def index():
return render_template_string("""<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Test</title>
<script type="text/javascript" src="http://code.jquery.com/jquery-1.8.0.min.js"></script>
<script type="text/javascript">
function updater() {
$.get('/data', function(data) {
$('#time').html(data); // update page with new data
});
};
setInterval(updater, 1000); // run `updater()` every 1000ms (1s)
</script>
</head>
<body>
Date & Time: <span id="time"><span>
</body>
</html>""")
#app.route('/data')
def data():
"""send current content"""
return datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
if __name__ == "__main__":
app.run(debug=True)
EDIT:
The same using standard fetch() without external libraries.
Code has to be after <span>
import datetime
from flask import Flask, render_template_string
app = Flask(__name__)
#app.route("/")
def index():
return render_template_string("""<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Test</title>
</head>
<body>
Date & Time: <span id="time"><span>
<script type="text/javascript">
var time_span = document.getElementById("time");
function updater() {
fetch('/data')
.then(response => response.text())
.then(text => (time_span.innerHTML = text)); // update page with new data
}
setInterval(updater, 1000); // run `updater()` every 1000ms (1s)
</script>
</body>
</html>""")
#app.route('/data')
def data():
"""send current content"""
return datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
if __name__ == "__main__":
app.run(debug=True)
I am trying to create a 2-way communication between server and client using Flask and socket.io.
Everything works fine until server receives utf-8 string from client, which gets garbled. Sending from server to client works fine, and prior to sending from client to server, the client prints the message correctly.
Here is some code that reproduces the problem:
app.py:
import flask
from flask_socketio import SocketIO, emit, disconnect
import json
app = flask.Flask(__name__)
socket_io = SocketIO(app)
#socket_io.on('pull')
def socket_io_handle_pull():
json_msg = {
'msg': "abcćčddžđefghijklmnnjoprsštuvzž"
}
print("Pushing", json_msg)
socket_io.emit('response', json_msg)
#socket_io.on('push')
def socket_io_handle_push(json_msg):
print("Pushed:", json_msg)
#socket_io.on('disconnect')
def socket_io_handle_disconnect():
disconnect()
#app.route('/')
def root():
return flask.render_template(
'index.html'
)
if __name__ == '__main__':
socket_io.run(app)
index.html:
<!DOCTYPE html>
<html>
<head>
<script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/socket.io/1.3.6/socket.io.min.js"></script>
</head>
<body>
<script type="text/javascript">
var socket = io.connect('http://' + document.domain + ':' + location.port);
socket.on('response', json => {
socket.emit('push', json);
})
socket.emit('pull');
</script>
</body>
</html>
Output:
Pushing {'msg': 'abcćčddžđefghijklmnnjoprsštuvzž'}
Pushed: {'msg': 'abcÄ\x87Ä\x8dddA3Ä\x91efghijklmnnjoprsA!tuvzA3'}
You are using the 1.x versions of the Socket.IO client, which had known problems with double-encoding of UTF-8 strings. You should try the 2.x versions which have resolved this issue.
It seems that I was getting back a mojibake decoded using latin-1 and encoded with utf-8.
To fix this, I added:
json_str = json_str.encode('latin-1').decode('utf-8')
If you are having this problem, take a look at Miguel's answer.
I used servers socket.io js file from reverse proxy by adding socket.io.js end of reverse proxy path like this xxx.com/reverse_proxy_path/socket.io