I am trying to create a 2-way communication between server and client using Flask and socket.io.
Everything works fine until server receives utf-8 string from client, which gets garbled. Sending from server to client works fine, and prior to sending from client to server, the client prints the message correctly.
Here is some code that reproduces the problem:
app.py:
import flask
from flask_socketio import SocketIO, emit, disconnect
import json
app = flask.Flask(__name__)
socket_io = SocketIO(app)
#socket_io.on('pull')
def socket_io_handle_pull():
json_msg = {
'msg': "abcćčddžđefghijklmnnjoprsštuvzž"
}
print("Pushing", json_msg)
socket_io.emit('response', json_msg)
#socket_io.on('push')
def socket_io_handle_push(json_msg):
print("Pushed:", json_msg)
#socket_io.on('disconnect')
def socket_io_handle_disconnect():
disconnect()
#app.route('/')
def root():
return flask.render_template(
'index.html'
)
if __name__ == '__main__':
socket_io.run(app)
index.html:
<!DOCTYPE html>
<html>
<head>
<script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/socket.io/1.3.6/socket.io.min.js"></script>
</head>
<body>
<script type="text/javascript">
var socket = io.connect('http://' + document.domain + ':' + location.port);
socket.on('response', json => {
socket.emit('push', json);
})
socket.emit('pull');
</script>
</body>
</html>
Output:
Pushing {'msg': 'abcćčddžđefghijklmnnjoprsštuvzž'}
Pushed: {'msg': 'abcÄ\x87Ä\x8dddA3Ä\x91efghijklmnnjoprsA!tuvzA3'}
You are using the 1.x versions of the Socket.IO client, which had known problems with double-encoding of UTF-8 strings. You should try the 2.x versions which have resolved this issue.
It seems that I was getting back a mojibake decoded using latin-1 and encoded with utf-8.
To fix this, I added:
json_str = json_str.encode('latin-1').decode('utf-8')
If you are having this problem, take a look at Miguel's answer.
I used servers socket.io js file from reverse proxy by adding socket.io.js end of reverse proxy path like this xxx.com/reverse_proxy_path/socket.io
Related
In the supplied source code,
The client-side has two event-handlers: connect and server_to_client. When the page loads for the first time, the connect emits texts: hello! and world!.
Then, on the server side, the function server_to_client() received this message, prints it on the console, and subsequently emits another message received from the server to the client.
Finally, on the client side, the event-handler server_to_client prints the server side message to an H2-tag.
As you can see, two functions are working at the client side, and only one function is working at the server side.
On the server side, the same function is handling client_to_server event and raising server_to_client event.
Say, I want to use two different functions on the server side. I.e. one function will print hello! world!, and a different function will emit received from the server to the client.
Is it possible?
server_sends_client_receives.html
<!DOCTYPE HTML>
<html>
<head>
<title>Long task</title>
<script src="https://code.jquery.com/jquery-3.6.3.js"></script>
<script src="https://cdn.socket.io/4.5.4/socket.io.min.js"></script>
<script type="text/javascript" charset="utf-8">
$(document).on('click', '.widget input', function (event) {
namespace = '/test';
var socket = io(namespace);
socket.on('connect', function() {
$('#messages').append('<br/>' + $('<div/>').text('Requesting task to run').html());
////myText = $("messages").text()
socket.emit('client_to_server', {'hello': 'hello!', 'world': 'world!'});
});
socket.on('server_to_client', function(msg, cb) {
$('#messages').text(msg.data);
if (cb)
cb();
});
event.preventDefault();
});
</script>
</head>
<body>
<div class="widget">
<input type="submit" value="Click me" />
</div>
<h3>Messages</h3>
<H2 id="messages" >xxx yyy zzz</H2>
</body>
</html>
server_sends_client_receives.py
from flask import Flask, render_template
from flask_socketio import SocketIO, emit, disconnect
from time import sleep
async_mode = None
app = Flask(__name__)
socketio_obj = SocketIO(app, async_mode=async_mode)
#app.route('/')
def index():
return render_template('server_sends_client_receives.html',
sync_mode=socketio_obj.async_mode)
#socketio_obj.on('client_to_server', namespace='/test')
def server_to_client(arg1):
try:
print('received from the client : ', arg1['hello'])
print('received from the client : ', arg1['world'])
emit('server_to_client', {'data': 'received from the server'})
disconnect()
except Exception as ex:
print(ex)
if __name__ == '__main__':
socketio_obj.run(app, host='0.0.0.0', port=80, debug=True)
Remove the disconnect() call, which terminates the connection with the client.
Define a separate handler that listens to 'client_to_server2' event:
#socketio_obj.on('client_to_server', namespace='/test')
def server_to_client(arg1):
try:
print('received from the client : ', arg1['hello'])
print('received from the client : ', arg1['world'])
except Exception as ex:
print(ex)
#socketio_obj.on('client_to_server2', namespace='/test')
def server_to_client2():
try:
emit('server_to_client', {'data': 'received from the server'})
except Exception as ex:
print(ex)
Emit 'client_to_server2' event from client:
socket.emit('client_to_server', {'hello': 'hello!', 'world': 'world!'});
socket.emit('client_to_server2');
It appears that Flask assumes that the server is returning html to the client (browser).
Here's a simple example;
import json
from flask import Flask
app = Flask(__name__)
#app.route("/")
def home():
msg = ['Hello, world!']
return json.dumps(msg) + '\n'
This code works as expected and returns the desired json;
$ curl -s http://localhost:5000/
["Hello, world!"]
But if I introduce an error;
import json
from flask import Flask
app = Flask(__name__)
#app.route("/")
def home():
msg = ['Hello, world!']
return json.dumps(XXmsg) + '\n'
Then Flask emits the error wrapped in several pages worth of html, starting like;
$ curl -s http://localhost:5000/
<!DOCTYPE html>
<html>
<head>
<title>NameError: name 'XXmsg' is not defined
// Werkzeug Debugger</title>
<link rel="stylesheet" href="?__debugger__=yes&cmd=resource&f=style.css">
<link rel="shortcut icon"
href="?__debugger__=yes&cmd=resource&f=console.png">
<script src="?__debugger__=yes&cmd=resource&f=debugger.js"></script>
<script>
var CONSOLE_MODE = false,
EVALEX = true,
EVALEX_TRUSTED = false,
SECRET = "Mq5TSy6QE4OuOHUfvk8b";
</script>
</head>
<body style="background-color: #fff">
<div class="debugger">
Emitting html makes sense if you're creating a page load app. But I'm creating an api that only returns json.
Is there anyway to prevent Flask from emitting html at all?
Thanks
Mike
Have a look at the section Returning API Errors as JSON of the Flask docs.
Basically, you have to replace the default error handler with a function that returns the error as json. A very basic example:
#app.errorhandler(HTTPException)
def handle_exception(exception):
response = exception.get_response()
response.content_type = "application/json"
response.data = json.dumps({"code": exception.code})
return response
The accepted response gives a good hint for handling HTTPException but it won't work for all exceptions unless you create a handler for the mother of all exceptions:Exception. And you might not want to do this for security reasons, if you have some custom defined exceptions with sensible data it'll get handled by this handler.
I suspect the true reason you have those lengthy html responses is because you started your flask app with the --debug option.
I'm building a little server that sends a page with a script in javascript, it works when I try to open it in my browser, but if I request the page from the server, the page is recived, but not the script, so I get these errors:
Script.js is missing between the files:
Looks strange because from the network session i can see a request for the script with the status 200:
The js file i'm tryng to add is Chart.js, so I can't add it internally, it would become impossible to work with it, but for now the server is just a few lines of code in python that use the SimpleHTTPRequestHandler, and I'm probably going to replace it, may it be because the SimpleHTTPRequestHandler can't handle multiple file requests?
Here's the code, tried to make a snippet but it does't work there too (probably that's just me never wrote a snippet before):
HTML:
<!doctype html>
<html>
<head>
<script src="script.js"></script>
</head>
<body>
<p id = "paragraph"></p>
<script>
document.getElementById('paragraph').innerHTML = sayHello();
</script>
</body>
</html>
JS:
function sayHello(){
return "HelloWorld!"
}
Here is the python server script:
from http.server import HTTPServer, BaseHTTPRequestHandler
class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.end_headers()
with open("index.html", "r") as page:
self.wfile.write(page.read())
httpd = HTTPServer(("192.168.1.100", 8000), SimpleHTTPRequestHandler)
httpd.serve_forever()
I think you get an element with id, tag name or class Name and add file
var head = document.getElementsByTagName('head')[0];
var script = document.createElement('script');
script.type = 'text/javascript';
script.onload = function() {
callFunctionFromScript();
}
script.src = 'path/to/your-script.js';
head.appendChild(script);
also check this link
I am trying to access an external url https://data.dublinked.ie/cgi-bin/rtpi/realtimebusinformation?stopid=184&format=json in a Flask view
I get the error,
Not Found
The requested URL was not found on the server. If you entered the URL
manually please check your spelling and try again.
Is that my local server that flask is looking for this url on. And if so why? I am running flask locally.
The view, services.py
from flask import Flask, Response
import json
import urllib2
app = Flask(__name__)
#app.route('/')
def test():
return 'Everything is running!'
#app.route('/stopid')
def stopid():
dublin_bus_url = "https://data.dublinked.ie/cgi-bin/rtpi/realtimebusinformation?stopid=184&format=json"
response = urllib2.urlopen(dublin_bus_url)
json_response = json.load(response)
routes = set()
for result in json_response["results"]:
routes.add(result["route"])
return json.dumps(list(routes))
if __name__ == '__main__':
app.run()
The index.html and script is,
<!DOCTYPE html>
<html>
<head>
<script src="http://d3js.org/d3.v3.min.js" charset="utf-8"></script>
</head>
<body>
<script>
d3.json("/stopid", function(error, routes) {
routes.forEach(function(route) {
console.log(route)
});
});
</script>
</body>
</html>
I am new to Flask but this must not be the way to deal with an external link in a view.
The code above is adopted from this excellent tutorial for the Donorschoose api.
https://youtu.be/bzl4hCH2CdY
https://github.com/VidyaSource/starting-with-data
Thanks,
If we assume that the HTML file is not being served by flask:
You need to enable Cross origin resource sharing. You can do this by creating a response and setting it's header Access-Control-Allow-Origin to *: that is everyone. Or you can set it to your own domain when deploying.
resp.headers['Access-Control-Allow-Origin'] = '*'
Also, you're calling d3.json("/stopid" ... you need to change this to:
d3.json("http://localhost:5000/stopid" ...
Complete code:
from flask import Flask, Response, jsonify
import json
import urllib2
app = Flask(__name__)
#app.route('/')
def test():
return 'Everything is running!'
#app.route('/stopid')
def stopid():
dublin_bus_url = "https://data.dublinked.ie/cgi-bin/rtpi/realtimebusinformation?stopid=184&format=json"
my_response = urllib2.urlopen(dublin_bus_url)
json_response = json.load(my_response)
routes = set()
for result in json_response["results"]:
routes.add(result["route"])
resp = jsonify(list(routes))
resp.headers['Access-Control-Allow-Origin'] = '*'
return resp
if __name__ == '__main__':
app.run()
If the HTML is being served by flask, there is no need to enable cross origin sharing.
#app.route('/d3')
def d3():
return render_template('d3.html')
Call the link to this url using:
d3.json("{{ url_for('stopid') }}", ...
But this isn't exactly reliable, because you don't want to use the api using javascript when you can do it in flask itself.
I'm trying to build a simple web socket server that loads a file with some tweets in it (as CSV) and then just sends the string of the tweet to a web browser through a websocket. Here is a gist with the sample that I'm using for testing. Here's the Autobahn server component (server.py):
import random
import time
from twisted.internet import reactor
from autobahn.websocket import WebSocketServerFactory, \
WebSocketServerProtocol, \
listenWS
f = open("C:/mypath/parsed_tweets_sample.csv")
class TweetStreamProtocol(WebSocketServerProtocol):
def sendTweet(self):
tweet = f.readline().split(",")[2]
self.sendMessage(tweet, binary=False)
def onMessage(self, msg, binary):
self.sendTweet()
if __name__ == '__main__':
factory = WebSocketServerFactory("ws://localhost:9000", debug = False)
factory.protocol = TweetStreamProtocol
listenWS(factory)
reactor.run()
And here is the web component (index.html):
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<script type="text/javascript">
var ws = new WebSocket("ws://localhost:9000");
ws.onmessage = function(e) {
document.getElementById('msg').textContent = e.data; //unescape(encodeURIComponent(e.data));
console.log("Got echo: " + e.data);
}
</script>
</head>
<body>
<h3>Twitter Stream Visualization</h3>
<div id="msg"></div>
<button onclick='ws.send("tweetme");'>
Get Tweet
</button>
</body>
</html>
When the tweet arrives in the browser, the UTF-8 characters aren't properly displayed. How can I modify these simple scripts to display the proper UTF-8 characters in the browser?
This works for me:
from autobahn.twisted.websocket import WebSocketServerProtocol, \
WebSocketServerFactory
class TweetStreamProtocol(WebSocketServerProtocol):
def sendTweets(self):
for line in open('gistfile1.txt').readlines():
## decode UTF8 encoded file
data = line.decode('utf8').split(',')
## now operate on data using Python string functions ..
## encode and send payload
payload = data[2].encode('utf8')
self.sendMessage(payload)
self.sendMessage((u"\u03C0"*10).encode("utf8"))
def onMessage(self, payload, isBinary):
if payload == "tweetme":
self.sendTweets()
if __name__ == '__main__':
import sys
from twisted.python import log
from twisted.internet import reactor
log.startLogging(sys.stdout)
factory = WebSocketServerFactory("ws://localhost:9000", debug = False)
factory.protocol = TweetStreamProtocol
reactor.listenTCP(9000, factory)
reactor.run()
Notes:
above code is for Autobahn|Python 0.7 and above
I'm not sure if you sample Gist is properly UTF8 encoded file
However, the "last" pseudo Tweet is 10x "pi", and that properly shows in the browser, so
it works in principle ..
Also note: for reasons too long to explain here, Autobahn's sendMessage function expects payload to be already UTF8 encoded if isBinary == False. A "normal" Python string is Unicode, which needs to be encoded like above to UTF8 for sending.
instead of <meta http-equiv="content-type" content="text/html; charset=UTF-8">< try <meta charset = utf-8>.
if you're using XHTML then write <meta charset = utf-8 />