We Keep Coding

Python socketio badnamespaceerror exception on one device but not working on another

Hi all I have the following code but for some reason I keep getting the following error but it seems to work on a colleagues pc. We can't seem to figure out why this won't work on mine.
We have also double checked that we're importing the same socketio using dir()
I've tried specifying the namespace both on sio.connect and in the sio.emit but still no luck!
socketio.exceptions.BadNamespaceError: / is not a connected namespace.
bearerToken = 'REDACT'
core = 'REDACT'
output = 'REDACT'
import socketio
import json
def getListeners(token, coreUrl, outputId):
sio = socketio.Client(reconnection_attempts=5, request_timeout=5)
sio.connect(url=coreUrl, transports='websocket')
#sio.on('mwedge:batch:stats')
def batchStats(data):
if (outputId in data['outputStats']):
listeners = data['outputStats'][outputId][16]
print("Number of listeners ", len(listeners))
ips = []
for listener in listeners:
ips.append(listener[1])
print("Ips", ips)
def authCallback(data):
print(json.dumps(data))
sio.emit(event='auth',
data={
'token': token
},
callback=authCallback)
getListeners(bearerToken, core, output)

The Socket.IO connection involves a number of exchanges between the client and the server. The connect() function initiates this process, but this continues in the background. The connection ends when the handler for your connect event is invoked. At this point you can emit.
The problem with your code is that you are not waiting until the connection handshakes are completed, so your emit() call happens before there is a connection established. The solution is to add a connect event handler, and move your emit() call there.
As an additional note, I suggest you set up your event handlers before you call the connect() function.

gRPC-python: switching Servicer while gRPC server is running ? (simulation - real mode switching)

For our new, open lab device automation standard (https://gitlab.com/SiLA2/sila_python) we would like to run the devices (=gRPC servers) in two modes: a simulation mode and a real mode (with the same set of remote calls, but in the first case it shall just return simulated responses in the second it should communicate with the hardware.
My first idea was to create two almost identical gRPC servicer python classes in separated modules, like in this example:
in hello_sim.py:
class SayHello(SayHello_pb2_grpc.SayHelloServicer):
#... implementation of the simulation servicer
def SayHello(self, request, context):
# simulation code ...
return SayHello_pb2.SayHello_response("simulation")
in hello_real.py:
class SayHello(SayHello_pb2_grpc.SayHelloServicer):
#... implementation of the real servicer
def SayHello(self, request, context):
# real hardware code
return SayHello_pb2.SayHello_response("real")
and then, after creating the gRPC server in server.py I could switch between simulation and real mode by re-registration of the servcier at the gRPC server like, e.g.:
server.py
# imports, init ...
grpc_server = GRPCServer(ThreadPoolExecutor(max_workers=10))
sh_sim = SayHello_sim.SayHello()
sh_real = SayHello_real.SayHello()
SayHello_pb2_grpc.add_SayHelloServicer_to_server(sh_sim, grpc_server)
grpc_server.run()
# ..... and later, still while the same grpc server is running, re-register, like
SayHello_pb2_grpc.add_SayHelloServicer_to_server(sh_real, grpc_server)
to be able to call the real hardware code;
or by exchanging the reference to the servicer object, like, e.g.:
# imports, init ...
grpc_server = GRPCServer(ThreadPoolExecutor(max_workers=10))
sh_sim = SayHello_sim.SayHello()
sh_real = SayHello_real.SayHello()
sh_current = sh_sim
SayHello_pb2_grpc.add_SayHelloServicer_to_server(sh_current , grpc_server)
grpc_server.run()
# ..... and then later, still while the same grpc server is running, re-register the real Servicer, like
sh_current = sh_real
# so that the server would just use the other servicer object for the next calls ...
but both strategies are not working :(
When calling the server in simulation mode from a gRPC client, I would expect that it should reply (according to the example): "simulation"
gRPC_client.py
# imports, init ....
response = self.SayHello_stub.SayHello()
print(response)
>'simulation'
and after switching to real-mode (by any mechanism) "real":
# after switching to real mode ...
response = self.SayHello_stub.SayHello()
print(response)
>'real'
What is the most clean and elegant solution to achieve this mode switching without completely shutting down the gRPC server (and by that loosing the connection to the client) ?
Thanks a lot for your help in advance !
PS:(Shutting down the gRPC server and re-registering would of course work, but this is not, what we want.)

A kind colleague of mine provided me with a good suggestion:
One could use the dependency injection concept (s. https://en.wikipedia.org/wiki/Dependency_injection), like:
class SayHello(pb2_grpc.SayHelloServicer):
def inject_implementation(SayHelloImplmentation):
self.implementation = SayHelloImplementation;
def SayHello(self, request, context):
return self.implementation.sayHello(request, context)
for a full example, see
https://gitlab.com/SiLA2/sila_python/tree/master/examples/simulation_real_mode_demo

(flask + socket.IO) Result of emit callback is the response of my REST endpoint

Just to give a context here, I'm a node.JS developer, but I'm on a project that I need to work with Python using Flask framework.
The problem is, when a client request to an endpoint of my rest flask app, I need to emit an event using socket.IO, and get some data from the socket server, then this data is the response of the endpoint. But I didn't figured out how to send this, because flask needs a "return" statement saying what is the response, and my callback is in another context.
Sample of what I'm trying to do: (There's some comments explaining)
import socketio
import eventlet
from flask import Flask, request
sio = socketio.Server()
app = Flask(__name__)
#app.route('/test/<param>')
def get(param):
def ack(data):
print (data) #Should be the response
sio.emit('event', param, callback=ack) # Socket server call my ack function
#Without a return statement, the endpoint return 500
if __name__ == '__main__':
app = socketio.Middleware(sio, app)
eventlet.wsgi.server(eventlet.listen(('', 8000)), app)
Maybe, the right question here is: Is this possible?

I'm going to give you one way to implement what you want specifically, but I believe you have an important design flaw in this, as I explain in a comment above. In the way you have this coded, your socketio.Server() object will broadcast to all your clients, so will not be able to get a callback. If you want to emit to one client (hopefully not the same one that sent the HTTP request), then you need to add a room=client_sid argument to the emit. Or, if you are contacting a Socket.IO server, then you need to use a Socket.IO client here, not a server.
In any case, to block your HTTP route until the callback function is invoked, you can use an Event object. Something like this:
from threading import Event
from flask import jsonify
#app.route('/test/<param>')
def get(param):
ev = threading.Event()
result = None
def ack(data):
nonlocal result
nonlocal ev
result = {'data': data}
ev.set() # unblock HTTP route
sio.emit('event', param, room=some_client_sid, callback=ack)
ev.wait() # blocks until ev.set() is called
return jsonify(result)

I had a similar problem using FastAPI + socketIO (async version) and I was stuck at the exact same point. No eventlet so could not try out the monkey patching option.
After a lot of head bangings it turns out that, for some reason, adding asyncio.sleep(.1) just before ev.wait() made everything work smoothly. Without that, emitted event actually never reach the other side (socketio client, in my scenario)

Python: Making a Flask Rest API Asynchronous and Deploying it

I have a python server that is currently keeping track of the location of all the buses in my university and generating predictions of arrivals to specific locations.
Now, I wanted to attach a lightweight REST API to this server but I have been running intro problems.
I tried using flask with the following code:
from flask import Flask, jsonify
from PredictionWrapper import *
import threading
class RequestHandler():
def __init__(self,predictionWrapper):
self.app = Flask(__name__)
self.predictor = predictionWrapper
self.app.debug = False
self.app.add_url_rule('/<route>/<int:busStop>','getSinglePrediction',self.getSinglePrediction)
t = threading.Thread(target=self.app.run, kwargs={'host':'0.0.0.0', 'port':80, 'threaded':True})
t.start()
def getSinglePrediction(self, route, busStop):
# TODO Get the actual prediction with given parameters
prediction = self.predictor.getPredictionForStop(route, busStop)
return jsonify({'busStop': busStop, 'prediction': prediction})
def getStopPrediction(self, busStop):
# TODO Get the actual prediction with given parameters
return jsonify({'busStop': busStop, 'prediction': 2})
def run(self):
self.app.run(host='0.0.0.0', port=80, threaded=True)
The problem is that I have been encountering the error below after about half a day of running the server. Note that no requests were made to the server around the time it failed with the following error:
ERROR:werkzeug: - - [01/May/2016 09:55:55] code 400, message Bad request syntax ('\x02\xfd\xb1\xc5!')
After investigating I believe I need to deploy to a WSGI production server. But I have no clue what it means in this specific approach given that 1)the flask server is being threaded in order to run the rest of the prediction application, and 2)I am using classes which none of the documentation uses.
Any help on how to setup the wsgi file with apache, gunicorn, or the technology of your choice would be appreciated. Also, any comments on a better approach on making a non-blocking REST API would be helpful.
Let me know if you need any further clarification!

Not sure if this can actually solve your problem but you can use the coroutine based web server gevent. They have a WSGI server that you can use if that's what you meant by saying that you need to deploy a WSGI production server.
If you want to implement the server to your flask application just do the following:
from gevent.pywsgi import WSGIServer
app = Flask(__name__)
http_server = WSGIServer(('', 5000), app)
http_server.serve_forever()
Gevent in general is a very powerful tool and by issuing context switches as necessary it can handle multiple clients very easily. Also, gevent fully supports flask.

First thing to do would be to put exception handling to deal with bad JSON request data (which maybe is what's happening) something like
def getSinglePrediction(self, route, busStop):
try:
prediction = self.predictor.getPredictionForStop(route, busStop)
return jsonify({'busStop': busStop, 'prediction': prediction})
except:
return jsonify({'busStop': 'error', 'prediction': 'error'})

Websocket/event-source/... implementation to expose a two-way RPC to a python/django application

for a django application I'm working on, I need to implement a two ways RPC so :
the clients can call RPC methods from the platform and
the platform can call RPC methods from each client.
As the clients will mostly be behind NATs (which means no public IPs, and unpredictable weird firewalling policies), the platform to client way has to be initiated by the client.
I have a pretty good idea on how I can write this from scratch, I also think I can work something out of the publisher/subscriber model of twisted, but I've learned that there is always a best way to do it in python.
So I'm wondering what would be the best way to do it, that would also integrate the best to django. The code will have to be able to scope with hundreds of clients in short term, and (we hope) with thousands of clients in medium/long term.
So what library/implementation would you advice me to use ?
I'm mostly looking to starting points for RTFM !

websocket is a moving target, with new specifications from time to time. Brave developpers implements server side library, but few implements client side. The client for web socket is a web browser.
websocket is not the only way for a server to talk to a client, event source is a simple and pragmatic way to push information to a client. It's just a never ending page. Twitter fire hose use this tricks before its specification. The client open a http connection and waits for event. The connection is kept open, and reopen if there is some troubles (connection cut, something like that).
No timeout, you can send many events in one connection.
The difference between websocket and eventsource is simple. Websocket is bidirectionnal and hard to implement. Eventsource is unidirectionnal and simple to implement, both client and server side.
You can use eventsource as a zombie controller. Each client connects and reconnect to the master and wait for instruction. When instruction is received, the zombie acts and if needed can talk to its master, with a classical http connection, targeting the django app.
Eventsource keep the connection open, so you need an async server, like tornado. Django need a sync server, so, you need both, with a dispatcher, like nginx. Django or a cron like action talks to the async server, wich talks to the right zombie. Zombie talks to django, so, the async server doesn't need any peristance, it's just a hub with plugged zombies.
Gevent is able to handle such http server but there is no decent doc and examples for this point. It's a shame. I want a car, you give me a screw.

You can also use Tornado + Tornadio + Socket.io. That's what we are using right now for notifications, and the amount of code that you should write is not that much.
from tornadio2 import SocketConnection, TornadioRouter, SocketServer
class RouterConnection(SocketConnection):
__endpoints__ = {'/chat': ChatConnection,
'/ping': PingConnection,
'/notification' : NotificationConnection
}
def on_open(self, info):
print 'Router', repr(info)
MyRouter = TornadioRouter(RouterConnection)
# Create socket application
application = web.Application(
MyRouter.apply_routes([(r"/", IndexHandler),
(r"/socket.io.js", SocketIOHandler)]),
flash_policy_port = 843,
flash_policy_file = op.join(ROOT, 'flashpolicy.xml'),
socket_io_port = 3001,
template_path=os.path.join(os.path.dirname(__file__), "templates/notification")
)
class PingConnection(SocketConnection):
def on_open(self, info):
print 'Ping', repr(info)
def on_message(self, message):
now = dt.utcnow()
message['server'] = [now.hour, now.minute, now.second, now.microsecond / 1000]
self.send(message)
class ChatConnection(SocketConnection):
participants = set()
unique_id = 0
#classmethod
def get_username(cls):
cls.unique_id += 1
return 'User%d' % cls.unique_id
def on_open(self, info):
print 'Chat', repr(info)
# Give user unique ID
self.user_name = self.get_username()
self.participants.add(self)
def on_message(self, message):
pass
def on_close(self):
self.participants.remove(self)
def broadcast(self, msg):
for p in self.participants:
p.send(msg)

here is a really simple solution I could came up with :
import tornado.ioloop
import tornado.web
import time
class MainHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self):
self.set_header("Content-Type", "text/event-stream")
self.set_header("Cache-Control", "no-cache")
self.write("Hello, world")
self.flush()
for i in range(0, 5):
msg = "%d<br>" % i
self.write("%s\r\n" % msg) # content
self.flush()
time.sleep(5)
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
and
curl http://localhost:8888
gives output when it comes !
Now, I'll just have to implement the full event-source spec and some kind of data serialization between the server and the clients, but that's trivial. I'll post an URL to the lib I'll write here when it'll be done.

I've recently played with Django, Server-Sent Events and WebSocket, and I've wrote an article about it at http://curella.org/blog/2012/jul/17/django-push-using-server-sent-events-and-websocket/
Of course, this comes with the usual caveats that Django probably isn't the best fit for evented stuff, and both protocols are still drafts.