We Keep Coding

Unit testing using Tornado Websocket - no attribute 'io_loop' error

I have stitched together a tornado websocket client code and using it in my python unit test case. This is my first time use of tornado websocket and not very familiar with its unit test API. Looking for some help to understand the use of tornado websocket asynchronous unit test code and the below case working.
Client class code:
import logging
import logging.config
import ssl
import time
import traceback
from tornado.concurrent import Future
from tornado import gen
from tornado.httpclient import HTTPError, HTTPRequest
from tornado.log import gen_log, app_log
from tornado.web import Application, RequestHandler
class TorWebSocketClient():
def __init__(self, ip_addr, port, cookie):
self.ip_addr = ip_addr
self.port = port
self.cookie = cookie
self.sockConnected = False
self.logger = logging.getLogger(__name__)
def Connect(self):
# Creating the websocket client for each test case.
url = "ws://{0}:{1}/socket{2}".format(str(self.ip_addr), str(self.port), self.cookie)
self.logger.debug('Websocket URL: ' + url)
sslopt={"cert_reqs": ssl.CERT_NONE,
"check_hostname": False,
"ssl_version": ssl.PROTOCOL_TLSv1}
self.logger.debug('New web socket connection is being establshed by the client')
self.ws = websocket.websocket_connect(HTTPRequest(url, headers=headers, ssl_options=sslopt), io_loop=self.io_loop)
# Start the websocket client thread. A wait is added till connection is established.
self.sockConnected = True
def send(self, data):
# Wait till websocket is connected.
if not self.ws.sock.connected:
self.logger.debug('Send failed; Websocket connection is not yet established')
return
self.logger.info('Sending data to the server: ' + data)
self.ws.write_message(data)
def recv(self, expValues):
# Read data from the response message.
resp = yield self.ws.read_message()
print '>>>> Response: ', resp
def stop(self):
self.logger.debug('Client closing the websocket connection with the server')
self.ws.close()
Unit test function is below:
import functools
import json
import logging
import logging.config
import time
# These are couple of custom classes.
import TorWebSocketClient
from infra.serverbase import Server
from tornado.testing import AsyncHTTPTestCase, gen_test, bind_unused_port, ExpectLog
class TornadoTest(AsyncHTTPTestCase):
def get_app(self):
app = tornado.web.Application([('/', EchoWebSocketHandler)])
return app
#gen_test
def testToradoWSConection(self):
# Login to the server to get the cookie.
logger = logging.getLogger(__name__)
server = Server(self.ipaddr, self.port, self.username, self.password)
result = server.Login()
self.assertEqual(result, True, 'login failed')
webSocClient = yield TorWebSocketClient(self.ipaddr, self.port, server.GetCookie())
result = webSocClient.Connect()
self.assertEqual(result, True, 'Websocket connection failed')
Error I am getting:
Traceback (most recent call last):
File "/users/usr1/pyvenv/venv/lib/python2.7/site-packages/tornado/testing.py", line 527, in post_coroutine
return self.io_loop.run_sync(
AttributeError: TornadoTest instance has no attribute 'io_loop'
----------------------------------------------------------------------
Ran 1 tests in 0.002s
FAILED (errors=1)

Did you have your own setUp function?
The io_loop is created under AsyncTestCase's setUp function, I think you need to call super's setUp function.

tornado - WebSocketClientConnection - How to catch and handle connection failures?

I am a Tornado and also Websocket newbie. There are many resources how to implement a websocket server application with Tornado. However, I haven't found a complete example that contains a websocket client application built on top of Tornado.
Server Application (server.py)
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import tornado.web
import tornado.websocket
import tornado.ioloop
import tornado.options
from tornado.options import define, options
define("port", default=3000, help="run on the given port", type=int)
class Application(tornado.web.Application):
def __init__(self):
handlers = [(r"/", MainHandler)]
settings = dict(debug=True)
tornado.web.Application.__init__(self, handlers, **settings)
class MainHandler(tornado.websocket.WebSocketHandler):
def check_origin(self, origin):
return True
def open(self):
logging.info("A client connected.")
def on_close(self):
logging.info("A device disconnected")
def main():
tornado.options.parse_command_line()
app = Application()
app.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
Client Application (client.py):
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from tornado.ioloop import IOLoop
from tornado.websocket import websocket_connect
if __name__ == "__main__":
ws = websocket_connect("ws://localhost:3000")
IOLoop.instance().start()
Problems & Questions
I don't have any problems with server application. However, I can't say same thing for client application. Why?
If server goes down, client application doesn't react. It does not say "Hey, server is down buddy.". I write my own WebSocketClientConnection class (inherited by the original) and also websocket_connect function (same function, but this one uses my WebSocketClientConnection class). I override on_connection_close method in my WebSocketClientConnection class, so I can catch "server down on middle of the connection" failure. However, I cannot make it reconnect. How do I make it reconnect to server?
If server is already down before the connection, client application does not raise anything and seems perfectly. I don't know how to catch that failure. How do I catch this connection failure?
My WebSocketClientConnection class:
class MyWebSocketClientConnection(tornado.websocket.WebSocketClientConnection):
def on_connection_close(self):
super(MyWebSocketClientConnection, self).on_connection_close()
print "connection closed"
Thanks.

By the way, after I figured out, I did write an example Tornado WebSocket client/server pair to demonstrate how to do it.
https://github.com/ilkerkesen/tornado-websocket-client-example
I hope it helps someone.

websocket_connect returns a Future; to see whether it succeeded or failed you must examine the Future (probably by yielding it in a coroutine). Furthermore, after establishing the connection you should go into a read_message loop. Even if you don't expect the client to send any messages, you should still call read_message: this is how you will connections that are closed after being established (read_message will return None).
#gen.coroutine
def my_websocket_client(url):
ws = yield websocket_connect(url)
while True:
msg = yield ws.read_message()
if msg is None: break
# do stuff with msg

Websockets with Tornado: Get access from the "outside" to send messages to clients

I'm starting to get into WebSockets as way to push data from a server to connected clients. Since I use python to program any kind of logic, I looked at Tornado so far. The snippet below shows the most basic example one can find everywhere on the Web:
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
class WSHandler(tornado.websocket.WebSocketHandler):
def open(self):
print 'new connection'
self.write_message("Hello World")
def on_message(self, message):
print 'message received %s' % message
self.write_message('ECHO: ' + message)
def on_close(self):
print 'connection closed'
application = tornado.web.Application([
(r'/ws', WSHandler),
])
if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().start()
As it is, this works as intended. However, I can't get my head around how can get this "integrated" into the rest of my application. In the example above, the WebSocket only sends something to the clients as a reply to a client's message. How can I access the WebSocket from the "outside"? For example, to notify all currently connected clients that some kind event has occured -- and this event is NOT any kind of message from a client. Ideally, I would like to write somewhere in my code something like:
websocket_server.send_to_all_clients("Good news everyone...")
How can I do this? Or do I have a complete misundersanding on how WebSockets (or Tornado) are supposed to work. Thanks!

You need to keep track of all the clients that connect. So:
clients = []
def send_to_all_clients(message):
for client in clients:
client.write_message(message)
class WSHandler(tornado.websocket.WebSocketHandler):
def open(self):
send_to_all_clients("new client")
clients.append(self)
def on_close(self):
clients.remove(self)
send_to_all_clients("removing client")
def on_message(self, message):
for client in clients:
if client != self:
client.write_message('ECHO: ' + message)

This is building on Hans Then's example. Hopefully it helps you understand how you can have your server initiate communication with your clients without the clients triggering the interaction.
Here's the server:
#!/usr/bin/python
import datetime
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
class WSHandler(tornado.websocket.WebSocketHandler):
clients = []
def open(self):
print 'new connection'
self.write_message("Hello World")
WSHandler.clients.append(self)
def on_message(self, message):
print 'message received %s' % message
self.write_message('ECHO: ' + message)
def on_close(self):
print 'connection closed'
WSHandler.clients.remove(self)
#classmethod
def write_to_clients(cls):
print "Writing to clients"
for client in cls.clients:
client.write_message("Hi there!")
application = tornado.web.Application([
(r'/ws', WSHandler),
])
if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().add_timeout(datetime.timedelta(seconds=15), WSHandler.write_to_clients)
tornado.ioloop.IOLoop.instance().start()
I made the client list a class variable, rather than global. I actually wouldn't mind using a global variable for this, but since you were concerned about it, here's an alternative approach.
And here's a sample client:
#!/usr/bin/python
import tornado.websocket
from tornado import gen
#gen.coroutine
def test_ws():
client = yield tornado.websocket.websocket_connect("ws://localhost:8888/ws")
client.write_message("Testing from client")
msg = yield client.read_message()
print("msg is %s" % msg)
msg = yield client.read_message()
print("msg is %s" % msg)
msg = yield client.read_message()
print("msg is %s" % msg)
client.close()
if __name__ == "__main__":
tornado.ioloop.IOLoop.instance().run_sync(test_ws)
You can then run the server, and have two instances of the test client connect. When you do, the server prints this:
bennu#daveadmin:~$ ./torn.py
new connection
message received Testing from client
new connection
message received Testing from client
<15 second delay>
Writing to clients
connection closed
connection closed
The first client prints this:
bennu#daveadmin:~$ ./web_client.py
msg is Hello World
msg is ECHO: Testing from client
< 15 second delay>
msg is Hi there! 0
And the second prints this:
bennu#daveadmin:~$ ./web_client.py
msg is Hello World
msg is ECHO: Testing from client
< 15 second delay>
msg is Hi there! 1
For the purposes of the example, I just had the server send the message to the clients on a 15 second delay, but it could be triggered by whatever you want.

my solution for this: first add "if __name__ == '__main__':" - to the main.py. then import main.py into the websocket module. e.g. (import main as MainApp) . it is now possible to call a function in 'main.py' from within the ws.py/WebSocketHandler-function. - inside the Handler pass the message like so:
MainApp.function(message)
i dunno if this is the opposite of elegant but it works for me.
..plus create and import a custom 'config.py' (thats looks like: someVar = int(0) ) into the 'mainApp.py' .. like so: import config as cfg --> now you can alter variables with cfg.someVar = newValue from inside the function in 'main.py' that once was called by the Handler from 'ws.py'.

Implement SSL in tornadoweb websockets server

Good Day to all,
I have a websocket server written in python that serves clients changes done to a postgresql database table. Presently it uses the standard ws protocol. I would Like to implement SSL wss but there seems to be no documentation on the tornadoweb site. Any help appreciated!
Code as follows:
import tornado.web
import tornado.websocket
import tornado.ioloop
import tornado.httpserver
import threading
import select
import psycopg2
import psycopg2.extensions
# This is a global variable to store all connected clients
websockets = []
# Connect to DB to listen for notifications
conn = psycopg2.connect("host=xxx.xxx.xxx.xxx dbname=mydb user=xxx port=yyy")
conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
curs = conn.cursor()
curs.execute("LISTEN notifier;")
# Create the websocket
class WebSocketHandler(tornado.websocket.WebSocketHandler):
#tornado.web.asynchronous
# Add to global variable the connected client
def open(self):
self.set_nodelay(True)
# on disconnect remove client
def on_close(self):
# Search for it in the list. If it matches with us (the client) then remove it as he has quit the con
try:
for websocket in websockets:
if websocket[0] == self:
websockets.remove(websocket)
except ValueError:
print ValueError
def on_message(self, message):
if self not in websockets:
websockets.append([self,message])
def on_pong(self, data):
print data
# This is the function that polls the database for changes, then it sends change to clients through websockets
def db_listener():
while 1:
if select.select([conn],[],[],5) == ([],[],[]):
for websocket in websockets:
# ping the client regularly to avoid disconnect
websocket[0].ping("ping")
else:
conn.poll()
while conn.notifies:
notify = conn.notifies.pop()
details = notify.payload.split(",")
if len(details) > 1:
for websocket in websockets:
if details[34] in websocket[1]:
websocket[0].write_message(notify.payload)
application = tornado.web.Application([
(r"/websocket", WebSocketHandler),
])
if __name__ == "__main__":
# Start a separate thread for every client so that we do not block the main websockets program!
threading.Thread(target=db_listener).start()
application.listen(5252)
tornado.ioloop.IOLoop.instance().start()
Any suggestions?
Thanks to all!!

try this:
server = tornado.httpserver.HTTPServer(application, ssl_options = {
"certfile": "path-to-crt-file",
"keyfile": "path-to-key-file",
})
server.listen(5252)

Tornado server stuck in loop, not accepting clients

I've written a code in tornado that connects to a server that is pushing an infinite data stream, processes the data stream and sends it out on a websocket server.
The problem is that the way I implemented it the server became blocked on a particular function and doesn't accept any more clients since it never exits the function serving the data to the websocket. I want the connection to the server and the data retrieved from it processed only once but send the processed data to all the clients that connect to my tornado server. Could someone please help me, I can't figure out a way to do it. Here's my code with processing of data removed:
import socket
import ssl
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
websockets = []
class WSHandler(tornado.websocket.WebSocketHandler):
def readData(self):
while True:
line = self.ssl_sock.read()
#PROCESS THE READ LINE AND CONVERT INTO RESULTING DATA
if(toSend):
self.write_message(result)
def makeConnection(self):
self.ssl_sock.connect(self.address)
self.readData()
def open(self):
print 'New connection was opened'
self.s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.ssl_sock=ssl.wrap_socket(self.s, cert_reqs=ssl.CERT_NONE)
self.address=('SERVER_ADDRESS',5000)
self.nodes=[]
self.edges=[]
if self not in websockets:
print ('added')
websockets.append(self)
if(len(websockets)==1):
print('executing make conn')
self.makeConnection()
else:
self.readData()
print('executing read data')
def on_message(self, message):
print 'Incoming message:', message
self.write_message("You said: " + message)
def on_close(self):
print 'Connection was closed...'
application = tornado.web.Application([
(r'/ws', WSHandler),
])
if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().start()

Tornado is an asynchronous framework, that is, all your IO must be run within its event loop, otherwise the whole server gets stuck.
Try having a look at Tornado Async Client.