I have one function orderbook(pair) that opens up a websocket, subscribes to a channel and listens for messages. I want to be able to subscribe to multiple channels and listen to messages in parallel. Is there any way to do this?
As of now, I am attempting the following:
from dev.orderbook import *
from multiprocessing import Process
def main():
process01 = Process(target=orderbook('BTC-USD'))
process02 = Process(target=orderbook('ETH-USD'))
process01.start()
process02.start()
if __name__ == '__main__':
main()
However, all this does is run process01 and only when I stop the script it runs process02. Which is not the behavior I am looking for. Is there any way I can subscribe to both these channels at once and listen and print messages/responses concurrently? If not, then is there any way I can start one process - let it run for a specified time and then start the next process?
My websocket mirrors the following:
import websocket
class orderbook(object):
def __init__(self, pair):
self.pair = pair
websocket.enableTrace(True)
ws = websocket.WebSocketApp("ws://echo.websocket.org/",
on_message = self.on_message,
on_error = self.on_error,
on_close = self.on_close)
ws.on_open = self.on_open
ws.run_forever()
def on_message(self, ws, message):
print(message)
def on_error(self, ws, error):
print(error)
def on_close(self, ws):
print("### closed ###")
def on_open(self, ws):
ws.send(self.pair)
Your Process constructors include actual calls to your target function. So, the function is called, in the parent process, in the course of evaluating the constructor line, and the result is passed to Process() as target.
You should rather pass the function itself, and pass its arguments separately:
process01 = Process(target=orderbook, args=('BTC-USD',))
Related
I am using the websocket library in Python and I am new to this.
I want to create multiple different connections to websockets. This happens through my custom WebsocketProcess class which opens the connection, receives the event, keeps a record Id and then calls an API to grab the information for this particular record.
I am having trouble running them in parallel.
Please, see below (ignore the numerous imports)
Main:
#if __name__ == "__main__":
async def main():
#The length AccessTokens, ClientDescriptions and SQLTablesFix determines how many websockets we need to open
L = await asyncio.gather(
properties[0].websocket_starting(AccessTokens[0], ClientDescriptions[0], SQLTablesFix[0]),
properties[1].websocket_starting(AccessTokens[1], ClientDescriptions[1], SQLTablesFix[1]),
...
...
)
asyncio.run(main())
The WebsocketProcess class is as follows:
class WebsocketProcess:
"""description of class"""
def on_error(self, ws, error):
#{Relevant Code Here}
def on_open(self, ws):
print("\nOn Open\n")
def run(*args):
while True:
try:
time.sleep(1)
except TimeoutError:
pass
ws.close()
def on_close(self):
#{Relevant Code Here}
def on_message(self, ws, message):
#{Relevant Code Here}
ws.close()
def connect_websocket(self, AccessToken, ClientDescription, SQLTablesFix):
ws = websocket.WebSocketApp("_______url_here_____",
on_open = self.on_open,
on_message = self.on_message,
on_error = self.on_error,
on_close = self.on_close,
cookie = "ClientToken=_______; AccessToken=%s" % AccessToken)
ws.run_forever()
async def websocket_starting(self, AccessToken, ClientDescription, SQLTablesFix):
print("\nwebsocket_starting")
self.AccessToken = AccessToken
self.ClientDescription = ClientDescription
self.SQLTablesFix = SQLTablesFix
self.connect_websocket(self.AccessToken, self.ClientDescription, self.SQLTablesFix)
As you can see from the above, I have changed the main to asynchronous to run multiple instances of the websocket_process class in parallel. It opens a connection to the first websocket and it stops there waiting for events, without proceeding to open a second websocket.
I tried making the WebsocketProcess class entirely asynchronous but the errors I am receiving an error specifying that coroutine 'run' was never awaited (in the connect_websocket method).
Do you guys have any suggestions on how to run multiple instances of the websocket_process class in parallel?
Thanks!
Your websocket operations are blocking operations, to use websocket in asyncio, use other async libraries like websockets, Tornado
I'm trying to implement a websocket client in python using websockets and the apparently mandatory asyncio which I never used before (and I have a hard time to understand...).
I've read a lot on the subject and saw (too) many examples here and everywhere, but I can't find a way to properly make a websocket client with a persistent connection.
I need to have a persistent connection because the commands need to be requested on the same connection, the first one being an authentication command.
The remote server is a 3rd party API I don't have any control over.
I suppose I could run an authentication request along with each command my program sends but that does not feel right to open > auth > request > close for each command instead of keeping one connection alive during the whole program's life
My implementation is a library using many classes and I need to wrap the websocket connector/handler in one of them
Here's what I have right now, based on examples I found here and there (with some obfuscated data) :
import json
import asyncio
from websockets import connect
URL = 'wss://server.com/endpoint'
class Websocket:
async def __aenter__(self):
self._conn = connect(URL)
self.websocket = await self._conn.__aenter__()
return self
async def __aexit__(self, *args, **kwargs):
await self._conn.__aexit__(*args, **kwargs)
async def send(self, message):
await self.websocket.send(message)
async def receive(self):
return await self.websocket.recv()
class Handler:
def __init__(self):
self.wws = Websocket()
self.loop = asyncio.get_event_loop()
def command(self, cmd):
return self.loop.run_until_complete(self.__async__command(cmd))
async def __async__command(self, cmd):
async with self.wws as echo:
await echo.send(json.dumps(cmd))
return await echo.receive()
def main():
handler = Handler()
foo = handler.command('authentication command')
print('auth: ', foo)
bar = handler.command('another command to run depending on the first authentication')
print('command: ', bar)
if __name__ == '__main__':
main()
Basically right now I get these answers (simplified and obfuscated) :
auth: Ok, authenticated
command: Command refused, not authenticated
I suppose my problem is that the block async with self.wws as echo: kind of create the connection, runs its code then drop it instead of keeping the connection alive. Since we are not using a usual __init__ here but some asyncio voodoo I don't understand, I'm kind of stuck.
I think your diagnosis is correct, the problem is that the async context manager it creating and closing a connection for each call of Handler.command ... really not want you want.
Instead you could just synchronously establish the websocket connection during the init of Handler and then store the connection websocket (instance of type WebSocketClientProtocol) as a class member for later use, as in this sample code:
import json
import asyncio
from websockets import connect
URL = 'ws://localhost:8000'
class Handler:
def __init__(self):
self.ws = None
self.loop = asyncio.get_event_loop()
# perform a synchronous connect
self.loop.run_until_complete(self.__async__connect())
async def __async__connect(self):
print("attempting connection to {}".format(URL))
# perform async connect, and store the connected WebSocketClientProtocol
# object, for later reuse for send & recv
self.ws = await connect(URL)
print("connected")
def command(self, cmd):
return self.loop.run_until_complete(self.__async__command(cmd))
async def __async__command(self, cmd):
await self.ws.send(json.dumps(cmd))
return await self.ws.recv()
def main():
handler = Handler()
foo = handler.command('authentication command')
print('auth: ', foo)
bar = handler.command('another command to run depending on the first authentication')
print('command: ', bar)
if __name__ == '__main__':
main()
I figure this is pretty basic, but can't seem to figure out even how to ask google the right question. I am using this python websocket client to make some websocket connections. Let's just assume I'm using the code example similar to that page:
import websocket
import thread
import time
def on_message(ws, message):
print(message)
def on_error(ws, error):
print(error)
def on_close(ws):
print("### closed ###")
def on_open(ws):
def run(*args):
ws.send("Hello")
time.sleep(1)
ws.close()
print("thread terminating...")
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp("ws://echo.websocket.org/",
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()
So what I am trying to do is add more arguments to the on_open function, something like this:
def on_open(ws, more_arg):
def run(*args):
ws.send("Hello %s" % more_arg)
time.sleep(1)
ws.close()
print("thread terminating...")
thread.start_new_thread(run, ())
But i can't figure out how to pass these arguments in, so I tried in the main thread:
ws.on_open = on_open("this new arg")
But i get the error:
TypeError: on_open() takes exactly 2 arguments (1 given)
How am I going to pass these new arguments to my on_open function?
Keep in mind that you need to assign a callback. You are instead calling a function and passing the return value to ws, which is incorrect.
You can use functools.partial to curry a function to a higher order one:
from functools import partial
func = partial(on_open, "this new arg")
ws.on_open = func
When func is invoked, it will invoke on_open with the first argument as "this new arg", followed by any other arguments passed to func. Look at the implementation of partial in the doclink for more details.
You can use a lambda to wrap the call:
ws.on_open = lambda *x: on_open("this new arg", *x)
First I should say I am completely new to asyncio or its' paradigm.
So I am trying implement asyncio (python 3.4) client with asyncio.open_connection() able to send requests on tcp socket (telnet) and read it's responses and listen to what other side may send in the same time.
In other words I need bi-directional communication which I am initializing, therefore I am client. Yet all examples I found so far using StreamReader and StreamWriter were forced to break loop after reading empty line. Also I need somehow deal with if incoming message (every message is exactly one line) is response to previouse request or original message of other side.
I was thinking something like this might work.
class MyHandler:
#asyncio.coroutine
def connect(self):
self.reader, self.writer = asyncio.open_connection('localhost', 2020)
while True:
msg = self.reader.readline()
if msg is None:
asyncio.sleep(1)
continue
self.handle_msg(msg)
#asyncio.coroutine
def request(self, msg):
self.writer.write(msg)
return self.reader.readline()
if __name__ == '__main__':
h = MyHandler()
loop = asyncio.get_event_loop()
loop.run_until_complete(h.connect)
loop.close()
I've written this code only during creating this question. Now that I start run it, it fails even before getting to main point. Somehow loop.run_until_complete(h.connect) fails with TypeError: A Future or coroutine is required
h.connect is a function, whereas h.connect() is a coroutine. this means the correct code would be:
class MyHandler:
#asyncio.coroutine
def connect(self):
self.reader, self.writer = asyncio.open_connection('localhost', 2020)
while True:
msg = yield from self.reader.readline() # Yield from since it's a coroutine
if msg.strip() is None:
yield from asyncio.sleep(1) # Also a coroutine
continue
self.handle_msg(msg)
#asyncio.coroutine
def request(self, msg):
self.writer.write(msg)
return (yield from self.reader.readline())
if __name__ == '__main__':
h = MyHandler()
loop = asyncio.get_event_loop()
loop.run_until_complete(h.connect())
loop.close()
I am currently developing a server program in Python that uses the websockets and asyncio packages.
I got a basic script handling websockets working (Exhibit A). This script locks when waiting for input, which is not what I want.
The solution for this that I imagine is I can start two asynchronous tasks - one that handles inputs and one that handles outputs - and start them in a secondary event loop. I had to do some research about coroutines, and I came up with Exhibit B as a proof of concept for running two things simultaneously in an event loop.
Now what I'm stuck on is Exhibit C. When I attempted to use this in a practical scenario with the websockets package, I found that websocket.recv() never finishes (or the coroutine never un-pauses - I'm not sure what's going on exactly). In exhibit A it works fine, and I've determined that the coroutine definitely runs at least up until that point.
Any ideas?
Exhibit A:
#!/usr/bin/python3
import asyncio
import websockets
import time
# This works great!
async def hello(websocket, path):
while True:
# This line waits for input from socket
name = await websocket.recv()
print("< {}".format(name))
# "echo... echo... echo... echo... echo..."
greeting = ''.join(name + "... " for x in range(5))
await websocket.send(greeting)
print("> {}".format(greeting))
time.sleep(0.1);
start_server = websockets.serve(hello, '', 26231)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Exhibit B:
#!/usr/bin/python3
import asyncio
import time
class Thing:
def __init__(self):
pass
async def test(self):
for x in range(10):
print("Hello"+str(x))
await asyncio.sleep(0)
def run(self):
# Add the task to the event loop twice
asyncio.ensure_future(self.test())
asyncio.ensure_future(self.test())
t = Thing()
t.run()
loop = asyncio.get_event_loop();
loop.run_forever()
Exhibit C:
#!/usr/bin/python3
import asyncio
import websockets
import time
class WebsocketRequest:
def __init__(self, websocket):
self.websocket = websocket
# Works great
async def handle_oputs(self):
# This works fine - sends a message
# every 10 seconds to the client
while True:
print("sending...")
await self.websocket.send("Hello")
print("> {}".format("Hello"))
time.sleep(10)
# Doesn't work
async def handle_iputs(self):
# This stops at the await and never receives
# any input from the client :/
while True:
try:
print("receiving...")
# This is the line that causes sadness
data = await self.websocket.recv()
print("< {}".format(data))
except:
# This doesn't happen either
print("Listener is dead")
async def run(self):
# This is the part where the coroutine for
# a client get split off into two of them
# to handle input and output separately.
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
asyncio.ensure_future(self.handle_iputs())
asyncio.ensure_future(self.handle_oputs())
loop.run_forever()
class WebsocketServer:
def __init__(self, address):
self.ip = address[0]
self.port = address[1]
async def hello(self, websocket, path):
req = WebsocketRequest(websocket)
await req.run()
def run(self):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
start_server = websockets.serve(self.hello, self.ip, self.port)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Maybe the module websocket (unlike websockets) can help you.
The use of WebsocketApp is very simple:
from websocket import WebSocketApp
class ExampleClass(object):
def __init__(self):
websocket.enableTrace(True)
self.ws = websocket.WebSocketApp("ws://echo.websocket.org",
on_message=on_message,
on_error=on_error,
on_close=on_close)
def on_message(ws, msg):
print "Message Arrived:" + msg
def on_error(ws, error):
print error
def on_close(ws):
print "Connection Closed"
def on_open(ws):
ws.send("Hello!")
To download this module: https://pypi.python.org/pypi/websocket-client
time.sleep() is a blocking operation, so any other tasks cannot interrupt and are not scheduled. Use await asyncio.sleep() instead.