We Keep Coding

Websocket buffer delayed Python

I
setting up a Websocket that receives market data from 33 pairs, process the data and insert it into a local mysql database.
what I've tried so far :
Setting up the websocket works fine, then process the data on each new message function and insert it directly into the database
--> problem was that with 33 pairs the websocket was stacking up the buffer with market data, and after a few minutes I would get a delay in the database of at least 10 seconds
Then I tried processing the data through a thread : the on_message function would execute a thread that is simply putting the market data into an array, like below
datas=[]
def add_queue(symbol,t,a,b,r_n):
global datas
datas.append([symbol,t,a,b,r_n])
if json_msg['ev']=="C":
symbol=json_msg['p'].replace("/","-")
round_number=pairs_dict_new[symbol]
t = Thread(target=add_queue, args=(symbol,json_msg['t'],json_msg['a'],json_msg['b'],round_number,))
t.start()
and then another function, with a loop thread would pick it up to insert it into the database
def add_db():
global datas
try:
# db = mysql.connector.connect(
# host="104.168.157.164",
# user="bvnwurux_noe_dev",
# password="Tickprofile333",
# database="bvnwurux_tick_values"
# )
while True:
for x in datas:
database.add_db(x[0],x[1],x[2],x[3],x[4])
if x in datas:
datas.remove(x)
except KeyboardInterrupt:
print("program ending..")
t2 = Thread(target=add_db)
t2.start()
still giving a delay, and the threaded process wasn't actually using a lot of CPU but more of RAM and it just was even worse.
instead of using a websocket with a thread, I tried simple webrequests to the API call, so with 1 thread per symbol, it would loop through a webrequest and in everythread send it to the database. my issues here were that mysql connections don't like threads (sometimes they would make a request with the same connection at the same time and crash) or it would still be delayed by the time to process the code, even without buffer. the code was taking too long to process the answered request that it couldnt keep it under 10s of delay.
Here is a little example of the basic code I used to get the data.
pairs={'AUDCAD':5,'AUDCHF':5,'AUDJPY':3,'AUDNZD':5,'AUDSGD':2,'AUDUSD':5,'CADCHF':5,'CADJPY':3,'CHFJPY':3,'EURAUD':5,'EURCAD':5,'EURCHF':5,'EURGBP':5,'EURJPY':3,'EURNZD':5,'EURSGD':5,'EURUSD':5,'GBPAUD':5,'GBPCAD':5,'GBPCHF':5,'GBPJPY':3,'GBPNZD':5,'GBPSGD':5,'GBPUSD':5,'NZDCAD':5,'NZDCHF':5,'NZDJPY':3,'NZDUSD':5,'USDCAD':5,'USDCHF':5,'USDJPY':3,'USDSGD':5,'SGDJPY':3}
def on_open(ws):
print("Opened connection")
ws.send('{"action":"auth","params":"<API KEY>"}') #connecting with secret api key
def on_message(ws, message):
print("msg",message)
json_msg = json.loads(message)[0]
if json_msg['status'] == "auth_success": # successfully authenticated
r = ws.send('{"action":"subscribe","params":"C.*"}') # subscribing to currencies
print("should subscribe to " + pairs)
#once the websocket is connected to all the pairs, process the data
--> process json_msg
if __name__ == "__main__":
# websocket.enableTrace(True) # just to show all the requests made (debug mode)
ws = websocket.WebSocketApp("wss://socket.polygon.io/forex",
on_open=on_open,
on_message=on_message)
ws.run_forever(dispatcher=rel) # Set dispatcher to automatic reconnection
rel.signal(2, rel.abort) # Keyboard Interrupt
rel.dispatch()
method I tried multiprocess, but this was on the other crashing my server because it would use 100% CPU, and then the requests made on the apache server would not reach or take a long time loading. Its really a balance problem
I'm using an ubuntu server with 32CPUS, based in london and the API polygon is based in NYC.
I also tried with 4 CPUS in seattle to NYC, but still no luck.
Even with 4 pairs and 32CPUS , it would eventually reach 10s delay. I think this is more of a code structure problem.

How to use Async / Await in Python with messaging based invocation

After years of NodeJS dev, I decided to give Python a shot. So far so good, but I just ran into a wall that I would really like some help with.
I am working on a library that communicates with a remote machine using MQTT. When invoking a function on that library, a message is posted for processing on that remote machine. Once the processing is done, it posts a new message on the bus that my library picks up on and returns the result back to the calling code (the code that invoked the library function).
In Javascript, this is done by returning a Promise, that has a resolve & reject function, that can be stored within the library until the remote message comes back through the broker with the result (intercepted in a different function elsewhere in the library), at which point I can simply invoke the 'resolve' function stored previously to return control to the calling code (the code that invoked the async function of my library). This library function would simply be invoked using the async keyword.
Now in Python, async/await does not use resolve and reject functions that can conveniently be stored away for later, so the logic must be implemented differently I suppose. Using a simple callback function rather than an async/await workflow works, but makes in inconvenient when invoked multiple times in sequence for similar back and forth communications, given that each result handling callback is a separate function.
Here is a basic example of what this would look like in Javascript (for illustration only):
let TASKS = {};
....
mqttClient.on('message', (topic, message) => {
if (topic == "RESULT_OK/123") {
TASKS["123"].resolve(message);
} else if (topic == "RESULT_KO/123") {
TASKS["123"].reject(message);
}
});
...
let myAsyncLibraryFunction = (someCommand) => {
return new Promise((res, rej) => {
TASKS["123"] = {
resolve: res,
reject: rej
};
mqttClient.publish("REQUEST/123", someCommand);
});
}
To call this, I would simply have to do:
try{
let response1 = await myAsyncLibraryFunction("do this");
let response2 = await myAsyncLibraryFunction("now do that");
...
} catch(e) {
...
}
NodeJS is an event loop based language, that's why this is very appropriate for those types of use cases. But this type of application logic is common when dealing with message-based disparate backends, so I am sure there are good ways of solving this in Python as well.
This is a test Python code snippet that I am working on, that attempts to use a future object to achieve something similar:
import paho.mqtt.client as mqtt
import asyncio
import threading
# Init a new asyncio event loop
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
# define a global future placeholder
_future = None
# Create MQTT client
mqttClient = mqtt.Client()
# MQTT Event - on connect
def on_connect(client, userdata, flags, rc):
print("Connected")
client.subscribe("YAVA/#")
# We start a new thread to test our workflow.
#
# If I had done this on the current thread, then the MQTT event loop
# would get stuck (does not process incoming and outgoing messages anymore) when
# calling "await" on the future object later on.
taskThread = threading.Thread(target=_simulateClient, args=())
taskThread.start()
# MQTT Event - on incoming message
def on_message(client, userdata, msg):
global _future
if msg.topic.startswith("YAVA/API/TASK_DONE/") == True:
payload = str(msg.payload.decode("utf-8", "ignore"))
# Resolve the future object
_future.set_result(payload)
mqttClient.on_connect = on_connect
mqttClient.on_message = on_message
# Use asyncio to call a async function and test the workflow
def _simulateClient():
asyncio.run(performAsyncTask())
# This async function will ask for a task to be performed on a remote machine,
# and wait for the response to be sent back
async def performAsyncTask():
result = await pubAndWhaitForResponse("YAVA/API/TASK_START", "")
print(result)
# perform the actual MQTT async logic
async def pubAndWhaitForResponse(topic, message):
# Create a future object that can be resolved in the MQTT event "on_message"
global _future
_future = asyncio.get_running_loop().create_future()
# Publish message that will start the task execution remotely somewhere
global mqttClient
mqttClient.publish(topic, message)
# Now block the thread until the future get's resolved
result = await _future
# Return the result
return result
# Start the broker and loop^forever in the main thread
mqttClient.connect("192.168.1.70", 1883, 60)
# The MQTT library will start a new thread that will continuously
# process outgoing and incoming messages through that separate thread.
# The main thread will be blocked so that the program does not exit
mqttClient.loop_forever()
It all runs fine, but the _future.set_result(payload) line does not seem to resolve the future. I never see the result printed.
It feels like there is not much missing to get this sorted. Any suggestions would be great.
Thanks

I think we are using the asyncio library the bad way, mixing it with multi-process/multi-threading parallelism.
Here is an implementation based on the multiprocessing module. When submitting a task for your remote, your library can return a Queue that the caller can use with the get() method: it return the value if available, else it suspend the thread, waiting for the value. Hence, the Queue acts as a Scala's Future or a JS Promise.
import multiprocessing
import time
from concurrent.futures.thread import ThreadPoolExecutor
import paho.mqtt.client as mqtt
import logging
logging.basicConfig(level=logging.DEBUG,
format="%(asctime)s ## %(thread)d ## %(funcName)s ## %(message)s")
def remote_on_connect(client, *_):
logging.info("Connected")
client.subscribe("YAVA/API/TASK_START")
def remote_on_message(client, _, _1):
logging.info("Remotely processing your data")
time.sleep(1)
logging.info("Publishing result")
client.publish("YAVA/API/TASK_DONE", 42)
class Lib:
def __init__(self):
self.client = mqtt.Client()
self.executor = ThreadPoolExecutor(max_workers=1)
self.client.on_connect = Lib.on_connect
self.client.connect("test.mosquitto.org")
self.client.loop_start()
def stop(self):
self.client.loop_stop()
def execute(self):
cb, queue = self.get_cb()
self.client.on_message = cb
self.client.publish("YAVA/API/TASK_START", "foo")
return queue
#staticmethod
def on_connect(client, *_):
logging.info("Connected")
client.subscribe("YAVA/API/TASK_DONE")
def get_cb(self):
queue = multiprocessing.Queue(maxsize=1)
def cb(_0, _1, msg):
self.client.on_message = None
logging.info("Fetching back the result")
logging.info(str(msg.payload.decode("utf-8", "ignore")))
queue.put(42)
logging.info("Queue filled")
return cb, queue
def main():
remote_client = mqtt.Client()
remote_client.on_connect = remote_on_connect
remote_client.on_message = remote_on_message
remote_client.connect("test.mosquitto.org")
remote_client.loop_start()
lib = Lib()
future = lib.execute()
logging.info("Result is:")
logging.info(future.get())
remote_client.loop_stop()
lib.stop()
logging.info("Exiting")
if __name__ == '__main__':
main()
2019-11-19 15:08:34,433 ## 139852611577600 ## remote_on_connect ## Connected
2019-11-19 15:08:34,450 ## 139852603184896 ## on_connect ## Connected
2019-11-19 15:08:34,452 ## 139852632065728 ## main ## Result is:
2019-11-19 15:08:34,467 ## 139852611577600 ## remote_on_message ## Remotely processing your data
2019-11-19 15:08:35,469 ## 139852611577600 ## remote_on_message ## Publishing result
2019-11-19 15:08:35,479 ## 139852603184896 ## cb ## Fetching back the result
2019-11-19 15:08:35,479 ## 139852603184896 ## cb ## 42
2019-11-19 15:08:35,480 ## 139852603184896 ## cb ## Queue filled
2019-11-19 15:08:35,480 ## 139852632065728 ## main ## 42
2019-11-19 15:08:36,481 ## 139852632065728 ## main ## Exiting
As you can see in the output, the main method execute up to the future.get method (as show by the Result is: line early in the log). Then, processing happen in another thread, until putting a value inside the shared Queue. Now the future.get returns (because the value is available) and the main method proceed to the end.
Hope this can help you to achieve what you want, but any insights about better ways to achieve this, either with asyncio or with smaller data structure than Queue, are welcome.

First thing I can see is that you publish in the YAVA/API/TASK_START topic while checking that the topic is YAVA/API/TASK_DONE/ in your on_message callback. Hence, your _future never gets a result and the await _future never returns...
I advice you to add log. Add these lines at the start of your code:
import logging
logging.basicConfig(level=logging.DEBUG,
format="%(asctime)s ## %(thread)d ## %(funcName)s ## %(message)s")
Then use logging.info(...) to trace your execution order.
I added some to your code (as well as changing the condition in on_message) and here is the output.
2019-11-19 11:37:10,440 ## 140178907485888 ## __init__ ## Using selector: EpollSelector
2019-11-19 11:37:10,478 ## 140178907485888 ## on_connect ## Connected
2019-11-19 11:37:10,478 ## 140178887976704 ## _simulateClient ## Enter simulate client
2019-11-19 11:37:10,479 ## 140178887976704 ## __init__ ## Using selector: EpollSelector
2019-11-19 11:37:10,480 ## 140178887976704 ## performAsyncTask ## Perform async task
2019-11-19 11:37:10,480 ## 140178887976704 ## pubAndWhaitForResponse ## Pub and wait
2019-11-19 11:37:10,481 ## 140178887976704 ## pubAndWhaitForResponse ## Publish
2019-11-19 11:37:10,481 ## 140178887976704 ## pubAndWhaitForResponse ## Await future: <Future pending created at /usr/lib/python3.7/asyncio/base_events.py:391>
2019-11-19 11:37:10,499 ## 140178907485888 ## on_message ## New message
2019-11-19 11:37:10,499 ## 140178907485888 ## on_message ## Topic: YAVA/API/TASK_DONE
2019-11-19 11:37:10,499 ## 140178907485888 ## on_message ## Filling future: <Future pending cb=[<TaskWakeupMethWrapper object at 0x7f7df0f5fd10>()] created at /usr/lib/python3.7/asyncio/base_events.py:391>
I also added a log after the _future.set_result(payload) line, but it never appears. So the set_result seems to hang or something like that...
You probably have to dig inside it to know why/where it hangs.
Edit
By the way, you are mixing many concepts: asyncio, threading, and mqtt (with its own loop).
Moreover, the asyncio.Future is not thread-safe, I think it's dangerous to use it as you do. While using debugger, to go inside the set_result method, I encounter an exception in the mqtt client class:
Non-thread-safe operation invoked on an event loop other than the current one
It is never reported on stdout/stderr, but you can maybe catch it in the on_log callback of your client.
Edit 2
Here is a more Pythonic example of your code. In this one, the set_result does not hang (the log just after is displayed) but it is the await in the main.
import asyncio
import time
import paho.mqtt.client as mqtt
import logging
logging.basicConfig(level=logging.DEBUG,
format="%(asctime)s ## %(thread)d ## %(funcName)s ## %(message)s")
def remote_on_connect(client, *_):
logging.info("Connected")
client.subscribe("YAVA/API/TASK_START")
def remote_on_message(client, _, _1):
logging.info("Remotely processing your data")
time.sleep(1)
logging.info("Publishing result")
client.publish("YAVA/API/TASK_DONE", 42)
class Lib:
def __init__(self):
self.client = mqtt.Client()
self.client.on_connect = Lib.on_connect
self.client.on_log = lambda x: logging.info("Log: %s", x)
self.client.connect("test.mosquitto.org")
self.client.loop_start()
def stop(self):
self.client.loop_stop()
def execute(self):
self.client.publish("YAVA/API/TASK_START", "foo")
cb, fut = Lib.get_cb()
self.client.on_message = cb
return fut
#staticmethod
def on_connect(client, *_):
logging.info("Connected")
client.subscribe("YAVA/API/TASK_DONE")
#staticmethod
def get_cb():
fut = asyncio.get_event_loop().create_future()
def cb(_0, _1, msg):
logging.info("Fetching back the result")
logging.info(str(msg.payload.decode("utf-8", "ignore")))
fut.set_result(42)
logging.info("Future updated")
return cb, fut
async def main():
remote_client = mqtt.Client()
remote_client.on_connect = remote_on_connect
remote_client.on_message = remote_on_message
remote_client.connect("test.mosquitto.org")
remote_client.loop_start()
lib = Lib()
future = lib.execute()
logging.info("Result is:")
await future
logging.info(future.result())
remote_client.loop_stop()
lib.stop()
logging.info("Exiting")
if __name__ == '__main__':
asyncio.run(main())

Mixing Synchronous and A-sync code in Python

I'm trying to convert a synchronous flow in Python code which is based on callbacks to an A-syncronious flow using asyncio.
Basically the code interacts a lot with TCP/UNIX sockets. It reads data from the sockets, manipulates it to make decisions and writes stuff back to the other side. This is going on over multiple sockets at once and data is shared between the contexts to make decisions sometimes.
EDIT :: The code currently is mostly based on registering a callback to a central entity for a specific socket, and having that entity run the callback when the relevant socket is readable (something like "call this function when that socket has data to be read"). Once the callback is called - a bunch of stuff happens, and eventually a new callback is registered for when new data is available. The central entity runs a select over all sockets registered to figure out which callbacks should be called.
I'm trying to do this without refactoring my entire code and making this as seamless as possible to the programmer - so I was trying to think about it like so - all code should run the same way as it does today - but whenever the current code does a socket.recv() to get new data - the process would yield execution to other tasks. When the read returns, it should go back to handling the data from the same point using the new data it got.
To do this, I wrote a new class called AsyncSocket - which interacts with the IO streams of asyncIO and placed the Async/await statements almost solely in there - thinking that I would implement the recv method in my class to make it look like a "regular IO socket" to the rest of my code.
So far - this is my understanding of what A-sync programming should allow.
Now to the problem :
My code awaits for clients to connect - when it does, each client's context is allowed to read and write from it's own connection.
I've simplified to flow to the following to clarify the problem:
class AsyncSocket():
def __init__(self,reader,writer):
self.reader = reader
self.writer = writer
def recv(self,numBytes):
print("called recv!")
data = self.read_mitigator(numBytes)
return data
async def read_mitigator(self,numBytes):
print("Awaiting of AsyncSocket.reader.read")
data = await self.reader.read(numBytes)
print("Done Awaiting of AsyncSocket.reader.read data is %s " % data)
return data
def mit2(aSock):
return mit3(aSock)
def mit3(aSock):
return aSock.recv(100)
async def echo_server(reader, writer):
print ("New Connection!")
aSock = AsyncSocket(reader,writer) # create a new A-sync socket class and pass it on the to regular code
while True:
data = await some_func(aSock) # this would eventually read from the socket
print ("Data read is %s" % (data))
if not data:
break
writer.write(data) # echo everything back
async def main(host, port):
server = await asyncio.start_server(echo_server, host, port)
await server.serve_forever()
asyncio.run(main('127.0.0.1', 5000))
mit2() and mit3() are synchronous functions that do stuff with the data on the way back before returning to the main client's loop - but here I'm just using them as empty functions.
The problem starts when I play with the implementation of some_func().
A pass through implementation (edit: kind-of-works) - but still has issues :
def some_func(aSock):
try:
return (mit2(aSock)) # works
except:
print("Error!!!!")
While an implementation which reads the data and does something with it - like adding a suffix before returning, throws an error:
def some_func(aSock):
try:
return (mit2(aSock) + "something") # doesn't work
except:
print("Error!!!!")
The error (as far as I understand it) means it's not really doing what it should:
New Connection!
called recv!
/Users/user/scripts/asyncServer.py:36: RuntimeWarning: coroutine 'AsyncSocket.read_mitigator' was never awaited
return (mit2(aSock) + "something") # doesn't work
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Error!!!!
Data read is None
And the echo server obviously doesn't work.
Obviously my code looks more like option #2 with a lot more stuff in some_func(),mit2() and mit3() - but I can't get this to work. I'm fairly new in using asyncio/async/await - so what (rather basic concept I guess) am I missing?

This code won't work as envisioned:
def recv(self,numBytes):
print("called recv!")
data = self.read_mitigator(numBytes)
return data
async def read_mitigator(self,numBytes):
...
You cannot call an async function from a sync function and get the result, you must await it, which ensures that you return to the event loop in case the data is not yet ready. This mismatch between async and sync code is sometimes referred to as the issue of function color.
Since your code is already using non-blocking sockets and an event loop, a good approach to porting it to asyncio might be to first switch to the asyncio event loop. You can use event loop methods like sock_recv to request data:
def start():
loop = asyncio.get_event_loop()
sock = make_socket() # make sure it's non-blocking
future_data = loop.sock_recv(sock, 1024)
future_data.add_done_callback(continue_read)
# return to the event loop - when some data is ready
# continue_read will be invoked
def continue_read(future):
data = future.result()
print('got', data)
# ... do something with data, e.g. process it
# and call sock_sendall with the response
asyncio.get_event_loop().call_soon(start())
asyncio.get_event_loop().run_forever()
Once you have the program working in that mode, you can start moving to coroutines, which allow the code to look like sync code, but work in exactly the same way:
async def start():
loop = asyncio.get_event_loop()
sock = make_socket() # make sure it's non-blocking
data = await loop.sock_recv(sock, 1024)
# data is available "immediately", meaning the coroutine gets
# automatically suspended when awaiting data that is not yet
# ready, and automatically re-scheduled when the data is ready
print('got', data)
asyncio.run(start())
The next step can be eliminating make_socket and switching to asyncio streams.

How to write python code that will work with threads or coroutines and will complete in deterministic time?

What I mean by "deterministic time"? For example AWS offer a service "AWS Lambda". The process started as lambda function has time limit, after that lambda function will stop execution and will assume that task was finished with error. And example task - send data to http endpoint. Depending of a network connection to http endpoint, or other factors, process of sending data can take a long time. If I need to send the same data to the many endpoints, then full process time will take one process time times endpoints amount. Which increase a chance that lambda function will be stopped before all data will be send to all endpoints.
To solve this I need to send data to different endpoints in parallel mode using threads.
The problem with threads - started thread can't be stopped. If http request will take more time than it dedicated by lambda function time limit, lambda function will be aborted and return error. So I need to use timeout with http request, to abort it, if it take more time than expected.
If http request will be canceled by timeout or endpoint will return error, I need to save not processed data somewhere to not lost the data. The time needed to save unprocessed data can be predicted, because I control the storage where data will be saved.
And the last part that consume time - procedure or loop where threads are scheduled executor.submit(). If there is only one endpoint or small number of them then the consumed time will be small. And there is no necessary to control this. But if I have deal with many endpoints, I have to take this into account.
So basically full time will consists of:
scheduling threads
http request execution
saving unprocessed data
There is example of how I can manage time using threads
import concurrent.futures
from functools import partial
import requests
import time
start = time.time()
def send_data(data):
host = 'http://127.0.0.1:5000/endpoint'
try:
result = requests.post(host, json=data, timeout=(0.1, 0.5))
# print('done')
if result.status_code == 200:
return {'status': 'ok'}
if result.status_code != 200:
return {'status': 'error', 'msg': result.text}
except requests.exceptions.Timeout as err:
return {'status': 'error', 'msg': 'timeout'}
def get_data(n):
return {"wait": n}
def done_cb(a, b, future):
pass # save unprocessed data
def main():
executor = concurrent.futures.ThreadPoolExecutor()
futures = []
max_time = 0.5
for i in range(1):
future = executor.submit(send_data, *[{"wait": 10}])
future.add_done_callback(partial(done_cb, 2, 3))
futures.append(future)
if time.time() - s_time > max_time:
print('stopping creating new threads')
# save unprocessed data
break
try:
for item in concurrent.futures.as_completed(futures, timeout=1):
item.result()
except concurrent.futures.TimeoutError as err:
pass
I was thinking of how I can use asyncio library instead of threads, to do the same thing.
import asyncio
import time
from functools import partial
import requests
start = time.time()
def send_data(data):
...
def get_data(n):
return {"wait": n}
def done_callback(a,b, future):
pass # save unprocessed data
def main(loop):
max_time = 0.5
futures = []
start_appending = time.time()
for i in range(1):
event_data = get_data(1)
future = (loop.run_in_executor(None, send_data, event_data))
future.add_done_callback(partial(done_callback, 2, 3))
futures.append(future)
if time.time() - s_time > max_time:
print('stopping creating new futures')
# save unprocessed data
break
finished, unfinished = loop.run_until_complete(
asyncio.wait(futures, timeout=1)
)
_loop = asyncio.get_event_loop()
result = main(_loop)
Function send_data() the same as in previous code snipped.
Because request library is not async code I use run_in_executor() to create future object. The main problems I have is that done_callback() is not executed when the thread that started but executor done it's job. But only when the futures will be "processed" by asyncio.wait() expression.
Basically I seeking the way to start execute asyncio future, like ThreadPoolExecutor start execute threads, and not wait for asyncio.wait() expression to call done_callback(). If you have other ideas how to write python code that will work with threads or coroutines and will complete in deterministic time. Please share it, I will be glad to read them.
And other question. If thread or future done its job, it can return result, that I can use in done_callback(), for example to remove message from queue by id returned in result. But if thread or future was canceled, I don't have result. And I have to use functools.partial() pass in done_callback additional data, that can help me to understand for what data this callback was called. If passed data are small this is not a problem. If data will be big, I need to put data in array/list/dictionary and pass in callback only index of array or put "full data: in callback.
Can I somehow get access to variable that was passed to future/thread, from done_callback(), that was triggered on canceled future/thread?

You can use asyncio.wait_for to wait for a future (or multiple futures, when combined with asyncio.gather) and cancel them in case of a timeout. Unlike threads, asyncio supports cancellation, so you can cancel a task whenever you feel like it, and it will be cancelled at the first blocking call it makes (typically a network call).
Note that for this to work, you should be using asyncio-native libraries such as aiohttp for HTTP. Trying to combine requests with asyncio using run_in_executor will appear to work for simple tasks, but it will not bring you the benefits of using asyncio, such as being able to spawn a massive number of tasks without encumbering the OS, or the possibility of cancellation.

Need to set a variable equal to the result of a part of a function

I am attempting to create a script that uses a value in a remote database to create a wol packet.
I need to pass the value of a (something that I honestly don't know what to call) to a variable. I am unable to just set it to the variable, and I can't figure out how to import it from somewhere else. I need "payload" which is printed and returned under "def message" to be saved to the variable "data"
Below is my code, and I will link the MQTT code that this relies on.
# Import standard python modules.
import sys
# Import Adafruit IO MQTT client.
from Adafruit_IO import MQTTClient
from Adafruit_IO import *
# Set to your Adafruit IO key & username below.
ADAFRUIT_IO_KEY = 'where the api key goes'
ADAFRUIT_IO_USERNAME = 'my username' # See https://accounts.adafruit.com
# to find your username.
# Set to the ID of the feed to subscribe to for updates.
FEED_ID = 'test1'
# Define callback functions which will be called when certain events happen.
def connected(client):
# Connected function will be called when the client is connected to Adafruit IO.
# This is a good place to subscribe to feed changes. The client parameter
# passed to this function is the Adafruit IO MQTT client so you can make
# calls against it easily.
print ('Connected to Adafruit IO! Listening for {0} changes...').format(FEED_ID)
# Subscribe to changes on a feed named DemoFeed.
client.subscribe(FEED_ID)
def disconnected(client):
# Disconnected function will be called when the client disconnects.
print ('Disconnected from Adafruit IO!')
sys.exit(1)
def message(client, feed_id, payload):
# Message function will be called when a subscribed feed has a new value.
# The feed_id parameter identifies the feed, and the payload parameter has
print ('Feed {0} received new value: {1}').format(FEED_ID, payload)
return (payload == payload)
data = message(x, x, x)
# Create an MQTT client instance.
client = MQTTClient(ADAFRUIT_IO_USERNAME, ADAFRUIT_IO_KEY)
# Setup the callback functions defined above.
client.on_connect = connected
client.on_disconnect = disconnected
client.on_message = message
# Connect to the Adafruit IO server.
client.connect()
# Start a message loop that blocks forever waiting for MQTT messages to be
# received. Note there are other options for running the event loop like doing
# so in a background thread--see the mqtt_client.py example to learn more.
if data == '1':
print('Latest value from Test: {0}'.format(data.value))
wol.send_magic_packet('my mac addy')
time.sleep(3)
# Send a value to the feed 'Test'.
aio.send('test1', 0)
print ("worked this time")
client.loop_blocking()
Here is the link to the other code it relies on https://github.com/adafruit/io-client-python/blob/master/Adafruit_IO/mqtt_client.py

First of all, you don't need to import the same module twice
from Adafruit_IO import MQTTClient # this imports part MQTTClient
from Adafruit_IO import * # this imports everything, including MQTTClient again
As for your problem, you've returned (payload == payload) which will always be True, i'm not sure what you're trying to do here but it should look something like this:
def message(client, feed_id, payload):
...
print ('Feed {0} received new value: {1}').format(FEED_ID, payload)
return payload == payload_from_database # what you return will be saved as the variable data
data = message("Steve the happy client", 85, "£100") # in this case, data will be payload