We Keep Coding

Nim: How can I improve concurrent async response time and quota to match cpythons asyncio?

For an upcoming project this year, I wanted to look into some languages that I haven't really used yet, but that repeatedly catch my interest. Nim is one of them 😊.
I wrote the following code to make async requests:
import asyncdispatch, httpclient, strformat, times, strutils
let urls = newHttpClient().getContent("https://gist.githubusercontent.com/tobealive/b2c6e348dac6b3f0ffa150639ad94211/raw/31524a7aac392402e354bced9307debd5315f0e8/100-popular-urls.txt").splitLines()[0..99]
proc getHttpResp(client: AsyncHttpClient, url: string): Future[string] {.async.} =
try:
result = await client.getContent(url)
echo &"{url} - response length: {len(result)}"
except Exception as e:
echo &"Error: {url} - {e.name}"
proc requestUrls(urls: seq[string]) {.async.} =
let start = epochTime()
echo "Starting requests..."
var futures: seq[Future[string]]
for url in urls:
var client = newAsyncHttpClient()
futures.add client.getHttpResp(&"http://www.{url}")
for i in 0..urls.len-1:
discard await futures[i]
echo &"Requested {len(urls)} websites in {epochTime() - start}."
waitFor requestUrls(urls)
Results doing some loops:
Iterations: 10. Total errors: 94.
Average time to request 100 websites: 9.98s.
The finished application will only request from a single ressource. So for example, when requesting Google search queries (for simplicity just the numbers from 1 to 100), the result look like:
Iterations: 1. Total errors: 0.
Time to request 100 google searches: 3.75s.
Compared to Python, there are still significant differences:
import asyncio, time, requests
from aiohttp import ClientSession
urls = requests.get(
"https://gist.githubusercontent.com/tobealive/b2c6e348dac6b3f0ffa150639ad94211/raw/31524a7aac392402e354bced9307debd5315f0e8/100-popular-urls.txt"
).text.split('\n')
async def getHttpResp(url: str, session: ClientSession):
try:
async with session.get(url) as resp:
result = await resp.read()
print(f"{url} - response length: {len(result)}")
except Exception as e:
print(f"Error: {url} - {e.__class__}")
async def requestUrls(urls: list[str]):
start = time.time()
print("Starting requests...")
async with ClientSession() as session:
await asyncio.gather(*[getHttpResp(f"http://www.{url}", session) for url in urls])
print(f"Requested {len(urls)} websites in {time.time() - start}.")
# await requestUrls(urls) # jupyter
asyncio.run(requestUrls(urls))
Results:
Iterations: 10. Total errors: 10.
Average time to request 100 websites: 7.92s.
When requesting only google search queries:
Iterations: 1. Total errors: 0.
Time to request 100 google searches: 1.38s.
Additionally: the difference in response time remains when comparing a single response to an individual URL and just getting the response status code.
(I'm not big into python, but when using it, it's often impressive what it's C libraries deliver.)
To improve the Nim code, I thought it might be worth trying to add channels and multiple clients (this is from a still very limited point of view on my second day of programming in Nim + generally not having a lot of experience with concurrent requests). But I haven't really figured out how to get it to work.
Doing a lot of request to the same endpoint in the nim example (e.g. when doing the google searches) it also may result in a Too Many Requests error if this amount of Google searches are performed repeatedly. In python this doesn't seem to be the case.
So it would be great if you could share your approach on what can be done to improve the response quota and request time!
If anyone wants a repo for cloning and tinkering, this one contains the example with the loop:
https://github.com/tobealive/nim-async-requests-example

I tried to remember how Nims async works, and can unfortunately see no real issue in your code. Compiling with -d:release seems to make not a big difference. One idea is the timeout, which may be different for Python. From https://nim-lang.org/docs/httpclient.html#timeouts we learn that there is no timeout for async, so a very slow page may keep the connection open for a long time. Maybe Python does a time-out? I was not able to test the Python module, aiohttp is missing on my box. Below is a test of mine, not that different from yours. I made main() not async, by using waitFor all(f). Sorry that I could not really help you, maybe you should really try the chronos variant.
# nim r -d:ssl -d:release t.nim
import std/[asyncdispatch, httpclient, strutils, strformat, times]
const
UrlSource = "https://gist.githubusercontent.com/tobealive/" &
"b2c6e348dac6b3f0ffa150639ad94211/raw/31524a7aac392402e354bced9307debd5315f0e8/" &
"100-popular-urls.txt"
proc getHttpResp(client: AsyncHttpClient, url: string): Future[string] {.async.} =
try:
result = await client.getContent(url)
echo &"{url} - response length: {len(result)}"
except Exception as e:
echo &"Error: {url} - {e.name}"
proc main =
let start = epochTime()
echo "Starting requests..."
var urls = newHttpClient().getContent(UrlSource).splitLines
if urls.len > 100: # in case that there are more than 100, clamp it
urls.setLen(100)
# urls.setLen(3) # for fast tests with only a few urls
var f: seq[Future[string]]
for url in urls:
let client = newAsyncHttpClient()
f.add(client.getHttpResp(&"http://www.{url}"))
let res: seq[string] = waitFor all(f)
for x in res:
echo x.len
echo fmt"Requested {len(urls)} websites in {epochTime() - start:.2f} seconds."
main()
Testing with an extended version of the above program, I get the feeling that the total transfer rate is just limited to a few MB/s, and my idea about timeouts was very wrong. I did some Google search about the topic, was not able to find much useful info. As you wrote in your initial post already, Nim's async from standard library is not parallel, but it is (theoretical) possible to use it with multiple threads. When I have more free time, I may do a test with Chronos.
# nim r -d:ssl -d:release t.nim
import std/[asyncdispatch, httpclient, strutils, strformat, times]
const
UrlSource = "https://gist.githubusercontent.com/tobealive/" &
"b2c6e348dac6b3f0ffa150639ad94211/raw/31524a7aac392402e354bced9307debd5315f0e8/" &
"100-popular-urls.txt"
proc getHttpResp(client: AsyncHttpClient, url: string): Future[string] {.async.} =
let start = epochTime()
try:
result = await client.getContent(url)
stdout.write &"{url} - response length: {len(result)}"
except Exception as e:
stdout.write &"Error: {url} - {e.name}"
echo fmt" --- Request took {epochTime() - start:.2f} seconds."
proc main =
var transferred: int = 0
let start = epochTime()
echo "Starting requests..."
var urls = newHttpClient().getContent(UrlSource).splitLines
if urls.len > 100: # in case that there are more than 100, clamp it
urls.setLen(100)
#urls.setLen(3) # for fast tests with only a few urls
var f: seq[Future[string]]
for url in urls:
let client = newAsyncHttpClient()
f.add(client.getHttpResp(&"http://www.{url}"))
let res: seq[string] = waitFor all(f)
for x in res:
transferred += x.len
echo fmt"Sum of transferred data: {transferred} bytes. ({transferred.float / (1024 * 1024).float / (epochTime() - start):.2f} MBytes/s)"
echo fmt"Requested {len(urls)} websites in {epochTime() - start:.2f} seconds."
main()
References:
https://xmonader.github.io/nimdays/day04_asynclinkschecker.html

Websockets messages only sent at the end and not in instances using async / await, yield in nested for loops

I have a computationally heavy process that takes several minutes to complete in the server. So I want to send the results of every iteration to the client via websockets.
The overall application works but my problem is that all the messages are arriving at the client in one big chunk after the entire simulation finishes. I must be missing something here as I expect the await websocket.send_json() to send the message during the process and not all of them at the end.
Server python (FastAPI)
# A very simplified abstraction of the actual app.
def simulate_intervals(data):
for t in range(data.n_intervals):
state = interval(data) # returns a JAX NumPy array
yield state
def simulate(data):
for key in range(data.n_trials):
trial = simulate_intervals(data)
yield trial
#app.websocket("/ws")
async def socket(websocket: WebSocket):
await websocket.accept()
while True:
# Get model inputs from client
data = await websocket.receive_text()
# Minimal computation
nodes = distributions(data)
nodosJson = json.dumps(nodes, cls=NumpyEncoder)
# I expect this message to be sent early on,
# but the client gets it at the end with all the other messages.
await websocket.send_json({"tipo": "nodos", "datos": json.loads(nodosJson)})
# Heavy computation
trials = simulate(data)
for trialI, trial in enumerate(trials):
for stateI, state in enumerate(trial):
stateString = json.dumps(state, cls=NumpyEncoder)
await websocket.send_json(
{
"tipo": "estado",
"datos": json.loads(stateString),
"trialI": trialI,
"stateI": stateI,
}
)
await websocket.send_json({"tipo": "estado", "msg": "fin"})
For completeness, here is the basic client code.
Client
const ws = new WebSocket('ws://localhost:8000/ws');
ws.onopen = () => {
console.log('Conexión exitosa');
};
ws.onmessage = (e) => {
const mensaje = JSON.parse(e.data);
console.log(mensaje);
};
botonEnviarDatos.onclick = () => {
ws.send(JSON.stringify({...}));
}

I was not able to make it work as posted in my question, still interested in hearing from anyone who understands why it is not possible to send multiple async messages without them getting blocked.
For anyone interested, here is my current solution:
Ping pong messages from client and server
I changed the logic so the server and client are constantly sending each other messages and not trying to stream the data in a single request from the client.
This actually works much better than my original attempt because I can detect when a sockets gets disconnected and stop processing in the server. Basically, if the client disconnects, no new requests for data are sent from that client and the server never continues the heavy computation.
Server
# A very simplified abstraction of the actual app.
def simulate_intervals(data):
for t in range(data.n_intervals):
state = interval(data) # returns a JAX NumPy array
yield state
def simulate(data):
for key in range(data.n_trials):
trial = simulate_intervals(data)
yield trial
#app.websocket("/ws")
async def socket(websocket: WebSocket):
await websocket.accept()
while True:
# Get messages from client
data = await websocket.receive_text()
# "tipo" is basically the type of data being sent from client or server to the other one.
# In this case, "tipo": "inicio" is the client sending inputs and requesting for a certain data in response.
if data["tipo"] == "inicio":
# Minimal computation
nodes = distributions(data)
nodosJson = json.dumps(nodes, cls=NumpyEncoder)
# In this first interaction, the client gets the first message without delay.
await websocket.send_json({"tipo": "nodos", "datos": json.loads(nodosJson)})
# Since this is a generator (def returns yield) it does not actually
# trigger that actual computationally heavy process.
trials = simulate(data)
# define some initial variables to count the iterations
trialI = 0
stateI = 0
trialsLen = args.number_trials
statesLen = 600
# load the first trial (also a generator)
# without the for loop used before, the counters and next()
# allow us to do the same as being done before in the for loop
trial = next(trials)
# With the use of generators and next() it is possible to keep
# this first message light on the server and send the first response
# as quickly as possible.
# This type of message asks for the next instance of the simluation
# without processing the entire model.
elif data["tipo"] == "sim":
# check if we are within the limits (before this was a nested for loop)
if trialI < trialsLen and stateI < statesLen:
# Trigger the next instance of the simulation
state = next(trial)
# update counter
stateI = stateI + 1
# Send the message with 1 instance of the simulation.
#
stateString = json.dumps(state, cls=NumpyEncoder)
await websocket.send_json(
{
"tipo": "estado",
"datos": json.loads(stateString),
"trialI": trialI,
"stateI": stateI,
}
)
# Check if the second loop is done
if stateI == statesLen:
# update counter of first loop
trialI = trialI + 1
# update counter of second loop
stateI = 0
# Check if there are more pending trials,
# otherwise stop and notify the client we are done.
try:
trial = next(trials)
except StopIteration:
await websocket.send_json({"tipo": "fin"})
Client
Just the part that actually changed:
ws.onmessage = (e) => {
const mensaje = JSON.parse(e.data);
// Simply check the type of incoming message so it can be processed
if (mensaje.tipo === 'fin') {
viz.calcularResultados();
} else if (mensaje.tipo === 'nodos') {
viz.pintarNodos(mensaje.datos);
} else if (mensaje.tipo === 'estado') {
viz.sumarEstado(mensaje.datos);
}
// After receiving a message, ping the server for the next one
ws.send(
JSON.stringify({
tipo: 'sim',
})
);
};
This seems like reasonable solution to keep the server and client working together. I am able to show in the client the progress of a long simulation and the user experience is much better than having to wait for a long time for the server to respond. Hope it helps other with a similar problem.

I got a similar issue, and was able to resolve it by adding a small await asyncio.sleep(0.1) after sending json messages. I have not dived into asyncios internals yet, but my guess is that websocker.send shedules a message to be sent, but since the async function continues to run it never has a chance to do it in the background. Sleeping the async function makes asyncio pick up other tasks while it is waiting.

Multiple Await in Python Async Function

I am using aiohttp session along with a semaphore within a custom class:
async def get_url(self, url):
async with self.semaphore:
async with self.session.get(url) as response:
try:
text_response = await response.text()
read_response = await response.read()
json_response = await response.json()
await asyncio.sleep(random.uniform(0.1, 0.5))
except aiohttp.client_exceptions.ContentTypeError:
json_response = {}
return {
'json': json_response,
'text': text_response,
'read': read_response,
'status': response.status,
'url': response.url,
}
I have two questions:
Is it correct/incorrect to to have multiple await statements within a single async function? I need to return both the response.text() and response.read(). However, depending on the URL, the response.json() may or may not be available so I've thrown everything into a try/except block to catch this exception.
Since I am using this function to loop through a list of different RESTful API endpoints, I am controlling the number of simultaneous requests through the semaphore (set to max of 100) but I also need to stagger the requests so they aren't log jamming the host machine. So, I thought I could accomplish this by adding an asyncio.sleep that is randomly chosen between 0.1-0.5 seconds. Is this the best way to enforce a small wait in between requests? Should I move this to the beginning of the function instead of near the end?

It is absolutely fine to have multiple awaits in one async function, as far as you know what you are awaiting for, and each of them are awaited one by one, just like the very normal sequential execution. One thing to mention about aiohttp is that, you'd better call read() first and catch UnicodeDecodeError too, because internally text() and json() call read() first and process its result, you don't want the processing to prevent returning at least read_response. You don't have to worry about read() being called multiple times, it is simply cached in the response instance on the first call.
Random stagger is an easy and effective solution for sudden traffic. However if you want to control exactly the minimum time interval between any two requests - for academic reasons, you could set up two semaphores:
def __init__(self):
# something else
self.starter = asyncio.Semaphore(0)
self.ender = asyncio.Semaphore(30)
Then change get_url() to use them:
async def get_url(self, url):
await self.starter.acquire()
try:
async with self.session.get(url) as response:
# your code
finally:
self.ender.release()
Because starter was initialized with zero, so all get_url() coroutines will block on starter. We'll use a separate coroutine to control it:
async def controller(self):
last = 0
while self.running:
await self.ender.acquire()
sleep = 0.5 - (self.loop.time() - last) # at most 2 requests per second
if sleep > 0:
await asyncio.sleep(sleep)
last = self.loop.time()
self.starter.release()
And your main program should look something like this:
def run(self):
for url in [...]:
self.loop.create_task(self.get_url(url))
self.loop.create_task(self.controller())
So at first, the controller will release starter 30 times evenly in 15 seconds, because that is the initial value of ender. After that, the controller would release starter as soon as any get_url() ends, if 0.5 seconds have passed since the last release of starter, or it will wait up to that time.
One issue here: if the URLs to fetch is not a constant list in memory (e.g. coming from network constantly with unpredictable delays between URLs), the RPS limiter will fail (starter released too early before there is actually a URL to fetch). You'll need further tweaks for this case, even though the chance of a traffic burst is already very low.

Make API request every x seconds in Python 3

I am trying to do stress test on a server using Python 3. The idea is to send an HTTP request to the API server every 1 second for 30 minutes. I tried using requests and apscheduler to do this but I kept getting
Execution of job "send_request (trigger: interval[0:00:01], next run at: 2017-05-23 11:05:46 EDT)"
skipped: maximum number of running instances reached (1)
How can I make this work? Below is my code so far:
import requests, json, time, ipdb
from apscheduler.schedulers.blocking import BlockingScheduler as scheduler
def send_request():
url = 'http://api/url/'
# Username and password
credentials = { 'username': 'username', 'password': 'password'}
# Header
headers = { 'Content-Type': 'application/json', 'Client-Id': 'some string'}
# Defining payloads
payload = dict()
payload['item1'] = 1234
payload['item2'] = 'some string'
data_array = [{"id": "id1", "data": "some value"}]
payload['json_data_array'] = [{ "time": int(time.time()), "data": data_array]
# Posting data
try:
request = requests.post(url, headers = headers, data = json.dumps(payload))
except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as err:
print("Error while trying to POST pid data")
print(err)
finally:
request.close()
print(request.content)
return request.content
if __name__ == '__main__':
sched = scheduler()
print(time.time())
sched.add_job(send_request, 'interval', seconds=1)
sched.start()
print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))
try:
# This is here to simulate application activity (which keeps the main thread alive).
while true:
pass
except (KeyboardInterrupt, SystemExit):
# Not strictly necessary if daemonic mode is enabled but should be done if possible
scheduler.shutdown()
I tried searching on stack overflow but none of the other questions does what I want so far, or maybe I missed something. I would appreciate someone to point me to the correct thread if that is the case. Thank you very much!

I think your error is described well by the duplicate that I marked as well as the answer by #jeff
Edit: Apparently not.. so here I'll describe how to fix the maximum instances problem:
Maximum instances problem
When you're adding jobs to the scheduler there is an argument you can set for the number of maximum allowed concurrent instances of the job. You can should read about this here:
BaseScheduler.add_job()
So, fixing your problem is just a matter of setting this to something higher:
sch.add_job(myfn, 'interval', seconds=1, max_instances=10)
But, how many concurrent requests do you want? If they take more than one second to respond, and you request one per second, you will always eventually get an error if you let it run long enough...
Schedulers
There are several scheduler options available, here are two:
BackgroundScheduler
You're importing the blocking scheduler - which blocks when started. So, the rest of your code is not being executed until after the scheduler stops. If you need other code to be executed after starting the scheduler, I would use the background scheduler like this:
from apscheduler.schedulers.background import BackgroundScheduler as scheduler
def myfn():
# Insert your requests code here
print('Hello')
sch = scheduler()
sch.add_job(myfn, 'interval', seconds=5)
sch.start()
# This code will be executed after the sceduler has started
try:
print('Scheduler started, ctrl-c to exit!')
while 1:
# Notice here that if you use "pass" you create an unthrottled loop
# try uncommenting "pass" vs "input()" and watching your cpu usage.
# Another alternative would be to use a short sleep: time.sleep(.1)
#pass
#input()
except KeyboardInterrupt:
if sch.state:
sch.shutdown()
BlockingScheduler
If you don't need other code to be executed after starting the scheduler, you can use the blocking scheduler and it's even easier:
apscheduler.schedulers.blocking import BlockingScheduler as scheduler
def myfn():
# Insert your requests code here
print('Hello')
# Execute your code before starting the scheduler
print('Starting scheduler, ctrl-c to exit!')
sch = scheduler()
sch.add_job(myfn, 'interval', seconds=5)
sch.start()

I have never used the scheduler in python before, however this other stackOverflow question seems to deal with that.
It means that the task is taking longer than one second and by default only one concurrent execution is allowed for a given job... -Alex Grönholm
In your case I imagine using threading would meet your needs.
If you created a class that inherited threads in python, something like:
class Requester(threading.Thread):
def __init__(self, url, credentials, payload):
threading.Thread._init__(self)
self.url = url
self.credentials = credentials
self.payload = payload
def run(self):
# do the post request here
# you may want to write output (errors and content) to a file
# rather then just printing it out sometimes when using threads
# it gets really messing if you just print everything out
Then just like how you handle with a slight change.
if __name__ == '__main__':
url = 'http://api/url/'
# Username and password
credentials = { 'username': 'username', 'password': 'password'}
# Defining payloads
payload = dict()
payload['item1'] = 1234
payload['item2'] = 'some string'
data_array = [{"id": "id1", "data": "some value"}]
payload['json_data_array'] = [{ "time": int(time.time()), "data": data_array]
counter = 0
while counter < 1800:
req = Requester(url, credentials, payload)
req.start()
counter++
time.sleep(1)
And of course finish the rest of it however you would like to, if you want to you could make it so that the KeyboardInterrupt is what actually finishes the script.
This of course is a way to get around the scheduler, if that is what the issue is.

Multiple simultaneous HTTP requests

I'm trying to take a list of items and check for their status change based on certain processing by the API. The list will be manually populated and can vary in number to several thousand.
I'm trying to write a script that makes multiple simultaneous connections to the API to keep checking for the status change. For each item, once the status changes, the attempts to check must stop. Based on reading other posts on Stackoverflow (Specifically, What is the fastest way to send 100,000 HTTP requests in Python? ), I've come up with the following code. But the script always stops after processing the list once. What am I doing wrong?
One additional issue that I'm facing is that the keyboard interrup method never fires (I'm trying with Ctrl+C but it does not kill the script.
from urlparse import urlparse
from threading import Thread
import httplib, sys
from Queue import Queue
requestURLBase = "https://example.com/api"
apiKey = "123456"
concurrent = 200
keepTrying = 1
def doWork():
while keepTrying == 1:
url = q.get()
status, body, url = checkStatus(url)
checkResult(status, body, url)
q.task_done()
def checkStatus(ourl):
try:
url = urlparse(ourl)
conn = httplib.HTTPConnection(requestURLBase)
conn.request("GET", url.path)
res = conn.getresponse()
respBody = res.read()
conn.close()
return res.status, respBody, ourl #Status can be 210 for error or 300 for successful API response
except:
print "ErrorBlock"
print res.read()
conn.close()
return "error", "error", ourl
def checkResult(status, body, url):
if "unavailable" not in body:
print status, body, url
keepTrying = 1
else:
keepTrying = 0
q = Queue(concurrent * 2)
for i in range(concurrent):
t = Thread(target=doWork)
t.daemon = True
t.start()
try:
for value in open('valuelist.txt'):
fullUrl = requestURLBase + "?key=" + apiKey + "&value=" + value.strip() + "&years="
print fullUrl
q.put(fullUrl)
q.join()
except KeyboardInterrupt:
sys.exit(1)
I'm new to Python so there could be syntax errors as well... I'm definitely not familiar with multi-threading so perhaps I'm doing something else wrong as well.

In the code, the list is only read once. Should be something like
try:
while True:
for value in open('valuelist.txt'):
fullUrl = requestURLBase + "?key=" + apiKey + "&value=" + value.strip() + "&years="
print fullUrl
q.put(fullUrl)
q.join()
For the interrupt thing, remove the bare except line in checkStatus or make it except Exception. Bare excepts will catch all exceptions, including SystemExit which is what sys.exit raises and stop the python process from terminating.
If I may make a couple comments in general though.
Threading is not a good implementation for such large concurrencies
Creating a new connection every time is not efficient
What I would suggest is
Use gevent for asynchronous network I/O
Pre-allocate a queue of connections same size as concurrency number and have checkStatus grab a connection object when it needs to make a call. That way the connections stay alive, get reused and there is no overhead in creating and destroying them and the increased memory use that goes with it.