When the loadbalancer in front of the tested https web site fails-over, this generates some HTTPError 500 for few seconds, then Locust hangs:
The response time graph stops (empty graph)
The total requests per second turns to a wrong green flat line.
If I just stop & start the test, locust restart monitoring properly the response time.
We can see some HTTPError 500 in the Failures tab
Is this a bug ?
How can I make sure Locust kills and restarts users, either manually or when timeout ?
My attempt to regularly "RescheduleTaskImmediately" did not help.
My locustfile.py:
#!/usr/bin/env python
import time
import random
from locust import HttpUser, task, between, TaskSet
from locust.exception import InterruptTaskSet, RescheduleTaskImmediately
URL_LIST = [
"/url1",
"/url2",
"/url3",
]
class QuickstartTask(HttpUser):
wait_time = between(0.1, 0.5)
connection_timeout = 15.0
network_timeout = 20.0
def on_start(self):
# Required to use the http_proxy & https_proxy
self.client.trust_env = True
print("New user started")
self.client.timeout = 5
self.client.get("/")
self.client.get("/favicon.ico")
self.getcount = 0
def on_stop(self):
print("User stopped")
#task
def track_and_trace(self):
url = URL_LIST[random.randrange(0,len(URL_LIST))]
self.client.get(url, name=url[:50])
self.getcount += 1
if self.getcount > 50 and (random.randrange(0,1000) > 990 or self.getcount > 200):
print(f'Reschedule after {self.getcount} requests.')
self.client.cookies.clear()
self.getcount = 0
raise RescheduleTaskImmediately
Each locust runs in a thread. If the thread gets blocked, it doesn't take further actions.
self.client.get(url, name=url[:50], timeout=.1)
Something like this is probably what you need, potentially with a try/except to do something different when you get an http timeout exception.
In my experience, the problem you're describing with the charts on the Locust UI has nothing to do with the errors your Locust users are hitting. I've seen this behavior if you have multiple people attempting to access the Locust UI simultaneously. Locust uses Flask to create and serve the UI. Flask by itself (at the way Locust is using it) doesn't do well with multiple connections.
If Person A starts using Locust UI and starts a test, they'll see stats and everything working fine until Person B loads the Locust UI. Person B will then see things working fine but Person A will experience issues as you describe, with the test seemingly stalling and charts not updating properly. In that state, sometimes starting a new test would resolve it temporarily, other times you need to refresh. Either way, A and B would be fighting between each other for a working UI.
The solution in this case would be to put Locust behind a reverse proxy using something such as Nginx. Nginx then maintains a single connection to Locust and all users connect through Nginx. Locust's UI should then continue to work for all connected users with correctly updating stats and charts.
Related
I want to parallely send a GET request for the specified count say 100 times. How to achieve this using JMeter or Python ?
I tried bzm parallel executor but that doesn't workout.
import requests
import threading
totalRequests = 0
numberOfThreads = 10
threads = [0] * numberOfThreads
def worker(thread):
r = requests.get("url")
threads[thread] = 0 # free thread
while totalRequests < 100:
for thread in range(numberOfThreads):
if threads[thread] == 0:
threads[thread] = 1 # occupy thread
t = threading.Thread(target=worker, args=(thread,))
t.start()
totalRequests += 1
In JMeter:
Add Thread Group to your Test Plan and configure it like:
Add HTTP Request sampler as a child of the Thread Group and specify protocol, host, port, path and parameters:
if you're not certain regarding properly configuring the HTTP Request sampler - you can just record the request using your browser and JMeter's HTTP(S) Test Script Recorder or JMeter Chrome Extension
For Python the correct would be using Locust framework as I believe you're interested in metrics like response times, latencies and so on. The official website is down at the moment
so in the meantime you can check https://readthedocs.org/projects/locust/
I have a time-consuming task in route /fibo and a route /home that return hello.
from flask import Flask, request
from fibo import fib_t
from datetime import datetime
app = Flask(__name__)
#app.route('/home')
def hello():
return 'hello '
#app.route('/fibo')
def cal_fibo():
request_rec_t = datetime.now().strftime("%M:%S")
v, duretion = fib_t(37)
return f'''
<div><h1>request recieved by server:</h1>
<p>{request_rec_t}</p>
<p>
<ul>
<li><b>calculated value</b>:{v}</li>
<li><b>that takes</b>: {duretion} seconds</li>
<ul>
</p>
</div>
<div>
<h1>resonse send by server:</h1>
<p>{datetime.now().strftime("%M:%S")}</p>
</div>'''
if __name__ == '__main__':
app.run(debug=True)
the fibo module
import time
def fib(a):
if a == 1 or a==2:
return 1
else:
return fib(a-1) + fib(a-2)
def fib_t(a):
t = time.time()
v = fib(a)
return v, f'{time.time()-t:.2f}'
if I made two requests to /fibo the second one will start running after the first one finishes his work.
but if I send a request to the time-consuming task and another one to /home I will receive the /home route response immediately while still first request isn't resolved.
if this is matters, I make the requests from different tabs of the browser.
for example, if I request /fibo in two different tabs. the page starts to load. the captured time for the first request is 10:10 and 10:20(the execution of Fibonacci number is tasked 10 seconds) during this request to be finished, the second tab still is loading. after that the second request is finished i see that the process started at the 10:20(the time that first request finishes process) and finished in 10:29(the execution of Fibonacci number is tasked 9 seconds).
but if I request /home while the /fibo in the first tab isn't returned, I will get the response.
why this is happening?
my general question is how and by whom the multiple requests are being handled?
how can I log the time requests reached the server?
EDIT:
if I add another route fibo2 with the same logic as cal_fibo view function, now I can run them concurrently(like /fibo and /home that I showed that are running concurrently, /fibo and /fibo2 are also running concurrently.)
EDIT 2:
as shown in the image, the second request was sent after the first request's response was received. why and how did this has happened?
(the related TCP handshake of each request has happened close to each other but how and who managed that the related HTTP get request is sent after the first request's response received?)
actually you have only one process running with only one thread all the time to enable threading and running more than one process this depends on the context (development or production) .
for development purposes
to enable threading you need to run
app.run(host="your.host", port=4321, threaded=True)
also to enable running more than one process e.g. 3 according to documentation
you need to run the following line
app.run(host="your.host", port=4321, processes=3)
for production purposes
when you run the app in production environment it is the responsability of the WSGI gateway (eg. gunicorn) that you configure on the cloud provider like heroku or aws to support the handling of multiple requests.
we use this method in production as it is more robust to crashes and also more efficent.
this was just a summary of the answers in this question
I am trying to build REST API with only one call.
Sometimes it takes up to 30 seconds for a program to return a response. But if user thinks that service is lagging - he makes a new call and my app returns response with error code 500 (Internal Server Error).
For now it is enough for me to block any new requests if last one is not ready. Is there any simple way to do it?
I know that there is a lot of queueing managers like Celery, but I prefer not to overload my app with any large dependencies/etc.
You could use Flask-Limiter to ignore new requests from that remote address.
pip install Flask-Limiter
Check this quickstart:
from flask import Flask
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
app = Flask(__name__)
limiter = Limiter(
app,
key_func=get_remote_address,
default_limits=["200 per day", "50 per hour"]
)
#app.route("/slow")
#limiter.limit("1 per day")
def slow():
return "24"
#app.route("/fast")
def fast():
return "42"
#app.route("/ping")
#limiter.exempt
def ping():
return "PONG"
As you can see, you could ignore the remote IP address for a certain amount of time meanwhile you finish the process you´re running
DOCS
Check these two links:
Flasf-Limiter Documentation
Flasf-Limiter Quick start
I need a webserver which routes the incoming requests to back-end workers by batching them every 0.5 second or when it has 50 http requests whichever happens earlier. What will be a good way to implement it in python/tornado or any other language?
What I am thinking is to publish the incoming requests to a rabbitMQ queue and then somehow batch them together before sending to the back-end servers. What I can't figure out is how to pick multiple requests from the rabbitMq queue. Could someone point me to right direction or suggest some alternate apporach?
I would suggest using a simple python micro web framework such as bottle. Then you would send the requests to a background process via a queue (thus allowing the connection to end).
The background process would then have a continuous loop that would check your conditions (time and number), and do the job once the condition is met.
Edit:
Here is an example webserver that batches the items before sending them to any queuing system you want to use (RabbitMQ always seemed overcomplicated to me with Python. I have used Celery and other simpler queuing systems before). That way the backend simply grabs a single 'item' from the queue, that will contain all required 50 requests.
import bottle
import threading
import Queue
app = bottle.Bottle()
app.queue = Queue.Queue()
def send_to_rabbitMQ(items):
"""Custom code to send to rabbitMQ system"""
print("50 items gathered, sending to rabbitMQ")
def batcher(queue):
"""Background thread that gathers incoming requests"""
while True:
batcher_loop(queue)
def batcher_loop(queue):
"""Loop that will run until it gathers 50 items,
then will call then function 'send_to_rabbitMQ'"""
count = 0
items = []
while count < 50:
try:
next_item = queue.get(timeout=.5)
except Queue.Empty:
pass
else:
items.append(next_item)
count += 1
send_to_rabbitMQ(items)
#app.route("/add_request", method=["PUT", "POST"])
def add_request():
"""Simple bottle request that grabs JSON and puts it in the queue"""
request = bottle.request.json['request']
app.queue.put(request)
if __name__ == '__main__':
t = threading.Thread(target=batcher, args=(app.queue, ))
t.daemon = True # Make sure the background thread quits when the program ends
t.start()
bottle.run(app)
Code used to test it:
import requests
import json
for i in range(101):
req = requests.post("http://localhost:8080/add_request",
data=json.dumps({"request": 1}),
headers={"Content-type": "application/json"})
I'm trying to send ~400 HTTP GET requests and collect the results.
I'm running from django.
My solution was to use celery with gevent.
To start the celery tasks I call get_reports :
def get_reports(self, clients, *args, **kw):
sub_tasks = []
for client in clients:
s = self.get_report_task.s(self, client, *args, **kw).set(queue='io_bound')
sub_tasks.append(s)
res = celery.group(*sub_tasks)()
reports = res.get(timeout=30, interval=0.001)
return reports
#celery.task
def get_report_task(self, client, *args, **kw):
report = send_http_request(...)
return report
I use 4 workers:
manage celery worker -P gevent --concurrency=100 -n a0 -Q io_bound
manage celery worker -P gevent --concurrency=100 -n a1 -Q io_bound
manage celery worker -P gevent --concurrency=100 -n a2 -Q io_bound
manage celery worker -P gevent --concurrency=100 -n a3 -Q io_bound
And I use RabbitMq as the broker.
And although it works much faster than running the requests sequentially (400 requests took ~23 seconds), I noticed that most of that time was overhead from celery itself, i.e. if I changed get_report_task like this:
#celery.task
def get_report_task(self, client, *args, **kw):
return []
this whole operation took ~19 seconds.
That means that I spend 19 seconds only on sending all the tasks to celery and getting the results back
The queuing rate of messages to rabbit mq is seems to be bound to 28 messages / sec and I think that this is my bottleneck.
I'm running on a win 8 machine if that matters.
some of the things I've tried:
using redis as broker
using redis as results backend
tweaking with those settings
BROKER_POOL_LIMIT = 500
CELERYD_PREFETCH_MULTIPLIER = 0
CELERYD_MAX_TASKS_PER_CHILD = 100
CELERY_ACKS_LATE = False
CELERY_DISABLE_RATE_LIMITS = True
I'm looking for any suggestions that will help speed things up.
Are you really running on Windows 8 without a Virtual Machine? I did the following simple test on 2 Core Macbook 8GB RAM running OS X 10.7:
import celery
from time import time
#celery.task
def test_task(i):
return i
grp = celery.group(test_task.s(i) for i in range(400))
tic1 = time(); res = grp(); tac1 = time()
print 'queued in', tac1 - tic1
tic2 = time(); vals = res.get(); tac2 = time()
print 'executed in', tac2 - tic2
I'm using Redis as broker, Postgres as a result backend and default worker with --concurrency=4. Guess what is the output? Here it is:
queued in 3.5009469986
executed in 2.99818301201
Well it turnes out I had 2 separate issues.
First off, the task was a member method. After extracting it out of the class, the time went down to about 12 seconds. I can only assume it has something to do with the pickling of self.
The second thing was the fact that it ran on windows.
After running it on my linux machine, the run time was less than 2 seconds.
Guess windows just isn't cut for high performance..
How about using twisted instead? You can reach for much simpler application structure. You can send all 400 requests from the django process at once and wait for all of them to finish. This works simultaneously because twisted sets the sockets into non-blocking mode and only reads the data when its available.
I had a similar problem a while ago and I've developed a nice bridge between twisted and django. I'm running it in production environment for almost a year now. You can find it here: https://github.com/kowalski/featdjango/. In simple words it has the main application thread running the main twisted reactor loop and the django view results is delegated to a thread. It use a special threadpool, which exposes methods to interact with reactor and use its asynchronous capabilities.
If you use it, your code would look like this:
from twisted.internet import defer
from twisted.web.client import getPage
import threading
def get_reports(self, urls, *args, **kw):
ct = threading.current_thread()
defers = list()
for url in urls:
# here the Deferred is created which will fire when
# the call is complete
d = ct.call_async(getPage, args=[url] + args, kwargs=kw)
# here we keep it for reference
defers.append(d)
# here we create a Deferred which will fire when all the
# consiting Deferreds are completed
deferred_list = defer.DeferredList(defers, consumeErrors=True)
# here we tell the current thread to wait until we are done
results = ct.wait_for_defer(deferred_list)
# the results is a list of the form (C{bool} success flag, result)
# below unpack it
reports = list()
for success, result in results:
if success:
reports.append(result)
else:
# here handle the failure, or just ignore
pass
return reports
This still is something you can optimize a lot. Here, every call to getPage() would create a separate TCP connection and close it when its done. This is as optimal as it can be, providing that each of your 400 requests is sent to a different host. If this is not a case, you can use a http connection pool, which uses persistent connections and http pipelineing. You instantiate it like this:
from feat.web import httpclient
pool = httpclient.ConnectionPool(host, port, maximum_connections=3)
Than a single request is perform like this (this goes instead the getPage() call):
d = ct.call_async(pool.request, args=(method, path, headers, body))