I am trying to do stress test on a server using Python 3. The idea is to send an HTTP request to the API server every 1 second for 30 minutes. I tried using requests and apscheduler to do this but I kept getting
Execution of job "send_request (trigger: interval[0:00:01], next run at: 2017-05-23 11:05:46 EDT)"
skipped: maximum number of running instances reached (1)
How can I make this work? Below is my code so far:
import requests, json, time, ipdb
from apscheduler.schedulers.blocking import BlockingScheduler as scheduler
def send_request():
url = 'http://api/url/'
# Username and password
credentials = { 'username': 'username', 'password': 'password'}
# Header
headers = { 'Content-Type': 'application/json', 'Client-Id': 'some string'}
# Defining payloads
payload = dict()
payload['item1'] = 1234
payload['item2'] = 'some string'
data_array = [{"id": "id1", "data": "some value"}]
payload['json_data_array'] = [{ "time": int(time.time()), "data": data_array]
# Posting data
try:
request = requests.post(url, headers = headers, data = json.dumps(payload))
except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as err:
print("Error while trying to POST pid data")
print(err)
finally:
request.close()
print(request.content)
return request.content
if __name__ == '__main__':
sched = scheduler()
print(time.time())
sched.add_job(send_request, 'interval', seconds=1)
sched.start()
print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))
try:
# This is here to simulate application activity (which keeps the main thread alive).
while true:
pass
except (KeyboardInterrupt, SystemExit):
# Not strictly necessary if daemonic mode is enabled but should be done if possible
scheduler.shutdown()
I tried searching on stack overflow but none of the other questions does what I want so far, or maybe I missed something. I would appreciate someone to point me to the correct thread if that is the case. Thank you very much!
I think your error is described well by the duplicate that I marked as well as the answer by #jeff
Edit: Apparently not.. so here I'll describe how to fix the maximum instances problem:
Maximum instances problem
When you're adding jobs to the scheduler there is an argument you can set for the number of maximum allowed concurrent instances of the job. You can should read about this here:
BaseScheduler.add_job()
So, fixing your problem is just a matter of setting this to something higher:
sch.add_job(myfn, 'interval', seconds=1, max_instances=10)
But, how many concurrent requests do you want? If they take more than one second to respond, and you request one per second, you will always eventually get an error if you let it run long enough...
Schedulers
There are several scheduler options available, here are two:
BackgroundScheduler
You're importing the blocking scheduler - which blocks when started. So, the rest of your code is not being executed until after the scheduler stops. If you need other code to be executed after starting the scheduler, I would use the background scheduler like this:
from apscheduler.schedulers.background import BackgroundScheduler as scheduler
def myfn():
# Insert your requests code here
print('Hello')
sch = scheduler()
sch.add_job(myfn, 'interval', seconds=5)
sch.start()
# This code will be executed after the sceduler has started
try:
print('Scheduler started, ctrl-c to exit!')
while 1:
# Notice here that if you use "pass" you create an unthrottled loop
# try uncommenting "pass" vs "input()" and watching your cpu usage.
# Another alternative would be to use a short sleep: time.sleep(.1)
#pass
#input()
except KeyboardInterrupt:
if sch.state:
sch.shutdown()
BlockingScheduler
If you don't need other code to be executed after starting the scheduler, you can use the blocking scheduler and it's even easier:
apscheduler.schedulers.blocking import BlockingScheduler as scheduler
def myfn():
# Insert your requests code here
print('Hello')
# Execute your code before starting the scheduler
print('Starting scheduler, ctrl-c to exit!')
sch = scheduler()
sch.add_job(myfn, 'interval', seconds=5)
sch.start()
I have never used the scheduler in python before, however this other stackOverflow question seems to deal with that.
It means that the task is taking longer than one second and by default only one concurrent execution is allowed for a given job... -Alex Grönholm
In your case I imagine using threading would meet your needs.
If you created a class that inherited threads in python, something like:
class Requester(threading.Thread):
def __init__(self, url, credentials, payload):
threading.Thread._init__(self)
self.url = url
self.credentials = credentials
self.payload = payload
def run(self):
# do the post request here
# you may want to write output (errors and content) to a file
# rather then just printing it out sometimes when using threads
# it gets really messing if you just print everything out
Then just like how you handle with a slight change.
if __name__ == '__main__':
url = 'http://api/url/'
# Username and password
credentials = { 'username': 'username', 'password': 'password'}
# Defining payloads
payload = dict()
payload['item1'] = 1234
payload['item2'] = 'some string'
data_array = [{"id": "id1", "data": "some value"}]
payload['json_data_array'] = [{ "time": int(time.time()), "data": data_array]
counter = 0
while counter < 1800:
req = Requester(url, credentials, payload)
req.start()
counter++
time.sleep(1)
And of course finish the rest of it however you would like to, if you want to you could make it so that the KeyboardInterrupt is what actually finishes the script.
This of course is a way to get around the scheduler, if that is what the issue is.
Related
I am trying to make a brute forcer for my ethical hacking class using multiprocessing, I want it to iterate through the list of server IP's and try one login for each of them, but it is printing every single IP before trying to make connections, and then once all the IP's have been printed, it will start trying to make connections then print a couple IP's, then try to make another connection, and so on.
I just want it to iterate through the list of IP's and try to connect to each one, one process for each connection and try about 20 processes at a time
import threading, requests, time, os, multiprocessing
global count2
login_list=[{"username":"admin","password":"Password1"}]
with open('Servers.txt') as f:
lines = [line.rstrip() for line in f]
count=[]
for number in range(len(lines)):
count.append(number)
count2 = count
def login(n):
try:
url = 'http://'+lines[n]+'/api/auth'
print(url)
if '/#!/init/admin' in url:
print('[~] Admin panel detected, saving url and moving to next...')
x = requests.post(url, json = login_list)
if x.status_code == 422:
print('[-] Failed to connect, trying again...')
print(n)
if x.status_code == 403:
print('[!] 403 Forbidden, "Access denied to resource", Possibly to many tries. Trying again in 20 seconds')
time.sleep(20)
print(n)
if x.status_code == 200:
print('\n[~] Connection successful! Login to '+url+' saved.\n')
print(n)
except:
print('[#] No more logins to try for '+url+' moving to next server...')
print('--------------')
if __name__ == "__main__":
# creating a pool object
p = multiprocessing.Pool()
# map list to target function
result = p.map(login, count2)
An example of the Server.txt file:
83.88.223.86:9000
75.37.144.153:9000
138.244.6.184:9000
34.228.116.82:9000
125.209.107.178:9000
33.9.12.53:9000
Those are not real IP adresses
I think you're confused about how the subprocess map function passes values to the relevant process. Perhaps this will make matters clearer:
from multiprocessing import Pool
import requests
import sys
from requests.exceptions import HTTPError, ConnectionError
IPLIST = ['83.88.223.86:9000',
'75.37.144.153:9000',
'138.244.6.184:9000',
'34.228.116.82:9000',
'125.209.107.178:9000',
'33.9.12.53:9000',
'www.google.com']
PARAMS = {'username': 'admin', 'password': 'passw0rd'}
def err(msg):
print(msg, file=sys.stderr)
def process(ip):
with requests.Session() as session:
url = f'http://{ip}/api/auth'
try:
(r := session.post(url, json=PARAMS, timeout=1)).raise_for_status()
except ConnectionError:
err(f'Unable to connect to {url}')
except HTTPError:
err(f'HTTP {r.status_code} for {url}')
except Exception as e:
err(f'Unexpected exception {e}')
def main():
with Pool() as pool:
pool.map(process, IPLIST)
if __name__ == '__main__':
main()
Additional notes: You probably want to specify a timeout otherwise unreachable addresses will take a long time to process due to default retries. Review the exception handling.
The first thing I would mention is that this is a job best suited for multithreading since login is mostly waiting for network requests to complete and it is far more efficient to create threads than to create processes. In fact you should create a thread pool whose size is equal to the number of URLs you will be posting to up to a maximum of say a 1000 (and you would not want to create a multiprocessing pool of that size).
Second, when you are doing multiprocessing or multithreading your worker function, login in this case, is processing a single element of the iterable that is being passed to the map function. I think you get that. But instead of passing to map the list of servers you are passing a list of numbers (which are indices) and then login is using that index to get the information from the lines list. That is rather indirect. Also, the way you build the list of indices could have been simplified with one line: count2 = list(range(len(lines))) or really just count2 = range(len(lines)) (you don't need a list).
Third, in your code you say that you are retrying certain errors but there is actually no logic to do so.
import requests
from multiprocessing.pool import ThreadPool
from functools import partial
import time
# This must be a dict not a list:
login_params = {"username": "admin", "password": "Password1"}
with open('Servers.txt') as f:
servers = [line.rstrip() for line in f]
def login(session, server):
url = f'http://{server}/api/auth'
print(url)
if '/#!/init/admin' in url:
print(f'[~] Admin panel detected, saving {url} and moving to next...')
# To move on the next, you simply return
# because you are through with this URL:
return
try:
for retry_count in range(1, 4): # will retry up to 3 times certain errors:
r = session.post(url, json=login_params)
if retry_count == 3:
# This was the last try:
break
if r.status_code == 422:
print(f'[-] Failed to connect to {url}, trying again...')
elif r.status_code == 403:
print(f'[!] 403 Forbidden, "Access denied to resource", Possibly to many tries. Trying {url} again in 20 seconds')
time.sleep(20)
else:
break # not something we retry
r.raise_for_status() # test status code
except Exception as e:
print('Got exception: ', e)
else:
print(f'\n[~] Connection successful! Login to {url} saved.\n')
if __name__ == "__main__":
# creating a pool object
with ThreadPool(min(len(servers), 1000)) as pool, \
requests.Session() as session:
# map will return list of None since `login` returns None implicitly:
pool.map(partial(login, session), servers)
I recently encountered an issue that I have not been able to solve, despite calling the Bloomberg helpdesk and researching thoroughly the internet for similar cases.
In short, I am using the official Python blpapi from Bloomberg (https://github.com/msitt/blpapi-python) and now am experiencing some connectivity issue: I cannot leave a session opened.
Here is the code I am running: https://github.com/msitt/blpapi-python/blob/master/examples/SimpleHistoryExample.py
I simply added a "while True loop" and a "time.sleep" in it so that I can keep the session open and refresh my data every 30 seconds (this is my use case).
This use to run perfectly fine for days, however, since last Friday, I am now getting those log messages:
22FEB2021_08:54:18.870 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
22FEB2021_08:54:23.755 29336:26880 WARN blpapi_platformcontroller.cpp:377 blpapi.session.platformcontroller.{1} Connectivity lost, no connected endpoints.
22FEB2021_08:54:31.867 29336:26880 WARN blpapi_platformcontroller.cpp:344 blpapi.session.platformcontroller.{1} Connectivity restored.
22FEB2021_08:54:32.731 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
which goes on and on and on, along with those responses as well:
SessionConnectionDown = {
server = "localhost:8194"
}
ServiceDown = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
SessionConnectionUp = {
server = "localhost:8194"
encryptionStatus = "Clear"
compressionStatus = "Uncompressed"
}
ServiceUp = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
I still can pull the data from the bloomberg API: I see the historical data request results just fine. However:
Those service/session status messages messes up my code (I could still ignore them)
For some reason the connect/reconnect also messes my Excel BBG in the background and prevent me from using the BBG excel add-in at all! I now have those "#N/A Connection" outputs in all of my workbooks using bloomberg formulas.
screenshot from excel
Has anyone ever encountered such cases? If yes, please do not hesitate to share your experience, any help is more than appreciated!
Wishing you all a great day,
Adrien
I cannot comment yet so I will try to "answer" it. I use blpapi everyday and pull data all day. I am a BBG Anywhere user and never have any session issues. If you log in from a different device it will kill your session for the Python app. Once you log back in where the python app is running it will connect again.
Why do you have another while loop and sleep to keep the session alive? You should create a separate session and always call it to run your request. You should not need any "keep alive" code inside the request. Just don't call session.stop(). This is what I ended up doing after much trial and error from not knowing what to do.
I run my model using Excel, trying to move away from any substantial code in Excel and use it as a GUI until I can migrate to a custom GUI. I also have BDP functions in my Excel and they work fine.
import blpapi
# removed optparse because it is deprecated.
from argparse import ArgumentParser
SERVICES = {}
def parseCmdLine():
parser = ArgumentParser(description='Retrieve reference data.')
parser.add_argument('-a',
'--ip',
dest='host',
help='server name or IP (default: %(default)s)',
metavar='ipAddress',
default='localhost')
parser.add_argument('-p',
dest='port',
type=int,
help='server port (default: %(default)s)',
metavar='tcpPort',
default=8194)
args = parser.parse_args()
return args
def start_session():
"""Standard session for synchronous refdata requests. Upon creation
the obj is held in SERVICES['session'].
Returns:
obj: A session object.
"""
args = parseCmdLine()
# Fill SessionOptions
sessionOptions = blpapi.SessionOptions()
sessionOptions.setServerHost(args.host)
sessionOptions.setServerPort(args.port)
# Create a Session
session = blpapi.Session(sessionOptions)
# Start a Session
session.start()
SERVICES['session'] = session
return SERVICES['session']
def get_refDataService():
"""Create a refDataService object for functions to use. Upon creation
it is held in SERVICES['refDataService'].
Returns:
obj: refDataService object.
"""
# return the session and request because requests need session.send()
global SERVICES
if 'session' not in SERVICES:
start_session()
session = SERVICES['session']
# Check for SERVICES['refdata'] not needed because start_session()
# is called by this function and start_session() is never called on its own.
session.openService("//blp/refdata")
refDataService = session.getService("//blp/refdata")
SERVICES['refDataService'] = refDataService
session = SERVICES['session']
refDataService = SERVICES['refDataService']
print('get_refDataService called. Curious when this is called.')
return session, refDataService
# sample override request
def ytw_oride_muni(cusip_dict):
"""Requests the Price To Worst for a dict of cusips and Yield To Worst values.
The dict must be {'cusip' : YTW}. Overrides apply to each request, so this
function is designed for different overrides for each cusip. Although they could
all be the same as that is not a restriction.
Returns: Single level nested dict
{'cusip Muni': {'ID_BB_SEC_NUM_DES': 'val', 'PX_ASK': 'val1', 'YLD_CNV_ASK': 'val2'}}
"""
session, refDataService = get_refDataService()
fields1 = ["ID_BB_SEC_NUM_DES", "PX_ASK", "YLD_CNV_ASK"]
try:
values_dict = {}
# For different overrides you must send separate requests.
# This loops and creates separate messages.
for cusip, value in cusip_dict.items():
request = refDataService.createRequest("ReferenceDataRequest")
# append security to request
request.getElement("securities").appendValue(f"{cusip} Muni")
# append fields to request
request.getElement("fields").appendValue(fields1[0])
request.getElement("fields").appendValue(fields1[1])
request.getElement("fields").appendValue(fields1[2])
# add overrides
overrides = request.getElement("overrides")
override1 = overrides.appendElement()
override1.setElement("fieldId", "YLD_CNV_ASK")
override1.setElement("value", f"{value}")
session.sendRequest(request)
# Process received events
while(True):
# We provide timeout to give the chance to Ctrl+C handling:
ev = session.nextEvent(500)
# below msg.messageType == ReferenceDataResponse
for msg in ev:
if msg.messageType() == "ReferenceDataResponse":
if msg.hasElement("responseError"):
print(msg)
if msg.hasElement("securityData"):
data = msg.getElement("securityData")
num_cusips = data.numValues()
for i in range(num_cusips):
sec = data.getValue(i).getElement("security").getValue()
try:
des = data.getValue(i).getElement("fieldData").getElement("ID_BB_SEC_NUM_DES").getValue()
except:
des = None
try:
ptw = data.getValue(i).getElement("fieldData").getElement("PX_ASK").getValue()
except:
ptw = None
try:
ytw = data.getValue(i).getElement("fieldData").getElement("YLD_CNV_ASK").getValue()
except:
ytw = None
values = {'des': des, 'ptw': ptw, 'ytw': ytw}
# Response completly received, so we could exit
if ev.eventType() == blpapi.Event.RESPONSE:
values_dict.update({sec: values})
break
finally:
# Stop the session
# session.stop()
return values_dict
I wrote a pythonic server with socket. that should receives requests at the same time(parallel) and respond them parallel.
When i send more than one request to it, the time of answering increase more than i expected.
server:
import datetime
import asyncio, timeit
import json, traceback
from asyncio import get_event_loop
requestslist = []
loop = asyncio.get_event_loop()
async def handleData(reader, writer):
message = ''
clientip = ''
data = bytearray()
print("Async HandleData", datetime.datetime.utcnow())
try:
start = timeit.default_timer()
data = await reader.readuntil(separator=b'\r\n\r\n')
msg = data.decode(encoding='utf-8')
len_csharp_message = int(msg[msg.find('content-length:') + 15:msg.find(';dmnid'):])
data = await reader.read(len_csharp_message)
message = data.decode(encoding='utf-8')
clientip = reader._transport._extra['peername'][0]
clientport = reader._transport._extra['peername'][1]
print('\nData Received from:', clientip, ':', clientport)
if (clientip, message) in requestslist:
reader._transport._sock.close()
else:
requestslist.append((clientip, message))
# adapter_result = parallel_members(message_dict, service, dmnid)
adapter_result = '''[{"name": {"data": "data", "type": "str"}}]'''
body = json.dumps(adapter_result, ensure_ascii=False)
print(body)
contentlen = len(bytes(str(body), 'utf-8'))
header = bytes('Content-Length:{}'.format(contentlen), 'utf-8')
result = header + bytes('\r\n\r\n{', 'utf-8') + body + bytes('}', 'utf-8')
stop = timeit.default_timer()
print('total_time:', stop - start)
writer.write(result)
writer.close()
writer.close()
# del writer
except Exception as ex:
writer.close()
print(traceback.format_exc())
finally:
try:
requestslist.remove((clientip, message))
except:
pass
def main(*args):
print("ready")
loop = get_event_loop()
coro = asyncio.start_server(handleData, 'localhost', 4040, loop=loop, limit=204800000)
srv = loop.run_until_complete(coro)
loop.run_forever()
if __name__ == '__main__':
main()
When i send single request, it tooke 0.016 sec.
but for more request, this time increase.
cpu info : intel xeon x5650
client:
import multiprocessing, subprocess
import time
from joblib import Parallel, delayed
def worker(file):
subprocess.Popen(file, shell=False)
def call_parallel (index):
print('begin ' , index)
p = multiprocessing.Process(target=worker(index))
p.start()
print('end ' , index)
path = r'python "/test-Client.py"' # ## client address
files = [path, path, path, path, path, path, path, path, path, path, path, path]
Parallel(n_jobs=-1, backend="threading")(delayed(call_parallel)(i) for index,i in enumerate(files))
for this client that send 12 requests synchronous, total time for per request is 0.15 sec.
I expect for any number requests, the time be fixed.
What is request
Single request (roughly saying) consists of the following steps:
write data to network
waste time waiting for answer
read answer from network
№1/№3 processed by your CPU very fast. Step №2 - is a bytes journey from your PC to some server (in another city, for example) and back by wires: it usually takes much more time.
How asynchronous requests work
Asynchronous requests are not really "parallel" in terms of processing: it's still your single CPU core that can process one thing at a time. But running multiple async requests allows you to use step №2 of some request to do steps №1/№3 of other request instead of just wasting huge amount of time. That's a reason why multiple async requests usually would finish earlier then same amount of sync ones.
Running async code without network delay
But when you run things locally, step №2 doesn't take much time: your PC and server are the same thing and bytes don't go to network journey. There is just no time that can be used in step №2 to start new request. Only your single CPU core works processing one thing at a time.
You should test requests against server that answers with some delay to see results you expect.
I have a lambda that I require to gracefully shutdown and log to an external system. After looking some literature on the matter, I reached the following solution using threading:
def lambda_handler(event, context):
threshold_millis = 10 * 1000 # leave when there are only 10 seconds left
que = queue.Queue()
t = threading.Thread(target=lambda q, ev: q.put(do_work(ev)), args=(que, event))
t.daemon = True
t.start()
while True:
if context.get_remaining_time_in_millis() < threshold_millis:
# Do some logging notifying the timeout
return {
"isBase64Encoded": False,
"statusCode": 408,
"headers": {'Content-Type': "application/json"},
"body": "Request timed out"
}
elif not t.isAlive():
response = que.get()
return response
time.sleep(1)
Although it works I was wondering: is there any better practice than this one to gracefully handle timeouts in AWS Lambda?
Watching get_remaining_time_in_millis is the best/recommended way of preempting a timeout in lambda; there are no special events that get invoked to let you know you're going to timeout.
Without knowing the specifics, you could infer a timeout on the client side by looking at how long it took for you to receive an error. However I would prefer your solution of having the lambda be explicit about it.
I'm trying to take a list of items and check for their status change based on certain processing by the API. The list will be manually populated and can vary in number to several thousand.
I'm trying to write a script that makes multiple simultaneous connections to the API to keep checking for the status change. For each item, once the status changes, the attempts to check must stop. Based on reading other posts on Stackoverflow (Specifically, What is the fastest way to send 100,000 HTTP requests in Python? ), I've come up with the following code. But the script always stops after processing the list once. What am I doing wrong?
One additional issue that I'm facing is that the keyboard interrup method never fires (I'm trying with Ctrl+C but it does not kill the script.
from urlparse import urlparse
from threading import Thread
import httplib, sys
from Queue import Queue
requestURLBase = "https://example.com/api"
apiKey = "123456"
concurrent = 200
keepTrying = 1
def doWork():
while keepTrying == 1:
url = q.get()
status, body, url = checkStatus(url)
checkResult(status, body, url)
q.task_done()
def checkStatus(ourl):
try:
url = urlparse(ourl)
conn = httplib.HTTPConnection(requestURLBase)
conn.request("GET", url.path)
res = conn.getresponse()
respBody = res.read()
conn.close()
return res.status, respBody, ourl #Status can be 210 for error or 300 for successful API response
except:
print "ErrorBlock"
print res.read()
conn.close()
return "error", "error", ourl
def checkResult(status, body, url):
if "unavailable" not in body:
print status, body, url
keepTrying = 1
else:
keepTrying = 0
q = Queue(concurrent * 2)
for i in range(concurrent):
t = Thread(target=doWork)
t.daemon = True
t.start()
try:
for value in open('valuelist.txt'):
fullUrl = requestURLBase + "?key=" + apiKey + "&value=" + value.strip() + "&years="
print fullUrl
q.put(fullUrl)
q.join()
except KeyboardInterrupt:
sys.exit(1)
I'm new to Python so there could be syntax errors as well... I'm definitely not familiar with multi-threading so perhaps I'm doing something else wrong as well.
In the code, the list is only read once. Should be something like
try:
while True:
for value in open('valuelist.txt'):
fullUrl = requestURLBase + "?key=" + apiKey + "&value=" + value.strip() + "&years="
print fullUrl
q.put(fullUrl)
q.join()
For the interrupt thing, remove the bare except line in checkStatus or make it except Exception. Bare excepts will catch all exceptions, including SystemExit which is what sys.exit raises and stop the python process from terminating.
If I may make a couple comments in general though.
Threading is not a good implementation for such large concurrencies
Creating a new connection every time is not efficient
What I would suggest is
Use gevent for asynchronous network I/O
Pre-allocate a queue of connections same size as concurrency number and have checkStatus grab a connection object when it needs to make a call. That way the connections stay alive, get reused and there is no overhead in creating and destroying them and the increased memory use that goes with it.