I am using sleekxmpp as the xmpp client for python. The requests are coming which I am further forwarding to other users/agents.
Now the use case is, if a user is not available we need to check the availability every 10 seconds and transfer to him when he is available. We need to send a message to customer only 5 times but check the availability for a long time.
I am using time.sleep() if the user is not available to check again in 10 seconds, but the problem is it is blocking the entire thread and no new requests are coming to the server.
send_msg_counter = 0
check_status = False
while not check_status:
check_status = requests.post(transfer_chat_url, data=data)
if send_msg_counter < 5:
send_msg("please wait", customer)
send_msg_counter += 1
time.sleep(10)
This is true that time.sleep(10) will block your active thread. You may actually find that using Python 3's async/await to be the way to go. Sadly I don't have much experience with those keywords yet, but another route might be to implement python's threading.
https://docs.python.org/3/library/threading.html
Here might be one way to implement this feature.
import threading
def poll_counter(customer, transfer_chat_url, data, send_count=5, interval=10):
send_msg_counter = 0
check_status = False
while not check_status:
check_status = requests.post(transfer_chat_url, data=data)
if send_msg_counter < send_count:
send_msg("please wait", customer)
send_msg_counter += 1
time.sleep(interval)
# If we're here, check status became true
return None
... pre-existing code ...
threading.Thread(target=poll_counter, args=(customer, transfer_chat_url, data)).start()
... proceed to handle other tasks while the thread runs in the background.
Now, I won't go into detail, but there are use cases where threading is a major mistake. This shouldn't be one of them, but here is a good read for you to understand those use cases.
https://realpython.com/python-gil/
Also, for more details on asyncio (async/await) here is a good resource.
https://docs.python.org/3/library/asyncio-task.html
Try implementing
delay = min(self.reconnect_delay * 2, self.reconnect_max_delay)
delay = random.normalvariate(delay, delay * 0.1)
log.debug('Waiting %s seconds before connecting.', delay)
elapsed = 0
try:
while elapsed < delay and not self.stop.is_set():
time.sleep(0.1)
elapsed += 0.1
except KeyboardInterrupt:
self.set_stop()
return False
except SystemExit:
self.set_stop()
return False
Source Link
Related
I have a pyzmq Publisher which sends around 1000 messages per second. I am trying to start around 10 Subscribers in an asyncio event_loop.
It works but around 2.5 times slower than speed of the only one Subscriber.
What could possibly be wrong with the code?
import asyncio
import zmq
import json
from zmq.backend.cython.constants import NOBLOCK
from zmq.asyncio import Context, Poller
from loop_ import Loop
class Client:
REQUEST_TIMEOUT = 35000
SERVER_ENDPOINT = "tcp://localhost:6666"
def __init__(self, id_):
self.id = id_
def get_task(self):
return asyncio.create_task(self.client_coroutine())
async def client_coroutine(self):
context = Context.instance()
socket = context.socket(zmq.SUB)
socket.connect(self.SERVER_ENDPOINT)
socket.setsockopt(zmq.SUBSCRIBE, b'4')
poller = Poller()
poller.register(socket, zmq.POLLIN)
while True:
event = dict(await poller.poll(self.REQUEST_TIMEOUT))
if event.get(socket) == zmq.POLLIN:
reply = await socket.recv_multipart(flags=NOBLOCK)
if not reply:
break
else:
print(eval(json.loads(reply[1].decode('utf-8'))))
else:
print("No response from server, retrying...")
socket.setsockopt(zmq.LINGER, 0)
socket.close()
poller.unregister(socket)
async def tasks():
_tasks = [Client(id_).get_task() for id_ in range(10)]
done, pending = await asyncio.wait(_tasks, return_when=asyncio.FIRST_EXCEPTION)
loop = asyncio.get_event_loop()
loop.run_until_complete(tasks())
Q : What could possibly be wrong with the code?
Given the code is using the same localhost ( as seen from using the address ), the suspect number one is, that having 10x more work to process, the such workload will always stress the localhost's O/S and the CPU, won't it?
Next comes the choice of the transport-class. Given all the SUB-s are co-located on the same localhost as the PUB, there is all the L3-stack-based TCP/IP protocol work going wasted. To compare the relative costs ( the add-on effect of using the tcp:// transport-class for this hardware-singular messaging ), test the very same with using inproc:// transport-class, where none of the protocol-related TCP/IP-stack add-on processing will take place.
Last, but not least, my code will never mix different event-loops ( using ZeroMQ since v2.11, so someone may consider my a bit old-fashioned in avoiding relying on async-decorated capabilities available in recent py3.6+ )
My code will use an explicit, non-blocking, zero-waiting test for a presence of a message per-aSocketINSTANCE, as in aSocketINSTANCE.poll( zmq.POLLIN, 0 ) rather than using any "externally" added decoration, which may report the same, but via some additional (expensive and outside of my code domain of control) event-handling. All real-time, low-latency use-cases strive to bear as minimum latency/overheads as possible, so using explicit control will always win in my Projects, to any "modern" syntax-sugar sweetened tricks.
Anyway, enjoy the Zen-of-Zero
I am developing a bot with module pyTelegramBotAPI, which is working via webhooks with installed Flask+Gunicorn as server for webhooks. Gunicorn is working with 5 workers for better speed, structure of my project looks like that:
app.py
bot.py
In bot.py i have a function for processing updates:
def process_the_update(update):
logger.info(update)
update = types.Update.de_json(update)
bot.process_new_updates([update])
In app.py i imported this function, so, whenever an update comes, app.py will call this function, and bot will handle the update. In my bot user can call a command, which will use an external api to get some information. The problem is, that this external api has limit for 3 requests per second. I need to configure a bot with such rate limit. First i thought of doing it with Queue with code like this:
lock_queue = Queue(1)
requests_queue = Queue(3)
def api_request(argument):
if lock_queue.empty():
try:
requests_queue.put_nowait(time.time())
except queue.Full:
lock_queue.put(1)
first_request_time = requests_queue.get()
logger.info('First request time: ' + str(first_request_time))
current_time = time.time()
passed_time = current_time - first_request_time
if passed_time >= 1:
requests_queue.put_nowait(time.time())
lock_queue.get()
else:
logger.info(passed_time)
time.sleep(1 - passed_time)
requests_queue.put_nowait(time.time())
lock_queue.get()
else:
lock_queue.put(1)
first_request_time = vk_requests_queue.get()
logger.info('First request time: ' + str(first_request_time))
current_time = time.time()
passed_time = current_time - first_request_time
if passed_time >= 1:
requests_queue.put_nowait(time.time())
lock_queue.get()
else:
logger.info(passed_time)
time.sleep(1 - passed_time)
requests_queue.put_nowait(time.time())
lock_queue.get()
result = make_api_request(argument) # requests are made too by external module.
return result
The logic was that, as i thought, because module pyTelegramBotAPI uses threads for faster updates handling, all threads would check requests_queue, which will have time of 3 last api_requests, and so the time of the first of 3 requests made will be compared to the current time (to check, if a second passed). And, because i needed to be sure, that only one thread would do this kind of comparison simultaneously, i made lock_queue.
But, the problem is that, firstly, gunicorn uses 5 workers, so there will be always possibility, that all messages from users will be handled in different processes, and these processes will have their own queues. And, secondly, even if i set number of workers to default (1 worker), i still get 429 error, so i think, that my code won't work as i wanted at all.
I thought of making rate limit with redis, so every time in every thread and process bot will check the time of last 3 requests, but still i am not sure, that this is the right way, and i am not sure, how to write this.
Would be glad, if someone suggests any ideas or even examples of code (external api does not provide any x-rate-limit header)
Wrote this function, using redis to count requests (based on this https://www.binpress.com/tutorial/introduction-to-rate-limiting-with-redis/155 tutorial)
import redis
r_db = redis.Redis(port=port, db=db)
def limit_request(request_to_make, limit=3, per=1, request_name='test', **kwargs):
over_limit_lua_ = '''
local key_name = KEYS[1]
local limit = tonumber(ARGV[1])
local duration = ARGV[2]
local key = key_name .. '_num_of_requests'
local count = redis.call('INCR', key)
if tonumber(count) > limit then
local time_left = redis.call('PTTL', key)
return time_left
end
redis.call('EXPIRE', key, duration)
return -2
'''
if not hasattr(r_db, 'over_limit_lua'):
r_db.over_limit_lua = r_db.register_script(over_limit_lua_)
request_possibility = int(r_db.over_limit_lua(keys=request_name, args=[limit, per]))
if request_possibility > 0:
time.sleep(request_possibility / 1000.0)
return limit_request(request_to_make, limit, per, request_name, **kwargs)
else:
request_result = request_to_make(**kwargs)
return request_result
I have a script that gets launched on boot, and it is possible that it would be launched before networking is fully up.
The following code fails if it is run before networking is up, if it gets called again later it succeeds.
Even if I increase the tries to 5 minutes, it will still continue until the 5 minutes and then return false, even though networking comes up probably less than 30 seconds after the script launches.
Rather than just sleeping for 1 minute before making any attempt, is there a way to make the following code work and not die if the ethernet is not up ?
self.TRIES = 60
self.URL="http://www.somedomain.com"
## Do we have internet
def isup():
try:
urllib2.urlopen(self.URL).close()
return True
except urllib2.URLError,e:
pass
return False
## Try the lookup
while (self.TRIES > 0):
if isup():
check()
break
self.TRIES = self.TRIES-1
time.sleep(1)
Edit
During OS bootup (Arch Linux) the adapter (eth0 in this case) and the networking service are initially not running, and are started during the bootup process.
It appears that urllib2 (and other network-related calls) die if it is called before networking service is fully up, and subsequent calls will always result in a fail.
This is NOT the same as just disconnecting the ethernet cable, if you just unplug the ethernet cable and call the function (class) then it will succeed, but if it is called BEFORE the networking service is fully up, it will fail and die.
I can solve this problem by adding a time.sleep(30) to the top of the code, this then gives enough time for the O/S network service to fully start and the script works 100% as expected.
use requests and check the status code?
import requests
In [36]: r = requests.get('http://httpbin.org/get')
In [37]: r.status_code == requests.codes.ok
Out[37]: True
In [38]: r.status_code
Out[38]: 200
200
In [33]: r = requests.get('http://httpbin.org/bad')
In [34]: r.status_code
Out[34]: 404
In [35]: r.status_code == requests.codes.ok
Out[35]: False
def isup():
try:
r = requests.get(self.URL)
return r.status_code == requests.codes.ok
except Exception, e:
print e
return False
You could do it as:
def isup():
try:
urllib2.urlopen(self.URL).close()
return True
except urllib2.URLError,e:
pass
return False
## Try the lookup
while not isup():
pass #or replace pass with time.sleep(1) or time.sleep(0.5)
check()
Why do you not use ping?
def isUp():
ret = os.system("ping -c 1 www.google.com")
return (ret != 0)
NOTE: this does not work on Windows as is but you got the idea...
We are using celery to get flights data from different travel
agencies, every request takes ~20-30 seconds(most agencies require
request sequence - authorize, send request, poll for results).
Normal
celery task looks like this:
from eventlet.green import urllib2, time
def get_results(attr, **kwargs):
search, provider, minprice = attr
data = XXX # prepared data
host = urljoin(MAIN_URL, "RPCService/Flights_SearchStart")
req = urllib2.Request(host, data, {'Content-Type': 'text/xml'})
try:
response_stream = urllib2.urlopen(req)
except urllib2.URLError as e:
return [search, None]
response = response_stream.read()
rsp_host = urljoin(MAIN_URL, "RPCService/FlightSearchResults_Get")
rsp_req = urllib2.Request(rsp_host, response, {'Content-Type':
'text/xml'})
ready = False
sleeptime = 1
rsp_response = ''
while not ready:
time.sleep(sleeptime)
try:
rsp_response_stream = urllib2.urlopen(rsp_req)
except urllib2.URLError as e:
log.error('go2see: results fetch failed for %s IOError %s'%
(search.id, str(e)))
else:
rsp_response = rsp_response_stream.read()
try:
rsp = parseString(rsp_response)
except ExpatError as e:
return [search, None]
else:
ready = rsp.getElementsByTagName('SearchResultEx')
[0].getElementsByTagName('IsReady')[0].firstChild.data
ready = (ready == 'true')
sleeptime += 1
if sleeptime > 10:
return [search, None]
hash = "%032x" % random.getrandbits(128)
open(RESULT_TMP_FOLDER+hash, 'w+').write(rsp_response)
# call to parser
parse_agent_results.apply_async(queue='parsers', args=[__name__,
search, provider, hash])
This tasks are run in eventlet pool with concurency 300,
prefetch_multiplier = 1, broker_limit = 300
When ~100-200 task are fetched from queue - CPU usage raises up to 100%
( whole CPU core is used) and task fetching from queue is performed
with delays.
Could you please point on possible issues - blocking
operations( eventlet ALARM DETECTOR gives no exceptions ), wrong
architecture or whatever.
A problem occurs if you fire 200 requests to a server, responses could be delayed and therefore urllib.urlopen will hang.
Another thing i noticed: If an URLError is raised, the program stays in the while loop until sleeptime is greater than 10. So an URLError error will let this script sleep for 55 sec (1+2+3.. etc)
Sorry for late response.
Thing i would try first in such situation is to turn off Eventlet completely in both Celery and your code, use process or OS thread model. 300 threads or even processes is not that much load for OS scheduler (although you may lack memory to run many processes). So i would try it and see if CPU load drops dramatically. If it does not, then problem is in your code and Eventlet can't magically fix it. If it does drop, however, we would need to investigate the issue closer.
If bug still persists, please, report it via any of these ways:
https://bitbucket.org/which_linden/eventlet/issues/new
https://github.com/eventlet/eventlet/issues/new
email to eventletdev#lists.secondlife.com
How can I run background tasks on App Engine?
You may use the Task Queue Python API.
GAE is very useful tool to build scalable web applications. Few of the limitations pointed out by many are no support for background tasks, lack of periodic tasks and strict limit on how much time each HTTP request takes, if a request exceeds that time limit the operation is terminated, which makes running time consuming tasks impossible.
How to run background task ?
In GAE the code is executed only when there is a HTTP request. There is a strict time limit (i think 10secs) on how long the code can take. So if there are no requests then code is not executed. One of the suggested work around was use an external box to send requests continuously, so kind of creating a background task. But for this we need an external box and now we dependent on one more element. The other alternative was sending 302 redirect response so that client re-sends the request, this also makes us dependent on external element which is client. What if that external box is GAE itself ? Everyone who has used functional language which does not support looping construct in the language is aware of the alternative ie recursion is the replacement to loop. So what if we complete part of the computation and do a HTTP GET on the same url with very short time out say 1 second ? This creates a loop(recursion) on php code running on apache.
<?php
$i = 0;
if(isset($_REQUEST["i"])){
$i= $_REQUEST["i"];
sleep(1);
}
$ch = curl_init("http://localhost".$_SERVER["PHP_SELF"]."?i=".($i+1));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 1);
curl_exec($ch);
print "hello world\n";
?>
Some how this does not work on GAE. So what if we do HTTP GET on some other url say url2 which does HTTP GET on the first url ? This seem to work in GAE. Code for this looks like this.
class FirstUrl(webapp.RequestHandler):
def get(self):
self.response.out.write("ok")
time.sleep(2)
urlfetch.fetch("http://"+self.request.headers["HOST"]+'/url2')
class SecondUrl(webapp.RequestHandler):
def get(self):
self.response.out.write("ok")
time.sleep(2)
urlfetch.fetch("http://"+self.request.headers["HOST"]+'/url1')
application = webapp.WSGIApplication([('/url1', FirstUrl), ('/url2', SecondUrl)])
def main():
run_wsgi_app(application)
if __name__ == "__main__":
main()
Since we found out a way to run background task, lets build abstractions for periodic task (timer) and a looping construct which spans across many HTTP requests (foreach).
Timer
Now building timer is straight forward. Basic idea is to have list of timers and the interval at which each should be called. Once we reach that interval call the callback function. We will use memcache to maintain the timer list. To find out when to call callback, we will store a key in memcache with interval as expiration time. We periodically (say 5secs) check if that key is present, if not present then call the callback and again set that key with interval.
def timer(func, interval):
timerlist = memcache.get('timer')
if(None == timerlist):
timerlist = []
timerlist.append({'func':func, 'interval':interval})
memcache.set('timer-'+func, '1', interval)
memcache.set('timer', timerlist)
def checktimers():
timerlist = memcache.get('timer')
if(None == timerlist):
return False
for current in timerlist:
if(None == memcache.get('timer-'+current['func'])):
#reset interval
memcache.set('timer-'+current['func'], '1', current['interval'])
#invoke callback function
try:
eval(current['func']+'()')
except:
pass
return True
return False
Foreach
This is needed when we want to do long taking computation say doing some operation on 1000 database rows or fetch 1000 urls etc. Basic idea is to maintain list of callbacks and arguments in memcache and each time invoke callback with the argument.
def foreach(func, args):
looplist = memcache.get('foreach')
if(None == looplist):
looplist = []
looplist.append({'func':func, 'args':args})
memcache.set('foreach', looplist)
def checkloops():
looplist = memcache.get('foreach')
if(None == looplist):
return False
if((len(looplist) > 0) and (len(looplist[0]['args']) > 0)):
arg = looplist[0]['args'].pop(0)
func = looplist[0]['func']
if(len(looplist[0]['args']) == 0):
looplist.pop(0)
if((len(looplist) > 0) and (len(looplist[0]['args']) > 0)):
memcache.set('foreach', looplist)
else:
memcache.delete('foreach')
try:
eval(func+'('+repr(arg)+')')
except:
pass
return True
else:
return False
# instead of
# foreach index in range(0, 1000):
# someoperaton(index)
# we will say
# foreach('someoperaton', range(0, 1000))
Now building a program which fetches list of urls every one hour is straight forward. Here is the code.
def getone(url):
try:
result = urlfetch.fetch(url)
if(result.status_code == 200):
memcache.set(url, '1', 60*60)
#process result.content
except :
pass
def getallurl():
#list of urls to be fetched
urllist = ['http://www.google.com/', 'http://www.cnn.com/', 'http://www.yahoo.com', 'http://news.google.com']
fetchlist = []
for url in urllist:
if (memcache.get(url) is None):
fetchlist.append(url)
#this is equivalent to
#for url in fetchlist: getone(url)
if(len(fetchlist) > 0):
foreach('getone', fetchlist)
#register the timer callback
timer('getallurl', 3*60)
complete code is here http://groups.google.com/group/httpmr-discuss/t/1648611a54c01aa
I have been running this code on appengine for few days without much problem.
Warning: We make heavy use of urlfetch. The limit on no of urlfetch per day is 160000. So be careful not to reach that limit.
You can find more about cron jobs in Python App Engine here.
Up and coming version of runtime will have some kind of periodic execution engine a'la cron. See this message on AppEngine group.
So, all the SDK pieces appear to work, but my testing indicates this isn't running on the production servers yet-- I set up an "every 1 minutes" cron that logs when it runs, and it hasn't been called yet
Hard to say when this will be available, though...
Using the Deferred Python Library is the easiest way of doing background task on Appengine using Python which is built on top of TaskQueue API.
from google.appengine.ext import deferred
def do_something_expensive(a, b, c=None):
logging.info("Doing something expensive!")
# Do your work here
# Somewhere else
deferred.defer(do_something_expensive, "Hello, world!", 42, c=True)
If you want to run background periodic tasks, see this question (AppEngine cron)
If your tasks are not periodic, see Task Queue Python API or Task Queue Java API
There is a cron facility built into app engine.
Please refer to:
https://developers.google.com/appengine/docs/python/config/cron?hl=en
Use the Task Queue - http://code.google.com/appengine/docs/java/taskqueue/overview.html