I have been working on prometheus and Python where I want to be able to have multiple scripts that writes to Promethethus.
Currently I have done 2 scripts:
sydsvenskan.py
import time
import requests
from prometheus_client import Counter
REQUEST_COUNT = Counter(
namespace="scraper",
name="request_count",
documentation="Count the total requests",
labelnames=['http_status']
)
def monitor_feed():
while True:
with requests.get("https://sydsvenskan.se") as rep:
print("Request made!")
REQUEST_COUNT.labels(http_status=rep.status_code).inc()
time.sleep(10)
if __name__ == '__main__':
monitor_feed()
BBC.py
import time
import requests
from prometheus_client import Counter
REQUEST_COUNT = Counter(
namespace="scraper",
name="request_count",
documentation="Count the total requests",
labelnames=['http_status']
)
def monitor_feed():
while True:
with requests.get("https://bbc.com") as rep:
print("Request made!")
REQUEST_COUNT.labels(http_status=rep.status_code).inc()
time.sleep(10)
if __name__ == '__main__':
monitor_feed()
and then I have another script that just starts the promethethus http_server:
from prometheus_client import start_http_server
if __name__ == '__main__':
start_http_server(8000)
however the problem is it seems like nothing goes through the promethethus from the sydsvenskan.py and bbc.py and I wonder what am I doing wrong? I do not see any statistics growing when running the sydsvenskan and bbc at the same time
You need to combine the start_http_server function with your monitor_feed functions.
You can either combine everything under a single HTTP server.
Or, as I think you want, you'll need to run 2 HTTP servers, one with each monitor_feed:
import time
import requests
from prometheus_client import Counter
from prometheus_client import start_http_server
REQUEST_COUNT = Counter(
namespace="scraper",
name="request_count",
documentation="Count the total requests",
labelnames=['http_status']
)
def monitor_feed():
while True:
with requests.get("https://bbc.com") as rep:
print("Request made!")
REQUEST_COUNT.labels(http_status=rep.status_code).inc()
time.sleep(10)
if __name__ == '__main__':
start_http_server(8000)
monitor_feed()
In the latter case, if you run both servers on the same host machine, you'll need to use 2 different ports (you can't use 8000 for both).
Related
tl,dr: How can I programmably execute a python module (not function) as a separate process from a different python module?
On my development laptop, I have a 'server' module containing a bottle server. In this module, the name==main clause starts the bottle server.
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
if __name__ == '__main__':
serve(bt_app, port=localhost:8080)
I also have a 'test_server' module containing pytests. In this module, the name==main clause runs pytest and displays the results.
def test_something():
_rtn = some_server_function()
assert _rtn == desired
if __name__ == '__main__':
_rtn = pytest.main([__file__])
print("Pytest returned: ", _rtn)
Currently, I manually run the server module (starting the web server on localhost), then I manually start the pytest module which issues html requests to the running server module and checks the responses.
Sometimes I forget to start the server module. No big deal but annoying. So I'd like to know if I can programmatically start the server module as a separate process from the pytest module (just as I'm doing manually now) so I don't forget to start it manually.
Thanks
There is my test cases dir tree:
test
├── server.py
└── test_server.py
server.py start a web server with flask.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
test_server.py make request to test.
import sys
import requests
import subprocess
import time
p = None # server process
def start_server():
global p
sys.path.append('/tmp/test')
# here you may want to do some check.
# whether the server is already started, then pass this fucntion
kwargs = {} # here u can pass other args needed
p = subprocess.Popen(['python','server.py'], **kwargs)
def test_function():
response = requests.get('http://localhost:5000/')
print('This is response body: ', response.text)
if __name__ == '__main__':
start_server()
time.sleep(3) # waiting server started
test_function()
p.kill()
Then you can do python test_server to start the server and do test cases.
PS: Popen() needs python3.5+. if older version, use run instead
import logging
import threading
import time
def thread_function(name):
logging.info("Thread %s: starting", name)
time.sleep(2)
logging.info("Thread %s: finishing", name)
if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
datefmt="%H:%M:%S")
threads = list()
for index in range(3):
logging.info("Main : create and start thread %d.", index)
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for index, thread in enumerate(threads):
logging.info("Main : before joining thread %d.", index)
thread.join()
logging.info("Main : thread %d done", index)
With threading you can run multiple processes at once!
Wim baasically answered this question. I looked into the subprocess module. While reading up on it, I stumbled on the os.system function.
In short, subprocess is a highly flexible and functional program for running a program. os.system, on the other hand, is much simpler, with far fewer functions.
Just running a python module is simple, so I settled on os.system.
import os
server_path = "python -m ../src/server.py"
os.system(server_path)
Wim, thanks for the pointer. Had it been a full fledged answer I would have upvoted it. Redo it as a full fledged answer and I'll do so.
Async to the rescue.
import gevent
from gevent import monkey, spawn
monkey.patch_all()
from gevent.pywsgi import WSGIServer
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
def test_something():
_rtn = some_server_function()
assert _rtn == desired
print("Pytest returned: ",_rtn)
sleep(0)
if __name__ == '__main__':
spawn(test_something) #runs async
server = WSGIServer(("0.0.0.0", 8080, bt_app)
server.serve_forever()
I'd want to make a script that automatically sends alot of requests to a URL via Python.
Example link: https://page-views.glitch.me/badge?page_id=page.id
I've tried selenium but thats very slow.
pip install requests
import requests
for i in range(100): # Or whatever amount of requests you wish to send
requests.get("https://page-views.glitch.me/badge?page_id=page.id")
Or if you really wanted to hammer the address you could use multiprocessing
import multiprocessing as mp
import requests
def my_func(x):
for i in range(x):
print(requests.get("https://page-views.glitch.me/badge?page_id=page.id"))
def main():
pool = mp.Pool(mp.cpu_count())
pool.map(my_func, range(0, 100))
if __name__ == "__main__":
main()
You can send send multiple get() requests in a loop as follows:
for i in range(100):
driver.get(https://page-views.glitch.me/badge?page_id=page.id)
I'm trying to build a web server to collect "commands" via AJAX and then distribute the commands to clients via long-polling.
The goal is that someone POSTs some data to /add-command.
Another client implements a long-polling client hitting /poll waiting for a command to execute.
I think a queue is the right data structure to use to hold commands waiting for attention. I'd like the commands to essentially be distributed immediately to any long-polling client but held if no client is currently polling.
Here's my python script.
import os
import time
import tornado.httpserver
import tornado.ioloop
import tornado.web
import tornado.gen
import Queue
import multiprocessing.pool
import mysql.connector
import urlparse
import uuid
import json
_commandQueue = Queue.Queue()
_commandPollInterval = 0.2
_commandPollTimeout = 10
class HomeHandler(tornado.web.RequestHandler):
def get(self):
self.render("home.htm")
class AddCommandHandler(tornado.web.RequestHandler):
def post(self):
d = urlparse.parse_qs(self.request.body)
_commandQueue.put(d)
self.write(str(True))
class PollHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
self.write("start")
d = 1
d = yield self.getCommand()
self.write(str(d))
self.write("end")
self.finish()
#tornado.gen.coroutine
def getCommand(self):
start = time.time()
while (time.time() - start) < _commandPollTimeout * 1000:
if not _commandQueue.empty:
return _commandQueue.get()
else:
time.sleep(_commandPollInterval)
return None
def main():
application = tornado.web.Application(
[
(r"/", HomeHandler),
(r"/add-command", AddCommandHandler),
(r"/poll", PollHandler),
],
debug=True,
template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
)
tornado.httpserver.HTTPServer(application).listen(int(os.environ.get("PORT", 5000)))
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
The AddCommandHandler works fine to put items in the _commandQueue.
The PollHandler request just times out. If I call the PollHandler, it seems to lock the _commandQueue and I can't put or get from it.
I suspect I need to join the queue, but I can't seem to find the right time to do that in the code.
UPDATE -- Here's my final code thanks to the answers
import os
import time
import datetime
import tornado.httpserver
import tornado.ioloop
import tornado.web
import tornado.gen
import tornado.queues
import urlparse
import json
_commandQueue = tornado.queues.Queue()
_commandPollInterval = 0.2
_commandPollTimeout = 10
class HomeHandler(tornado.web.RequestHandler):
def get(self):
self.render("home.htm")
class AddCommandHandler(tornado.web.RequestHandler):
def get(self):
cmd = urlparse.parse_qs(self.request.body)
_commandQueue.put(cmd)
self.write(str(cmd))
def post(self):
cmd = urlparse.parse_qs(self.request.body)
_commandQueue.put(cmd)
self.write(str(cmd))
class PollHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
cmd = yield self.getCommand()
self.write(str(cmd))
#tornado.gen.coroutine
def getCommand(self):
try:
cmd = yield _commandQueue.get(
timeout=datetime.timedelta(seconds=_commandPollTimeout)
)
raise tornado.gen.Return(cmd)
except tornado.gen.TimeoutError:
raise tornado.gen.Return()
def main():
application = tornado.web.Application(
[
(r"/", HomeHandler),
(r"/add-command", AddCommandHandler),
(r"/poll", PollHandler),
],
debug=True,
template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"),
)
tornado.httpserver.HTTPServer(application).listen(int(os.environ.get("PORT", 5000)))
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
In async model you should omit blocking operation, time.sleep is evil in your code. Moreover, I think that the best way is to use tornado's (in async interface) queue - tornado.queue.Queue and use async get:
import datetime
import tornado.gen
import tornado.queues
_commandQueue = tornado.queues.Queue()
# ...rest of the code ...
#tornado.gen.coroutine
def getCommand(self):
try:
# wait for queue item if cannot obtain in timeout raise exception
cmd = yield _commandQueue.get(
timeout=datetime.timedelta(seconds=_commandPollTimeout)
)
return cmd
except tornado.gen.Timeout:
return None
Note: Module tornado.queues si available since Tornado 4.x, if you use older one, Toro will help.
You can NOT use sleep in listener, since it blocks reading from input stream. time.sleep(_commandPollInterval). What you should use is yield gen.sleep(_commandPollInterval)
I have 2 servers in python, I want to mix them up in one single .py and run together:
Server.py:
import logging, time, os, sys
from yowsup.layers import YowLayerEvent, YowParallelLayer
from yowsup.layers.auth import AuthError
from yowsup.layers.network import YowNetworkLayer
from yowsup.stacks.yowstack import YowStackBuilder
from layers.notifications.notification_layer import NotificationsLayer
from router import RouteLayer
class YowsupEchoStack(object):
def __init__(self, credentials):
"Creates the stacks of the Yowsup Server,"
self.credentials = credentials
stack_builder = YowStackBuilder().pushDefaultLayers(True)
stack_builder.push(YowParallelLayer([RouteLayer, NotificationsLayer]))
self.stack = stack_builder.build()
self.stack.setCredentials(credentials)
def start(self):
self.stack.broadcastEvent(YowLayerEvent(YowNetworkLayer.EVENT_STATE_CONNECT))
try:
logging.info("#" * 50)
logging.info("\tServer started. Phone number: %s" % self.credentials[0])
logging.info("#" * 50)
self.stack.loop(timeout=0.5, discrete=0.5)
except AuthError as e:
logging.exception("Authentication Error: %s" % e.message)
if "<xml-not-well-formed>" in str(e):
os.execl(sys.executable, sys.executable, *sys.argv)
except Exception as e:
logging.exception("Unexpected Exception: %s" % e.message)
if __name__ == "__main__":
import sys
import config
logging.basicConfig(stream=sys.stdout, level=config.logging_level, format=config.log_format)
server = YowsupEchoStack(config.auth)
while True:
# In case of disconnect, keeps connecting...
server.start()
logging.info("Restarting..")
App.py:
import web
urls = (
'/', 'index'
)
app = web.application(urls, globals())
class index:
def GET(self):
greeting = "Hello World"
return greeting
if __name__ == "__main__":
app.run()
I want to run both together from single .py file together.
If I try to run them from one file, either of the both starts and other one starts only when first one is done working.
How can I run 2 servers in python together?
import thread
def run_app1():
#something goes here
def run_app2():
#something goes here
if __name__=='__main__':
thread.start_new_thread(run_app1)
thread.start_new_thread(run_app2)
if you need to pass args to the functions you can do:
thread.start_new_thread(run_app1, (arg1,arg2,....))
if you want more control in your threads you could go:
import threading
def app1():
#something here
def app2():
#something here
if __name__=='__main__':
t1 = threading.Thread(target=app1)
t2 = threading.Thread(target=app2)
t1.start()
t2.start()
if you need to pass args you can go:
t1 = threading.Thread(target=app1, args=(arg1,arg2,arg3.....))
What's the differences between thread vs threading? Threading is higher level module than thread and in 3.x thread got renamed to _thread... more info here: http://docs.python.org/library/threading.html but that's for another question I guess.
So in your case, just make a function that runs the first script, and the second script, and just spawn threads to run them.
I'm trying to run a Tornado server with Couchbase 4.0 Developer preview.
import tornado.web
import tornado.httpserver
import tornado.options
import tornado.ioloop
import tornado.websocket
import tornado.httpclient
from tornado import gen
import os.path
from tornado.options import define, options, parse_command_line
import time
#from couchbase.bucket import Bucket
from twisted.internet import reactor
from txcouchbase.bucket import Bucket
from couchbase.n1ql import N1QLQuery, N1QLError
from pprint import pprint
server = "x.x.x.x"
bucketname = "zips"
Connection = "couchbase://" + server + "/" + bucketname
bkt = Bucket(Connection)
class IndexHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self):
print "entered"
query = "SELECT * FROM `zips` where pincode= '632014'"
q = N1QLQuery(query)
#self.bkt = bkt
t0 = time.time()
res = bkt.n1qlQueryAll(q)
res.addCallback(self.on_ok)
reactor.run()
t1 = time.time()
print t1-t0
self.write("Hello World")
def on_ok(self,response):
print "LOl"
for each in res:
print each
reactor.stop()
self.finish()
handlers = [
(r'/',IndexHandler),
]
if __name__ == "__main__":
parse_command_line()
# template path should be given here only unlike handlers
app = tornado.web.Application(handlers, template_path=os.path.join(os.path.dirname(__file__), "templates"),
static_path=os.path.join(os.path.dirname(__file__), "static"), cookie_secret="61oETzKXQAGaYdkL5gEmGeJJFuYh7EQnp2XdTP1o/Vo=", debug=True)
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(8888, address='0.0.0.0')
tornado.ioloop.IOLoop.instance().start()
After I run this, for some reason the callback function is never called. I could not find any proper documentation for this, and had to go through the source code to write this. I'm still confused as I'm new to asynchronous programming. Can someone please tell me where I'm going wrong and if there is a better way of doing this?
In asynchronous programming, you only want to start an event loop (like IOLoop.start() or reactor.run()) once, at the top of your program. You're calling IOLoop.start(), so instead of calling reactor.run() you want to tell Twisted to use the Tornado IOLoop as its reactor. Before the import of reactor, do
import tornado.platform.twisted
tornado.platform.twisted.install()
from twisted.internet import reactor
See http://www.tornadoweb.org/en/stable/twisted.html#twisted-on-tornado for more.
Once you've done this, you can call twisted libraries without having to start and stop the reactor.