Is there any tools that catches python logging (socket or http) handlers' reports, and serves a http service so that I can check my logs through a http page?
Thanks
Finally... I found a working server that will do the job...
LoggingWebMonitor
UPDATE
I found Sentry on github, It seems more sophiscated and production-ready.
There are many to tools to easily create simple RESTful HTTP webservices. My favorite is itty.
from itty import get, run_itty
import glob, gzip, json, os, functools
def jsonify(origfunc):
#functools.wraps(origfunc)
def wrapper(*args, **kwds):
result = origfunc(*args, **kwds)
return json.dumps(result, indent=4)
return wrapper
#get('/logs')
#jsonify
def list_logfiles(request):
return glob.glob('/var/log/myserver/*.gz')
#get('/logs/(?P<name>\w+)')
def show_logfile(request, name):
fullname = os.path.join('/var/log/myserver', name)
with gzip.open(fullname, 'rb') as f:
return f.read()
run_itty(host='localhost', port=8080)
i recommend you bootle.py is a nice framework for this cases.
Here the link to project website: http://bottlepy.org/docs/dev/
Try Splunk, simple to set up and has a nice interface. You'd listen to your logfiles or simply send logs to splunk. Even works remotly for logs in multiple servers. And you can do a lot more than just checking logs.
Related
I'm using the twittytwister module to implement a Twitter client but it looks like it's no longer being developed. I need to connect to an HTTP proxy to be able to reach the internet but the module does not have such an option so I'm looking at modifying it.
def __downloadPage(factory, *args, **kwargs):
downloader = factory(*args, **kwargs)
if downloader.scheme == 'https':
from twisted.internet import ssl
contextFactory = ssl.ClientContextFactory()
reactor.connectSSL(downloader.host, downloader.port,
downloader, contextFactory)
else:
reactor.connectTCP(downloader.host, downloader.port,
downloader)
return downloader
def getPage(url, *args, **kwargs):
return __downloadPage(client.HTTPClientFactory, url, *args, **kwargs)
What can I do here to make it connect with my proxy? Do I replace client.HTTPClientFactory with something else?
The newer (added in Twisted 9.0.0) HTTP client API, twisted.web.client.Agent, includes support for connecting to an HTTP proxy. For example, you could write:
from twisted.web.client import Agent, ProxyAgent
from twisted.internet.endpoints import clientFromString
from twisted.internet import reactor
from os import environ
try:
proxy = environ["HTTP_PROXY"]
except KeyError:
agent = Agent(reactor)
else:
agent = ProxyAgent(clientFromString(reactor, proxy))
See the endpoints documentation for details about the expected format of the HTTP_PROXY environment variable this example uses.
Unfortunately it looks like twittytwister uses the older HTTP client API so you'll need to port it to use Agent before you can benefit from ProxyAgent. Fortunately the getPage style API is fairly limited so it shouldn't be very hard to replace it with Agent-style code.
I am trying have a set of python scripts report their status to a set of micro controllers.
So my idea for this is to have the python scripts each create their own webpage that can be viewed by the micro controllers, but is there anyway to have the script itself keeping the page served, i.e. an apache library so that if the script crashes or is not running the page is not served or a way to make the page have a default value if the script is not running.
You can also have a look at twisted.web
A very basic example:
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet import reactor
class StatusPageResource(Resource):
isLeaf = True
def __init__(self, param1):
self.param1 = param1
# Call the constructor of the super class
Resource.__init__(self)
def render_GET(self, request):
return "<html><body>%s</body></html>" % self.param1
my_res = Resource()
my_res.putChild('GetStatusPage1', StatusPageResource(param1='abc'))
my_res.putChild('GetStatusPage2', StatusPageResource(param1='xyz'))
factory = Site(my_res)
reactor.listenTCP(8080, factory)
print 'Runnning on port 8080'
reactor.run()
Now point your browser to http://localhost:8080/GetStatusPage1 (for example)
You could use http://docs.python.org/library/simplehttpserver.html or some minimal http server framework like http://flask.pocoo.org/ or http://www.cherrypy.org/.
If you want to feed "live" information to your micro controllers also have a look at comet style long polling requests. You essentially keep downloading "the page" forever and analyse it as a data stream while the server keeps adding updated info at the "end of the page".
So i've looked around at a few things involving writting an HTTP Proxy using python and the Twisted framework.
Essentially, like some other questions, I'd like to be able to modify the data that will be sent back to the browser. That is, the browser requests a resource and the proxy will fetch it. Before the resource is returned to the browser, i'd like to be able to modify ANY (HTTP headers AND content) content.
This ( Need help writing a twisted proxy ) was what I initially found. I tried it out, but it didn't work for me. I also found this ( Python Twisted proxy - how to intercept packets ) which i thought would work, however I can only see the HTTP requests from the browser.
I am looking for any advice. Some thoughts I have are to use the ProxyClient and ProxyRequest classes and override the functions, but I read that the Proxy class itself is a combination of the both.
For those who may ask to see some code, it should be noted that I have worked with only the above two examples. Any help is great.
Thanks.
To create ProxyFactory that can modify server response headers, content you could override ProxyClient.handle*() methods:
from twisted.python import log
from twisted.web import http, proxy
class ProxyClient(proxy.ProxyClient):
"""Mangle returned header, content here.
Use `self.father` methods to modify request directly.
"""
def handleHeader(self, key, value):
# change response header here
log.msg("Header: %s: %s" % (key, value))
proxy.ProxyClient.handleHeader(self, key, value)
def handleResponsePart(self, buffer):
# change response part here
log.msg("Content: %s" % (buffer[:50],))
# make all content upper case
proxy.ProxyClient.handleResponsePart(self, buffer.upper())
class ProxyClientFactory(proxy.ProxyClientFactory):
protocol = ProxyClient
class ProxyRequest(proxy.ProxyRequest):
protocols = dict(http=ProxyClientFactory)
class Proxy(proxy.Proxy):
requestFactory = ProxyRequest
class ProxyFactory(http.HTTPFactory):
protocol = Proxy
I've got this solution by looking at the source of twisted.web.proxy. I don't know how idiomatic it is.
To run it as a script or via twistd, add at the end:
portstr = "tcp:8080:interface=localhost" # serve on localhost:8080
if __name__ == '__main__': # $ python proxy_modify_request.py
import sys
from twisted.internet import endpoints, reactor
def shutdown(reason, reactor, stopping=[]):
"""Stop the reactor."""
if stopping: return
stopping.append(True)
if reason:
log.msg(reason.value)
reactor.callWhenRunning(reactor.stop)
log.startLogging(sys.stdout)
endpoint = endpoints.serverFromString(reactor, portstr)
d = endpoint.listen(ProxyFactory())
d.addErrback(shutdown, reactor)
reactor.run()
else: # $ twistd -ny proxy_modify_request.py
from twisted.application import service, strports
application = service.Application("proxy_modify_request")
strports.service(portstr, ProxyFactory()).setServiceParent(application)
Usage
$ twistd -ny proxy_modify_request.py
In another terminal:
$ curl -x localhost:8080 http://example.com
For two-way proxy using twisted see the article:
http://sujitpal.blogspot.com/2010/03/http-debug-proxy-with-twisted.html
Can anyone recommend some simple code to set up a simple JSON RPC client and server using twisted?
I found txJSON-RPC, but I was wondering if someone had some experience using some of these anc could recommend something.
txJSONRPC is great. I use it and it works. I suggest you give it a try.
SERVER:
from txjsonrpc.web import jsonrpc
from twisted.web import server
from twisted.internet import reactor
class Math(jsonrpc.JSONRPC):
"""
An example object to be published.
"""
def jsonrpc_add(self, a, b):
"""
Return sum of arguments.
"""
return a + b
reactor.listenTCP(7080, server.Site(Math()))
reactor.run()
CLIENT:
from twisted.internet import reactor
from txjsonrpc.web.jsonrpc import Proxy
def printValue(value):
print "Result: %s" % str(value)
def printError(error):
print 'error', error
def shutDown(data):
print "Shutting down reactor..."
reactor.stop()
proxy = Proxy('http://127.0.0.1:7080/')
d = proxy.callRemote('add', 3, 5)
d.addCallback(printValue).addErrback(printError).addBoth(shutDown)
reactor.run()
As a bonus, I will leave some alternative: amp.
http://amp-protocol.net
If you are looking for a framework-independent approach, this lib I pushed (using mixin) might be helpful:
Cyclone, a Tornado async web server implementation written using twisted, has a built-in json-rpc request handler that uses the python json/simplejson module. Example server and client code is here.
wikipedia has a bunch of implementations listed for python: https://en.wikipedia.org/wiki/JSON-RPC#Implementations
That said, txjason feels like the one best integrated with twisted. It seems to support out of order responses out of the box for example. Most of it would be portable to python3 using six. The most horrible part is the parameter validation, which is not exposed in the normal public API anyway.
For me this worked better then "libraries" , speaking of client.
TESTDATA = {'id': 1234,
'method': 'getbalance',
}
URL = 'http://localhost:7777'
d= getPage(URL,method="POST",postdata=json.dumps(TESTDATA))
d.addBoth(lambda x :print(json.loads(x)))
I am working my way through learning Twisted, and have stumbled across something I'm not sure I'm terribly fond of - the "Twisted Command Prompt". I am fiddling around with Twisted on my Windows machine, and tried running the "Chat" example:
from twisted.protocols import basic
class MyChat(basic.LineReceiver):
def connectionMade(self):
print "Got new client!"
self.factory.clients.append(self)
def connectionLost(self, reason):
print "Lost a client!"
self.factory.clients.remove(self)
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.message(line)
def message(self, message):
self.transport.write(message + '\n')
from twisted.internet import protocol
from twisted.application import service, internet
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
application = service.Application("chatserver")
internet.TCPServer(1025, factory).setServiceParent(application)
However, to run this application as a Twisted server, I have to run it via the "Twisted Command Prompt", with the command:
twistd -y chatserver.py
Is there any way to change the code (set Twisted configuration settings, etc) so that I can simply run it via:
python chatserver.py
I've Googled, but the search terms seem to be too vague to return any meaningful responses.
Thanks.
I don't know if it's the best way to do this but what I do is instead of:
application = service.Application("chatserver")
internet.TCPServer(1025, factory).setServiceParent(application)
you can do:
from twisted.internet import reactor
reactor.listenTCP(1025, factory)
reactor.run()
Sumarized if you want to have the two options (twistd and python):
if __name__ == '__main__':
from twisted.internet import reactor
reactor.listenTCP(1025, factory)
reactor.run()
else:
application = service.Application("chatserver")
internet.TCPServer(1025, factory).setServiceParent(application)
Hope it helps!
Don't confuse "Twisted" with "twistd". When you use "twistd", you are running the program with Python. "twistd" is a Python program that, among other things, can load an application from a .tac file (as you're doing here).
The "Twisted Command Prompt" is a Twisted installer-provided convenience to help out people on Windows. All it is doing is setting %PATH% to include the directory containing the "twistd" program. You could run twistd from a normal command prompt if you set your %PATH% properly or invoke it with the full path.
If you're not satisfied with this, perhaps you can expand your question to include a description of the problems you're having when using "twistd".
On windows you can create .bat file with your command in it, use full paths, then just click on it to start up.
For example I use:
runfileserver.bat:
C:\program_files\python26\Scripts\twistd.py -y C:\source\python\twisted\fileserver.tac
Maybe one of run or runApp in twisted.scripts.twistd modules will work for you. Please let me know if it does, it will be nice to know!
I haven't used twisted myself. However, you may try seeing if the twistd is a python file itself. I would take a guess that it is simply managing loading the appropriate twisted libraries from the correct path.
I am successfully using the simple Twisted Web server on Windows for Flask web sites.
Are others also successfully using Twisted on Windows, to validate that configuration?
new_app.py
if __name__ == "__main__":
reactor_args = {}
def run_twisted_wsgi():
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
resource = WSGIResource(reactor, reactor.getThreadPool(), app)
site = Site(resource)
reactor.listenTCP(5000, site)
reactor.run(**reactor_args)
if app.debug:
# Disable twisted signal handlers in development only.
reactor_args['installSignalHandlers'] = 0
# Turn on auto reload.
import werkzeug.serving
run_twisted_wsgi = werkzeug.serving.run_with_reloader(run_twisted_wsgi)
run_twisted_wsgi()
old_app.py
if __name__ == "__main__":
app.run()