I have two PCs and I want to monitor the Internet connectivity in both of them and make it available in a page as to whether they're currently online and running. How can I do that?
I'm thinking of a cron job that gets executed every minute that sends a POST to a file located in a server, which in turn would write the connectivity status "online" to a file. In the page where the statuses are displayed, read from both the status files and display whether they're online or not. But this feels like a sloppy idea. What alternative suggestion do you have?
(The answer doesn't necessarily have to be code; I'm not looking for copy-paste solutions. I just want an idea, a nudge in the right directio,)
I would suggest just a GET request (you just need a ping to indicate that the PC is on) sent periodically to maybe a Django server and if you query a page on the Django server, it shows a webpage indicating the status of each.
In the Django server have a loop where the time each GET is received is indicated, if the time between the last GET and current time is too large, set a flag to false.
That flag will later be visible when the URL is queried, via the views.
I don't think this would end up sloppy, just a trivial solution where you don't really have to dig too deep to make it work.
I have used Nagios in the past I like it a lot. It is free and open source. I have used it to monitor several Web, DNS, Mail servers and a proxy. You can check it here: https://www.nagios.com/products/nagioscore
Related
I have a python code that performs some fairly intense computations, and then generates a plot (png file) for display on the web. I'm using python, flask, mod_wsgi, and apache. Since the computation takes several seconds (around 10 seconds), I'd like to display a "LOADING" type image while the computation is happening so that the viewer doesn't think the server is messed up, and then the actual image when computations are complete. How can I do this from the server side (not using javascript in the web browser)? In the past I remember seeing a lot of web pages where it seemed like the server was pushing a new page to the browser (from what a recall most it was search engines on message forums). The answer to this question I believe is really an http related question, so it doesn't necessarily have to be specific to serving an image (it could be an html page), or using python, flask, mod_wsgi, or apache, but it would be great if an example could be giving for just that configuration.
Before Javascript I did implement this by generating a page that had a refresh in the HTML header, with a delay of 2-3 seconds.
This page would redisplay itself until the code generating that page noticed that the 'result' was finished then generating different HTML code (without the refresh).
<HEAD>
<META http-equiv="refresh" content="3">
</HEAD>
I'm aware that this question is a bit old now, but I feel like there is not enough information available on this topic. I was struggling with this myself to find a relevant information, so I hope this will help:
Suggested solution
There are different technologies that can be used here, but I believe the simplest would be Server Sent Events. The implementation for Flask can be found here. The last part of the documentation is really important:
Subscribers will connect and block for a long time, so you should seriously consider running under an asynchronous WSGI server, such as gunicorn+gevent
So make sure fulfil the requirement. Also, it's pretty important to understand that this approach is good if you want to send the messages from your server to the client. In case you have an external worker that does the calculations for you this method will only make it more complicated for you, since your server will have to play the role of a middle man between the browser and the worker machine. On some hostings it may even not work as expected (e.g. Heroku - still not sure why it misbehaves, looks like too many updates from the worker and are not propagated properly to the client). In case you use the same host for your app and the workers, you should have no problem though.
Alternate solution
In my opinion this type of calculations belong to background, so this solution assumes that we have some kind of workers doing the job for you (like I had when I first encountered the problem). Note that this solution is not a server->client communication, but it's based on polling. I think this may be the only option if you don't run on the asynchronous server in production.
So let's assume you have a worker which status you can check, for example Iron Worker. The user visits your page and this invokes the calculations on the worker. From this point on you should use AJAX calls to get the status update directly from your worker. What I did in my app, I used jQuery to poll the worker web api and learn about it's status. After you discover that your worker is done, you can just reload the page or just the image or whatever else you need.
Additional information
If you need to update many places at the same time (not only the browser), you can use queue services, for example ironMQ, which allows you to propagate your messages to a special queue, and then subscribe to this queue with a client and receive the messages from it. This is what I did before I discovered I can query the worker directly for it's status.
I think I don't completely understand the deployment process. Here is what I know:
when we need to do hot deployment -- meaning that we need to change the code that is live -- we can do it by reloading the modules, but
imp.reload is a bad idea, and we should restart the application instead of reloading the changed modules
ideally the running code should be a clone of your code repository, and any time you need to deploy, you just pull the changes
Now, let's say I have multiple instances of wsgi app running behind a reverse proxy like nginx (on ports like 8011, 8012, ...). And, let's also assume that I get 5 requests per second.
Now in this case, how should I update my code in all the running instances of the application.
If I stop all the instances, then update all of them, then restart them all -- I will certainly lose some requests
If I update each instance one by one -- then the instances will be in inconsistent states (some will be running old code, and some new) until all of them are updated. Now if a request hits an updated instance, and then a subsequent (and related) request hits an older instance (yet to be updated) -- then I will get wrong results.
Can somebody explain thoroughly how busy applications like this are hot-deployed?
For deployment across several hot instances that are behind a load balancer like nginx I like to do rolling deployments with a tool like Fabric.
Fabric connects you to Server 1
Shut down the web-server
Deploy changes, either by using your VCS or transferring tarball with the new application
Start up the web-server
GOTO1 and connect to the next server.
That way you're never offline, and it's seamless as nginx knows when a webserver is taken down when it tries to round-robin to it and will move onto the next one instead, and as soon as the node/instance is back up it will be back into production usage.
EDIT:
You can use the ip_hash module in nginx to ensure all requests from one IP Address goes to the same server for the length of the session
This directive causes requests to be distributed between upstreams based on the IP-address of the client.
The key for the hash is the class-C network address of the client. This method guarantees that the client request will always be transferred to the same server. But if this server is considered inoperative, then the request of this client will be transferred to another server. This gives a high probability clients will always connect to the same server.
What this means to you, is that once your web-server is updated and a client has connected to the new instance, all connections for that session will continue to be forwarded to the same server.
This does leave you in the situation of
Client connects to site, gets served from Server 1
Server 1 is updated before client finishes whatever they're doing
Client potentially left in a state of limbo?
This scenario begs the question, are you removing things from your API/Site which could potentially leave the client in a state of limbo ? If all you're doing is for example updating UI elements or adding pages etc but not changing any back-end APIs you should not have any problems. If you are removing API functions, you might end up with issues.
Couldn't you take half your servers offline (say by pulling them out of the load balancing pool) and then update those. Then bring them back online while simultaneously pulling down the other half. Then update those and bring them back online.
This will ensure that you stay online while also ensuring that you never have the old and new versions of your application online at the same time. Yes, this will mean that your site would run at half its capacity during the time. But that might be ok?
I am working on a project for work that requires me to pull information from a logfile and send a notification anytime it finds a the specific information. For example the exact issue I am working on is I am needing to create a python script that will look into may /var/log/auth.log (FreeBSD system) and pull any invalid SSH login attempts, then proceed to email me and another co-worker anytime there is an offense.
I've been looking all over for a few days now and have had minimal success any help would be greatly appreciated.
I think what you're really after is a daemon like fail2ban, which is specifically designed to examine log files for intrusion attempts.
From the fail2ban wiki:
Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs
that show the malicious signs -- too many password failures, seeking
for exploits, etc. Generally Fail2Ban then used to update firewall
rules to reject the IP addresses for a specified amount of time,
although any arbitrary other action (e.g. sending an email, or
ejecting CD-ROM tray) could also be configured. Out of the box
Fail2Ban comes with filters for various services (apache, curier, ssh,
etc).
This would probably work better than any solution you baked yourself.
That said, if you did want to roll your own, the naive way to implement periodic checking of a file is simply to read it every five minutes and see if it's changed.
The smarter way is to use the operating system's file monitoring service, which hooks into the filesystem driver and notifies you as soon as the file changes. This has the dual benefits that your code will take less CPU time, and it will respond immediately whenever the file changes.
On Linux the service is called inotify. BSD and Windows have an equivalent feature.
You could run a cron job every few minutes that checks for changes in that file. If there are any changes, it will email you, by using, for example, smtplib. Here is an example of smtplib usage with sendgrid: http://docs.sendgrid.com/documentation/get-started/integrate/examples/python-email-example-using-smtp/
How do you find out if a file was modified?
You keep a copy of the file as it looked in the previous script run, and compare that to the current contents
You check the file's last modification time.
This is just a general idea that can be tweaked, and all the 'ingredients' can be found on google, so you should be able to implement it by googling yourself.
Hope this helps.
As a rough idea for a cron job:
with open('/var/log/auth.log') as auth:
for line in auth:
if 'blahblah' in line:
# send email
You'll want to check out the email module for emailing details. You'll also want a way to keep track of what's already been scanned, so you don't end up sending duplicate emails.
I hope you guys can spare a moment with some ideas on how to develop my idea.
I have an Asterisk-based telephone switch . When an incoming call is arriving, I can make sure the server runs an external script of any language.
Here comes my development work. I would like to notify a group of listening clients about the call, and probably open a browser page on their computer.
What kind of approach would you take for this sort of server-based push notification? (with no iPhone involved)
I am open to any language.
Thanks
You can use or build off a tool like callPopPy, which interacts with Asterisk's existing notification subsystem. More options are listed here.
Maybe have a look at www.pubnub.com .. its commercial, but lets you send 5 million messages a month for free. Essentially it lets you create a named channel, and have X number of clients connect to it and send messages back and forth.
Using one of these services would of course require you to write a client to distribute to your users (in your language of choice) and tie's you in somewhat (shouldn't really be a problem as you could swap in some other solution later if they go under or whatever.)
The upside would be, very good x-platform support and a very clean API, infrastructure taken care of for you (for example clients can still connect the the channel even if your asterisk box is down or whatever)
(and no, I don't work for pubnub! but it seems like a no-brainer to use it with the 5mil messages free deal!)
I'm writing an application that streams the output (by this I mean both sys.stdout and sys.stderr) of a python script excited on the server, in real time to the browser.
The users on the site will be allowed to select the script to run, excite and kill their chosen script, and change some parameters, so I will need a different thread per user on the site (user A can start, stop and change a script, whilst user B can do the same with a different script).
I know I need to use comet for the web clients, and seeing as the rest of the project is written in python, I'd like to use twisted for the server, however I'm not really sure of what I need to do next!
There are a daunting number of options (Divmod Mantissa, Divmod Nevow, twisted.web, STOMP, etc), and some are better documented that others, making the whole thing rather tricky!
I have a working demo using stompservice on orbited, using Orbited.TCPSocket for the javascript side of things, however I'm starting to think that STOMPs channel model isn't going to work for multithreading, multi-running scripts (unless I open a new channel per run, but that seems like the wrong use of the channel model).
Can anyone point me in the right direction, or some sample code I can learn from?
Thanks!
Nevow Athena is a framework specifically for AJAX and COMET applications and in theory is exactly the sort of thing you are looking for.
However, I am not sure that it is well used or supported at this time - looking at mailing list traffic and google search results suggests that it may not be.
There are a couple of tutorials you could look at to help you decide on it:
one on the 'official' site: http://divmod.org/trac/wiki/DivmodNevow/Athena/Tutorials/LiveElement
and one other that I found:
http://divmodsphinx.funsize.net/nevow/chattutorial/part01/index.html
The code for the latter seems to be included in the Nevow distribution when you download it under /doc/listings/partxx (I think...)
You can implement a very simple "HTTP streaming" by keeping the http connection open and appending javascript chunks that update the dom contents. This works since the browser evaluates the "script" chunks as they arrive.
I wrote a blog entry a while ago with a running example using twisted and very few lines of javascript: Simple HTTP streaming with Twisted & Javascript
You can easily mix this pattern with a publisher/subscriber pattern to make it multiuser, etc. I use this pattern to watch live log streams via web.
An example of serving for long-polling clients with Twisted is slosh. This might not be what you want, but because it's not a large framework, it can help you figure out how to use Twisted.