So I have been getting my feet wet with python, attempting to build a reminder system that ties into the gnome notification ui. The basic idea is you type a command into your shell like remind me to check on dinner in 20 min and then in 20 min you get a desktop notification saying "check on dinner". The way I am doing this is by having a script parse the message and write the time the notification should be sent and the message that should be sent to a log file.
The notifications are getting triggered by a python daemon. I am using this daemon design I found online. The issue I am seeing is when this daemon is running it is taking 100% of my cpu! I stripped down all the code the daemon was doing and it I still have this problem when all the daemon is doing is
while True:
last_modified = os.path.getmtime(self.logfile)
I presume that this is a bad approach and I should instead be notifying the daemon when there is a new reminder and then most of the time the reminder daemon should be sleeping. Now this is just an idea but I am having a hard time finding resources on 'how to notify a process' when all I know is the daemons pid. So if I have suspend the daemon with something like time.sleep(time_to_next_notification) would there be a way for me to send a signal to to the daemon letting it know that there was a new reminder?
Though I believe you're better off using a server - client type solution that listens on a port, what you are asking is 100% possible using the signal and os libraries. This approach will not work well with multi threaded programs however as signals are only handled by the parent thread in python. Additionally windows doesn't implement signals in the same way so the options are more limited.
Signals
The "client" process can send arbitrary signals using os.kill(pid, signal). You will have to go through the available signals and determine which one you want to use (signal.NSIG may be a good option because it shouldn't stomp on any other default behavior).
The "daemon" process on startup must register a handler for what to do when it receives your chosen signal. The handler is a function you must define that receives the signal itself that was received as well as the current stack frame of execuiton (def handler(signum, frame):). If you're only doing one thing with this handler, and it doesn't need to know what was happening when it was called, you can probably ignore both these parameters. Then you must register the handler with signal.signal ex: signal.signal(signal.NSIG, handler).
From there you will want to find some appropriate way to wait until the next signal without consuming too many resources. This could be as simple as looping on a os.sleep
command, or you could try to get fancy. I'm not sure 100% how execution resumes on returning from a signal handler, so you may need to concern yourself with recursion depth (ie, make sure you don't recurse every time a signal is handled or you'll only ever be able to handle a limited number of signals before needing to re-start).
Server
Having a process listen on a port (generally referred to as a server, but functionally the same as your 'daemon' description) instead of listen for operating system signals has several main advantages.
Ports are able to send data where signals are only able to trigger events
Ports are more similar cross-platform
Ports play nice[r] with multi-threading
Ports make it easy to send messages across a network (ie: create reminder from phone and execute on PC)
Waiting for multiple things at once
In order to address the need to wait for multiple processes at once (listening for input as well as waiting to deliver next notification) you have quite a few options:
Signals actually may be a good use case as signal.SIGALRM can be used as a conveniently re-settable alarm clock (if you're using UNIX). You would set up the handler in the same way as before, and simply set an alarm for the next notification. After setting the alarm, you could simply resume listening on the port for new tasks. If a new task comes in, setting the alarm again will override the existing one, so the handler would need to retrieve the next queued notification and re-set the alarm once done with the first task.
Threads could either be used to poll a queue of notification tasks, or an individual thread could be created to wait for each task. This is not a particularly elegant solution, however it would be effective and easy to implement.
The most elegant solution would likely be to use asyncio co-routines, however I am not as well versed in asyncio, and will admit they're a bit more confusing than threads.
Related
In an attempt to make my terminal based program survive longer I was told to look into forking the process off of system. I can't find much specifying a PID to which I want to spawn a new process off of.
is this possible in Linux? I am a Windows guy mainly.
My program is going to be dealing with sockets and if my application crashed then I would lose lots of information. I was under the impression that if it was forked from system the sockets would stay alive?
EDIT: Here is what I am trying to do. I have multiple computers that I want to communicate with. So I am building a program that lets me listen on a socket(simple). Then I will connect to it from each of my remote computers(simple).
Once I have a connection I want to open a new terminal, and use my program to start interacting with the remote computer(simple).
The questions came from this portion.. The client shell will send all traffic to the main shell who will then send it out to the remote computer. When a response is received it goes to main shell and forwards it to client shell.
The issue is keeping each client shell in the loop. I want all client shells to know who is connected to who on each client shell. So client shell 1 should tell me if I have a client shell 2, 3, 4, 5, etc and who is connected to it. This jumped into sharing resources between different processes. So I was thinking about using local sockets to send data between all these client shells. But then I ran into a problem if the main shell were to die, everything is lost. So I wanted a way to try and secure it.
If that makes sense.
So, you want to be able to reload a program without losing your open socket connections?
The first thing to understand is that when a process exits, all open file descriptors are closed. This includes socket connections. Running as a daemon does not change that. A process becomes a daemon by becoming independent of your terminal sesssion, so that it will continue to run when your terminal sesssion ends. But, like any other process, when a daemon terminates for any reason (normal exit, crashed, killed, machine is restarted, etc), then all connections to it cease to exist. BTW this is not specific to unix, Windows is the same.
So, the short answer to your question is NO, there's no way to tell unix/linux to not close your sockets when your process stops, it will close them and that's that.
The long answer is, there are a few ways to re-engineer things to get around this:
1) You can have your program exec() itself when you send it a special message or signal (eg SIGHUP). In unix, exec (or its several variants), does not end or start any process, it simply loads code into the current process and starts execution. The new code takes the place of the old within the same process. Since the process remains the same, any open files remain open. However you will lose any data that you had in memory, so the sockets will be open, but your program will know nothing about them. On startup you'd have to use various system calls to discover which descriptors are open in your process and whether any of them are socket connections to clients. One way to get around this would be to pass critical information as command line arguments or environment variables which can be passed through the exec() call and thus preserved for use of the new code when it starts executing.
Keep in mind that this only works when the process calls exec ITSELF while it is still running. So you cannot recover from a crash or any other cause of your process ending.. your connections will be gone. But this method does solve the problem of you wanting to load new code without losing your connections.
2) You can bypass the issue by dividing your server (master) into two processes. The first (call it the "proxy") accepts the TCP connections from the clients and keeps them open. The proxy can never exit, so it should be kept so simple that you'll rarely want to change that code. The second process runs the "worker", which is the code that implements your application logic. All the code you might want to change often should go in the worker. Now all you need do establish interprocess communication from the proxy to the worker, and make sure that if the worker exits, there's enough information in the proxy to re-establish your application state when the worker starts up again. In a really simple, low volume application, the mechanism can be as simple as the proxy doing a fork() + exec() of the worker each time it needs to do something. A fancier way to do this, which I have used with good results, is a unix domain datagram (SOCK_DGRAM) socket. The proxy receives messages from the clients, forwards them to the worker through the datagram socket, the worker does the work, and responds with the result back to the proxy, which in turn forwards it back to the client. This works well because as long as the proxy is running and has opened the unix domain socket, the worker can restart at will. Shared memory can also work as a way to communicate between proxy and worker.
3) You can use the unix domain socket along with the sendmesg() and recvmsg() functions along with the SCM_RIGHTS flag to pass not the client data itself, but to actually send the open socket file descriptors from the old instance to the new. This is the only way to pass open file descriptors between unrelated processes. Using this mechanism, there are all sorts of strategies you can implement.. for example, you could start a new instance of your master program, and have it connect (via a unix domain socket) to the old instance and transfer all the sockets over. Then your old instance can exit. Or, you can use the proxy/worker model, but instead of passing messages through the proxy, you can just have the proxy hand the socket descriptor to the worker via the unix domain socket between them, and then the worker can talk directly to the client using that descriptor. Or, you could have your master send all its socket file descriptors to another "stash" process that holds on to them in case the master needs to restart. There are all sorts of architectures possible. Keeping in mind that the operating system just provides the ability to ship the descriptors around, all the other logic you have to code for yourself.
4) You can accept that no matter how careful you are, inevitably connections will be lost. Networks are unreliable, programs crash sometimes, machines are restarted. So rather than going to significant effort to make sure your connections don't close, you can instead engineer your system to recover when they inevitably do.
The simplest approach to this would be: Since your clients know who they wish to connect to, you could have your client processes run a loop where, if the connection to the master is lost for any reason, they periodically try to reconnect (let's say every 10-30 seconds), until they succeed. So all the master has to do is to open up the rendezvous (listening) socket and wait, and the connections will be re-established from every client that is still out there running. The client then has to re-send any information it has which is necessary to re-establish proper state in the master.
The list of connected computers can be kept in the memory of the master, there is no reason to write it to disk or anywhere else, since when the master exits (for any reason), those connections don't exist anymore. Any client can then connect to your server (master) process and ask it for a list of clients that are connected.
Personally, I would take this last approach. Since it seems that in your system, the connections themselves are much more valuable than the state of the master, being able to recover them in the event of a loss would be the first priority.
In any case, since it seems that the role of the master is to simply pass data back and forth among clients, this would be a good application of "asynchronous" socket I/O using the select() or poll() functions, this allows you to communicate between multiple sockets in one process without blocking. Here's a good example of a poll() based server that accepts multiple connections:
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_71/rzab6/poll.htm
As far as running your process "off System".. in Unix/Linux this is referred to running as a daemon. In *ix, these processes are children of process id 1, the init process.. which is the first process that starts when the system starts. You can't tell your process to become a child of init, this happens automatically when the existing parent exits. All "orphaned" processes are adopted by init. Since there are many easily found examples of writing a unix daemon (at this point the code you need to write to do this has become pretty standardized), I won't paste any code here, but here's one good example I found: http://web.archive.org/web/20060603181849/http://www.linuxprofilm.com/articles/linux-daemon-howto.html#ss4.1
If your linux distribution uses systemd (a recent replacement for init in some distributions), then you can do it as a systemd service, which is systemd's idea of a daemon but they do some of the work for you (for better or for worse.. there's a lot of complaints about systemd.. wars have been fought just about)...
Forking from your own program, is one approach - however a much simpler and easier one is to create a service. A service is a little wrapper around your program that deals with keeping it running, restarting it if it fails and providing ways to start and stop it.
This link shows you how to write a service. Although its specifically for a web server application, the same logic can be applied to anything.
https://medium.com/#benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6
Then to start the program you would write:
sudo systemctl start my_service_name
To stop it:
sudo systemctl stop my_service_name
To view its outputs:
sudo journalctl -u my_service_name
In my (python) code I have a thread listening for changes from a couchdb feed (continuous changes). The changes request has a timeout parameter which is too big in certain circumstances (for example when a user wants to interrupt the program manually with ^C).
How can I abort a long-running blocking http request?
Is this possible, or do I need to reduce the timeout to make my program more responsive?
This would be unfortunate, because having a timeout small enough to make the program really responsive (say, 1s), means that there are lots of connections being created (one per second!), which defeats the purpose of listening to changes, and makes it very difficult to make sure that we are not missing any changes (in the re-connecting timespan we can indeed miss changes, so that special code is needed to handle that case)
The other option is to forcefully abort the thread, but that is not really an option in python.
If I understand correctly it looks like you are waiting too long between requests before deciding whether to respond to the users or not. You are right continuously closing and creating new connections will defeat the purpose of changes feed.
A solution could be to use heartbeat query parameter in which couchdb will keep sending newlines to tell the client that the connection is still alive.
http://localhost:5984/hello/_changes?feed=continuous&heartbeat=1000&include_docs=true
as long as you are getting heartbeats (newlines) you can be sure that you are getting new changes. A new line will indicate that no changes have occurred. Where as an actual change will be reported back. No need to close the connection. Respond to your clients if resp!="/n"
Blocking the thread execution in general prevents the thread from beeing terminated. You need to wait until the request timed out. But this is already clear.
Using a library that supports non blocking requests is maybe a solution, but I don't know if there is any.
Anyway ... you've mentioned that reducing the timeout will lead to more connections. I'd suggest to implement a waiting loop between requests that can be interrupted by an external signal to terminate the thread. with this loop you can control the number of requests independent from the timeout.
I was wondering what the best practice solution would be to constantly monitor and resart processes, because there are multiple ways in doing it.
Additional info:
I have a unix program which uses multiple processes to work. There's a main process, it always starts first and is not likely to die or terminate without stopping the program.
Then I spawn multiple "module" processes, which take care of some work and communicate through the main process. Those modules sometimes die because of exceptions, and because it's an external program I can't resolve the issues, so I have to restart them if they die.
I've made a program to check if any of the modules died and restart them, but I need to run it manually. My program checks if the pid files of the modules exist and if they listen on a specific tcp port. If the pid file doesn't exist or the socket can't establish connection, it restarts the module.
My thoughts so far:
Cron job to run the checks every minute and restart any dead modules. (kind of an overkill, because they don't die that frequently)
Daemon running in the background, which starts the modules and receives notifications if they die, so it doesn't have to check them constantly. (SIGCHLD signal, os.wait)
If I use the daemon method, how should I communicate with the daemon through my interface? (socket, or maybe a file which gets read if the daemon receives a specific signal)
Usually I would just go with the daemon because it seems to be the best practice method to restart the modules asap(cron only runs once a minute), but I've wanted to get some opinions from more experienced users. (I've never done something like this before, and asking doesn't hurt anyone :D)
I apologize if these questions are answered somewhere else, but I couldn't find any related question.
P.S. If I forgot something or you need more infos, please feel free to ask. :)
I would investigate running the monitoring process as part of a dedicated monitoring framework. Monit is one example, however there are of course others.
This has the advantage of providing additional features which might be useful, such as email alerts and analytics. In my experience, you should be able to use your existing program without too much modification, and Monit itself uses few system resources if that is a concern.
I have a python script that can run for long time in the background, and am trying to find a way of getting a status update from it. Basically we're considering to send it a SIGUSR1 signal, and then have it report back a status update.
Catching the signal in Python is not the issue, lots of information about that.
But how to get back information to the process initiating the signal? It seems that there is no way to figure out the pid of the initiating process by the receiving process, which could provide a way to send information back. A single reply message is enough here (in the tune of 'busy uploading; at 55% now; will finish at such a time'); a continuing update would be fantastic but not necessary.
What I've come up with is to write this data to a temporary file with predetermined name - has the issue of leaving stale files behind, and need some kind of clean-up routine then. But that sounds like a hack. Is there anything better available?
The way the running process is signalled doesn't matter, it doesn't have to be kill -SIGUSR1 pid. Any way to communicate with it would do. As long as the communication can be initiated from a new process that's started after the main process runs, possibly running under as different user.
Signals are not designed to be general inter-process communication mechanisms that allow for passing data. They can't do much more than provide a notification. What the target process does in response can be fairly general (generating output to a particular file that the sender then knows to go look at, for example), but passing data directly back to the sender would require a different mechanism like a pipe, shared memory, message queue, etc. Also note that, in general, a process receiving a signal can't really determine who sent the signal, so it wouldn't know where to send a response anyway.
I am learning Twisted, especially its XMPP side. I am writing a Jabber client which must send and recieve messages. Here is my code: http://pastebin.com/m71225776
As I understood the workflow is like this:
1. I create handlers for important network events (i.e. connecting, message recieving, disconnecting, etc)
2. I run reactor. At this moment starts the loop which is waiting for any event. When event happens it is passed to specified handler.
The problem is in sending messages. Sending is not associated with any network event so I can't create handler for it. Also I can't do anything with reactor until its loop stops working. But the goal is "To send messages when I need and recieve data when it comes".
I think am not fully understand philosophy of twisted, so give me a right way please.
You just need to find what events will trigger sending a message.
For example, in a GUI client, sending happens when the user types something. You should integrate with a graphics toolkit, using the Twisted reactor for its mainloop (there's a Gtk+ Twisted reactor for example). Then you'll be able to listen for some interface events, like the user typing enter in a text area; and you'll be able to react to that event by sending a message.
Other sources of events could be Twisted timers, any kind of protocol, including IPC, webhooks…
Incidentally, if you need a higher-level library for XMPP with Twisted, have a look at Wokkel.
More accurately, you can't do anything with the reactor until it calls one of your callbacks. You don't call twisted, twisted calls you.
One way to experiment is to have one of your setup handlers which you know will be called (or just test code put in after you start the reactor) call callLater() or loopingCall().