We Keep Coding

How to do I write signal-safe Python code?

I'll start with an example code, that "should never" fail:
counter1 = 0
counter2 = 0
def increment():
global counter1, counter2
counter1 += 1
counter2 += 1
while True:
try:
increment()
except:
pass
assert counter1 == counter2
The counters represent an internal structure that should keep its integrity no matter what. By a quick look, the assertion will never be False, and the structure will be intact.
One small signal however, that occurs in the middle of the function (for example SIGINT or KeyboardInterrupt in Python) causes the internal structure to break. In real-world scenario it may cause memory corruption, deadlocks, and all other types of mayhem.
Is there any way to make this function signal safe? If any arbitrary code can run anywhere and cause it to malfunction, how can we program library code that is safe and secure?
Even if we attempt to guard it using try...finally..., we might still receive a signal at the finally and prevent it from running.
While the example is in Python, I believe this question applies to all programming languages as a whole.
EDIT:
Do keep in mind the counters are just an example. In my actual use case it's different Locks and threading.Events. There's no real way (that I know of) to make the operation atomic.

One solution to your example is to update the data atomically, if possible:
counters = [0, 0]
def increment():
global counters
# "complex" preparation. not atomic at all
next0 = counters[0] + 1
next1 = counters[1] + 1
temp = [next0, next1]
# actual atomic update
counters = temp
try:
while True:
increment()
finally:
assert counters[0] == counters[1]
No matter where the execution of the code stops, counters will always be in a consistent state, even without any cleanup.
Of course, this is a special case, which can't be always applied. A more general concept would be transactions where multiple actions are bundled and either all get executed or none.
Errors during rollback/abort of a transaction can be still problematic, though.
There is a transaction package, but I did not look into that.

best way is to used the techniques here. https://julien.danjou.info/atomic-lock-free-counters-in-python/
unfortunately other answers have some complex concurrency issues.

python - flow control when executing multiple functions inside a function

I've been writing some python app and unfortunately I stumbled upon (I believe...) a Design Pattern/Flow Control problem.
Let's say that I have some big function that includes multiple other function and outcome of the big function is strictly determined by small function success.
So the big function simply contains a series of operations.
If just one small function fails then we should abort the big function, unfortunately the deeper we go... then the small functions make more modifications to some objects, so I think that I just cannot abort the big function, because I have to revert/repair some of those unfinished changes made by small functions - "perform cleanup".
Is there a pythonic way to check/control execution of small functions inside a big function? Because the solution that I have now seems extremely ugly and not very ZENish...
Here is some pseudo code that represents the solution that I have now:
def small_func():
try:
# doing something here
return True # if operation successful
except Error:
return False
def big_func():
# state var helping determine if we need to cleanup if some of the
# funtions were unsuccessful
state = True
if not small_func1():
state = False
if not small_func2():
state = False
if not small_func3():
state = False
if not small_func4():
state = False
if not small_func4():
state = False
etc...
if not state:
#perform some cleanup after failed function since we can't
#finish the big function - we need to abort and clean unfinished stuff

You could save all states first, use a single try-block in the big function and reset in the except block if something goes wrong. Like this:
def small_func():
# no try-block
def big_func():
# Save all states here
try:
small_func1()
small_func2()
small_func3()
small_func4()
except Error:
# Reset all states here

Some events completed make another event in python

I am currently under a hard problems.
Say that I have 2+ events, such as:
(A): User do an action A
(B): User do an action B
Each event is separated from the other and can be triggered from web request by user, and I do not know their order (which one occurs first).
Then, when both of A and B are occurred, it will fire another event called C: A + B => C
Is there any programming model or library make it easier in Python, I prefer pure python just because I am working with GAE.
I have thought about an usage of blinker with some addition codes, but it is still much of works.
Thank you,

Since blinker is pure Python, using it on GAE should be quite feasible.
So you're left with the issue of triggering a signal when two other signals have been received -- an and gate, which I believe blinker doesn't provide -- definitely not when the incoming signals can come "well separated" in time and space.
So I'd define a model such as:
class AndGate(ndb.Model):
a_received = ndb.BooleanProperty()
b_received = ndb.BooleanProperty()
using users.get_user().userid() as the id in its entities' key.
Then we'd have signal(A).connect(got_a) and signal(B).connect(got_b) with
def got_a(*_):
k = ndb.Key('AndGate', users.get_user().userid())
ag = k.get()
if ag is None:
AndGate(a_received=True, b_received=False, key=k).put()
return
if ag.b_received:
k.delete()
signal('C').send('')
else:
ag.a_received = True
ag.put()
This looks a bit redundant with just two signals, I'm thinking forward to gates with N>2 signals but you can surely simplify this if that's not an issue for you.
Yes there's work left to do -- atomic transactions if needed, periodic "garbage collection" tasks looking for gates where only one signal was received and are older than some threshold X and reporting about them, and so on. But, it doesn't seem "much of works" to me, if I understood your specifications correctly.

This may help:
1.- Avoid a race condition: If you're using a DB use a transaction/lock table/lock row to write A or B, if you are using threads use Threading.Lock() or something similar
2.- Check inside the transacction/lock if the trigger condition is met, and then execute it.
#lock
def a():
do_a
if b:
do_c
#lock
def b():
do_b
if a:
do_c

How to have input while a while loop ( or similar) is executed

I do believe that thread may accomplish this, although I am not sure. Most of the threads out there that address this problem doesn't address it to match my problem. I have created a simple mud-like fighting system that executes when you 'fight' an NPC. I have the code that runs under a while loop that checks health between you and NPC and if one of you dies then the loop ends.
However
During the loop I want to have it where a user can type in commands, instead of being stuck watching a looping code block without you being able to do anything. From what I have read online it looks like thread module may be of some help to me? Also if anyone has PyGame experience, maybe looking into that would be a solution? Please let me know what you think.
Below is a very simple example of what I am trying to accomplish.
import time
fighting = True
while fighting:
# do the magic here
time.sleep(4) # to give it a nice even pace between loop intervals
Although at any time i want to be able do input a command like a skill or spell.
Any ideas or suggestions?

You can separate your human interface and fight game into separate threads. The fight game uses a queue for input, which uses a timeout to continue. Here is a very simple queue structure that should minimally do what you want.
import time
import threading
import Queue
def fighter(input_queue):
while True:
start = time.time()
# do stuff
wait = time.time() - start()
if wait <= 0.0:
wait = 0
try:
msg = input_queue.get(wait, wait)
if msg == 'done':
return
# do something else with message
except Queue.Empty:
pass
def main():
input_queue = Queue.Queue()
fight_thread = threading.Thread(target=fighter, args=(input_queue,))
fight_thread.start()
while True:
msg = raw_input('hello ') # py 2.x
input_queue.put(msg)
if msg == 'done':
break
fight_thread.join()

If you only want this to work on Windows, and you want to keep your simple event loop:
fighting = True
inputbuf = ''
while fighting:
# do the magic here
while msvcrt.khbit():
newkey = msvcrt.getwche()
inputbuf += newkey
if newkey == '\r':
process_command(inputbuf)
inputbuf = ''
time.sleep(4) # to give it a nice even pace between loop intervals
On the other hand, if you want to use a background thread, it would be a lot simpler:
def background():
for line in sys.stdin:
process_command(line)
bt = threading.Thread(target=background)
bt.start
fighting = True
while fighting:
# do the magic here
time.sleep(4) # to give it a nice even pace between loop intervals
This works cross-platform, and it gives normal line-buffered input (including full readline support), which people will probably like.
However, I'm assuming you want that process_command to share information with the # do the magic here code, and possibly even to set fighting = False. If you do that without any thread synchronization, it will no longer work cross-platform. (It may work on both Windows CPython and Unix CPython, but will probably not work on IronPython or Jython—or, worse, it will work most of the time but randomly fail just often enough that you have to fix it but not often enough that you can debug it…)

What you may be looking for is a non-blocking raw_input implementation. This would allow the loop to keep going while allowing the user a possibility at entering commands.
There is an example of an implementation of this here and here. Maybe you can adapt one of them to suit your purpose.
Edit:
Or, if you're working on Windows...

Python Mock Process for Unit Testing

Background:
I am currently writing a process monitoring tool (Windows and Linux) in Python and implementing unit test coverage. The process monitor hooks into the Windows API function EnumProcesses on Windows and monitors the /proc directory on Linux to find current processes. The process names and process IDs are then written to a log which is accessible to the unit tests.
Question:
When I unit test the monitoring behavior I need a process to start and terminate. I would love if there would be a (cross-platform?) way to start and terminate a fake system process that I could uniquely name (and track its creation in a unit test).
Initial ideas:
I could use subprocess.Popen() to open any system process but this runs into some issues. The unit tests could falsely pass if the process I'm using to test is run by the system as well. Also, the unit tests are run from the command line and any Linux process I can think of suspends the terminal (nano, etc.).
I could start a process and track it by its process ID but I'm not exactly sure how to do this without suspending the terminal.
These are just thoughts and observations from initial testing and I would love it if someone could prove me wrong on either of these points.
I am using Python 2.6.6.
Edit:
Get all Linux process IDs:
try:
processDirectories = os.listdir(self.PROCESS_DIRECTORY)
except IOError:
return []
return [pid for pid in processDirectories if pid.isdigit()]
Get all Windows process IDs:
import ctypes, ctypes.wintypes
Psapi = ctypes.WinDLL('Psapi.dll')
EnumProcesses = self.Psapi.EnumProcesses
EnumProcesses.restype = ctypes.wintypes.BOOL
count = 50
while True:
# Build arguments to EnumProcesses
processIds = (ctypes.wintypes.DWORD*count)()
size = ctypes.sizeof(processIds)
bytes_returned = ctypes.wintypes.DWORD()
# Call enum processes to find all processes
if self.EnumProcesses(ctypes.byref(processIds), size, ctypes.byref(bytes_returned)):
if bytes_returned.value &lt size:
return processIds
else:
# We weren't able to get all the processes so double our size and try again
count *= 2
else:
print "EnumProcesses failed"
sys.exit()
Windows code is from here

edit: this answer is getting long :), but some of my original answer still applies, so I leave it in :)
Your code is not so different from my original answer. Some of my ideas still apply.
When you are writing Unit Test, you want to only test your logic. When you use code that interacts with the operating system, you usually want to mock that part out. The reason being that you don't have much control over the output of those libraries, as you found out. So it's easier to mock those calls.
In this case, there are two libraries that are interacting with the sytem: os.listdir and EnumProcesses. Since you didn't write them, we can easily fake them to return what we need. Which in this case is a list.
But wait, in your comment you mentioned:
"The issue I'm having with it however is that it really doesn't test
that my code is seeing new processes on the system but rather that the
code is correctly monitoring new items in a list."
The thing is, we don't need to test the code that actually monitors the processes on the system, because it's a third party code. What we need to test is that your code logic handles the returned processes. Because that's the code you wrote. The reason why we are testing over a list, is because that's what your logic is doing. os.listir and EniumProcesses return a list of pids (numeric strings and integers, respectively) and your code acts on that list.
I'm assuming your code is inside a Class (you are using self in your code). I'm also assuming that they are isolated inside their own methods (you are using return). So this will be sort of what I suggested originally, except with actual code :) Idk if they are in the same class or different classes, but it doesn't really matter.
Linux method
Now, testing your Linux process function is not that difficult. You can patch os.listdir to return a list of pids.
def getLinuxProcess(self):
try:
processDirectories = os.listdir(self.PROCESS_DIRECTORY)
except IOError:
return []
return [pid for pid in processDirectories if pid.isdigit()]
Now for the test.
import unittest
from fudge import patched_context
import os
import LinuxProcessClass # class that contains getLinuxProcess method
def test_LinuxProcess(self):
"""Test the logic of our getLinuxProcess.
We patch os.listdir and return our own list, because os.listdir
returns a list. We do this so that we can control the output
(we test *our* logic, not a built-in library's functionality).
"""
# Test we can parse our pdis
fakeProcessIds = ['1', '2', '3']
with patched_context(os, 'listdir', lamba x: fakeProcessIds):
myClass = LinuxProcessClass()
....
result = myClass.getLinuxProcess()
expected = [1, 2, 3]
self.assertEqual(result, expected)
# Test we can handle IOERROR
with patched_context(os, 'listdir', lamba x: raise IOError):
myClass = LinuxProcessClass()
....
result = myClass.getLinuxProcess()
expected = []
self.assertEqual(result, expected)
# Test we only get pids
fakeProcessIds = ['1', '2', '3', 'do', 'not', 'parse']
.....
Windows method
Testing your Window's method is a little trickier. What I would do is the following:
def prepareWindowsObjects(self):
"""Create and set up objects needed to get the windows process"
...
Psapi = ctypes.WinDLL('Psapi.dll')
EnumProcesses = self.Psapi.EnumProcesses
EnumProcesses.restype = ctypes.wintypes.BOOL
self.EnumProcessses = EnumProcess
...
def getWindowsProcess(self):
count = 50
while True:
.... # Build arguments to EnumProcesses and call enun process
if self.EnumProcesses(ctypes.byref(processIds),...
..
else:
return []
I separated the code into two methods to make it easier to read (I believe you are already doing this). Here is the tricky part, EnumProcesses is using pointers and they are not easy to play with. Another thing is, that I don't know how to work with pointers in Python, so I couldn't tell you of an easy way to mock that out =P
What I can tell you is to simply not test it. Your logic there is very minimal. Besides increasing the size of count, everything else in that function is creating the space EnumProcesses pointers will use. Maybe you can add a limit to the count size but other than that, this method is short and sweet. It returns the windows processes and nothing more. Just what I was asking for in my original comment :)
So leave that method alone. Don't test it. Make sure though, that anything that uses getWindowsProcess and getLinuxProcess get's mocked out as per my original suggestion.
Hopefully this makes more sense :) If it doesn't let me know and maybe we can have a chat session or do a video call or something.
original answer
I'm not exactly sure how to do what you are asking, but whenever I need to test code that depends on some outside force (external libraries, popen or in this case processes) I mock out those parts.
Now, I don't know how your code is structured, but maybe you can do something like this:
def getWindowsProcesses(self, ...):
'''Call Windows API function EnumProcesses and
return the list of processes
'''
# ... call EnumProcesses ...
return listOfProcesses
def getLinuxProcesses(self, ...):
'''Look in /proc dir and return list of processes'''
# ... look in /proc ...
return listOfProcessses
These two methods only do one thing, get the list of processes. For Windows, it might just be a call to that API and for Linux just reading the /proc dir. That's all, nothing more. The logic for handling the processes will go somewhere else. This makes these methods extremely easy to mock out since their implementations are just API calls that return a list.
Your code can then easy call them:
def getProcesses(...):
'''Get the processes running.'''
isLinux = # ... logic for determining OS ...
if isLinux:
processes = getLinuxProcesses(...)
else:
processes = getWindowsProcesses(...)
# ... do something with processes, write to log file, etc ...
In your test, you can then use a mocking library such as Fudge. You mock out these two methods to return what you expect them to return.
This way you'll be testing your logic since you can control what the result will be.
from fudge import patched_context
...
def test_getProcesses(self, ...):
monitor = MonitorTool(..)
# Patch the method that gets the processes. Whenever it gets called, return
# our predetermined list.
originalProcesses = [....pids...]
with patched_context(monitor, "getLinuxProcesses", lamba x: originalProcesses):
monitor.getProcesses()
# ... assert logic is right ...
# Let's "add" some new processes and test that our logic realizes new
# processes were added.
newProcesses = [...]
updatedProcesses = originalProcessses + (newProcesses)
with patched_context(monitor, "getLinuxProcesses", lamba x: updatedProcesses):
monitor.getProcesses()
# ... assert logic caught new processes ...
# Let's "kill" our new processes and test that our logic can handle it
with patched_context(monitor, "getLinuxProcesses", lamba x: originalProcesses):
monitor.getProcesses()
# ... assert logic caught processes were 'killed' ...
Keep in mind that if you test your code this way, you won't get 100% code coverage (since your mocked methods won't be run), but this is fine. You're testing your code and not third party's, which is what matters.
Hopefully this might be able to help you. I know it doesn't answer your question, but maybe you can use this to figure out the best way to test your code.

Your original idea of using subprocess is a good one. Just create your own executable and name it something that identifies it as a testing thing. Maybe make it do something like sleep for a while.
Alternately, you could actually use the multiprocessing module. I've not used python in windows much, but you should be able to get process identifying data out of the Process object you create:
p = multiprocessing.Process(target=time.sleep, args=(30,))
p.start()
pid = p.getpid()