We Keep Coding

Linux IPC: Locking, but not file based locking

I need a way to ensure only one python process is processing a directory.
The lock/semaphore should be local to the machine (linux operating system).
Networking or NFS is not involved.
I would like to avoid file based locks, since I don't know where I should put the lock file.
There are libraries which provide posix IPC at pypi.
Is there no way to use linux semaphores with python without a third party library?
The lock provided by multiprocessing.Lock does not help, since both python interpreter don't share one the same parent.
Threading is not involved. All processes have only one thread.
I am using Python 2.7 on linux.
How to to synchronize two python scripts on linux (without file based locking)?
Required feature: If one process dies, then the lock/semaphore should get released by the operating system.

flock the directory itself — then you never need worry about where to put the lock file:
import errno
import fcntl
import os
import sys
# This will work on Linux
dirfd = os.open(THE_DIRECTORY, os.O_RDONLY) # FIXME: FD_CLOEXEC
try:
fcntl.flock(dirfd, fcntl.LOCK_EX|fcntl.LOCK_NB)
except IOError as ex:
if ex.errno != errno.EAGAIN:
raise
print "Somebody else is working here; quitting." # FIXME: logging
sys.exit(1)
do_the_work()
os.close(dirfd)

I would like to avoid file based locks, since I don't know where I should put the lock file.
You can lock the existing file or directory (the one being processed).
Required feature: If one process dies, then the lock/semaphore should get released by the operating system.
That is exactly how file locks work.

PyCuda / Multiprocessing Issue on OS X 10.8

I'm working on a project where I distribute compute tasks to multiple python Processes each associated with its own CUDA device.
When spawning the subprocesses, I use the following code:
import pycuda.driver as cuda
class ComputeServer(object):
def _init_workers(self):
self.workers = []
cuda.init()
for device_id in range(cuda.Device.count()):
print "initializing device {}".format(device_id)
worker = CudaWorker(device_id)
worker.start()
self.workers.append(worker)
The CudaWorker is defined in another file as follows:
from multiprocessing import Process
import pycuda.driver as cuda
class CudaWorker(Process):
def __init__(self, device_id):
Process.__init__(self)
self.device_id = device_id
def run(self):
self._init_cuda_context()
while True:
# process requests here
def _init_cuda_context(self):
# the following line fails
cuda.init()
device = cuda.Device(self.device_id)
self.cuda_context = device.make_context()
When I run this code on Windows 7 or Linux, I have no issues. When running the code on my MacBook Pro with OSX 10.8.2, Cuda 5.0, and PyCuda 2012.1 I get the following error:
Process CudaWorker-1:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/Users/tombnorwood/pymodules/computeserver/worker.py", line 32, in run
self._init_cuda_context()
File "/Users/tombnorwood/pymodules/computeserver/worker.py", line 38, in _init_cuda_context
cuda.init()
RuntimeError: cuInit failed: no device
I have no issues running PyCuda scripts without forking new processes on my Mac. I only get this issue when spawning a new Process.
Has anyone run into this issue before?

This is really just an educated guess based on my experienced, but I suspect that the OS X implementation of CUDA (or possibly PyCuda) relies on some APIs that can't be used safely after fork, while the linux implementation does not.* Since the POSIX implementation of multiprocessing uses fork without exec to create child processes, this would explain why it fails on OS X but not linux. (And on Windows, there is no fork, just a spawn equivalent, so this isn't an issue.)
The simplest solution would be to drop multiprocessing. If CUDA and PyCUDA are thread-safe (I don't know if they are), and your code is not CPU-bound (just GPU-bound), you might be able to just drop in threading.Thread in place of multiprocessing.Process and be done with it. Or you could consider one of the other parallel-processing libraries that provide similar APIs to multiprocessing. (There are a few people who use pp only because it always execs…)
However, it's pretty easy to hack up multiprocessing to exec/spawn a new Python interpreter and then do everything Windows-style instead of POSIX-style. (Getting every case right is difficult, but getting one specific use case right is easy.)
Or, if you look at bug #8713, there's some work being done on making this work right in general. And there are working patches. Those patches are for 3.3, not 2.7, so you'd probably need a bit of massaging, but it shouldn't be very much. So, just cp $MY_PYTHON_LIB/multiprocessing.py $MY_PROJECT_DIR/mymultiprocessing.py, patch it, use mymultiprocessing in place of multiprocessing, and add the appropriate call to pick spawn/fork+exec/whatever the mode is called in the latest patch before you do anything else.
* The OP says he suspected the same thing, so I probably don't need to explain this to him, but for future readers: It's not about a difference between Darwin and other Unixes, but about the fact that Apple ships a lot of non-Unix-y mid-level libraries like CoreFoundation.framework, Accelerate.framework, etc. that use unsafe-after-fork functionality (or just assert that they're not being used after a fork because Apple doesn't want to put in the rigorous testing that would be warranted before they could say "as of 10.X, Foo.framework is safe after fork"). Also, if you compare the way OS X and linux deal with graphics and other hardware, there's a lot more mid-level in-each-process-userspace going on in OS X.

How to start daemon process from python on windows?

Can my python script spawn a process that will run indefinitely?
I'm not too familiar with python, nor with spawning deamons, so I cam up with this:
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_NEW_CONSOLE
subprocess.Popen(executable, close_fds = True, startupinfo = si)
The process continues to run past python.exe, but is closed as soon as I close the cmd window.

Using the answer Janne Karila pointed out this is how you can run a process that doen't die when its parent dies, no need to use the win32process module.
DETACHED_PROCESS = 8
subprocess.Popen(executable, creationflags=DETACHED_PROCESS, close_fds=True)
DETACHED_PROCESS is a Process Creation Flag that is passed to the underlying CreateProcess function.

This question was asked 3 years ago, and though the fundamental details of the answer haven't changed, given its prevalence in "Windows Python daemon" searches, I thought it might be helpful to add some discussion for the benefit of future Google arrivees.
There are really two parts to the question:
Can a Python script spawn an independent process that will run indefinitely?
Can a Python script act like a Unix daemon on a Windows system?
The answer to the first is an unambiguous yes; as already pointed out; using subprocess.Popen with the creationflags=subprocess.CREATE_NEW_PROCESS_GROUP keyword will suffice:
import subprocess
independent_process = subprocess.Popen(
'python /path/to/file.py',
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
)
Note that, at least in my experience, CREATE_NEW_CONSOLE is not necessary here.
That being said, the behavior of this strategy isn't quite the same as what you'd expect from a Unix daemon. What constitutes a well-behaved Unix daemon is better explained elsewhere, but to summarize:
Close open file descriptors (typically all of them, but some applications may need to protect some descriptors from closure)
Change the working directory for the process to a suitable location to prevent "Directory Busy" errors
Change the file access creation mask (os.umask in the Python world)
Move the application into the background and make it dissociate itself from the initiating process
Completely divorce from the terminal, including redirecting STDIN, STDOUT, and STDERR to different streams (often DEVNULL), and prevent reacquisition of a controlling terminal
Handle signals, in particular, SIGTERM.
The reality of the situation is that Windows, as an operating system, really doesn't support the notion of a daemon: applications that start from a terminal (or in any other interactive context, including launching from Explorer, etc) will continue to run with a visible window, unless the controlling application (in this example, Python) has included a windowless GUI. Furthermore, Windows signal handling is woefully inadequate, and attempts to send signals to an independent Python process (as opposed to a subprocess, which would not survive terminal closure) will almost always result in the immediate exit of that Python process without any cleanup (no finally:, no atexit, no __del__, etc).
Rolling your application into a Windows service, though a viable alternative in many cases, also doesn't quite fit. The same is true of using pythonw.exe (a windowless version of Python that ships with all recent Windows Python binaries). In particular, they fail to improve the situation for signal handling, and they cannot easily launch an application from a terminal and interact with it during startup (for example, to deliver dynamic startup arguments to your script, say, perhaps, a password, file path, etc), before "daemonizing". Additionally, Windows services require installation, which -- though perfectly possible to do quickly at runtime when you first call up your "daemon" -- modifies the user's system (registry, etc), which would be highly unexpected if you're coming from a Unix world.
In light of that, I would argue that launching a pythonw.exe subprocess using subprocess.CREATE_NEW_PROCESS_GROUP is probably the closest Windows equivalent for a Python process to emulate a traditional Unix daemon. However, that still leaves you with the added challenge of signal handling and startup communications (not to mention making your code platform-dependent, which is always frustrating).
That all being said, for anyone encountering this problem in the future, I've rolled a library called daemoniker that wraps both proper Unix daemonization and the above strategy. It also implements signal handling (for both Unix and Windows systems), and allows you to pass objects to the "daemon" process using pickle. Best of all, it has a cross-platform API:
from daemoniker import Daemonizer
with Daemonizer() as (is_setup, daemonizer):
if is_setup:
# This code is run before daemonization.
do_things_here()
# We need to explicitly pass resources to the daemon; other variables
# may not be correct
is_parent, my_arg1, my_arg2 = daemonizer(
path_to_pid_file,
my_arg1,
my_arg2
)
if is_parent:
# Run code in the parent after daemonization
parent_only_code()
# We are now daemonized, and the parent just exited.
code_continues_here()

For that purpose you could daemonize your python process or as you are using windows environment you would like to run this as a windows service.
You know i like to hate posting only web-links:
But for more information according to your requirement:
A simple way to implement Windows Service. read all comments it will resolve any doubt
If you really want to learn more
First read this
what is daemon process or creating-a-daemon-the-python-way
update:
Subprocess is not the right way to achieve this kind of thing

How can I watch for a filesystem change without drastically affecting system performance?

Time for another newbie question, I fear. I'm attempting to use Python 3.2.2 (the version is important, in this case) to monitor a particular Windows path for changes. The simplest method, and the method I'm using, is:
original_state = os.listdir(path_string)
while os.listdir(path_string) == original_state:
time.sleep(1)
change_time = datetime.datetime.now()
I'm writing this code to do some timing tests of another application. With that goal in mind, the Python script needs to (a) not adversely affect system performance, and (b) be relatively precise -- a margin of error of +/- 1 second is the absolute maximum I can justify. Unfortunately, this method doesn't meet the first criterion: When running this particular bit of code, the virtual environment is hammered, drastically slowing down the operations whose performance I'm trying to accurately measure.
I've read how to watch a File System for change, How do I watch a file for changes?, and http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html (an article recommended as a solution to that second SO question.) Unfortunately, Tim Golden's code appears to be Python 2.x code -- as near as I can tell, the pywin32 module isn't supported in Python 3.
What can I do in Python 3 to monitor this particular path without running into the same performance problems?

According to the ActivePython 3.2 Documentation, their pywin32 now supports Python 3.x

On Linux there is iNotify and pyNotify. Similar asynchronous notification mechanism on windows is FindFirstChangeNotification function which is a part of FileSystemWatcher Class
Please look at solutions on the Tim Golden's page:
http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html#use_findfirstchange

It is also possible to monitor a file or directory using GFileMonitor with Gio taking care of the underlying operating system details. Although, granted you likely won't be using Gtk if this is a Windows program. For posterity:
from gi.repository import Gio
gfile = Gio.file_new_for_path('/home/user/Downloads')
gfilemonitor = gfile.monitor(Gio.FileMonitorFlags.NONE, None)
gfilemonitor.connect('changed', callback_func)

Memory leak when using shared library with thread local storage via ctypes in a python program

I am using the ctypes module in python to load a shared c-library , which contains thread local storage. Its a quite large c-library with a long history, that we are trying to make thread safe. The library contains lots of global variables and statics, so our initial strategy towards thread safety has been to use thread local storage. We want our libarary to be platform independent, and have been compiling and testing thread safety on both win32, win64 and 64-bit Ubuntu. From a pure c-process there doesn't seem to be any problems.
However in python (2.6 and 2.7) on win32 and on Ubuntu we are seeing memory leaks. It seems like the thread local storage is not being released properly when a python thread terminates. Or at least that somehow the python process is not "aware" about that the memory is freed. The same problem is also seen in a c#-program on win32 actually, but it is not present on our win64 server test machine (running python 2.7 also).
The problem can be reproduced with a simple toy example like this:
Create a c-file containing (on linux/unix remove __declspec(dllexport)):
#include <stdio.h>
#include <stdlib.h>
void __declspec(dllexport) Leaker(int tid){
static __thread double leaky[1024];
static __thread int init=0;
if (!init){
printf("Thread %d initializing.", tid);
int i;
for (i=0;i<1024;i++) leaky[i]=i;
init=1;}
else
printf("This is thread: %d\n",tid);
return;}
Compile wit MINGW on windows/gcc on linux like:
gcc -o leaky.dll (or leaky.so) -shared the_file.c
On windows we could have compiled with Visual Studio, replacing __thread with __declspec(thread). However on win32 (up to winXP I believe), this does not work if the library is to be loaded in runtime with LoadLibrary.
Now create a python program like:
import threading, ctypes, sys, time
NRUNS=1000
KEEP_ALIVE=5
REPEAT=2
lib=ctypes.cdll.LoadLibrary("leaky.dll")
lib.Leaker.argtypes=[ctypes.c_int]
lib.Leaker.restype=None
def UseLibrary(tid,repetitions):
for i in range(repetitions):
lib.Leaker(tid)
time.sleep(0.5)
def main():
finished_threads=0
while finished_threads<NRUNS:
if threading.activeCount()<KEEP_ALIVE:
finished_threads+=1
thread=threading.Thread(target=UseLibrary,args=(finished_threads,REPEAT))
thread.start()
while threading.activeCount()>1:
print("Active threads: %i" %threading.activeCount())
time.sleep(2)
return
if __name__=="__main__":
sys.exit(main())
That is enough to reproduce the error. Explicitly import the garbage collector, doing a collect gc.collect() when starting each new thread does not help.
For a while I thought that the problem had to do with incompatible runtimes (python compiled with Visual Studio, my library with MINGW). But the problem is also on Ubuntu, but not on a win64 server, even when the library is cross compiled with MINGW.
Hope that anyone can help!
Cheers,
Simon Kokkendorff, National Survey and Cadastre of Denmark.

This seems not to be ctypes' or Python's fault at all. I can reproduce the same leak, leaking at the same rate, by writing only C code.
Strangely, at least on Ubuntu Linux 64, the leak occurs if the Leaker() function with the __thread variables is compiled as an .so and called from a program with dlopen(). It does not occur when running exactly the same code but with both parts compiled together as a regular C program.
I suspect that the fault is some interaction between dynamically linked libraries and thread-local storage. Still, it looks like a rather bad bug (is it really undocumented?).

My guess is that not joining with the threads is the problem. From the man page for pthread_join:
Failure to join with a thread that is joinable (i.e., one that is not
detached), produces a "zombie thread". Avoid doing this, since each
zombie thread consumes some system resources, and when enough zombie
threads have accumulated, it will no longer be possible to create new
threads (or processes).
If you modify your loop to collect the thread objects and use .isAlive() and .join() on them in that last while loop I think it should take care of your memory leak.