This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Python: single instance of program
I need to prevent a cron job from running concurrent instances when a job takes longer to complete than the launcher interval. I'm trying to use the flock concept to achieve this, but fcntl module is not behaving the way I expect.
Can anyone tell me why this works to prevent two concurrent instances:
import sys
import time
import fcntl
file_path = '/var/lock/test.py'
file_handle = open(file_path, 'w')
try:
fcntl.lockf(file_handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
print 'no other instance is running'
for i in range(5):
time.sleep(1)
print i + 1
except IOError:
print 'another instance is running exiting now'
sys.exit(0)
And why this does not work:
import sys
import time
import fcntl
def file_is_locked(file_path):
file_handle = open(file_path, 'w')
try:
fcntl.lockf(file_handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
return False
except IOError:
return True
file_path = '/var/lock/test.py'
if file_is_locked(file_path):
print 'another instance is running exiting now'
sys.exit(0)
else:
print 'no other instance is running'
for i in range(5):
time.sleep(1)
print i + 1
My humble opinion (although I may be totally wrong) is that file_handle is local to the function (in the second case) and therefore, it gets destroyed once the function is done.
The following code seems to work as expected:
#!/usr/bin/env python
#http://stackoverflow.com/questions/14406562/prevent-running-concurrent-instances-of-a-python-script
import sys
import time
import fcntl
file_handle = None
def file_is_locked(file_path):
global file_handle
file_handle= open(file_path, 'w')
try:
fcntl.lockf(file_handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
return False
except IOError:
return True
file_path = '/var/lock/test.py'
if file_is_locked(file_path):
print 'another instance is running exiting now'
sys.exit(0)
else:
print 'no other instance is running'
for i in range(5):
time.sleep(1)
print i + 1
Notice that the only thing I did is setting file_handle as global variable (although I copied the whole code to have a working example)
As I mentioned in my commen on #BorrajaX's answer, since it looks like you are POSIX-bound anyway, you could try using a native named semaphore.
You could use the setlock program from D. J. Bernstein's daemontools instead:
http://cr.yp.to/daemontools/setlock.html
Easiest way would be to create a file /tmp/scriptlock at the start of the script and check if that file exists before doing work. Make sure the lock file is removed though at the end of processing.
Related
I want to remote control a python application which uses urwid for the user interface.
My idea was to create a file, pass it's name as command line argument to the application and whenever I write to the file the application reads from that file.
Urwid's event loop has a method watch_file(fd, callback).
This method is described as "Call callback() when fd has some data to read."
This sounds exactly like what I want to have, but it causes an infinite loop.
callback is executed as often as possible, despite the fact that the file is empty.
Even if I delete the file, callback is still called.
#!/usr/bin/env python3
import urwid
import atexit
def onkeypress(key, size=None):
if key == 'q':
raise urwid.ExitMainLoop()
text.set_text(key)
def onfilechange():
text.set_text(cmdfile.read())
# clear file so that I don't read already executed commands again
# and don't run into an infinite loop - but I am doing that anyway
with open(cmdfile.name, 'w') as f:
pass
cmdfile = open('/tmp/cmd', 'rt')
atexit.register(cmdfile.close)
text = urwid.Text("hello world")
filler = urwid.Filler(text)
loop = urwid.MainLoop(filler, unhandled_input=onkeypress)
loop.watch_file(cmdfile, onfilechange)
if __name__ == '__main__':
loop.run()
(My initial idea was to open the file only for reading instead of keeping it open all the time but fd has to be a file object, not a path.)
Urwid offers several different event loops.
By default, SelectEventLoop is used.
GLibEventLoop has the same behaviour, it runs into an infinite loop.
AsyncioEventLoop instead throws an "operation not permitted" exception.
TwistedEventLoop and TornadoEventLoop would need additional software to be installed.
I have considered using the independent watchdog library but it seems accessing the user interface from another thread would require to write a new loop, see this stack overflow question.
The answer to that question recommends polling instead which I would prefer to avoid.
If urwid specifically provides a method to watch a file I cannot believe that it does not work in any implementation.
So what am I doing wrong?
How do I react to a file change in a python/urwid application?
EDIT:
I have tried using named pipes (and removed the code to clear the file) but visually it has the same behaviour: the app does not start.
Audibly, however, there is a great difference: It does not go into the infinite loop until I write to the file.
Before I write to the file callback is not called but the app is not started either, it just does nothing.
After I write to the file, it behaves as described above for regular files.
I have found the following work around: read a named pipe in another thread, safe each line in a queue and poll in the UI thread to see if something is in the queue.
Create the named pipe with mkfifo /tmp/mypipe.
Then write to it with echo >>/tmp/mypipe "some text".
#!/usr/bin/env python3
import os
import threading
import queue
import urwid
class App:
POLL_TIME_S = .5
def __init__(self):
self.text = urwid.Text("hello world")
self.filler = urwid.Filler(self.text)
self.loop = urwid.MainLoop(self.filler, unhandled_input=self.onkeypress)
def watch_pipe(self, path):
self._cmd_pipe = path
self.queue = queue.Queue()
threading.Thread(target=self._read_pipe_thread, args=(path,)).start()
self.loop.set_alarm_in(0, self._poll_queue)
def _read_pipe_thread(self, path):
while self._cmd_pipe:
with open(path, 'rt') as pipe:
for ln in pipe:
self.queue.put(ln)
self.queue.put("!! EOF !!")
def _poll_queue(self, loop, args):
while not self.queue.empty():
ln = self.queue.get()
self.text.set_text(ln)
self.loop.set_alarm_in(self.POLL_TIME_S, self._poll_queue)
def close(self):
path = self._cmd_pipe
# stop reading
self._cmd_pipe = None
with open(path, 'wt') as pipe:
pipe.write("")
os.remove(path)
def run(self):
self.loop.run()
def onkeypress(self, key, size=None):
if key == 'q':
raise urwid.ExitMainLoop()
self.text.set_text(key)
if __name__ == '__main__':
a = App()
a.watch_pipe('/tmp/mypipe')
a.run()
a.close()
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
I was messing around with a zip file cracker and decided to use the multiprocessing module to speed the process up. It was a complete pain since it was my first time using the module and I don't even fully understand it yet. However, I got it to work.
The problem is that it doesn't complete the word list; it just stops at random puts during the word list, and if the password is found it continues to go through the word list instead of just stopping the process.
Does anyone know why it's exhibiting this behaviour?
Source Code For ZipFile Cracker
#!/usr/bin/env python3
import multiprocessing as mp
import zipfile # Handeling the zipfile
import sys # Command line arguments, and quiting application
import time # To calculate runtime
def usage(program_name):
print("Usage: {0} <path to zipfile> <dictionary>".format(program_name))
sys.exit(1)
def cracker(password):
try:
zFile.extractall(pwd=password)
print("[+] Password Found! : {0}".format(password.decode('utf-8')))
pool.close()
except:
pass
def main():
global zFile
global pool
if len(sys.argv) < 3:
usage(sys.argv[0])
zFile = zipfile.ZipFile(sys.argv[1])
print("[*] Started Cracking")
startime = time.time()
pool = mp.Pool()
for i in open(sys.argv[2], 'r', errors='ignore'):
pswd = bytes(i.strip('\n'), 'utf-8')
pool.apply_async(cracker, (pswd,))
print (pswd)
runtime = round(time.time() - startime, 5)
print ("[*] Runtime:", runtime, 'seconds')
sys.exit(0)
if __name__ == "__main__":
main()
You are terminating your program too early. To test this out, add a harmless time.sleep(10) in the cracker method and observe your program still terminating within a second.
Call join to wait for the pool to finish:
pool = mp.Pool()
for i in open(sys.argv[2], 'r', errors='ignore'):
pswd = bytes(i.strip('\n'), 'utf-8')
pool.apply_async(cracker, (pswd,))
pool.close() # Indicate that no more data is coming
pool.join() # Wait for pool to finish processing
runtime = round(time.time() - startime, 5)
print ("[*] Runtime:", runtime, 'seconds')
sys.exit(0)
Additionally, once you find the right password, calling close just indicates that no more future tasks are coming - all tasks already submitted will still be done. Instead, call terminate to kill the pool without processing any more tasks.
Furthermore, depending on the implementation details of multiprocessing.Pool, the global variable pool may not be available when you need it (and its value isn't serializable anyways). To solve this problem, you can use a callback, as in
def cracker(password):
try:
zFile.extractall(pwd=password)
except RuntimeError:
return
return password
def callback(found):
if found:
pool.terminate()
...
pool.apply_async(cracker, (pswd,), callback=cb)
Of course, since you now look at the result all the time, apply is not the right way to go. Instead, you can write your code using imap_unordered:
with open(sys.argv[2], 'r', errors='ignore') as passf, \
multiprocessing.Pool() as pool:
passwords = (line.strip('\n').encode('utf-8') for line in passf)
for found in pool.imap_unordered(cracker, passwords):
if found:
break
Instead of using globals, you may also want to open the zip file (and create a ZipFile object) in each process, by using an initializer for the pool. Even better (and way faster), forgo all of the I/O in the first place and read just the bytes you need once and then pass them on to the children.
phihag's answer is the correct solution.
I just wanted to provide an additional detail regarding calling terminate() when you've found the correct password. The pool variable in cracker() was not defined when I ran the code. So trying to invoke it from there simply threw an exception:
NameError: name 'pool' is not defined
(My fork() experience is weak, so I don't completely understand why the global zFile is copied to the child processes successfully while pool is not. Even if it were copied, it would not be the same pool in the parent process, right? So any methods invoked on it would have no effect on the real pool in the parent process. Regardless, I prefer this advice listed within the multiprocessing module's Programming guidelines section: Explicitly pass resources to child processes.)
My suggestion is to make cracker() return the password if it is correct, otherwise return None. Then pass a callback to apply_async() that records the correct password, as well as terminating the pool. Here's my take at modifying your code to do this:
#!/usr/bin/env python3
import multiprocessing as mp
import zipfile # Handeling the zipfile
import sys # Command line arguments, and quiting application
import time # To calculate runtime
import os
def usage(program_name):
print("Usage: {0} <path to zipfile> <dictionary>".format(program_name))
sys.exit(1)
def cracker(zip_file_path, password):
print('[*] Starting new cracker (pid={0}, password="{1}")'.format(os.getpid(), password))
try:
time.sleep(1) # XXX: to simulate the task taking a bit of time
with zipfile.ZipFile(zip_file_path) as zFile:
zFile.extractall(pwd=bytes(password, 'utf-8'))
return password
except:
return None
def main():
if len(sys.argv) < 3:
usage(sys.argv[0])
print('[*] Starting main (pid={0})'.format(os.getpid()))
zip_file_path = sys.argv[1]
password_file_path = sys.argv[2]
startime = time.time()
actual_password = None
with mp.Pool() as pool:
def set_actual_password(password):
nonlocal actual_password
if password:
print('[*] Found password; stopping future tasks')
pool.terminate()
actual_password = password
with open(password_file_path, 'r', errors='ignore') as password_file:
for pswd in password_file:
pswd = pswd.strip('\n')
pool.apply_async(cracker, (zip_file_path, pswd,), callback=set_actual_password)
pool.close()
pool.join()
if actual_password:
print('[*] Cracked password: "{0}"'.format(actual_password))
else:
print('[*] Unable to crack password')
runtime = round(time.time() - startime, 5)
print("[*] Runtime:", runtime, 'seconds')
sys.exit(0)
if __name__ == "__main__":
main()
Here's an implementation of the advice from #phihag's and #Equality 7-2521's answers:
#!/usr/bin/env python3
"""Brute force zip password.
Usage: brute-force-zip-password <zip archive> <passwords>
"""
import sys
from multiprocessing import Pool
from time import monotonic as timer
from zipfile import ZipFile
def init(archive): # run at the start of a worker process
global zfile
zfile = ZipFile(open(archive, 'rb')) # open file in each process once
def check(password):
assert password
try:
with zfile.open(zfile.infolist()[0], pwd=password):
return password # assume success
except Exception as e:
if e.args[0] != 'Bad password for file':
# assume all other errors happen after the password was accepted
raise RuntimeError(password) from e
def main():
if len(sys.argv) != 3:
sys.exit(__doc__) # print usage
start = timer()
# decode passwords using the preferred locale encoding
with open(sys.argv[2], errors='ignore') as file, \
Pool(initializer=init, initargs=[sys.argv[1]]) as pool: # use all CPUs
# check passwords encoded using utf-8
passwords = (line.rstrip('\n').encode('utf-8') for line in file)
passwords = filter(None, passwords) # filter empty passwords
for password in pool.imap_unordered(check, passwords, chunksize=100):
if password is not None: # found
print("Password: '{}'".format(password.decode('utf-8')))
break
else:
sys.exit('Unable to find password')
print('Runtime: %.5f seconds' % (timer() - start,))
if __name__=="__main__":
main()
Note:
each worker process has its own ZipFile object and the zip file is opened once per process: it should make it more portable (Windows support) and improve time performance
the content is not extracted: check(password) tries to open and immediately closes an archive member on success: it is safer and it should improve time performance (no need to create directories, etc)
all errors except 'Bad password for file' while decrypting the archive member are assumed to happen after the password is accepted: the rational is to avoid silencing unexpected errors -- each exception should be considered individually
check(password) expects nonempty passwords
chunksize parameter may drastically improve performance
a rare for/else syntax is used, to report cases when the password is not found
the with-statement calls pool.terminate() for you
I'm testing to see if the atexit module runs properly by attempting to print to file. The actual purpose of atexit will be to close the open port, however this is to test if atexit is working in the first place. However, atexit is not printing to file, any ideas why not? Any help is appreciated.
import atexit
import scala5
from scala5 import sharedvars
scalavars = sharedvars()
import serial
myPort=serial.Serial()
myPort.baudrate = 9600
myPort.port = "COM4"
myPort.parity=serial.PARITY_NONE
myPort.stopbits=serial.STOPBITS_ONE
myPort.bytesize=serial.EIGHTBITS
myPort.timeout=2
myPort.writeTimeout=2
try:
myPort.close()
except:
print("didn't need to close")
def port_exit():
fo = open("test.txt", "w")
fo.write("This works!")
fo.close()
myPort.open()
while True:
x = myPort.readline()
if x == "1\r\n":
scalavars.door = x
scala5.ScalaPlayer.Sleep(10)
atexit.register(port_exit)
port_exit()
You are registering the function after your while loop, which I am assuming you are exiting by killing the script (meaning the function port_exit isn't registered). If you register the function before the while then it should trigger and write the file when the script exits.
atexit.register(port_exit)
while True:
x = myPort.readline()
if x == "1\r\n":
scalavars.door = x
scala5.ScalaPlayer.Sleep(10)
I have a few scripts that i want to run simultaneously, they read a CSV file, im trying the following;
import sys
import csv
out = open("C:\PYDUMP\PYDUMPINST.csv","r")
dataf=csv.reader(out)
for row in dataf:
take = row[0]
give = row[1]
def example():
try:
lfo = int(take)
if lfo > 0:
#code
except Exception, e:
pass
example()
This is saved as takefile1.py. I have 20 scripts with similar structures that i want to run simultaneously. So im using(which i have been using for running other batches of scripts trouble free) the following;
import csv
import sys
from threading import Thread
def firstsend_lot():
execfile("C:\Users\takefile1.py")
execfile("C:\Users\takefile2.py")
def secondsend_lot():
execfile("C:\Users\takefile3.py")
execfile("C:\Users\takefile4.py")
if __name__ == '__main__':
Thread(target = firstsend_lot).start()
Thread(target = secondsend_lot).start()
So i am getting the error "global name 'take' is not defined". Anyone got any suggestions? Im pretty hopeless at Python so pretend you are talking to an idiot.
Your function example(), do not have access to take. Try adding a line in it:
def example():
global take
try:
lfo = int(take)
if lfo > 0:
#code
except Exception, e:
pass
I would expect next code to be executed simultaneously and all filenames from os.walk iterations , that got 0 at random , will get in result dictionary. And all threads that have some timeout would get into deamon mode and will be killed as soon as script reaches end. However, script respects all timeouts for each thread.
Why is this happening? Should it put all threads in backgroung and kill them if they will not finish and return result before the end of script execution? thank you.
import threading
import os
import time
import random
def check_file(file_name,timeout):
time.sleep(timeout)
print file_name
result.append(file_name)
result = []
for home,dirs,files in os.walk("."):
for ifile in files :
filename = '/'.join([home,ifile])
t = threading.Thread(target=check_file(filename,random.randint(0,5)))
t.setDaemon(True)
t.start()
print result
Solution: I found my mistake:
t = threading.Thread(target=check_file(filename,random.randint(0,5)))
has to be
t = threading.Thread(target=check_file, args=(filename,random.randint(0,5)))
In this case, threading will spawn a thread with function as object ang give it arguments. In my initial example, function with args has to be resolved BEFORE thread spawns. And this is fair.
However, example above works for me at 2.7.3 , but at 2.7.2 i cannot make it working.
I `m getting got exception that
function check_file accepts exactly 1 argument (34 is given).
Soulution :
in 2.7.2 i had to put ending coma in args tuple , considering that i have 1 variable only . God knows why this not affects 2.7.3 version . It was
t = threading.Thread(target=check_file, args=(filename))
and started to work with
t = threading.Thread(target=check_file, args=(filename,))
I understand what you were trying to do, but you're not using the right format for threading. I fixed your example...look up the Queue class on how to do this properly.
Secondly, never ever do string manipulation on file paths. Use the os.path module; there's a lot more than adding separators between strings that you and I don't think about most of the time.
Good luck!
import threading
import os
import time
import random
import Queue
def check_file():
while True:
item = q.get()
time.sleep(item[1])
print item
q.task_done()
q = Queue.Queue()
result = []
for home,dirs,files in os.walk("."):
for ifile in files:
filename = os.path.join(home, ifile)
q.put((filename, random.randint(0,5)))
number_of_threads = 25
for i in range(number_of_threads):
t = threading.Thread(target=check_file)
t.daemon = True
t.start()
q.join()
print result