i'm quite new with python/programming and i've stumbled upon this "issue".
I' d like to have my main script run a separate script multiple times, get all the processes that i have open this way and the possibility to shut them off.
i need to specify some variables every time from my main code, in order for those process to run separately, each with his own "setting" that i specify from the terminal on my main file.
Can You point me to the right direction? what do i need to read/search to adress this stuff?
basic example:
my main.py file will ask for a string eg. val = input("value: "). I type "Hello".
my process.py file will run in a loop, printing "hello" until i say stop.
While first process is running, i can specify another string in my main eg. val = "bye" and process.py would start again with the string "bye", whithout shutting down the one with "hello".
until i specify so, they both have to run.
and i'd also like to have a list of tot n. of process.py running, with their "specs", and the possibility to shut them of eg. with an id assigned to each one of them.
Am i doing it all wrong? there's a better way to do this? i don' t know at all.
Thanks for helping me
You can spawn subprocesses running your process.py script with different parameters.
These subprocesses will continue running independently of the parent process, or you can monitor and kill them from your main.py, which could look like this.
main.py
import subprocess
import time
def spawn_process_py(arguments):
args = [
sys.executable, # use the same Python this script is using
'process.py', # the script to run
] + list(arguments)
return subprocess.Popen(args)
p1 = spawn_process_py(['hello'])
p2 = spawn_process_py(['ohai'])
time.sleep(5) # wait for a bit
p1.terminate() # kill process 1
time.sleep(5) # wait for a bit
p2.terminate() # kill process 2
You can naturally keep track of the subprocesses using a dict or an array instead of just keeping them in free variables like p1 and p2.
process.py
import sys
import time
while True:
print('Process.py:', sys.argv[1:])
time.sleep(.2)
Other methods are e.g. the multiprocessing module, which adds inter-process communication, etc, but this is arguably the most straightforward.
Related
In a program I am writing in python I need to completely restart the program if a variable becomes true, looking for a while I found this command:
while True:
if reboot == True:
os.execv(sys.argv[0], sys.argv)
When executed it returns the error [Errno 8] Exec format error. I searched for further documentation on os.execv, but didn't find anything relevant, so my question is if anyone knows what I did wrong or knows a better way to restart a script (by restarting I mean completely re-running the script, as if it were been opened for the first time, so with all unassigned variables and no thread running).
There are multiple ways to achieve the same thing. Start by modifying the program to exit whenever the flag turns True. Then there are various options, each one with its advantages and disadvantages.
Wrap it using a bash script.
The script should handle exits and restart your program. A really basic version could be:
#!/bin/bash
while :
do
python program.py
sleep 1
done
Start the program as a sub-process of another program.
Start by wrapping your program's code to a function. Then your __main__ could look like this:
def program():
### Here is the code of your program
...
while True:
from multiprocessing import Process
process = Process(target=program)
process.start()
process.join()
print("Restarting...")
This code is relatively basic, and it requires error handling to be implemented.
Use a process manager
There are a lot of tools available that can monitor the process, run multiple processes in parallel and automatically restart stopped processes. It's worth having a look at PM2 or similar.
IMHO the third option (process manager) looks like the safest approach. The other approaches will have edge cases and require implementation from your side to handle edge cases.
This has worked for me. Please add the shebang at the top of your code and os.execv() as shown below
#!/usr/bin/env python3
import os
import sys
if __name__ == '__main__':
while True:
reboot = input('Enter:')
if reboot == '1':
sys.stdout.flush()
os.execv(sys.executable, [sys.executable, __file__] + [sys.argv[0]])
else:
print('OLD')
I got the same "Exec Format Error", and I believe it is basically the same error you get when you simply type a python script name at the command prompt and expect it to execute. On linux it won't work because a path is required, and the execv method is basically encountering the same error.
You could add the pathname of your python compiler, and that error goes away, except that the name of your script then becomes a parameter and must be added to the argv list. To avoid that, make your script independently executable by adding "#!/usr/bin/python3" to the top of the script AND chmod 755.
This works for me:
#!/usr/bin/python3
# this script is called foo.py
import os
import sys
import time
if (len(sys.argv) >= 2):
Arg1 = int(sys.argv[1])
else:
sys.argv.append(None)
Arg1 = 1
print(f"Arg1: {Arg1}")
sys.argv[1] = str(Arg1 + 1)
time.sleep(3)
os.execv("./foo.py", sys.argv)
Output:
Arg1: 1
Arg1: 2
Arg1: 3
.
.
.
I am using the multiprocessing module in python to launch few processes in parallel. These processes are independent of each other. They generate their own output and write out the results in different files. Each process calls an external tool using the subprocess.call method.
It was working fine until I discovered an issue in the external tool where due to some error condition it goes into a 'prompt' mode and waits for the user input. Now in my python script I use the join method to wait till all the processes finish their tasks. This is causing the whole thing to wait for this erroneous subprocess call. I can put a timeout for each of the process but I do not know in advance how long each one is going to run and hence this option is ruled out.
How do I figure out if any child process is waiting for an user input and how do I send an 'exit' command to it? Any pointers or suggestions to relevant modules in python will be really appreciated.
My code here:
import subprocess
import sys
import os
import multiprocessing
def write_script(fname,e):
f = open(fname,'w')
f.write("Some useful cammnd calling external tool")
f.close()
subprocess.call(['chmod','+x',os.path.abspath(fname)])
return os.path.abspath(fname)
def run_use(mname,script):
print "ssh "+mname+" "+script
subprocess.call(['ssh',mname,script])
if __name__ == '__main__':
dict1 = {}
dict['mod1'] = ['pp1','ext2','les3','pw4']
dict['mod2'] = ['aaa','bbb','ccc','ddd']
machines = ['machine1','machine2','machine3','machine4']
log_file.write(str(dict1.keys()))
for key in dict1.keys():
arr = []
for mod in dict1[key]:
d = {}
arr.append(mod)
if ((mod == dict1[key][-1]) | (len(arr)%4 == 0)):
for i in range(0,len(arr)):
e = arr.pop()
script = write_script(e+"_temp.sh",e)
d[i] = multiprocessing.Process(target=run_use,args=(machines[i],script,))
d[i].daemon = True
for pp in d:
d[pp].start()
for pp in d:
d[pp].join()
Since you're writing a shell script to run your subcommands, can you simply tell them to read input from /dev/null?
#!/bin/bash
# ...
my_other_command -a -b arg1 arg2 < /dev/null
# ...
This may stop them blocking on input and is a really simple solution. If this doesn't work for you, read on for some other options.
The subprocess.call() function is simply shorthand for constructing a subprocess.Popen instance and then calling the wait() method on it. So, your spare processes could instead create their own subprocess.Popen instances and poll them with poll() method on the object instead of wait() (in a loop with a suitable delay). This leaves them free to remain in communication with the main process so you can, for example, allow the main process to tell the child process to terminate the Popen instance with the terminate() or kill() methods and then itself exit.
So, the question is how does the child process tell whether the subprocess is awaiting user input, and that's a trickier question. I would say perhaps the easiest approach is to monitor the output of the subprocess and search for the user input prompt, assuming that it always uses some string that you can look for. Alternatively, if the subprocess is expected to generate output continually then you could simply look for any output and if a configured amount of time goes past without any output then you declare that process dead and terminate it as detailed above.
Since you're reading the output, actually you don't need poll() or wait() - the process closing its output file descriptor is good enough to know that it's terminated in this case.
Here's an example of a modified run_use() method which watches the output of the subprocess:
def run_use(mname,script):
print "ssh "+mname+" "+script
proc = subprocess.Popen(['ssh',mname,script], stdout=subprocess.PIPE)
for line in proc.stdout:
if "UserPrompt>>>" in line:
proc.terminate()
break
In this example we assume that the process either gets hung on on UserPrompt>>> (replace with the appropriate string) or it terminates naturally. If it were to get stuck in an infinite loop, for example, then your script would still not terminate - you can only really address that with an overall timeout, but you didn't seem keen to do that. Hopefully your subprocess won't misbehave in that way, however.
Finally, if you don't know in advance the prompt that will be giving from your process then your job is rather harder. Effectively what you're asking to do is monitor an external process and know when it's blocked reading on a file descriptor, and I don't believe there's a particularly clean solution to this. You could consider running a process under strace or similar, but that's quite an awful hack and I really wouldn't recommend it. Things like strace are great for manual diagnostics, but they really shouldn't be part of a production setup.
Return a value early to a calling process in Python?
Hello, I want to ask is there a way to have one script call another script, have both those scripts running at the same time, and have the child script send a value to the parent script long before that child script is done running (WITHOUT exiting that child script early)? I'm looking for a solution in Python, but any information or clues would help, thankyou.
I think one way to do this is to print the value that you want to send back to the parent script to the standard output and then have the the parent script redirect it or pick it up some how, but there must be a better solution, because what if the child script prints other things? (then the parent script has to know how to isolate that exact part of the output with something like Unix head and tail commands, and what if you don't want to use the standard output at all?)
I have searched for answers on this, but I cannot find any.
You could use multiprocessing to launch the child script from the parent script.
A mp.Queue could be used to communicate output from the child script back to the parent. Here is a simple example:
parent.py:
import multiprocessing as mp
import child
if __name__ == '__main__':
queue = mp.Queue()
proc = mp.Process(target=child.main, args=(queue,))
proc.daemon = True
# This launches the child process, calling child.main()
proc.start()
for i in range(10):
result = queue.get() # Get results from child.main
print(result)
child.py:
import time
def main(queue=None):
for i in range(10):
# do a computation
result = i
if queue:
# Put a result in the queue for the parent to get
queue.put(result)
time.sleep(.5)
if __name__=='__main__':
# We reach here only when child.py is run as a script
# (as opposed to child being imported as a module).
main()
Note that the result passed through the queue must be picklable.
It is probably best to use the multiprocessing module which is designed for exactly this purpose.
I'm sure I'm missing something simple, but when using the subprocess module, there is a very significant wait (> 10 seconds) to starting the first subprocess. The second one starts shortly after the first. Is there any way to fix this? Code below:
EDIT: To add, HWAccess (in proc.py) links a dll. Could this have anything to do with it?
EDIT2: I've boiled the test down to starting a SINGLE subprocess and it takes significantly longer to import HWAccess than if I just run proc.py directly from cmd prompt. I don't see how this has anything to do with the dll specifically if it loads fast from cmd, but not as a sub-process through test.py
test.py:
import subprocess
import os
import time
print 'STARTING'
proc0 = subprocess.Popen(['python','proc.py','0'])
proc1 = subprocess.Popen(['python','proc.py','1'])
while True:
try: pass
except KeyboardInterrupt:
os._exit(0)
except ValueError:
pass
proc.py:
print 'Process starting...'
import HWAccess
print 'HWAccess imported...'
import sys
print 'sys imported...'
import time
print 'time imported...'
print 'hi from ',sys.argv[1]
Edit: After putting the prints in, there is around 5s to reach the first 'Process starting...', the second process prints 'Process starting...' immediately afterwards. Then there is a ~30 second pause to import HWAccess (takes a matter of seconds running on an individual process), the second process then immediately prints that it too has imported HWAccess... from then on execution is fast. HWAccess links a .dll so I'm wondering if two processes trying to import HWAccess result in some sort of race condition that takes a while to negotiate.
I am not sure if this is the right track, but I remember seeing such delays when starting a process wayyy back (and not at all Python related), and it turned out they were related to some badly configured network settings on my computer. Upon subprocess start-up, it has to set up interprocess communication, and those settings might interfere.
I remember my problems were related to using a false hostname for the machine, which was not properly configured on the network - can you check to see if it is your case? If it is not a production machine, try not setting a hostname at all, leaving it as "localhost".
I am working with a cluster system over linux (www.mosix.org) that allows me to run jobs and have the system run them on different computers. Jobs are run like so:
mosrun ls &
This will naturally create the process and run it on the background, returning the process id, like so:
[1] 29199
Later it will return. I am writing a Python infrastructure that would run jobs and control them. For that I want to run jobs using the mosrun program as above, and save the process ID of the spawned process (29199 in this case). This naturally cannot be done using os.system or commands.getoutput, as the printed ID is not what the process prints to output... Any clues?
Edit:
Since the python script is only meant to initially run the script, the scripts need to run longer than the python shell. I guess it means the mosrun process cannot be the script's child process. Any suggestions?
Thanks
Use subprocess module. Popen instances have a pid attribute.
Looks like you want to ensure the child process is daemonized -- PEP 3143, which I'm pointing to, documents and points to a reference implementation for that, and points to others too.
Once your process (still running Python code) is daemonized, be it by the means offered in PEP 3143 or others, you can os.execl (or other os.exec... function) your target code -- this runs said target code in exactly the same process which we just said is daemonized, and so it keeps being daemonized, as desired.
The last step cannot use subprocess because it needs to run in the same (daemonized) process, overlaying its executable code -- exactly what os.execl and friends are for.
The first step, before daemonization, might conceivably be done via subprocess, but that's somewhat inconvenient (you need to put the daemonize-then-os.exec code in a separate .py): most commonly you'd just want to os.fork and immediately daemonize the child process.
subprocess is quite convenient as a mostly-cross-platform way to run other processes, but it can't really replace Unix's good old "fork and exec" approach for advanced uses (such as daemonization, in this case) -- which is why it's a good thing that the Python standard library also lets you do the latter via those functions in module os!-)
Thanks all for the help. Here's what I did in the end, and seems to work ok. The code uses python-daemon. Maybe something smarter should be done about transferring the process id from the child to the father, but that's the easier part.
import daemon
def run_in_background(command, tmp_dir="/tmp"):
# Decide on a temp file beforehand
warnings.filterwarnings("ignore", "tempnam is a potential security")
tmp_filename = os.tempnam(tmp_dir)
# Duplicate the process
pid = os.fork()
# If we're child, daemonize and run
if pid == 0:
with daemon.DaemonContext():
child_id = os.getpid()
file(tmp_filename,'w').write(str(child_id))
sp = command.split(' ')
os.execl(*([sp[0]]+sp))
else:
# If we're a parent, poll for the new file
n_iter = 0
while True:
if os.path.exists(tmp_filename):
child_id = int(file(tmp_filename, 'r').read().strip())
break
if n_iter == 100:
raise Exception("Cannot read process id from temp file %s" % tmp_filename)
n_iter += 1
time.sleep(0.1)
return child_id