I have seven python scripts that all manipulate some files in my folder system based on some information on an MSSQL server. The code is written in a way that a script should just restart whenever it has finished. Additionally, the scripts should run in parallel. The order does not matter as long as they are all executed every now and again (that is, it would be bad if Script 1 (reading a file) runs endlessly while Script 7 (deleting all the files that are already read) is never executed. However, it wouldn't matter if Script 1 is run several times before Script 7 is run once).
So far, I've found a solution with PowerShell. I have 7 separate PowerShell Scripts (process1.ps1, process2.ps1, ..., process7.ps1) that all look as follows:
while($true)
{
$i++
Start-Process -NoNewWindow -Wait python F:\somewhere\something.py
Start-Sleep -Seconds 10
}
This works if I open 7 different PowerShell consoles and start one .ps1 in each like this:
& "F:\PowerShellScripts\process1.ps1"
However, opening and monitoring seven sessions every time is cumbersome. Is there a way to start all these processes in one go but still ensure that they are parallelized correctly?
You are able to parallelize command in powershell via:
Jobs
Runspaces
Workflows
For easiest use, take jobs (my recommendation for your needs). For best performance, use runspaces. I have not tried workflows yet.
A starter for jobs:
$scriptpaths = "C:\temp\1.py", "C:\temp\2.py"
foreach ($path in $scriptpaths){
start-job -ScriptBlock {
Param($path)
while($true)
{
Start-Process -NoNewWindow -Wait python $path
Start-Sleep -Seconds 10
}
} -ArgumentList $path
}
Please do read the documentations though, this code is far from ideal. Also, this code does not synchronize your scripts. If one runs faster than the others, they will desynchronize.
Related
Suppose that I run an MPI program involving 25 processes on 25 different machines. The program is initiated at one of them called the "master" with a command like
mpirun -n 25 --hostfile myhostfile.txt python helloworld.py
This is executed on Linux with some bash script and it uses mpi4py. Sometimes, in the middle of execution, I want to stop the program in all machines. I don't care if this is done graciously or not since the data I might need is already saved.
Usually, I press Ctrl + C on terminal of the "master" and I think it works as described above. Is this true? In other words, will it stop this specific MPI program in all machines?
Another method I tried is to get the PID of the process in the "master" and kill it. I am not sure about this either.
Do the above methods work as described? If no, what else do you suggest? Note that I want to avoid the use of MPI calls for that purpose like MPI_Abort that some other discussions here and here suggest.
I have written a python (python version is 3) script that runs 24/7. The way I run my script in my Windows machine is the following. I right click on the .py file, then click on "Edit with IDLE" and then "Run". The script has no issue but, due to the many instructions printed in the python shell (I use a logger), after a couple of days this python shell gets very heavy. My newbie question is the following. Is there to limit the number of rows temporarily saved in the python shell to a specific number? Or perhaps somebody has a better suggestion to run this constantly running script that prints a lot of the script steps in the shell? Please, notice how I'm not asking how to run a script 24/7, it's my understanding the best way to do it is though a VPS. My problem is that the data saved in the displayed python shell gets bigger and bigger every day, so I only wonder how to limit the data temporarily displayed/saved in it. Thanks
I have it running a python script in a loop:
#ECHO OFF
:loop
D:\Python\python.exe C:\pythonfile.py
goto loop
PAUSE
How do I make it so when I double-click the batch file, the process runs in the background and the console window isn't showing in the taskbar?
Make the process be hidden completely with windows built in vbs, then call the batch from it. You will then run the vbs file instead of the batch, which will call the batch file.
no_see.vbs
Set MyScript = CreateObject("WScript.Shell")
MyScript.Run "C:\wherever\script\is\batchfile.cmd", 0, False
This will run the batch file completely hidden.
The python script however might still call a console window, so you should consider this.
#ECHO OFF
:loop
D:\Python\pythonw.exe C:\pythonfile.pyw
goto loop
PAUSE
This will run without a console, meaning everything will be hidden.
One concern I have, you are running the .py file in a loop, no sleep nothing meaning you will spawn thousands of python instances. What exactly are you attemping? If you can give me an idea, I can perhaps help in making the solution better.
I'm not sure if there is a way to do that using batch, at least not one that I know of. However, I would recommend doing this with a scheduled task instead if you do not want it running in the taskbar.
Open the Task Scheduler and in the library, create a new task. From there, you can set conditions, triggers, and plenty of other options for how you want it to run, including automatic start at logon, etc.
Just thought of this...alternatively, you could also use the Task View of Windows 10. Open and run the batch file, and then move it to another virtual desktop.
I run Python script on Windows using IDE PyCharm.
Sometimes process is stopped. How can I configure that process will be re-run automatically after interruption?
There is also 3 ways as in this answer.
1 way create infinity batch loop
:loop
python your_script.py
goto loop
2 and 3rd way are already described here.
Also you can start python script as windows service but it much harder.
I'm facing a problem in python:
My script, at a certain point, has to run some test script written in bash, and I have to do it in parallel, and wait until they end.
I've already tried :
os.system("./script.sh &")
inside a for loop but it did not worked.
Any suggest?
Thank you!
edit
I have nt correctly explained my situation:
My phyton script resides in the home dir;
my sh scripts resides in other dirs, for instance /tests/folder1 and /tests/folder2;
Trying to use os.system implies the usage of os.chdir prior to call os.system (to avoid troubles on "no such files or directory", my .sh scripts contains some relative references), and also this method is blocking my terminal output.
Trying to use Popen and passing all the path fro home folder to my .sh lead to launch zombie processes without any responses or other.
Hope to find a solution,
Thank you guys!
Have you looked at subprocess? The convenience functions call and check_output block, but the default Popen object doesn't:
processes = []
processes.append(subprocess.Popen(['script.sh']))
processes.append(subprocess.Popen(['script2.sh']))
...
return_codes = [p.wait() for p in processes]
Can you use GNU Parallel?
ls test_scripts*.sh | parallel
Or:
parallel ::: script1.sh script2.sh ... script100.sh
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel