We Keep Coding

subprocess.call command runs but doesn't actually do anything

I get no error message when running the following code, however it doesn't perform the required task (to be more precise, the output is an empty file).
import subprocess
chemin_in_troncon = r"C:\Users\HBT\Desktop\Run\1OUTIL_CREATION_DB\donnees_out\2-FAX0W0\FAX0W0900_2.in"
chemin_strapontin = r"C:\Users\HBT\Desktop\STRAPONTIN\strapontin9120.exe"
result900 = subprocess.call([chemin_strapontin, chemin_in_troncon], shell=True)
However, when I execute the equivalent task directly from the shell with the following command, the .exe file does its job and gives me a non-empty output :
strapontin9120.exe ../Run/1OUTIL_CREATION_DB/donnees_out/2-FAX0W0/FAX0W0900_2.in
Here, I wrote this shell command while in the directory containing "strapontin9120.exe".
Any idea what is wrong in my code?

& was unexpected when unzipping bz2 file

Hello I am trying to write a script to unzip bz2 files with 7zip and python.
First when I write
PS> & 'C:\Program Files\7-zip\7z.exe' e D:\path\example.bz2
with powershell it is working perfectly.
So I tried that with Python:
import glob
import subprocess
target = r"D:\path\*.bz2"
tab = glob.glob(target)
for i in range(len(tab)):
subprocess.call("& 'C:\\Program Files\\7-zip\\7z.exe' e %s" %tab[i], shell=True)
And I've got that error message: & was unexpected.
Does someone has an idea why ?
I am using Python 3.9.2

By using shell=True, you're opting to pass the command line to the platform-native shell, which on Windows is cmd.exe, not PowerShell, so your PowerShell command line fundamentally cannot be expected to work as-is.
If we take a step back: you don't need to involve the shell at all in your 7z.exe call, and not doing so also speeds up your operation.
By omitting shell=True, the target executable and its arguments must then be passed as the elements of an array rather than as a single command-line string.
for i in range(len(tab)):
exitCode = subprocess.call([ 'C:\\Program Files\\7-zip\\7z.exe', 'e', tab[i] ])
Note the use of exitCode = , which captures 7z.exe's exit code and therefore allows you to check for failure.
Alternatively, you could let Python raise an exception on failure automatically, by using subprocess.check_call() rather than subprocess.call()

Calling cmd and passing arguments with Python subprocess

I am trying to get a command to run within python that otherwise works fine as a bat file or in the cmd command prompt.
Python version 2.7.9.
The following works fine in the command prompt:
"C:\Program Files (x86)\Common Files\Aquatic Informatics\AQUARIUS\runreport.exe" -Report="NHC_chart_1Week_1Series" -TimeSeries="Precip Total.24hr#FK Intake Forecast" -Server="aqserver-van" -username="admin" -password="admin" -label="Battery Voltage Data" -OutputFile="C:\Aquarius\Reports\Altagas\Summary\24hr_Precip_Forecast.pdf"
This basically calls a program (runreport.exe) and passes a bunch of arguments to it. If things go well a file is created.
It seems I should be able to do the same in python with subprocess.call
I have tried many different versions of the code but none of them run the program correctly. Below is my current code:
import subprocess
run_report_program_path=r'C:\Program Files (x86)\Common Files\Aquatic Informatics\AQUARIUS\runreport.exe'
reportname="-Report=NHC_chart_1Week_1Series"
timeseries="-TimeSeries=Precip Total.24hr#FK Intake Forecast"
server="-Server=aqserver-van"
user="-username=admin"
passwrd="-password=admin"
output="-OutputFile=C:\Aquarius\Reports\Altagas\Summary\24hr_Precip_Forecast.pdf"
result=subprocess.check_output([run_report_program_path, reportname, timeseries, server, user, passwrd, output], shell= True)
The program runreport's does 'pop up' but runs to quick and doesn't output the desired result, instead I get the error below. Does anyone see the issue?
Traceback (most recent call last):
File "C:\aquarius\scripts\API_import_and_trigger_reports_py\python_scripts\call_run_reports\test2.py", line 10, in <module>
result=subprocess.check_output([run_report_program_path, reportname, timeseries, server, user, passwrd, output], shell= True)
File "C:\Python27\lib\subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['C:\\Program Files (x86)\\Common Files\\Aquatic Informatics\\AQUARIUS\\runreport.exe', '-Report=NHC_chart_1Week_1Series', '-TimeSeries=Precip Total.24hr#FK Intake Forecast', '-Server=aqserver-van', '-username=admin', '-password=admin', '-OutputFile=C:\\Aquarius\\Reports\\Altagas\\Summary\x14hr_Precip_Forecast.pdf']' returned non-zero exit status 1
Also, in the command prompt it usually returns a nice set of errors (like time series not found, or report created successfully), I also don't get these. Is it possible to get these returned within the python session?

subprocess.check_output raises an exception if your program does not return zero. Try insert an r before the output string, that maybe is why your program fails.
Also it passes the output as a return value, so you would not see any output even if your program passed.

Changing the output to
output=r'-OutputFile=C:\Aquarius\Reports\Altagas\Summary\24hr_Precip_Forecast.pdf'
Fixed the issue. Thanks you

import a python module whose location is unknown

I would like to import a function (function.py) from a given module in python (MOD.py), whose location I do not know. For it, I have performed two steps:
First step, I get the path to the directory that contains the module:
path = subprocess.check_output(['find', 'home/scripts','-iname','MOD.py','|','sed','s/\/MOD.py//g']).rstrip()
Secondly, I point at this directory to get the function from the module:
sys.path.insert(0,'{0}'.format(path))
from MOD import function
The code written is failing in the first step, particularly in the sed. Why is it not working? Is there a clearer way to do the first step? Is it necessary to do two steps, or is it possible to do it with one python instruction?
Thanks!

First note that you could not use pipe like that ! for using pipe you have to pass shell=True so instead of check_output use Popen, also your code failed in path argument of find add a / before home .
If the executed command returns a nonzero exit code, an exception is raised. you can use a try-except with subprocess.CalledProcessError to catch errors and getting the output created along with the exit code :
import subprocess
try:
ps = subprocess.Popen(['find', '/home/scripts','-iname','MOD.py','|','sed','s/\/MOD.py//g'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
path= ps.communicate()[0]
except subprocess.CalledProcessError as e:
out_bytes = e.output
code= e.returncode
in addition as a more secure method i suggest don't use shell=True instead use tow command :
ps = subprocess.Popen(['find', '/home/scripts','-iname','MOD.py'], stdout=subprocess.PIPE)
path = subprocess.check_output(['sed','s/\/MOD.py//g'], stdin=ps.stdout)

Issues with subprocess.Popen() replacing os.system()

I am trying to generate a set of files through a python script on a 48-core cluster which actually has a master-and 3 slave machines. I need to generate a set of files, run some scripts on them, collect results and then delete them. I again repeat the process- regenerate files, execute, delete etc.,
When I delete and regenerate files with the same name, I see that the slave machine complains that it cannot find the files.
I am running python script through os.system()
I learnt from this post that it is better to use subprocess.Popen() rather than os.system so that it actually waits for my script to generate my files, before proceeding with the execution. I could use os.system("pause") or time.sleep(whatever) for waiting, but I want to convert my os.systems to subprocess.popens or subprocess.calls and I am stuck here.
I ran through the python documentation and tried out subprocess.Popen('ls'), but I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
It might sound silly but how do I execute such a simple command like cd through subprocess rather than os.system('cd')?
Then, I actually want to convert the following into subprocess. How do I do it?
import os,optparse
from optparse import OptionParser
parser.add_option("-m", "--mod",dest='module', help='Enter the entity name')
parser.add_option("-f", "--folder", dest="path",help="Enter the path")
module=options.module
path=options.path
os.system('python %s/python_repeat_deckgen_remote.py -m %s' %(path,module))
I just replaced os.system with subprocess.Popen.
But it gave me a lot of complaints:
File "/usr/lib64/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

As NPE already noted, new processes don't affect the existing one (which means os.system('cd /some/where') also has no effect on the current process). In this case, though, I think you're tripping over the fact that os.system invokes a shell to interpret the command you pass in, while subprocess.Popen does not do so by default. But you can tell it to do so:
proc = subprocess.Popen(
'python %s/python_repeat_deckgen_remote.py -m %s' % (path, module),
shell = True)
status = proc.wait()
If you're invoking a shell built-in command or using shell expansions, it's necessary to invoke the shell (assuming you're not willing to simulate it):
>>> import subprocess
>>> x = subprocess.Popen('echo $$')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/local/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
>>> x = subprocess.Popen('echo $$', shell = True); x.wait()
81628
0
>>>
but you can bypass the shell—which can help with security issues—if your problem permits, by breaking up the command arguments into a list:
>>> x = subprocess.Popen(['echo', '$$']); x.wait()
$$
0
>>>
Note that this time the output here is the string $$, rather than a process ID, since this time the shell did not interpret the string.
For your original example, for instance, you might use:
proc = subprocess.Popen(['python',
os.path.join(path, 'python_repeat_deckgen_remote.py'),
'-m',
module])
which avoids issues if path and/or module contain characters special to the shell.
(You might even want to call subprocess.Popen with cwd = path and eliminate the call to os.path.join, although that depends on other things.)

I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
Each process's current working directory is independent of other processes in the system.
When you start a subprocess (using either of these mechanisms), and get it to cd into another directory, that has no effect on the parent process, i.e. on your Python script.
To change the current working directory of your script, you should use os.chdir().

I might be not answering you question directly, but for such tasks which require running subprocess, I always use plumbum.
IMO, It makes the task much simpler and more intuitive, including running on remote machines.
Using plumbum, in order to set subprocess's working directory, you can run the command in a with local.cwd(path): my_cmd() context.