We Keep Coding

A shell script initiated by Python's os.system fails to run, but the script do run when called from terminal

I have a Python3 script that needs to call a shell script with some parameters. When I call this shell script directly form the terminal - it works. The shell script call from terminal:
source $HW/scripts/gen.sh -top $TOP -proj opy_fem -clean
But when I try to call the shell script exactly the same way from Python 3 using os.system (or os.popen - same result), the shell script fails to run. Python call to the shell script:
os.system("source $HW/scripts/gen.sh -top $TOP -proj opy_fem -clean")
Get the next errors:
/project/users/alona/top_fabric_verif_env/logic/hw/scripts/gen.sh: line 18: syntax error near unexpected token `('
/project/users/alona/top_fabric_verif_env/logic/hw/scripts/gen.sh: line 18: `foreach i ( $* )'
Could you please shed light on why the same shell script fails to run from Python?
Thank you for any help

foreach is a C-shell command. csh (and derivates like tcsh) are not standard system shells in Unix/Linux.
If you need to use a specific shell, for instance the C-shell:
os.system('/bin/csh -c "put the command here"')
This will execute the /bin/csh in the standard shell, but starting two shells instead of one creates an additional overhead. A better solution is:
subprocess.run(['/bin/csh', '-c', 'put the command here'])
Note that using the shell's source ... command does not make much sense when the shell exits after the command.

Pythons subprocess check_call doesn't give the same result as the same command executed in the command line

I am using an anaconda environment both for the python code and the terminal.
When I want to execute a program in the shell (Windows CMD) with the environment activated. The program ogr2ogr returns the correct output with the given parameter. The tool ogr2ogr has been installed via a conda package.
But when I execute the my python code, the ogr2ogr returns an errors output. I thought it might be to different installations used due to usage of different environments (without my knowledge), but this is ownly a guess.
The python code goes as follows:
from pathlib import Path
from subprocess import check_call, STDOUT
...
file_path = Path(file_name)
destination = str(file_path.with_suffix(".gpkg"))
command = f"ogr2ogr -f GPKG -s_srs EPSG:25833 -t_srs EPSG:25833 {destination} GMLAS:{file_name} -oo REMOVE_UNUSED_LAYERS=YES"
check_call(command, stderr=STDOUT, shell=True)
ogr2ogr translates a file into another format. Which is also done, but when I open the file, I see, it's not done 100 % correctly.
When I copy the value of the string command and copy it to the shell and execute the command the execution is done correctly!
How can I correct the behaviour of using subprocess.check_call

How to solve 'AttributeError: 'module' object has no attribute 'run'' from subprocess.run()

I am creating a bash script which calls a python script that in turn runs other processes in bash using subprocess.run(). However, when the bash script runs the python script within it, in the line where subprocess.run is called, I get an error message:
run_metric = subprocess.run(command, shell=True, stdout = subprocess.PIPE, universal_newlines = True)
AttributeError: 'module' object has no attribute 'run'
1) I made sure I ran the script using python 3 by activating a conda environment with python=3.6, which should not bring me any problem to call subprocess.run. The interesting thing is that if I change subprocess.run() to subprocess.Popen() the script works, but I could not work out how to get run_metric.stdout properly.
2) I do not have any subprocess.py file within any directory I am working in
3) the result of print(subprocess.__file__) is showing me that python is not 3.6: /usr/lib/python2.7/subprocess.pyc
Also, I tried to use something like
from subprocess import run
and making sure in both the python script and the function I had import subprocess
The bash script is as follows:
SWC_FOLDER_PATH=$(pwd)
sudo chmod +x /media/leandroscholz/KINGSTON/Results_article/Tracing_data/run_metrics.py
echo "run /media/leandroscholz/Tracing_data/run_metrics.py ${SWC_FOLDER_PATH} /media/leandroscholz/KINGSTON/Results_article/TREEStoolbox_tree_fixed.swc"
python /media/leandroscholz/Tracing_data/run_metrics.py ${SWC_FOLDER_PATH} /media/leandroscholz/TREEStoolbox_tree_fixed.swc
And the python script I run calls a certain function that uses subprocess.run() this way (just part of the code where the problem arises):
import subprocess
import glob
import numpy as np
def compute_metrics(swc_folder_path, gt_file_path):
# first get list of files in swc_folder_path
swc_files = (glob.glob(swc_folder_path+"/*_fixed.swc"))
n_swc_files = len(swc_files)
workflow_dict = gets_workflow_dict(swc_files)
n_images = get_n_images(swc_files)
n_workflows = len(workflow_dict)
for swc in range(0,n_swc_files):
command = "java -jar /home/leandroscholz/DiademMetric.jar -G " + swc_files[swc] +" -T " + gt_file_path
run_metric = subprocess.run(command, shell=True, stdout = subprocess.PIPE, universal_newlines = True)
I am using subprocess.run within python because, in the end, I want to get a string of the run_metric.stdout after running the process in bash so I can later store it in an array and save it to a txt file.
I hope I was sufficiently clear and provided enough information.
Thanks!

After the comments received, I tested the output of print(subprocess.__file__), which showed that python being used was python2.7,
Thus, I changed the call of the python script from python script.py to python3 script.py. I've found this question, which also shows another way to call python programs from terminal.
Running Python File in Terminal

Executing PowerShell commands with Python 3.5

I have been working on an issue that requires a Python script to run via the PowerShell command line. The script should pass the command to the command line and save the output. However, I'm running into an issue where some command line arguments are not recognized.
import subprocess
try:
output = subprocess.check_output\
(["Write-Output 'Hello world'"], shell=True)
# (["dir"], shell=True)
except subprocess.CalledProcessError as e:
print(e.output)
print('^Error Output^')
If I use the current command with the check_output command, I get an error stating that:
'"Write-Output 'Hello world'"' is not recognized as an internal or external command,
operable program or batch file.
If I just use the "dir" line, the script runs just fine. I'm at odds here as to why this would be happening. This is not the exact script that I'm running, but it produces the same problem on my machine. If I just type the problem command into the command line, it would output "Hello world" onto the new line just as expected.
Any insight as to why this would be happening would be greatly appreciated. If it's of relevance, I would like to not use any sort of admin privilege workaround.

I believe this is because in Windows your default Shell is not PowerShell, you could Execute a Powershell command, calling the executable by executing Powershell with the arguments you need.
For Example
POWERSHELL_COMMAND = r'C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe'
subprocess.Popen([POWERSHELL_COMMAND,
'-ExecutionPolicy', 'Unrestricted',
'Write-Output', 'Hello World'],
stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
if powershell is not in path you could use the full path for the executable
or if it's in path you could use just POWERSHELL_COMMAND = "powershell" as command, becareful, with the backslashed windows paths, to avoid errors you could use raw strings.
To verify that you have powershell in path, you could go to the configurations and check, or you could just open a cmd and type powershell and if It works, then you could assume that powershell is in path.
From the docs:
On Windows with shell=True, the COMSPEC environment variable specifies the default shell.
So set COMSPEC=powershell allows to make shell=True use powershell as default instead of cmd

how to run python script from the console as if called from command prompt?

The python script I would use (source code here) would parse some arguments when called from the command line. However, I have no access to the Windows command prompt (cmd.exe) in my environment. Can I call the same script from within a Python console? I would rather not rewrite the script itself.

%run is a magic in IPython that runs a named file inside IPython as a program almost exactly like running that file from the shell. Quoting from %run? referring to %run file args:
This is similar to running at a system prompt python file args,
but with the advantage of giving you IPython's tracebacks, and of
loading all variables into your interactive namespace for further use
(unless -p is used, see below). (end quote)
The only downside is that the file to be run must be in the current working directory or somewhere along the PYTHONPATH. %run won't search $PATH.
%run takes several options which you can learn about from %run?. For instance: -p to run under the profiler.

If you can make system calls, you can use:
import os
os.system("importer.py arguments_go_here")

You want to spawn a new subprocess.
There's a module for that: subprocess
Examples:
Basic:
import sys
from subprocess import Popen
p = Popen(sys.executable, "C:\test.py")
Getting the subprocess's output:
import sys
from subprocess import Popen, PIPE
p = Popen(sys.executable, "C:\test.py", stdout=PIPE)
stdout = p.stdout
print stdout.read()
See the subprocess API Documentation for more details.