We Keep Coding

A shell script initiated by Python's os.system fails to run, but the script do run when called from terminal

I have a Python3 script that needs to call a shell script with some parameters. When I call this shell script directly form the terminal - it works. The shell script call from terminal:
source $HW/scripts/gen.sh -top $TOP -proj opy_fem -clean
But when I try to call the shell script exactly the same way from Python 3 using os.system (or os.popen - same result), the shell script fails to run. Python call to the shell script:
os.system("source $HW/scripts/gen.sh -top $TOP -proj opy_fem -clean")
Get the next errors:
/project/users/alona/top_fabric_verif_env/logic/hw/scripts/gen.sh: line 18: syntax error near unexpected token `('
/project/users/alona/top_fabric_verif_env/logic/hw/scripts/gen.sh: line 18: `foreach i ( $* )'
Could you please shed light on why the same shell script fails to run from Python?
Thank you for any help

foreach is a C-shell command. csh (and derivates like tcsh) are not standard system shells in Unix/Linux.
If you need to use a specific shell, for instance the C-shell:
os.system('/bin/csh -c "put the command here"')
This will execute the /bin/csh in the standard shell, but starting two shells instead of one creates an additional overhead. A better solution is:
subprocess.run(['/bin/csh', '-c', 'put the command here'])
Note that using the shell's source ... command does not make much sense when the shell exits after the command.

Run Perl file with integer arguments from a Python script

I am trying to run the Perl script rtpc.pl from a Python script from the command line with two input arguments. Normally, I run the Perl script from the command line with two arguments:
perl rtpc.pl [ARG1] [ARG2]
What I want is a separate python script that I can call from the command line that I can input [ARG1] and [ARG2] into while compiling the Perl script rtpc.pl.
Something like:
python3 pyscript.py
So far, in my python script, I am using the subprocess module, but I am a little unsure how to pass arguments for my perl script.
pipe = subprocess.Popen(["perl", "rtpc.pl"], stdout=subprocess.PIPE)
How would I input the arguments needed to equivalently run the Perl script terminal command? I should also mention that the shell that I am using is tcsh.

Add them to the list you pass to subprocess.Popen().
arg1 = 'foo'
arg2 = 'bar'
pipe = subprocess.Popen(["perl", "rtpc.pl", arg1, arg2], stdout=subprocess.PIPE)

Calling Python debugger within a Python script - ends immediately and doesn't debug

I have a Python script that I want to be able to debug from the command line. Let's say it's called simple.py, and it contains the following code:
var1 = "Hello"
var2 = "World"
print(var1, var2)
I want to debug this. Of course I can use the debugger from the command line:
python -m pdb simple.py
But I want to debug this specifically from within another python file. I've tried using subprocess to do this, by putting this code into another script, debug.py:
import subprocess
subprocess.Popen(["bash", "-ic", "python -m pdb simple.py"])
When I run it in a Bash shell, this happens:
$ python debug.py
$ > /[path]/simple.py(1)<module>()
-> var1 = "Hello"
(Pdb) n
bash: n: command not found
$
So I've typed "n", expecting the debugger to move to the next line. Instead, it seems to just go straight back to Bash, and the debugger doesn't do anything.
Any idea why this happens? Is there a way to spawn a Bash shell containing a Python debugger from within a Python script, which I can actually use to debug?

Your debug.py finish its run and you are back to the shell (try typing ls instead, and see what happens)
What you are looking for is a way to interact with the other process, you need to get the input from your stdin and pass it to the other process stdin. It can looks something like:
import subprocess
p = subprocess.Popen(["python3", "-m", "pdb", "test.py"])
while True:
cmd = input()
p.stdin.write(cmd.encode())

How can I open a .py file from another file in a separate console?

How can I open one python file from another in a separate console window. I have tried using import file, but this runs it in the same console as the original program.

You want to execute another python instance for that.
Create a list where arg[0] is the python executable path (sys.executable).
This is like running the Python script from a command prompt or bash shell.
import subprocess, sys
script_name = 'my_other_script.py'
# set arg1, arg2, etc. to match the script arguments
cmd_line[sys.executable, script_name, arg1, arg2]
subprocess.check_call(cmd_line)

How to execute a command in the terminal from a Python script?

I want to execute a command in terminal from a Python script.
./driver.exe bondville.dat
This command is getting printed in the terminal, but it is failing to execute.
Here are my steps:
echo = "echo"
command="./driver.exe"+" "+"bondville.dat"
os.system(echo + " " + command)
It should execute the command, but it's just printing it on terminal. When feeding the same thing manually it's executing. How do I do this from a script?

The echo terminal command echoes its arguments, so printing the command to the terminal is the expected result.
Are you typing echo driver.exe bondville.dat and is it running your driver.exe program?
If not, then you need to get rid of the echo in the last line of your code:
os.system(command)

You can use the subprocess.check_call module to run the command, you don't need to echo to run the command:
from subprocess import check_call
check_call(["./driver.exe", "bondville.dat"])
Which is equivalent to running ./driver.exe bondville.dat from bash.
If you wanted to get the output you would use check_outout:
from subprocess import check_output
out = check_output(["./driver.exe", "bondville.dat"])
In your own code you are basically echoing the string command not actually running the command i.e echo "./driver.exe bondville.dat" which would output ./driver.exe bondville.dat in your shell.

Try this:
import subprocess
subprocess.call("./driver.exe bondville.dat")