We Keep Coding

How to add environment variables to the bash opened by subprocess module?

I need to use the wget in a Python script with the subprocess.call function, but it seems the "wget" command cannot be identified by the bash subprocess opened by python.
I have added the environment variable (the path where wget is):
export PATH=/usr/local/bin:$PATH
to the ~/.bashrc file and the ~/.bash_profile file on my mac and guaranteed to have sourced them.
And the python script looks like:
import subprocess as sp
cmd = 'wget'
process = sp.Popen(cmd ,stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE, shell=True ,executable='/bin/bash')
(stdoutdata, stderrdata) = process.communicate()
print stdoutdata, stderrdata
The expected output should be like
wget: missing URL
Usage: wget [OPTION]... [URL]...
But the result is always
/bin/bash: wget: command not found
Interestingly I can get the help output if I type in wget directly in a bash terminal, but it never works in the python script. How could it be?
PS:
If I change the command to
cmd = '/usr/local/bin/wget'
then it works. So I am sure I got wget installed.

You can pass an env= argument to the subprocess functions.
import os
myenv = os.environ.copy
myenv['PATH'] = '/usr/local/bin:' + myenv['PATH']
subprocess.run(..., env=myenv)
However, you probably want to avoid running a shell at all, and instead augment the PATH that Python uses to find the binary to run in the subprocess call.
import subprocess as sp
import os
os.environ['PATH'] = '/usr/local/bin:' + os.environ['PATH']
cmd = 'wget'
# use run instead of Popen
# don't needlessly use a shell
# and thus put [cmd] as a list
process = sp.run([cmd], stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE,
universal_newlines=True)
print(process.stdout, process.stderr)
Running Bash commands in Python explains the changes I made in more detail.
However, there is no good reason to use an external utility for this; Python requests does pretty everything wget does, often more naturally and with more control over what exactly it does.

Python Popen fails in compound command (PowerShell)

I am trying to use Python's Popen to change my working directory and execute a command.
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()
However, powershell returns "The system cannot find the path specified." The path does exist.
If I run
pg = subprocess.Popen("cd c:/mydirectory ;", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
it returns the same thing.
However, if i run this: (without the semicolon)
pg = subprocess.Popen("cd c:/mydirectory",stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
The command returns without an error. This leads me to believe that the semicolon is issue. What is the cause for this behavior and how can I get around it?
I know I can just do c:/mydirectory/runExecutable.exe --help, but I would like to know why this is happening.
UPDATE :
I have tested passing the path to powershell as the argument for Popen's executable parameter. Just powershell.exe may not be enough. To find the true absolute path of powershell, execute where.exe powershell. Then you can pass it into Popen. Note that shell is still true. It will use the default shell but pass the command to powershell.exe
powershell = C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, executable=powershell)
buff,buffErr = pg.communicate()
//It works!

In your subprocess.Popen() call, shell=True means that the platform's default shell should be used.
While the Windows world is - commendably - moving from CMD (cmd.exe) to PowerShell, Python determines what shell to invoke based on the COMSPEC environment variable, which still points to cmd.exe, even in the latest W10 update that has moved toward PowerShell in terms of what the GUI offers as the default shell.
For backward compatibility, this will not change anytime soon, and will possibly never change.
Therefore, your choices are:
Use cmd syntax, as suggested in Maurice Meyer's answer.
Do not use shell = True and invoke powershell.exe explicitly - see below.
Windows only: Redefine environment variable COMSPEC before using shell = True - see below.
A simple Python example of how to invoke the powershell binary directly, with command-line switches followed by a single string containing the PowerShell source code to execute:
import subprocess
args = 'powershell', '-noprofile', '-command', 'set-location /; $pwd'
subprocess.Popen(args)
Note that I've deliberately used powershell instead of powershell.exe, because that opens up the possibility of the command working on Unix platforms too, once PowerShell Core is released.
Windows only: An example with shell = True, after redefining environment variable COMSPEC to point to PowerShell first:
import os, subprocess
os.environ["COMSPEC"] = 'powershell'
subprocess.Popen('Set-Location /; $pwd', shell=True)
Note:
COMSPEC is only consulted on Windows; on Unix platforms, the shell executable is invariably /bin/sh
As of Windows PowerShell v5.1 / PowerShell Core v6-beta.3, invoking powershell with just -c (interpreted as -Command) still loads the profiles files by default, which can have unexpected side effects (with the explicit invocation of powershell used above, -noprofile suppresses that).
Changing the default behavior to not loading the profiles is the subject of this GitHub issue, in an effort to align PowerShell's CLI with that of POSIX-like shells.

You can concat multiple commands using '&' character instead of a semicolon. Try this:
pg = subprocess.Popen("cd c:/mydirectory & ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()

Console in new window

I want the commands that run a python file with console
are in an independent window
my code:
def update(self):
self.prombt("sh /usr/script/update.sh")
self.close(None)
def prombt(self, com):
self.session.open(Console,_("sTaRt ShElL cOm: %s") % (com), ["%s" % com])
it's possible?
Tank's

You can realize this using the subprocess module.
import subprocess
subprocess.call(["gnome-terminal", "-x", "sh", "/usr/script/update.sh"])
In this example I used "gnome-terminal" as my terminal emulator. On your system you may not have this emulator and you should replace it with the one you use (e.g. Konsole for KDE).
You must then also find the appropriate parameter (in this case "-x") to execute the command, when opening the emulator.

To accomplish this, you can use either subprocess or os.system().
Whichever one you use, the bash command to do so would be:
gnome-terminal -e sh /usr/script/update.sh
for subprocess:
import subprocess
subprocess.call(["gnome-terminal", "-x", "sh", "/usr/script/update.sh"])
for 'os.system()'
import os
os.system("gnome-terminal -e "sh /usr/script/update.sh"")
It is recommended you use subprocess.call() for anything more complex than simple commands as os.system() is outdated.

Calling alias Command from python script

I need to run an OpenFOAM command by automatized python script.
My python code contains the lines
subprocess.Popen(['OF23'], shell=True)
subprocess.Popen(['for i in *; do surfaceConvert $i file_path/$i.stlb; done', shell=True)
where OF23 is a shell command is defined in alias as
alias OF23='export PATH=/usr/lib64/openmpi/bin/:$PATH;export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib/:$LD_LIBRARY_PATH;source /opt/OpenFOAM/OpenFOAM-2.3.x/etc/bashrc'
This script runs the OpenFOAM command in terminal and the file_path defines the stl files which are converted to binary format
But when I run the script, I am getting 'OF23' is not defined.
How do I make my script to run the alias command and also perform the next OpenFOAM file conversion command

That's not going to work, even once you've resolved the alias problem. Each Python subprocess.Popen is run in a separate subshell, so the effects of executing OF23 won't persist to the second subprocess.Popen.
Here's a brief demo:
import subprocess
subprocess.Popen('export ATEST="Hello";echo "1 $ATEST"', shell=True)
subprocess.Popen('echo "2 $ATEST"', shell=True)
output
1 Hello
2
So whether you use the alias, or just execute the aliased commands directly, you'll need to combine your commands into one subprocess.Popen call.
Eg:
subprocess.Popen('''export PATH=/usr/lib64/openmpi/bin/:$PATH;
export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib/:$LD_LIBRARY_PATH;
source /opt/OpenFOAM/OpenFOAM-2.3.x/etc/bashrc;
for i in *;
do surfaceConvert $i file_path/$i.stlb;
done''', shell=True)
I've used a triple-quoted string so I can insert linebreaks, to make the shell commands easier to read.
Obviously, I can't test that exact command sequence on my machine, but it should work.

You need to issue shopt -s expand_aliases to activate alias expansion. From bash(1):
Aliases are not expanded when the shell is not interactive, unless the expand_aliases shell option is set using shopt [...]
If that does not help, check if the shell executed from your Python program is actually Bash (e.g. by echoing $BASH).

If your command may use bash-isms then you could pass executable parameter otherwise /bin/sh is used. To expand aliases, you could use #Michael Jaros' suggestion:
#!/usr/bin/env python
import subprocess
subprocess.check_call("""
shopt -s expand_aliases
OF23
for i in *; do surfaceConvert $i file_path/$i.stlb; done
"""], shell=True, executable='/bin/bash')
If you already have a working bash-script then just call it as is.
Though to make it more robust and maintainable, you could convert to Python parts that provide the most benefit e.g., here's how you could emulate the for-loop:
#!/usr/bin/env python
import subprocess
for entry in os.listdir():
subprocess.check_call(['/path/to/surfaceConvert', entry,
'file_path/{entry}.stlb'.format(entry)])
It allows filenames to contain shell meta-characters such as spaces.
To configure the environment for a child process, you could use Popen's env parameter e.g., env=dict(os.environ, ENVVAR='value').
It is possible to emulate source bash command in Python but you should probably leave the parts that depend on it in bash-script.

Starting gnome-terminal with arguments

Using Python , I would like to start a process in a new terminal window, because so as to show the user what is happening and since there are more than one processes involved.
I tried doing:
>>> import subprocess
>>> subprocess.Popen(['gnome-terminal'])
<subprocess.Popen object at 0xb76a49ac>
and this works as I want, a new window is opened.
But how do I pass arguments to this? Like, when the terminal starts, I want it to say, run ls. But this:
>>> subprocess.Popen(['gnome-terminal', 'ls'])
<subprocess.Popen object at 0xb76a706c>
This again works, but the ls command doesn't: a blank terminal window starts.
So my question is, how do I start the terminal window with a command specified, so that the command runs when the window opens.
PS: I am targetting only Linux.

$ gnome-terminal --help-all
...
-e, --command Execute the argument to this option inside the terminal
...
If you want the window to stay open then you'll need to run a shell or command that keeps it open afterwards.

In [5]: import subprocess
In [6]: import shlex
In [7]: subprocess.Popen(shlex.split('gnome-terminal -x bash -c "ls; read -n1"'))
Out[7]: <subprocess.Popen object at 0x9480a2c>

this is the system that I use to launch a gnome-terminal from notepad++ in WINE,
1:notepad++ command to launch
#!/usr/bin/python
#this program takes three inputs:::
#$1 is the directory to change to (in case we have path sensitive programs)
#$2 is the linux program to run
#$3+ is the command line arguments to pass the program
#
#after changing directory, it launches a gnome terminal for the new spawned linux program
#so that your windows program does not eat all the stdin and stdout (grr notepad++)
import sys
import os
import subprocess as sp
dir = sys.argv[1]
dir = sp.Popen(['winepath','-u',dir], stdin=sp.PIPE, stdout=sp.PIPE).stdout.read()[:-1]
os.chdir(os.path.normpath(os.path.realpath(dir)))
print os.getcwd()
print "running '%s'"%sys.argv[2]
cmd=['gnome-terminal','-x','run_linux_program_sub']
for arg in sys.argv[2:]:
cmd.append(os.path.normpath(os.path.realpath(sp.Popen(['winepath','-u',arg], stdin=sp.PIPE, stdout=sp.PIPE).stdout.read()[:-1])))
print cmd
p = sp.Popen(cmd, stdin=sp.PIPE, stdout=sp.PIPE)
2: run sub script, which I use to run bash after my program quits (python in this case normally)
#!/bin/sh
#$1 is program to run, $2 is argument to pass
#afterwards, run bash giving me time to read terminal, or do other things
$1 "$2"
echo "-----------------------------------------------"
bash