I am using /bin/tcsh as my default shell.
However, the tcsh style command os.system('setenv VAR val') doesn't work for me. But os.system('export VAR=val') works.
So my question is how can I know the os.system() run command under which shell?
Was just reading Executing BASH from Python, then 17.1. subprocess — Subprocess management — Python v2.7.3 documentation, and I saw the executable argument; and it seems to work:
$ python
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:16:52)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print os.popen("echo $0").read()
sh
>>> import subprocess
>>> print subprocess.call("echo $0", shell=True).read()
/bin/sh
>>> print subprocess.Popen("echo $0", stdout=subprocess.PIPE, shell=True).stdout.read()
/bin/sh
>>> print subprocess.Popen("echo $0", stdout=subprocess.PIPE, shell=True, executable="/bin/bash").stdout.read()
/bin/bash
>>> print subprocess.Popen("cat <(echo TEST)", stdout=subprocess.PIPE, shell=True).stdout.read()
/bin/sh: Syntax error: "(" unexpected
>>> print subprocess.Popen("cat <(echo TEST)", stdout=subprocess.PIPE, shell=True, executable="/bin/bash").stdout.read()
TEST
Hope this helps someone,
Cheers!
These days you should be using the Subprocess module instead of os.system(). According to the documentation there, the default shell is /bin/sh. I believe that os.system() works the same way.
Edit: I should also mention that the subprocess module allows you to set the environment available to the executing process through the env parameter.
os.system() just calls the system() system call ("man 3 system"). On most *nixes this means you get /bin/sh.
Note that export VAR=val is technically not standard syntax (though bash understands it, and I think ksh does too). It will not work on systems where /bin/sh is actually the Bourne shell. On those systems you need to export and set as separate commands. (This will work with bash too.)
If your command is a shell file, and the file is executable, and the file begins with "#!", you can pick your shell.
#!/bin/zsh
Do Some Stuff
You can write this file and then execute it with subprocess.Popen(filename,shell=True) and you'll be able to use any shell you want.
Also, be sure to read this about os.system and subprocess.Popen.
Related
I am trying to background a Python Flask based service via nohup on a redhat linux server, using the subprocess library and here is the issue:
I have a python flask service at location: /home/user/service_location
Name of service: service.py
I have to start this service from another python file on the same server.
Location of the starter script: /home/user/service_regen
Name of this script: restart_services.py
After spending much time on it, here is what is inside restart_services.py now:
working_directory = '/home/user/service_location'
command = 'nohup python service.py &'
ps = subprocess.Popen([command], cwd=working_directory, shell=True, stdout=open('/dev/null', 'w'), stderr=open('logfile.log', 'a'))
stdout, stderr = ps.communicate()
print("stdout: {}, stderr: {}".format(str(stdout),str(stderr)))
But this still seems to not start the service!
I am open to any other solution/alternative also if a feasible solution in subprocess does not exist.
Also, the output of the command does not matter to me, so you don't need to provide any solution for that. Just that I want the service to be started and backgrounded via nohup using Python.
I think it could be due to the command you send in, or the absolute path.
Such operation works on my side, please try out :
import os
import subprocess
import shlex
working_directory = '/home/user/service_location'
ab_path = os.path.abspath(working_directory)
command = 'nohup python service.py &'
args = shlex.split(command) # use shlex to parse the command
ps = subprocess.Popen(args, cwd=ab_path, shell=True, stdout=open('/dev/null', 'w'), stderr=open('logfile.log', 'a'))
stdout, stderr = ps.communicate()
print("stdout: {}, stderr: {}".format(str(stdout),str(stderr)))
Okay, after putting in 5 hours into it. Here is what a senior of mine came up with which is now working!
command = 'nohup python3.6 service.py &'
The exact reasons of this working are not clear to us at the moment. Maybe it has got to do something with python distributions and PATHs in this particular server. (because, the above posted code seemed to work just fine in one of our dev servers). Strangely, both servers have the same distribution installed via the same way!
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec XX XXXX, 12:22:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
I am trying to use Python's Popen to change my working directory and execute a command.
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()
However, powershell returns "The system cannot find the path specified." The path does exist.
If I run
pg = subprocess.Popen("cd c:/mydirectory ;", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
it returns the same thing.
However, if i run this: (without the semicolon)
pg = subprocess.Popen("cd c:/mydirectory",stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
The command returns without an error. This leads me to believe that the semicolon is issue. What is the cause for this behavior and how can I get around it?
I know I can just do c:/mydirectory/runExecutable.exe --help, but I would like to know why this is happening.
UPDATE :
I have tested passing the path to powershell as the argument for Popen's executable parameter. Just powershell.exe may not be enough. To find the true absolute path of powershell, execute where.exe powershell. Then you can pass it into Popen. Note that shell is still true. It will use the default shell but pass the command to powershell.exe
powershell = C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, executable=powershell)
buff,buffErr = pg.communicate()
//It works!
In your subprocess.Popen() call, shell=True means that the platform's default shell should be used.
While the Windows world is - commendably - moving from CMD (cmd.exe) to PowerShell, Python determines what shell to invoke based on the COMSPEC environment variable, which still points to cmd.exe, even in the latest W10 update that has moved toward PowerShell in terms of what the GUI offers as the default shell.
For backward compatibility, this will not change anytime soon, and will possibly never change.
Therefore, your choices are:
Use cmd syntax, as suggested in Maurice Meyer's answer.
Do not use shell = True and invoke powershell.exe explicitly - see below.
Windows only: Redefine environment variable COMSPEC before using shell = True - see below.
A simple Python example of how to invoke the powershell binary directly, with command-line switches followed by a single string containing the PowerShell source code to execute:
import subprocess
args = 'powershell', '-noprofile', '-command', 'set-location /; $pwd'
subprocess.Popen(args)
Note that I've deliberately used powershell instead of powershell.exe, because that opens up the possibility of the command working on Unix platforms too, once PowerShell Core is released.
Windows only: An example with shell = True, after redefining environment variable COMSPEC to point to PowerShell first:
import os, subprocess
os.environ["COMSPEC"] = 'powershell'
subprocess.Popen('Set-Location /; $pwd', shell=True)
Note:
COMSPEC is only consulted on Windows; on Unix platforms, the shell executable is invariably /bin/sh
As of Windows PowerShell v5.1 / PowerShell Core v6-beta.3, invoking powershell with just -c (interpreted as -Command) still loads the profiles files by default, which can have unexpected side effects (with the explicit invocation of powershell used above, -noprofile suppresses that).
Changing the default behavior to not loading the profiles is the subject of this GitHub issue, in an effort to align PowerShell's CLI with that of POSIX-like shells.
You can concat multiple commands using '&' character instead of a semicolon. Try this:
pg = subprocess.Popen("cd c:/mydirectory & ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()
I'm trying to pass some secrets to shell commands using environment variables. The Popen seems using "single quote concept" to escape the arguments so the commands cannot get the variable value.
Python 2.7.1 (r271:86832, Mar 3 2017, 10:25:58)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux3
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.Popen(['echo','$PATH'])
$PATH
Expected output is the real $PATH values like "/bin:/sbin:/usr/bin..."
Thanks!
You need a shell to evaluate your line:
subprocess.Popen('echo $PATH', shell=True)
But if this is all you want then consider this:
print os.getenv('PATH')
Use shell=True, but be careful https://docs.python.org/3/library/subprocess.html#security-considerations
subprocess.Popen('echo $PATH', shell=True)
Using shell is dangerous if the command line is filed from external sources, so use it with caution.
Also, there is an optional parameter to decide where the shell should be when launching the command.
command = 'echo $PATH'
p = subprocess.Popen(command, shell=True, cwd='./')
p.communicate() # it would wait until the subprocess is complete.
If I execute following python code on Windows:
import subprocess
subprocess.Popen( [ 'python', 'foo' ], shell = True ).communicate()
I got error written into stdout, as expected:
python: can't open file 'foo': [Errno 2] No such file or directory
But if i execute same code on linux (ubuntu, OSX - any) I got interactive python REPL started instead of this text! Like this:
user#debian:~/Documents$ python test.py
Python 2.7.3 (default, Jab 2 2013, 16:53:07)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information
>>>
Why such strange behaviour? executing python interpreter with argument ('foo') must put it into evaluation mode on all platforms, not into REPL mode.
This is spelled out in the documentation:
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence.
On Unix with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.
(emphasis mine)
I have a shell script where certain parameters are being set like:
k.sh:
export var="value"
export val2="value2"
Then I have a python script where i am calling the shell script and want to use these enviornment variables
ex1.py:
import subprocess
import os
subprocess.call("source k.sh",shell=True)
print os.environ["var"]
But I am getting a KeyError
How can I use these shell variables in my Python script?
subprocess.call starts a shell in a new process, which calls source. There is no way to modify the environment within your process from a child process.
You could source k.sh and run a Python one-liner to print the contents of os.environ as JSON. Then use json.loads to convert that output back into a dict in your main process:
import subprocess
import os
import json
PIPE = subprocess.PIPE
output = subprocess.check_output(
". ~/tmp/k.sh && python -c 'import os, json; print(json.dumps(dict(os.environ)))'",
shell=True)
env = json.loads(output)
print(env["var"])
yields
value
If you want to set the environment and then run a Python script, well, set the environment and run the Python script using runner:
runner:
#! /bin/bash
. k.sh
exec ex1.py
and that's it.
ex1.py:
#import subprocess
import os
#subprocess.call("source k.sh",shell=True)
print os.environ["var"]
As pointed out by chepner. You'r subprocess part is runnign individually.
Working with environment variables has to be done prior to launching the python script..
For instance:
C:\Users\anton\Desktop\githubs>echo %x%
y
C:\Users\anton\Desktop\githubs>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['x']
'y'
>>>
Sourcing your environment variables prior to launching the script will traverse down however, or if you execute multiple commands to the subprocess call that would also be great after you sourced it. for instance:
import subprocess
import os
x = subprocess.call("source k.sh",shell=True, STDIN=subprocess.PIPE, STDOUT=subprocess.PIPE)
y = subprocess.call("echo $var",shell=True, STDIN=x.stdout, STDOUT=subprocess.PIPE)
Never tried that tho, as mentioned. Source it before launch.
/u/unutbu already answered this question. However I fixed couple of bugs in his code:
def run_external_script(script):
if is_windows():
command = script+" && python -c \"import os; print dict(os.environ)\""
else:
command = ". "+ script+" && python -c 'import os; print dict(os.environ)'"
output = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True).communicate()[0]
r = re.compile('{.*}')
m = r.search(output)
try:
env = eval(m.group(0))
except:
error( "Something went wrong in " + script )
error( output )
return env
There are couple of small differences:
This code works both on windows/linux
I replaced subprocess.check_output with subprocess.call, check_output requires Python 2.7
When I ran his code, there std out of the script would also got printed in the output variables. So I used a re to grab the dictionary everything between two {}, such as {'var1'=1, 'var2'='x'}.
instead of using json, I used python's eval. There is a chance of injection, so use it at your own risk. such as {; exit(1); }
If your on windows you could set an environment variable with powershell
os.system("powershell.exe [System.Environment]::SetEnvironmentVariable('key', 'value','User')")
its worth noting before your application can use the variables you will need to restart you application to allow it to read the updated variables...
then you can access with
yourValue = os.environ['key']