I have a shell script where certain parameters are being set like:
k.sh:
export var="value"
export val2="value2"
Then I have a python script where i am calling the shell script and want to use these enviornment variables
ex1.py:
import subprocess
import os
subprocess.call("source k.sh",shell=True)
print os.environ["var"]
But I am getting a KeyError
How can I use these shell variables in my Python script?
subprocess.call starts a shell in a new process, which calls source. There is no way to modify the environment within your process from a child process.
You could source k.sh and run a Python one-liner to print the contents of os.environ as JSON. Then use json.loads to convert that output back into a dict in your main process:
import subprocess
import os
import json
PIPE = subprocess.PIPE
output = subprocess.check_output(
". ~/tmp/k.sh && python -c 'import os, json; print(json.dumps(dict(os.environ)))'",
shell=True)
env = json.loads(output)
print(env["var"])
yields
value
If you want to set the environment and then run a Python script, well, set the environment and run the Python script using runner:
runner:
#! /bin/bash
. k.sh
exec ex1.py
and that's it.
ex1.py:
#import subprocess
import os
#subprocess.call("source k.sh",shell=True)
print os.environ["var"]
As pointed out by chepner. You'r subprocess part is runnign individually.
Working with environment variables has to be done prior to launching the python script..
For instance:
C:\Users\anton\Desktop\githubs>echo %x%
y
C:\Users\anton\Desktop\githubs>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['x']
'y'
>>>
Sourcing your environment variables prior to launching the script will traverse down however, or if you execute multiple commands to the subprocess call that would also be great after you sourced it. for instance:
import subprocess
import os
x = subprocess.call("source k.sh",shell=True, STDIN=subprocess.PIPE, STDOUT=subprocess.PIPE)
y = subprocess.call("echo $var",shell=True, STDIN=x.stdout, STDOUT=subprocess.PIPE)
Never tried that tho, as mentioned. Source it before launch.
/u/unutbu already answered this question. However I fixed couple of bugs in his code:
def run_external_script(script):
if is_windows():
command = script+" && python -c \"import os; print dict(os.environ)\""
else:
command = ". "+ script+" && python -c 'import os; print dict(os.environ)'"
output = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True).communicate()[0]
r = re.compile('{.*}')
m = r.search(output)
try:
env = eval(m.group(0))
except:
error( "Something went wrong in " + script )
error( output )
return env
There are couple of small differences:
This code works both on windows/linux
I replaced subprocess.check_output with subprocess.call, check_output requires Python 2.7
When I ran his code, there std out of the script would also got printed in the output variables. So I used a re to grab the dictionary everything between two {}, such as {'var1'=1, 'var2'='x'}.
instead of using json, I used python's eval. There is a chance of injection, so use it at your own risk. such as {; exit(1); }
If your on windows you could set an environment variable with powershell
os.system("powershell.exe [System.Environment]::SetEnvironmentVariable('key', 'value','User')")
its worth noting before your application can use the variables you will need to restart you application to allow it to read the updated variables...
then you can access with
yourValue = os.environ['key']
Related
I have a simple python script which I am using to automate updates to a dhcp config file.
The Idea is that it puts the new config file in the dhcpd directory runs a check and if that returns ok it can restart the service. My code looks like this:
syslog.syslog(syslog.LOG_INFO, 'INFO: file copied to /etc/dhcp/conf.d')
return_code = subprocess.call(['dhcpd -t -cf /etc/dhcp/dhcpd.conf'], shell=True)
if return_code != 0:
print('dhcp config test failed, exiting script')
syslog.syslog(syslog.LOG_ERR, 'ERROR: dhcp config test failed, exiting script')
sys.exit()
else:
print('dhcp config test passed restarting service')
syslog.syslog(syslog.LOG_INFO, 'INFO: config check passed, restarting service')
return_code = subprocess.call(['service', conf['service_name'], 'restart'])
if return_code != 0:
print('dhcpd service failed to restart')
syslog.syslog(syslog.LOG_ERR, 'ERROR: dhcpd service failed to restart')
else:
print('dhcpd service restarted')
syslog.syslog(syslog.LOG_INFO, 'INFO: service restarted')
email_results()
This script is kicked off by a cron job, when it runs it always fails at this bit:
print('dhcp config test failed, exiting script')
If I run the script manually it always works fine and continues to the end as expected.
If I open the python shell and run the important commands by hand it seems to work fine:
python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> return_code = subprocess.call(['dhcpd -t -cf /etc/dhcp/dhcpd.conf'], shell=True)
Internet Systems Consortium DHCP Server 4.3.3
Copyright 2004-2015 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Config file: /etc/dhcp/dhcpd.conf
Database file: /var/lib/dhcp/dhcpd.leases
PID file: /var/run/dhcpd.pid
>>> print(return_code)
0
I have tried using "shell=True" and also tried without.
I have also tried subprocess.check_call with the same results.
Where am I going wrong here?
Use absolute paths instead of just command names like dhcpd in your script.
Try if your script still works when you call it after setting an empty PATH.
The arguments to subprocess should be either an array of strings or a single string. Passing in an array of a single string is an error, though it might happen to work on some platforms which are fundamentally broken anyway.
You want
return_code = subprocess.call(['dhcpd', '-t', '-cf', '/etc/dhcp/dhcpd.conf']) # shell=False implicitly
or
return_code = subprocess.call('dhcpd -t -cf /etc/dhcp/dhcpd.conf', shell=True)
but really, you should avoid shell=True whenever possible; see also Actual meaning of 'shell=True' in subprocess
And of course, if dhcpd is not in the PATH that you get from cron, you want to update the PATH correspondingly, or use an explicit hard-coded path like /usr/sbin/dhcpd (I generally recommend the former).
I need to use the wget in a Python script with the subprocess.call function, but it seems the "wget" command cannot be identified by the bash subprocess opened by python.
I have added the environment variable (the path where wget is):
export PATH=/usr/local/bin:$PATH
to the ~/.bashrc file and the ~/.bash_profile file on my mac and guaranteed to have sourced them.
And the python script looks like:
import subprocess as sp
cmd = 'wget'
process = sp.Popen(cmd ,stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE, shell=True ,executable='/bin/bash')
(stdoutdata, stderrdata) = process.communicate()
print stdoutdata, stderrdata
The expected output should be like
wget: missing URL
Usage: wget [OPTION]... [URL]...
But the result is always
/bin/bash: wget: command not found
Interestingly I can get the help output if I type in wget directly in a bash terminal, but it never works in the python script. How could it be?
PS:
If I change the command to
cmd = '/usr/local/bin/wget'
then it works. So I am sure I got wget installed.
You can pass an env= argument to the subprocess functions.
import os
myenv = os.environ.copy
myenv['PATH'] = '/usr/local/bin:' + myenv['PATH']
subprocess.run(..., env=myenv)
However, you probably want to avoid running a shell at all, and instead augment the PATH that Python uses to find the binary to run in the subprocess call.
import subprocess as sp
import os
os.environ['PATH'] = '/usr/local/bin:' + os.environ['PATH']
cmd = 'wget'
# use run instead of Popen
# don't needlessly use a shell
# and thus put [cmd] as a list
process = sp.run([cmd], stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE,
universal_newlines=True)
print(process.stdout, process.stderr)
Running Bash commands in Python explains the changes I made in more detail.
However, there is no good reason to use an external utility for this; Python requests does pretty everything wget does, often more naturally and with more control over what exactly it does.
I am using python 3.6.3 and subprocess module to run another python script
# main.py
#!/bin/env python
from subprocess import Popen,PIPE
from sys import executable
p = Popen([executable, 'test.py', 'arg1'],shell=True, stdout=PIPE)
p.wait()
print(p.stdout.read().decode())
and
# test.py
import sys
print(sys.argv)
I expect it will run and execute test.py. However, it opens an python interpreter in interactive mode!
I tested shell=False option, it works. I tested string form rather than list form of args, it works.
I am not sure if it is a bug or not.
You need to remove shell=True or change the first argument to be executable + ' test.py arg1' instead of [executable, 'test.py', 'arg1'].
As explained in the documentation, with shell = True, it will run it as /bin/sh -c python test.py arg1, which means python will be run without arguments.
I'm using sh in python 2.7.5 to call shell programs like curl and mkdir, but in PyDev plugin 2.7.5 under Eclipse 4.3.0. the following line gives an Unresolved Import error:
from sh import curl, printenv, mkdir, cat
I'm able to run the above code in a python shell. I do have the path to sh included in the Libraries pane in the Interpreter - Python window in Preferences, so I don't think that's the problem.
Try using the subprocess module to call console commands. For example:
from subprocess import call
dir_name = '/foo/bar/'
call('mkdir %s'%dir_name, shell=True)
Like Bill said, subprocess is a good choice here. I'd personally recommend using the Popen because it doesn't block, and allows you to wait for commands to finish with its communicate() method, which also returns stdout and stderr. Also, avoid using shell=True when possible. Usage:
import subprocess
testSubprocess = subprocess.Popen(['mkdir', dir_name], stdout=subprocess.PIPE)
testOut, testErr = testSubprocess.communicate()
I am using /bin/tcsh as my default shell.
However, the tcsh style command os.system('setenv VAR val') doesn't work for me. But os.system('export VAR=val') works.
So my question is how can I know the os.system() run command under which shell?
Was just reading Executing BASH from Python, then 17.1. subprocess — Subprocess management — Python v2.7.3 documentation, and I saw the executable argument; and it seems to work:
$ python
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:16:52)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print os.popen("echo $0").read()
sh
>>> import subprocess
>>> print subprocess.call("echo $0", shell=True).read()
/bin/sh
>>> print subprocess.Popen("echo $0", stdout=subprocess.PIPE, shell=True).stdout.read()
/bin/sh
>>> print subprocess.Popen("echo $0", stdout=subprocess.PIPE, shell=True, executable="/bin/bash").stdout.read()
/bin/bash
>>> print subprocess.Popen("cat <(echo TEST)", stdout=subprocess.PIPE, shell=True).stdout.read()
/bin/sh: Syntax error: "(" unexpected
>>> print subprocess.Popen("cat <(echo TEST)", stdout=subprocess.PIPE, shell=True, executable="/bin/bash").stdout.read()
TEST
Hope this helps someone,
Cheers!
These days you should be using the Subprocess module instead of os.system(). According to the documentation there, the default shell is /bin/sh. I believe that os.system() works the same way.
Edit: I should also mention that the subprocess module allows you to set the environment available to the executing process through the env parameter.
os.system() just calls the system() system call ("man 3 system"). On most *nixes this means you get /bin/sh.
Note that export VAR=val is technically not standard syntax (though bash understands it, and I think ksh does too). It will not work on systems where /bin/sh is actually the Bourne shell. On those systems you need to export and set as separate commands. (This will work with bash too.)
If your command is a shell file, and the file is executable, and the file begins with "#!", you can pick your shell.
#!/bin/zsh
Do Some Stuff
You can write this file and then execute it with subprocess.Popen(filename,shell=True) and you'll be able to use any shell you want.
Also, be sure to read this about os.system and subprocess.Popen.