I am trying to source a bash script containing some environment variables in python. I followed one other thread to do it. But, there seems that one of the variable is malformed, as can be seen in the given snippet.
COLORTERM=gnome-terminal
mc=() { . /usr/share/mc/mc-wrapper.sh
}
_=/usr/bin/env
I am using the following code to set up the current environment.
import os
import pprint
import subprocess
command = ['bash', '-c', 'source init_env && env']
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
(key, _, value) = line.partition("=")
os.environ[key] = value
proc.communicate()
'
If I change the above code a little like putting a condition:
for line in proc.stdout:
(key, _, value) = line.partition("=")
if not value:
continue
os.environ[key] = value
then things are working but the environment is corrupted because of one missing bracket as can be seen from the snippet of environment variable that the bracket is appearing on new line. Because of this corruption, If I run some other command like
os.system("ls -l")
it gives me the following error
sh: mc: line 1: syntax error: unexpected end of file
sh: error importing function definition for `mc'
What could be the possible solutions for this problem?
Thanks alot
Probably the best way to do this is to create a separate program that writes out the environment variables in a way that is easily and unambiguously processed by your own program; then call that program instead of env. Using the standard pickle module, that separate program can be as simple as this:
import os
import sys
import pickle
pickle.dump(os.environ, sys.stdout)
which you can either save into its own .py file, or else put directly in a Bash command:
python -c 'import os, sys, pickle; pickle.dump(os.environ, sys.stdout)'
In either case, you can process its output like this:
import os
import pprint
import subprocess
import pickle
command = [
'bash',
'-c',
'source init_env && ' +
'python -c "import os, sys, pickle; ' +
'pickle.dump(os.environ, sys.stdout)"'
]
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for k, v in pickle.load(proc.stdout).iteritems():
os.environ[k] = v
proc.communicate()
Related
I am trying to execute several lines of bash in Python 3 and check the status of each line separately.
I first tried to use gestatusoutput from subprocess, but each line is run in a separated process that does not communicate with the others (for the sake of simplicity, the given MWE consists of setting a variable, but what I intend to do in my actual code is more complex than that — and I know about os.environ for this very specific example):
from subprocess import getstatusoutput as cmd
stat, out = cmd("export TEST=1")
stat, out = cmd("echo $TEST")
will therefore returns:
>>> print(out)
(0, "")
I then tried the following:
cmdline = """export TEST=1
echo $TEST"""
stat, out = cmd(cmdline)
That works but forces me to parse the output, specially if I want to check the status of the first command (if echo works, the status returns by cmd is 0 whatever happens before), that is not very robust.
I saw some things using Popen (still from subprocess) but was unable to use it efficiently.
Any help would be appreciated!
To me, you are trying to share the environment variable between two process, which is not possible.
It looks like this:
Process 1 python main.py #TEST = ""
|Process 2-->"export TEST=1" #Change Process2 env variable TEST to '1'
|Process 3-->"echo $TEST" #Print Process3 env variable TEST (get from process 1)
You can use os.environ[] to change the current environment first (Process 1 variable),Later on use the variable after fork.
Something like this
import os
import subprocess
import sys
os.environ['TEST'] = '1'
out = subprocess.check_call('echo $TEST',shell = True)
I resulted doing the following:
create a launch command wrapping subprocess.Popen to launch my bash commands, that in addition allows me either to retrieve the current environment or to pass a custom environment
create a get_env to parse the return from the previous command and get a dict of the environment
launch wrapper
import os
import subprocess as sp
def launch(cmd_, env=os.environ, get_env=False):
if get_env: cmd_ += " && printenv"
load = sp.Popen(cmd_, shell=True, stdout=sp.PIPE, stderr=sp.PIPE, env=env)
out = load.communicate()
err = load.returncode
return(err, out)
Retrieve the environment
def get_env(out, encoding='utf-8'):
lout = str(out[0], encoding).split('\n')
new_env = {}
for line in lout:
if len(line.split('=')) <= 1:
pass
else:
k = line.split("=")[0]
v = "=".join(line.split("=")[1:])
new_env[k] = v
return new_env
(This is a simple version, it may be more complicated if you have things like functions in your environment — it happens.)
Results:
I can use it as follow:
err, out = launch("export TEST=1", get_env=True)
if not err: new_env = get_env(out)
err, out = launch("echo $TEST", env=new_env)
and therefore:
>>> print(str(out[0], encoding='utf-8'))
1
I am running a python script on Centos, which has some bash commands using subprocess:
import ConfigParser
import fileinput
import sys
import subprocess
config = ConfigParser.ConfigParser()
config.readfp(open(r'config.file'))
host = config.get('section-1', 'machine_hostname')
##changing the hostname of the machine
change_hostname = "sudo hostnamectl set-hostname new-hostname"
process = subprocess.Popen(change_hostname.split(),
stdout=subprocess.PIPE)
output, error = process.communicate()
I am importing the variables from a config file
how to pass the "new-hostname" as a variable "host" to the command I am executing, which could be dynamically assigned from the config file
It seems like you just want to assemble a string, you can use the format command:
change_hostname = "sudo {} set-hostname new-hostname".format(host)
should give you what you want, if you are using a fairly recent version of python(3.6.4+ iirc) you can also do:
change_hostname = f"sudo {host} set-hostname new-hostname"
I have been looking for an answer for how to execute a java jar file through python and after looking at:
Execute .jar from Python
How can I get my python (version 2.5) script to run a jar file inside a folder instead of from command line?
How to run Python egg files directly without installing them?
I tried to do the following (both my jar and python file are in the same directory):
import os
if __name__ == "__main__":
os.system("java -jar Blender.jar")
and
import subprocess
subprocess.call(['(path)Blender.jar'])
Neither have worked. So, I was thinking that I should use Jython instead, but I think there must a be an easier way to execute jar files through python.
Do you have any idea what I may do wrong? Or, is there any other site that I study more about my problem?
I would use subprocess this way:
import subprocess
subprocess.call(['java', '-jar', 'Blender.jar'])
But, if you have a properly configured /proc/sys/fs/binfmt_misc/jar you should be able to run the jar directly, as you wrote.
So, which is exactly the error you are getting?
Please post somewhere all the output you are getting from the failed execution.
This always works for me:
from subprocess import *
def jarWrapper(*args):
process = Popen(['java', '-jar']+list(args), stdout=PIPE, stderr=PIPE)
ret = []
while process.poll() is None:
line = process.stdout.readline()
if line != '' and line.endswith('\n'):
ret.append(line[:-1])
stdout, stderr = process.communicate()
ret += stdout.split('\n')
if stderr != '':
ret += stderr.split('\n')
ret.remove('')
return ret
args = ['myJarFile.jar', 'arg1', 'arg2', 'argN'] # Any number of args to be passed to the jar file
result = jarWrapper(*args)
print result
I used the following way to execute tika jar to extract the content of a word document. It worked and I got the output also. The command I'm trying to run is "java -jar tika-app-1.24.1.jar -t 42250_EN_Upload.docx"
from subprocess import PIPE, Popen
process = Popen(['java', '-jar', 'tika-app-1.24.1.jar', '-t', '42250_EN_Upload.docx'], stdout=PIPE, stderr=PIPE)
result = process.communicate()
print(result[0].decode('utf-8'))
Here I got result as tuple, hence "result[0]". Also the string was in binary format (b-string). To convert it into normal string we need to decode with 'utf-8'.
With args: concrete example using Closure Compiler (https://developers.google.com/closure/) from python
import os
import re
src = test.js
os.execlp("java", 'blablabla', "-jar", './closure_compiler.jar', '--js', src, '--js_output_file', '{}'.format(re.sub('.js$', '.comp.js', src)))
(also see here When using os.execlp, why `python` needs `python` as argv[0])
How about using os.system() like:
os.system('java -jar blabla...')
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.
I have a python script that calls a shell scrips, that in turn calls a .exe called iv4_console. I need to print the stdout of iv4_console for debugging purposes. I used this:
Python:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["c.sh", var], shell=True, stdout=subprocess.PIPE)
output = ''
for line in iter(proc.stdout.readline, ""):
print line
output += line
Shell:
start_dir=$PWD
release=$1
echo Release inside shell: $release
echo Directory: $start_dir
cd $start_dir
cd ../../iv_system4/ports/visualC12/Debug
echo Debug dir: $PWD
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.xml
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.xml
exit
But this didn't work, it prints nothing. What do you think?
See my comment, best approach (i.m.o) would be to just use python only.
However, in answer of your question, try:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["/bin/bash", "/full/path/to/c.sh"], stdout=subprocess.PIPE)
# Best to always avoid shell=True because of security vulnerabilities.
proc.wait() # To make sure the shell script does not continue running indefinitely in the background
output, errors = proc.communicate()
print(output.decode())
# Since subprocess.communicate() returns a bytes-string, you can use .decode() to print the actual output as a string.
You can use
import subprocess
subprocess.call(['./c.sh'])
to call the shell script in python file
or
import subprocess
import shlex
subprocess.call(shlex.split('./c.sh var'))
This question already has answers here:
Running Bash commands in Python
(11 answers)
Closed 9 months ago.
I read this somewhere a while ago but cant seem to find it. I am trying to find a command that will execute commands in the terminal and then output the result.
For example: the script will be:
command 'ls -l'
It will out the result of running that command in the terminal
There are several ways to do this:
A simple way is using the os module:
import os
os.system("ls -l")
More complex things can be achieved with the subprocess module:
for example:
import subprocess
test = subprocess.Popen(["ping","-W","2","-c", "1", "192.168.1.70"], stdout=subprocess.PIPE)
output = test.communicate()[0]
I prefer usage of subprocess module:
from subprocess import call
call(["ls", "-l"])
Reason is that if you want to pass some variable in the script this gives very easy way for example take the following part of the code
abc = a.c
call(["vim", abc])
import os
os.system("echo 'hello world'")
This should work. I do not know how to print the output into the python Shell.
Custom standard input for python subprocess
In fact any question on subprocess will be a good read
https://stackoverflow.com/questions/tagged/subprocess
for python3 use subprocess
import subprocess
s = subprocess.getstatusoutput(f'ps -ef | grep python3')
print(s)
You can also check for errors:
import subprocess
s = subprocess.getstatusoutput('ls')
if s[0] == 0:
print(s[1])
else:
print('Custom Error {}'.format(s[1]))
# >>> Applications
# >>> Desktop
# >>> Documents
# >>> Downloads
# >>> Library
# >>> Movies
# >>> Music
# >>> Pictures
import subprocess
s = subprocess.getstatusoutput('lr')
if s[0] == 0:
print(s[1])
else:
print('Custom Error: {}'.format(s[1]))
# >>> Custom Error: /bin/sh: lr: command not found
You should also look into commands.getstatusoutput
This returns a tuple of length 2..
The first is the return integer (0 - when the commands is successful)
second is the whole output as will be shown in the terminal.
For ls
import commands
s = commands.getstatusoutput('ls')
print s
>> (0, 'file_1\nfile_2\nfile_3')
s[1].split("\n")
>> ['file_1', 'file_2', 'file_3']
In python3 the standard way is to use subprocess.run
res = subprocess.run(['ls', '-l'], capture_output=True)
print(res.stdout)
The os.popen() is pretty simply to use, but it has been deprecated since Python 2.6.
You should use the subprocess module instead.
Read here: reading a os.popen(command) into a string
Jupyter
In a jupyter notebook you can use the magic function !
!echo "execute a command"
files = !ls -a /data/dir/ #get the output into a variable
ipython
To execute this as a .py script you would need to use ipython
files = get_ipython().getoutput('ls -a /data/dir/')
execute script
$ ipython my_script.py
You could import the 'os' module and use it like this :
import os
os.system('#DesiredAction')
Running: subprocess.run
Output: subprocess.PIPE
Error: raise RuntimeError
#! /usr/bin/env python3
import subprocess
def runCommand (command):
output=subprocess.run(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
if output.returncode != 0:
raise RuntimeError(
output.stderr.decode("utf-8"))
return output
output = runCommand ([command, arguments])
print (output.stdout.decode("utf-8"))