I am working on a project with Python in which I am supposed to execute shell script with Python.
I have written a simple program from which I am able to execute shell script with my python code. But now I need to pass certain parameters from my Python code to the shell script and then print out those parameters by executing the shell script.
For the simplicity sake, Currently I am executing shell script which will print out Hello World but now I want to pass hostname and pp and sp values to my shell script and then print out those values from the shell script when it is getting execute by Python client.
#!/usr/bin/python
import subprocess
import json
import socket
hostname = socket.gethostname()
jsonData = '{"desc": "some information about the host", "pp": [0,3,5,7,9], "sp": [1,2,4,6,8]}'
jj = json.loads(jsonData)
print jj['pp'] # printing it from Python program for now
print jj['sp'] # printing it from Python program for now
print hostname # printing it from Python program for now
# pass the above values to my shell script
jsonStr = '{"script":"#!/bin/bash\\necho Hello World\\n"}'
j = json.loads(jsonStr)
print "start"
subprocess.call(j['script'], shell=True)
print "end"
In general, I want to pass, hostname, pp and sp values to my shell script as shown in jsonStr and then print out those values from the shell script itself when I run my Python code.
So it should print out like this whenever I execute my shell script in jsonStr-
start
Hello world
[0, 3, 5, 7, 9]
[1, 2, 4, 6, 8]
myhostname
end
Is this possible to do it in Python?
There are two ways to pass variables to a script: as arguments, or through the environment.
Since you're trying to execute a script as if it were a giant command line (which won't actually work—especially if you're escaping your newlines as \\n so the shell sees the whole thing as one line—but let's pretend it would), you can't pass arguments, so you will need to pass an environment.
This is trivial:
env = {}
env.update(os.environ)
env.update(jj)
subprocess.call('echo ${pp}', shell=True, env=env)
This will print out whatever was in jj[pp], and return 0.
Why? Well, in bash, ${pp} means "whatever is in the environment variable pp". And we copied every key-value pair from jj into the env environment, so the environment variable pp has whatever value was in jj[pp]. (In some cases you might want to quote things, e.g. "${pp}", but for echo there's no reason to do that, and without knowing what you're going to do in your real-life code I can't guess what you might need.)
If you actually had a script that you wanted to call, stored in a file, then of course you could pass it arguments the same way you do with any other program you run via subprocess, and inside the script you could reference them as $1, etc.
However, I don't see why you need to pass arguments to a script you're building on the fly. Just build the values into the script. While that's usually a bad idea, that's mainly because building a script on the fly is a bad idea, and you've already committed to that part for some reason. Format your strings in Python, where you have access to the full power of str.format or % (as you prefer), and the entire Python stdlib. For example:
script = 'echo {}'.format(shlex.quote(jj['pp']))
subprocess.call(script, shell=True)
Now the script doesn't have to do anything to access the value; it's hard-coded into the script.
Related
I am currently trying to utilize strace to automatically trace a programm 's system calls. To then parse and process the data obtained, I want to use a Python script.
I now wonder, how would I go about calling strace from Python?
Strace is usually called via command line and I don't know of any C library compiled from strace which I could utilize.
What is the general way to simulate an access via command line via Python?
alternatively: are there any tools similar to strace written natively in Python?
I'm thankful for any kind of help.
Nothing, as I'm clueless
You need to use the subprocess module.
It has check_output to read the output and put it in a variable, and check_call to just check the exit code.
If you want to run a shell script you can write it all in a string and set shell=True, otherwise just put the parameters as strings in a list.
import subprocess
# Single process
subprocess.check_output(['fortune', '-m', 'ciao'])
# Run it in a shell
subprocess.check_output('fortune | grep a', shell=True)
Remember that if you run stuff in a shell, if you don't escape properly and allow user data to go in your string, it's easy to make security holes. It is better to not use shell=True.
You can use commands as the following:
import commands
cmd = "strace command"
result = commands.getstatusoutput(cmd)
if result[0] == 0:
print result[1]
else:
print "Something went wrong executing your command"
result[0] contains the return code, and result[1] contains the output.
Python 2 and Python 3 (prior 3.5)
Simply execute:
subprocess.call(["strace", "command"])
Execute and return the output for processing:
output = subprocess.check_output(["strace", "command"])
Reference: https://docs.python.org/2/library/subprocess.html
Python 3.5+
output = subprocess.run(["strace", "command"], caputure_output=True)
Reference: https://docs.python.org/3.7/library/subprocess.html#subprocess.run
Running into issues executing a PowerShell script from within Python.
The Python itself is simple, but it seems to be passing in \n when invoked and errors out.
['powershell.exe -ExecutionPolicy Bypass -File', '$Username = "test";\n$Password = "password";\n$URL
This is the code in full:
import os
import subprocess
import urllib2
fetch = urllib2.urlopen('https://raw.githubusercontent.com/test')
script = fetch.read()
command = ['powershell.exe -ExecutionPolicy Bypass -File', script]
print command #<--- this is where I see the \n.
#\n does not appear when I simply 'print script'
So I have two questions:
How do I correctly store the script as a variable without writing to disk while avoiding \n?
What is the correct way to invoke PowerShell from within Python so that it would run the script stored in $script?
How do I correctly store the script as a variable without writing to disk while avoiding \n?
This question is essentially a duplicate of this one. With your example it would be okay to simply remove the newlines. A safer option would be to replace them with semicolons.
script = fetch.read().replace('\n', ';')
What is the correct way to invoke PowerShell from within Python so that it would run the script stored in $script?
Your command must be passed as an array. Also you cannot run a sequence of PowerShell statements via the -File parameter. Use -Command instead:
rc = subprocess.call(['powershell.exe', '-ExecutionPolicy', 'Bypass', '-Command', script])
I believe this is happening because you are opening up PowerShell and it is automatically formatting it a specific way.
You could possibly do a for loop that goes through the command output and print without a /n.
I am invoking the bash script from python script.
I want the bash script to add an element to dictionary "d" in the python script
abc3.sh:
#!/bin/bash
rank=1
echo "plugin"
function reg()
{
if [ "$1" == "what" ]; then
python -c 'from framework import data;data(rank)'
echo "iamin"
else
plugin
fi
}
plugin()
{
echo "i am plugin one"
}
reg $1
python file:
import sys,os,subprocess
from collections import *
subprocess.call(["./abc3.sh what"],shell=True,executable='/bin/bash')
def data(rank,check):
d[rank]["CHECK"]=check
print d[1]["CHECK"]
If I understand correctly, you have a python script that runs a shell script, that in turn runs a new python script. And you'd want the second Python script to update a dictionnary in the first script. That will not work like that.
When you run your first python script, it will create a new python process, which will interpret each instruction from your source script.
When it reaches the instruction subprocess.call(["./abc3.sh what"],shell=True,executable='/bin/bash'), it will spawn a new shell (bash) process which will in turn interpret your shell script.
When the shell script reaches python -c <commands>, it invokes a new python process. This process is independant from the initial python process (even if you run the same script file).
Because each of theses scripts will run in a different process, they don't have access to each other data (the OS makes sure that each process is independant from each other, excepted for specific inter-process communications methods).
What you need to do: use some kind of interprocess mechanism, so that the initial python script gets data from the shell script. You may for example read data from the shell standard output, using https://docs.python.org/3/library/subprocess.html#subprocess.check_output
Let's suppose that you have a shell plugin that echoes the value:
echo $1 12
The mockup python script looks like (I'm on windows/MSYS2 BTW, hence the strange paths for a Linux user):
import subprocess
p = subprocess.Popen(args=[r'C:\msys64\usr\bin\sh.exe',"-c","C:/users/jotd/myplugin.sh myarg"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
o,e= p.communicate()
p.wait()
if len(e):
print("Warning: error found: "+e.decode())
result = o.strip()
d=dict()
d["TEST"] = result
print(d)
it prints the dictionary, proving that argument has been passed to the shell, and went back processed.
Note that stderr has been filtered out to avoid been mixed up with the results, but is printed to the console if occurs.
{'TEST': b'myarg 12'}
I am trying to achive following:
I am having a python script which is calling a shell script and passing a parameter as well. Shell script creates a tar.gz file using that parameter passed in some location.
Now shell script should pass the name and location of the tar.gz so created. Python script uses that to form a JSON and pass to some other code.
Along with this I want to add some check to make sure if tar.gz is generated then only value is returned to python otherwise not.
Here is the code:
Python script:
#!/usr/bin/python
import json
import subprocess
json_data='{"name": "StackOverflow", "uid": "8fa36334-ce51"}'
data = json.loads(json_data)
for keys,values in data.items():
print(keys)
print(values)
UID = data.get('uid')
rc = subprocess.check_output(["/home/cyc/Cyc-
Repo/cyc_core/cyc_platform/src/package/cyc_bsc/scripts/test.sh",
UID])
print rc
if rc != 0:
print "failed for passed in uid"
data_op = {}
data_op['pathTOCompressfile'] = 'value_should_be_return_from_shell'
data_op['status'] = 'OK'
json_data_op = json.dumps(data_op)
print json_data_op
shell script:
#!/bin/bash
if [ "$1" != "" ]; then
uid=$1
echo "Positional parameter 1 contains something $1"
else
echo "Positional parameter 1 is empty"
fi
LOG_TMP="tmp_log_file_location"
log_location="log_file_location"
filename="${log_location}/$uid.tar.gz"
echo $filename
tar -zcvf $filename -C $LOG_TMP/dc .
This is what i am not able to understand:
How to pass back the value of variable "filename" back to python script if tar -zcvf command is successful.
In python script how can i verify take value of filename and create JSON using that
In case value cannot be generated STATUS becomes fail in JSON ( within python ) so capture that as well.
Your shell script writes the name of the generated file to standard output, so your question boils down to how to catch stdandard output of a subprocess started from Python. This has been ansered here. BTW, when asking questions about Python, it would be a good idea to always specify the Python version you are using.
In your case however, I would redesign your shell script a bit:
Your script outputs not only the generated filename, but also messages about the "positional parameter", and this means that you would have to fiddle them apart in your script, whether it is an message or a valid output. You could send the messages to standard error, to keep them apart.
BTW, if there is no positional parameter, the generated file name is just .tar.gz. Is this really what you want to have? Wouldn't it better to do a exit 1, if there is no parameter?
I am using Centos 7.0 and PyDEv in Eclipse. I am trying to pass the variable in Python into c shell script. But I am getting error:
This is my Python script named raw2waveconvert.py
num = 10
print(num)
import subprocess
subprocess.call(["csh", "./test1.csh"])
Output/Error when I run the Python script:
10
num: Undefined variable.
The file test1.csh contains:
#!/bin/csh
set nvar=`/home/nishant/workspace/codec_implement/src/NTTool/raw2waveconvert.py $num`
echo $nvar
Okey, so apparently it's not so easy to find a nice and clear duplicate. This is how it's usually done. You either pass the value as an argument to the script, or via an environmental variable.
The following example shows both ways in action. Of course you can drop whatever you don't like.
import subprocess
import shlex
var = "test"
env_var = "test2"
script = "./var.sh"
#prepare a command (append variable to the scriptname)
command = "{} {}".format(script, var)
#prepare environment variables
environment = {"test_var" : env_var}
#Note: shlex.split splits a textual command into a list suited for subprocess.call
subprocess.call( shlex.split(command), env = environment )
This is corresponding bash script, but from what I've read addressing command line variables is the same, so it should work for both bash and csh set as default shells.
var.sh:
#!/bin/sh
echo "I was called with a command line argument '$1'"
echo "Value of enviormental variable test_var is '$test_var'"
Test:
luk32$ python3 subproc.py
I was called with a command line argument 'test'
Value of enviormental variable test_var is 'test2'
Please note that the python interpreter needs to have appropriate access to the called script. In this case var.sh needs to be executable for the user luk32. Otherwise, you will get Permission denied error.
I also urge to read docs on subprocess. Many other materials use shell=True, I won't discuss it, but I dislike and discourage it. The presented examples should work and be safe.
subprocess.call(..., env=os.environ + {'num': num})
The only way to do what you want here is to export/pass the variable value through the shell environment. Which requires using the env={} dictionary argument.
But it is more likely that what you should do is pass arguments to your script instead of assuming pre-existing variables. Then you would stick num in the array argument to subprocess.call (probably better to use check_call unless you know the script is supposed to fail) and then use $1/etc. as normal.