I am trying to mirror the following shell command using subprocess.Popen():
echo "SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient -U PortManUser -P plum45\\torts -q
It currently looks like:
stdout = subprocess.Popen(["echo", "\"SELECT", "employeeid", "FROM", "Users", "WHERE", "samaccountname=${1};\"", "|", "bsqldb", "arg1etc"], stdout=subprocess.PIPE)
for line in stdout.stdout.readlines():
print line
It seems that this is wrong, it returns the following standard out:
"SELECT employeeid FROM Users WHERE samaccountname=${1};" | bsqldb arg1etc
Does anyone know where my syntax for subprocess.Popen() has gone wrong?
The problem is that you're trying to run a shell command without the shell. What happens is that you're passing all of those strings—including "|" and everything after is—as arguments to the echo command.
Just add shell=True to your call to fix that.
However, you almost definitely want to pass the command line as a string, instead of trying to guess at the list that will be joined back up into the string to pass to the shell.
Or, even better, don't use the shell, and instead pipe within Python. The docs have a nice section about Replacing shell pipeline (and all kinds of other things) with subprocess code.
But in your case, the thing you're trying to pipe is just echo, which is quite silly, since you already have exactly what echo would return, and can just feed it as the input to the second program.
Also, I'm not sure what you expect that ${1} to get filled in with. Presumably you're porting a shell script that took some arguments on the command line; your Python script may have the same thing in sys.argv[1], but without knowing more about what you're doing, that's little more than a guess.
The analog of echo some string | command arg1 arg2 shell pipeline in Python is:
from subprocess import Popen, PIPE
p = Popen(["command", "arg1", "arg2"], stdin=PIPE)
p.communicate("some string")
In your case, you could write it as:
import shlex
import sys
from subprocess import Popen, PIPE
cmd = shlex.split("bsqldb -S mdw2k8sqlp02.dow.com -D PhoneBookClient "
"-U PortManUser -P plum45\\torts -q")
sql = """SELECT employeeid FROM Users
WHERE samaccountname={name};""".format(name=sql_escape(sys.argv[1]))
p = Popen(cmd, stdin=PIPE)
p.communicate(input=sql)
sys.exit(p.returncode)
Related
I am facing difficulties calling a command line from my script.I run the script but I don't get any result. Through this command line in my script I want to run a tool which produces a folder that has the output files for each line.The inputpath is already defined. Can you please help me?
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
os.system(cmd)
You really want to use the Python standard library module subprocess. Using functions from that module, you can construct you command line as a list of strings, and each would be processed as one file name, option or value. This bypasses the shell's escaping, and eliminates the need to massage you script arguments before calling.
Besides, your code would not work, because the body block of the for statement is not indented. Python would simply not accept this code (could be you pasted into the questiong without the proper indentations).
as mentioned before, executing command vias: os.system(command) is not recomended. please use subprocess (read in python docs about this modulesubprocess_module_docs). see the code here:
for command in input_file:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
# use this if you want to communicate with child process
# p = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()
# --- do the rest
I usually do like this for static command
from subprocess import check_output
def sh(command):
return check_output(command, shell=True, universal_newlines=True)
output = sh('echo hello world | sed s/h/H/')
BUT THIS IS NOT SAFE!!! It's vunerable to shell injection you should do
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
output = sh('echo hello world')
The difference is subtle but important. shell=True will create a new shell, so pipes, etc will work. I use this when I have a big command line with pipes and that is static, I mean, it do not depend on user input. This is because this variant is vunerable to shell injection, a user can input something; rm -rf / and it will run.
The second variant only accepts one command, it will not spawn a shell, instead it will run the command directly. So no pipes and such shell things will work, and is safer.
universal_newlines=True is for getting output as string instead of bytes. Use it for text output, if you need binary output just ommit it. The default is false.
So here is the full example
from subprocess import check_output
from shlex import split
def sh(command):
return check_output(split(command), universal_newlines=True)
for line in inputFile:
cmd = 'python3 CRISPRcasIdentifier.py -f %s/%s.fasta -o %s/%s.csv -st dna -co %s/'%(inputpath,line.strip(),outputfolder,line.strip(),outputfolder)
sh(cmd)
Ps: I didn't test this
I'm working on a windows machine and I want to set a variable in the shell and want to use it with another shell command, like:
set variable = abc
echo %variable%
I know that I could do this using os.system(com1 && com2) but I also know, that this is considered 'bad style' and it should be possible by using the subprocess module, but I don't get how.
Here is what I got so far:
proc = Popen('set variable=abc', shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
proc.communicate(input=b'echo %variable%)
But neither line seems to work, both commands don't get executed. Also, if I type in nonexisting commands, I don't get an error. How is the proper way to do it?
Popen can only execute one command or shell script. You can simply provide the whole shell script as single argument using ; to separate the different commands:
proc = Popen('set variable=abc;echo %variable%', shell=True)
Or you can actually just use a multiline string:
>>> from subprocess import call
>>> call('''echo 1
... echo 2
... ''', shell=True)
1
2
0
The final 0 is the return-code of the process
The communicate method is used to write to the stdin of the process. In your case the process immediately ends after running set variable and so the call to communicate doesn't really do anything.
You could spawn a shell and then use communicate to write the commands:
>>> proc = Popen(['sh'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
>>> proc.communicate('echo 1; echo 2\n')
('1\n2\n', '')
Note that communicate also closes the streams when it is done, so you cannot call it mulitple times. If you want an interactive session you hvae to write directly to proc.stdin and read from proc.stdout.
By the way: you can specify an env parameter to Popen so depending on the circumstances you may want to do this instead:
proc = Popen(['echo', '%variable%'], env={'variable': 'abc'})
Obviously this is going to use the echo executable and not shell built-in but it avoids using shell=True.
I would like to run a subprocess from Python. Inside the command string are several subcommands with backticks:
subprocess = subprocess.Popen(["echo", "COMMAND [`date +%%s`] SCHEDULE_HOST_DOWNTIME;%s;`date +%%s`;`date -d 'now + %d sec' +%%s`;1;;;%s;Downtime comment" % (hostname, 300, username)], stdout=subprocess.PIPE)
Though the date commands in the backticks are not executed. The stdout of this command is:
COMMAND [`date +%s`] SCHEDULE_HOST_DOWNTIME;example.com;`date +%s`;`date -d 'now + 300 sec' +%s`;1;;;my-username;Downtime comment
I also tried to use $(date +%s) instead of backticks and explicitly sent it to bash via subprocess.Popen(["/bin/bash", "-c", "echo", "..."], with the same result.
How can this be solved? I know I can of course use Pythons datetime module in this specific case. But I want to know why this is not working and how to solve it without tearing apart the command. While I'm here able to run the timestamp calculation easily in Python, in other cases it might be more complicated, where I would be forced to run several subprocesses which gets quickly very ugly.
Backticks are a shell syntax feature, but you are not running your command in a shell. The subprocess module runs the command directly.
Provide one string rather than a list, and set shell=True if the shell needs to process the command as one entry:
subprocess = subprocess.Popen(
'echo "COMMAND [`date +%%s`] SCHEDULE_HOST_DOWNTIME;%s;`date +%%s`;`date -d \'now + %d sec\' +%%s`;1;;;%s;Downtime comment"' % (hostname, 300, username),
stdout=subprocess.PIPE
shell=True)
I am trying to execute a shell script(not command) from python:
main.py
-------
from subprocess import Popen
Process=Popen(['./childdir/execute.sh',str(var1),str(var2)],shell=True)
execute.sh
----------
echo $1 //does not print anything
echo $2 //does not print anything
var1 and var2 are some string that I am using as an input to shell script. Am I missing something or is there another way to do it?
Referred: How to use subprocess popen Python
The problem is with shell=True. Either remove that argument, or pass all arguments as a string, as follows:
Process=Popen('./childdir/execute.sh %s %s' % (str(var1),str(var2),), shell=True)
The shell will only pass the arguments you provide in the 1st argument of Popen to the process, as it does the interpretation of arguments itself.
See a similar question answered here. What actually happens is your shell script gets no arguments, so $1 and $2 are empty.
Popen will inherit stdout and stderr from the python script, so usually there's no need to provide the stdin= and stderr= arguments to Popen (unless you run the script with output redirection, such as >). You should do this only if you need to read the output inside the python script, and manipulate it somehow.
If all you need is to get the output (and don't mind running synchronously), I'd recommend trying check_output, as it is easier to get output than Popen:
output = subprocess.check_output(['./childdir/execute.sh',str(var1),str(var2)])
print(output)
Notice that check_output and check_call have the same rules for the shell= argument as Popen.
you actually are sending the arguments ... if your shell script wrote a file instead of printing you would see it. you need to communicate to see your printed output from the script ...
from subprocess import Popen,PIPE
Process=Popen(['./childdir/execute.sh',str(var1),str(var2)],shell=True,stdin=PIPE,stderr=PIPE)
print Process.communicate() #now you should see your output
If you want to send arguments to shellscript from python script in a simple way.. You can use python os module :
import os
os.system(' /path/shellscriptfile.sh {} {}' .format(str(var1), str(var2))
If you have more arguments.. Increase the flower braces and add the args..
In shellscript file.. This will read the arguments and u can execute the commands accordingly
Hi all I am learning python and shell script and for GUI i am using wxpython.
I have said to make a automated tool which does some operation , deb creation is also one of that.
for deb creation , command is:
./myfile -u username
I have tried os.Popen, os.system, subprocess.Popen , subprocess.call.But no use everytime "-u" wont take effect, "-u " is must. I have tried by storing "-u" in variable and the passed it but still no use.
Please suggest me the exact way or where i am doing wrong
No error message but "myfile" output shows that "-u" has not given in command.
Code is:
1. cmd = ['./myfile', '-u', 'username']
Popen(cmd,shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
2. user = "-u"
name = "username"
sbprocess.call("./myfile %s %s" %(str(user),str(name)), shell=True)
same kind using "os" command also
The first code example in your question passes incorrect command because shell=True changes the meaning of the first parameter, from the subprocess docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
..[snip]..
If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
The second code example from you question should work if the following command works in a shell:
$ /bin/sh -c './myfile -u username'
To fix your command, you could omit possibly unnecessary shell=True and use check_call():
import subprocess
subprocess.check_call(["./myfile", "-u", "username"])
Try 'bash' instead of './'. If its bash script or else use respective. Its work only if your myfile is shell script.
subprocess.call("bash myfile -u %s" % str(name)), shell=True)