We're having an issue kicking off a subprocess.Popen within an airflow operator. We're using the following code to kick off sqlcl:
import subprocess
cmd = '/usr/local/bin/sqlcl -V'
p = subprocess.Popen(
cmd, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
executable='/bin/bash')
for line in iter(p.stdout.readline, ''):
self.log.info('%s', line)
p.wait()
# we have also tried p.communicate() and p.poll() here
The snippet above works when run from ipython but hangs with no output when run from within airflow. Any suggestions?
Related
Hey i'm trying to run a shell Script with python using the Following lines:
import subprocess
shellscript = subprocess.Popen(["displaySoftware.sh"], stdin=subprocess.PIPE)
shellscript.stdin.write("yes\n")
shellscript.stdin.close()
returncode = shellscript.wait()
But when I run the Program it says that it can't find the .sh file.
Your command is missing "sh", you have to pass "shell=True" and "yes\n" has to be encoded.
Your sample code should look like this:
import subprocess
shellscript = subprocess.Popen(["sh displaySoftware.sh"], shell=True, stdin=subprocess.PIPE )
shellscript.stdin.write('yes\n'.encode("utf-8"))
shellscript.stdin.close()
returncode = shellscript.wait()
This method might be better:
import subprocess
shellscript = subprocess.Popen(["displaySoftware.sh"], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
returncode = shellscript.communicate(input='yes\n'.encode())[0]
print(returncode)
When running this on my machine the "displaySoftware.sh" script, that is in the same directory as the python script, is successfully executed.
How to execute multiple commands to a command line program(.exe) one by one using python. I need to send commands and need to read the reply on the command line for each command. All in one session since login and settings has to be done.
I tried the below code(python 3.6) but it's not working
from subprocess import Popen, PIPE
process = Popen( "cmd.exe", shell=False, universal_newlines=True,
stdin=PIPE, stdout=PIPE, stderr=PIPE )
cmd= "\"C:\temp\demo.exe\"\n"
out, err = process.communicate(cmd)
print(out)
out, err = process.communicate( 'login\n' )
print(out)
this way you can execute multiple commands
import subprocess
command_list=["C:\\temp\\demo.exe","echo hello"]
for command in command_list:
proc = subprocess.Popen(command, shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = proc.communicate()
print("{} : {}".format(command,out.decode()))
Because shell=True is not the need for cmd.exe
I cant figure out how to start the server using a python command.
s = subprocess.Popen('"D:\MC SERVER 2k19\server.jar" -jar server.jar java', stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
This code runs without error but doesn't start the server in cmd.
Thanks.
It's got to do with how you're passing your arguments.
subprocess.Popen(['java', '-jar', 'server.jar'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True, cwd='D:\MC SERVER 2k19')
If you need to start a Java program using CMD process from Python and show the window, you can use subprocess to call open another CMD terminal and run the command.
In Windows you will need to CMD-escape spaces in the path you passing to the secondary CMD process. This is done with the carrot ^
proc = subprocess.Popen(
['start', 'cmd', '/k', "D:\\MC^ SERVER^ 2k19\\server.jar",
'-jar', 'server.jar', 'java'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
Keep in mind you will NOT be able to retrieve any output from the secondary CMD process from Python.
I.e. the process will return nothing.
proc.communicate()
# returns:
(b'', b'')
I have a python program which executes subprocess.Popen, like this;
process = subprocess.Popen(stand_alone_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = process.communicate()
print "out: ", out
print "err: ", err
If my stand_alone_command will run forever, how do I get whatever stand_alone_command is throwing at STDOUT and STDERR so that I can log it.
Try reading from stdout instead of calling communicate() such as..
import subprocess
sac = ['tail', '-f', '/var/log/syslog']
process = subprocess.Popen(sac, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while 1:
line = process.stdout.readline()
if line:
print(line)
I think you'll need to set shell=False but I'm on Linux and Windows is a bit different.
Whenever i call a c code executable from Python using the method below I get a "Bad file descriptor" error. When I run the code from the command prompt it works fine. Help please!?
import subprocess
p = subprocess.call(['C:\Working\Python\celp.exe', '-o', 'ofile'], stdin=subprocess.PIPE, stderr=subprocess.STDOUT)
In [84]: %run "C:\Working\Python\test.py"
Error: : Bad file descriptor
You forgot to add the stdout flag. add stdout = subprocess.PIPE, like this:
p = subprocess.call(
['C:\Working\Python\celp.exe', '-o', 'ofile'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, # needed for the next line to be sensible
stderr=subprocess.STDOUT,
)
now, try run your script again
Try this
import subprocess
p = subprocess.call(['C:\Working\Python\celp.exe', '-o', 'ofile'], stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell = True)