We Keep Coding

Popen prepending path to executable

I am using popen to run the following command on a windows vm
'tf changeset ...'
however when I run it using
commandLine = 'tf changeset /noprompt /latest /loginType:OAuth /login:.,***'
process = Popen(commandLine, shell=True, stdout=PIPE, stderr=PIPE)
I see the following being executed in the logs
'C:\Azure\Agent-1/externals/tf/tf changeset ...'
Meaning that 'C:\Azure\Agent-1/externals/tf/' has been prepended to my command. I was just expecting to see
'tf changeset ...'
Unfortunately adding the path to the execution breaks the command, is there any way to stop python from doing this?

Try passing the commandLine to Popen as a list of arguments:
commandLine = ["tf", "changeset", "/noprompt", "/latest", "/loginType:OAuth", "/login:.,***'"]
process = Popen(commandLine, stdout=PIPE, stderr=PIPE)

Python by itself does no such thing. Perhaps the shell=True is doing more than you hoped or bargained for? But we would need access to your shell's configuration to get beyond mere speculation around this.
Calling Popen on the result from Popen is obviously not well-defined; but perhaps this is just an error in your transcription of your real code?
Removing the first process =Popen( would fix this with minimal changes. As per the above, I would also remove shell=True as at least superfluous and at worst directly harmful.
commandLine = 'tf changeset /noprompt /latest /loginType:OAuth /login:.,***'
process = Popen(commandLine, stdout=PIPE, stderr=PIPE)
Like the subprocess documentation tells you, shell=True is only useful on Windows when your command is a cmd built-in.
For proper portability, you should break the command into tokens, manually or by way of shlex.split() if you are lazy or need the user to pass in a string to execute.
commandLine = ['tf ', 'changeset', '/noprompt', '/latest', '/loginType:OAuth', '/login:.,***']
process = Popen(commandLine, stdout=PIPE, stderr=PIPE)
This avoids the other dangers of shell=True and will be portable to non-Windows platforms (assuming of course that the command you are trying to run is available on the target platform).

Python script using subprocess to get PID and kill it acts weird when launched from outside its sitting directory

Thank you in advance for the time you'll give to read this question. I am learning Python and I looked up a lot before asking here, please forgive me for the newbie question.
So I created this script in python 3 using subprocess module to search for another python script's PID, while only knowing the beginning of the script's name and terminate it nicely.
Basically I run python clocks on my LCD screen through Raspberry and I2C, and I terminate the script, clear the LCD and turn it off. This "off" script code is provided below.
The issue is that when I run it from the directory it sits in with a:
python3 off.py
It works perfectly, getting parsing and terminating the PID, then turning off the LCD display.
Ideally I want to trigger it through telegram-cli because I did it in bash and it worked nicely, I find it to be a nice feature. In python it fails.
So I tested and it appears that when I try to launch it from another directory like this:
python3 ~/code/off.py
The grep subprocess returns more than the one PID it returns normally when launched from the script residing directory. For instance (with python3 -v):
kill: failed to parse argument: '25977
26044'
The second PID number is from a sub process created by the script, I can't seem to find what it is as it terminates when the script ends but fails it initial purpose.
Any help in understanding what is happening here would be really appreciated.
I came so far, as show below, from two ugly lines of bash mixed with a call to an dummy four lines python scripts, so I really feel I am getting close to a proper way of achieving my first real python script.
I tried to decompose the script line by line in the interpreter and could not reproduce the error, everything behave as expected. I only get this double PID result when running the script from an outer location.
Thank you in advance for any helpful insight on how to understand what is happening!
#!/usr/bin/env python3
import subprocess
import I2C_LCD_driver
import string
# Defining variables for searched strings and string encoding
searched_process_name = 'lcd_'
cut_grep_out_of_results = 'grep'
result_string_encoding = 'utf-8'
mylcd = I2C_LCD_driver.lcd()
LCD_NOBACKLIGHT = 0x00
run = True
def kill_script():
# Listing processes and getting the searched process
ps_process = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE)
grep_process = subprocess.Popen(["grep", "-i", searched_process_name], stdin=ps_process.stdout, stdout=subprocess.PIPE)
# The .stdout.close() lines below allow the previous process to receive a SIGPIPE if the next process exits.
ps_process.stdout.close()
# Cleaning the result until only the PID number is returned in a string
grep_cutout = subprocess.Popen(["grep", "-v", cut_grep_out_of_results], stdin=grep_process.stdout, stdout=subprocess.PIPE)
grep_process.stdout.close()
awk = subprocess.Popen(["cut", "-c", "10-14"], stdin=grep_cutout.stdout, stdout=subprocess.PIPE)
grep_cutout.stdout.close()
output = awk.communicate()[0]
clean_output = output.decode(result_string_encoding)
clean_output_no_new_line = clean_output.rstrip()
clean_output_no_quote = clean_output_no_new_line.replace("'", '')
PID = clean_output_no_quote
# Terminating the LCD script process
subprocess.Popen(["kill", "-9", PID])
while run:
kill_script()
# Cleaning and shutting off LCD screen
mylcd.lcd_clear()
mylcd.lcd_device.write_cmd(LCD_NOBACKLIGHT)
break

I found out the reason of this weird comportment. An error on my end:
I forgot I called some directories with a name including the characters string I was running grep -i against provoking the double result when running the script from outside its directory using its full path.
Turns out the script runs pretty well using subprocess.
So in the end, I renamed the scripts I wanted to terminate with disp_ rather than lcd_ and added shell=False to my subprocesses to make sure there were no risk of unwantedly sending the output to bash while the running the script.

Python subprocess.call not waiting for process to finish blender

I have a python script in blender where it has
subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
followed by many other code which depends on this shell script to finish. What happens is that it doesn't wait for it to finish, I don't know why? I even tried using Popen instead of call as shown:
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
p1.wait()
and I tried using commuincate but it still didn't work:
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True).communicate()
this shell script works great on MacOS (after changing paths) and waits when using subprocess.call(['sh', '/userA/Test/run-my-script.sh'])
but on Windows this is what happens, I run the below python script in Blender then once it gets to the subprocess line Git bash is opened and runs the shell script while blender doesn't wait for it to finish it just prints Hello in its console without waiting for the Git Bash to finish. Any help?
import bpy
import subprocess
subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
print('Hello')

You can use subprocess.call to do exactly that.
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)
Run the command described by args. Wait for command to complete, then return the returncode attribute.
Edit: I think I have a hunch on what's going on. The command works on your Mac because Macs, I believe, support Bash out of the box (at least something functionally equivalent) while on Windows it sees your attempt to run a ".sh" file and instead fires up Git Bash which I presume performs a couple forks when starting.
Because of this Python thinks that your script is done, the PID is gone.
If I were you I would do this:
Generate a unique, non-existing, absolute path in your "launching" script using the tempfile module.
When launching the script, pass the path you just made as an argument.
When the script starts, have it create a file at the path. When done, delete the file.
The launching script should watch for the creation and deletion of that file to indicate the status of the script.
Hopefully that makes sense.

You can use Popen.communicate API.
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
sStdout, sStdErr = p1.communicate()
The command
Popen.communicate(input=None, timeout=None)
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for the process to terminate.

subprocess.run will by default wait for the process to finish.

Use subprocess.Popen and Popen.wait:
process = subprocess.Popen(['D:/Test/run-my-script.sh'],shell=True, executable="/bin/bash")
process.wait()
You could also use check_call() instead of Popen.

You can use os.system, like this:
import bpy
import os
os.system("sh "+os.path.abspath('D:/Test/run-my-script.sh'))
print('Hello')

There are apparently cases when the run command fails.
This is my workaround:
def check_has_finished(pfi, interval=1, timeout=100):
if os.path.exists(pfi):
if pfi.endswith('.nii.gz'):
mustend = time.time() + timeout
while time.time() < mustend:
try:
# Command is an ad hoc one to check if the process has finished.
subprocess.check_output('command {}'.format(pfi), shell=True)
except subprocess.CalledProcessError:
print "Caught CalledProcessError"
else:
return True
time.sleep(interval)
msg = 'command {0} not working after {1} tests. \n'.format(pfi, timeout)
raise IOError(msg)
else:
return True
else:
msg = '{} does not exist!'.format(pfi)
raise IOError(msg)

A wild try, but are you running the shell as Admin while Blender as regular user or vice versa?
Long story short (very short), Windows UAC is a sort of isolated environment between admin and regular user, so random quirks like this can happen. Unfortunately I can't remember the source of this, the closest I found is this.
My problem was the exact opposite of yours, the wait() got stuck in a infinite loop because my python REPL was fired from an admin shell and wasn't able to read the state of the regular user subprocess. Reverting to normal user shell got it fixed. It's not the first time I'm bit from this UAC snafu.

Subprocess in script doesn't work, when started manually it does

I have a script that reads from an mssql database and passes the read data to a subprocess of some.exe.
The data fetching works, fine but as soon as it is supposed to start proc = subprocess.(["C:\\absolute\\path\\some.exe ", fetched_data]) proc.wait() it seems to skip it and goes on for the next "fetched_data".. I also tried to use subprocess.call(["C:\\absolute\\path\\some.exe ", fetched_data])
If I start python in the console (windows cmd) and do the exact same thing it works.
Why does calling the subprocess in the script not work and if issued manually in the console it does?
edit: The problem was that the subprocess started in the script again used another.exe, which couldn't be found by the subprocess (as the it used the python path). When started from directory where some.exe and another.exe are, the script runs fine.

fetched_data is an additional argument therefore:
proc = subprocess.call(["C:\\absolute\\path\\some.exe ", fetched_data])
It's an argument LIST not a string, what subprocess expects.

Capture mongo shell output using subprocess.Popen

I am trying the following:
#!/usr/bin/python
import os, subprocess
func = 'print("Hello World")'
x = subprocess.Popen(['mongo', '--eval', func], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, stdin=subprocess.PIPE)
print x.stdout.read()
print x.stderr.read()
But all I am getting is:
MongoDB shell version: 2.2.3
followed by two new lines. How do I capture the output of function execution?

Reading the pipes gets whatever is currently inside said pipe. Your mongo is waiting to connect to the localhost. Since it doesn't return quickly enough, your read command is not getting the results. This may be because you don't have mongo running locally, but you will run into this problem repeatedly if you don't wait for the subprocess to complete.

Also, keep in mind that subprocess.Popen, to my knowledge, doesn't block. You would probably need to make a x.wait() call if you want the function to complete before trying to grab the output.