We Keep Coding

Running python in subprocess.call(cmd, ...) get error with message /bin/sh: -c: line 0: syntax error near unexpected token `('

My code (python3 run in virtualenv)
try:
cmd = "cd NST/experiments && python main.py eval --model models/21styles.model --content-image " + directory + "/" + filename + " --style-image " + STYLE_IMAGE_UPLOAD + "wikiart/" + choose_file() + " --output-image " + OUTPUT_IMAGE_UPLOAD + filename + " --content-size 600" " --cuda=0"
returned_value = subprocess.call(cmd, shell=True) # returns the exit code in unix
print('returned value:', returned_value)
except Exception as e:
print(e)
I got this error during run the script
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `cd NST/experiments && python main.py eval --model models/21styles.model --content-image /Users/kanel/Documents/Developments/ML/applications/static/nst/content_image/en_20191208/20.jpg --style-image /Users/kanel/Documents/Developments/ML/applications/static/nst/style_image/wikiart/facsimile-of-may-courtly-figures-on-horseback(1).jpg --output-image /Users/kanel/Documents/Developments/ML/applications/static/nst/output_image/20.jpg --content-size 600 --cuda=0'

You have unquoted strings in the command line. In general, it is nice to have any passed values/variables to system calls (or subprocess or etc) quoted by default or it is possible to get bugs like you have now.
I've a bit restructured your source code, added quoting and did it added more readability to the source code block.
Here is the code which should work for you well :)
try:
from pipes import quote as quote_path
except ImportError:
from shlex import quote as quote_path
try:
cmd = (
"cd NST/experiments && python main.py eval --model models/21styles.model "
"--content-image {} "
"--style-image {} "
"--output-image {} "
"--content-size 600 --cuda=0"
).format(
quote_path(directory + " / " + filename),
quote_path(STYLE_IMAGE_UPLOAD + " wikiart / " + choose_file()),
quote_path(OUTPUT_IMAGE_UPLOAD + filename)
)
returned_value = subprocess.call(cmd, shell=True) # returns the exit code in unix
print('returned value:', returned_value)
except Exception as e:
print(e)

subprocess.check_call gives error returned non-zero exit status 1

I have a bash script that gets called at the end of Python script. If I call the script manually with a simple Python script (test.py below) it works fine. However when called from my actual script (long.py) it fails. So long.py runs, calling rename.sh at the end, and passing it a linux directory path, source_dir. rename.sh renames a file in said path. Here's a relevant excerpt of that script:
long.py
PATH = '/var/bigbluebutton/published/presentation/'
LOGS = '/var/log/bigbluebutton/download/'
source_dir = PATH + meetingId + "/"
...
def main():
...
try:
create_slideshow(dictionary, length, slideshow, bbb_version)
ffmpeg.trim_audio_start(dictionary, length, audio, audio_trimmed)
ffmpeg.mux_slideshow_audio(slideshow, audio_trimmed, result)
serve_webcams()
# zipdir('./download/')
copy_mp4(result, source_dir + meetingId + '.mp4')
finally:
print >> sys.stderr, source_dir
#PROBLEM LINE
subprocess.check_call(["/scripts/./rename.sh", str(source_dir)])
print >> sys.stderr, "Cleaning up temp files..."
cleanup()
print >> sys.stderr, "Done"
if __name__ == "__main__":
main()
Here's the problem:
long.py uses the above line to call rename.sh:
subprocess.check_call(["/scripts/./rename.sh", str(source_dir)])
it gives the error:
subprocess.CalledProcessError: Command '['/scripts/./rename.sh', '/var/bigbluebutton/published/presentation/5b64bdbe09fdefcc3004c987f22f163ca846f1ea-1574708322765/']' returned non-zero exit status 1
The script otherwise works perfectly.
test.py, a shortened version of long.py, contains only the following two lines:
test.py
source_dir = '/var/bigbluebutton/published/presentation/5b64bdbe09fdefcc3004c987f22f163ca846f1ea-1574708322765/'
subprocess.check_call(["/scripts/./rename.sh", str(source_dir)])
It does not encounter the error when ran using python test.py.
Here's the contents of rename.sh:
rename.sh
#!/bin/bash
i=$1
a=$(grep '<name>' $i/metadata.xml | sed -e 's/<name>\(.*\)<\/name>/\1/' | tr -d ' ')
b=$(grep '<meetingName>' $i/metadata.xml | sed -e 's/<meetingName>\(.*\)<\/meetingName>/\1/' | tr -d ' ')
c=$(ls -alF $i/*.mp4 | awk '{ gsub(":", "_"); print $6"-"$7"-"$8 }')
d=$(echo $b)_$(echo $c).mp4
cp $i/*.mp4 /root/mp4s/$d
test.py and long.py are in the same location.
I'm not executing long.py manually; it gets executed by another program.
print >> sys.stderr, source_dir
confirms that the exact same value as I define explicitly in test.py is getting passed by long.py to rename.sh

Split bash script into separate commands and run them them one-by-one

I want to be able to run a bash shell script from Python (using something like subprocess.Popen), but display on my GUI the command that will be executed along with the output live as it comes in stdout and stderr. The GUI should show Input (newline) Output (newline) Input and so on. I am able to currently implement it for one-line bash commands, but I need a more sophisticated parser for multi-line commands. Here is my code which only works for simple one-line commands in bash.
test.sh:
ping -c 2 www.google.com
if [ "abc" = "ghi" ]; then
echo expression evaluated as true
else
echo expression evaluated as false
fi
My python file:
with open("test.sh", "r") as script:
for line in script:
if not line.strip().startswith('#') and not line.strip() == "":
print("Debug: Running " + line.strip() + " ...")
proc = subprocess.Popen(shlex.split(line), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
output = str(proc.stdout.readline().strip().decode()) + ""
err = str(proc.stderr.readline().strip().decode()) + ""
if output == '' and proc.poll() is not None:
print("Debug: Command completed...")
time.sleep(1)
break
if output:
# Code for updating a Gtk TextView buffer
GLib.idle_add(self.updateConsoleText, output + "\n")
if err:
# Code for updating a Gtk TextView buffer
GLib.idle_add(self.updateConsoleText, err + "\n")
As expected, it does not work with multi-line code that involves if-else, loops, heredocs, etc. I am looking for a bash parser that at least recognises when a command ends and can split a bash script into separate commands in such cases where multi-line commands are used.
Do you think you could help me find such a library/tool?

You can use the trap command.
Here a little example to demonstrate the concept:
#!/bin/bash
# redirect stderr to a logfile
exec 2>/tmp/test.log
# print commands and arguments at execution
set -x
# set trap for every step, sleep one second.
# "sleep 1" could be every bash command
trap "sleep 1" DEBUG
echo -n "Name: "; read -r name
echo -n "Age: "; read -r age
echo "$name is $age years old"
Parallel to the run of this script you can use tail -f /tmp/test.log to track the call of the commands and their parameters.

The term 'Select-String' is not recognized

I am trying to get only the PID by running this python script.
My current error:
Select string is not recognized as an internal or external command
To deal with this error, I thought that I needed to escape | by adding ^ --> But apparently it doesn't work
I added in some \ to escape ", hopefully it's correct?
cmd = "netstat -ano | findstr " + str(o)
print (cmd)
cmd += " | Select-String \"TCP\s+(.+)\:(.+)\s+(.+)\:(\d+)\s+(\w+)\s+(\d+)\" | ForEach-Object { Write-Output $_.matches[0].Groups[6].value }"
print (cmd)
pid = run_command(cmd)
The run_command method does this:
def run_command(cmd_array,os_name='posix'):
p = subprocess.Popen(cmd_array,shell=True,cwd=os.getcwd())
output,err = p.communicate()
print('output=%s'%output)
print('err=%s'%err)
return output
Expected Result
When I run solely the command in command prompt, it gives me the PID --> which is 7556 in this case.
Not too sure why it is not working for the script but working in command prompt by itself.

This question is specific to windows OS
Answer to my own question
With help from the comments, i did not use my run_command method since it was using shell = True.
shell = True refers to cmd.exe on Windows, not powershell.
The commands i have written are powershell commands.
Directly use subprocess.call to run powershell commands
Python scripts
cmd = "netstat -ano | findstr 8080"
cmd += " | Select-String \"TCP\s+(.+)\:(.+)\s+(.+)\:(\d+)\s+(\w+)\s+(\d+)\" | ForEach-Object { Write-Output $_.matches[0].Groups[6].value }"
subprocess.call(["powershell.exe", cmd])
#this does the job but the code will print extra zeros along with PID. It was not what i was looking for.
Result:
6492 (Prints out the PID Along with some additional zeros)
What worked for me - For those who are trying to get only PID and kill port using PID in python script
cmd = "for /f \"tokens=5\" %a in ('netstat -aon ^| find \":8080"
cmd += "\" ^| find \"LISTENING\"\') do taskkill /f /pid %a"
#I added in some \ to escape the "
run_command(cmd)
Result:
SUCCESS: The process with PID 2072 has been terminated

Does the ampersand need to be escaped?

I am attempting to create a function in python 2.7.8 to delete files that are no longer needed on a remote PC.
If I run it from the command line I am able to do this:
C:\>del /f /q \\RemotePC\d$\temp\testfile.txt && if exist \\RemotePC\d$\temp\testfile.txt (set errorlevel=1) else (set errorlevel=0)
From this I get the expected result:
C:\>echo %errorlevel%
0
But when I attempt to put this into my python function it does not work. The test variable does not get set.
Here is what I have in my python function:
def DelFile(self, address, del_file):
cmd_line = "del /f /q \\\\" + address + del_file + "\&\& if exist \"\\\\" + address + del_file + "\" (set errorlevel=1) else (set errorlevel=0)"
myP = subprocess.Popen(cmd_line, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
out = timeout(myP.communicate, (), {}, 120, (None, None))
result = out[0]
error = out[1]
error_code = None
error_code = myP.returncode
notes = []
return cmd_line, result, error, error_code, notes
I have verified that the address and del_file variables are formatted correctly in order to get me to the desired file. I have also verified that it deletes or does not delete the intended file based on the circumstances. But for some reason the test variable never gets set.
It is my understanding that the ampersand character needs to be escaped, like so "\&" when being used in a string. Is this not true or am I not escaping it correctly?

Your cmd_line should look like this
cmd_line = "del /f /q \\\\{0}{1} & if exist \\\\{0}{1} (set test=1) else (set test=0)".format(address, del_file)
Try using one &

I figured out the issue, turns out I actually had a few. The first being that the ampersand apparently does not need to be escaped (Thank you FirebladeDan), the second being that I was missing a space before the ampersands (Thank you again to FirebladeDan for the tip with formatting, that is how I noticed the missing space). The last being more of a CMD issue than python, I was trying to directly set the errorlevel variable, when I should have been using the exit /b with the respective 1 or 0.
As a side note, I also added in the use of psexec, it was not necessary for the solution, it just helps with the performance of the overall program.
Here is what I ended up with:
def DelFile(self, address, del_file):
cmd_line = r'psexec \\{0} cmd /c "del /f /q "{1}" && if exist "{1}" (exit /b 1) else (exit /b 0)"'.format(address, del_file)
myP = subprocess.Popen(cmd_line, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
out = timeout(myP.communicate, (), {}, 60, (None, None))
result = out[0]
error = out[1]
error_code = None
error_code = myP.returncode
notes = []
return cmd_line, result, error, error_code, notes