We Keep Coding

subprocess.run simple scenario fails

I am trying to run python subprocess.run function to execute following command:
pdftoppm -jpeg -f 1 -scale-to 200 data/andromeda.pdf and-page
pdftoppm - is part of poppler utility and it generates images from pdf files.
File data/andromeda.pdf exists. Folder data is on same level with python script and/or where I run command from.
Command basically will generate a jpeg file, from page 1 (-f 1) 200px wide (-scale-to) from given file of and-page-1.jpeg format (so called ppmtroot).
Long story short: from command line it works as expected i.e. if I call the above command either from zsh or bash shell, manually - it generates thumbnail as expected. However if I run it from python subprocess module - it fails it returns 99 error code!
Following is python code (file name is sc_02_thumbnails.py):
import subprocess
import sys
def main(filename, ppmroot):
cmd = [
'pdftoppm',
'-f 1',
'-scale-to 200',
'-jpeg',
filename,
ppmroot
]
result = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
if result.returncode:
print("Failed to generate thumbnail. Return code: {}. stderr: {}".format(
result.returncode,
result.stderr
))
print("Used cmd: {}".format(' '.join(cmd)))
sys.exit(1)
else:
print("Success!")
if __name__ == "__main__":
if len(sys.argv) > 2:
filename = sys.argv[1]
ppmroot = sys.argv[2]
else:
print("Usage: {} <pdffile> <ppmroot>".format(sys.argv[0]))
sys.exit(1)
main(filename, ppmroot)
And here is repo which includes data/andromeda.pdf file as well.
I call my script with as (from zsh):
$ chmod +x ./sc_02_thumbnauils.py
$ ./sc_02_thumbnails.py data/andromeda.pdf and-page
and ... thumbnail generating fails!
I have tried executing python script from both, from zsh and bash shells :(
What I am doing wrong?

The quoting is wrong, you should have '-f', '1', etc

sh: 1: Syntax error: redirection unexpected Python/Bash

I have problem with my python cmd script.
I don't know why it does not work. Maybe something wrong with my code.
Im trying to run the program in cmdline through my python script.
And Im getting error in bash "sh: 1: Syntax error: redirection unexpected"
pls help Im just biologist :)
Im using spyder (anaconda)/Ubuntu
#!/usr/bin/python
import sys
import os
input_ = sys.argv[1]
output_file = open(sys.argv[2],'a+')
names = input_.rsplit('.')
for name in names:
os.system("esearch -db pubmed -query %s | efetch -format xml | xtract -pattern PubmedArticle -element AbstractText >> %s" % (name, output_file))
print("------------------------------------------")

output_file is a file object. When you do "%s" % output_file, the resulting string is something like "<open file 'filename', mode 'a+' at 0x7f1234567890>". This means that the os.system call is running a command like
command... >> <open file 'filename', mode 'a+' at 0x7f1234567890>
The < after the >> causes the "Syntax error: redirection unexpected" error message.
To fix that, don't open the output file in your Python script, just use the filename:
output_file = sys.argv[2]

I got similar error on following line:
os.system('logger Status changed on %s' %s repr(datetime.now())
Indeed, as nomadictype stated the problem is in running plain OS command. The command may include special characters. In my case this was <.
So instead of changing OS command significantly, I just added quotes and this works:
os.system('logger "Status changed on %s"' %s repr(datetime.now())
Quotes make content of passed parameter invisible for shell.

python script to open a file in sublime text

Context:
I am modifying a small python script, so that the output of a lldb command that I run from Xcode debugger, will be output to a file and open up in sublime text.
import lldb
import os
import subprocess
def print_to_file(debugger, command, result, dict):
#Change the output file to a path/name of your choice
f=open("/Users/venkat13/pcomponents.txt","w")
debugger.SetOutputFileHandle(f,True);
debugger.HandleCommand(command)
path = "/Users/venkat13/pcomponents.txt"
sublimePath = '/Applications/Sublime\ Text.app'
subprocess.Popen("%s %s" % (sublimePath, path))
def __lldb_init_module (debugger, dict):
debugger.HandleCommand('command script add -f po.print_to_file print_to_file ')
Problem :
This above script is generating the file, but it does not open in sublime text. Where am I going wrong ?

That's because Popen will not directly pass the entire string to the shell unless you specify shell=True (It just seems to me that you expecting the command to work the same way you would typically type on a shell).
Instead use:
subprocess.Popen([sublimePath, path])
Alternatively, (NOT RECOMMENDED):
subprocess.Popen("%s %s" % (sublimePath, path), shell=True)

Keyerror in reading csv file

I am trying to write a script where I pass file name as argument from shell script to python script and python script processes that script.It is giving me keyerror but if I run the same script hardcoding the file name it works fine.
#!/bin/sh
LOCKFILE=./test.txt
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
echo "already running"
exit
fi
trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT
echo $$ > ${LOCKFILE}
# do stuff
FILES=/home/sugoi/script/csv/*
for file in $FILES
do
python ./csvTest.py $file
#mv $file ./archive
done
rm -f ${LOCKFILE}
exit
Python:
from pymongo import MongoClient
import csv
import json
import sys
client = MongoClient()
db = client.test
for arg in sys.argv:
try:
csvfile = open(arg, 'r')#if i hardcode file name here it works fine
except IOError as e:
#write to error log
sys.exit(100)
reader = csv.DictReader(csvfile)
header=reader.next()
for each in reader:
row={}
for field in header:
row[field]=each[field]
db.test.update({"_id": row["CustomerId"]}, {"$push": {"activities":{"action": row["Action"],"date" :row["Timestamp"],"productId":row["productId"]}}},True)
What am I doing wrong ?

Two issues.
Your shell script isn't expanding the file list correctly.
FILES=/home/sugoi/script/csv/* needs to be something like:
FILES=`ls -1 /home/sugoi/script/csv/*;`
Your argument to the python script will only be one file at a time, so why loop through sys.argv?
Just use the argument itself, sys.argv[1]. As #Brian Besmanoff pointed out, that needs to be indexed 1 because the script name itself is stored in sys.argv[0].
try:
csvfile = open(sys.argv[1], 'r')
except IOError as e:
(...)
Finally: you can just parse directories with Python instead of looping in a shell script. Look at the os module, particularly os.listdir(). A little more work and you can have the whole thing running inside one Python script instead of juggling between shell and calling a script.

The first value in sys.argv is going to be the name of the script. reference

awk calling in python script responding as file cannot be open/read

I am trying to open a file via python script by calling awk by giving folder and file name as %s.
import commands
genotype = 'rice'
filename = 'model.txt'
dear = commands.getoutput('''awk '{print $0}' /home/angad/Desktop/Python_result/%s/%s''') % (genotype,filename)
print dear
But it is returning as:
awk: cmd. line:1: fatal: cannot open file '/home/angad/Desktop/Python_result/rice/model.txt' for reading (No such file or directory)
The concerned file is indeed present there.
As if, I write input file path directly instead of raw_input(%s), it successfully open and print the file content.

One of your parentheses is in the wrong place. Try:
dear = commands.getoutput('''awk '{print $0}' /home/angad/Desktop/Python_result/%s/%s''' % (genotype,filename))
Or better yet:
cmd_str = '''awk '{print $0}' /home/angad/Desktop/Python_result/%s/%s''' % (genotype,filename)
dear = commands.getoutput(cmd_str)
The reason the error message shows the correct file path in your original example is that the result of getoutput, which included the error message with the %s formats still present, was being substituted before being printed. Very misleading!