We Keep Coding

sh: 1: Syntax error: redirection unexpected Python/Bash

I have problem with my python cmd script.
I don't know why it does not work. Maybe something wrong with my code.
Im trying to run the program in cmdline through my python script.
And Im getting error in bash "sh: 1: Syntax error: redirection unexpected"
pls help Im just biologist :)
Im using spyder (anaconda)/Ubuntu
#!/usr/bin/python
import sys
import os
input_ = sys.argv[1]
output_file = open(sys.argv[2],'a+')
names = input_.rsplit('.')
for name in names:
os.system("esearch -db pubmed -query %s | efetch -format xml | xtract -pattern PubmedArticle -element AbstractText >> %s" % (name, output_file))
print("------------------------------------------")

output_file is a file object. When you do "%s" % output_file, the resulting string is something like "<open file 'filename', mode 'a+' at 0x7f1234567890>". This means that the os.system call is running a command like
command... >> <open file 'filename', mode 'a+' at 0x7f1234567890>
The < after the >> causes the "Syntax error: redirection unexpected" error message.
To fix that, don't open the output file in your Python script, just use the filename:
output_file = sys.argv[2]

I got similar error on following line:
os.system('logger Status changed on %s' %s repr(datetime.now())
Indeed, as nomadictype stated the problem is in running plain OS command. The command may include special characters. In my case this was <.
So instead of changing OS command significantly, I just added quotes and this works:
os.system('logger "Status changed on %s"' %s repr(datetime.now())
Quotes make content of passed parameter invisible for shell.

/bin/sh: line 62: to: command not found

I have a python code in which I am calling a shell command. The part of the code where I did the shell command is:
try:
def parse(text_list):
text = '\n'.join(text_list)
cwd = os.getcwd()
os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
synnet_output = subprocess.check_output(["echo '%s' | syntaxnet/demo.sh 2>/dev/null"%text], shell = True)
os.chdir(cwd)
return synnet_output
except Exception as e:
sys.stdout.write(str(e))
Now, when i run this code on a local file with some sample input (I did cat /home/sree/example.json | python parse.py) it works fine and I get the required output. But I am trying to run the code with an input on my HDFS (the same cat command but input file path is from HDFS) which contains exactly the same type of json entries and it fails with an error:
/bin/sh: line 62: to: command not found
list index out of range
I read similar questions on Stack Overflow and the solution was to include a Shebang line for the shell script that is being called. I do have the shebang line #!/usr/bin/bash in demo.sh script.
Also, which bash gives /usr/bin/bash.
Someone please elaborate.

You rarely, if ever, want to combine passing a list argument with shell=True. Just pass the string:
synnet_output = subprocess.check_output("echo '%s' | syntaxnet/demo.sh 2>/dev/null"%(text,), shell=True)
However, you don't really need a shell pipeline here.
from subprocess import check_output
from StringIO import StringIO # from io import StringIO in Python 3
synnet_output = check_output(["syntaxnet/demo.sh"],
stdin=StringIO(text),
stderr=os.devnull)

There was a problem with some special characters appearing in the text string that i was inputting to demo.sh. I solved this by storing text into a temporary file and sending the contents of that file to demo.sh.
That is:
try:
def parse(text_list):
text = '\n'.join(text_list)
cwd = os.getcwd()
with open('/tmp/data', 'w') as f:
f.write(text)
os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
synnet_output = subprocess.check_output(["cat /tmp/data | syntaxnet/demo.sh 2>/dev/null"%text], shell = True)
os.chdir(cwd)
return synnet_output
except Exception as e:
sys.stdout.write(str(e))

Path not being found in Windows via bash/python script

This is .csv file containing target,path:
[root#server ~]# head /usr/local/CustomAppResults/App/targets.txt
server1.com,C:\\Program Files\\App\\
I loop through the .csv, line by line in a Bash script ..
IFS=","
while read target path
do
# Windows target
result=$(python /root/apache-tomcat-7.0.37/webapps/App/tools/remoteStafCommand.py "$target" "dir $path /B")
resultCode=$?
echo "$result"
done
The bash script passes the contents of the csv. file into a Python script, it's contents are below.
The Python script runs a staf subprocess and parses the results, which is returned and stored inside result of the Bash script.
[root#server ~]# cat App/tools/remoteStafCommand.py
import subprocess,re,sys
from subprocess import CalledProcessError, check_output
print "1st arg: %s" % (sys.argv[1])
print "2nd arg: %s" % (sys.argv[2])
try:
output = subprocess.check_output(["staf", "%s" % (sys.argv[1]) , "PROCESS", "START", "SHELL", "COMMAND", "%s" % (sys.argv[2]), "WAIT", "RETURNSTDOUT", "STDERRTOSTDOUT"])
result = re.findall(r'Data\s*:\s*((?:[^\n]*(?:[\r\n]+(?!\s*}))?)+)', output, re.DOTALL)[0]
print result
except CalledProcessError as e:
print(e.returncode)
sys.exit(e.returncode)
This is the output I get, why can it not find the path? It exists.
1st arg: server1.com
2nd arg: dir C:\Program Files\App\ /B
The system cannot find the path specified.

The read command tokenizes on whitespace, not comma. You can override this by changing the value of IFS. Or if the text file isn't used for anything else, change its format to have a space instead of a comma between the fields. Or better still, have your Python script read and parse the file directly -- the simple Bash script begs to be factored away.

Wrapping path with double quotes directly in .csv file for escaping ..
[root#server ~]# head /usr/local/CustomAppResults/App/targets.txt
server1.com,"C:\\Program Files\\App\\"

What did I do wrong in my python script to get output?

I think that's should work. I want to get output of this command
"grep -l %s *.py" % tag for each tag,
on created file with tag name.
import os
import sys
results_dir = '/home/ks/one/work/tn/format-description 15852/results'
converters_dir = '/home/ks/one/work/cvr'
export_dir = '/home/ks/one/work/epr'
all_dirs = {'cvr':cvrs_dir,
'epr':eprs_dir}
tags = [tag.strip() for tag in open('new file').readlines()]
for coe, directory in all_dirs.items(): # coe - type of our file
os.chdir(results_dir)
for tag in tags:
tag_file = open(coe + ' ' + tag, 'w')
sys.stdout = tag_file
os.chdir(directory)
os.system("grep -l %s *.py" % tag)
tag_file.close()
But all what I see when script is runned - it's output in my console.

Use the subprocess module of Python
http://docs.python.org/library/subprocess.html
The documentation contains extensive documentation and examples.
The other (poor) option with os.system() is redirecting the output to a file
os.system('do_something >/tmp/myoutput')
and then later reading the output file from within Python. However this is ugly
and likely not very portable.

The most minimal change I can think of so you can get at the output in python is to change:
os.system("grep -l %s *.py" % tag)
for:
output = commands.getoutput("grep -l %s *.py" % tag)
This way, the output of the command will end up in the output variable.
I'm not sure why other people dogmatically tell you to change your code to use subprocess; it will need more rewriting to do so.

mp3info giving an error with variable filename

I have been using mp3info to calculate my file length using the following syntax:
mp3info -p "%S" /path/to/file
whenever I use the code with the filename I get the correct output:
mp3info -p "%S" far_from_love.mp3
However, on storing the filename in a string variable and then using the variable I get an error:
SyntaxError: invalid syntax
Could anyone tell me how to correctly use the command with a variable filename?
this is the python code which uses mp3info
listing=os.listdir("C:\\Python27")
for f in listing:
if fnmatch.fnmatch(f,'*.mp3'):
ext=f[:-4] #extract name of file without extension
WAVE_OUTPUT_FILENAME="%s.wav"%ext
print WAVE_OUTPUT_FILENAME#save output filename as wav extension
print f
x=os.popen('mp3info -p "%S" f).read()
print x

x=os.popen('mp3info -p "%S" f).read()
is missing a closing quote:
x=os.popen('mp3info -p "%S" ' + f).read()
You may also want to use the safer subprocess module:
import subprocess
x = subprocess.check_output(['mp3info', '-p', '%S', f])