We Keep Coding

issue parsing shell program through python

My function run_deinterleave() is meant to copy code from the file deinterleave.sh then replace the placeholder (sra_data) with a file name which has been input by the user and then run it on the command line.
def run_deinterleave():
codes = open('Project/CODE/deinterleave.sh')
codex = codes.read()
print(inp_address)
codex = codex.replace('sra_data', inp_address)
#is opening this twice creating another pipeline?
stream = os.popen(codex)
codes.close()
self.txtarea.insert(END,codex)
#stuff
However, I keep getting this error:
/bin/sh: 5: Syntax error: "(" unexpected
The code in deinterleave.sh works fine and produces two individual files given an interleaved paired end sra_file (an output file from genetic sequencing machines, I think :P)
#1deinterleave paired end fastq file
paste - - - - - - - - < sra_data \
| tee >(cut -f 1-4 | tr "\t" "\n" > /home/lols/Project/reads-1.fq) \
| cut -f 5-8 | tr "\t" "\n" > /home/lols/Project/reads-2.fq

As the error message shows, the code was interpreted by /bin/sh; if you executed
/bin/sh Project/CODE/deinterleave.sh, you'd get the same error, because the process substitution >(…) is a Bash extension not understood by /bin/sh.
Besides, since you don't communicate with the shell code, we don't need pipes at all. So instead of os.popen I'd use subprocess.run, which allows to specify Bash as the shell.
subprocess.run(codex, shell=True, executable="bash")

The absolutely best fix is probably to replace the shell script with native Python code; but without a specification and/or sample input, I don't think we can tell you exactly how to do that.
An immediate and trivial fix is to change deinterlace so that it accepts an input file parameter.
#!/usr/bin/env bash
paste - - - - - - - - < "${1-sra_data}" |
tee >(cut -f 1-4 | tr "\t" "\n" > "${2-/home/lols/Project/reads-1.fq}") |
cut -f 5-8 | tr "\t" "\n" > "${3-/home/lols/Project/reads-2.fq}"
This refactoring also allows you to specify the names of the output files as the second and third command-line arguments.
Also, a Bash script really should not have a .sh extension, so probably take that out.
Explictly naming Bash in the shebang line should solve the error message you got when running Bash code in sh; perhaps see also Difference between sh and bash
With that, your Python code can be reduced to something like
subprocess.run(
['Project/CODE/deinterleave', inp_address],
# probably a good idea
check=True)
though I don't exactly understand the rest of the surrounding function, so it's not clear how exactly to rewrite it.
I think the shell script could be reimplemented something like
with open(inp_address, 'r') as sra_data, open(
'/home/lols/Project/reads-1.fq', 'w') as first, open(
'/home/lols/Project/reads-2.fq', 'w') as second:
for idx in range(4):
first.write(sra_data.readline())
for idx in range(4):
second.write(sra_data.readline())

How to separate the parts of this unix command to use in Python subprocess module?

I would like to add a Linux command to my Python script (using the subprocess module). This is the command I would like to add:
awk '/^>/ {printf("%s%s\t",(N>0?"\n":""),$0);N++;next;} {printf("%s",$0);} END {printf("\n");}' < input.fa
And this is currently what my code looks like:
out = subprocess.Popen(["awk '/^>/ {printf("%s%s\t",(N>0?"\n":""),$0);N++;next;} {printf("%s",$0);} END {printf("\n");}' < input.fa"],
stdout = subprocess.PIPE,
stderr = subprocess.STDOUT)
stdout, stderr = out.communicate()
print(stdout)
print(stderr)
However, it appears that my syntax is wrong somewhere. In some other examples I have seen, it seems like the command needs to be split into sections with apostrophes - however, this command is quite complex, and I do not know where the splitting would be required.
I would appreciate any help in identifying the underlying issue and correcting the syntax of this script.

I have managed to circumvent my problem by using a command which does the same thing but does not include any "\n", which is what seems to have been the cause of my issues. Also, using different quotation marks (i.e. " and ') helps to prevent this causing issues. The code now looks like this and does what I need it to do:
print(subprocess.Popen('perl -ne "chomp;print;" chr11', shell=True, stdout=subprocess.PIPE).stdout.read())

Determine the terminal cursor position with an ANSI sequence in Python 3

I want to write a little script which prints images to the terminal with /usr/lib/w3mimgdisplay(like in mac osx lsi). Therefore i need the the actual cursor position (or caret position) when the script has started. So far i figured out to get the cursor position in a shell with an ANSI sequence:
$ echo -en "\e[6n"
^[[2;1R$ 1R
This is how the response of this ANSI sequence look like (urvxt and bash - don't know if that is important). So this sequence prints out the result (^[[2;1R) immediately. And this is what i don't understand. How is this done? If i write a very simple shell script, just with that instruction and strace the script, this doesn't clear up things ether. What the? Then i try to figure out how this happens by looking in the terminfo manpage. Can't find it here (maybe i didn't try hard enough). At this point i find myself very confused about this concept. Does the terminal write the position even to stdout?
Terminal
#!/bin/bash
echo -en "\e[6n"
$ strace sh curpos.sh
[...]
read(255, "#!/bin/bash\necho -en \"\\e[6n\"\n", 29) = 29
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 8), ...}) = 0
write(1, "\33[6n", 4) = 4
^[[54;21Rread(255, "", 29) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(0) = ?
+++ exited with 0 +++
Python
First i tried to use subprocess.check_output which of course just returns the the string i've echoed. How do i capture the response to this ANSI sequence?
>>> subprocess.check_output(["echo", "-en", "\x1b[6n"])
b"\x1b[6n"
I also tried a lot of other things, like reading the stdin and stdout!? With and without threads, but all of that was more a guessing and mocking around rather than knowing what to do. I also search the Internet for quite a while, hoping to find an example of how to do this, but whit no luck. I've found this answer to the same question: https://stackoverflow.com/a/35526389/2787738 but this don't work. Actually i don't know if this has ever worked, because in this answer the ANSI sequence is written to stdout before it starts reading from stdin? Here i realised again that i don't understand the concept/mechanism how these ANSI sequences really work. So at this point every explanation which clears things a very much appreciated. The most helpful post i found was this one: https://www.linuxquestions.org/questions/programming-9/get-cursor-position-in-c-947833/. In this thread someone posted this bash script:
#!/bin/bash
# Restore terminal settings when the script exits.
termios="$(stty -g)"
trap "stty '$termios'" EXIT
# Disable ICANON ECHO. Should probably also disable CREAD.
stty -icanon -echo
# Request cursor coordinates
printf '\033[6n'
# Read response from standard input; note, it ends at R, not at newline
read -d "R" rowscols
# Clean up the rowscols (from \033[rows;cols -- the R at end was eaten)
rowscols="${rowscols//[^0-9;]/}"
rowscols=("${rowscols//;/ }")
printf '(row %d, column %d) ' ${rowscols[0]} ${rowscols[1]}
# Reset original terminal settings.
stty "$termios"
Here we can see that indeed the response is somehow magically appears one the screen :). This is why this script disables echoing on the terminal and after reading the response it resets the original terminal settings via stty.

Here is a POC snippet, how to read the current cursor position via an ansi/vt100 controll sequence.
I suggest you to put the following code inside a curpos.py file and import this file in your code (import curpos).
import re, sys, termios, tty
def getpos():
buff = ''
stdin = sys.stdin.fileno()
tattr = termios.tcgetattr(stdin)
try:
tty.setcbreak(stdin, termios.TCSANOW)
sys.stdout.write('\033[6n')
sys.stdout.flush()
while True:
buff += sys.stdin.read(1)
if buff[-1] == 'R':
break
finally:
termios.tcsetattr(stdin, termios.TCSANOW, tattr)
matches = re.match(r'^\033\[(\d*);(\d*)R', buff)
if matches == None: return None
groups = matches.groups()
return (int(groups[0]), int(groups[1]))
An example of how your code should/could be:
import curpos
res = getpos()
if res == None:
print('Problem, I cannot get the cursor possition')
else:
print(f'The cursor position is: {res}')
warning
This is not perfect. To make it more robust, a routine which sorts out keystrokes from the user while reading from stdin would be nice.

Python subprocess library: Running grep command from Python

I am trying to run grep command from my Python module using the subprocess library. Since, I am doing this operation on the doc file, I am using Catdoc third party library to get the content in a plan text file. I want to store the content in a file. I don't know where I am going wrong but the program fails to generate a plain text file and eventually to get the grep result. I have gone through the error log but its empty. Thanks for all the help.
def search_file(name, keyword):
#Extract and save the text from doc file
catdoc_cmd = ['catdoc', '-w' , name, '>', 'testing.txt']
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
output = catdoc_process.communicate()[0]
grep_cmd = []
#Search the keyword through the text file
grep_cmd.extend(['grep', '%s' %keyword , 'testing.txt'])
print grep_cmd
p = subprocess.Popen(grep_cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
stdoutdata = p.communicate()[0]
print stdoutdata

On UNIX, specifying shell=True will cause the first argument to be treated as the command to execute, with all subsequent arguments treated as arguments to the shell itself. Thus, the > won't have any effect (since with /bin/sh -c, all arguments after the command are ignored).
Therefore, you should actually use
catdoc_cmd = ['catdoc -w "%s" > testing.txt' % name]
A better solution, though, would probably be to just read the text out of the subprocess' stdout, and process it using re or Python string operations:
catdoc_cmd = ['catdoc', '-w' , name]
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
for line in catdoc_process.stdout:
if keyword in line:
print line.strip()

I think you're trying to pass the > to the shell, but that's not going to work the way you've done it. If you want to spawn a process, you should arrange for its standard out to be redirected. Fortunately, that's really easy to do; all you have to do is open the file you want the output to go to for writing and pass it to popen using the stdout keyword argument, instead of PIPE, which causes it to be attached to a pipe which you can read with communicate().

Avoid subprocess.Popen auto escaping my backslashes in grep

I'm trying to write an svn pre-commit hook in python. Part of this involves checking the diff file to see if there are any actual file changes (as opposed to just property changes).
I have a working grep command which I can execute fine on the shell
grep "^\(Added: \|Modified: \|Deleted: \)" diff filename | grep -v 'svn:'
However when I put it through subprocess.POpen it escapes all my backslashes, which knackers the regexp.
Executing command: ['grep', '"^\\Added: \\|Modified: \\|Deleted: \\)", ...]
How do I avoid this?
NB: I'm aware that I can pipe results between subprocesses and I can do the two greps that way. I need help getting the first one working first though :/
NB2: I also tried using filterdiff --clean instead and couldn't get it to work. Searching for Added, Modified or Deleted lines, removing those with 'svn:' in and checking I had some results seemed to work though.
Python code:
command = ['grep', '"^\(Added: \|Modified: \|Deleted: \)"', filename]
sys.stdout.write('Executing command: %s\n' % (command))
p = subprocess.Popen(command,
stdin = subprocess.PIPE
stdout = subprocess.PIPE
stderr = subprocess.STDOUT
shell = True)
data = p.stdout.read()
if len(data) == 0:
sys.stdout.write("Diff does not contain any file modifications./n")
exit(0)

You need to consider what you want grep to see in its command line arguments.
The first argument needs to be the literal string "^\(Added: \|Modified: \|Deleted: \)", so that means that it shouldn't include the double quotes but should include the backslashes.
The way to express this kind of string is to use Python raw strings:
command = ['grep', r'^\(Added: \|Modified: \|Deleted: \)', filename]
A good way to check what you're actually running is to replace grep by echo so you can at least see what you're passing to the command.