We Keep Coding

subprocess for long string of commands python error message [duplicate]

This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 1 year ago.
I want to write a python script to execute to following line:
ls -1 -d ??? | cut -c 1-2 | sort | uniq | while read NN; do echo $NN; tar -cjf fcllistRPC_${NN}.bz2 ${NN}?; done
I tried:
def loop(jobname):
"""
Loop over gen_fcl output and store as tarbull
"""
loop = ["ls", "-1", "-d", "???", "|", "cut", "-c", "1-2", "|", "sort", "|", "uniq", "|", "while", "read", "NN;", "do", "echo", "$NN;", "tar", "-cjf", "fcllist_"+str(jobname)+".bz2", "${NN}?;", "done"]
subproc = subprocess.run(loop)
return subproc
But I am seeing this error:
ls: invalid option -- 'j'
Try 'ls --help' for more information.
Does anyone have any ideas?

in your code you have -cjf. each letter is an "option" for a command. this is causing the error. if you remove it it will run
also if you want it to run a line of code then run that line of code.
codeToRun = "ls -1 -d ??? | cut -c 1-2 | sort | uniq | while read NN; do echo $NN; tar -cjf fcllistRPC_${NN}.bz2 ${NN}?; done"
subproc = subprocess.run(codeToRun)

How to format plain Null ('\0') character for query string in python

For an macOS system uninstall command, need to insert and format the command with plain null ('\0') character.
However, python formatter convert that null character string into it's own hex format '\x00' which is in-correct for the requirement
How do I format following uninstall command in python
Command = 'pkgutil --only-dirs --files <your-package-id> | tail -r | tr '\n' '\0' | xargs -n 1 -0 sudo rmdir'
Tried following method but not working as expected
PACKAGE_ID = 'com.abcd.bdr'
TR_COMMAND = 'tr \'\n\' \'\0\''
command = u'pkgutil --only-dirs --files {} | tail -r | {} | xargs -n 1 -0 sudo rm -rif'.format(PACKAGE_ID, TR_COMMAND)
prog = Popen(command, shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
_outs, _errs = prog.communicate(b'')
Python converts the Null character '\0' string to '\x00' which is not correct for me
Expected result was :
Command = 'pkgutil --only-dirs --files <your-package-id> | tail -r | tr '\n' **'\0'** | xargs -n 1 -0 sudo rmdir'
But python complier gives :
Command = 'pkgutil --only-dirs --files <your-package-id> | tail -r | tr '\n' '**\x00'** | xargs -n 1 -0 sudo rmdir'

How to implement find command to use input from sys.argv in Python

I have a bunch of search strings in a dictionary that I use to parse information.
my_func_dict = {
'index_one': r'pattern1'
'index_two': r'pattern2'
etc
}
Then I use the following to capture the path to evaluate and apply my search strings which is working fine.
if len(sys.argv) >= 2:
location = sys.argv[1]
else:
location = raw_input("Enter the path to evaluate...>: ")
Then, I iterate the dictionary items to apply the search command:
search_cmd = 'grep -h -r'.split()
for name, pattern in my_func_dict.items():
with open('{}.txt'.format(name), 'a') as output:
cmd = search_cmd + [pattern, location]
subprocess.call(cmd, stdout=output)
This is working fine for few search patterns and few files to evaluate. But in my case I have many search patterns and apply these to folders with several files which includes several extension types: *.txt, *log, etc, and this is taking ages. I would like to use the find option to look only for the specific file type inside in the folder path first and then apply the grep more precise in order to obtain the output results much faster.
But following attempt:
search_cmd = 'find $location -name "*test.txt" -print0 | xargs -0 grep -h -r'.split()
for name, pattern in my_func_dict.items():
with open('{}.txt'.format(name), 'a') as output:
cmd = search_cmd + [pattern, location]
subprocess.call(cmd, stdout=output)
Gives me an error:
find: |: unknown primary or operator
find: |: unknown primary or operator
find: |: unknown primary or operator
find: |: unknown primary or operator
How can I implement my search_cmd to avoid this issue?. I need to use -print0 and xargs -0 as attributes for find because folder name in path has spaces, for example: /This is the path/for/This Folder. Thanks

You could use subprocess with a full string using shell=True with Popen. We can also use Python to split the output on newlines.
import subprocess
mydict = {'.': 'patte', './': '".atty"'}
results = []
for path, pattern in mydict.items():
cmd = 'find ' + path + ' -type f -name "*.txt" | xargs fgrep -h --basic-regex ' + pattern
sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
response = sp.communicate()
response = [x.decode('utf-8').strip().split('\n') for x in response if x]
if response:
response = response[0]
results.append(response)
results
[['pattern1', 'pattern2'], ['pattycakes', 'patty']]

Insert changing string from filename at the beginning of every line in terminal

I have zero experience with bash so I'm struggling with the syntax - I gave up attempting in python as I thought that it might be easier. I would like to extract part of the filename (before the .xyz extension, and after the prefix), insert it on every line (starting from the third line) and pipe the output to a new file. I would also like to do this for multiple files, where the string changes.
My input files are as follows:
blahblah-0.xyz
blahblah-1.xyz
blahblah-2xyz
So far I know that I can do:
sed '3,$ s/^/500 /' file-500.xyz > output
and this will insert the string on every line. But I don't want to do this 100 times for each directory! I also tried the following from here: awk parse filename and add result to the end of each line:
for filename in ratio*; do
num=$(echo $filename | grep -Eo '[^ratio_distances]+\.xyz' | cut -d. -f1)
sed -i "s/\^/\t$num" $filename
done
Just to add, this is just being performed in the standard mac terminal, as I've had errors crop up in regards to the 'sed -i' command.
EDIT:
I got it to work in python, but I'd still be interested to know the bash commands. Python code should any one else be after the same thing:
import sys
import os
import glob
list_of_files = glob.glob("./blah-blah*.xyz")
for file in list of files:
for i in range (0, 80):
P = 10*i
if str(P) in file:
with open(file, 'r') as infile:
lines = infile.readlines()
lines[:]=lines[2:]
lines = [str(P)+' '+line for line in lines]
with open(file.replace('blahblah','output'),'w') as outfile:
outfile.writelines(lines)
infile.close()
outfile.close()
Thanks very much for any insight,
Anna

Assuming you can just prefix the old file names with "new_" to create the new file names:
awk '
FNR==1 { pfx = FILENAME; sub(/.*\./,"",pfx) }
FNR>=3 { $0 = pfx $0 }
{ print > ("new_"FILENAME) }
' ratio*

You can use bash's parameter expansion to extract the number from the file name. The -i is not supported in Mac's sed, so you have to use a temp file:
#! /bin/bash
for filename in ratio* ; do
num=${filename#ratio_distances-} # Remove from the left.
num=${num%.xyz} # Remove from the right.
sed "3,\$ s/^/$num /" "$filename" > new"$num"
mv new"$num" "$filename" # Replace the original with the tempfile.
done

#!/bin/bash
PrefixFile="blahblah"
awk -v "Prefix=${PrefixFile}" '
# At each new file starting
FNR == 1 {
# take number from current file name
gsub( "^" Prefix "-|[.]xyz$", "", Index = FILENAME)
}
# at each line (so for every files)
{
# print the index file (current) followed by original line
# to the (corresponding) filename.New
print Index $0 > ( FILENAME ".New" )
}
' ${PrefixFile}*.xyz
Using awk, using all files at once from shell expansion
assume prefix is without the - (easily adapteable if not)
ouput culd be any other file except itself (modification of aw could also change the name at the end but better in bash itself)
Thanks to #EdMorton for extra info about behaviour of awk about file redirection

Run shell commands from python and string concatenation

I am trying to adjust the "post to dropbox" services for Snow loepard (http://dl.dropbox.com/u/1144075/Post%20to%20Dropbox.zip).
I dont want the public URL, but a shortened one from goo.gl
Therefor I am using these shell commands:
curl -s --data-urlencode "url=http://link.com" http://googl/action/shorten | grep "googl" | awk -F\" '{print $(NF-1)}' | awk 'BEGIN { FS = "=" } ; { print $2}' | pbcopy
Now the python script does this to copy a dropbox URL for all the files it just copies in the public folder to the clipboard:
pasteURLs = []
for file in copied_files: # for all elements in our list
components = file.split(os.sep) # seperate the path
local_dir = os.sep.join(components[5:]) # cut off the beginning
local_dir = urllib.quote(local_dir) # convert it to a URL (' ' -> '%20', etc.)
#construct the URL
finalURL = 'http://dl.dropbox.com/u/%s/%s' % ( dropbox_id, local_dir )
pasteURLs.append(finalURL) # add the current URL to the string
copy_string = "\n".join(pasteURLs)
os.system( "echo '%s' | pbcopy" % (copy_string) ) # put the string into clipboard
I have to admit I dont know anything about python, but from what it looks like, I need to change the last two lines with this:
shortURL = []
for thisURL in pasteURLs:
shortURL = os.system( curl -s --data-urlencode "url=http://link.com" http://googl/action/shorten | grep "goo.gl" | awk -F\" '{print $(NF-1)}' | awk 'BEGIN { FS = "=" } ; { print $2}' | pbcopy )
shortURLs.append(shortURL)
copy_string = "\n".join(shortURLs)
os.system( "echo '%s' | pbcopy" % (copy_string) ) # put the string into clipboard
But my problem is, how to put the correct URL in the command? As u can see it says http://link.com But it should use thisURL instead.
Any ideas? Thanks in advance!

I think your os.system call should look something like this:
os.system("curl -s --data-urlencode \"url=%s\" http://goo.gl/action/shorten | grep \"goo.gl\" | awk -F\\\" '{print $(NF-1)}' | awk 'BEGIN { FS = \"=\" } ; { print $2}' | pbcopy " % thisURL)

UPDATE I wrote the script for you and used a much simpler command pipeline. Not that the entire thing could be done in python without curl, but here it is.
import subprocess
thisURL = 'http://whatever.com'
pipeline = []
pipeline.append('curl -s -i --data-urlencode "url=%s" ' % thisURL +
'http://goo.gl/action/shorten')
pipeline.append('grep Location:')
pipeline.append('cut -d = -f 2-')
#pipeline.append('pbcopy')
command = "|".join(pipeline)
link, ignore = subprocess.Popen(command, stdout=subprocess.PIPE,
shell=True).communicate()
print link

Other answers have already provided the core of this: use quotation marks around your command, use a format string to insert the value and consider using subprocess in order to actually get the output from the command.
However, if you, like me, think this is getting a bit too convoluted, go have a look at this example on how to do the actual shortening in python. If you're new to python, this might mean you'll need to read up on your exception handling to understand it. (It also looks like you might need a custom module, but then again it appears to only be used if you get an exception...)