We Keep Coding

how to use popen with command line arguments contains single quote and double quote?

I want to run following jq command with subprocess.Popen() in python3.
$ jq 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}' image_data_annotation.json > image_data_annotation_with_label.json
Note that first command line argument contains dot, dollar sign, double quotes within single quote.
FYI, jq is JSON processor utility for processing json files.
I wrote following python3 script for automating JSON file processing with jq utility.
#!python3
# file name: letter_image_tool.py
import os, subprocess
"""
command line example to automate
$ jq 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}' image_data_annotation.json > image_data_annotation_with_label.json
"""
# define first command line argument
jq_filter='\'INDEX(.images[]; .id) as $imgs | { "filename_with_label" : [ .annotations[] | select(.attributes.type=="letter" ) | $imgs[.image_id] + {label:.text} | {id:.id} + {filename:.file_name} + {label:.label} ] }\''
input_json_files= [ "image_data_annotation.json"]
output_json_files= []
for input_json in input_json_files:
print("Processing %s" %(input_json))
filename, ext = os.path.splitext(input_json)
output_json = filename + "_with_label" + ext
output_json_files.append(output_json)
print("output file is : %s" %(output_json))
#jq_command ='jq' + " " + jq_filter, input_json + ' > ' + output_json
jq_command =['jq', jq_filter, input_json + ' > ' + output_json]
print(jq_command)
subprocess.Popen(jq_command, shell=True)
Running the above python script on bash results in folowing:
$ ./letter_image_tool.py
Processing image_data_annotation.json
output file is : image_data_annotation_with_label.json
['jq', '\'INDEX(.images[]; .id) as $imgs | { "filename_with_label" : [ .annotations[] | select(.attributes.type=="letter" ) | $imgs[.image_id] + {label:.text} | {id:.id} + {filename:.file_name} + {label:.label} ] }\'', 'image_data_annotation.json > image_data_annotation_with_label.json']
jq - commandline JSON processor [version 1.6-124-gccc79e5-dirty]
Usage: jq [options] <jq filter> [file...]
jq [options] --args <jq filter> [strings...]
jq [options] --jsonargs <jq filter> [JSON_TEXTS...]
jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.
The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).
For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq
Example:
$ echo '{"foo": 0}' | jq .
{
"foo": 0
}
For a listing of options, use jq --help.
It does not handle the first argument of jq utility:
'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}'
The first argument should be enclosed with single quote as above snipet but my script does not handle it.
I think the main problems are related to the dot, dollar sign, single quote and double quote used in the first command line argument (jq_filter in the above python script). But I don't know how to treat this kind of complex meta character related to bash.
What should I do to solve above problems?
Thanks for your kind reading.
Update with my solution
With triple quote for jq_filter defintion, and space seprated join as follows
#!python3
# file name: letter_image_tool.py
import os, subprocess
"""
command line example to automate
$ jq 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}' image_data_annotation.json > image_data_annotation_with_label.json
"""
# define first command line argument with triple quotes
jq_filter=""" 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label" : [
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label} ] } ' """
input_json_files= [ "image_data_annotation.json"]
output_json_files= []
for input_json in input_json_files:
print("Processing %s" %(input_json))
filename, ext = os.path.splitext(input_json)
output_json = filename + "_with_label" + ext
output_json_files.append(output_json)
print("output file is : %s" %(output_json))
#jq_command ='jq' + " " + jq_filter, input_json + ' > ' + output_json
# jq command composed with space separated join
jq_command =' '.join['jq', jq_filter, input_json, ' > ', output_json]
print(jq_command)
# shell keyword argument should be set True
subprocess.Popen(jq_command, shell=True)
With triple double quotes, jq_filter can be more readable using multi-lined definition instead of single line defintion.

The reason you need single quotes is to prevent the shell from doing any expansion of your argument. This is a problem, only when using shell=True. If this is not set, the shell will never touch your arguments and there is no need to "protect" them.
However, the shell is also responsible for the stdout redirect (i.e. [... '>', output_json]). Not using the shell, requires that the redirect is handled in the Python code instead. That, however, is as simple as adding the argument stdout=... to Popen.
All-in-all this means that your code can be rewritten as
import os
import subprocess
# Still define first command line argument with triple quotes for readability
# Note that there are no single quotes though
jq_filter = """INDEX(.images[]; .id) as $imgs | {
"filename_with_label" : [
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label} ] }"""
input_json_files = ["image_data_annotation.json"]
output_json_files = []
for input_json in input_json_files:
print("Processing %s" % (input_json))
filename, ext = os.path.splitext(input_json)
output_json = filename + "_with_label" + ext
output_json_files.append(output_json)
print("output file is : %s" % (output_json))
# Keep command as list, since this is what we need when NOT using shell=True
# Note also that the redirect and the output file are not parts of the argument list
jq_command = ['jq', jq_filter, input_json]
# shell keyword argument should NOT be set True
# Instead redirect stdout to an out_file
# (We must open the file for writing before redirecting)
with open(output_json, "w") as out_file:
subprocess.Popen(jq_command, stdout=out_file)
Generally it is recommended to not use shell=True anyway, since that opens up another vector of attack against the code, since an injection attack can give full access to the shell. Also, another small benefit with not using the shell, is that it will reduce the number of created subprocesses, since no extra shell process is needed.

how to use jq tools or python as a output encoder to make result to translate to json formot is a difficult question

for a example
[root#test ~]# mysql -uroot -p'123123' -e"select user,host from mysql.user;"
+-------------------+-----------+
| user | host |
+-------------------+-----------+
| root | % |
| test | % |
| sqlaudit_test_mon | % |
| sysbase_test | % |
| mysql.session | localhost |
| mysql.sys | localhost |
+-------------------+-----------+
how to make search the result quick to convert to json format can you jq tools or python
such as that out put
[
{
"user":"root","host":"%"},
{
"user":"test","host":"%"},
{
"user":"sqlaudit_test_mon","host":"%"},
{
"user":"sysbase_test","host":"%"},
{
"user":"mysql.session","host":"localhost"},
{
"user":"mysql.sys","host":"localhost"}
]
i just want to know how to quick make search result to json,thank you!
it is better to user jq or python script it can make me search result to json format.

Just do it in your SELECT instead of pulling another program into a pipeline. MySQL has JSON functions. Ones of interest here are JSON_ARRAYAGG() and JSON_OBJECT(). Something like:
SELECT json_arrayagg(json_object('user', user, 'host', host)) FROM mysql.user;
should do it, plus whatever's needed to not print out that fancy table ascii art.

Here's an all-jq solution that assumes an invocation like this:
jq -Rcn -f program.jq sql.txt
Note in particular the -R ("raw input") and -n options.
def trim: sub(" *$";"") | sub("^ *";"");
# input: an array of values
def objectify($headers):
. as $in
| reduce range(0; $headers|length) as $i ({}; .[$headers[$i]] = ($in[$i]) ) ;
def preprocess:
select( startswith("|") )
| split("|")
| .[1:-1]
| map(trim) ;
reduce (inputs|preprocess) as $in (null;
if . == null then {header: $in}
else .header as $h
| .table += [$in|objectify($h)]
end )
| .table

remove python multiline comments and do not remove string

the question: I want to use shell scripts to remove:
"""
111
222
"""
or
'''
111
222
'''
but do not remove
s = """
111
222
"""
I test the following ways:
find -name *.py | xargs -i sed -i "/^\s*\"\"\".*\"\"\"$/d" {}
find -name *.py | xargs -i sed -i '/"""/,/"""/d' {}
but i have no idea about
s = """
111
222
"""
please help
the test code like this,thanks
>
"""
template
class Foo {
public:
virtual int Bar();
};
"""
source = """
class Foo {
public:
virtual int Bar();
};
"""
"""
template<class T>
class Foo {
public:
virtual int Bar();
};
"""
source = """
class Foo {
public:
virtual void Bar(bool flag) const;
};
"""
source = """
class Foo {
public:
virtual int Bar(void);
};
"""
uptonow:
the
"""
aaa
"""
and the
="""
bbb
"""
have been test ok
function delete_multiline_comments
function delete_multiline_comments
grep -n '\"\"\"' gmock_class_test.py | sed '/=/,+1d' >a.txt
line=$(wc -l a.txt| awk '{print $1}')
for (( i=0;i<$line/2;i++ ))
do
second=`tail -1 a.txt | tr -cd "[0-9]"`
sed -i '$d' a.txt
first=`tail -1 a.txt | tr -cd "[0-9]"`
sed -i '$d' a.txt
sed -i "${first},${second}d" gmock_class_test.py
done
end
the following line used to remove """XXXX""" and '''XXX''' line
sed -ie "/'''.*'''/d"
sed -i "/^\s*\"\"\".*\"\"\"$/d"

Not sed, but you can try vims.
For your problem, this is a quick way to do it (it swaps the """ with a key: eh3UgT):
cat myfile.py | \
vims -e '=\s*"""' 'f\"xxxaeh3UgT\<esc>/\"\"\"\<enter>xxxaeh3UgT' \
'"""' 'V/\"\"\"\<enter>d' -t '%s/eh3UgT/"""/g'
It works for me on your test file!
Explanation:
'=s\*"""' - Match all lines with = and then """ somewhere after
'f\"xxxaeh3UgT\<esc>/\"\"\"\<enter>xxxaeh3UgT' - On that match, move to the first " with f\" (a vim command), then delete the quotes xxx, then enter insert mode a, then type eh3UgT, then move to the next triple quotes \<esc>/\"\"\"\<enter>, then delete them xxx, then type the key again a then eh3UgT.
'"""' - Match all remaining triple quotes
'V/\"\"\"\<enter>d' - Start highlighting V, move to next triple quotes /\"\"\"\<enter>, then delete d.
-t '%s/eh3UgT/"""/g' - Turn on normal vim command-line mode, replace all keys with triple quotes
Here, 1. and 2. act to "save" the string variables by replacing their quotes with a key, then 3. and 4. delete everything contained within triple quotes, then 5. replaces the key back with the triple quotes.
If you are worried about the key eh3UgT being matched elsewhere by accident, just make it longer. If you are worried that this is insecure (say this is a recurring script), then randomly generate the key each time.

Need a help on fetching value based on a key from a config file

I have a file containing similar data
[xxx]
name = xxx
address = bangalore
[yyy]
name = yyy
address = sjc
Please help me getting a regex that I can fetch the address/name value based on xxx or yyy (xxx or yyy and address or name is the input)

You can do something like this with awk if your file is just like that (i.e., the name is the same as the section and it is before the address):
$ awk -v nm='yyy' -F ' *= *' '$1=="name" && $2==nm{infi=1; next}
$1=="address" && infi {print $2; infi=0}' file
sjc
Or, better still you can get the section and then fetch the key, value as they occur and print them and then exit:
$ awk -v sec='yyy' -v key='address' '
BEGIN{
FS=" *= *"
pat=sprintf("^\\[%s\\]", sec)}
$0 ~ pat {secin=$1; next}
NF==2 && $1==key && secin ~ pat {print $2; exit}' file
sjc
If you want to gather all sections with their key/value pairs, you can do (with gawk):
$ gawk 'BEGIN{FS=" *= *"}
/^\[[^\]]+\]/ && NF==1 {sec=$1; next}
NF==2 {d[sec][$1]=$2}
END{ for (k in d){
printf "%s: ",k
for (v in d[k])
printf "\t%s = %s\n", v, d[k][v]
}
}' file
[xxx]: address = bangalore
name = xxx
[yyy]: address = sjc
name = yyy
Config or .ini files can have quoting like csv, so it is best to use a full config file parser. You can use Perl or Python that have robust libraries for parsing .ini or config type files.
Python example:
#!/usr/bin/python
import ConfigParser
config = ConfigParser.ConfigParser()
config.read("/tmp/file")
Then you can grab the sections, the items in each section, or a specific items in a specific section:
>>> config.sections()
['xxx', 'yyy']
>>> config.items("yyy")
[('name', 'yyy'), ('address', 'sjc')]
>>> config.get("xxx", "address")
'bangalore'

Regex to the rescue!
This approach splits the entries into single elements and parses the key-value-pairs afterwards. In the end, you can simply ask your resulting dictionary for ie. values['xxx'].
See a demo on ideone.com.
import re
string = """
[xxx]
name = xxx
address = bangalore
[yyy]
name = yyy
address = sjc
"""
rx_item = re.compile(r'''
^\[(?P<name>[^][]*)\]
.*?
(?=^\[[^][]*\]$|\Z)
''', re.X | re.M | re.DOTALL)
rx_value = re.compile(r'^(?P<key>\w+)\s*=\s*(?P<value>.+)$', re.MULTILINE)
values = {item.group('name'): {
m.group('key'): m.group('value')
for m in rx_value.finditer(item.group(0))}
for item in rx_item.finditer(string)
}
print(values)
# {'xxx': {'name': 'xxx', 'address': 'bangalore'}, 'yyy': {'name': 'yyy', 'address': 'sjc'}}

It's not clear if you're trying to search for the value inside the square brackets or the value of the "name" tag but here's a solution to one possible interpretation of your question:
$ cat tst.awk
BEGIN { FS=" *= *" }
!NF { next }
NF<2 { prt(); k=$0 }
{ map[$1] = $2 }
END { prt() }
function prt() { if (k=="["key"]") print map[tag]; delete map }
$ awk -v key='yyy' -v tag='address' -f tst.awk file
sjc
$ awk -v key='xxx' -v tag='address' -f tst.awk file
bangalore
$ awk -v key='xxx' -v tag='name' -f tst.awk file
xxx

error while concatinating git command string in python

Following this: Find out git branch creator
I making a python script that provides me a sorted set of emails out of the result of
git for-each-ref --format='%(authoremail)%09%(refname)' | sort -k5n -k2M -k3n -k4n | grep remotes | awk -F "\t" '{ printf "%-32s %-27s %s\n", $1, $2, $3 }'
so that I can email them that these are you branches up on remote please delete them.
but when I try to put it together in python I getting error
intitial = "git for-each-ref --format='%(authoremail)%09%(refname)' | sort -k5n -k2M -k3n -k4n | grep remotes | awk -F "
addTab = "\t"
printf = '{ printf "%-32s %-27s %s\n", $1, $2, $3 }'
gitCommnad = "%s%s %s " % (intitial, addTab, printf)
def _exec_git_command(command, verbose=False):
""" Function used to get data out of git commads
and errors in case of failure.
Args:
command(string): string of a git command
verbose(bool): whether to display every command
and its resulting data.
Returns:
(tuple): string of Data and error if present
"""
# converts multiple spaces to single space
command = re.sub(' +',' ',command)
pr = subprocess.Popen(command, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
msg = pr.stdout.read()
err = pr.stderr.read()
if err:
print err
if 'Could not resolve host' in err:
return
if verbose and msg:
print "Executing '%s' %s" % (command, msg)
return msg, err
print _exec_git_command(gitCommnad)

The issue is that you are not putting \t or { printf "%-32s %-27s %s\n", $1, $2, $3 } inside quotes, hence awk reports a Syntax error. You should use -
intitial = "git for-each-ref --format='%(authoremail)%09%(refname)' | sort -k5n -k2M -k3n -k4n | grep remotes | awk -F"
addTab = "\t"
printf = '{ printf "%-32s %-27s %s\n", $1, $2, $3 }'
gitCommnad = "%s \"%s\" '%s' " % (intitial, addTab, printf)