We Keep Coding

Transform JSON String to Dictionary using shell without escape

I am calling a python script from the shell, with one input argument.
python main.py """{"key1":"value1", "key2":"value2"}"""
All keys and values are strings. Once in python, I would like to convert the JSON string to a dictionary, so I have acess to the values by using the keys.
I tried the following
import json
import sys
dict_in = json.loads(sys.argv[1])
But dict_in would end up a string like that {key1:value1, key2:value2}
So it seems like I need to find a way to pass the string with quotation marks from the shell to python. I can not use escape characters since the string is provided by a different program.
Is there an elegant way to solve this?

I've found a python 2 module which can handle such cases.
Suppose you have this string:
>>> str = '{foo: bar, id: 23}'
Then you can use yaml as follows:
>>> import yaml
>>> dict = yaml.load(str)
>>> dict
{'foo': 'bar', 'id': 23}
>>> dict['foo']
'bar'
Now you have what you needed.
More info (and also python 3 support and etc.) can be found here: https://pyyaml.org/wiki/PyYAMLDocumentation

Not sure if what you passing in is important but you can pass following and get desired output:
"{\"key1\":\"value1\", \"key2\":\"value2\"}"
or
'{"key1":"value1", "key2":"value2"}'
Here is the code and output:
$cat json_convert.py
import json
import sys
dict_in = json.loads(sys.argv[1])
print (dict_in)
$ python json_convert.py '{"key1":"value1", "key2":"value2"}'
{'key1': 'value1', 'key2': 'value2'}
Also what you are passing """{"key1":"value1", "key2":"value2"}""" translates to "" + "{" + key1 + ":" + value1 + ", " + + key2 + ":" + value2 + "}" + "" if you are asking bash, if you were calling the function with that as a argument from the python itself you would get the desired results.
So really goes down to what you are calling it from.
If you still like quotes go ahead and pass """{"'"key1"'":"'"value1"'", "'"key2"'":"'"value2"'"}""" to get desired result :)

use either:
$ your_other_program | python main.py
to send the output of the other program to python, or use base64.b64encode(json.dumps(blah)) and you'll get pretty code like
'eyJtQXV0b21hdGljVGVzdExpc3QiOiBbeyJtWSI6IDguMTE0MTA1LCAibU5hbWUiOiAiYWNjZWxlcmF0b3JFbnRpdHkiLCAibVRlc3RTdGF0dXMiOiB0cnVlLCAibVgiOiAzLjgwNDM1MTgsICJtWiI6IC0zLjM4OTU3MjF9LCB7Im1OYW1lIjogImJhcm9tZXRlckVudGl0eSIsICJtVmFsdWUiOiAwLCAibVRlc3RTdGF0dXMiOiBmYWxzZX1dLCAibUF1dG9tYXRpY1Rlc3RDb21wbGV0ZWQiOiB0cnVlfQ=='
to put in the command line, and then decode it back from base64 into JSON.
Or, even better, use:
$ your_other_program >output_file.tmp
$ python main.py < output_file.tmp
$ rm output_file.tmp

Ok so here is what is my test script:
print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(yaml.load(sys.argv[1]))
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")
dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)
This is what I type into the console
python test_script.py """{foo:bar, id:23}"""
And This is the output
original sys.argv output
"{foo:bar, id:23}"
first transformation
{foo:bar, id:23}
This is the dictionary
{'foo': 'bar', 'id': 23}
This only workds if I use tripple quotes ("""). If I use (") or (') to define the input string I get an error.
Alternatively one can remove the (") from the sys.argv[1]
print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(sys.argv[1])[1:-1]
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")
dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)

Trying to generate an array with an exec function in Python throws an error

I am trying to generate the next code in Python:
atribute_name = 'atr1'
exec("list_%s = []", atribute_name)
So the expected result should be:
list_atr1 = []
But when I execute it I get the next error message:
TypeError: exec: arg 2 must be a dictionary or None
Why is it happening?

Instead of comma, you should replace it with % for string concatenation.
exec("list_%s = []" % atribute_name)

You are passing second argument for exec function.
Instead you just need to format your string.
Below sample code can help you.
atribute_name = 'atr1'
str1 = "list_%s = []" % atribute_name
print str1
exec(str1)
print list_atr1

Sending two element from the list in python to the unix shell command

I have a file which contains the list of filesnames:
List:
Sample1_R1_L1.bam
Sample1_R2_L1.bam
Sample2_R1_L1.bam
Sample2_R2_L1.bam
.......
I want to run a unix command that merges each pair of files:
$ samtools merge Sample1_merged_output.bam Sample1_R1_L1.bam Sample1_R2_L1.bam
I was thinking I can achieve this by using the for loop in python which takes two elements from the list of file names and runs the "subprocess" to call the unix command. I found a post which helped to access two elements at a time but I can not pass the names of the file to the unix shell:
for i,d in enumerate(list):
if i < (len(list) - 1):
print d + ' ' + list[i+1]
# print d + ' ' + list[i+1]
any suggestions to achieve this are welcome. Thanks.

You should use argv to get arguments.
from sys import argv
print argv[1]
Result will be:
merge
Then your code would be something like:
list2merge = argv[2:]
for i,d in enumerate(list2merge):
print str(i) + ' ' + str(d)
UPDATE
You should not use list as variable name, since it's reserved.

concatenating string in python

I am converting a command line to a python string. The command line is:
../src/clus -INFILE=../input/tua40.sq -OUTPUT=OUT
The python statement is:
c_dir = '~/prj/clus/'
c_bin = c_dir + 'src/clus'
c_data = c_dir + 'input/tua40.sq'
c = LiveProcess()
c.executable = c_bin
c.cwd = c_dir
c.cmd = [c.executable] + ['-INFILE=', 'c_data, '-OUTPUT=OUT']
Problem is the c.cmd at the end looks like
~/prj/clus/src/clus -INFILE= ~/prj/clus/input/tua40.sq ...
Not that there is a 'space' after '=' which causes the program to report an error.
How can I concatenate '=' to the path?

LiveProcess is expecting an argv-style list of arguments. Where you want to make one argument, you need to provide one string. So use concatenation to make the string:
c.cmd = [c.executable] + ['-INFILE='+c_data, '-OUTPUT=OUT']
Also, no need for the list addition:
c.cmd = [c.executable, '-INFILE='+c_data, '-OUTPUT=OUT']

Why don't you just concatenate string like this:
a = 'A'+'B'
then
a == 'AB'
that is in your example
['-INFILE=' + c_data, '-OUTPUT=OUT']

Given that it looks like you're concatenating paths, you should be using os.path.join, not regular string concat.

Try this:
c.cmd = [c.executable] + ['-INFILE='+c_data, '-OUTPUT=OUT']

Python: Combining string and lists

I have a list of counters
counters = ['76195087', '963301809', '830123644', '60989448', '0', '0', '76195087', '4006066839', '390361581', '101817210', '0', '0']
and I would like to create a string using some of these counters....
cmd = 'my_command' + counters[0:1]
But I find that I am unable to concatenate strings and lists.
What I must have at the end is a string that looks like this:
my_command 76195087
How do I get these numbers out of their list and get them to behave like strings?

You can join strings in a list with, well, join:
cmd = 'my_command' + ''.join(counters[:1])
But you shouldn't construct a command like that in the first place and give it to os.popen or os.system. Instead, use the subprocess module, which handles the internals (and escapes problematic values):
import subprocess
# You may want to set some options in the following line ...
p = subprocess.Popen(['my_command'] + counters[:1])
p.communicate()

If you just want a single element of the list, just index that element:
cmd = 'my_command ' + counters[0]
If you want to join several elements, use the 'join()' method of strings:
cmd = 'my_command ' + " ".join(counters[0:2]) # add spaces between elements

If you just want to append a single counter, you can use
"my_command " + counters[0]
or
"%s %s" % (command, counters[0])
where command is a variable containing the command as a string. If you want to append more than one counter, ' '.join() is your friend:
>>> ' '.join([command] + counters[:3])
'my_command 76195087 963301809 830123644'

You have to access an element of the list, not sublists of the list, like this:
cmd = 'my_command' + counters[0]
Since I guess you're interested in using all the counters at some point, use a variable to store the index you're currently using, and increment it where you see fit (possibly inside a loop)
idx = 0
cmd1 = 'my_command' + counters[idx]
idx += 1
cmd2 = 'my_command' + counters[idx]
Of course, being careful of not incrementing the index variable beyond the size of the list.