We Keep Coding

Transform JSON String to Dictionary using shell without escape

I am calling a python script from the shell, with one input argument.
python main.py """{"key1":"value1", "key2":"value2"}"""
All keys and values are strings. Once in python, I would like to convert the JSON string to a dictionary, so I have acess to the values by using the keys.
I tried the following
import json
import sys
dict_in = json.loads(sys.argv[1])
But dict_in would end up a string like that {key1:value1, key2:value2}
So it seems like I need to find a way to pass the string with quotation marks from the shell to python. I can not use escape characters since the string is provided by a different program.
Is there an elegant way to solve this?

I've found a python 2 module which can handle such cases.
Suppose you have this string:
>>> str = '{foo: bar, id: 23}'
Then you can use yaml as follows:
>>> import yaml
>>> dict = yaml.load(str)
>>> dict
{'foo': 'bar', 'id': 23}
>>> dict['foo']
'bar'
Now you have what you needed.
More info (and also python 3 support and etc.) can be found here: https://pyyaml.org/wiki/PyYAMLDocumentation

Not sure if what you passing in is important but you can pass following and get desired output:
"{\"key1\":\"value1\", \"key2\":\"value2\"}"
or
'{"key1":"value1", "key2":"value2"}'
Here is the code and output:
$cat json_convert.py
import json
import sys
dict_in = json.loads(sys.argv[1])
print (dict_in)
$ python json_convert.py '{"key1":"value1", "key2":"value2"}'
{'key1': 'value1', 'key2': 'value2'}
Also what you are passing """{"key1":"value1", "key2":"value2"}""" translates to "" + "{" + key1 + ":" + value1 + ", " + + key2 + ":" + value2 + "}" + "" if you are asking bash, if you were calling the function with that as a argument from the python itself you would get the desired results.
So really goes down to what you are calling it from.
If you still like quotes go ahead and pass """{"'"key1"'":"'"value1"'", "'"key2"'":"'"value2"'"}""" to get desired result :)

use either:
$ your_other_program | python main.py
to send the output of the other program to python, or use base64.b64encode(json.dumps(blah)) and you'll get pretty code like
'eyJtQXV0b21hdGljVGVzdExpc3QiOiBbeyJtWSI6IDguMTE0MTA1LCAibU5hbWUiOiAiYWNjZWxlcmF0b3JFbnRpdHkiLCAibVRlc3RTdGF0dXMiOiB0cnVlLCAibVgiOiAzLjgwNDM1MTgsICJtWiI6IC0zLjM4OTU3MjF9LCB7Im1OYW1lIjogImJhcm9tZXRlckVudGl0eSIsICJtVmFsdWUiOiAwLCAibVRlc3RTdGF0dXMiOiBmYWxzZX1dLCAibUF1dG9tYXRpY1Rlc3RDb21wbGV0ZWQiOiB0cnVlfQ=='
to put in the command line, and then decode it back from base64 into JSON.
Or, even better, use:
$ your_other_program >output_file.tmp
$ python main.py < output_file.tmp
$ rm output_file.tmp

Ok so here is what is my test script:
print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(yaml.load(sys.argv[1]))
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")
dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)
This is what I type into the console
python test_script.py """{foo:bar, id:23}"""
And This is the output
original sys.argv output
"{foo:bar, id:23}"
first transformation
{foo:bar, id:23}
This is the dictionary
{'foo': 'bar', 'id': 23}
This only workds if I use tripple quotes ("""). If I use (") or (') to define the input string I get an error.
Alternatively one can remove the (") from the sys.argv[1]
print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(sys.argv[1])[1:-1]
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")
dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)

Trying to generate an array with an exec function in Python throws an error

I am trying to generate the next code in Python:
atribute_name = 'atr1'
exec("list_%s = []", atribute_name)
So the expected result should be:
list_atr1 = []
But when I execute it I get the next error message:
TypeError: exec: arg 2 must be a dictionary or None
Why is it happening?

Instead of comma, you should replace it with % for string concatenation.
exec("list_%s = []" % atribute_name)

You are passing second argument for exec function.
Instead you just need to format your string.
Below sample code can help you.
atribute_name = 'atr1'
str1 = "list_%s = []" % atribute_name
print str1
exec(str1)
print list_atr1

Appending print commands with .decode() method

I have a script of about 300 lines (part of which is pasted below) with a lot of print commands. I am trying to cleanup the output it produces. If I leave it the way it is then all the print commands print bytes with \r\n on to the console.
I figured if I add .decode('utf-8') in front of the variable that I need to print then the output is what I should be expecting (uni-code string). For example, compare print (data1) and print (data3) commands below. What I want to do is to go through all of the code and append .decode() to every print statement.
All the print commands are in this format: Print (dataxxxx)
import telnetlib
import time
import sys
import random
from xlwt import Workbook
shelfIp = "10.10.10.10"
shelf = "33"
print ("Shelf IP is: " + str(shelfIp))
print ("Shelf number is: " + str(shelf))
def addCard():
tn = telnetlib.Telnet(shelfIp)
### Telnet session
tn.read_until(b"<",5)
cmd = "ACT-USER::ADMIN:ONE::ADMIN;"
tn.write(bytes(cmd,encoding="UTF-8"))
data1 = tn.read_until(b"ONE COMPLD", 5)
print (data1.decode('utf-8'))
### Entering second network element
cmd = "ENT-CARD::CARD" + shelf + "-" + shelf + ":TWO:xyz:;"
tn.write(bytes(cmd,encoding="UTF-8"))
data3 = tn.read_until(b"TWO COMPLD", 5)
print (data3)
### Entering third network element
cmd = "ENT-CARD::CARD-%s-%s:ADM:ABC:;" %(shelf,shelf)
tn.write(bytes(cmd,encoding="UTF-8"))
dataAmp = tn.read_until(b"ADM COMPLD", 5)
print (dataAmp)
tn.close()
addCard()

If you are looking into doing some sort of find-replace on the code, you can try this:
import re
f = open('script.py','rb')
script = f.read()
f.close()
newscript = re.sub("(print\(.*)\)", "\g<1>.decode('utf-8'))", script)
f = open('script.py', 'wb')
f.write(newscript)
f.close()
What I did in the regular expression:
Catch text that contains print(......) and save the print(..... part into group 1
Replace the text after the print(.... which is ) with: .decode('utf-8')) using the syntax \g<1> which takes the saved group number 1 and put that as the prefix in the replaced text.

Appending .decode() to print() statements will fail because .decode() is a string method.
>>> x=u"testing"
>>> print(x).decode('utf-8')
testing
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'decode'
You must apply .decode('utf-8') to the variables you wish to decode, which is not easily accomplished using regex based tools.

Getting a list by sys.argv

I pass mylist as an argument to be get after by sys.argv I do this:
mylist = str(list)
nbre = str(nbre)
comm = 'python2.6 file.py ' + mylist + ' ' + nbre + ' &'
os.system(comm)
In file.py, I am expected to get mylist by this way and which contains [machine1,machine2] but when doing:
mylist = sys.argv[1]
I get [machine1, which is wrong. When I display sys.argv I found:
['file.py','[machine1,','machine2]','1']
I didn't understand why my list is composed like that?

Apart from this being a terrible way to communicate a list from one python script to another, you'd need to use quotes around the list to prevent it from being split by the shell:
comm = 'python2.6 file.py "%s" "%s" &' % (mylist, nbre)
I've used string formatting to put the quotes around mylist and nbre.
You really want to look into the subprocess module to invoke other processes without the shell getting in the way.

Not all of arguments converted during string formatting

Im wrtiting a script which saves the current date and time as a filename but I get an error stating "TypeError: not all arguments converted during string formatting" I am new to Python andmay of missed something obvious. Code below:
from subprocess import Popen
import datetime
today = datetime.date.today()
today = str(today)
print today
f = open("%s.sql", "w" % (today))
x = Popen(["mysqldump", "-u", "root", "-pucsdrv", "normalisationtion"], stdout = f)
x.wait()
f.close()

You're putting the string formatting in the wrong place; it needs to be right after the string that's being formatted:
f = open("%s.sql" % (today), "w")
It's legal to not pass any formatting arguments, like you did with "%s.sql", but it's not legal to pass arguments but not the right amount ("w" % (today) passes one, but there's no string formatting in "w", so you get an error that not all of the arguments were used)

f = open("%s.sql" % today, "w")