We Keep Coding

How do I take a GET request and turn it into a JSON object?

I don't actually want to make any calls to a website but I have to take this GET request and turn it into a JSON object.
?goal=NOT_GOAL_SETTING&kpi=lfa&sec_kpi=not_being_used&numdays=31&budget=13000000&channel=not_being_used&channel_max=not_being_used&brand=Ram&nameplate=namplate1_nameplate2_&target=800000&nameplate_min_spend=0_0_0_0_0_0_0&nameplate_max_spend=0_0_0_0_0_0_0&max_lfas=70000_100000_4000_400000_90000_15000_2000&search_digital_min_spend=0_0&search_digital_max_spend=0_0&search_digital_min_lfas=0_0&search_digital_max_lfas=0_0
I want every variable that is defined after the = and I want to split the variables by _.
A smaller request is like this-
?variable1=1_2_3_4&variable2=string
What I want is the following:
{"variable1":[1,2,3,4], "variable2":"string"}

I've built a simple function for this before which uses urllib:
import sys
import urllib.parse
def parseGetUrl(url):
result = {}
for data in url.split("&"):
key, val = urllib.parse.unquote(data).split("=")
if val.find('_') != -1:
val = val.split('_')
result[key] = val
return result
if __name__ == "__main__":
url = sys.argv[1][1:] # Gets argument then removes ?
parsedData = parseGetUrl(url)
print(parsedData)
You need to wrap your url inside quotes(")
python3 app.py "?goal=102&value=1_0_0_0"
Do note though that depending on which python version you use urrlib might throw an error:
# python 3
import urrlib.parse
...
key, val = urllib.parse.unquote(data).split("=")
# python 2
import urllib
...
key, val = urllib.unquote(data).split("=")

Python new section in config class

I am trying to write a dynamic config .ini
Where I could add new sections with key and values also I could add key less values.
I have written a code which create a .ini. But the section is coming as 'default'.
Also it is just overwriting the file every time with out adding new section.
I have written a code in python 3 to create a .ini file.
import configparser
"""Generates the configuration file with the config class.
The file is a .ini file"""
class Config:
"""Class for data in uuids.ini file management"""
def __init__(self):
self.config = configparser.ConfigParser()
self.config_file = "conf.ini"
# self.config.read(self.config_file)
def wrt(self, config_name={}):
condict = {
"test": "testval",
'test1': 'testval1',
'test2': 'testval2'
}
for name, val in condict.items():
self.config.set(config_name, name, val)
#self.config.read(self.config_file)
with open(self.config_file, 'w+') as out:
self.config.write(out)
if __name__ == "__main__":
Config().wrt()
I should be able to add new sections with key or with out keys.
Append keys or value.
It should have proper section name.

Some problems with your code:
The usage of mutable objects as default parameters can be a little
tricky and you may see unexpected behavior.
You are using config.set() which is legacy.
you are defaulting config_name to a dictionary, why?
Too much white space :p
You don't need to iterate through the dictionary items to write them using the newer (none legacy) function, as shown below
This should work:
"""Generates the configuration file with the config class.
The file is a .ini file
"""
import configparser
import re
class Config:
"""Class for data in uuids.ini file management."""
def __init__(self):
self.config = configparser.ConfigParser()
self.config_file = "conf.ini"
# self.config.read(self.config_file)
def wrt(self, config_name='DEFAULT', condict=None):
if condict is None:
self.config.add_section(config_name)
return
self.config[config_name] = condict
with open(self.config_file, 'w') as out:
self.config.write(out)
# after writing to file check if keys have no value in the ini file (e.g: key0 = )
# the last character is '=', let us strip it off to only have the key
with open(self.config_file) as out:
ini_data = out.read()
with open(self.config_file, 'w') as out:
new_data = re.sub(r'^(.*?)=\s+$', r'\1', ini_data, 0, re.M)
out.write(new_data)
out.write('\n')
condict = {"test": "testval", 'test1': 'testval1', 'test2': 'testval2'}
c = Config()
c.wrt('my section', condict)
c.wrt('EMPTY')
c.wrt(condict={'key': 'val'})
c.wrt(config_name='NO_VALUE_SECTION', condict={'key0': '', 'key1': ''})
This outputs:
[DEFAULT]
key = val
[my section]
test = testval
test1 = testval1
test2 = testval2
[EMPTY]
[NO_VALUE_SECTION]
key1
key0

How we convert string into json

I want to convert ansible-init file into json. So, I just use this code:
common_shared file:
[sql]
x.com
[yps_db]
y.com
[ems_db]
c.com
[scc_db]
d.com
[all:vars]
server_url="http://x.com/x"
app_host=abc.com
server_url="https://x.com"
[haproxy]
1.1.1.1 manual_hostname=abc instance_id=i-dddd
2.2.2.2 manual_hostname=xyz instance_id=i-cccc
For converting Ansible INI file in JSON:
import json
options= {}
f = open('common_shared')
x = f.read()
config_entries = x.split()
for key,value in zip(config_entries[0::2], config_entries[1::2]):
cleaned_key = key.replace("[",'').replace("]",'')
options[cleaned_key]=value
print json.dumps(options,indent=4,ensure_ascii=False)
But it will print this result:
{
"scc_db": "xxx",
"haproxy": "x.x.x.x",
"manual_hostname=xxx": "instance_id=xx",
"ems_db": "xxx",
"yps_db": "xxx",
"all:vars": "yps_server_url=\"xxx\"",
"1.1.1.5": "manual_hostname=xxx",
"sql": "xxx",
"xxx": "scc_server_url=xxxx\""
}
But I wanted to print result in proper JSON format but not able to understand how. I tried config parser but didn't get help to print it in desired format.

You can use ConfigParser to read in your file, and then do the conversion to a dict to dump.
from ConfigParser import ConfigParser
from collections import defaultdict
config = ConfigParser()
config.readfp(open('/path/to/file.ini'))
def convert_to_dict(config):
config_dict = defaultdict(dict)
for section in config.sections():
for key, value in config.items(section):
config_dict[section][key] = value
return config_dict
print convert_to_dict(config)
EDIT
As you stated in your comment, some line items are just 'things' with no value, the below might work for you.
import re
from collections import defaultdict
SECTION_HEADER_RE = re.compile('^\[.*\]$')
KEY_VALUE_RE = re.compile('^.*=.*$')
def convert_ansible_to_dict(filepath_and_name):
ansible_dict = defaultdict(dict)
with open(filepath_and_name) as input_file:
section_header = None
for line in input_file:
if SECTION_HEADER_RE.findall(line.strip()):
section_header = SECTION_HEADER_RE.findall(line.strip())[0]
elif KEY_VALUE_RE.findall(line.strip()):
if section_header:
# Make sure you have had a header section prior to the line
key, value = KEY_VALUE_RE.findall(line.strip())[0].split('=', 1)
ansible_dict[section_header][key] = value
else:
if line.strip() and section_header:
# As they're just attributes without value, assign the value None
ansible_dict[section_header][line.strip()] = None
return ansible_dict
This is a naive approach, and might not catch all corner cases for you, but perhaps it's a step in the right direction. If you have any 'bare-attributes' prior to your first section header, they will not be included in the dictionary as it would not know where to apportion it to, and the regex for key=value pairs is working on the assumption that there will be only 1 equals sign in the line. I'm sure there might be many other cases I'm not seeing right now, but hopefully this helps.

Christian's answer is the correct one: Use ConfigParser.
Your issue with his solution is that you have an incorrectly formatted INI file.
You need to change all your properties to:
key=value
key: value
e.g.
[sql]
aaaaaaa: true
https://wiki.python.org/moin/ConfigParserExamples
https://en.wikipedia.org/wiki/INI_file#Keys_.28properties.29

using ConfigParser and dictionary in Python

I am trying some basic python scripts using ConfigParser and converting to a dictionary. I am reading a file named "file.cfg" which contains three sections - root, first, second. Currently the code reads the file and converts everything within the file to a dictionary.
My requirement is to convert only sections named "first" and "second" and so on, its key value pair to a dictionary. What would be best way of excluding the section "root" and its key value pair?
import urllib
import urllib2
import base64
import json
import sys
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('file.cfg')
print parser.get('root', 'auth')
config_dict = {}
for sect in parser.sections():
config_dict[sect] = {}
for name, value in parser.items(sect):
config_dict[sect][name] = value
print config_dict
Contents of file.cfg -
~]# cat file.cfg
[root]
username = admin
password = admin
auth = http://192.168.1.1/login
[first]
username = pete
password = sEcReT
url = http://192.168.1.1/list
[second]
username = ron
password = SeCrET
url = http://192.168.1.1/status
Output of the script -
~]# python test4.py
http://192.168.1.1/login
{'second': {'username': 'ron', 'url': 'http://192.168.1.1/status', 'password': 'SeCrEt'}, 'root': {'username': 'admin', 'password': 'admin', 'auth': 'http://192.168.1.1/login'}, 'first': {'username': 'pete', 'url': 'http://192.168.1.1/list', 'password': 'sEcReT'}}

You can remove root section from parser.sections() as follows:
parser.remove_section('root')
Also you don't have to iterate over each pair in each section. You can just convert them to dict:
config_dict = {}
for sect in parser.sections():
config_dict[sect] = dict(parser.items(sect))
Here is one liner:
config_dict = {sect: dict(parser.items(sect)) for sect in parser.sections()}

Bypass the root section by comparison.
for sect in parser.sections():
if sect == 'root':
continue
config_dict[sect] = {}
for name, value in parser.items(sect):
config_dict[sect][name] = value
Edit after acceptance:
ozgur's one liner is a much more concise solution. Upvote from me. If you don't feel like removing sections from the parser directly, the entry can be deleted afterwards.
config_dict = {sect: dict(parser.items(sect)) for sect in parser.sections()} # ozgur's one-liner
del config_dict['root']

Maybe a bit off topic, but ConfigParser is a real pain when in comes to store int, floats and booleans. I prefer using dicts which I dump into configparser.
I also use funtcions to convert between ConfigParser objects and dicts, but those deal with variable type changing, so ConfigParser is happy since it requests strings, and my program is happy since 'False' is not False.
def configparser_to_dict(config: configparser.ConfigParser) -> dict:
config_dict = {}
for section in config.sections():
config_dict[section] = {}
for key, value in config.items(section):
# Now try to convert back to original types if possible
for boolean in ['True', 'False', 'None']:
if value == boolean:
value = bool(boolean)
# Try to convert to float or int
try:
if isinstance(value, str):
if '.' in value:
value = float(value)
else:
value = int(value)
except ValueError:
pass
config_dict[section][key] = value
# Now drop root section if present
config_dict.pop('root', None)
return config_dict
def dict_to_configparser(config_dict: dict) -> configparser.ConfigParser:
config = configparser.ConfigParser()
for section in config_dict.keys():
config.add_section(section)
# Now let's convert all objects to strings so configparser is happy
for key, value in config_dict[section].items():
config[section][key] = str(value)
return config

How can I use a dict in python to build a multi-dimension array with boto and amazon ec2?

I'm attempting to use python and boto to print a list of instances and IPs from Amazon EC2.
I'm used to PHP's nice multidimensional arrays and the similar JSON syntax but I'm having a lot of trouble in python. I tried using AutoVivification as mentioned in What's the best way to initialize a dict of dicts in Python? but am not having luck with access objects in it.
Here is my code:
import sys
import os
import boto
import string
import urllib2
from pprint import pprint
from inspect import getmembers
from datetime import datetime
class AutoVivification(dict):
"""Implementation of perl's autovivification feature."""
def __getitem__(self, item):
try:
return dict.__getitem__(self, item)
except KeyError:
value = self[item] = type(self)()
return value
conn = boto.connect_ec2_endpoint(ec2_url, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
tags = conn.get_all_tags()
myInstances = AutoVivification()
for tag in tags:
if ( tag.res_type == 'instance' and tag.name == 'Name'):
if( tag.res_id ):
myInstances[tag.res_id]['myid'] = tag.res_id
myInstances[tag.res_id]['name'] = tag.value
addrs = conn.get_all_addresses()
for a in addrs:
if( a.instance_id ):
myInstances[a.instance_id]['ip'] = a.public_ip
pprint( myInstances )
for i in myInstances:
print i.name.rjust(25), i.myid
If I do pprint( myInstances ) then I am able to see the multidimensional dict that I have created, but i am not able to access the sub-arrays with i.myid - I get errors like:
AttributeError: 'unicode' object has no attribute 'myid'
AttributeError: 'unicode' object has no attribute 'name'
Doing pprint( myInstances) gives me something like:
{u'i-08148364': {'myid': u'i-18243322', 'name': u'nagios', 'ip': u'1.2.3.4'}}
so I don't understand why I can't access these items.

Your problem is just in how you're trying to access the items:
for i in myInstances:
# i iterates over the KEYS in myInstances
print i.name.rjust(25), i.myid
This attempts, for each key in myInstances, to print i.name.rjust(25) and so on. What you want is to access the value of the given keys (and to use the right Python syntax for accessing dictionary elements):
for i in myInstances:
# i iterates over the KEYS in myInstances
print myInstances[i]["name"].rjust(25), myInstances[i]["myid"]
Or if you don't need the keys at all, just iterate over the values in the first place:
for i in myInstances.values():
# i iterates over the VALUES in myInstances
print i["name"].rjust(25), i["myid"]
Or finally, per request, if you really want to iterate over keys and values at once:
for k, v in myInstances.iteritems():
print k, v["name"].rjust(25), v["myid"]