I am trying some basic python scripts using ConfigParser and converting to a dictionary. I am reading a file named "file.cfg" which contains three sections - root, first, second. Currently the code reads the file and converts everything within the file to a dictionary.
My requirement is to convert only sections named "first" and "second" and so on, its key value pair to a dictionary. What would be best way of excluding the section "root" and its key value pair?
import urllib
import urllib2
import base64
import json
import sys
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('file.cfg')
print parser.get('root', 'auth')
config_dict = {}
for sect in parser.sections():
config_dict[sect] = {}
for name, value in parser.items(sect):
config_dict[sect][name] = value
print config_dict
Contents of file.cfg -
~]# cat file.cfg
[root]
username = admin
password = admin
auth = http://192.168.1.1/login
[first]
username = pete
password = sEcReT
url = http://192.168.1.1/list
[second]
username = ron
password = SeCrET
url = http://192.168.1.1/status
Output of the script -
~]# python test4.py
http://192.168.1.1/login
{'second': {'username': 'ron', 'url': 'http://192.168.1.1/status', 'password': 'SeCrEt'}, 'root': {'username': 'admin', 'password': 'admin', 'auth': 'http://192.168.1.1/login'}, 'first': {'username': 'pete', 'url': 'http://192.168.1.1/list', 'password': 'sEcReT'}}
You can remove root section from parser.sections() as follows:
parser.remove_section('root')
Also you don't have to iterate over each pair in each section. You can just convert them to dict:
config_dict = {}
for sect in parser.sections():
config_dict[sect] = dict(parser.items(sect))
Here is one liner:
config_dict = {sect: dict(parser.items(sect)) for sect in parser.sections()}
Bypass the root section by comparison.
for sect in parser.sections():
if sect == 'root':
continue
config_dict[sect] = {}
for name, value in parser.items(sect):
config_dict[sect][name] = value
Edit after acceptance:
ozgur's one liner is a much more concise solution. Upvote from me. If you don't feel like removing sections from the parser directly, the entry can be deleted afterwards.
config_dict = {sect: dict(parser.items(sect)) for sect in parser.sections()} # ozgur's one-liner
del config_dict['root']
Maybe a bit off topic, but ConfigParser is a real pain when in comes to store int, floats and booleans. I prefer using dicts which I dump into configparser.
I also use funtcions to convert between ConfigParser objects and dicts, but those deal with variable type changing, so ConfigParser is happy since it requests strings, and my program is happy since 'False' is not False.
def configparser_to_dict(config: configparser.ConfigParser) -> dict:
config_dict = {}
for section in config.sections():
config_dict[section] = {}
for key, value in config.items(section):
# Now try to convert back to original types if possible
for boolean in ['True', 'False', 'None']:
if value == boolean:
value = bool(boolean)
# Try to convert to float or int
try:
if isinstance(value, str):
if '.' in value:
value = float(value)
else:
value = int(value)
except ValueError:
pass
config_dict[section][key] = value
# Now drop root section if present
config_dict.pop('root', None)
return config_dict
def dict_to_configparser(config_dict: dict) -> configparser.ConfigParser:
config = configparser.ConfigParser()
for section in config_dict.keys():
config.add_section(section)
# Now let's convert all objects to strings so configparser is happy
for key, value in config_dict[section].items():
config[section][key] = str(value)
return config
Related
I am trying to pass a data back to a URL fetch request. We are using Python 3.x
user_type_data = {'user_type': 'admin',
'user_name': 'myname',
'user_check_flag': 'yes'}
return_data = json.dumps({
l_user_type_data : user_type_data
},default = date_handler)
return return_data
When we do this for a dict I am getting the following error - TypeError("unhashable type: 'dict'"). According to this, it states that we cannot use a dict that is not hashabale - but how do we do this?
How do we fix this?
A valid dictionary key string should be enveloped by quotes or double quotes.
a_dict = {'key': 'value'} # Valid
b_dict = {"key": "value"} # Valid
Or if you wish to assign string that was stored in a variable to be the dictionary key, you can do this instead:
st = "key"
a_dict = dict()
a_dict[st] = 'value'
Since json_dumps requires a valid python dictionary, you may need to rearrange your code.
If the l_user_type_data is a variable contains a string, you should do:
temp_dict = dict()
temp_dict[l_user_type_data] = user_type_data
result = json.dumps(temp_dict, default = date_handler)
Otherwise, if l_user_type_data is a string for the key, just simply enclose that with either single quote or double quotes.
return_data = json.dumps({
"l_user_type_data" : user_type_data
},default = date_handler)
I am trying to write a dynamic config .ini
Where I could add new sections with key and values also I could add key less values.
I have written a code which create a .ini. But the section is coming as 'default'.
Also it is just overwriting the file every time with out adding new section.
I have written a code in python 3 to create a .ini file.
import configparser
"""Generates the configuration file with the config class.
The file is a .ini file"""
class Config:
"""Class for data in uuids.ini file management"""
def __init__(self):
self.config = configparser.ConfigParser()
self.config_file = "conf.ini"
# self.config.read(self.config_file)
def wrt(self, config_name={}):
condict = {
"test": "testval",
'test1': 'testval1',
'test2': 'testval2'
}
for name, val in condict.items():
self.config.set(config_name, name, val)
#self.config.read(self.config_file)
with open(self.config_file, 'w+') as out:
self.config.write(out)
if __name__ == "__main__":
Config().wrt()
I should be able to add new sections with key or with out keys.
Append keys or value.
It should have proper section name.
Some problems with your code:
The usage of mutable objects as default parameters can be a little
tricky and you may see unexpected behavior.
You are using config.set() which is legacy.
you are defaulting config_name to a dictionary, why?
Too much white space :p
You don't need to iterate through the dictionary items to write them using the newer (none legacy) function, as shown below
This should work:
"""Generates the configuration file with the config class.
The file is a .ini file
"""
import configparser
import re
class Config:
"""Class for data in uuids.ini file management."""
def __init__(self):
self.config = configparser.ConfigParser()
self.config_file = "conf.ini"
# self.config.read(self.config_file)
def wrt(self, config_name='DEFAULT', condict=None):
if condict is None:
self.config.add_section(config_name)
return
self.config[config_name] = condict
with open(self.config_file, 'w') as out:
self.config.write(out)
# after writing to file check if keys have no value in the ini file (e.g: key0 = )
# the last character is '=', let us strip it off to only have the key
with open(self.config_file) as out:
ini_data = out.read()
with open(self.config_file, 'w') as out:
new_data = re.sub(r'^(.*?)=\s+$', r'\1', ini_data, 0, re.M)
out.write(new_data)
out.write('\n')
condict = {"test": "testval", 'test1': 'testval1', 'test2': 'testval2'}
c = Config()
c.wrt('my section', condict)
c.wrt('EMPTY')
c.wrt(condict={'key': 'val'})
c.wrt(config_name='NO_VALUE_SECTION', condict={'key0': '', 'key1': ''})
This outputs:
[DEFAULT]
key = val
[my section]
test = testval
test1 = testval1
test2 = testval2
[EMPTY]
[NO_VALUE_SECTION]
key1
key0
I use following YAML data:
Document:
InPath: /home/me
OutPath: /home/me
XLOutFile: TestFile1.xlsx
Sheets:
- Sheet: Test123
InFile: Test123.MQSC
Server: Testsystem1
- Sheet: Test345
InFile: Test345.MQSC
Server: Testsystem2
Title:
A: "Server Name"
B: "MQ Version"
C: "Broker Version"
Fields:
A: ServerName
B: MQVersion
C: BrokerVersion
and following code:
import yaml
class cfgReader():
def __init__(self):
self.stream = ""
self.ymldata = ""
self.ymlkey = ""
self.ymld = ""
def read(self,infilename):
self.stream = self.stream = file(infilename, 'r') #Read the yamlfile
self.ymldata = yaml.load(self.stream) #Instanciate yaml object and parse the input "stream".
def docu(self):
print self.ymldata
print self.ymldata['Sheets']
for self.ymlkey in self.ymldata['Document']: #passes String to iterator
print self.ymlkey
for sheets in self.ymldata['Sheets']: #passes Dictionary to iterator
print sheets['Sheet']
for title in self.ymldata['Title']:
print title
for fields in self.ymldata['Fields']:
print fields
The print output is:
{'Fields': {'A': 'ServerName', 'C': 'BrokerVersion', 'B': 'MQVersion'}, 'Document': {'XLOutFile': 'TestFile1.xlsx', 'InPath': '/home/me', 'OutPath': '/home/me'}, 'Sheets': [{'Sheet': 'Test123', 'InFile': 'Test123.MQSC', 'Server': 'Testsystem1'}, {'Sheet': 'Test345', 'InFile': 'Test345.MQSC', 'Server': 'Testsystem2'}], 'Title': {'A': 'Server Name', 'C': 'Broker Version', 'B': 'MQ Version'}}
[{'Sheet': 'Test123', 'InFile': 'Test123.MQSC', 'Server': 'Testsystem1'}, {'Sheet': 'Test345', 'InFile': 'Test345.MQSC', 'Server': 'Testsystem2'}]
X
I
O
Test123
Test345
A
C
B
A
C
B
I could not find out how to control the way data is passed to the iterator. What I want is to pass it as dictionaries so that I can access the value through the key. This works for "Sheets" but I don't understand why. The documentation was not describing it clearly : http://pyyaml.org/wiki/PyYAMLDocumentation
In your code self.ymldata['Sheets'] is a list of dictionaries because your YAML source for that:
- Sheet: Test123
InFile: Test123.MQSC
Server: Testsystem1
- Sheet: Test345
InFile: Test345.MQSC
Server: Testsystem2
is a sequence of mappings (and this is the value for the key Sheets of the top-level mapping in your YAML file).
The values for the other top-level keys are all mappings (and not sequences of mappings), which get loaded as Python dict. And if you iterate over a dict as you do, you get the key values.
If you don't want to iterate over these dictionaries then you should not start a for loop. You might want to test what the value for a toplevel keys is and then act accordingly, e.g. to print out all dictionaries loaded from the YAML file except for the top-level mapping do:
import ruamel.yaml as yaml
class CfgReader():
def __init__(self):
self.stream = ""
self.ymldata = ""
self.ymlkey = ""
self.ymld = ""
def read(self, infilename):
self.stream = open(infilename, 'r') # Read the yamlfile
self.ymldata = yaml.load(self.stream) # Instanciate yaml object and parse the input "stream".
def docu(self):
for k in self.ymldata:
v = self.ymldata[k]
if isinstance(v, list):
for elem in v:
print(elem)
else:
print(v)
cfg_reader = CfgReader()
cfg_reader.read('in.yaml')
cfg_reader.docu()
which prints:
{'InFile': 'Test123.MQSC', 'Sheet': 'Test123', 'Server': 'Testsystem1'}
{'InFile': 'Test345.MQSC', 'Sheet': 'Test345', 'Server': 'Testsystem2'}
{'B': 'MQVersion', 'A': 'ServerName', 'C': 'BrokerVersion'}
{'B': 'MQ Version', 'A': 'Server Name', 'C': 'Broker Version'}
{'XLOutFile': 'TestFile1.xlsx', 'InPath': '/home/me', 'OutPath': '/home/me'}
Please also note some general things, you should be aware off
I use ruamel.yaml (disclaimer: I am the author of that package), which supports YAML 1.2 (PyYAML supports the 1.1 standard from 2005). For your purposes they act the same.
don't use file() it is not available in Python3, use open()
assigning the same value twice to the same attribute makes no sense (self.stream = self.stream = ...)
your opened file/stream never gets closed, you might want to consider using
with open(infilename) as self.stream:
self.ymldata = yaml.load(self.stream)
class names, by convention, should start with an upper case character.
I am getting JIRA data using the following python code,
how do I store the response for more than one key (my example shows only one KEY but in general I get lot of data) and print only the values corresponding to total,key, customfield_12830, summary
import requests
import json
import logging
import datetime
import base64
import urllib
serverURL = 'https://jira-stability-tools.company.com/jira'
user = 'username'
password = 'password'
query = 'project = PROJECTNAME AND "Build Info" ~ BUILDNAME AND assignee=ASSIGNEENAME'
jql = '/rest/api/2/search?jql=%s' % urllib.quote(query)
response = requests.get(serverURL + jql,verify=False,auth=(user, password))
print response.json()
response.json() OUTPUT:-
http://pastebin.com/h8R4QMgB
From the the link you pasted to pastebin and from the json that I saw, its a you issues as list containing key, fields(which holds custom fields), self, id, expand.
You can simply iterate through this response and extract values for keys you want. You can go like.
data = response.json()
issues = data.get('issues', list())
x = list()
for issue in issues:
temp = {
'key': issue['key'],
'customfield': issue['fields']['customfield_12830'],
'total': issue['fields']['progress']['total']
}
x.append(temp)
print(x)
x is list of dictionaries containing the data for fields you mentioned. Let me know if I have been unclear somewhere or what I have given is not what you are looking for.
PS: It is always advisable to use dict.get('keyname', None) to get values as you can always put a default value if key is not found. For this solution I didn't do it as I just wanted to provide approach.
Update: In the comments you(OP) mentioned that it gives attributerror.Try this code
data = response.json()
issues = data.get('issues', list())
x = list()
for issue in issues:
temp = dict()
key = issue.get('key', None)
if key:
temp['key'] = key
fields = issue.get('fields', None)
if fields:
customfield = fields.get('customfield_12830', None)
temp['customfield'] = customfield
progress = fields.get('progress', None)
if progress:
total = progress.get('total', None)
temp['total'] = total
x.append(temp)
print(x)
I want to convert ansible-init file into json. So, I just use this code:
common_shared file:
[sql]
x.com
[yps_db]
y.com
[ems_db]
c.com
[scc_db]
d.com
[all:vars]
server_url="http://x.com/x"
app_host=abc.com
server_url="https://x.com"
[haproxy]
1.1.1.1 manual_hostname=abc instance_id=i-dddd
2.2.2.2 manual_hostname=xyz instance_id=i-cccc
For converting Ansible INI file in JSON:
import json
options= {}
f = open('common_shared')
x = f.read()
config_entries = x.split()
for key,value in zip(config_entries[0::2], config_entries[1::2]):
cleaned_key = key.replace("[",'').replace("]",'')
options[cleaned_key]=value
print json.dumps(options,indent=4,ensure_ascii=False)
But it will print this result:
{
"scc_db": "xxx",
"haproxy": "x.x.x.x",
"manual_hostname=xxx": "instance_id=xx",
"ems_db": "xxx",
"yps_db": "xxx",
"all:vars": "yps_server_url=\"xxx\"",
"1.1.1.5": "manual_hostname=xxx",
"sql": "xxx",
"xxx": "scc_server_url=xxxx\""
}
But I wanted to print result in proper JSON format but not able to understand how. I tried config parser but didn't get help to print it in desired format.
You can use ConfigParser to read in your file, and then do the conversion to a dict to dump.
from ConfigParser import ConfigParser
from collections import defaultdict
config = ConfigParser()
config.readfp(open('/path/to/file.ini'))
def convert_to_dict(config):
config_dict = defaultdict(dict)
for section in config.sections():
for key, value in config.items(section):
config_dict[section][key] = value
return config_dict
print convert_to_dict(config)
EDIT
As you stated in your comment, some line items are just 'things' with no value, the below might work for you.
import re
from collections import defaultdict
SECTION_HEADER_RE = re.compile('^\[.*\]$')
KEY_VALUE_RE = re.compile('^.*=.*$')
def convert_ansible_to_dict(filepath_and_name):
ansible_dict = defaultdict(dict)
with open(filepath_and_name) as input_file:
section_header = None
for line in input_file:
if SECTION_HEADER_RE.findall(line.strip()):
section_header = SECTION_HEADER_RE.findall(line.strip())[0]
elif KEY_VALUE_RE.findall(line.strip()):
if section_header:
# Make sure you have had a header section prior to the line
key, value = KEY_VALUE_RE.findall(line.strip())[0].split('=', 1)
ansible_dict[section_header][key] = value
else:
if line.strip() and section_header:
# As they're just attributes without value, assign the value None
ansible_dict[section_header][line.strip()] = None
return ansible_dict
This is a naive approach, and might not catch all corner cases for you, but perhaps it's a step in the right direction. If you have any 'bare-attributes' prior to your first section header, they will not be included in the dictionary as it would not know where to apportion it to, and the regex for key=value pairs is working on the assumption that there will be only 1 equals sign in the line. I'm sure there might be many other cases I'm not seeing right now, but hopefully this helps.
Christian's answer is the correct one: Use ConfigParser.
Your issue with his solution is that you have an incorrectly formatted INI file.
You need to change all your properties to:
key=value
key: value
e.g.
[sql]
aaaaaaa: true
https://wiki.python.org/moin/ConfigParserExamples
https://en.wikipedia.org/wiki/INI_file#Keys_.28properties.29