Related
I want to dump a python object into a YAML string that only contains a single line. However, ruamel.yaml.safe_dump appends newline characters as well as (sometimes) '...'
Dumping for example list or dict objects appends a single newline character:
import ruamel.yaml as yaml
yaml.safe_dump([1, None], default_flow_style=None)
Outputs: '[1, null]\n'
The output I need is: '[1, null]'
When dumping "scalar" objects, even more is appended:
import ruamel.yaml as yaml
yaml.safe_dump(None, default_flow_style=None)
Outputs: 'null\n...\n'
The output I need is: 'null'
Both expected outputs are valid YAML syntax I think, i.e.
yaml.safe_load('null')
correctly returns None.
Is there any way (besides manually removing the trailing line breaks and '...', which is very hacky) to achieve what I want?
You should not be using the old API in ruamel.yaml , it's deprecated and about to be removed.
If you want everything on one line, you should probably use .default_flow_style = True
depending on how complex your data-structure can become, and widen the output so you
don't get linewraps.
Contrary to JSON, YAML normally appends a newline, so it is best to just transform the output
to hack anything of after the first one.
import sys
import ruamel.yaml
class DSL:
def __init__(self):
pass
#property
def yaml(self):
try:
return self._yaml
except AttributeError:
pass
yaml = ruamel.yaml.YAML(typ='safe')
yaml.default_flow_style = True
yaml.width = 2048
self._yaml = yaml
return yaml
def __call__(self, data, stream=sys.stdout):
def strip_nl(s):
result, rest = s.split('\n', 1)
if rest not in ['', '...\n']:
print('multi-line YAML output', repr(rest))
sys.exit(1)
return result
self.yaml.dump(data, stream, transform=strip_nl)
dsl = DSL()
sys.stdout.write('|')
dsl([1, None])
sys.stdout.write('|\n')
sys.stdout.write('|')
dsl(None)
sys.stdout.write('|\n')
sys.stdout.write('|')
dsl(dict(a=[1, None], b=42))
sys.stdout.write('|\n')
which gives:
|[1, null]|
|null|
|{a: [1, null], b: 42}|
The typical ConfigParser generated file looks like:
[Section]
bar=foo
[Section 2]
bar2= baz
Now, is there a way to index lists like, for instance:
[Section 3]
barList={
item1,
item2
}
Related question: Python’s ConfigParser unique keys per section
I am using a combination of ConfigParser and JSON:
[Foo]
fibs: [1,1,2,3,5,8,13]
just read it with:
>>> json.loads(config.get("Foo","fibs"))
[1, 1, 2, 3, 5, 8, 13]
You can even break lines if your list is long (thanks #peter-smit):
[Bar]
files_to_check = [
"/path/to/file1",
"/path/to/file2",
"/path/to/another file with space in the name"
]
Of course i could just use JSON, but i find config files much more readable, and the [DEFAULT] Section very handy.
There is nothing stopping you from packing the list into a delimited string and then unpacking it once you get the string from the config. If you did it this way your config section would look like:
[Section 3]
barList=item1,item2
It's not pretty but it's functional for most simple lists.
I recently implemented this with a dedicated section in a config file for a list:
[paths]
path1 = /some/path/
path2 = /another/path/
...
and using config.items( "paths" ) to get an iterable list of path items, like so:
path_items = config.items( "paths" )
for key, path in path_items:
#do something with path
One thing a lot of people don't know is that multi-line configuration-values are allowed. For example:
;test.ini
[hello]
barlist =
item1
item2
The value of config.get('hello','barlist') will now be:
"\nitem1\nitem2"
Which you easily can split with the splitlines method (don't forget to filter empty items).
If we look to a big framework like Pyramid they are using this technique:
def aslist_cronly(value):
if isinstance(value, string_types):
value = filter(None, [x.strip() for x in value.splitlines()])
return list(value)
def aslist(value, flatten=True):
""" Return a list of strings, separating the input based on newlines
and, if flatten=True (the default), also split on spaces within
each line."""
values = aslist_cronly(value)
if not flatten:
return values
result = []
for value in values:
subvalues = value.split()
result.extend(subvalues)
return result
Source
Myself, I would maybe extend the ConfigParser if this is a common thing for you:
class MyConfigParser(ConfigParser):
def getlist(self,section,option):
value = self.get(section,option)
return list(filter(None, (x.strip() for x in value.splitlines())))
def getlistint(self,section,option):
return [int(x) for x in self.getlist(section,option)]
Note that there are a few things to look out for when using this technique
New lines that are items should start with whitespace (e.g. a space or a tab)
All following lines that start with whitespace are considered to be part of the previous item. Also if it has an = sign or if it starts with a ; following the whitespace.
No mention of the converters kwarg for ConfigParser() in any of these answers was rather disappointing.
According to the documentation you can pass a dictionary to ConfigParser that will add a get method for both the parser and section proxies. So for a list:
example.ini
[Germ]
germs: a,list,of,names, and,1,2, 3,numbers
Parser example:
cp = ConfigParser(converters={'list': lambda x: [i.strip() for i in x.split(',')]})
cp.read('example.ini')
cp.getlist('Germ', 'germs')
['a', 'list', 'of', 'names', 'and', '1', '2', '3', 'numbers']
cp['Germ'].getlist('germs')
['a', 'list', 'of', 'names', 'and', '1', '2', '3', 'numbers']
This is my personal favorite as no subclassing is necessary and I don't have to rely on an end user to perfectly write JSON or a list that can be interpreted by ast.literal_eval.
If you want to literally pass in a list then you can use:
ast.literal_eval()
For example configuration:
[section]
option=["item1","item2","item3"]
The code is:
import ConfigParser
import ast
my_list = ast.literal_eval(config.get("section", "option"))
print(type(my_list))
print(my_list)
output:
<type'list'>
["item1","item2","item3"]
I landed here seeking to consume this...
[global]
spys = richard.sorge#cccp.gov, mata.hari#deutschland.gov
The answer is to split it on the comma and strip the spaces:
SPYS = [e.strip() for e in parser.get('global', 'spys').split(',')]
To get a list result:
['richard.sorge#cccp.gov', 'mata.hari#deutschland.gov']
It may not answer the OP's question exactly but might be the simple answer some people are looking for.
This is what I use for lists:
config file content:
[sect]
alist = a
b
c
code :
l = config.get('sect', 'alist').split('\n')
it work for strings
in case of numbers
config content:
nlist = 1
2
3
code:
nl = config.get('sect', 'alist').split('\n')
l = [int(nl) for x in nl]
thanks.
So another way, which I prefer, is to just split the values, for example:
#/path/to/config.cfg
[Numbers]
first_row = 1,2,4,8,12,24,36,48
Could be loaded like this into a list of strings or integers, as follows:
import configparser
config = configparser.ConfigParser()
config.read('/path/to/config.cfg')
# Load into a list of strings
first_row_strings = config.get('Numbers', 'first_row').split(',')
# Load into a list of integers
first_row_integers = [int(x) for x in config.get('Numbers', 'first_row').split(',')]
This method prevents you from needing to wrap your values in brackets to load as JSON.
I completed similar task in my project with section with keys without values:
import configparser
# allow_no_value param says that no value keys are ok
config = configparser.ConfigParser(allow_no_value=True)
# overwrite optionxform method for overriding default behaviour (I didn't want lowercased keys)
config.optionxform = lambda optionstr: optionstr
config.read('./app.config')
features = list(config['FEATURES'].keys())
print(features)
Output:
['BIOtag', 'TextPosition', 'IsNoun', 'IsNomn']
app.config:
[FEATURES]
BIOtag
TextPosition
IsNoun
IsNomn
Only primitive types are supported for serialization by config parser. I would use JSON or YAML for that kind of requirement.
To take Grr's answer (my favorite) a step further, instead of enclosing list items in quotes in the .ini file, you can use the map function. This allows you to pythonically specify list item datatypes.
Config file:
[section]
listKey1: 1001, 1002, 1003
listKey2: AAAA, BBBB, CCCC
Code:
cfgFile = 'config.ini'
parser = ConfigParser(converters={'list': lambda x: [i.strip() for i in x.split(',')]})
parser.read(cfgFile)
list1 = list(map(int, parser.getlist('section', 'listKey1')))
list2 = list(map(str, parser.getlist('section', 'listKey2')))
print(list1)
print(list2)
Output:
[1001, 1002, 1003]
['AAAA', 'BBBB', 'CCCC']
If this is your config.ini:
[Section 3]
barList=item1,item2
Then with configparser you could do this:
from configparser import ConfigParser
config = ConfigParser()
config.read('config.ini')
my_list = config['Section 3']['barList'].split(',')
You will get:
my_list = ['item1', 'item2']
The split()-method will return a list, see Python string docs.
If you have white spaces in your config.ini like this:
[Section 3]
barList= item1, item2
Then you'd better do this:
my_list = [x.strip() for x in config['Section 3']['barList'].split(',')]
If your items are numbers (integers for instance), just apply:
my_list_of_ints = list(map(int, my_list))
You will get:
my_list_of_ints = [item1, item2]
I faced the same problem in the past. If you need more complex lists, consider creating your own parser by inheriting from ConfigParser. Then you would overwrite the get method with that:
def get(self, section, option):
""" Get a parameter
if the returning value is a list, convert string value to a python list"""
value = SafeConfigParser.get(self, section, option)
if (value[0] == "[") and (value[-1] == "]"):
return eval(value)
else:
return value
With this solution you will also be able to define dictionaries in your config file.
But be careful! This is not as safe: this means anyone could run code through your config file. If security is not an issue in your project, I would consider using directly python classes as config files. The following is much more powerful and expendable than a ConfigParser file:
class Section
bar = foo
class Section2
bar2 = baz
class Section3
barList=[ item1, item2 ]
import ConfigParser
import os
class Parser(object):
"""attributes may need additional manipulation"""
def __init__(self, section):
"""section to retun all options on, formatted as an object
transforms all comma-delimited options to lists
comma-delimited lists with colons are transformed to dicts
dicts will have values expressed as lists, no matter the length
"""
c = ConfigParser.RawConfigParser()
c.read(os.path.join(os.path.dirname(__file__), 'config.cfg'))
self.section_name = section
self.__dict__.update({k:v for k, v in c.items(section)})
#transform all ',' into lists, all ':' into dicts
for key, value in self.__dict__.items():
if value.find(':') > 0:
#dict
vals = value.split(',')
dicts = [{k:v} for k, v in [d.split(':') for d in vals]]
merged = {}
for d in dicts:
for k, v in d.items():
merged.setdefault(k, []).append(v)
self.__dict__[key] = merged
elif value.find(',') > 0:
#list
self.__dict__[key] = value.split(',')
So now my config.cfg file, which could look like this:
[server]
credentials=username:admin,password:$3<r3t
loggingdirs=/tmp/logs,~/logs,/var/lib/www/logs
timeoutwait=15
Can be parsed into fine-grained-enough objects for my small project.
>>> import config
>>> my_server = config.Parser('server')
>>> my_server.credentials
{'username': ['admin'], 'password', ['$3<r3t']}
>>> my_server.loggingdirs:
['/tmp/logs', '~/logs', '/var/lib/www/logs']
>>> my_server.timeoutwait
'15'
This is for very quick parsing of simple configs, you lose all ability to fetch ints, bools, and other types of output without either transforming the object returned from Parser, or re-doing the parsing job accomplished by the Parser class elsewhere.
json.loads & ast.literal_eval seems to be working but simple list within config is treating each character as byte so returning even square bracket....
meaning if config has fieldvalue = [1,2,3,4,5]
then config.read(*.cfg)
config['fieldValue'][0] returning [ in place of 1
As mentioned by Peter Smit (https://stackoverflow.com/a/11866695/7424596)
You might want to extend ConfigParser, in addition, an Interpolator can be used to automatically convert into and from the list.
For reference at the bottom you can find code which automatically converts config like:
[DEFAULT]
keys = [
Overall cost structure, Capacity, RAW MATERIALS,
BY-PRODUCT CREDITS, UTILITIES, PLANT GATE COST,
PROCESS DESCRIPTION, AT 50% CAPACITY, PRODUCTION COSTS,
INVESTMENT, US$ MILLION, PRODUCTION COSTS, US ¢/LB,
VARIABLE COSTS, PRODUCTION COSTS, MAINTENANCE MATERIALS
]
So if you request keys you will get:
<class 'list'>: ['Overall cost structure', 'Capacity', 'RAW MATERIALS', 'BY-PRODUCT CREDITS', 'UTILITIES', 'PLANT GATE COST', 'PROCESS DESCRIPTION', 'AT 50% CAPACITY', 'PRODUCTION COSTS', 'INVESTMENT', 'US$ MILLION', 'PRODUCTION COSTS', 'US ¢/LB', 'VARIABLE COSTS', 'PRODUCTION COSTS', 'MAINTENANCE MATERIALS']
Code:
class AdvancedInterpolator(Interpolation):
def before_get(self, parser, section, option, value, defaults):
is_list = re.search(parser.LIST_MATCHER, value)
if is_list:
return parser.getlist(section, option, raw=True)
return value
class AdvancedConfigParser(ConfigParser):
_DEFAULT_INTERPOLATION = AdvancedInterpolator()
LIST_SPLITTER = '\s*,\s*'
LIST_MATCHER = '^\[([\s\S]*)\]$'
def _to_list(self, str):
is_list = re.search(self.LIST_MATCHER, str)
if is_list:
return re.split(self.LIST_SPLITTER, is_list.group(1))
else:
return re.split(self.LIST_SPLITTER, str)
def getlist(self, section, option, conv=lambda x:x.strip(), *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self._get_conv(
section, option,
lambda value: [conv(x) for x in self._to_list(value)],
raw=raw,
vars=vars,
fallback=fallback,
**kwargs
)
def getlistint(self, section, option, *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self.getlist(section, option, int, raw=raw, vars=vars,
fallback=fallback, **kwargs)
def getlistfloat(self, section, option, *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self.getlist(section, option, float, raw=raw, vars=vars,
fallback=fallback, **kwargs)
def getlistboolean(self, section, option, *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self.getlist(section, option, self._convert_to_boolean,
raw=raw, vars=vars, fallback=fallback, **kwargs)
Ps keep in mind importance of indentdation. As reads in ConfigParser doc string:
Values can span multiple lines, as long as they are indented deeper
than the first line of the value. Depending on the parser's mode, blank
lines may be treated as parts of multiline values or ignored.
you can use list in config file then parse it in python
from ast import literal_eval
literal_eval("[1,2,3,4]")
import json
json.loads("[1,2,3,4]")
and also you can use json file behind your config file like this:
your config file :
[A]
json_dis = .example.jason
--------------------
your code :
import configparser
config = configparser.ConfigParser()
config.read('config.ini')
# getting items of section A
config.items('A')
# result is a list of key-values
An improvement on split(',') might be to treat the comma separated values as a record in a CSV file
import csv
my_list = list(csv.reader([config['Section 3']['barList']], dialect=csv.excel))[0]
You can configure a dialect to parse whatever style of CSV you like.
I want to convert a string into a dictionary. I saved this dictionary previously in a text file.
The problem is now, that I am not sure, how the structure of the keys are. The values are generated with Counter(dictionaryName). The dictionary is really large, so I cannot check every key to see how it would be possible.
The keys can contain simple quotes like ', double quotes ", commas and maybe other characters. So is there any possibility to convert it back into a dictionary?
For example this is stored in the file:
Counter({'element0':512, "'4,5'element1":50, '4:55foobar':23,...})
I found previous solutions with for example json, but I have problems with the double quotes and I cannot simply split for the commas.
If you trust the source, load from collections import Counter and eval() the string
How about something like:
>> from collections import Counter
>> line = '''Counter({'element0':512, "'4,5'element1":50, '4:55foobar':23})'''
>> D = eval(line)
>> D
Counter({"'4,5'element1": 50, '4:55foobar': 23, 'element0': 512})
You could remove the Counter( and ) parts, then parse the rest with ast.literal_eval as long as it only involves basic Python data types:
import ast
def parse_Counter_string(s):
s = s.strip()
if not (s.startswith('Counter(') and s.endswith(')')):
raise ValueError('String does not match expected format')
# Counter( is 8 characters
# 12345678
s = s[8:-1]
return Counter(ast.literal_eval(s))
In the future, I recommend picking a different way to serialize your data.
you can use demjson library for doing this, you can have the text directly in your program
import demjson
counter = demjson.decode("enter your text here")
if it is in the file ,you can do the following steps :
WD = dirname(realpath(__file__))
file = open(WD, "filename"), "r")
counter = demjson.decode(file.read())
file.close()
I have a dictionary in the form
{'from.x': 'from.changed.x',...}
possibly quite big, and I have to substitute in text files accordingly to that dictionary in a quite big directory structure.
I didn't find anything which might any nice solution and I end up:
using os.walk
iterating through the dictionary
writing everything out
WIth something like:
def fix_imports(top_dir, not_ui_keys):
"""Walk through the directory and substitute the wrong imports
"""
repl = {}
for n in not_ui_keys:
# interleave a model in between
dotted = extract_dotted(n)
if dotted:
repl[dotted] = add_model(dotted)
for root, dirs, files in walk(top_dir):
py_files = [path.join(root, x) for x in files if x.endswith('.py')]
for py in py_files:
res = replace_text(open(py).read(), repl)
def replace_text(orig_text, replace_map):
res = orig_text
# now try to grep all the keys, using a translate maybe
# with a dictionary of the replacements
for to_replace in replace_map:
res.replace(to_replace, replace_map[to_replace])
# now print the differences
for un in unified_diff(res.splitlines(), orig_text.splitlines()):
print(un)
return res
Is there any better/nicer/faster way to do it?
EDIT:
Clarifying a bit the problem, the substitution are generated from a function, and they are all in the form:
{'x.y.z': 'x.y.added.z', 'x.b.a': 'x.b.added.a'}
And yes, sure I should better use regexps, I just thought I didn't need them this time.
I don't think it can help much, however, because I can't really formalize the whole range of substitutions with only one (or multiple) regexps..
I would write the first function using generators:
def fix_imports(top_dir, not_ui_keys):
"""Walk through the directory and substitute the wrong imports """
from itertools import imap,ifilter
gen = ifilter(None,imap(extract_dotted, not_ui_keys))
repl = dict((dotted,add_model(dotted)) for dotted in gen)
py_files = (path.join(root, x)
for root, dirs, files in walk(top_dir)
for x in files if x[-3:]=='.py')
for py in py_files:
with open(py) as opf:
res = replace_text(opf.read(), repl)
x[-3:]=='.py' is faster than x.endswith('.py')
Thank you everyone, and about the problem of substituting from a mapping in many files, I think I have a working solution:
def replace_map_to_text(repl_map, text_lines):
"""Take a dictionary with the replacements needed and a list of
files and return a list with the substituted lines
"""
res = []
concat_st = "(%s)" % "|".join(repl_map.keys())
# '.' in non raw regexp means one of any characters, so must be
# quoted ore we need a way to make the string a raw string
concat_st = concat_st.replace('.', '\.')
combined_regexp = re.compile(concat_st)
for line in text_lines:
found = combined_regexp.search(line)
if found:
expr = found.group(1)
new_line = re.sub(expr, repl_map[expr], line)
logger.info("from line %s to line %s" % (line, new_line))
res.append(new_line)
else:
res.append(line)
return res
def test_replace_string():
lines = ["from psi.io.api import x",
"from psi.z import f"]
expected = ["from psi.io.model.api import x",
"from psi.model.z import f"]
mapping = {'psi.io.api': 'psi.io.model.api',
'psi.z': 'psi.model.z'}
assert replace_map_to_text(mapping, lines) == expected
In short I compose a big regexp in the form
(first|second|third)
Then I search for it in every line and substitute with re.sub if something was found.
Still a bit rough but the simple test after works fine.
EDIT: fixed a nasty bug in the concatenation, because if it's not a raw string '.' means only one character, not a '.'
The typical ConfigParser generated file looks like:
[Section]
bar=foo
[Section 2]
bar2= baz
Now, is there a way to index lists like, for instance:
[Section 3]
barList={
item1,
item2
}
Related question: Python’s ConfigParser unique keys per section
I am using a combination of ConfigParser and JSON:
[Foo]
fibs: [1,1,2,3,5,8,13]
just read it with:
>>> json.loads(config.get("Foo","fibs"))
[1, 1, 2, 3, 5, 8, 13]
You can even break lines if your list is long (thanks #peter-smit):
[Bar]
files_to_check = [
"/path/to/file1",
"/path/to/file2",
"/path/to/another file with space in the name"
]
Of course i could just use JSON, but i find config files much more readable, and the [DEFAULT] Section very handy.
There is nothing stopping you from packing the list into a delimited string and then unpacking it once you get the string from the config. If you did it this way your config section would look like:
[Section 3]
barList=item1,item2
It's not pretty but it's functional for most simple lists.
I recently implemented this with a dedicated section in a config file for a list:
[paths]
path1 = /some/path/
path2 = /another/path/
...
and using config.items( "paths" ) to get an iterable list of path items, like so:
path_items = config.items( "paths" )
for key, path in path_items:
#do something with path
One thing a lot of people don't know is that multi-line configuration-values are allowed. For example:
;test.ini
[hello]
barlist =
item1
item2
The value of config.get('hello','barlist') will now be:
"\nitem1\nitem2"
Which you easily can split with the splitlines method (don't forget to filter empty items).
If we look to a big framework like Pyramid they are using this technique:
def aslist_cronly(value):
if isinstance(value, string_types):
value = filter(None, [x.strip() for x in value.splitlines()])
return list(value)
def aslist(value, flatten=True):
""" Return a list of strings, separating the input based on newlines
and, if flatten=True (the default), also split on spaces within
each line."""
values = aslist_cronly(value)
if not flatten:
return values
result = []
for value in values:
subvalues = value.split()
result.extend(subvalues)
return result
Source
Myself, I would maybe extend the ConfigParser if this is a common thing for you:
class MyConfigParser(ConfigParser):
def getlist(self,section,option):
value = self.get(section,option)
return list(filter(None, (x.strip() for x in value.splitlines())))
def getlistint(self,section,option):
return [int(x) for x in self.getlist(section,option)]
Note that there are a few things to look out for when using this technique
New lines that are items should start with whitespace (e.g. a space or a tab)
All following lines that start with whitespace are considered to be part of the previous item. Also if it has an = sign or if it starts with a ; following the whitespace.
No mention of the converters kwarg for ConfigParser() in any of these answers was rather disappointing.
According to the documentation you can pass a dictionary to ConfigParser that will add a get method for both the parser and section proxies. So for a list:
example.ini
[Germ]
germs: a,list,of,names, and,1,2, 3,numbers
Parser example:
cp = ConfigParser(converters={'list': lambda x: [i.strip() for i in x.split(',')]})
cp.read('example.ini')
cp.getlist('Germ', 'germs')
['a', 'list', 'of', 'names', 'and', '1', '2', '3', 'numbers']
cp['Germ'].getlist('germs')
['a', 'list', 'of', 'names', 'and', '1', '2', '3', 'numbers']
This is my personal favorite as no subclassing is necessary and I don't have to rely on an end user to perfectly write JSON or a list that can be interpreted by ast.literal_eval.
If you want to literally pass in a list then you can use:
ast.literal_eval()
For example configuration:
[section]
option=["item1","item2","item3"]
The code is:
import ConfigParser
import ast
my_list = ast.literal_eval(config.get("section", "option"))
print(type(my_list))
print(my_list)
output:
<type'list'>
["item1","item2","item3"]
I landed here seeking to consume this...
[global]
spys = richard.sorge#cccp.gov, mata.hari#deutschland.gov
The answer is to split it on the comma and strip the spaces:
SPYS = [e.strip() for e in parser.get('global', 'spys').split(',')]
To get a list result:
['richard.sorge#cccp.gov', 'mata.hari#deutschland.gov']
It may not answer the OP's question exactly but might be the simple answer some people are looking for.
This is what I use for lists:
config file content:
[sect]
alist = a
b
c
code :
l = config.get('sect', 'alist').split('\n')
it work for strings
in case of numbers
config content:
nlist = 1
2
3
code:
nl = config.get('sect', 'alist').split('\n')
l = [int(nl) for x in nl]
thanks.
So another way, which I prefer, is to just split the values, for example:
#/path/to/config.cfg
[Numbers]
first_row = 1,2,4,8,12,24,36,48
Could be loaded like this into a list of strings or integers, as follows:
import configparser
config = configparser.ConfigParser()
config.read('/path/to/config.cfg')
# Load into a list of strings
first_row_strings = config.get('Numbers', 'first_row').split(',')
# Load into a list of integers
first_row_integers = [int(x) for x in config.get('Numbers', 'first_row').split(',')]
This method prevents you from needing to wrap your values in brackets to load as JSON.
I completed similar task in my project with section with keys without values:
import configparser
# allow_no_value param says that no value keys are ok
config = configparser.ConfigParser(allow_no_value=True)
# overwrite optionxform method for overriding default behaviour (I didn't want lowercased keys)
config.optionxform = lambda optionstr: optionstr
config.read('./app.config')
features = list(config['FEATURES'].keys())
print(features)
Output:
['BIOtag', 'TextPosition', 'IsNoun', 'IsNomn']
app.config:
[FEATURES]
BIOtag
TextPosition
IsNoun
IsNomn
Only primitive types are supported for serialization by config parser. I would use JSON or YAML for that kind of requirement.
To take Grr's answer (my favorite) a step further, instead of enclosing list items in quotes in the .ini file, you can use the map function. This allows you to pythonically specify list item datatypes.
Config file:
[section]
listKey1: 1001, 1002, 1003
listKey2: AAAA, BBBB, CCCC
Code:
cfgFile = 'config.ini'
parser = ConfigParser(converters={'list': lambda x: [i.strip() for i in x.split(',')]})
parser.read(cfgFile)
list1 = list(map(int, parser.getlist('section', 'listKey1')))
list2 = list(map(str, parser.getlist('section', 'listKey2')))
print(list1)
print(list2)
Output:
[1001, 1002, 1003]
['AAAA', 'BBBB', 'CCCC']
If this is your config.ini:
[Section 3]
barList=item1,item2
Then with configparser you could do this:
from configparser import ConfigParser
config = ConfigParser()
config.read('config.ini')
my_list = config['Section 3']['barList'].split(',')
You will get:
my_list = ['item1', 'item2']
The split()-method will return a list, see Python string docs.
If you have white spaces in your config.ini like this:
[Section 3]
barList= item1, item2
Then you'd better do this:
my_list = [x.strip() for x in config['Section 3']['barList'].split(',')]
If your items are numbers (integers for instance), just apply:
my_list_of_ints = list(map(int, my_list))
You will get:
my_list_of_ints = [item1, item2]
I faced the same problem in the past. If you need more complex lists, consider creating your own parser by inheriting from ConfigParser. Then you would overwrite the get method with that:
def get(self, section, option):
""" Get a parameter
if the returning value is a list, convert string value to a python list"""
value = SafeConfigParser.get(self, section, option)
if (value[0] == "[") and (value[-1] == "]"):
return eval(value)
else:
return value
With this solution you will also be able to define dictionaries in your config file.
But be careful! This is not as safe: this means anyone could run code through your config file. If security is not an issue in your project, I would consider using directly python classes as config files. The following is much more powerful and expendable than a ConfigParser file:
class Section
bar = foo
class Section2
bar2 = baz
class Section3
barList=[ item1, item2 ]
import ConfigParser
import os
class Parser(object):
"""attributes may need additional manipulation"""
def __init__(self, section):
"""section to retun all options on, formatted as an object
transforms all comma-delimited options to lists
comma-delimited lists with colons are transformed to dicts
dicts will have values expressed as lists, no matter the length
"""
c = ConfigParser.RawConfigParser()
c.read(os.path.join(os.path.dirname(__file__), 'config.cfg'))
self.section_name = section
self.__dict__.update({k:v for k, v in c.items(section)})
#transform all ',' into lists, all ':' into dicts
for key, value in self.__dict__.items():
if value.find(':') > 0:
#dict
vals = value.split(',')
dicts = [{k:v} for k, v in [d.split(':') for d in vals]]
merged = {}
for d in dicts:
for k, v in d.items():
merged.setdefault(k, []).append(v)
self.__dict__[key] = merged
elif value.find(',') > 0:
#list
self.__dict__[key] = value.split(',')
So now my config.cfg file, which could look like this:
[server]
credentials=username:admin,password:$3<r3t
loggingdirs=/tmp/logs,~/logs,/var/lib/www/logs
timeoutwait=15
Can be parsed into fine-grained-enough objects for my small project.
>>> import config
>>> my_server = config.Parser('server')
>>> my_server.credentials
{'username': ['admin'], 'password', ['$3<r3t']}
>>> my_server.loggingdirs:
['/tmp/logs', '~/logs', '/var/lib/www/logs']
>>> my_server.timeoutwait
'15'
This is for very quick parsing of simple configs, you lose all ability to fetch ints, bools, and other types of output without either transforming the object returned from Parser, or re-doing the parsing job accomplished by the Parser class elsewhere.
json.loads & ast.literal_eval seems to be working but simple list within config is treating each character as byte so returning even square bracket....
meaning if config has fieldvalue = [1,2,3,4,5]
then config.read(*.cfg)
config['fieldValue'][0] returning [ in place of 1
As mentioned by Peter Smit (https://stackoverflow.com/a/11866695/7424596)
You might want to extend ConfigParser, in addition, an Interpolator can be used to automatically convert into and from the list.
For reference at the bottom you can find code which automatically converts config like:
[DEFAULT]
keys = [
Overall cost structure, Capacity, RAW MATERIALS,
BY-PRODUCT CREDITS, UTILITIES, PLANT GATE COST,
PROCESS DESCRIPTION, AT 50% CAPACITY, PRODUCTION COSTS,
INVESTMENT, US$ MILLION, PRODUCTION COSTS, US ¢/LB,
VARIABLE COSTS, PRODUCTION COSTS, MAINTENANCE MATERIALS
]
So if you request keys you will get:
<class 'list'>: ['Overall cost structure', 'Capacity', 'RAW MATERIALS', 'BY-PRODUCT CREDITS', 'UTILITIES', 'PLANT GATE COST', 'PROCESS DESCRIPTION', 'AT 50% CAPACITY', 'PRODUCTION COSTS', 'INVESTMENT', 'US$ MILLION', 'PRODUCTION COSTS', 'US ¢/LB', 'VARIABLE COSTS', 'PRODUCTION COSTS', 'MAINTENANCE MATERIALS']
Code:
class AdvancedInterpolator(Interpolation):
def before_get(self, parser, section, option, value, defaults):
is_list = re.search(parser.LIST_MATCHER, value)
if is_list:
return parser.getlist(section, option, raw=True)
return value
class AdvancedConfigParser(ConfigParser):
_DEFAULT_INTERPOLATION = AdvancedInterpolator()
LIST_SPLITTER = '\s*,\s*'
LIST_MATCHER = '^\[([\s\S]*)\]$'
def _to_list(self, str):
is_list = re.search(self.LIST_MATCHER, str)
if is_list:
return re.split(self.LIST_SPLITTER, is_list.group(1))
else:
return re.split(self.LIST_SPLITTER, str)
def getlist(self, section, option, conv=lambda x:x.strip(), *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self._get_conv(
section, option,
lambda value: [conv(x) for x in self._to_list(value)],
raw=raw,
vars=vars,
fallback=fallback,
**kwargs
)
def getlistint(self, section, option, *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self.getlist(section, option, int, raw=raw, vars=vars,
fallback=fallback, **kwargs)
def getlistfloat(self, section, option, *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self.getlist(section, option, float, raw=raw, vars=vars,
fallback=fallback, **kwargs)
def getlistboolean(self, section, option, *, raw=False, vars=None,
fallback=_UNSET, **kwargs):
return self.getlist(section, option, self._convert_to_boolean,
raw=raw, vars=vars, fallback=fallback, **kwargs)
Ps keep in mind importance of indentdation. As reads in ConfigParser doc string:
Values can span multiple lines, as long as they are indented deeper
than the first line of the value. Depending on the parser's mode, blank
lines may be treated as parts of multiline values or ignored.
you can use list in config file then parse it in python
from ast import literal_eval
literal_eval("[1,2,3,4]")
import json
json.loads("[1,2,3,4]")
and also you can use json file behind your config file like this:
your config file :
[A]
json_dis = .example.jason
--------------------
your code :
import configparser
config = configparser.ConfigParser()
config.read('config.ini')
# getting items of section A
config.items('A')
# result is a list of key-values
An improvement on split(',') might be to treat the comma separated values as a record in a CSV file
import csv
my_list = list(csv.reader([config['Section 3']['barList']], dialect=csv.excel))[0]
You can configure a dialect to parse whatever style of CSV you like.