it's a little bit I'm out of python syntax and I have a problem in reading a .ini file with interpolated values.
this is my ini file:
[DEFAULT]
home=$HOME
test_home=$home
[test]
test_1=$test_home/foo.csv
test_2=$test_home/bar.csv
Those lines
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('config.ini')
print parser.get('test', 'test_1')
does output
$test_home/foo.csv
while I'm expecting
/Users/nkint/foo.csv
EDIT:
I supposed that the $ syntax was implicitly included in the so called string interpolation (referring to the manual):
On top of the core functionality, SafeConfigParser supports
interpolation. This means values can contain format strings which
refer to other values in the same section, or values in a special
DEFAULT section.
But I'm wrong. How to handle this case?
First of all according to the documentation you should use %(test_home)s to interpolate test_home. Moreover the key are case insensitive and you can't use both HOME and home keys. Finally you can use SafeConfigParser(os.environ) to take in account of you environment.
from ConfigParser import SafeConfigParser
import os
parser = SafeConfigParser(os.environ)
parser.read('config.ini')
Where config.ini is
[DEFAULT]
test_home=%(HOME)s
[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv
You can write custom interpolation in case of Python 3:
import configparser
import os
class EnvInterpolation(configparser.BasicInterpolation):
"""Interpolation which expands environment variables in values."""
def before_get(self, parser, section, option, value, defaults):
value = super().before_get(parser, section, option, value, defaults)
return os.path.expandvars(value)
cfg = """
[section1]
key = value
my_path = $PATH
"""
config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])
If you want to expand some environment variables, you can do so using os.path.expandvars before parsing a StringIO stream:
import ConfigParser
import os
import StringIO
with open('config.ini', 'r') as cfg_file:
cfg_txt = os.path.expandvars(cfg_file.read())
config = ConfigParser.ConfigParser()
config.readfp(StringIO.StringIO(cfg_txt))
the trick for proper variable substitution from environment is to use the ${} syntax for the environment variables:
[DEFAULT]
test_home=${HOME}
[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv
ConfigParser.get values are strings, even if you set values as integer or True. But ConfigParser has getint, getfloat and getboolean.
settings.ini
[default]
home=/home/user/app
tmp=%(home)s/tmp
log=%(home)s/log
sleep=10
debug=True
config reader
>>> from ConfigParser import SafeConfigParser
>>> parser = SafeConfigParser()
>>> parser.read('/home/user/app/settings.ini')
>>> parser.get('defaut', 'home')
'/home/user/app'
>>> parser.get('defaut', 'tmp')
'/home/user/app/tmp'
>>> parser.getint('defaut', 'sleep')
10
>>> parser.getboolean('defaut', 'debug')
True
Edit
Indeed you could get name values as environ var if you initialize SafeConfigParser with os.environ. Thanks for the Michele's answer.
Quite late, but maybe it can help someone else looking for the same answers that I had recently. Also, one of the comments was how to fetch Environment variables and values from other sections. Here is how I deal with both converting environment variables and multi-section tags when reading in from an INI file.
INI FILE:
[PKG]
# <VARIABLE_NAME>=<VAR/PATH>
PKG_TAG = Q1_RC1
[DELIVERY_DIRS]
# <DIR_VARIABLE>=<PATH>
NEW_DELIVERY_DIR=${DEL_PATH}\ProjectName_${PKG:PKG_TAG}_DELIVERY
Python Class that uses the ExtendedInterpolation so that you can use the ${PKG:PKG_TAG} type formatting. I add the ability to convert the windows environment vars when I read in INI to a string using the builtin os.path.expandvars() function such as ${DEL_PATH} above.
import os
from configparser import ConfigParser, ExtendedInterpolation
class ConfigParser(object):
def __init__(self):
"""
initialize the file parser with
ExtendedInterpolation to use ${Section:option} format
[Section]
option=variable
"""
self.config_parser = ConfigParser(interpolation=ExtendedInterpolation())
def read_ini_file(self, file='./config.ini'):
"""
Parses in the passed in INI file and converts any Windows environ vars.
:param file: INI file to parse
:return: void
"""
# Expands Windows environment variable paths
with open(file, 'r') as cfg_file:
cfg_txt = os.path.expandvars(cfg_file.read())
# Parses the expanded config string
self.config_parser.read_string(cfg_txt)
def get_config_items_by_section(self, section):
"""
Retrieves the configurations for a particular section
:param section: INI file section
:return: a list of name, value pairs for the options in the section
"""
return self.config_parser.items(section)
def get_config_val(self, section, option):
"""
Get an option value for the named section.
:param section: INI section
:param option: option tag for desired value
:return: Value of option tag
"""
return self.config_parser.get(section, option)
#staticmethod
def get_date():
"""
Sets up a date formatted string.
:return: Date string
"""
return datetime.now().strftime("%Y%b%d")
def prepend_date_to_var(self, sect, option):
"""
Function that allows the ability to prepend a
date to a section variable.
:param sect: INI section to look for variable
:param option: INI search variable under INI section
:return: Void - Date is prepended to variable string in INI
"""
if self.config_parser.get(sect, option):
var = self.config_parser.get(sect, option)
var_with_date = var + '_' + self.get_date()
self.config_parser.set(sect, option, var_with_date)
Based on #alex-markov answer (and code) and #srand9 comment, the following solution works with environment variables and cross-section references.
Note that the interpolation is now based on ExtendedInterpolation to allow cross-sections references and on before_read instead of before_get.
#!/usr/bin/env python3
import configparser
import os
class EnvInterpolation(configparser.ExtendedInterpolation):
"""Interpolation which expands environment variables in values."""
def before_read(self, parser, section, option, value):
value = super().before_read(parser, section, option, value)
return os.path.expandvars(value)
cfg = """
[paths]
foo : ${HOME}
[section1]
key = value
my_path = ${paths:foo}/path
"""
config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])
It seems in the last version 3.5.0, ConfigParser was not reading the env variables, so I end up providing a custom Interpolation based on the BasicInterpolation one.
class EnvInterpolation(BasicInterpolation):
"""Interpolation as implemented in the classic ConfigParser,
plus it checks if the variable is provided as an environment one in uppercase.
"""
def _interpolate_some(self, parser, option, accum, rest, section, map,
depth):
rawval = parser.get(section, option, raw=True, fallback=rest)
if depth > MAX_INTERPOLATION_DEPTH:
raise InterpolationDepthError(option, section, rawval)
while rest:
p = rest.find("%")
if p < 0:
accum.append(rest)
return
if p > 0:
accum.append(rest[:p])
rest = rest[p:]
# p is no longer used
c = rest[1:2]
if c == "%":
accum.append("%")
rest = rest[2:]
elif c == "(":
m = self._KEYCRE.match(rest)
if m is None:
raise InterpolationSyntaxError(option, section,
"bad interpolation variable reference %r" % rest)
var = parser.optionxform(m.group(1))
rest = rest[m.end():]
try:
v = os.environ.get(var.upper())
if v is None:
v = map[var]
except KeyError:
raise InterpolationMissingOptionError(option, section, rawval, var) from None
if "%" in v:
self._interpolate_some(parser, option, accum, v,
section, map, depth + 1)
else:
accum.append(v)
else:
raise InterpolationSyntaxError(
option, section,
"'%%' must be followed by '%%' or '(', "
"found: %r" % (rest,))
The difference between the BasicInterpolation and the EnvInterpolation is in:
v = os.environ.get(var.upper())
if v is None:
v = map[var]
where I'm trying to find the var in the enviornment before checking in the map.
Below is a simple solution that
Can use default value if no environment variable is provided
Overrides variables with environment variables (if found)
needs no custom interpolation implementation
Example:
my_config.ini
[DEFAULT]
HOST=http://www.example.com
CONTEXT=${HOST}/auth/
token_url=${CONTEXT}/oauth2/token
ConfigParser:
import os
import configparser
config = configparser.ConfigParser(interpolation=configparser.ExtendedInterpolation())
ini_file = os.path.join(os.path.dirname(__file__), 'my_config.ini')
# replace variables with environment variables(if exists) before loading ini file
with open(ini_file, 'r') as cfg_file:
cfg_env_txt = os.path.expandvars(cfg_file.read())
config.read_string(cfg_env_txt)
print(config['DEFAULT']['token_url'])
Output:
If no environtment variable $HOST or $CONTEXT is present this config will take the default value
user can override the default value by creating $HOST, $CONTEXT environment variable
works well with docker container
I am getting this error: configparser.ParsingError: Source contains parsing errors: 'my.ini'
although I am getting uncommented-values printed on the terminal.
my.ini:
[my]
# user
root
# passwd
password
I read here that # or ; could be used for commenting. This is how I am doing it:
import configparser
c = configparser.ConfigParser()
c.read('my.ini')
getval = c.items('my')
The problem is root and password don't have a value assigned to them. Since it appears that you want to allow that, just say so when you create the ConfigParser instance:
c = configparser.ConfigParser(allow_no_value=True)
Or in Python 2:
c = ConfigParser.ConfigParser(allow_no_value=True)
I am facing the same issue, but the below line resolves the error the as same above one.
from configparser import ConfigParser
config = ConfigParser(allow_no_value=True)
Thanks.
There is url like the following.
original = 'https://dev.s3.amazonaws.com/production/uploads/2017/11/filename.jpg'
I want to extract /production/uploads/2017/11 only from this.
It is possible to extract just the file name and extract the path including the file name, but please tell me if there is a good way to extract the path excluding the file name.
original_image_name = original.split('/')[-1] # 'filename.jpg'
from urllib.parse import urlparse
original_image_url = urlparse(original)
original_image_path = original_image_url.path # '/production/uploads/2017/11/filename.jpg'
You could use:
import os
os.path.dirname(original_image_path)
A quick way would be to use regular expressions, like this one:
(http[s]?:\/\/)?([^\/\s]+)(.*)
See this for a live demo: https://regex101.com/r/2jsyiI/2
urllib has the adequate tools for this:
https://docs.python.org/3.6/library/urllib.parse.html#module-urllib.parse
You will get expected output
original = "https://dev.s3.amazonaws.com/production/uploads/2017/11/filename.jpg"
test = original.split("/")
test = test[3:]
test = test[:-1]
sd = '/'.join(test)
print sd
OUTPUT
production/uploads/2017/11
This is the part of the mailer.py script:
config = pyfig.Pyfig(config_file)
svnlook = config.general.svnlook #svnlook path
sendmail = config.general.sendmail #sendmail path
From = config.general.from_email #from email address
To = config.general.to_email #to email address
what does this config variable contain? Is there a way to get the value for config variable without pyfig?
In this case config = a pyfig.Pyfig object initialised with the contents of the file named by the content of the string config_file.
To find out what that object does and contains you can either look at the documentation and/or the source code, both here, or you can print out, after the initialisation, e.g.:
config = pyfig.Pyfig(config_file)
print "Config Contains:\n\t", '\n\t'.join(dir(config))
if hasattr(config, "keys"):
print "Config Keys:\n\t", '\n\t'.join(config.keys())
or if you are using Python 3,
config = pyfig.Pyfig(config_file)
print("Config Contains:\n\t", '\n\t'.join(dir(config)))
if hasattr(config, "keys"):
print("Config Keys:\n\t", '\n\t'.join(config.keys()))
To get the same data without pyfig you would need to read and parse at the content of the file referenced by config_file within your own code.
N.B.: Note that pyfig seems to be more or less abandoned - no updates in over 5 years, web site no longer exists, etc., so I would strongly recommend converting the code to use a json configuration file instead.
How can one write comments to a given file within sections?
If I have:
import ConfigParser
with open('./config.ini', 'w') as f:
conf = ConfigParser.ConfigParser()
conf.set('DEFAULT', 'test', 1)
conf.write(f)
I will get the file:
[DEFAULT]
test = 1
But how can I get a file with comments inside [DEFAULT] section, like:
[DEFAULT]
; test comment
test = 1
I know I can write codes to files by doing:
import ConfigParser
with open('./config.ini', 'w') as f:
conf = ConfigParser.ConfigParser()
conf.set('DEFAULT', 'test', 1)
conf.write(f)
f.write('; test comment') # but this gets printed after the section key-value pairs
Is this a possibility with ConfigParser? And I don't want to try another module because I need to keep my program as "stock" as possible.
You can use the allow_no_value option if you have Version >= 2.7
This snippet:
import ConfigParser
config = ConfigParser.ConfigParser(allow_no_value=True)
config.add_section('default_settings')
config.set('default_settings', '; comment here')
config.set('default_settings', 'test', 1)
with open('config.ini', 'w') as fp:
config.write(fp)
config = ConfigParser.ConfigParser(allow_no_value=True)
config.read('config.ini')
print config.items('default_settings')
will create an ini file like this:
[default_settings]
; comment here
test = 1
Update for 3.7
I've been dealing with configparser lately and came across this post. Figured I'd update it with information relevant to 3.7.
Example 1:
config = configparser.ConfigParser(allow_no_value=True)
config.set('SECTION', '; This is a comment.', None)
Example 2:
config = configparser.ConfigParser(allow_no_value=True)
config['SECTION'] = {'; This is a comment':None, 'Option':'Value')
Example 3: If you want to keep your letter case unchanged (default is to convert all option:value pairs to lowercase)
config = configparser.ConfigParser(allow_no_value=True)
config.optionxform = str
config.set('SECTION', '; This Comment Will Keep Its Original Case', None)
Where "SECTION" is the case-sensitive section name you want the comment added to. Using "None" (no quotes) instead of an empty string ('') will allow you to set the comment without leaving a trailing "=".
You can create variable that starts by # or ; character:
conf.set('default_settings', '; comment here', '')
conf.set('default_settings', 'test', 1)
created conf file is
[default_settings]
; comment here =
test = 1
ConfigParser.read function won't parse first value
config = ConfigParser.ConfigParser()
config.read('config.ini')
print config.items('default_settings')
gives
[('test','1')]
You could also use ConfigUpdater. It has many more convenience options to update configuration files in a minimal invasive way.
You would basically do:
from configupdater import ConfigUpdater
updater = ConfigUpdater()
updater.add_section('DEFAULT')
updater.set('DEFAULT', 'test', 1)
updater['DEFAULT']['test'].add_before.comment('test comment', comment_prefix=';')
with open('./config.ini', 'w') as f:
updater.write(f)
Freaky solution for the above :)
Note there is a side-effect, see if that suites you
config = configparser.ConfigParser(comment_prefixes='///')
config.set('section', '# cmt', 'comment goes here')
configparse will treat comments as variables, but real software would not.
This would even preserve the comments on writes done after read of the same ini file, which is a real game changer (disappearing comments are just horrible) :) and you don't need to do allow_no_value=True to allow empty value, just minor visual candy :)
so the ini file would look like:
[section]
# cmt = comment goes here
which pretty much gets the job done :)
please make sure to initialize comment_prefixes with a string that would never appear in your ini file just in case
This worked for me in 3.9.
Side effect on writing the already existing comments. They would not disappear which was normal default, but will be converted to a similar form # first = <remaining>, where first - first word of comment, remaining - remaining of the comment, which would change how file looks, so be carefull...