We Keep Coding

How should I create and read a user editable configuration file in ~/.config or similar?

I am planning a command line Python application that I intend to distribute through PyPi.
When the application is installed with pip, I want to create a user-editable configuration file in the appropriate location on the user's filesystem.
For example, in Ubuntu, the file would be something like ~/.config/foo/config.ini
On installation I want to create the file (if possible) and be able to specify another config file to use instead with a command line parameter.
What is the usual scheme for getting this done?

I think appdirs package on PyPI is what you need, isn’t it?

You should use the appdirs module for this as it reliably handles differences across different platforms:
>>> from appdirs import user_config_dir
>>> appname = "SuperApp"
>>> appauthor = "Acme"
>>> user_config_dir(appname)
'/home/trentm/.config/SuperApp'

A project such as platformdirs can help with such a task. It implements Freedesktop's "XDG Base Directory Specification".
Don't create the file at installation time though. It is preferable to create the file when it is actually needed, this might be during the first run of the application for example.
Something like the following should get you started with the configuration directory:
>>> import platformdirs
>>> platformdirs.user_config_dir('foo')
'/home/sinoroc/.config/foo'
Or without external dependency, it could roughly look like this:
#!/usr/bin/env python3
import argparse
import os
import pathlib
import platform
def get_config_file_path(project_name, file_name):
path = None
config_path = None
platform_system = platform.system()
if platform_system == 'Windows':
if 'APPDATA' in os.environ:
config_path = pathlib.Path(os.environ['APPDATA'])
else:
config_path = pathlib.Path.home().joinpath('AppData', 'Roaming')
elif platform_system == 'Linux':
if 'XDG_CONFIG_HOME' in os.environ:
config_path = pathlib.Path(os.environ['XDG_CONFIG_HOME'])
else:
config_path = pathlib.Path.home().joinpath('.config')
if config_path:
path = config_path.joinpath(project_name, file_name)
return path
def main():
default_config_file_path = get_config_file_path('foo', 'config.ini')
args_parser = argparse.ArgumentParser()
args_parser.add_argument(
'--config', '-c',
default=str(default_config_file_path),
type=argparse.FileType('r'),
)
args = args_parser.parse_args()
if __name__ == '__main__':
main()

Error while installing debian packages programmitically using apt_pkg

I've built some debian packages and put them on my host. I want to install these packages in a python script. So I've written the install function using apt_pkg as follows :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import apt
import apt_pkg
import sys
class my_pkg_manager:
def __init__(self):
apt_pkg.init()
self.cache = apt_pkg.Cache()
self.sources = apt_pkg.SourceList()
self.pkg_records = apt_pkg.PackageRecords(self.cache)
self.depcache = apt_pkg.DepCache(self.cache)
self.pkg_manager = apt_pkg.PackageManager(self.depcache)
self.fetcher = apt_pkg.Acquire()
def find_my_package(self):
for pkg in self.cache.packages:
if len(pkg.version_list)>0:
package_site = pkg.version_list[0].file_list[0][0].site
if package_site == 'my_host_address.com':
if pkg.name == 'my_pkg_name':
return pkg
return None
def install_package(self, pkg):
self.depcache.mark_install(pkg)
self.sources.read_main_list()
self.pkg_manager.get_archives(self.fetcher, self.sources,
self.pkg_records)
log_file = open('install_log','w')
res = self.pkg_manager.do_install(log_file.fileno())
if res == self.pkg_manager.RESULT_COMPLETED:
print('result completed!')
elif res == self.pkg_manager.RESULT_INCOMPLETE:
print('result incomplete!')
else:
print('result failed!')
def run(self):
my_pkg = self.find_my_package()
if my_pkg != None:
self.install_package(my_pkg)
else:
print('Can't find package!')
if __name__ == '__main__':
my_package_manager = my_pkg_manager()
my_package_manager.run()
If I use apt-get install, packages will be installed without any problem, but with this script the following error occurs :
res = self.pkg_manager.do_install(log_file.fileno()) apt_pkg.Error:
E:Internal Error, Pathname to install is not absolute
'myPackage_1.0.0_all.deb'
I've placed .deb file and Packages.gz in a directory named 'debian' in my host and added the following line to my sources.list :
deb http://my_host_address.com debian/
I don't know what is wrong with my script?

deb http://my_host_address.com debian/
should be:
deb http://my_host_address.com /debian/
Absolute path needs to start with a slash.

Compile Python code with Openpyxl into executable using PyInstaller Error cannot import '__versions__' [duplicate]

My traceback from running pandas takes me to:
site-packages\pandas\io\excel.py line 58, in get_writer
AttributeError: 'module' object has no attribute '__version__'
I found this link to a git issue in the PyInstaller repo
https://github.com/pyinstaller/pyinstaller/issues/1890 and found my openpyxl version, manually added it into the get_writer method like so:
def get_writer(engine_name):
if engine_name == 'openpyxl':
try:
import openpyxl
#remove when conda update available
openpyxl.__version__ = '2.3.2'
# with version-less openpyxl engine
# make sure we make the intelligent choice for the user
if LooseVersion(openpyxl.__version__) < '2.0.0':
return _writers['openpyxl1']
elif LooseVersion(openpyxl.__version__) < '2.2.0':
return _writers['openpyxl20']
else:
return _writers['openpyxl22']
except ImportError:
# fall through to normal exception handling below
pass
try:
return _writers[engine_name]
except KeyError:
raise ValueError("No Excel writer '%s'" % engine_name)
Still no dice. The line number given in the error traceback doesn't even change. I then updated the openpyxl version to 2.3.5, still receiving the error. The openpyxl init file has a version variable in it:
try:
here = os.path.abspath(os.path.dirname(__file__))
src_file = os.path.join(here, ".constants.json")
with open(src_file) as src:
constants = json.load(src)
__author__ = constants['__author__']
__author_email__ = constants["__author_email__"]
__license__ = constants["__license__"]
__maintainer_email__ = constants["__maintainer_email__"]
__url__ = constants["__url__"]
__version__ = constants["__version__"]
except IOError:
# packaged
pass
Any known or potential fixes or workarounds?

Edits were not making an impact because the process was compiled into an exe that these modules were running through. Exported the sections I needed outside of my anaconda environment and now the process works without a hitch.

I will add my workaround to this discussion as I was having the same problem using openpyxl 2.4.0 and maybe a few others are stuck too.
I found that to create a .exe file you have to revert to an older version of openpyxl. To do so:
Open the command prompt and uninstall openpyxl with 'pip uninstall openpyxl'
Reinstall openpyxl using an older version 'pip install openpyxl==2.3.5'
Now you should be able to create your .exe file using py2exe, cx_freeze, etc.

This is my fix: go to your openpyxl site-package folder (for me it's: C:\Python27\Lib\site-packages\openpyxl). Copy all variables in your .constant.json file directly into the _ init _.py file, so it looks like:
import json
import os
__author__= "See AUTHORS"
__author_email__= "eric.gazoni#gmail.com"
__license__= "MIT/Expat",
__maintainer_email__= "openpyxl-users#googlegroups.com"
__url__= "http://openpyxl.readthedocs.org"
__version__= "2.4.0"
try:
here = os.path.abspath(os.path.dirname(__file__))

How to verify that a package installed from a list of package names

I'm writing a program that will install certain packages from a whl file, however I need a way to verify that the packages where installed:
def verify_installs(self):
for pack in self.packages:
import pip
installed = pip.get_installed_distributions()
for name in list(installed):
if pack not in name:
print "{} failed to install.".format(pack)
This will throw the error:
Traceback (most recent call last):
File "run_setup.py", line 34, in <module>
test.verify_installs()
File "run_setup.py", line 29, in verify_installs
if pack not in name:
TypeError: argument of type 'Distribution' is not iterable
If I attempt to to run a loop of the packages and use import like so:
def verify_installs(self):
for pack in self.packages:
import pack
I'll get the error:
Traceback (most recent call last):
File "run_setup.py", line 29, in <module>
test.verify_installs()
File "run_setup.py", line 24, in verify_installs
import pack
ImportError: No module named pack
Is there a way I can loop through a list of packages and then try to import them and catch the import exception? Something like:
def verify_packs(pack_list):
for pack in pack_list:
try:
import pack
except ImportError:
print "{} failed to install".format(pack)

Say pack_list is a list of string names of modules:
import importlib
def verify_packs(pack_list):
for pack in pack_list:
try:
importlib.import_module(pack)
except ImportError:
print("{} failed to install".format(pack))
Note that this is not the preferable way to check if a module was installed and available.
Take a look here.

I figured out a way to check the installed packages:
def verify_installs(self):
for pack in self.packages:
import pip
items = pip.get_installed_distributions()
installed_packs = sorted(["{}".format(i.key) for i in items])
if pack not in installed_packs:
print "Package {} was not installed".format(pack)
Example:
test = SetUpProgram(["lxml", "test", "testing"], None, None)
test.verify_installs()
Output:
Package test was not installed
Package testing was not installed
Now to explain it, this part installed_packs = sorted(["{}".format(i.key) for i in items]) will create this:
['babelfish', 'backports.shutil-get-terminal-size', 'beautifulsoup', 'chardet',
'cmd2', 'colorama', 'cycler', 'decorator', 'django', 'easyprocess', 'gooey', 'gu
essit', 'hachoir-core', 'hachoir-metadata', 'hachoir-parser', 'ipython', 'ipytho
n-genutils', 'lxml', 'matplotlib', 'mechanize', 'mypackage', 'nose', 'numpy', 'p
athlib2', 'pickleshare', 'pillow', 'pip', 'prettytable', 'progressbar', 'prompt-
toolkit', 'pygments', 'pyinstaller', 'pyparsing', 'python-dateutil', 'python-geo
ip', 'python-geoip-geolite2', 'pytz', 'pyvirtualdisplay', 'rebulk', 'requests',
'scapy', 'scrappy', 'selenium', 'setuptools', 'simplegeneric', 'six', 'tinydb',
'traitlets', 'tvdb-api', 'twisted', 'wcwidth', 'win-unicode-console', 'zope.inte
rface']
A list of all locally installed packages on the computer, from there:
if pack not in installed_packs:
print "Package {} was not installed".format(pack)
Will run through the packages and check if any of the packages in a given list correspond to any of the actual install packages.

keyring module is not included while packaging with py2exe

I am making an app using python 2.7 on windows and keyring-3.2.1 . In my python code on eclipse, I used
import keyring
keyring.set_password("service","jsonkey",json_res)
json_res= keyring.get_password("service","jsonkey")
is working fine as I am storing json response in keyring. But, when I converted python code into exe by using py2exe, it shows import error keyring while making dist. Please suggest how to include keyring in py2exe.
Traceback (most recent call last):
File "APP.py", line 8, in <module>
File "keyring\__init__.pyc", line 12, in <module>
File "keyring\core.pyc", line 15, in <module>
File "keyring\util\platform_.pyc", line 4, in <module>
File "keyring\util\platform.pyc", line 29, in <module>
AttributeError: 'module' object has no attribute 'system'
platform_.py code is :
from __future__ import absolute_import
import os
import platform
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)
platform.py code is:
import os
import sys
# While we support Python 2.4, use a convoluted technique to import
# platform from the stdlib.
# With Python 2.5 or later, just do "from __future__ import absolute_import"
# and "import platform"
exec('__import__("platform", globals=dict())')
platform = sys.modules['platform']
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)

The issue you're reporting is due to an environment that contains invalid modules, perhaps from an improper installation of one version of keyring over another. You will want to ensure that you've removed remnants of the older version of keyring. In particular, make sure there's no file called keyring\util\platform.* in your site-packages.
After doing that, however, you'll encounter another problem. Keyring loads its backend modules programmatically, so py2exe won't detect them.
To work around that, you'll want to add a 'packages' declaration to your py2exe options to specifically include the keyring.backends package. I invoked the following setup.py script with Python 2.7 to convert 'app.py' (which imports keyring) to an exe:
from distutils.core import setup
import py2exe
setup(
console=['app.py'],
options=dict(py2exe=dict(
packages='keyring.backends',
)),
)
The resulting app.exe will import and invoke keyring.