We Keep Coding

Get Path to Python File Before Installation

I have a project that includes c++ binaries and python scripts, it's setup such that it should be installed using setuptools. One of the python files is intended to be both used as a script "
python3 script_name.py params
and for it's primary function to be used in other python projects from script_name import function.
The primary function calls a binary which is in a known relative location before the installation (the user is expected to call pip install project_folder). So in order to call the binary I want to get this files location (pre installation)
To get this I used something like
Path(__file__).resolve().parent
however, since the installation moves the file to another folder like ~/.local/... this doesn't work when imported after the installation.
Is there a way to get the original file path, or to make the installation save that path somewhere?
EDIT:
After #sinoroc 's suggestion I tried including the binary as a resource by putting an __init__.py in the build folder and putting
from importlib.resources import files
import build
binary = files(build).joinpath("binary")
in the main init. After that package.binary still gives me a path to my .local/lib and binary.is_file() still returns False
from importlib_resources import files
GenerateHistograms = files("build").joinpath("GenerateHistograms")
gave the same result

Since you are installing your package, you also need to include your C++ binary in the installation. You cannot have a mixed setup. I suggest something like this.
In your setup.py:
from setuptools import setup, find_packages
setup(
name="mypkg",
packages=find_packages(exclude=["tests"]),
package_data={
"mypkg": [
"binary", # relative path to your package directory
]
},
include_package_data=True,
)
Then in your module use pkg_resources:
from pathlib import Path
from pkg_resources import resource_filename
# "binary" is whatever relative path you put in package_data
path_to_binary = Path(resource_filename("mypkg", "binary"))
pkg_resources should be pulled in by setuptools.
EDIT: the recipe above might be a bit out of date; as #sinoroc suggests, using importlib.resources instead of pkg_resources is probably the modern equivalent.

So I managed to solve it in with #sinroc 's approach in the end
In setup.py
package_data={'package':['build/*']}
include_package_data=True
and in the primary __init.py__:
from importlib.resources import files
binary = files("package.build").joinpath("binary")
So I could then from package import binary to get the path
EDIT: Looks like someone also pointed the error in my ways out before I finished ^^

Proper way to copy configuration files during installation?

I am trying to distribute a mplstyle I wrote such that I can share it easily. It boils down to copying a text file to the proper configuration direction (which is known for any architecture) during installation. I want to be able to install using either python setup.py install or pip install .... Currently I do not seem to get either of the two ways robust (see current approach below).
Installing with pip install ... does not seem to invoke the copying at all.
Installing with python setup.py install works well on my machine, but ReadTheDocs gives me the following error:
python setup.py install --force
running install
error: [Errno 2] No such file or directory: u'/home/docs/.config/matplotlib/stylelib/goose.mplsty
What is the proper way to copy configuration files during installation in a robust way?
Current approach
File structure
setup.py
goosempl/
| __init__.py
| stylelib/
| goose.mplstyle
| ...
setup.py
from setuptools import setup
from setuptools.command.install import install
class PostInstallCommand(install):
def run(self):
import goosempl
goosempl.copy_style()
install.run(self)
setup(
name = 'goosempl',
...,
install_requires = ['matplotlib>=2.0.0'],
packages = ['goosempl'],
cmdclass = {'install': PostInstallCommand},
package_data = {'goosempl/stylelib':['goosempl/stylelib/goose.mplstyle']},
)
goosempl/__init__.py
def copy_style():
import os
import matplotlib
from pkg_resources import resource_string
files = [
'stylelib/goose.mplstyle',
]
for fname in files:
path = os.path.join(matplotlib.get_configdir(),fname)
text = resource_string(__name__,fname).decode()
print(path, text)
open(path,'w').write(text)
Upload to PyPi
python setup.py bdist_wheel --universal
twine upload dist/*

First of all, based on the project structure you've provided, you're not specifying the package_data correctly. If goosempl is a package and stylelib a directory inside it containing the mplstyle files (what I assume from your code), then your package_data configuration line should be:
package_data = {'goosempl': ['stylelib/goose.mplstyle']},
As described in Building and Distributing Packages with Setuptools:
The package_data argument is a dictionary that maps from package names to lists of glob patterns. The globs may include subdirectory names, if the data files are contained in a subdirectory of the package.
So your package is goosempl and stylelib/goose.mplstyle is the file to be included in package data for goosempl.
Your second issue (No such file or directory) is simple: in the copy_style() function, you never check if the parent directory of the file exists before writing the file. You should be able to reproduce this locally by removing the directory /home/<user>/.config/matplotlib/stylelib/ (or moving it temporarily).
The fix is also quite simple, actually there are lots of them. Use whatever you want to create missing directories.
distutils.dir_util.mkpath is suitable for both python2 and python3:
for fname in files:
path = os.path.join(matplotlib.get_configdir(), fname)
distutils.dir_util.mkpath(os.dirname(path))
My preferred one is using pathlib, but it is available only since Python 3.4:
for fname in files:
path = pathlib.Path(matplotlib.get_configdir(), fname)
path.parent.mkdir(parents=True, exist_ok=True)

Setuptools: how to distribute configuration files without erasing existing ones?

I'm trying to find a nice way to ship default configuration files with my python setuptools project.
For now I'm doing it like this:
from setuptools import setup
setup(
...
data_files = [('/usr/local/etc', ['files/myproject.conf', ...])]
...
)
The problem is that the configuration files are erased if I uninstall my package. Usually, when uninstalling a package on Linux or FreeBSD, the configuration files are not deleted. I think this is good, because sometimes you just want to uninstall a package to reinstall another version, or to install it with other options etc... You don't expect your configurations files to be deleted.
How to achieve the same with setuptools? How to install configurations files only if they don't already exist?

Why can't you? setup.py is also a python script, maybe you can manually add your configuration files to avoid adding them into metadata? Like the following code:
from setuptools import setup
from shutil import copyfile
import os
...
setup(
...
# no data_files option
...
)
if not os.path.exists(configuration_file):
copyfile(configuration, configuration_file)

setup.py not installing data files

I have a Python library that, in addition to regular Python modules, has some data files that need to go in /usr/local/lib/python2.7/dist-package/mylibrary.
Unfortunately, I have been unable to convince setup.py to actually install the data files there. Note that this behaviour is under install - not sdist.
Here is a slightly redacted version of setup.py
module_list = list_of_files
setup(name ='Modules',
version ='1.33.7',
description ='My Sweet Module',
author ='PN',
author_email ='email',
url ='url',
packages = ['my_module'],
# I tried this. It got installed in /usr/my_module. Not ok.
# data_files = [ ("my_module", ["my_module/data1",
# "my_module/data2"])]
# This doesn't install it at all.
package_data = {"my_module" : ["my_module/data1",
"my_module/data2"] }
)
This is in Python 2.7 (will have to run in 2.6 eventually), and will have to run on some Ubuntu between 10.04 and 12+. Developing it right now on 12.04.

UPD:
package_data accepts dict in format {'package': ['list', 'of?', 'globs*']}, so to make it work, one should specify shell globs relative to package dir, not the file paths relative to the distribution root.
data_files has a different meaning, and, in general, one should avoid using this parameter.
With setuptools you only need include_package_data=True, but data files should be under version control system, known to setuptools (by default it recognizes only CVS and SVN, install setuptools-git or setuptools-hg if you use git or hg...)
with setuptools you can:
- in MANIFEST.im:
include my_module/data*
- in setup.py:
setup(
...
include_package_data = True,
...
)

http://docs.python.org/distutils/setupscript.html#installing-additional-files
If directory is a relative path, it is interpreted relative to the
installation prefix (Python’s sys.prefix for pure-Python packages,
sys.exec_prefix for packages that contain extension modules).
This will probably do it:
data_files = [ ("my_module", ["local/lib/python2.7/dist-package/my_module/data1",
"local/lib/python2.7/dist-package/my_module/data2"])]
Or just use join to add the prefix:
data_dir = os.path.join(sys.prefix, "local/lib/python2.7/dist-package/my_module")
data_files = [ ("my_module", [os.path.join(data_dir, "data1"),
os.path.join(data_dir, "data2")])]

The following solution worked fine for me.
You should have MANIFEST.in file where setup.py is located.
Add the following code to the manifest file
recursive-include mypackage *.json *.md # can be extended with more extensions or file names.
Another solution is adding the following code to the MANIFEST.in file.
graft mypackage # will copy the entire package including non-python files.
global-exclude __pyache__ *.txt # list files you dont want to include here.
Now, when you do pip install all the necessary files will be included.
Hope this helps.
UPDATE:
Make sure that you also have include_package_data=True in the setup file

Python setuptools: how to include a config file for distribution into <prefix>/etc

How can I write setup.py so that:
The binary egg distribution (bdist_egg) includes a sample configuration file and
Upon installation puts it into the {prefix}/etc directory?
A sample project source directory looks like this:
bin/
myapp
etc/
myapp.cfg
myapp/
__init__.py
[...]
setup.py
The setup.py looks like this:
from distutils.command.install_data import install_data
packages = ['myapp', ]
scripts = ['bin/myapp',]
cmdclasses = {'install_data': install_data}
data_files = [('etc', ['etc/myapp.cfg'])]
setup_args = {
'name': 'MyApp',
'version': '0.1',
'packages': packages,
'cmdclass': cmdclasses,
'data_files': data_files,
'scripts': scripts,
# 'include_package_data': True,
'test_suite': 'nose.collector'
}
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
setup(**setup_args)
setuptools are installed in both the build environment and in the installation environment.
The 'include_package_data' commented out or not does not help.

I was doing some research on this issue and I think the answer is in the setuptools documentation: http://peak.telecommunity.com/DevCenter/setuptools#non-package-data-files
Next, I quote the extract that I think has the answer:
Non-Package Data Files
The distutils normally install general "data files" to a
platform-specific location (e.g. /usr/share). This feature intended to
be used for things like documentation, example configuration files,
and the like. setuptools does not install these data files in a
separate location, however. They are bundled inside the egg file or
directory, alongside the Python modules and packages. The data files
can also be accessed using the Resource Management API [...]
Note, by the way, that this encapsulation of data files means that you
can't actually install data files to some arbitrary location on a
user's machine; this is a feature, not a bug. You can always include a
script in your distribution that extracts and copies your the
documentation or data files to a user-specified location, at their
discretion. If you put related data files in a single directory, you
can use resource_filename() with the directory name to get a
filesystem directory that then can be copied with the shutil module.
[...]