I'm trying to find a nice way to ship default configuration files with my python setuptools project.
For now I'm doing it like this:
from setuptools import setup
setup(
...
data_files = [('/usr/local/etc', ['files/myproject.conf', ...])]
...
)
The problem is that the configuration files are erased if I uninstall my package. Usually, when uninstalling a package on Linux or FreeBSD, the configuration files are not deleted. I think this is good, because sometimes you just want to uninstall a package to reinstall another version, or to install it with other options etc... You don't expect your configurations files to be deleted.
How to achieve the same with setuptools? How to install configurations files only if they don't already exist?
Why can't you? setup.py is also a python script, maybe you can manually add your configuration files to avoid adding them into metadata? Like the following code:
from setuptools import setup
from shutil import copyfile
import os
...
setup(
...
# no data_files option
...
)
if not os.path.exists(configuration_file):
copyfile(configuration, configuration_file)
Related
I have forked a repo and now I have cloned it. When running the setup.py file inside, Python installs the package inside site-packages as an obscure name, that of which importing this within a Python file will not be viable.
For example, I fork and clone a repo called foo. I can also see this in the setup.py file:
setup(
name='foo',
version='3.3.0-rc6',
packages=find_packages('src'),
package_dir={'': 'src'},
include_package_data=True,
When I run python setup.py install, I find the package has been installed as foo-3.3.0rc6-py3.6.egg. I do not want to have to import the package as this name in every one of my projects utilizing it.
How can I just change the package name to foo (when running/installing via setup.py), so that I can run import foo and not import foo-3.3...?
I feel I can not just rename it, as if I wanted other users to clone the repo and not have to go through the same trouble as me. Is the package name embedded somewhere in the setup.py folder?
Let me know if you need anything else, I'm willing to have this issue resolved.
You don't have to import foo-3.3; actually you cannot import as it's SyntaxError.
You don't have to import foo-3.3 from foo-3.3.0rc6-py3.6.egg because distutils/setuptools configured correct import path for eggs. Look into easy-install.pth file and you find there ./foo-3.3.0rc6-py3.6.egg. Run python and verify sys.path — there have to be foo-3.3.0rc6-py3.6.egg entry so that import foo works.
That's just the name of the egg, and you needn't worry about it. Python knows where to look for the package, so when you do:
import foo
...it'll do the right thing.
I am trying to distribute a mplstyle I wrote such that I can share it easily. It boils down to copying a text file to the proper configuration direction (which is known for any architecture) during installation. I want to be able to install using either python setup.py install or pip install .... Currently I do not seem to get either of the two ways robust (see current approach below).
Installing with pip install ... does not seem to invoke the copying at all.
Installing with python setup.py install works well on my machine, but ReadTheDocs gives me the following error:
python setup.py install --force
running install
error: [Errno 2] No such file or directory: u'/home/docs/.config/matplotlib/stylelib/goose.mplsty
What is the proper way to copy configuration files during installation in a robust way?
Current approach
File structure
setup.py
goosempl/
| __init__.py
| stylelib/
| goose.mplstyle
| ...
setup.py
from setuptools import setup
from setuptools.command.install import install
class PostInstallCommand(install):
def run(self):
import goosempl
goosempl.copy_style()
install.run(self)
setup(
name = 'goosempl',
...,
install_requires = ['matplotlib>=2.0.0'],
packages = ['goosempl'],
cmdclass = {'install': PostInstallCommand},
package_data = {'goosempl/stylelib':['goosempl/stylelib/goose.mplstyle']},
)
goosempl/__init__.py
def copy_style():
import os
import matplotlib
from pkg_resources import resource_string
files = [
'stylelib/goose.mplstyle',
]
for fname in files:
path = os.path.join(matplotlib.get_configdir(),fname)
text = resource_string(__name__,fname).decode()
print(path, text)
open(path,'w').write(text)
Upload to PyPi
python setup.py bdist_wheel --universal
twine upload dist/*
First of all, based on the project structure you've provided, you're not specifying the package_data correctly. If goosempl is a package and stylelib a directory inside it containing the mplstyle files (what I assume from your code), then your package_data configuration line should be:
package_data = {'goosempl': ['stylelib/goose.mplstyle']},
As described in Building and Distributing Packages with Setuptools:
The package_data argument is a dictionary that maps from package names to lists of glob patterns. The globs may include subdirectory names, if the data files are contained in a subdirectory of the package.
So your package is goosempl and stylelib/goose.mplstyle is the file to be included in package data for goosempl.
Your second issue (No such file or directory) is simple: in the copy_style() function, you never check if the parent directory of the file exists before writing the file. You should be able to reproduce this locally by removing the directory /home/<user>/.config/matplotlib/stylelib/ (or moving it temporarily).
The fix is also quite simple, actually there are lots of them. Use whatever you want to create missing directories.
distutils.dir_util.mkpath is suitable for both python2 and python3:
for fname in files:
path = os.path.join(matplotlib.get_configdir(), fname)
distutils.dir_util.mkpath(os.dirname(path))
My preferred one is using pathlib, but it is available only since Python 3.4:
for fname in files:
path = pathlib.Path(matplotlib.get_configdir(), fname)
path.parent.mkdir(parents=True, exist_ok=True)
I can't get CX_Freeze to include the package ruamel.yaml with the build_exe.
I've also tried adding it to the "packages" option like
build_exe_options = {
...
"packages": [
...
"ruamel.yaml",
...
]
...
}
cx_Freeze.setup(
...
executables=[cx_Freeze.Executable("pyhathiprep/__main__.py",
targetName="pyhathiprep.exe", base="Console")],
)
and I get
File "C:\Users\hborcher\PycharmProjects\pyhathiprep\.env\lib\site-packages\cx_Freeze\finder.py", line 350, in _ImportModule
raise ImportError("No module named %r" % name)
ImportError: No module named 'ruamel.yaml'
I've tried adding it to the "namespace_packages" like
build_exe_options = {
...
"namespace_packages": ["ruamel.yaml"]
...
}
cx_Freeze.setup(
...
executables=[cx_Freeze.Executable("pyhathiprep/__main__.py",
targetName="pyhathiprep.exe", base="Console")],
)
and I get
File "C:\Users\hborcher\PycharmProjects\pyhathiprep\.env\lib\site-packages\cx_Freeze\finder.py", line 221, in _FindModule
return None, module.__path__[0], info
TypeError: '_NamespacePath' object does not support indexing
What am I doing wrong?
The doc for ruamel.yaml clearly states that you have to use a recent version of pip and setuptools to install ruamel.yaml.
CX_Freeze is not calling pip, nor does it support installing from the (correctly pre-configured) .whl files. Instead it does seem to call setup() in a way of its own.
What you can try to do is create a ruamel directory in your source directory, then in that directory create an empty __init__.py file and yaml directory. In that yaml directory copy all of the .py files from an unpacked latest version of ruamel.yaml skipping setup.py and all of the other install cruft. Alternatively you can check those files out from Bitbucket, but then there is even more unnecessary cruft to deal with, and you run the slight risk of having a non-released intermediate version if you don't check out by release tag.
Once that works you'll have a "pure" Python version of ruamel.yaml in your frozen application.
If you are using yaml = YAML(typ='safe') or yaml = YAML(typ='unsafe') and you expect the speed up from the C based loader and dumper, then you should look at using the Windows .whl files provided on PyPI. They include the _ruamel_yaml.cpXY-win_NNN.pyd files. If you don't know your target (python and/or win32|win_amd64 you should be able to include all of them and ruamel.yaml will pick the right one when it starts (actually it only does from _ruamel_yaml import CParser, CEmitter and assumes the Python interpreter knows what to do).
Okay I figured out a solution. I think it might be a bug in CX_Freeze. If I pip install ruamel.base and ruamel.yaml cx_freeze seems to install everything correctly. This is true, even if I ask it to only include ruamel.yaml.
If I have both ruamel.base and ruamel.yaml installed, then this works...
build_exe_options = {
...
"namespace_packages": ["ruamel.yaml"]
...
}
cx_Freeze.setup(
...
executables=[cx_Freeze.Executable("pyhathiprep/__main__.py",
targetName="pyhathiprep.exe", base="Console")],
)
I had this same problem with azure. The issue is the way microsoft structured the azure package - you can import azure.something.something_else.module, but you can't import azure directly. cx_freeze needs to be able to find the folder azure (or in your case, the folder ruamel) directly, not just the subfolders.
I had to go to each directory under the azure folder that I was accessing and ensure there was an init.py file there. After that, cx_freeze was able to find it perfectly.
Another option would be to just directly copy the folder from a path that you know (direct link to your site-packages, or copy the ruamel directory into your program directory and copy it from there) into the build folder as part of your setup. I do this for things like my data files:
import shutil
shutil.copytree("icons","build/exe.win32-3.6/icons")
I have a Python library that, in addition to regular Python modules, has some data files that need to go in /usr/local/lib/python2.7/dist-package/mylibrary.
Unfortunately, I have been unable to convince setup.py to actually install the data files there. Note that this behaviour is under install - not sdist.
Here is a slightly redacted version of setup.py
module_list = list_of_files
setup(name ='Modules',
version ='1.33.7',
description ='My Sweet Module',
author ='PN',
author_email ='email',
url ='url',
packages = ['my_module'],
# I tried this. It got installed in /usr/my_module. Not ok.
# data_files = [ ("my_module", ["my_module/data1",
# "my_module/data2"])]
# This doesn't install it at all.
package_data = {"my_module" : ["my_module/data1",
"my_module/data2"] }
)
This is in Python 2.7 (will have to run in 2.6 eventually), and will have to run on some Ubuntu between 10.04 and 12+. Developing it right now on 12.04.
UPD:
package_data accepts dict in format {'package': ['list', 'of?', 'globs*']}, so to make it work, one should specify shell globs relative to package dir, not the file paths relative to the distribution root.
data_files has a different meaning, and, in general, one should avoid using this parameter.
With setuptools you only need include_package_data=True, but data files should be under version control system, known to setuptools (by default it recognizes only CVS and SVN, install setuptools-git or setuptools-hg if you use git or hg...)
with setuptools you can:
- in MANIFEST.im:
include my_module/data*
- in setup.py:
setup(
...
include_package_data = True,
...
)
http://docs.python.org/distutils/setupscript.html#installing-additional-files
If directory is a relative path, it is interpreted relative to the
installation prefix (Python’s sys.prefix for pure-Python packages,
sys.exec_prefix for packages that contain extension modules).
This will probably do it:
data_files = [ ("my_module", ["local/lib/python2.7/dist-package/my_module/data1",
"local/lib/python2.7/dist-package/my_module/data2"])]
Or just use join to add the prefix:
data_dir = os.path.join(sys.prefix, "local/lib/python2.7/dist-package/my_module")
data_files = [ ("my_module", [os.path.join(data_dir, "data1"),
os.path.join(data_dir, "data2")])]
The following solution worked fine for me.
You should have MANIFEST.in file where setup.py is located.
Add the following code to the manifest file
recursive-include mypackage *.json *.md # can be extended with more extensions or file names.
Another solution is adding the following code to the MANIFEST.in file.
graft mypackage # will copy the entire package including non-python files.
global-exclude __pyache__ *.txt # list files you dont want to include here.
Now, when you do pip install all the necessary files will be included.
Hope this helps.
UPDATE:
Make sure that you also have include_package_data=True in the setup file
How can I write setup.py so that:
The binary egg distribution (bdist_egg) includes a sample configuration file and
Upon installation puts it into the {prefix}/etc directory?
A sample project source directory looks like this:
bin/
myapp
etc/
myapp.cfg
myapp/
__init__.py
[...]
setup.py
The setup.py looks like this:
from distutils.command.install_data import install_data
packages = ['myapp', ]
scripts = ['bin/myapp',]
cmdclasses = {'install_data': install_data}
data_files = [('etc', ['etc/myapp.cfg'])]
setup_args = {
'name': 'MyApp',
'version': '0.1',
'packages': packages,
'cmdclass': cmdclasses,
'data_files': data_files,
'scripts': scripts,
# 'include_package_data': True,
'test_suite': 'nose.collector'
}
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
setup(**setup_args)
setuptools are installed in both the build environment and in the installation environment.
The 'include_package_data' commented out or not does not help.
I was doing some research on this issue and I think the answer is in the setuptools documentation: http://peak.telecommunity.com/DevCenter/setuptools#non-package-data-files
Next, I quote the extract that I think has the answer:
Non-Package Data Files
The distutils normally install general "data files" to a
platform-specific location (e.g. /usr/share). This feature intended to
be used for things like documentation, example configuration files,
and the like. setuptools does not install these data files in a
separate location, however. They are bundled inside the egg file or
directory, alongside the Python modules and packages. The data files
can also be accessed using the Resource Management API [...]
Note, by the way, that this encapsulation of data files means that you
can't actually install data files to some arbitrary location on a
user's machine; this is a feature, not a bug. You can always include a
script in your distribution that extracts and copies your the
documentation or data files to a user-specified location, at their
discretion. If you put related data files in a single directory, you
can use resource_filename() with the directory name to get a
filesystem directory that then can be copied with the shutil module.
[...]