I have some fixture directories that contain xml files that I would like included with my python project when building the RPM with bdist_rpm. I thought I could do this by having MANIFEST.in do a recursive-include * *, however, it does not include anything other than *.py files. Is there anyway to have bdist_rpm include non python files in the package or specifically include *.xml files as well?
Where are you trying to install them? If you put them inside a package directory, like this...
myproject/
mypackage/
__init__.py
resources/
file1.xml
file2.xml
...you can use the package_data option in your setup.py file, like this:
from setuptools import setup, find_packages
setup(
name='myproject',
version='0.1',
description='A description.',
packages=find_packages(),
include_package_data=True,
package_data = { '': [ '*.xml' ] },
install_requires=[],
)
This will recursively include any *.xml files inside of any packages. They'll get installed with the rest of your package(s) somewhere inside of the Python library path. You can do the same thing with a MANIFEST.in that looks like this:
recursive-include * *.xml
If you're trying to install them into specific filesystem locations outside of the Python library, I'm not sure if you can do that via setup.py.
You can use data_files parameter of setup to do what you need. Something like this:
setup(
...
package_data = { '/usr/share/yourapp/xmls': [ 'xmls/1.xml', 'xmls/2.xml' ] },
...
)
This would install following files:
/usr/share/yourapp/xmls/1.xml
/usr/share/yourapp/xmls/2.xml
I generally create the list of files in a function like this:
def get_xmls():
xmlfiles = []
for filename in os.listdir('xmls/'):
if filename.endswith('.xml'):
xmlfiles.append('xmls/%s' % filename)
return xmlfiles
Related
Using python 3.8 I have the following structure in a test library:
testrepo
setup.py
Manifest.in
util/
mycode.py
data/
mydata.txt
The file setup.py looks like
setup(
name='testrepo',
version="0.1.0",
packages=['util'],
author='Tester',
description='Testing repo',
include_package_data=True,
package_data={
"util.data": ["*"]
},
)
and using the following Manifest.in:
include util/data/mycode.txt
and when I install this package I do not see any hint of the data folder in venv/lib/python3.8/site-packages/util (when installing the repo into a python virtual environment).
How to do it correctly, so I can read the content from the file util/data/mydata.txt using
from util import data
import importlib.resources as import_resources
text = import_resources.read_text(data, "mydata.txt")
or whatever...
Where can I find this completely documented, with examples etc.?
I guess what you have to do is to create the following basic structure of the repo:
myrepo
setup.py
Manifest.in
mypackage/
__init__.py
mycode.py
data/
__init__.py
mydata.txt
Just make sure to keep in mind 6 additional steps:
You need to put the data folder inside your package folder
You need to add __init__.py inside your data folder.
In setup.py you have to use packages=find_packages(), to find your packages.
In setup.py, you have to set include_package_data=True,
In setup.py, you have to specify the path to your data files:
package_data={
"mypackage.data": ["*"]
},
You also have to define a second file names Manifest.in containing again your data files as follows (using a placeholder here - you can also add each file in a separate line):
include util/data/*
It you are lucky, then you can include/use your data file like
from mypackage import data
import importlib.resources as import_resources
text = import_resources.read_text(data, "mydata.txt")
or
with import_resources.path(data, "mydata.txt") as filename:
myfilename = filename
to get the path to the data file.
Not sure this is documented anywhere.
I have a project directory structure:
myproject/
setup.py
myproject/
editors/
....
utilities/
...
find_inf.f90
All the files in the project are python, except for the one fortran file that i have indicated. Now, I can use setuptools to install my project without the fortran file just fine, but to include the fortran file i have to use numpy.distutils.core.Extension. So I have a setup files like this:
from setuptools import find_packages
from numpy.distutils.core import Extension
ext1 = Extension(name = 'myproject.find_inf',
sources = ['myproject/utilities/find_inf.f90'],
extra_f90_compile_args=['-fopenmp',],
libraries=['gomp'])
if __name__ == "__main__":
from numpy.distutils.core import setup
setup(name = 'myproject',
packages=find_packages(),
package_data={
......
},
entry_points={
'console_scripts': [....]
},
ext_modules = [ext1]
)
This creates and installs myproject-2.0-py2.7-macosx-10.6-x86_64.egg under the site-packages directory and the directory structure looks like:
myproject-2.0-py2.7-macosx-10.6-x86_64.egg
myproject
editors\
find_inf.pyc
find_inf.so.dSYM/
find_inf.py
find_inf.so*
__init__.py
__init__.pyc
So it looks to me that I should be able to import find_inf from myproject. But i can't! Writing from myproject import find_inf produces an import error. What am I doing wrong?
UPDATE:
If I chance the name of the extension from my project.find_inf to just find_inf then the installation puts the extension directly in myproject-2.0-py2.7-macosx-10.6-x86_64.egg. If then I manually move the find_inf files from there to myproject-2.0-py2.7-macosx-10.6-x86_64.egg/myproject then I can import the extension. I still can't make sense of this. Something is clearly wrong in my setup.py that it is not putting the extension in the right place....
UPDATE:
Figured it out. Answer below.
Okay, figured it out. The name= argument to Extension should have been name=myproject.utilities.find_inf.
Reason: So the issue is that there is no package named myproject.find_inf that setup() is aware of. The packages= argument to setup gets the names of the packages (as a list) and myproject.find_inf wasn't in the list because there is no directory under myproject called find_inf (as the directory structure in my question shows). This answer sheds important light on this issue. In order to have the compiled extensions placed under an appropriate sub-package, one needs:
those sub-packages to be present in the source directory structure
__init__.py files should exist in those packages, and
the names of those package have to be passed to the packages= argument in setup().
i.e. simply describing the extension hierarchy in the call to Extension() is not enough.
I have a Scrapy spider that is using gettext to translate some strings. The localization files is stored in /locale/.
When I load the translation I do it with the following code:
t = gettext.translation('sv', localedir=LOCALE_DIR, languages=['sv'])
LOCALE_DIR is set in settings.py with the following code:
import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
LOCALE_DIR = os.path.join(BASE_DIR, 'locale')
This works great when I run the code locally with scrapy crawl spider. But when I deploy it to scrapyd, it generates an .egg file and the localization files can no longer be found. When I print the LOCALE_DIR from the deployed version. It gives me /tmp/condobot-1428391146-4QuH3E.egg/locale.
I guess this is the reason why the files can not be found. The path is a subfolder of a file, which does not make sense. What I expected was that the .egg file would be extracted into a folder, and the path would point to /tmp/condobot-1428391146-4quh3e/locale.
Is there another, better way of setting the path to LOCALE_DIR than the way that I currently do it? I have also tried setting it to locale without any better results.
EDIT: I do use a setup.py file with the following code:
from setuptools import setup, find_packages
setup(
name = 'project',
version = '1.0',
packages = find_packages(),
entry_points = {'scrapy': ['settings = condobot.settings']},
package_data = {
# If any package contains *.txt or *.rst files, include them:
'': ['*.mo', '*.po', '*.txt'],
},
)
I also tried using the following setup.py with a MANIFEST.in file:
from setuptools import setup, find_packages
setup(
name = 'project',
install_requires = ['distribute'],
version = '1.0',
packages = find_packages(),
entry_points = {'scrapy': ['settings = condobot.settings']},
include_package_data = True,
zip_safe = False,
)
MANIFEST.in
recursive-include locale *
recursive-include condobot/locale *
My file structure looks like this:
- condobot
- locale
- sv
- LC_MESSAGES
sv.mo
pipelines.py
settings.py
- locale
- sv
- LC_MESSAGES
sv.mo
MANIFEST.in
requirements.txt
scrapy.cfg
setup.py
(I have placed the /locale/ folder both in / and in /condobot/ just to make sure that the path is not wrong.)
I have extracted the .egg file and I can confirm that it does include the /locale/ folder, and in the /locale/ folder there is /locale/sv/LC_MESSAGES/sv.mo and /locale/sv/LC_MESSAGES/sv.po.
So the issue does not seem to be that the setup.py file is not including the files in the .egg file. It seems to be that the path /......./file.egg/locale/ does not work.
One way to be sure that any other non-source file is accessible after the python module is packaged as an egg is to specify zip_safe as False to have the package fully extracted when installed, i.e.:
setup(
name = 'project',
version = '1.0',
...
zip_safe = False,
)
I have a project with this structure:
SomeProject/
bin/
CHANGES.txt
docs/
LICENSE.txt
MANIFEST.in
README.txt
setup.py
someproject/
__init__.py
location.py
utils.py
static/
javascript/
somescript.js
And a "setup.py" as follows:
#!/usr/bin/env python
import someproject
from os.path import exists
try:
from setuptools import setup, find_packages
except ImportError:
from distutils.core import setup, find_packages
setup(
name='django-some-project',
version=someproject.__version__,
maintainer='Some maintainer',
maintainer_email='some#manteiner.com',
packages=find_packages(),
include_package_data=True,
scripts=[],
url='https://github.com/xxx/some-project',
license='LICENSE',
description='Some project description.',
long_description=open('README.markdown').read() if exists("README.markdown") else "",
install_requires=[
"Django >= 1.4.0"
],
)
Then, when I upload it using the command:
python setup.py sdist upload
It seems ok, but there is no "static" folder with this "javascript" subfolder in the package. My "setup.py" was inspired on github.com/maraujop/django-crispy-forms that has a similar structure. Any hint on what is wrong on uploading this subfolders?
You should be able to add those files to source distributions by editing the MANIFEST.in file to add a line like:
recursive-include someproject/static *.js
or just:
include someproject/static/javascript/*.js
This will be enough to get the files included in source distributions. If the setuptools include_package_data option you're using isn't enough to get the files installed, you can ask for them to be installed explicitly with something like this in your setup.py:
package_data={'someproject': ['static/javascript/*.js']},
Use following
packages = ['.','templates','static','docs'],
package_data={'templates':['*'],'static':['*'],'docs':['*'],},
I cannot figure out how to write my setup.py script in order to include *.html files within the installed package.
Here is my attempt:
import os
from setuptools import setup, find_packages
setup(name='django-testapp',
version='0.1',
author='RadiantHex',
license='BSD',
keywords='testapp,django',
packages=['testapp']],
include_package_data=True,
data_files = os.walk('testapp'),
zip_safe = False,
)
The *.html files are contained within the testapp folder.
Any ideas?
Add following 'package_data' argument to the setup():
setup(...,
package_data={
'testapp' : ['testapp/*.html']
}, ...)