We Keep Coding

How can setup.py reference other files in the source repo?

I'm having trouble getting a python package to build with python -m build .. setup.py fails on:
FileNotFoundError: [Errno 2] No such file or directory: 'requirements/requirements.txt'
It's caused by the fact that build is copying files to a temporary directory first. But it's only copying the source/, README.md, setup.py, setup.cfg. It's not copying requirements/.
For complex reasons my setup.py needs to reference other files at the root of the source repo - a directory containing multiple requirements.txt files. It's not really worth discussing why it needs to be structured this way, I've already been through that long debate with colleagues.
This works fine when we install the package through pip install -e . or as a git dependency git+ssh://... but fails when building before we push to a pypi repo.
setup.cfg
setup.py
source/
source/my_package/
requirements/
requirements/requirements.txt
requirements/some-other-requirements.txt
setup.py references this directory before calling setup().
from pathlib import Path
from setuptools import setup, find_namespace_packages
requirements_dir = Path("requirements")
# This is the line that fails:
with (requirements_dir / "requirements.txt").open() as f:
install_requires = list(f)
setup(
packages=find_namespace_packages(where="source", include=["acme_corp.*"], exclude=["tests", "tests.*"]),
package_dir={"": "source"},
install_requires=install_requires,
extras_require=optional_packages,
)

build does the correct job here. After building the source dist, it checks whether the built result can be actually installed. And without including the files under requirements into the source dist, the source dist can not be installed and is thus unusable. Try it out:
$ python -m build --sdist # builds fine, but the tar is broken:
$ cd dist
$ tar tf pkg-0.0.1.tar.gz | grep requirements # will be empty
$ pip wheel pkg-0.0.1.tar.gz # will fail
Processing ./pkg-0.0.1.tar.gz
File was already downloaded
Preparing metadata (setup.py) ... error
ERROR: Command errored out with exit status 1:
command:
...
Complete output (9 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-wfzz27_k/setup.py", line 9, in <module>
with (requirements_dir / "requirements.txt").open() as f:
File "/usr/lib64/python3.9/pathlib.py", line 1252, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File "/usr/lib64/python3.9/pathlib.py", line 1120, in _opener
return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-wfzz27_k/requirements/requirements.txt'
To fix, write a MANIFEST.in alongside the setup.py that includes requirements directory in the source dist. Example file contents:
graft requirements
Building a source dist should work now; you can additionally verify that the sdist now contains all files to install from:
$ tar tf dist/pkg-0.0.1.tar.gz | grep requirements
pkg-0.0.1/requirements/
pkg-0.0.1/requirements/requirements.txt
...
This only concerns the source dist, as the wheel is built on host (your machine) already and does not contain a setup script anymore. Wheel building will also ignore the MANIFEST.in file.

Tox fails because setup.py can't find the requirements.txt

I have added tox to my project and my tox.ini is very simple:
[tox]
envlist = py37
[testenv]
deps =
-r{toxinidir}/requirements_test.txt
commands =
pytest -v
But when I run tox, I get the following error:
ERROR: invocation failed (exit code 1), logfile: /path/to/my_project/.tox/py37/log/py37-2.log
========================================================================================= log start ==========================================================================================
Processing ./.tox/.tmp/package/1/my_project-0+untagged.30.g6909bfa.dirty.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-ywna_4ks/setup.py", line 15, in <module>
with open(requirements_path) as requirements_file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-ywna_4ks/requirements.txt'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-ywna_4ks/
You are using pip version 10.0.1, however version 19.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
========================================================================================== log end ===========================================================================================
__________________________________________________________________________________________ summary ___________________________________________________________________________________________
ERROR: py37: InvocationError for command /path/to/my_project/.tox/py37/bin/python -m pip install --exists-action w .tox/.tmp/package/1/my_project-0+untagged.30.g6909bfa.dirty.zip (exited with code 1)
Here is my setup.py:
-*- coding: utf-8 -*-
import os
import sys
from setuptools import setup, find_packages
import versioneer
here = os.path.abspath(os.path.dirname(__file__))
sys.path.insert(0, here)
requirements_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')
with open(requirements_path) as requirements_file:
requires = requirements_file.readlines()
setup(
name='my_project',
version=versioneer.get_version(),
cmdclass=versioneer.get_cmdclass(),
maintainer='Hamed',
license='BSD',
py_modules=['my_project'],
packages=find_packages(),
package_data={'': ['*.csv', '*.yml', '*.html']},
include_package_data=True,
install_requires=requires,
long_description=open('README.md').read(),
zip_safe=False
]
},
)
python setup.py install works fine.
It seems that tox is looking for requirements in the tmp dir, but can't find it there. Is there something wrong with my configurations?
I am using tox==3.12.1, python==3.7.3, setuptools==41.0.1, and conda==4.6.9
I've tested this on Arch and SLES 12 and got the same result with both.

Based on the point from #phd, I found out that requirements.txt was not present in the source distribution. Adding requirements.txt to the MANIFEST.in solved the issue!

Complementing Hamed2005's answer:
I have my requirements split into different files (base_requirements.txt, dev_requirements.txt, etc), all of them in a requirements directory. In this case, you need to add this directory in the MANIFEST.in as
recursive-include requirements *

PIP: error: [Errno 2] No such file or directory: 'Python/Reference Documentation 3.2'

I've been trying to upload my package to PyPI by following these steps:
Registering on PyPI
Creating a hidden .pypirc file in my home directory (~/) containing:
[distutils]
index-servers = pypi
[pypi]
repository=https://pypi.python.org/pypi
username=my_username
password=my_password
Creating LICENSE.TXT, requirements.txt, and setup.cfg inside my package which contained this:
[metadata]
description-file = README.md
Creating setup.py in the same directory as my package was (Desktop) which contained this:
from distutils.core import setup
setup(
name='Package_name',
packages=['Package_name'],
version='1.0',
description='Description,
author= 'ShellRox',
author_email='Email',
url='Github url',
download_url='Github download url',
keywords=['authentication', 'steam', 'simple'],
classifiers=[],
)
And finally, executing this command:
python setup.py register -r pypitest
python setup.py sdist bdist_wininst upload
Which prints this error for some reason:
error: [Errno 2] No such file or directory: 'Python/Reference
Documentation 3.2
What could be the problem? Could there be something wrong with my setup.py?

Error when downloading package using git+ssh

I am working on creating a private package repository for my company and am trying to download the package from Github. I believe the package should compile, as I have uploaded this as a test function to PyPI.
The error I get is: FileNotFoundError: [Errno 2] No such file or directory: setup.py I am cloning the repo back into the exact same environment in which the function was created - what could be wrong?
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\tokenize.py", line 454, in open
buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory:
'C:\\Users\\ALLENB~1\\AppData\\Local\\Temp\\pip-07k46aoa-build\\setup.py'
Command "python setup.py egg_info" failed with error code 1 in
C:\Users\ALLENB~1\AppData\Local\Temp\pip-07k46aoa-build\setup.py

setup.py is not part of the Python/Anaconda 3 installation location!
It's included with the package that you wish to install. Change directories to the location of the source that you download it with pip install (or with: py -m pip install) and then run the setup.py file.
EDIT:
Add #subdirectory=folder_name to your code.

Error when trying to distribute via pip

I uploaded my package on pypi using this guide.
But it seems that there is an error with this line in setup.py
long-description=open(os.path.join(os.path.dirname(__file__),'README.md')).read()
which is giving me, on trying to install via pip
IO Error no such file or directory.
So how can I fix this? should a simple open('README.md')?
Is the long-description line really needed when I already have this in my setup.cfg
[metadata]
description-file = README.md

You need to add 'include README.md' in MANIFEST.in, if you do not have this file (the Manifest one), create it.
Just execute this on your repository root directory (where the setup.py file is located)
echo "include README.md" >> MANIFEST.in
Cheers.