Structure:
.
├── application
│ └── runner.py
└── dummyLibrary
├── helperFunctions.py
├── __init__.py
└── setup.py
runner.py:
import dummyLibrary
dummyLibrary.foo()
dummyLibrary.bar()
init.py:
(empty file)
helperFunctions.py:
def foo():
print("called foo()")
def bar():
print("called bar()")
setup.py:
#!/usr/bin/env python
from distutils.core import setup
setup(name='dummyLibrary', version='0.0.1')
After cd'ing into dummyLibrary/ ,
I tried installing dummyLibrary with
pip3 install -e .
This was the output:
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/Documents/pythonTest/dummyLibrary
Preparing metadata (setup.py) ... done
Installing collected packages: dummyLibrary
Running setup.py develop for dummyLibrary
Successfully installed dummyLibrary-0.0.1
I tried installing dummyLibrary with
python3 -m pip install -e .
This was the output:
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/Documents/pythonTest/dummyLibrary
Preparing metadata (setup.py) ... done
Installing collected packages: dummyLibrary
Attempting uninstall: dummyLibrary
Found existing installation: dummyLibrary 0.0.1
Uninstalling dummyLibrary-0.0.1:
Successfully uninstalled dummyLibrary-0.0.1
Running setup.py develop for dummyLibrary
Successfully installed dummyLibrary-0.0.1
After cd'ing into application/ and running
python3 runner.py
I get:
Traceback (most recent call last):
File "runner.py", line 1, in <module>
import dummyLibrary
ModuleNotFoundError: No module named 'dummyLibrary'
no matter how I try to install my library
Why is this?
Additional Information:
Using Ubuntu, Not using a virtual environment.
I tried rebooting after installing. Didn't help.
Welcome to Stack Overflow!
First thing: __init__.py and helperFunctions.py should be inside another nested folder with the same name as the package.
.
├── application
│ └── runner.py
└── dummyLibrary
├── dummyLibrary
│ ├── __init__.py
│ └── helperFunctions.py
└── setup.py
Secondly, with the code in runner.py as how you'd like it, the __init__.py should include the following line to import all of the functions in helperFunctions.py:
from .helperFunctions import *
Finally, your setup.py should also include the parameter packages as a list of folder names in the package, in this case, the parameter should be packages=['dummyLibrary'].
P.S.: it's a pythonic practice to name your files and packages in snake_case rather than camelCase.
Related
I used to be able to run the command runner.py from my module my-runner while using python setup.py develop. However, ever since that I reinstalled it using python setup.py install, I now get a pkg_resources.ResolutionError when calling runner.py.
This is the mini tree structure
.
├── bin
│ ├── some_other_file.py
│ ├── runner.py
├── setup.py
here is my setup.py
from setuptools import setup, find_packages
setup(
name='my-runner',
version='1.0.0',
license='private',
author='MyName',
author_email='myname#myemail.com',
description='My Runner',
packages=find_packages(),
scripts=['bin/runner.py', 'bin/some_other_file.py']
)
Running command runner.py returns the error
pkg_resources.ResolutionError: Script 'scripts/runner.py' not found in metadata at '/home/myname/module/my-runner.egg-info
I guess I have no clue why install would break it? I am guessing it has to do with the fact that develop does not read the egg-info dir but would like a solution to this problem.
A possible way is to remove the package(pip3 uninstall my-runner) and reinstall it (python setup.py install)
We have a private git monorepo which hosts a number of Python packages. Poetry was the dependency management tool initially chosen for the project. Anyway, due to this Poetry issue, it would not be accepted solution that involves creating new setup.py files.
A simplified version of the structure:
git-monorepo
├── pkg-1
│ ├── pkg
│ │ └── mod1.py
│ └── pyproject.toml
├── pkg-2
│ ├── pkg
│ │ └── mod2.py
│ └── pyproject.toml
└── lib
├── pkg
│ └── lib.py
└── pyproject.toml
The library distribution package lib is indepentent from any other package. However, pkg-1 depends on lib and pkg-2 depends on both pkg-1 and lib.
So, the question is:
How would be the proper way to use pip to install a package from this monorepo?
Let us consider as an example that we try to install pkg-1, where pkg-1/pyproject.toml includes the following lines:
...
[tool.poetry.dependencies]
lib = {path = "../lib/"}
...
The result from running pip, as explained in the VCS support documentation:
$ pip install -e git+https://gitlab.com/my-account/git-monorepo#"egg=pkg-1&subdirectory=pkg-1"
Traceback (most recent call last):
File "/home/hblanco/.local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
super(Requirement, self).__init__(requirement_string)
File "/home/hblanco/.local/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 115, in __init__
raise InvalidRequirement("Invalid URL: {0}".format(req.url))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: ../lib
The problem in the above setup is that the dependency is specified as a path dependency. When installing it, it uses that path dependency.
I ran into the same issue with a python monorepo, where I wanted to share the packages also to other projects.
I found 2 approaches to work for me:
in the CI/CD build pipeline, edit the pyproject.toml just before creating the wheel (which is published to a pypi repo)
First create the wheel (or .tar.gz) artifact, and then modify it afterwards (by extracting it, replacing the path dependencies, and zipping it again).
The full approach I've explained here.
However, it won't work with the git+https://...
You'll need a (private) pypi repo somewhere. Gitlab provides one for each project, which I utilize in the demo project here
I have a package which contains several modules and a cython extension. I am able to install the package with the standard pip install command in virtual environments, it compiles my cython extension and so on. Everything works great. Now, I would like to test all of those with pytest. Here again, everything works great locally (or when I install the package on any other machine). However, when I use a docker image (the one from circleci) the compiled cython extension is not found when I run my tests with pytest.
The structure of my package is as follow
/home/circleci/dspt
├── dspt
│ ├── connection
│ ├── data
│ └── sp
│ ├── bilateral
│ │ ├── filters.pyx
│ └── signal_processing.py
├── setup.py
├── tests
│ └── test_data.py
I install this package in a virtual environment /home/circleci/dspt/myvenv. So, under this venv, I tried theses:
(myvenv) /home/circleci/$ python
>>>import dspt.sp.filters # ok!
but if I change the directory
(myvenv) /home/circleci/dspt$ python
>>>import dspt.sp.filters # ModuleNotFoundError: No module named 'dspt.sp.filters'
I got the same error with pytest
(myvenv) /home/circle$ pytest dspt/tests/test_data.py
================================================= test session starts ==================================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/circleci/dspt/tests, inifile: pytest.ini
plugins: mock-1.10.1, metadata-1.8.0
collected 12 items / 1 errors / 11 selected
======================================================== ERRORS ========================================================
________________________________________ ERROR collecting test_ecg_analysis.py _________________________________________
ImportError while importing test module '/home/circleci/dspt/tests/test_data.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/guillaume/src/dspt/tests/test_data.py:3: in <module>
???
dspt/dspt/sp/signal_processing.py:9: in <module>
from dspt.sp.filters import bilateral_filter
E ModuleNotFoundError: No module named 'dspt.sp.filters'
My folder structure looks like
.
├── my_package
│ ├── A.py
│ └── __init__.py
├── my_package2
│ ├── B.py
│ └── __init__.py
└── setup.py
And my setup.py looks like
from setuptools import setup
if __name__ == '__main__':
setup(name='my_packages',
packages=['my_package'])
When I run
python3 setup.py develop
The egg file is created locally and an egg-link file is placed in my site-packages directory. Also, the folder is added to easy_install.pth which means that both my_package and my_package2 are importable. This is different than running python3 setup.py install (only my_package would be available per the keyword argument passed to the setup function).
This same behavior occurs when installing with pip using the -e flag.
Is this intended behavior? Is there any way to replicate the functionality of install using develop?
I wonder if as well as .deb packages for example, it is possible in my setup.py I configure the dependencies for my package, and run:
$ sudo python setup.py install
They are installed automatically. Already researched the internet but all I found out just leaving me confused, things like "requires", "install_requires" and "requirements.txt"
Just create requirements.txt in your lib folder and add all dependencies like this:
gunicorn
docutils>=0.3
lxml==0.5a7
Then create a setup.py script and read the requirements.txt in:
import os
lib_folder = os.path.dirname(os.path.realpath(__file__))
requirement_path = lib_folder + '/requirements.txt'
install_requires = [] # Here we'll get: ["gunicorn", "docutils>=0.3", "lxml==0.5a7"]
if os.path.isfile(requirement_path):
with open(requirement_path) as f:
install_requires = f.read().splitlines()
setup(name="mypackage", install_requires=install_requires, [...])
The execution of python setup.py install will install your package and all dependencies. Like #jwodder said it is not mandatory to create a requirements.txt file, you can just set install_requires directly in the setup.py script. But writing a requirements.txt file is a best practice.
In the setup function call, you also have to set version, packages, author, etc, read the doc for a complete example: https://docs.python.org/3/distutils/setupscript.html
You package dir will look like this:
├── mypackage
│ ├── mypackage
│ │ ├── __init__.py
│ │ └── mymodule.py
│ ├── requirements.txt
│ └── setup.py
Another possible solution
try:
# for pip >= 10
from pip._internal.req import parse_requirements
except ImportError:
# for pip <= 9.0.3
from pip.req import parse_requirements
def load_requirements(fname):
reqs = parse_requirements(fname, session="test")
return [str(ir.req) for ir in reqs]
setup(name="yourpackage", install_requires=load_requirements("requirements.txt"))
You generate egg information from your setup.py, then you use the requirements.txt from these egg information:
$ python setup.py egg_info
$ pip install -r <your_package_name>.egg-info/requires.txt
In Python 3.4+, it is possible to use the Path class from pathlib, to do effectively the same thing as #hayj answer.
from pathlib import Path
import setuptools
...
def get_install_requires() -> List[str]:
"""Returns requirements.txt parsed to a list"""
fname = Path(__file__).parent / 'requirements.txt'
targets = []
if fname.exists():
with open(fname, 'r') as f:
targets = f.read().splitlines()
return targets
...
setuptools.setup(
...
install_requires=get_install_requires(),
...
)