We Keep Coding

ModuleNotFoundError after pip install local python package

Structure:
.
├── application
│ └── runner.py
└── dummyLibrary
├── helperFunctions.py
├── __init__.py
└── setup.py
runner.py:
import dummyLibrary
dummyLibrary.foo()
dummyLibrary.bar()
init.py:
(empty file)
helperFunctions.py:
def foo():
print("called foo()")
def bar():
print("called bar()")
setup.py:
#!/usr/bin/env python
from distutils.core import setup
setup(name='dummyLibrary', version='0.0.1')
After cd'ing into dummyLibrary/ ,
I tried installing dummyLibrary with
pip3 install -e .
This was the output:
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/Documents/pythonTest/dummyLibrary
Preparing metadata (setup.py) ... done
Installing collected packages: dummyLibrary
Running setup.py develop for dummyLibrary
Successfully installed dummyLibrary-0.0.1
I tried installing dummyLibrary with
python3 -m pip install -e .
This was the output:
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/Documents/pythonTest/dummyLibrary
Preparing metadata (setup.py) ... done
Installing collected packages: dummyLibrary
Attempting uninstall: dummyLibrary
Found existing installation: dummyLibrary 0.0.1
Uninstalling dummyLibrary-0.0.1:
Successfully uninstalled dummyLibrary-0.0.1
Running setup.py develop for dummyLibrary
Successfully installed dummyLibrary-0.0.1
After cd'ing into application/ and running
python3 runner.py
I get:
Traceback (most recent call last):
File "runner.py", line 1, in <module>
import dummyLibrary
ModuleNotFoundError: No module named 'dummyLibrary'
no matter how I try to install my library
Why is this?
Additional Information:
Using Ubuntu, Not using a virtual environment.
I tried rebooting after installing. Didn't help.

Welcome to Stack Overflow!
First thing: __init__.py and helperFunctions.py should be inside another nested folder with the same name as the package.
.
├── application
│ └── runner.py
└── dummyLibrary
├── dummyLibrary
│ ├── __init__.py
│ └── helperFunctions.py
└── setup.py
Secondly, with the code in runner.py as how you'd like it, the __init__.py should include the following line to import all of the functions in helperFunctions.py:
from .helperFunctions import *
Finally, your setup.py should also include the parameter packages as a list of folder names in the package, in this case, the parameter should be packages=['dummyLibrary'].
P.S.: it's a pythonic practice to name your files and packages in snake_case rather than camelCase.

pkg_resources.ResolutionError with python setup.py install?

I used to be able to run the command runner.py from my module my-runner while using python setup.py develop. However, ever since that I reinstalled it using python setup.py install, I now get a pkg_resources.ResolutionError when calling runner.py.
This is the mini tree structure
.
├── bin
│   ├── some_other_file.py
│   ├── runner.py
├── setup.py
here is my setup.py
from setuptools import setup, find_packages
setup(
name='my-runner',
version='1.0.0',
license='private',
author='MyName',
author_email='myname#myemail.com',
description='My Runner',
packages=find_packages(),
scripts=['bin/runner.py', 'bin/some_other_file.py']
)
Running command runner.py returns the error
pkg_resources.ResolutionError: Script 'scripts/runner.py' not found in metadata at '/home/myname/module/my-runner.egg-info
I guess I have no clue why install would break it? I am guessing it has to do with the fact that develop does not read the egg-info dir but would like a solution to this problem.

A possible way is to remove the package(pip3 uninstall my-runner) and reinstall it (python setup.py install)

pip: module installed as editable not found

I'm trying to install the source checkout of my own package as an editable package using pip, but it doesn't get picked up as installed.
Yes, there are a whole lot of related questions, but they're mostly about non-standard directory structures. This here seems like it should "just work" and partially even does.
# starting from an empty venv:
(env) PS C:\test> pip install -e C:\path\to\my\PySymCircuit
Obtaining file:///C:/path/to/my/PySymCircuit
Preparing metadata (setup.py) ... done
Collecting sympy
Using cached sympy-1.9-py3-none-any.whl (6.2 MB)
Collecting mpmath>=0.19
Using cached mpmath-1.2.1-py3-none-any.whl (532 kB)
Installing collected packages: mpmath, sympy, SymCircuit
Looks good, but:
(env) PS C:\test> python -c 'import symcircuit'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'symcircuit'
(env) PS C:\test> pip show SymCircuit
WARNING: Package(s) not found: SymCircuit
The link file that is created is correct (as well as the egg-info in the source directory), and when I try to uninstall the package, suddenly it is recognized. Just not when it matters...
(env) PS C:\test> cat .\env\Lib\site-packages\SymCircuit.egg-link
C:\path\to\my\PySymCircuit
.
(env) PS C:\test> pip uninstall SymCircuit
Found existing installation: SymCircuit 0.1.0
Uninstalling SymCircuit-0.1.0:
Would remove:
c:\test\env\lib\site-packages\symcircuit.egg-link
Proceed (Y/n)? Y
WARNING: Cannot remove entries from nonexistent file c:\test\env\lib\site-packages\easy-install.pth
Successfully uninstalled SymCircuit-0.1.0
Plain setup.py develop produces the same link, also not usable.
What am I missing?
A non-editable install from the same source tree works as intended, so I believe it has something to do with the linking mechanism?
Update: so apparently the link is just for pip, the important part is that setuptools should add the path to easy-install.pth. It doesn't, hence the error message when uninstalling.
Now the question is, why...
Update as requested:
Local directory structure: git clone https://github.com/martok/py-symcircuit.git
py-symcircuit
│ LICENSE
│ README.md
│ setup.py
├───.git
├ ...
├───symcircuit
│ __init__.py
│ bode.py
│ spice.py
│ system.py
└───SymCircuit.egg-info
dependency_links.txt
PKG-INFO
requires.txt
SOURCES.txt
top_level.txt
Setup.py:
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(
name="SymCircuit",
version="0.1.0",
author="Martok",
author_email="martok#martoks-place.de",
description="Symbolic electronic circuit analysis",
long_description=open("README.md","rt").read(),
long_description_content_type="text/markdown",
url="https://github.com/martok/py-symcircuit",
project_urls={
"Bug Tracker": "https://github.com/martok/py-symcircuit/issues",
},
license="MIT",
classifiers=[...],
packages=find_packages(),
python_requires='>=3.6',
install_requires=[
"sympy",
],
extras_require={
"EE": [
"networkx",
"numpy",
"mplotkit"
],
},
)
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)] on win32
Package Version
---------------- ---------
pip 21.3.1
setuptools 59.4.0

Copy a non-Python file to specific directory during Pip Install

Problem Statement: when I install my pip package, a specific file inside the package get coped to Temp directory
Approach:
My Package directory Sturcture is following:
my-app/
├─ app/
│ ├─ __init__.py
│ ├─ __main__.py
├─ folder-with-extra-stuff/
│ ├─ __init__.py
│ ├─ file_I_want_to_cppy.tar.gz
├─ setup.py
├─ MANIFEST.in
I'm tweaking my setup.py file to do the job. Following is my setup.py
#!/usr/bin/env python
from setuptools import setup, find_packages
from setuptools.command.install import install
import os
import sys
import shutil
rootDir = os.path.abspath(os.path.dirname(__file__))
def run_custom_install():
print("--------Start running custom command -------")
temp_dir = r'c:\temp' if sys.platform == "win32" else r'/tmp'
temp_col_dir = temp_dir + os.sep + 'dump'
os.makedirs(temp_dir, exist_ok=True)
os.makedirs(temp_col_dir, exist_ok=True)
print("----------locate the zip file ---------------")
ColDirTests = os.path.abspath(os.path.join(rootDir, 'my-app','folder-with-extra-stuff'))
_src_file = os.path.join(ColDirTests , 'file_I_want_to_cppy.tar.gz ')
print(f"******{_src_file}**********")
if os.path.exists(_src_file):
print(f"-----zip file has been located at {_src_file}")
shutil.copy(_src_file, temp_col_dir)
else:
print("!!!!Couldn't locate the zip file for transfer!!!!")
class CustomInstall(install):
def run(self):
print("***********Custom run from install********")
install.run(self)
run_custom_install()
ver = "0.0.0"
setup(
name='my_pkg',
version=ver,
packages=find_packages(),
python_requires='>=3.6.0',
install_requires = getRequirements(),
include_package_data= True,
cmdclass={
'install' : CustomInstall,
}
)
MANIFEST.in
include README.md
include file_I_want_to_cppy.tar.gz
recursive-include my-app *
global-exclude *.pyc
include requirements.txt
prune test
Testing build:
> python setup.py bdist_wheel
It is working during build. I can see there is a directory formed C:\temp\dump and file_I_want_to_cppy.tar.gz inside it. But when I release the package in pip and try to install it from pip, the folder remains Empty!
Any idea what I might be doing wrong here?

After a lot of research I have figure out how to resolve this issue. Let me summarize my findings, it might be helpful for other who wants to do post_pip_install processing.
setup.py
Different options to install package: 1) pip install pkg_name, 2) python -m setup.py sdist
If you want to make them work in either ways, need to have install, egg_info and develop all 3 options repeated as shown in setup.py
If you create *.whl file by python -m setup.py bdist_wheel , post pip install processing won't be executed! Please upload .tar.gz format generated usingsdist to PyPi/Artifacts to make post pip install processing work. Again, Please note: It will not work when installing from a binary wheel
upload the pip package: twine upload dist/*.tar.gz
from setuptools import setup, find_packages
from setuptools.command.install import install
from setuptools.command.egg_info import egg_info
from setuptools.command.develop import develop
rootDir = os.path.abspath(os.path.dirname(__file__))
def run_post_processing():
print("--------Start running custom command -------")
# One can Run any Post Processing here that will be executed post pip install
class PostInstallCommand(install):
def run(self):
print("***********Custom run from install********")
install.run(self)
run_post_processing()
class PostEggCommand(egg_info):
def run(self):
print("***********Custom run from Egg********")
egg_info.run(self)
run_post_processing()
class PostDevelopCommand(develop):
def run(self):
print("***********Custom run from Develop********")
develop.run(self)
run_post_processing()
ver = "0.0.0"
setup(
name='my_pkg',
version=ver,
packages=find_packages(),
python_requires='>=3.6.0',
install_requires = getRequirements(),
include_package_data= True,
cmdclass={
'install' : PostInstallCommand,
'egg_info': PostEggCommand,
'develop': PostDevelopCommand
}
)
Few More Things from my research:
If you want to do pre-processing instead of post-processing, need to move install.run(self) at the end
while pip installing, if you want to see custom messages of pre/post instllation, use -vvv. Example: pip install -vvv my_pkg

Install Python package from monorepo

We have a private git monorepo which hosts a number of Python packages. Poetry was the dependency management tool initially chosen for the project. Anyway, due to this Poetry issue, it would not be accepted solution that involves creating new setup.py files.
A simplified version of the structure:
git-monorepo
├── pkg-1
│ ├── pkg
│ │ └── mod1.py
│ └── pyproject.toml
├── pkg-2
│ ├── pkg
│ │ └── mod2.py
│ └── pyproject.toml
└── lib
├── pkg
│ └── lib.py
└── pyproject.toml
The library distribution package lib is indepentent from any other package. However, pkg-1 depends on lib and pkg-2 depends on both pkg-1 and lib.
So, the question is:
How would be the proper way to use pip to install a package from this monorepo?
Let us consider as an example that we try to install pkg-1, where pkg-1/pyproject.toml includes the following lines:
...
[tool.poetry.dependencies]
lib = {path = "../lib/"}
...
The result from running pip, as explained in the VCS support documentation:
$ pip install -e git+https://gitlab.com/my-account/git-monorepo#"egg=pkg-1&subdirectory=pkg-1"
Traceback (most recent call last):
File "/home/hblanco/.local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
super(Requirement, self).__init__(requirement_string)
File "/home/hblanco/.local/lib/python3.8/site-packages/pip/_vendor/packaging/requirements.py", line 115, in __init__
raise InvalidRequirement("Invalid URL: {0}".format(req.url))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: ../lib

The problem in the above setup is that the dependency is specified as a path dependency. When installing it, it uses that path dependency.
I ran into the same issue with a python monorepo, where I wanted to share the packages also to other projects.
I found 2 approaches to work for me:
in the CI/CD build pipeline, edit the pyproject.toml just before creating the wheel (which is published to a pypi repo)
First create the wheel (or .tar.gz) artifact, and then modify it afterwards (by extracting it, replacing the path dependencies, and zipping it again).
The full approach I've explained here.
However, it won't work with the git+https://...
You'll need a (private) pypi repo somewhere. Gitlab provides one for each project, which I utilize in the demo project here