I have what I think is a pip bug, and I want to double-check that it's not actually a mistake of mine before submitting it as a formal issue. If it's not a bug, I'd appreciate an explanation of what I'm doing wrong.
I have a project structure like so:
project/
setup.py
project_src/
__init__.py
...
common_utils/
utils_src/
__init__.py
...
I want to be able to:
import code from "project/project_src" via import project_src (this isn't the issue, I just want to be comprehensive)
import code from "project/common_utils/utils_src" via import utils_src (note this strips the "common_utils" folder from the package path name)
In order to do this, the root-level "setup.py" looks something like this (abbreviated):
# setup.py
import setuptools
setuptools.setup(
...,
packages=['project_src', 'utils_src'],
package_dir={
'project_src': 'project_src',
'utils_src': 'common_utils/utils_src',
},
...,
)
Now here's my issue. When I then install this package locally via CL as pip install project/, I can then open an interpreter and successfully run import project_src and import utils_src. But if I install via pip install -e project/, import project_src works but import utils_src triggers a ModuleNotFoundError error. (This is a huge pain as I rely on using the -e flag for development.)
Again, please let me know if this appears to be a bug, or if this is a mistake on my part.
Not a bug in pip, your mistake. You want to use common_utils/ directory as a parent dir for a package but you want utils_src as a package inside it. So change your setup.py:
package_dir={
'project_src': 'project_src',
'utils_src': 'common_utils',
},
With this utils_src will be installed as a top-level package so you can do
import utils_src
PS. They say this doesn't work with
pip install -e
I didn't test the rumor yet.
Turns out this is a long-standing issue: pypa/setuptools #230: develop mode does not respect src structure
Thanks #sinoroc for the hint # comments on this answer
Related
I guess my question is a duplicate, but, unfortunately, I haven't found a solution that corresponds to my problem.
I have following project structure:
↓ project_root
↓ source_root
__init__.py
↓ inner_package
some_executable_file.py
some_library_file.py
So I would like to import a name from 'some_library_file' in the following way:
from source_root.inner_package.some_library_file import X
But when I do something like this, I see the following error:
ModuleNotFoundError: No module named 'source_root'
You need to install your own project to your (ideally virtual) environment. In order to do this, you need a setup.py file, which will be used by pip to find all your packages (i.e., folders containing __init__.py) during installation.
Minimal example:
project_root/setup.py
from setuptools import setup, find_packages
setup(
name='MyProjectName',
version='0.0.0',
packages=find_packages(),
)
Then from the command line, cd into your project_root ant type:
python -m pip install -e .
You now installed your own project into your Python environment. Files in the project are able to import their own packages with the from source_root.[...] import [...] syntax.
This process mimics a user pip installing your project from PyPI / github, so you can be sure that your imports are going to work on any other environment/machines, provided pip is used for installation.
The -e flag is used to install the project in editable mode, so you won't need to reinstall the package locally after changing one of the source files (which you'll be doing a lot during development).
A newb where pytest is concerned.
I set up my structure: my_proj\src with the app files under src, and my_proj\tests with the test files under there. Following this book, I then added files like my_proj\setup.py, which looks like this currently:
from setuptools import setup, find_packages
setup(
name='my_proj',
version='0.1.0',
license='proprietary',
...
packages=find_packages(where='src'),
package_dir={'': 'src'},
install_requires=[],
extras_require={},
entry_points={
'console_scripts': [
'py_lib = py_lib.cli:py_lib_cli',
]
},
)
Then, as per the book, I ran
> pip install -e .
... and by some magic beyond my understanding this then set up things so that under the my_proj\tests side of things, the test files were able to import .py files from under my_proj\src.
Then I wanted to add a new .py file to the src files, i.e. the application files, the files under my_proj\src. But the tests are completely unable to see it: "No module named 'xxx'".
I tried pip install -e . again. I also tried pip develop ., and I tried the --no-cache-dir flag. I even tried running setup.py directly, which apparently is not recommended.
Then I found that under my_proj\src there is a directory which has been generated, my_proj.egg-info. One of the files under there is SOURCES.txt, which appears to list all the files in the project. My new .py file under \src is indeed listed there. And yet the test framework apparently can't see it.
Finally I tried uninstalling the thing:
pip uninstall my_proj
...
Proceed (y/n)? y
...
and reinstalling
pip install -e .
Tests still can't see (i.e. import) the new file. Grrr.
Can anyone explain this and tell me how to resolve the problem?
Edit
In answer to the comment: I added a new file, under src\utils: new_file.py, like this:
# -*- coding: utf-8 -*-
conc_fh = False
line_of_stars=50*'*' + '\n'
The test file is my_proj\tests\unit\test_misc_utils.py. I am usually running pytest from my_proj\tests\unit, although I have also tried the parent and grandparent directories. In that file this is the import line:
import utils.new_file
Same fail (for the new file). Then I tried all those possible remedies again.
I also tried deleting things like the my_proj.egg-info directory, and the my_proj\tests.pytest_cache and my_proj\tests\__pycache__ directories.
I can only conclude (provisionally) that something under site-packages in the virtual environment is incapable of updating correctly.
I have even tried pip uninstall pytest and reinstall, followed by pip install -e . again. Even then: NO!
Can someone confirm that adding a file works without problems for them (ideally on a W10 OS)?
NB this is the fail output:
______________________________________ ERROR collecting unit/test_misc_utils.py _______________________________________
ImportError while importing test module 'D:\My documents\software projects\EclipseWorkspace\my_proj\tests\unit\test_misc_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\users\mike\appdata\local\programs\python\python39\lib\importlib\__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests\unit\test_misc_utils.py:3: in <module>
import utils.new_file
E ModuleNotFoundError: No module named 'utils.new_file'
Later
I completely created a new project, by copying the old one, but setting up an entirely new virtual environment.
I then installed pytest again, and again ran pip install -e ., and then pytest tests/unit from the project directory.
To my utter amazement, once again, these two new files, the ones recently created (and copied in the process of making a new project), and only them, failed to be recognised and could not be imported.
Now I am totally baffled. It would appear that there is something about the nature of these files which is "defective". I've tried looking at the lines in them, for things like spurious Unicode characters. Nothing so far...
Think I finally worked it out.
The question is: what is available (for import) to the modules and files under "my_proj/tests". I believe this is determined by setup.py (as outlined in the referenced book). The most bare-boned version of this is:
from setuptools import setup
setup(
name='my_proj2',
py_modules=['my_proj2']
)
When you then run pip install -e . (note dot), this means that modules under my_proj2/src become available for import. But perhaps not everything. The downloadable files (again for the book) include a module [root]/src/tasks ... and under that is an __init__.py, which explicitly imports various variables and classes from that module, for re-export to the test classes.
This approach (including anything you want to import into tests in __init__.py of an application module) is not the only way of accessing stuff in modules under [root]/src when running tests. Any valid path under [root]/src appears to work. I have no idea why it failed on my previous attempts. Maybe the previous setup.py file (tweaked from the downloaded version) contained something complicated which interfered with things.
I have a python package with the following standard directory structure:
package_name/
setup.py
package_name/
module_1.py
module_2.py
...
tests/
docs/
I've installed this package with pip3 install -e .. I've noticed an inconsistent importing issue. (Please read to the end!) If I restart terminal and run the following (1) within the interpreter:
>>> from package_name import module_1
I get an import error. If I instead run this (2):
>>> from package_name.package_name import module_1
it imports fine. If I then navigate to the directory and rerun pip3 install -e ., I can import in the standard way (following (1)). What on earth is causing this? To make things stranger, I can import in the standard way (1) in Jupyter and my IDE without reinstalling the package. This issue only comes up when I open/restart terminal.
This should be fixed by adding the main project folder package_name/ to your PATH
Also, try renaming your project folders with different names, in order to avoid confusion for yourself, people working with you and also to help python to find the right module location
You should also create the __init__.py files on each module folders, even if those are empty files. This also helps python to find the modules locations.
I'm trying to let users write code as a python module (folder with __init__.py defined) under whatever folder name they see fit. After that I want to install that module as a python package but define the import name myself.
The folder structure would be like this:
project_name/
user_defined_name/
__init__.py
...
setup.py
According to this I should be able to add this to my setup.py to get it working:
setuptools.setup(
package_dir={'my_defined_name': 'user_defined_name'},
packages=['user_defined_name']
)
But the only way that I was able to access the code was by using import user_defined_name. I tried installing the package without -e but that gave the same result. Leaving the packages=['..'] out of the setup functions also did not change the result.
My question is kind of the same as this one and there the only answers seem to be to change folder names and that is something that I would like to avoid. That question mentioned that it might be due to a problem in setuptools but that seemed fixed 3 years ago.
In short, it looks like you need something like that in your setup.py:
setuptools.setup(
package_dir={
'my_defined_name': 'user_defined_name',
},
packages=[
'my_defined_name',
],
)
as Ilia Novoselov said in a comment to your question.
This should work, if you package and install the project normally. You would be able to import my_defined_name.
Although, note that as far as I can tell, this will not work if you use an editable installation (python setup.py develop or python -m pip install --editable .). It will be impossible to import my_defined_name, but you would be able to import user_defined_name, which is not what you want.
#Oliver's answer here clarified this for me.
My TLDR is that to support both releases (python setup.py install and pip install .) and editable installs (python setup.py develop and pip install -e .) you must change your file structure to
project_name
setup.py
user_defined_name
my_defined_name
__init__.py
...
docs
tests
and your setup.py to
setuptools.setup(
package_dir={'': 'user_defined_name'},
packages=['my_defined_name']
)
You can support just releases (NOT editable installs) with
project_name
setup.py
user_defined_name
__init__.py
...
docs
tests
setuptools.setup(
package_dir={'my_defined_name': 'user_defined_name'},
packages=['my_defined_name']
)
I have forked a repo and now I have cloned it. When running the setup.py file inside, Python installs the package inside site-packages as an obscure name, that of which importing this within a Python file will not be viable.
For example, I fork and clone a repo called foo. I can also see this in the setup.py file:
setup(
name='foo',
version='3.3.0-rc6',
packages=find_packages('src'),
package_dir={'': 'src'},
include_package_data=True,
When I run python setup.py install, I find the package has been installed as foo-3.3.0rc6-py3.6.egg. I do not want to have to import the package as this name in every one of my projects utilizing it.
How can I just change the package name to foo (when running/installing via setup.py), so that I can run import foo and not import foo-3.3...?
I feel I can not just rename it, as if I wanted other users to clone the repo and not have to go through the same trouble as me. Is the package name embedded somewhere in the setup.py folder?
Let me know if you need anything else, I'm willing to have this issue resolved.
You don't have to import foo-3.3; actually you cannot import as it's SyntaxError.
You don't have to import foo-3.3 from foo-3.3.0rc6-py3.6.egg because distutils/setuptools configured correct import path for eggs. Look into easy-install.pth file and you find there ./foo-3.3.0rc6-py3.6.egg. Run python and verify sys.path — there have to be foo-3.3.0rc6-py3.6.egg entry so that import foo works.
That's just the name of the egg, and you needn't worry about it. Python knows where to look for the package, so when you do:
import foo
...it'll do the right thing.