Packaging python project with multiple directories - python
I need some explanation on working with setuptools and find_packages function.
I have a project structure like this:
├── project_dir_1
│ ├── module.py
│ ├── __init__.py
├── my_project
│ ├── cli.py
│ ├── subdir1
│ │ ├── __init__.py
│ │ ├── module.py
│ ├── conf
│ │ ├── module.py
│ │ ├── params
│ │ │ ├── config.yml
│ │ ├── __init__.py
│ ├── subdir2
│ │ ├── module.py
│ ├── __init__.py
│ └── version.py
├── project_dir_2
│ ├── subdir1
│ │ ├── module.py
│ │ ├── __init__.py
│ ├── __init__.py
├── README.md
├── requirements.txt
├── setup.py
└── tests
└── test_main.py
Actually all my code in the my_project dir and I also have two additonal dirs project_dir_1 and project_dir_2 that contains necessary external modules from where I should do imports both in the package code and in another projects code where this package will be installed in venv.
I have setup script like this:
setup(
name='my_project',
version='0.0.1',
description='Python library.',
license='license',
author='me',
author_email='my_email',
entry_points={'console_scripts': ['my_project=my_project.cli:main']},
python_requires='>=3.7',
packages=find_packages(
include=['my_project', 'project_dir_1', 'project_dir_2', 'my_project.*', 'project_dir_1.*', 'project_dir_2.*']
),
install_requires=list(open(join(dirname(__file__), 'requirements.txt')).read().split()),
)
When I activate venv in another project folder and trying to install package from package root folder like python ..\package_root\setup.py install everything seems to works fine during install. And pip list shows all dependencies and my_project 0.0.1. But if I'm trying to import something from my_project using the venv interpreter I got an error: ModuleNotFoundError: No module named 'my_project'. The same result if I'm trying to import something like from project_dir_1 import module which is also necessary. Also when I just run my_project from shell addressing to cli I got an error:
Traceback (most recent call last):
File "/home/developer/another_project/env/bin/my_project", line 11, in <module>
load_entry_point('my_project==0.0.1', 'console_scripts', 'my_project')()
File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'my_project'
So what is the right way to organize this complex project structure and include all necessary code in setup.py to get the setuptools install the package correctly? I need some better understanding of python projects packaging but still don't get an answers for this case from seeking the docs.
find_packages will resolve paths relative to current working directory, so calling it outside of the project root dir will effectively install nothing (check whether you see any sources installed by e.g. running
$ pip show -f my_project
, I bet nothing will be listed). You have to force switching to the project root dir in the setup script, e.g. add a magic line to your setup script:
# setup.py
import os
from setuptools import setup
# old-style for python 2
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))
# new style for python 3
from pathlib import Path
os.chdir(Path(__file__).parent.absolute())
setup(...)
Related
Importing module from diffrent custome package from the same project.... ModuleNotFoundError
I have created 2 diffrent directories(Packages) in my project: ├── Package1 │ ├── __init__.py │ └── database.py │ └── crud.py │ └── models.py │ └── schemas.py │ └── main.py ├── Package2 │ ├── __init_.py │ └── code_file.py In Package2.code_file.py I have import models from Package1 by adding the below line from Package1 import models When I'm trying to run the code_file.py from my CLI using (venv) systemdetails:~project_location$ python3 Package2/code_file.py I'm getting an error that states Traceback (most recent call last): File "/project_location/Package2/code_file.py", line 12, in <module> from Package1 import models ModuleNotFoundError: No module named 'Package1' But when I run the file using the PyCharm => right-click => run 'code_file.py' it runs perfectly. I'm unable to understand where the problem is... Can anyone point me in write direction? I have tried removing init.py in both the Packages.... Did not work
Setup.py duplicates sub packages as separate one
I have a following project structure: . └── Project/ ├── package_1/ │ ├── package_2 │ ├── __init__.py │ ├── file_1.py │ ├── file_2.py │ └── file_3.py └── __init__.py As the package_2 contains the ported files from another project, I want to be able to install package_1 using setuptools so that it doesn't conflict with the original project and import it like this : import package_1.package_2 here is my setup.py file content : setup( name="Project", ... packages=find_packages(exclude=["package_1"]), ) So far everything works perfectly except that in \Lib\site-packages directory in addition to package_1/package_2 I also have the package_2 as a separate module which I think it is not OK.
Can't import subfolders from python module on GitHub
I have a simple python package that I've published on GitHub. I installed the package locally on my machine using pip. I am trying to import a subfolder of the module but I keep getting a ModuleNotFoundError: No module named 'package_folder.subfolder1' ├── package_name/ │ ├── README.md │ ├── setup.py │ └── package_folder │ ├── __init__.py │ ├── file1.py │ ├── file2.py │ ├── subfolder1/ │ │ ├── __init__.py │ │ ├── file11.py │ │ └── file12.py I have the __init__.py files in both directories, so I'm not sure why I am unable to access the subfolder1 files. I am able to import file1.py and file2.py from the top-level package_folder with from package_folder import file1.py.
In the setup.py you have to include the subfolder in the packages as well. So, in setup.py instead of: packages=['package_folder'] You have to do: packages=['package_folder', 'package_folder/subfolder1']
python alternative package setup using setuptools
I am having some trouble adding packages with my particular setup: . ├── pkg_a │ ├── pkg_a │ │ ├── __init__.py │ │ └── module_a.py │ └── run_a.py ├── pkg_b │ ├── pkg_b │ │ ├── __init__.py │ │ └── module_b.py │ └── run_b.py └── setup.py My goal is to be able to import package modules without repeating package name twice. For example, in run_a.py I'd like to be able to call from pkg_a import module_a instead of calling from pkg_a.pkg_a import module_a I tried to follow Section 2.1 of doc here. By creating setup.py as follow: from setuptools import setup setup( name="test", packages=['pkg_a', 'pkg_b'], package_dir={'pkg_a':'pkg_a/pkg_a', 'pkg_b':'pkg_b/pkg_b'} ) But this does not achieve the desired effect as mentioned above as I tried to call python setup.py develop and then python -c 'from pkg_a import module_a'. Is this particular setup achievable? And what am I messing up here? Thanks all!
package_dir modifications do not work with editable (aka develop) installations. The only acceptable package_dir modification for editable installations is the one that covers the so-called src-layout: package_dir={'': 'src'},
No module named utils error on compiling py file
I'm trying to run a .py file through the command prompt using the command "python filename.py". I've already set the environment variables for python after I installed it, so I don't get any error when I type python. The file I'm running imports a few directories, all of which are preexistent in the same directory as the file I'm running, apart from the file web.py, which I can't seem to locate in the directory, so I'm assuming it's somewhere inside the python package, I have downloaded. But, I couldn't find it there either, so would I need to install an extension for python for the web.py file to be successfully imported or is there another way around this. I've downloaded Python 3.4, I'm using windows 7 as my operating system and the exact error I receive when I try to compile the file is ImportError: No module named 'utils' Can someone please explain or direct me to a page which shows in detail how to install extensions for python?
The specific error happens when the Python interpreter can't find a particular ".py" file. In your case, it is the file "utils.py". First you need to find which file is trying to import "utils.py". Starting with your main file, look up all the files you are importing. (I am guessing this issue is coming from one of the non-library files, but I could be wrong.) Once you have the "top level" import list, check each of those files to see what THEY are importing, and repeat the process for them. Eventually, you will find the .py file which is trying to import "utils". There might be a directory specification forcing Python to look in the wrong place. Finally, using windows' file manager, perform a search for "utils.py". As a temporary fix, you can copy it from its current location into your working directory. That will at least allow you to get your project up and running until you sort out the real cause.
This error occurs due to file(s)/folder(s) that are not in their respective locations. I had a very similar error with a Python Flask Framework app, it turns out that my manage.py and config.py files were inside the app folder with the other folders(they were supposed to be outside the app directory), and that cause the error in my situation. Once I placed the files in their proper location boom error was gone. So Check you application framework and make sure things are located were they're supposed to be. Good luck
I installed via apt (I use debian linux) and had this same error in one project. For me, the solution was to install via pip: $ pip install utils It should work for both python 2 and python 3.
So in my case I ran tree command in my Pipenv environment and it should be look like as below: I hope this helps. . ├── README.md ├── __init__.py ├── core.yaml ├── core_blueprints │ ├── __init__.py │ ├── ami_lookup.py │ ├── chef_buckets.py │ ├── custom_resources │ │ ├── __init__.py │ │ └── cfn_custom_classes.py │ ├── cw_alarm.py │ ├── roles.py │ ├── security_groups.py │ ├── shared_iam │ │ ├── __init__.py │ │ └── iam_policies.py │ ├── sns_subscription.py │ ├── sns_topic.py │ ├── ssm_chefrun_documents.py │ ├── tf_state.py │ ├── utils . #### This is not correct location. │ │ ├── __init__.py │ │ ├── standalone_output.py │ │ ├── version.py │ │ └── version_check.py │ ├── vpc.py │ ├── vpn_eip.py │ └── vpn_server.py ├── core_hooks │ ├── __init__.py │ ├── cookbook_archive.py │ ├── core_lambda.py │ ├── keypair.py │ ├── s3.py │ ├── s3_cache.py │ └── ssm.py ├── platform_version.py ├── prd1-ca-central-1.env ├── setup.py └── utils ###### This is a correct location. ├── __init__.py ├── standalone_output.py ├── version.py └── version_check.py