I have a python package with the following directory structure:
my_package/
my_subpackage/
__init__.py
my_module.py
__init__.py
setup.py
When I generate the Python wheel and pip install it, I am not going to have the my_package namespace, so my_subpackage is going to be part of the global namespace of my virtualenv or whatever.
I know that the solution here is to create another directory called my_package and put everything inside it:
my_package/
my_package/
my_subpackage/
__init__.py
my_module.py
__init__.py
setup.py
But let's say that I cannot change the directory structure for some reason. Is there a way to add a virtual my_package namespace in setup.py with the first layout?
You can specify the package_dir option in your setup.py.
Something like this should work:
setup(
...
package_dir={'': '..'},
packages=['my_package']
...
)
I never used relative paths with package_dir though, so your confirmation would be appreciated.
Related
I am trying to create multiple Python packages in the same namespace without re-creating the full folder structure for that namespace.
This is my project structure:
some_folder/
src/
subpackage1/
functions.py
setup.py
another_folder/
src/
subpackage2/
functions.py
setup.py
The first setup.py looks like this (the second one is the same, only with "subpackage2" instead of "common" in name and packages:
from setuptools import setup
setup(
name="foo-bar-subpackage1",
version="0.1",
package_dir={"foo.bar": "."},
packages=["foo.bar.subpackage1"]
)
So basically I want to define multiple different package in the same shared namespace, which namespace I also don't want to re-create as a folder structure (e.g. foo/bar/subpackage1) and just start with the lowest level where my code is. I am aiming to use "native namespace packaging" as explained here and in PEP 420, but that example doesn't use package_dir.
What I expect to achieve: that after running pip install . or pip install -e . in both some_folder/src/subpackage1 and another_folder/src/subpackage2 I am able to do this:
from foo.bar.subpackage1.functions import a
from foo.bar.subpackage2.functions import b
# there might be something else under foo.* or foo.bar.* that is provided by other modules in other projects, which should also be able to import
What I get instead:
AttributeError: module 'darwin.transform' has no attribute 'subpackage1'
When I remove package_dir parameter from setup(), change packages parameter value to ["foo", "foo.bar", "foo.bar.subpackage1"] and re-create the full namespace structure like this:
some_folder/
src/
foo/
bar/
subpackage1/
functions.py
setup.py
it works, but this structure feels highly excessive. Is there a way to shorten it like I'm intending above?
I have two projects (trysetup1 and trysetup2) with the following structure:
I want to pip install package1 and use module1 from project trysetup2
my setup.py that under package1 looks like this:
import setuptools
setuptools.setup(
name="common",
version="1.0.2",
packages=setuptools.find_packages(),
)
the way I want to use module1 is like this from package1.module1 import ClassOne because I still need to use it from package2
when trying to import from module2 it works just fine
but when trying to use it from module3 (in the different project after pip installing it) i'm having "Unresolved reference 'package1'" problem
I know I'm able to use module1 by putting it inside another package under package1 but I need this exact stracture in order to use it from the rest of the project 'trysetup1'
Thanks!
My answer was found here:
https://docs.python.org/3/distutils/examples.html
actually, all I needed to do was to change my setup.py file to look like this:
setuptools.setup(
name="common",
version="1.0.2",
package_dir={'package1': ''},
packages=['package1'],
)
by adding package_dir param, setup function tells all files under my root directory (package1) to be under package1 directory and by adding packages param it distributes package1 package and then if you go to:
/..../venv/lib/python3.8/site-packages/common-1.0.2-py3.8.egg-info/top_level.txt
you'll see the following content:
With a project setup as follows:
---------------------
root
FrameworkPackage1
__init__.py
sourcefile1.py
FrameworkPackage2
__init__.py
sourcefile2.py
apps
Project
src
MyApp
__init__.py
__main__.py
setup.py
README.md
---------------------
When I'm creating the setup.py, from what I understand, I use package_dir to set the location of these packages.
---------------------
packages=['MyApp', 'FrameworkPackage1', 'FrameworkPackage2'],
package_dir={'': 'src',
'FrameworkPackage1': '../../FrameworkPackage1',
'FrameworkPackage2': '../../FrameworkPackage2'}
---------------------
So this correctly builds a package with all the required files. However, when I try to install, it fails, and if I just try to untar/gz the package file it puts FrameworkPackage1/2 in the "../../.." dir from where the unzip happens.
Ideally I'd like the package to work as follows and install from pip so I could run the following:
import MyApp as ma
import FrameworkPackage1 as fp1
import FrameworkPackage2 as fp2
print(ma.Function())
print(fp1.OtherFunction())
print(fp2.OtherFunction())
Is there a way to set the frameworks to be found in "../../../" but install into the root of the distribution?
Firstly, as #a_guest suggested, shouldn't the package_dir look like this?
packages=['MyApp', 'FrameworkPackage1', 'FrameworkPackage2'],
package_dir={'': 'src',
'FrameworkPackage1': '../../FrameworkPackage1',
'FrameworkPackage2': '../../FrameworkPackage2'}
Alternatively, you could try adding a __init__.py to the root folder so that it is recognized as a python folder (based on this question)
Secondly, instead of using this bundled structure for your package, you could either:
If the Framework packages are used elsewhere: Treat each package separately. This would allow you to evolve them separately, and add them to your MyApp by simply including them in the requirements.txt (or equivalents). A con of this is that each one would have its own setup.py, but this is a much better packaging practice.
If the Framework packages are not used elsewhere (or you just want your local copy): Switch to a project setup with the setup.py directly in the main folder ( package_dir={'': 'src', 'FrameworkPackage1': 'src', 'FrameworkPackage2': 'src'}, with a structure looking like:
---------------------
...
Project
src
MyApp
__init__.py
__main__.py
FrameworkPackage1
__init__.py
sourcefile1.py
FrameworkPackage2
__init__.py
sourcefile2.py
setup.py
README.md
---------------------
I have a repository I inherited used by a lot of teams, lots of scripts call it, and it seems like its going to be a real headache to make any structural changes to it. I would like to make this repo installable somehow. It is structured like this:
my_repo/
scripts.py
If it was my repository, I would change the structure like so and make it installable, and run python setup.py install:
my_repo/
setup.py
my_repo/
__init__.py
scripts.py
If this is not feasible (and it sounds like it might not be), can I somehow do something like:
my_repo/
setup.py
__init__.py
scripts.py
And add something to setup.py to let it know that the repo is structured funny like this, so that I can install it?
You can do what you suggest.
my_repo/
setup.py
__init__.py
scripts.py
The only thing is you will need to import modules in your package via their name if they are in the base level. So for example if your structure looked like this:
my_repo/
setup.py
__init__.py
scripts.py
lib.py
pkg/
__init__.py
pkgmodule.py
Then your imports in scripts.py might look like
from lib import func1, func2
from pkg.pkgmodule import stuff1, stuff2
So in your base directory imports are essentially by module name not by package. This could screw up some of your other packages namespaces if you're not careful, like if there is another dependency with a package named lib. So it would be best if you have these scripts running in a virtualenv and if you test to ensure namespacing doesn't get messed up
There is a directive in setup.py file to set the name of a package to install and from where it should get it's modules for installation. That would let you use the desired directory structure. For instance with a given directory structure as :
my_repo/
setup.py
__init__.py
scripts.py
You could write a setup.py such as:
setup(
# -- Package structure ----
packages=['my_repo'],
package_dir={'my_repo': '.'})
Thus anyone installing the contents of my_repo with the command "./setup.py install" or "pip install ." would end up with an installed copy of my_repo 's modules.
As a side note; relative imports work differently in python 2 and python 3. In the latter, any relative imports need to explicitly specify the will to do so. This method of installing my_repo will work in python 3 when calling in an absolute import fashion:
from my_repo import scripts
I've got a python project (projectA), which I've included as a git submodule in a separate project (projectB) as a subfolder. The structure is laid out as
projectB/
projectB_file.py
projectA/ (repository)
projectA/ (source code)
module1/
file1.py (contains Class1)
file2.py (contains Class2)
tests/
test_file1.py
I'm trying to figure out how to layout __init__.py files so in projectB_file.py I can import Class1 and Class2 as
from projectA.module1 import Class1
from projectA import Class2
I think having the top level projectA part of the import will be a mistake. That will require you to write your imports with projectA duplicated (e.g. import projectA.projectA.module1.file1).
Instead, you should add the top projectA folder to the module search path in some way (either as part of an install script for projectB, or with a setting in your IDE). That way, projectA as a top-level name will refer to the inner folder, which you actually intend to be the projectA package.
You'll need an __init__.py in each subdirectory you want to treat as a package. Which in your case means one in:
projectA
projectA/projectA
projectA/projectA/module1
projectA/projectA/tests
It would definitely be better you could lose that first projectA folder.
This is an annoying issue. The main approach I use is to edit the PYTHONPATH. This is essentially how I do it.
ProjectB/ # Main Repository
projectb/ # Package/library
__init__.py
projectB_file.py
docs/
tests/
setup.py # Setup file to distribute the library
freeze.py # CX_Freeze file to make executables
ProjectA/ # Subrepository
projecta/ # Package/library
__init__.py
projectA_file.py
file2.py
submodule/
__init__.py
file3.py
pa_module1/ # Additional package/library in 1 project
__init__.py
file1.py
docs/
tests/
setup.py
With this setup I have to edit the python path before running ProjectB's code. I use this structure, because it is easier for deployment and distribution. The subrepository helps track the version of ProjectA for each release of ProjectB.
cd ProjectB
export PYTHONPATH=${PWD}:${PWD}/ProjectA
python projectb/projectB_file.py
projectB_file.py
from projecta import projectA_file
This is for my development environment. For a release environment someone would install with the below code.
# Need a correct setup.py to handle 2 packages for pa_module1 and projecta
pip install ProjectA (install projecta to site-packages)
cd ..
pip install ProjectB (install projectb to site-packages)
This means projectb and projecta are in site packages. From there any file can simply
from projectb import projectB_file
from projecta import projectA_file, file2
from pa_module1 import file1
# Personally I don't like imports that use project.sub.sub.sub ...
# I have seen it cause import confusion and problems, but it is useful in some situations.
from projecta.submodule import file3
#import projecta.projectA_file
#import projecta # if __init__.py has import code
file1.Class1()
file2.Class2()
file3.Class3()
Additionally with pip you can install a library as a developer environment which links to the real project directory instead of copying files to site-packages.
pip install -e ProjectA