I am trying to create multiple Python packages in the same namespace without re-creating the full folder structure for that namespace.
This is my project structure:
some_folder/
src/
subpackage1/
functions.py
setup.py
another_folder/
src/
subpackage2/
functions.py
setup.py
The first setup.py looks like this (the second one is the same, only with "subpackage2" instead of "common" in name and packages:
from setuptools import setup
setup(
name="foo-bar-subpackage1",
version="0.1",
package_dir={"foo.bar": "."},
packages=["foo.bar.subpackage1"]
)
So basically I want to define multiple different package in the same shared namespace, which namespace I also don't want to re-create as a folder structure (e.g. foo/bar/subpackage1) and just start with the lowest level where my code is. I am aiming to use "native namespace packaging" as explained here and in PEP 420, but that example doesn't use package_dir.
What I expect to achieve: that after running pip install . or pip install -e . in both some_folder/src/subpackage1 and another_folder/src/subpackage2 I am able to do this:
from foo.bar.subpackage1.functions import a
from foo.bar.subpackage2.functions import b
# there might be something else under foo.* or foo.bar.* that is provided by other modules in other projects, which should also be able to import
What I get instead:
AttributeError: module 'darwin.transform' has no attribute 'subpackage1'
When I remove package_dir parameter from setup(), change packages parameter value to ["foo", "foo.bar", "foo.bar.subpackage1"] and re-create the full namespace structure like this:
some_folder/
src/
foo/
bar/
subpackage1/
functions.py
setup.py
it works, but this structure feels highly excessive. Is there a way to shorten it like I'm intending above?
Related
I have some python modules that I currently have packaged from one directory tree, which looks like:
toplevel/
setup.cfg
setup.py
toplevel/
__init_.py
a.py
b.py
c.py
...
and these get imported by something that looks like
import toplevel.a
Because a.py ,b.py, and c.py have very different dependencies, I'd like to change this so that they are in separate packages with separate dependencies.
I've tried to do this using native namespace packages, but all the examples in the docs I have seen seem to result in a package where the modules are a sublevel lower, so that the imports look like
import toplevel.a.a
I've gotten that to work, but that isn't what I want. Can what I want be done with native namespace packages?
It looks to me like this isn't possible (I look forward to being corrected if it is).
However, there is a workaround that seems to work for me. If you have native namespace packaging working for imports of the form
import toplevel.a.a
with each separate package tree like
pyproject.toml
setup.cfg
src\
toplevel\
a\
__init__.py
a.py
then instead of having an empty __init.py__ at the leaf level, you can instead have one that contains
from .a import *
which then will load all the names from a.py into the toplevel.a namespace, at which point they are accessible via
import toplevel.a
I found https://towardsdatascience.com/whats-init-for-me-d70a312da583 helpful in thinking about this.
I have two projects (trysetup1 and trysetup2) with the following structure:
I want to pip install package1 and use module1 from project trysetup2
my setup.py that under package1 looks like this:
import setuptools
setuptools.setup(
name="common",
version="1.0.2",
packages=setuptools.find_packages(),
)
the way I want to use module1 is like this from package1.module1 import ClassOne because I still need to use it from package2
when trying to import from module2 it works just fine
but when trying to use it from module3 (in the different project after pip installing it) i'm having "Unresolved reference 'package1'" problem
I know I'm able to use module1 by putting it inside another package under package1 but I need this exact stracture in order to use it from the rest of the project 'trysetup1'
Thanks!
My answer was found here:
https://docs.python.org/3/distutils/examples.html
actually, all I needed to do was to change my setup.py file to look like this:
setuptools.setup(
name="common",
version="1.0.2",
package_dir={'package1': ''},
packages=['package1'],
)
by adding package_dir param, setup function tells all files under my root directory (package1) to be under package1 directory and by adding packages param it distributes package1 package and then if you go to:
/..../venv/lib/python3.8/site-packages/common-1.0.2-py3.8.egg-info/top_level.txt
you'll see the following content:
I have a python package with the following directory structure:
my_package/
my_subpackage/
__init__.py
my_module.py
__init__.py
setup.py
When I generate the Python wheel and pip install it, I am not going to have the my_package namespace, so my_subpackage is going to be part of the global namespace of my virtualenv or whatever.
I know that the solution here is to create another directory called my_package and put everything inside it:
my_package/
my_package/
my_subpackage/
__init__.py
my_module.py
__init__.py
setup.py
But let's say that I cannot change the directory structure for some reason. Is there a way to add a virtual my_package namespace in setup.py with the first layout?
You can specify the package_dir option in your setup.py.
Something like this should work:
setup(
...
package_dir={'': '..'},
packages=['my_package']
...
)
I never used relative paths with package_dir though, so your confirmation would be appreciated.
I have a package named 'firstlib' which I want to include completely in another package while keeping the code separated.
For example, the directory structure looks like this:
Proj1/
setup.py
firstlib/
__init__.py
mod1.py
subpack/
...
Proj2/
setup.py
otherlib/
__init__.py
mod2.py
...
Now i want to be able to use 'firstlib' as a standalone package but also as a subpackage of 'otherlib'. Assuming both packages are installed, what needs to be done to make these lines work:
from firstlib import mod1
from otherlib.firstlib import mod1
So basicaly I want the namespace of firstlib in the same namespace as modules of otherlib. In a way like a namespace package.
I have a repository I inherited used by a lot of teams, lots of scripts call it, and it seems like its going to be a real headache to make any structural changes to it. I would like to make this repo installable somehow. It is structured like this:
my_repo/
scripts.py
If it was my repository, I would change the structure like so and make it installable, and run python setup.py install:
my_repo/
setup.py
my_repo/
__init__.py
scripts.py
If this is not feasible (and it sounds like it might not be), can I somehow do something like:
my_repo/
setup.py
__init__.py
scripts.py
And add something to setup.py to let it know that the repo is structured funny like this, so that I can install it?
You can do what you suggest.
my_repo/
setup.py
__init__.py
scripts.py
The only thing is you will need to import modules in your package via their name if they are in the base level. So for example if your structure looked like this:
my_repo/
setup.py
__init__.py
scripts.py
lib.py
pkg/
__init__.py
pkgmodule.py
Then your imports in scripts.py might look like
from lib import func1, func2
from pkg.pkgmodule import stuff1, stuff2
So in your base directory imports are essentially by module name not by package. This could screw up some of your other packages namespaces if you're not careful, like if there is another dependency with a package named lib. So it would be best if you have these scripts running in a virtualenv and if you test to ensure namespacing doesn't get messed up
There is a directive in setup.py file to set the name of a package to install and from where it should get it's modules for installation. That would let you use the desired directory structure. For instance with a given directory structure as :
my_repo/
setup.py
__init__.py
scripts.py
You could write a setup.py such as:
setup(
# -- Package structure ----
packages=['my_repo'],
package_dir={'my_repo': '.'})
Thus anyone installing the contents of my_repo with the command "./setup.py install" or "pip install ." would end up with an installed copy of my_repo 's modules.
As a side note; relative imports work differently in python 2 and python 3. In the latter, any relative imports need to explicitly specify the will to do so. This method of installing my_repo will work in python 3 when calling in an absolute import fashion:
from my_repo import scripts