I have a package named 'firstlib' which I want to include completely in another package while keeping the code separated.
For example, the directory structure looks like this:
Proj1/
setup.py
firstlib/
__init__.py
mod1.py
subpack/
...
Proj2/
setup.py
otherlib/
__init__.py
mod2.py
...
Now i want to be able to use 'firstlib' as a standalone package but also as a subpackage of 'otherlib'. Assuming both packages are installed, what needs to be done to make these lines work:
from firstlib import mod1
from otherlib.firstlib import mod1
So basicaly I want the namespace of firstlib in the same namespace as modules of otherlib. In a way like a namespace package.
Related
I am trying to create multiple Python packages in the same namespace without re-creating the full folder structure for that namespace.
This is my project structure:
some_folder/
src/
subpackage1/
functions.py
setup.py
another_folder/
src/
subpackage2/
functions.py
setup.py
The first setup.py looks like this (the second one is the same, only with "subpackage2" instead of "common" in name and packages:
from setuptools import setup
setup(
name="foo-bar-subpackage1",
version="0.1",
package_dir={"foo.bar": "."},
packages=["foo.bar.subpackage1"]
)
So basically I want to define multiple different package in the same shared namespace, which namespace I also don't want to re-create as a folder structure (e.g. foo/bar/subpackage1) and just start with the lowest level where my code is. I am aiming to use "native namespace packaging" as explained here and in PEP 420, but that example doesn't use package_dir.
What I expect to achieve: that after running pip install . or pip install -e . in both some_folder/src/subpackage1 and another_folder/src/subpackage2 I am able to do this:
from foo.bar.subpackage1.functions import a
from foo.bar.subpackage2.functions import b
# there might be something else under foo.* or foo.bar.* that is provided by other modules in other projects, which should also be able to import
What I get instead:
AttributeError: module 'darwin.transform' has no attribute 'subpackage1'
When I remove package_dir parameter from setup(), change packages parameter value to ["foo", "foo.bar", "foo.bar.subpackage1"] and re-create the full namespace structure like this:
some_folder/
src/
foo/
bar/
subpackage1/
functions.py
setup.py
it works, but this structure feels highly excessive. Is there a way to shorten it like I'm intending above?
I have some python modules that I currently have packaged from one directory tree, which looks like:
toplevel/
setup.cfg
setup.py
toplevel/
__init_.py
a.py
b.py
c.py
...
and these get imported by something that looks like
import toplevel.a
Because a.py ,b.py, and c.py have very different dependencies, I'd like to change this so that they are in separate packages with separate dependencies.
I've tried to do this using native namespace packages, but all the examples in the docs I have seen seem to result in a package where the modules are a sublevel lower, so that the imports look like
import toplevel.a.a
I've gotten that to work, but that isn't what I want. Can what I want be done with native namespace packages?
It looks to me like this isn't possible (I look forward to being corrected if it is).
However, there is a workaround that seems to work for me. If you have native namespace packaging working for imports of the form
import toplevel.a.a
with each separate package tree like
pyproject.toml
setup.cfg
src\
toplevel\
a\
__init__.py
a.py
then instead of having an empty __init.py__ at the leaf level, you can instead have one that contains
from .a import *
which then will load all the names from a.py into the toplevel.a namespace, at which point they are accessible via
import toplevel.a
I found https://towardsdatascience.com/whats-init-for-me-d70a312da583 helpful in thinking about this.
I have two projects (trysetup1 and trysetup2) with the following structure:
I want to pip install package1 and use module1 from project trysetup2
my setup.py that under package1 looks like this:
import setuptools
setuptools.setup(
name="common",
version="1.0.2",
packages=setuptools.find_packages(),
)
the way I want to use module1 is like this from package1.module1 import ClassOne because I still need to use it from package2
when trying to import from module2 it works just fine
but when trying to use it from module3 (in the different project after pip installing it) i'm having "Unresolved reference 'package1'" problem
I know I'm able to use module1 by putting it inside another package under package1 but I need this exact stracture in order to use it from the rest of the project 'trysetup1'
Thanks!
My answer was found here:
https://docs.python.org/3/distutils/examples.html
actually, all I needed to do was to change my setup.py file to look like this:
setuptools.setup(
name="common",
version="1.0.2",
package_dir={'package1': ''},
packages=['package1'],
)
by adding package_dir param, setup function tells all files under my root directory (package1) to be under package1 directory and by adding packages param it distributes package1 package and then if you go to:
/..../venv/lib/python3.8/site-packages/common-1.0.2-py3.8.egg-info/top_level.txt
you'll see the following content:
Here we go with my first ever stackoverflow quesion. I did search for an answer, but couldn't find a clear one. Here's the situation. I've got a structure like this:
myapp
package/
__init.py__
main.py
mod1.py
mod2.py
Now, in this scenario, from main.py I am importing mod1.py, which also needs to be imported by mod2.py. Everything works fine, my imports look like this:
main.py:
from mod1 import Class1
mod2.py:
from mod1 import Class1
However, I need to move my main.py to the main folder structure, like this:
myapp
main.py
package/
__init.py__
mod1.py
mod2.py
And now what happens is that of course I need to change the way I import mod1 inside main.py:
from package.mod1 import Class1
However, what also happens is that in order not to get an "ImportError: No module named 'mod1'", I have make the same type of change inside mod2.py:
from package.mod1 import Class1
Why is that? mod2 is in the same folder/pakcage as mod1, so why - upon modifying main.py - am I expected to modify my import inside mod2?
The reason this is happening is because how python looks for modules and packages when you run a python script as the __main__ script.
When you run python main.py, python will add the parent directory of main.py to the pythonpath, meaning packages and modules within the directory will be importable. When you moved main.py, you changed the directory that was added to the pythonpath.
Generally, you don't want to rely on this mechanism for importing your modules, because it doesn't allow you to move your script and your package and modules are only importable when running that main script. What you should do is make sure your package is installed into a directory that is already in the pythonpath. There are several ways of doing this, but the most common is to create a setup.py script and actually install your python package for the python installation on your computer.
I have a repository I inherited used by a lot of teams, lots of scripts call it, and it seems like its going to be a real headache to make any structural changes to it. I would like to make this repo installable somehow. It is structured like this:
my_repo/
scripts.py
If it was my repository, I would change the structure like so and make it installable, and run python setup.py install:
my_repo/
setup.py
my_repo/
__init__.py
scripts.py
If this is not feasible (and it sounds like it might not be), can I somehow do something like:
my_repo/
setup.py
__init__.py
scripts.py
And add something to setup.py to let it know that the repo is structured funny like this, so that I can install it?
You can do what you suggest.
my_repo/
setup.py
__init__.py
scripts.py
The only thing is you will need to import modules in your package via their name if they are in the base level. So for example if your structure looked like this:
my_repo/
setup.py
__init__.py
scripts.py
lib.py
pkg/
__init__.py
pkgmodule.py
Then your imports in scripts.py might look like
from lib import func1, func2
from pkg.pkgmodule import stuff1, stuff2
So in your base directory imports are essentially by module name not by package. This could screw up some of your other packages namespaces if you're not careful, like if there is another dependency with a package named lib. So it would be best if you have these scripts running in a virtualenv and if you test to ensure namespacing doesn't get messed up
There is a directive in setup.py file to set the name of a package to install and from where it should get it's modules for installation. That would let you use the desired directory structure. For instance with a given directory structure as :
my_repo/
setup.py
__init__.py
scripts.py
You could write a setup.py such as:
setup(
# -- Package structure ----
packages=['my_repo'],
package_dir={'my_repo': '.'})
Thus anyone installing the contents of my_repo with the command "./setup.py install" or "pip install ." would end up with an installed copy of my_repo 's modules.
As a side note; relative imports work differently in python 2 and python 3. In the latter, any relative imports need to explicitly specify the will to do so. This method of installing my_repo will work in python 3 when calling in an absolute import fashion:
from my_repo import scripts