I have created a Python package which builds on the structure indicated in Kenneth Reitz' "Repository Structure and Python" (1). The main package path is:
/projects-folder (not site-packages)
/package
/package
__init__.py
Datasets.py
Draw.py
Gmaps.py
ShapeSVG.py
project.py
__init__.py
setup.py
With the current structure, I must use the following module import syntax:
import package.package.Datasets
I would prefer to type the following:
import package.Datasets
I am capable of typing the same word twice, of course, but it feels wrong in a deeper sense, i.e., I am structuring my package incorrectly or misunderstanding how Python interprets that structure.
The outer __init__.py is required for Python to detect this package at all, per the docs (2). But that sets up /package/ as the top level of the package and /package/package/ as a sub-package, forcing me into the unwieldy import syntax above.
To avoid this, it seems that my options are to:
Create a package in which the outer folder contains the top level of package modules.
Add the inner folder to my PYTHONPATH environment variable.
Yet both of these seem like suboptimal workarounds for something that shouldn't be an issue in the first place. What should I do?
You've misunderstood. You have two package packages for some reason, but the source you cite never said to do that. The outer folder, with setup.py, is not supposed to be a package.
It sounds like you're running Python in projects-folder and trying to import your package from there. That's not what you should be doing. You have several options to get your package into the import system. (I'll refer to the folder with setup.py in it as setupfolder, to distinguish it from the inner folder):
Build your package with setup.py, for example, python setup.py bdist-wheel --universal, and install the built package with pip.
Skip the build step and just run pip install path/to/setupfolder. Building the package produces an installer useful if you want to distribute your package, but maybe you don't want to do that.
"Install" the package's source tree in development mode with pip install -e path/to/setupfolder, so the Python import system will locate the package's source tree when performing imports. This is handy because you don't have to rebuild and reinstall if you edit the source repository, although you'll still want to restart any running Python processes that are using the package.
Run Python from directly inside the setupfolder.
Any of these options will cause your package to be importable directly as package instead of package.package, as it should be.
While I do not entirely agree with your package structure, you can make use of __all__ and possibly the one legitimate use for star imports I've seen so far. __init__.py can serve more purposes than just identifying your folder as a package or sub-package.
Using a Star Import
In package/package/__init__.py, add a variable __all__ that declares all the public elements you want to export:
__all__ = ['Datasets', 'Draw', 'Gmaps', 'ShapeSVG', 'project']
In package/__init__.py do from package.package import *. Now all the attributes that were available as package.package.x will also be available as package.x.
If you want to directly copy package.package.__all__ to package.__all__ (which is optional, but will allow you to do from package import * properly), you can do something like
from package.package import *
from package.package import __all__ as _all
__all__ = _all
del _all
Not Using a Star Import
You can accomplish the same thing without using package.package.__all__ at all. Just add __all__ directly to package/__init__.py and use from package.package import x-style imports:
from package.package import (
Datasets, Draw, Gmaps, ShapeSVG, project
)
# As before, package.__all__ is optional
__all__ = ['Datasets', 'Gmaps', 'ShapeSVG', 'project']
I would still recommend having a package.package.__all__ variable, but it is optional for this particular purpose.
Pros and Cons
Both approaches are pretty legitimate and I have seen both used in major projects. The first approach reduces redundancy. You only define the public exports in one place: package.package.__all__. The star imports and package.__all__ reference that definition directly, leading to one place that you really have to maintain. On the other hand, there are times when you want to separate the "full" package.package.x API from what you expose via package.x to the casual user. In that case, go with the second option. The only downside here is that you have to be more careful to keep package.__all__ and the corresponding imports synchronized properly.
Note
A number of projects I've seen (numpy especially comes to mind), export the attributes of the individual modules to the top level using this technique. For example, if you had a function package.package.Datasets.get_data, it would be listed in package.package.Datasets.__all__, which would be imported into pacakge.package.__init__, appended to package.package.__all__, and then be referenced by the top-level package and package.__all__.
Related
I have a solution with different folders and I wanted to create a common package for them to share reusable code. There are lot of examples for python packages are there. None of them addresses separate folder scenario in a simple way.
My folder structure is :
Client/
__init__.py
--logic.py
Common/
__init__.py
-- Constants.py
Server/
__init__.py
-- Test.py
My Test.py looks like this :
from Common import Constant
print(Constant.TEST_VALUE) #-- From common package
But the code not even passes the import statements.
Exception has occurred: ModuleNotFoundError No module named 'Common'
If I try with relative path : from ..Common import Constant
Error:
Exception has occurred: ImportError attempted relative import with no known parent package
My purpose is to reuse the constant.py file for both Cars and Server solution. There are examples for sys.path which I am not sure if its a good practice. Also the code in Common will not be having complex codes to proceed with installing them.
If these packages are meant to be part of the same conceptual project, then simply make a containing package. Now the relative import will work as long as you start from outside the package hierarchy, and ensure the containing folder is on sys.path. The easy ways to do that are to install the package (preferably to a virtual environment), or start python from that folder (in which case Cars, Common and Server appear as separate packages) or just outside (in which case the containing folder is a package and Cars, Common and Server are sub-packages - which is probably what you want; it's more consistent with how you will most likely install the code later).
If these packages are separate projects that happen to depend upon one another, then Python has no reason by default to look in any particular place for the dependency packages - aside from the places it ordinarily looks for anything. So, again, you can start python from the containing folder if there actually is such a containing folder; but there isn't a good project-organization reason for there to be a containing folder - so set up an actual project for each package, and install them all to the same virtual environment. Now you can do cross-package imports with absolute syntax (but not relative syntax, because that only works within a package hierarchy).
You can use this code found here:
# Test.py
import sys
# insert at 1, 0 is the script path (or '' in REPL)
sys.path.insert(1, '/path/to/Cars/')
sys.path.insert(1, '/path/to/Common/')
import Audi
import Constants
I know that python mechanism doesn't allow doing relative imports without a known parent package, but I want to know which the reason is.
If relative imports worked without known parent package it would make developers life much easier (I think, correct me if I am wrong)
Example:
From a python script a have to import a global definitions file. Right now I have to do it like this:
DEFS_PATH = '../../../'
sys.path.append(DEFS_PATH)
import DEFINITIONS as defs
If I could import this file just like this without having to specify the -m flag when executing the script or creating a __init__.py file that collects all packages. It would make everything much more easier.
from .... import DEFINITIONS as defs
Of course doing this raises the famous import error:
Obviously this is a toy example, but imagine having to repeat this in hundreds of python scripts...
Is there any workaround for importing relative packages without a known parent package that doesn't involve tha hacky ugly way (sys.path.append(...) or python -m myscript )?
I solved this problem but in a different way. I have a folder where I have lots of global functions that I used in different packages, I could create a Python Package of this folder, however I would have to rebuild it each time I changed something.
The solution that fitted me was to add user-packages.pth file in site-packages directory of my current environment, but it could also be added to global site-packages folder. Inside this user-packages.pth I added the absolute path to my directory where all the global utils are. And now I just have to do from any python script
from utils import data_processing as dp
from utils.database import database_connection as dc
Now I don't need to add in each file sys.path.append("path/to/myutils/")
Note:
The .pth file could have any file name (customName.pth) and paths inside the file should be separated by carriage return ("\n"). Also, paths should be absoulte.
For example:
C:\path\to\utils1
C:\path\to\other\utils2
I am creating a tool I plan to put on Github. It is very general however I want to also include some examples of its usage. As such I have set up the following folder structure.
This tool is for python 3.
repository/
commonTool.py
commonTool2.py
specificUsage/
runTheSpecificUsage.py
helpRunTheSpecificUsage.py
Now both of the scripts in the specificUsage/ folder will import the methods in commonTool.py and commonTool2.py.
My issue is ideally the user would be able to go
python repository/specificUsage/runTheSpecificUsage.py
However I cant get this to work. It never is able to import the functions that are in the folder above it. I have tried a variety of various posts on how to import a file from a super folder with no luck.
How should I be setting up these folders? Should I have one init.py or two? Where?
You can create a setup.py at the top level along with commonTool.py and commonTool2.py. Inside your setup.py, put the following line:
import setuptools
setuptools.setup(packages=setuptools.find_packages())
Then put an __init__.py in the same level, as well as in specificUsage/. You should be able to import from specificUsage like this:
import commonTool
After setting up your files, from the top level run:
pip install -e .
You may also want to consider that Python has strong naming conventions for modules and packages and frowns upon camelcase in general. You can read more here if you like.
First, I need to describe the environment I'm writing for. I'm writing Python code that will be loaded and executed by a Python runtime running within a CAD application. The CAD application uses Python as its scripting engine. As a result, I don't have access to the Python runtime and as a good citizen to all other scripts shouldn't modify any system settings. My script is just one of many that are loaded and running.
This all works fine except when I want to use non-standard libraries. In that case, I need to install a local copy of the library for my script to access. The problem I'm having is that most libraries expect to be installed and added to the sys path which is something I shouldn't do because it could create conflicts with what other scripts are doing. What I'm attempting to do instead is to set up a local copy of the library(s) and then edit their source so their imports are relative and they don't depend on the sys path. That way my program will have its own local copy of the libraries and not depend on anything else and won't disturb any other scripts.
I'm using the -t option of PIP to install Requests and PyOpenSSL into a "Packages" subfolder in my script folder. Here's an abbreviated list of what I have.
RequestsTest/
RequestsTest.py
Packages/
OpenSSL/
cryptography/
x509/
__init__.py
base.py
hazmat/
__init__.py
backends/
__init__.py
interfaces.py
openssl/
__init__.py
backend.py
x509.py
OpenSSL/
__init__.py
SSL.py
Requests/
chardet/
__init__.py
requests/
__init__.py
urllib3/
__init__.py
request.py
contrib/
__init__.py
pyopenssl.py
util/
__init__.py
request.py
ssl_.py
Although it's tedious to track down the various import statements and make then relative, it does seem to work. However, I'm having problems with one particular set of imports.
In Packages/Requests/urllib3/contrib/pyopenssl.py it contains the following imports, which I've modified:
from ....OpenSSL.OpenSSL import SSL
from ....OpenSSL.cryptography import x509
They were originally:
from OpenSSL import OpenSSL.SSL
from cryptography import x509
I get the error "ImportError: No module named 'OpenSSL'" for the first line and "ImportError: No module named 'cryptography'" for the second line. I'm fairly certain the path is correct because if I change the number of dots I get the no module named error but it lists the full path of the what it's trying to load and not just the name of the module.
I would appreciate some help with this specific issue but can also use some overall advice of how to set up and use private copies of libraries. Remember that my program is just one of many that the system is loading changing the system or setting up a virtual environment is not an option.
Check out the localimport module, which seems to be a solution for your particular use case. From the README:
Given your Python script, application or plugin comes with a directory that contains modules for import, you can use localimport to keep the global importer state clean.
app.py
res/modules/
some_package/
__init__.py
# app.py
with localimport('res/modules') as _importer:
import some_package
assert 'some_package' not in sys.modules
The tagline is "Isolated import of Python Modules for embedded applications." so it seems pretty relevant.
When using that module, the following may help keep things neat:
Put your actual script logic into its own file.
Have a wrapper script (which will be the one loaded by the CAD software) which does localimport as mentioned in the README then does a relative import of your module. If your module gets big enough maybe put it into its own package and consume it in the same way as everything else (just do from RequestsTest import * in the body of the with localimport(): ....
Try to have a clear boundary between the source code you write and the final organized set of files required to use that source code in the context of the CAD Python runtime. It is OK to have a build/packaging step that creates the localimport script, downloads the required packages, etc. It's better even because then it is automated and not something that was done manually that someone in the future may have to recreate.
I have the following structure on a Python program:
my_program/
main.py
packages/
__init.py__
package_to_share/
__init__.py
main_to_share.py
module_to_share.py
package_A/
__init__.py
main_A.py
some_module_A.py
package_B/
__init__.py
main_B.py
some_module_B.py
The package package_to_share provides functionality that every package in the packages folder uses and that main.py at the root folder uses.
I also want to be able to cd into each package and be able to run main_X.py.
So far I figured out how to access functionality from main.py:
import packages.package_A.some_module_A
import packages.package_to_share.module_to_share
but I am having problems accessing the functionality in package_to_share from regular packages (e.g. package_A)
For example, when in main_A.py or some_module_A.py, typing import packages.package_to_share.module_to_share fails.
This leads me to following questions questions:
Given the specifics of my problem, with packages to be shared (accessed) by files at the root folder and by other packages, is there a better way to organize my folders ? Does this organization of modules and files in general conform to good standards in Python?
The following looks incredibly hacky to me, but it's the best thing I came up with to make sure my regular modules see the "shared" modules:
p_this_file = os.path.dirname(os.path.realpath(__file__))
new_sys_path.append(os.path.join(p_cwd, '..')
new_sys_path.extend(sys.path)
sys.path = new_sys_path
It also does not prevent my regular packages from importing each other.
Rather than manipulating the path for imports (which I don't recommend), you could use relative imports within module_A:
from .. import shared
That is what I sometimes do, though I generally have my packages installed so I can reference them fully where I need to use them:
from my_module import shared