We Keep Coding

Add a virtual namespace for a Python package

I have a python package with the following directory structure:
my_package/
my_subpackage/
__init__.py
my_module.py
__init__.py
setup.py
When I generate the Python wheel and pip install it, I am not going to have the my_package namespace, so my_subpackage is going to be part of the global namespace of my virtualenv or whatever.
I know that the solution here is to create another directory called my_package and put everything inside it:
my_package/
my_package/
my_subpackage/
__init__.py
my_module.py
__init__.py
setup.py
But let's say that I cannot change the directory structure for some reason. Is there a way to add a virtual my_package namespace in setup.py with the first layout?

You can specify the package_dir option in your setup.py.
Something like this should work:
setup(
...
package_dir={'': '..'},
packages=['my_package']
...
)
I never used relative paths with package_dir though, so your confirmation would be appreciated.

Python / Import and design of program

I’m looking for solution for designing my program.
My program consists of 3 blocks:
Classes
Functions
Other utilities
I want to structure my program this way:
program_folder/
main.py
classes_folder/
class_1.py
class_2.py
functions_folder/
set_of_func_1.py
set_of_func_1.py
utilities_folder/
set_of_utilities_1.py
set_of_utilities_1.py
I want to:
any scripts in «classes_folder» were able to import any of scripts in
«functions_folder».
any scripts in «functions_folder» were able
to import any of scripts in «utilities_folder».
all scripts were
normally used by main.py.
all scripts in «classes_folder»,
«functions_folder» and «utilities_folder» could be tested when worked
as «main» (if __name__ == “__main__”: some tests)
«program_folder»
could be in any place in my computer (there shouldn’t be dependency
on exact path to «program_folder»).
From all the above I thought I have to:
Change import search path for all scripts in «classes_folder»,
«functions_folder» and «utilities_folder».
Set current working
directory to «program_folder» for all scripts?
Is there a way I can do it?
Does my idea look good or have I put there some unexpected problems?

You can create a skeleton project like the following:
/path/to/project/
setup.py
my_project/
__init__.py
a/
   __init__.py
  b/
  __init__.py
==> ./my_project/__init__.py <==
print('my_project/__init__.py')
==> ./my_project/a/__init__.py <==
import my_project
print('my_project/a/__init__.py')
==> ./my_project/b/__init__.py <==
import my_project.a
print('my_project/b/__init__.py')
==> ./setup.py <==
from distutils.core import setup
setup(name='my_project',
version='1.0',
description='my_project',
author='author',
packages=['my_project'])
Then you can install the project locally using pip install -e /path/to/project/ (the project folder is not copied, just gets registered; there's a dependency on the exact path, but this dependency is not hard-coded in project files themselves).
As the result, import my_project, import my_project.a etc. do that they mean:
$ python my_project/b/__init__.py
my_project/__init__.py
my_project/a/__init__.py
my_project/b/__init__.py

A common Python project structure could look like this:
project_name/
setup.py
requirements.txt
project_name/
__main__.py
classes/
__init__.py
class1.py
class2.py
functions/
__init__.py
functions.py
utils/
__init__.py
utils.py
Then, you could modify your imports from absolute to relative and run your package using something like:
$ /path/to/project_name> python -m project_name
Note that setup.py is only required if you want to install your package under some of your interpreters.
Note: see comments below also

How to exclude a single file from package with setuptools and setup.py

I am working on blowdrycss. The repository is here.
I want the settings file for blowdrycss_settings.py to be excluded from the final package on pypi. The intention is to dynamically build a custom settings file that will be placed in the users virtualenv / project folder.
In setup.py, I have the following:
packages=find_packages(exclude=['blowdrycss_settings.py', ]),
I also tried exclude_package_data:
exclude_package_data={
'': ['blowdrycss_settings.py'],
'': ['blowdrycss/blowdrycss_settings.py'],
'blowdrycss': ['blowdrycss_settings.py'],
},
I then run python setup.py sdist bdist.
However, when I look in the build folder I still see blowdrycss_settings.py:
- build
- lib
- blowdrycss_settings.py
It seems like it should be simple to just exclude a file.
How do I exclude blowdrycss_settings.py from the distributed package?

Imagine you have a project
root
├── setup.py
└── spam
├── __init__.py
├── bacon.py
└── eggs.py
and you want to exclude spam/eggs.py from packaging:
import fnmatch
from setuptools import find_packages, setup
from setuptools.command.build_py import build_py as build_py_orig
excluded = ['spam/eggs.py']
class build_py(build_py_orig):
def find_package_modules(self, package, package_dir):
modules = super().find_package_modules(package, package_dir)
return [
(pkg, mod, file)
for (pkg, mod, file) in modules
if not any(fnmatch.fnmatchcase(file, pat=pattern) for pattern in excluded)
]
setup(
packages=find_packages(),
cmdclass={'build_py': build_py}
)
Globs and multiple entries in excluded list will work too because it is consumed by fnmatch, so you can e.g. declare
excluded = [
'modules_in_directory/*.py',
'modules_in_subtree/**/*.py',
'path/to/other/module.py'
]
etc.
This recipe is based on my other answer to the question setup.py exclude some python files from bdist
. The difference is that this recipe excludes modules based on file globs, while the other one excludes modules based on qualnames, for example
excluded = ['spam.*', '*.settings']
will exclude all submodules of spam package and all modules named settings in every package and subpackage etc.

Here is my solution.
Underneath of blowdrycss, I created a new module called settings so the directory structure now looks like this:
blowdrycss
blowdrycss
settings
blowdrycss_settings.py
Based on this reference, inside of setup.py I have the following:
packages=find_packages(exclude=['*.settings', ]),
To build the distribution:
Delete the build, dist, and .egg-info folders.
Run python setup.py sdist bdist
In retrospect, it is good that I was unable to do what I was originally attempting. The new structure feels cleaner and is more modular.

The easiest way to remove a single, or at least a few specific files, from a package with setuptools is just to use the MANIFEST.in. For example, in a package you can exclude all files name foo.py by simply specifying global-exclude foo.py. There's no need for setuptools hacking or changing the structure of your package if you just use the MANIFEST.in method.
See the dedicated PyPA article on using MANIFEST.in for more commands you can use.

Directory Structure for Importing Python Package using __init__.py

I've got a python project (projectA), which I've included as a git submodule in a separate project (projectB) as a subfolder. The structure is laid out as
projectB/
projectB_file.py
projectA/ (repository)
projectA/ (source code)
module1/
file1.py (contains Class1)
file2.py (contains Class2)
tests/
test_file1.py
I'm trying to figure out how to layout __init__.py files so in projectB_file.py I can import Class1 and Class2 as
from projectA.module1 import Class1
from projectA import Class2

I think having the top level projectA part of the import will be a mistake. That will require you to write your imports with projectA duplicated (e.g. import projectA.projectA.module1.file1).
Instead, you should add the top projectA folder to the module search path in some way (either as part of an install script for projectB, or with a setting in your IDE). That way, projectA as a top-level name will refer to the inner folder, which you actually intend to be the projectA package.

You'll need an __init__.py in each subdirectory you want to treat as a package. Which in your case means one in:
projectA
projectA/projectA
projectA/projectA/module1
projectA/projectA/tests
It would definitely be better you could lose that first projectA folder.

This is an annoying issue. The main approach I use is to edit the PYTHONPATH. This is essentially how I do it.
ProjectB/ # Main Repository
projectb/ # Package/library
__init__.py
projectB_file.py
docs/
tests/
setup.py # Setup file to distribute the library
freeze.py # CX_Freeze file to make executables
ProjectA/ # Subrepository
projecta/ # Package/library
__init__.py
projectA_file.py
file2.py
submodule/
__init__.py
file3.py
pa_module1/ # Additional package/library in 1 project
__init__.py
file1.py
docs/
tests/
setup.py
With this setup I have to edit the python path before running ProjectB's code. I use this structure, because it is easier for deployment and distribution. The subrepository helps track the version of ProjectA for each release of ProjectB.
cd ProjectB
export PYTHONPATH=${PWD}:${PWD}/ProjectA
python projectb/projectB_file.py
projectB_file.py
from projecta import projectA_file
This is for my development environment. For a release environment someone would install with the below code.
# Need a correct setup.py to handle 2 packages for pa_module1 and projecta
pip install ProjectA (install projecta to site-packages)
cd ..
pip install ProjectB (install projectb to site-packages)
This means projectb and projecta are in site packages. From there any file can simply
from projectb import projectB_file
from projecta import projectA_file, file2
from pa_module1 import file1
# Personally I don't like imports that use project.sub.sub.sub ...
# I have seen it cause import confusion and problems, but it is useful in some situations.
from projecta.submodule import file3
#import projecta.projectA_file
#import projecta # if __init__.py has import code
file1.Class1()
file2.Class2()
file3.Class3()
Additionally with pip you can install a library as a developer environment which links to the real project directory instead of copying files to site-packages.
pip install -e ProjectA

How to structure a python project with three applications that use common module

My project contains three Python applications. Application 1 is a web app. Applications 2 and 3 contain scripts downloading some data.
All three apps need to use a module Common containing a "model" (classes that are saved to database) and common settings.
I have no clue how to structure this project. I could create three directories, one for each application, and copy Common three times into their directories (doesn't seem right).
Another idea that comes to mind is; create a main directory and put there all files from Common, including __init__.py. Then, crete three subdirectories (submodules), one for each application.
Another way would be installing Common using pip, but that means I would have to reinstall every time I change something in that module.
In previous projects I used .NET - the equivalent in that world would be a Solution with four projects, one of them being Common.
Any suggestions?

I have a similar project that is set up like this
project_root/
App1/
__init__.py
FlaskControlPanel/
app.py
static/
templates/
models/
__init__.py
mymodels.py
Then, I run everything from project_root. I have a small script (either batch or shell depending on my environment) that sets PYTHONPATH=. so that imports work correctly. This is done because I usually develop using PyCharm, where the imports "just work", but when I deploy the final product the path doesn't match what it did in my IDE.
Once the PYTHONPATH is set to include everything from your project root, you can do standard imports.
For example, from my FlaskControlPanel app.py, I have this line:
from models.mymodels import Model1, Model2, Model3
From the App1 __init__.py I have the exact same import statement:
from models.mymodels import Model1, Model2, Model3
I can start the Flask application by running this from my command line (in Windows) while I am in the project_root directory:
setlocal
SET PYTHONPATH=.
python FlaskControlPanel\app.py
The setlocal is used to ensure the PYTHONPATH is only modified for this session.

I like this approach
projects/
__init__.py
project1/
__init__.py
project2/
__init__.py
lib1/
__init__.py
libfile.py
lib2/
__init__.py
So, I need to cd into the projects folder.
To start a projects use
python -m project_name
This allows me to easily import from any external lib like
from lib1.libfile import [imoprt what you want]
or
from lib1 import libfile

Make standard Python modules from your apps. I recommend structure like this:
apps/
common/
setup.py
common/
__init__.py
models.py
app1/
setup.py
app1/
__init__.py
models.py
project/
requirements.txt
Basic setup.py for app common:
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(
name='common',
version='1.0.0',
packages=find_packages(),
zip_safe=False,
)
Make similar setup.py for other apps.
Set editable "-e" option for your apps in requirements.txt:
-e apps/common
-e apps/app1
Install requirements with pip:
$ pip install -r requirements.txt
Editable option means that source files will be linked into Python enviroment. Any change in source files of your apps will have immediate effect without reinstalling them.
Now you can import models from your common app (or any other app) anywhere (in other apps, project files, ...).

I would create a structure like this:
project_root/
app1/
__init__.py
script.py
common/
__init__.py
models.py (all "common" models)
app1/script.py
import os, sys
# add parent directory to pythonpath
basepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
if basepath not in sys.path:
sys.path.append(basepath)
from common.models VeryCommonModel
print VeryCommonModel
If you don't want to set the python path at runtime, set the python path before running the script:
$ export PYTHONPATH=$PYTHONPATH:/path/to/project_root
And then you can do:
python app1/script.py