How to write a Python module/package? - python

I've been making Python scripts for simple tasks at work and never really bothered packaging them for others to use. Now I have been assigned to make a Python wrapper for a REST API. I have absolutely no idea on how to start and I need help.
What I have:
(Just want to be specific as possible) I have the virtualenv ready, it's also up in github, the .gitignore file for python is there as well, plus, the requests library for interacting with the REST API. That's it.
Here's the current directory tree
.
├── bin
│   └── /the usual stuff/
├── include
│   └── /the usual stuff/
├── lib
│   └── python2.7
│   └── /the usual stuff/
├── local
│   └── /the usual stuff/
└── README.md
27 directories, 280 files
I don't even know where to put the .py files, if I ever make one.
What I wanted to do:
Make a python module install-able with "pip install ..."
If possible, I want a general step by step process on writing Python modules.


A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py
create hello.py then write the following function as its content:
def helloworld():
print "hello"
Then you can import hello:
>>> import hello
>>> hello.helloworld()
'hello'
>>>
To group many .py files put them in a folder. Any folder with an __init__.py is considered a module by python and you can call them a package
|-HelloModule
|_ __init__.py
|_ hellomodule.py
You can go about with the import statement on your module the usual way.
For more information, see 6.4. Packages.


Python 3 - UPDATED 18th November 2015
Found the accepted answer useful, yet wished to expand on several points for the benefit of others based on my own experiences.
Module: A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py appended.
Module Example: Assume we have a single python script in the current directory, here I am calling it mymodule.py
The file mymodule.py contains the following code:
def myfunc():
print("Hello!")
If we run the python3 interpreter from the current directory, we can import and run the function myfunc in the following different ways (you would typically just choose one of the following):
>>> import mymodule
>>> mymodule.myfunc()
Hello!
>>> from mymodule import myfunc
>>> myfunc()
Hello!
>>> from mymodule import *
>>> myfunc()
Hello!
Ok, so that was easy enough.
Now assume you have the need to put this module into its own dedicated folder to provide a module namespace, instead of just running it ad-hoc from the current working directory. This is where it is worth explaining the concept of a package.
Package: Packages are a way of structuring Python’s module namespace by using “dotted module names”. For example, the module name A.B designates a submodule named B in a package named A. Just like the use of modules saves the authors of different modules from having to worry about each other’s global variable names, the use of dotted module names saves the authors of multi-module packages like NumPy or the Python Imaging Library from having to worry about each other’s module names.
Package Example: Let's now assume we have the following folder and files. Here, mymodule.py is identical to before, and __init__.py is an empty file:
.
└── mypackage
├── __init__.py
└── mymodule.py
The __init__.py files are required to make Python treat the directories as containing packages. For further information, please see the Modules documentation link provided later on.
Our current working directory is one level above the ordinary folder called mypackage
$ ls
mypackage
If we run the python3 interpreter now, we can import and run the module mymodule.py containing the required function myfunc in the following different ways (you would typically just choose one of the following):
>>> import mypackage
>>> from mypackage import mymodule
>>> mymodule.myfunc()
Hello!
>>> import mypackage.mymodule
>>> mypackage.mymodule.myfunc()
Hello!
>>> from mypackage import mymodule
>>> mymodule.myfunc()
Hello!
>>> from mypackage.mymodule import myfunc
>>> myfunc()
Hello!
>>> from mypackage.mymodule import *
>>> myfunc()
Hello!
Assuming Python 3, there is excellent documentation at: Modules
In terms of naming conventions for packages and modules, the general guidelines are given in PEP-0008 - please see Package and Module Names
Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.


Since nobody did cover this question of the OP yet:
What I wanted to do:
Make a python module install-able with "pip install ..."
Here is an absolute minimal example, showing the basic steps of preparing and uploading your package to PyPI using setuptools and twine.
This is by no means a substitute for reading at least the tutorial, there is much more to it than covered in this very basic example.
Creating the package itself is already covered by other answers here, so let us assume we have that step covered and our project structure like this:
.
└── hellostackoverflow/
├── __init__.py
└── hellostackoverflow.py
In order to use setuptools for packaging, we need to add a file setup.py, this goes into the root folder of our project:
.
├── setup.py
└── hellostackoverflow/
├── __init__.py
└── hellostackoverflow.py
At the minimum, we specify the metadata for our package, our setup.py would look like this:
from setuptools import setup
setup(
name='hellostackoverflow',
version='0.0.1',
description='a pip-installable package example',
license='MIT',
packages=['hellostackoverflow'],
author='Benjamin Gerfelder',
author_email='benjamin.gerfelder#gmail.com',
keywords=['example'],
url='https://github.com/bgse/hellostackoverflow'
)
Since we have set license='MIT', we include a copy in our project as LICENCE.txt, alongside a readme file in reStructuredText as README.rst:
.
├── LICENCE.txt
├── README.rst
├── setup.py
└── hellostackoverflow/
├── __init__.py
└── hellostackoverflow.py
At this point, we are ready to go to start packaging using setuptools, if we do not have it already installed, we can install it with pip:
pip install setuptools
In order to do that and create a source distribution, at our project root folder we call our setup.py from the command line, specifying we want sdist:
python setup.py sdist
This will create our distribution package and egg-info, and result in a folder structure like this, with our package in dist:
.
├── dist/
├── hellostackoverflow.egg-info/
├── LICENCE.txt
├── README.rst
├── setup.py
└── hellostackoverflow/
├── __init__.py
└── hellostackoverflow.py
At this point, we have a package we can install using pip, so from our project root (assuming you have all the naming like in this example):
pip install ./dist/hellostackoverflow-0.0.1.tar.gz
If all goes well, we can now open a Python interpreter, I would say somewhere outside our project directory to avoid any confusion, and try to use our shiny new package:
Python 3.5.2 (default, Sep 14 2017, 22:51:06)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from hellostackoverflow import hellostackoverflow
>>> hellostackoverflow.greeting()
'Hello Stack Overflow!'
Now that we have confirmed the package installs and works, we can upload it to PyPI.
Since we do not want to pollute the live repository with our experiments, we create an account for the testing repository, and install twine for the upload process:
pip install twine
Now we're almost there, with our account created we simply tell twine to upload our package, it will ask for our credentials and upload our package to the specified repository:
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
We can now log into our account on the PyPI test repository and marvel at our freshly uploaded package for a while, and then grab it using pip:
pip install --index-url https://test.pypi.org/simple/ hellostackoverflow
As we can see, the basic process is not very complicated. As I said earlier, there is a lot more to it than covered here, so go ahead and read the tutorial for more in-depth explanation.


Once you have defined your chosen commands, you can simply drag and drop the saved file into the Lib folder in your python program files.
>>> import mymodule
>>> mymodule.myfunc()


Make a file named "hello.py"
If you are using Python 2.x
def func():
print "Hello"
If you are using Python 3.x
def func():
print("Hello")
Run the file. Then, you can try the following:
>>> import hello
>>> hello.func()
Hello
If you want a little bit hard, you can use the following:
If you are using Python 2.x
def say(text):
print text
If you are using Python 3.x
def say(text):
print(text)
See the one on the parenthesis beside the define? That is important. It is the one that you can use within the define.
Text - You can use it when you want the program to say what you want. According to its name, it is text. I hope you know what text means. It means "words" or "sentences".
Run the file. Then, you can try the following if you are using Python 3.x:
>>> import hello
>>> hello.say("hi")
hi
>>> from hello import say
>>> say("test")
test
For Python 2.x - I guess same thing with Python 3? No idea. Correct me if I made a mistake on Python 2.x (I know Python 2 but I am used with Python 3)


A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py
Create a file called hello.py with the following function as its content:
def helloworld():
print "hello"
Then you can
import hello
hello.helloworld()
To group many .py files, put them in a folder. Any folder with an init.py is considered a module by python and you can call them a package.
|-HelloModule
|_ init.py
|_ hellomodule.py

Related

Python, import subfolder from another subfolder [duplicate]

The following is my directory structure.
ankur
├── ankur1
│ ├── __init__.py
│ └── util.py
├── ankur2
│ └── main.py
└── __init__.py
In main.py, I am importing the following.
import ankur.ankur1.util
When I execute the code in Windows, it works perfectly fine. But in Linux, I get the following error.
ImportError: No module named ankur.ankur1.util
I also read the official python doc on Modules and Packages.

Your package structure is OK. Your import statement is OK. The only thing missing is for the package to be visible in sys.path, a list of locations where import statements can be resolved.
Usually we do this by "installing" the package locally with pip, which copies your code into site-packages†. This directory is one of the entries in sys.path, so when your code is installed in site-packages, the import statements can now be resolved as usual.
However, to install your code you'll need an installer (setup.py script) or a build system (pyproject.toml file) defined for the package. Your project doesn't appear to have any installer or build system, so you'll need to create one (see the Python Packaging User Guide for details about that) and then install the package with pip. If you don't want to learn Python packaging just yet, you'll need to find another way around.
It is possible to modify sys.path directly in main.py, which is subsequently enabling the statement import ankur.ankur1.util to be resolved. This is hacky and I recommend against that. It would add the restriction that executing main.py is the only entry point to the rest of the package, and so any other code wanting to import ankur will first need to know the path to main.py on the filesystem. That's a messy approach and should be avoided.
Another way around is to use the environment - there is an environment variable PYTHONPATH which can be used to augment the default search path for module files. In your shell:
export PYTHONPATH=/path/to/parent # linux/macOS
SET PYTHONPATH=C:/path/to/parent # Windows
Where parent is the directory containing ankur subdirectory.
† The exact location of site-packages depends on your OS/platform, but you can check with import sysconfig; sysconfig.get_paths()["purelib"]

How do I permanently add paths to PYTHONPATH in a script? [duplicate]

I've tried reading through questions about sibling imports and even the
package documentation, but I've yet to find an answer.
With the following structure:
├── LICENSE.md
├── README.md
├── api
│   ├── __init__.py
│   ├── api.py
│   └── api_key.py
├── examples
│   ├── __init__.py
│   ├── example_one.py
│   └── example_two.py
└── tests
│   ├── __init__.py
│   └── test_one.py
How can the scripts in the examples and tests directories import from the
api module and be run from the commandline?
Also, I'd like to avoid the ugly sys.path.insert hack for every file. Surely
this can be done in Python, right?

Tired of sys.path hacks?
There are plenty of sys.path.append -hacks available, but I found an alternative way of solving the problem in hand.
Summary
Wrap the code into one folder (e.g. packaged_stuff)
Create setup.py script where you use setuptools.setup(). (see minimal setup.py below)
Pip install the package in editable state with pip install -e <myproject_folder>
Import using from packaged_stuff.modulename import function_name
Setup
The starting point is the file structure you have provided, wrapped in a folder called myproject.
.
└── myproject
├── api
│ ├── api_key.py
│ ├── api.py
│ └── __init__.py
├── examples
│ ├── example_one.py
│ ├── example_two.py
│ └── __init__.py
├── LICENCE.md
├── README.md
└── tests
├── __init__.py
└── test_one.py
I will call the . the root folder, and in my example case it is located at C:\tmp\test_imports\.
api.py
As a test case, let's use the following ./api/api.py
def function_from_api():
return 'I am the return value from api.api!'
test_one.py
from api.api import function_from_api
def test_function():
print(function_from_api())
if __name__ == '__main__':
test_function()
Try to run test_one:
PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
File ".\myproject\tests\test_one.py", line 1, in <module>
from api.api import function_from_api
ModuleNotFoundError: No module named 'api'
Also trying relative imports wont work:
Using from ..api.api import function_from_api would result into
PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
File ".\tests\test_one.py", line 1, in <module>
from ..api.api import function_from_api
ValueError: attempted relative import beyond top-level package
Steps
Make a setup.py file to the root level directory
The contents for the setup.py would be*
from setuptools import setup, find_packages
setup(name='myproject', version='1.0', packages=find_packages())
Use a virtual environment
If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)
Create virtual env
python -m venv venv
Activate virtual env
source ./venv/bin/activate (Linux, macOS) or ./venv/Scripts/activate (Win)
To learn more about this, just Google out "python virtual env tutorial" or similar. You probably never need any other commands than creating, activating and deactivating.
Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis
PS C:\tmp\test_imports> python -m venv venv
PS C:\tmp\test_imports> .\venv\Scripts\activate
(venv) PS C:\tmp\test_imports>
and your folder tree should look like this**
.
├── myproject
│ ├── api
│ │ ├── api_key.py
│ │ ├── api.py
│ │ └── __init__.py
│ ├── examples
│ │ ├── example_one.py
│ │ ├── example_two.py
│ │ └── __init__.py
│ ├── LICENCE.md
│ ├── README.md
│ └── tests
│ ├── __init__.py
│ └── test_one.py
├── setup.py
└── venv
├── Include
├── Lib
├── pyvenv.cfg
└── Scripts [87 entries exceeds filelimit, not opening dir]
pip install your project in editable state
Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.
In the root directory, run
pip install -e . (note the dot, it stands for "current directory")
You can also see that it is installed by using pip freeze
(venv) PS C:\tmp\test_imports> pip install -e .
Obtaining file:///C:/tmp/test_imports
Installing collected packages: myproject
Running setup.py develop for myproject
Successfully installed myproject
(venv) PS C:\tmp\test_imports> pip freeze
myproject==1.0
Add myproject. into your imports
Note that you will have to add myproject. only into imports that would not work otherwise. Imports that worked without the setup.py & pip install will work still work fine. See an example below.
Test the solution
Now, let's test the solution using api.py defined above, and test_one.py defined below.
test_one.py
from myproject.api.api import function_from_api
def test_function():
print(function_from_api())
if __name__ == '__main__':
test_function()
running the test
(venv) PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
I am the return value from api.api!
* See the setuptools docs for more verbose setup.py examples.
** In reality, you could put your virtual environment anywhere on your hard disk.

Seven years after
Since I wrote the answer below, modifying sys.path is still a quick-and-dirty trick that works well for private scripts, but there has been several improvements
Installing the package (in a virtualenv or not) will give you what you want, though I would suggest using pip to do it rather than using setuptools directly (and using setup.cfg to store the metadata)
Using the -m flag and running as a package works too (but will turn out a bit awkward if you want to convert your working directory into an installable package).
For the tests, specifically, pytest is able to find the api package in this situation and takes care of the sys.path hacks for you
So it really depends on what you want to do. In your case, though, since it seems that your goal is to make a proper package at some point, installing through pip -e is probably your best bet, even if it is not perfect yet.
Old answer
As already stated elsewhere, the awful truth is that you have to do ugly hacks to allow imports from siblings modules or parents package from a __main__ module. The issue is detailed in PEP 366. PEP 3122 attempted to handle imports in a more rational way but Guido has rejected it one the account of
The only use case seems to be running scripts that happen
to be living inside a module's directory, which I've always seen as an
antipattern.
(here)
Though, I use this pattern on a regular basis with
# Ugly hack to allow absolute import from the root folder
# whatever its name is. Please forgive the heresy.
if __name__ == "__main__" and __package__ is None:
from sys import path
from os.path import dirname as dir
path.append(dir(path[0]))
__package__ = "examples"
import api
Here path[0] is your running script's parent folder and dir(path[0]) your top level folder.
I have still not been able to use relative imports with this, though, but it does allow absolute imports from the top level (in your example api's parent folder).

Here is another alternative that I insert at top of the Python files in tests folder:
# Path hack.
import sys, os
sys.path.insert(0, os.path.abspath('..'))

You don't need and shouldn't hack sys.path unless it is necessary and in this case it is not. Use:
import api.api_key # in tests, examples
Run from the project directory: python -m tests.test_one.
You should probably move tests (if they are api's unittests) inside api and run python -m api.test to run all tests (assuming there is __main__.py) or python -m api.test.test_one to run test_one instead.
You could also remove __init__.py from examples (it is not a Python package) and run the examples in a virtualenv where api is installed e.g., pip install -e . in a virtualenv would install inplace api package if you have proper setup.py.

I don't yet have the comprehension of Pythonology necessary to see the intended way of sharing code amongst unrelated projects without a sibling/relative import hack. Until that day, this is my solution. For examples or tests to import stuff from ..\api, it would look like:
import sys.path
import os.path
# Import from sibling directory ..\api
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/..")
import api.api
import api.api_key

For siblings package imports, you can use either the insert or the append method of the [sys.path][2] module:
if __name__ == '__main__' and if __package__ is None:
import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
import api
This will work if you are launching your scripts as follows:
python examples/example_one.py
python tests/test_one.py
On the other hand, you can also use the relative import:
if __name__ == '__main__' and if __package__ is not None:
import ..api.api
In this case you will have to launch your script with the '-m' argument (note that, in this case, you must not give the '.py' extension):
python -m packageName.examples.example_one
python -m packageName.tests.test_one
Of course, you can mix the two approaches, so that your script will work no matter how it is called:
if __name__ == '__main__':
if __package__ is None:
import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
import api
else:
import ..api.api

For readers in 2021: If you're not confident with pip install -e :
Consider this hierarchy, as recommended by an answer from Relative imports in Python 3:
MyProject
├── src
│ ├── bot
│ │   ├── __init__.py
│ │   ├── main.py
│ │   └── sib1.py
│ └── mod
│ ├── __init__.py
│ └── module1.py
└── main.py
The content of main.py, which is the starting point and we use absolute import (no leading dots) here:
from src.bot import main
if __name__ == '__main__':
main.magic_tricks()
The content of bot/main.py, which takes advantage of explicit relative imports:
from .sib1 import my_drink # Both are explicit-relative-imports.
from ..mod.module1 import relative_magic
def magic_tricks():
# Using sub-magic
relative_magic(in=["newbie", "pain"], advice="cheer_up")
my_drink()
# Do your work
...
Now here comes the reasoning:
When executing python MyProject/main.py, the path/to/MyProject is added into the sys.path.
The absolute import import src.bot will read it.
The from ..mod part means it will go up one level to MyProject/src.
Can we see it? YES, since path/to/MyProject is added into the sys.path.
So the point is:
We should put the main script next to MyProject/src, since that when doing relative-referencing, we won't go out of the src, and the absolute import import src. provides the just-fit scope for us: the src/ scope.
See also: ModuleNotFoundError: No module named 'sib1'

TLDR
This method does not require setuptools, path hacks, additional command line arguments, or specifying the top level of the package in every single file of your project.
Just make a script in the parent directory of whatever your are calling to be your __main__ and run everything from there. For further explanation continue reading.
Explanation
This can be accomplished without hacking a new path together, extra command line args, or adding code to each of your programs to recognize its siblings.
The reason this fails as I believe was mentioned before is the programs being called have their __name__ set as __main__. When this occurs the script being called accepts itself to be on the top level of the package and refuses to recognize scripts in sibling directories.
However, everything under the top level of the directory will still recognize ANYTHING ELSE under the top level. This means the ONLY thing you have to do to get files in sibling directories to recognize/utilize each other is to call them from a script in their parent directory.
Proof of Concept
In a dir with the following structure:
.
|__Main.py
|
|__Siblings
|
|___sib1
| |
| |__call.py
|
|___sib2
|
|__callsib.py
Main.py contains the following code:
import sib1.call as call
def main():
call.Call()
if __name__ == '__main__':
main()
sib1/call.py contains:
import sib2.callsib as callsib
def Call():
callsib.CallSib()
if __name__ == '__main__':
Call()
and sib2/callsib.py contains:
def CallSib():
print("Got Called")
if __name__ == '__main__':
CallSib()
If you reproduce this example you will notice that calling Main.py will result in "Got Called" being printed as is defined in sib2/callsib.py even though sib2/callsib.py got called through sib1/call.py. However if one were to directly call sib1/call.py (after making appropriate changes to the imports) it throws an exception. Even though it worked when called by the script in its parent directory, it will not work if it believes itself to be on the top level of the package.

You need to look to see how the import statements are written in the related code. If examples/example_one.py uses the following import statement:
import api.api
...then it expects the root directory of the project to be in the system path.
The easiest way to support this without any hacks (as you put it) would be to run the examples from the top level directory, like this:
PYTHONPATH=$PYTHONPATH:. python examples/example_one.py

Just in case someone using Pydev on Eclipse end up here: you can add the sibling's parent path (and thus the calling module's parent) as an external library folder using Project->Properties and setting External Libraries under the left menu Pydev-PYTHONPATH. Then you can import from your sibling, e. g. from sibling import some_class.

I wanted to comment on the solution provided by np8 but I don't have enough reputation so I'll just mention that you can create a setup.py file exactly as they suggested, and then do pipenv install --dev -e . from the project root directory to turn it into an editable dependency. Then your absolute imports will work e.g. from api.api import foo and you don't have to mess around with system-wide installations.
Documentation

If you're using pytest then the pytest docs describe a method of how to reference source packages from a separate test package.
The suggested project directory structure is:
setup.py
src/
mypkg/
__init__.py
app.py
view.py
tests/
__init__.py
foo/
__init__.py
test_view.py
bar/
__init__.py
test_view.py
Contents of the setup.py file:
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
Install the packages in editable mode:
pip install -e .
The pytest article references this blog post by Ionel Cristian Mărieș.

I made a sample project to demonstrate how I handled this, which is indeed another sys.path hack as indicated above. Python Sibling Import Example, which relies on:
if __name__ == '__main__': import os import sys sys.path.append(os.getcwd())
This seems to be pretty effective so long as your working directory remains at the root of the Python project.

in your main file add this:
import sys
import os
sys.path.append(os.path.abspath(os.path.join(__file__,mainScriptDepth)))
mainScriptDepth = the depth of the main file from the root of the project.
Here is your case mainScriptDepth = "../../". Then you can import by specifying the path (from api.api import * ) from the root of your project.

The problem:
You simply can not get import mypackage to work in test.py. You will need either an editable install, change to path, or changes to __name__ and path
demo
├── dev
│ └── test.py
└── src
└── mypackage
├── __init__.py
└── module_of_mypackage.py
--------------------------------------------------------------
ValueError: attempted relative import beyond top-level package
The solution:
import sys; sys.path += [sys.path[0][:-3]+"src"]
Put the above before attempting imports in test.py. Thats it. You can now import mypackage.
This will work both on Windows and Linux. It will also not care from which path you run your script. It is short enough to slap it anywhere you might need it.
Why it works:
The sys.path contains the places, in order, where to look for packages when attempting imports if they are not found in installed site packages. When you run test.py the first item in sys.path will be something like /mnt/c/Users/username/Desktop/demo/dev i.e.: where you ran your file. The oneliner will simply add the sibling folder to path and everything works. You will not have to worry about Windows vs Linux file paths since we are only editing the last folder name and nothing else. If you project structure is already set in stone for your repository we can also reasonably just use the magic number 3 to slice away dev and substitute src

for the main question:
call sibling folder as module:
from .. import siblingfolder
call a_file.py from sibling folder as module:
from ..siblingfolder import a_file
call a_function inside a file in sibling folder as module:
from..siblingmodule.a_file import func_name_exists_in_a_file
The easiest way.
go to lib/site-packages folder.
if exists 'easy_install.pth' file, just edit it and add your directory that you have script that you want make it as module.
if not exists, just make it one...and put your folder that you want there
after you add it..., python will be automatically perceive that folder as similar like site-packages and you can call every script from that folder or subfolder as a module.
i wrote this by my phone, and hard to set it to make everyone comfortable to read.

First, you should avoid having files with the same name as the module itself. It may break other imports.
When you import a file, first the interpreter checks the current directory and then searchs global directories.
Inside examples or tests you can call:
from ..api import api

Project
1.1 User
1.1.1 about.py
1.1.2 init.py
1.2 Tech
1.2.1 info.py
1.1.2 init.py
Now, if you want to access about.py module in the User package, from the info.py module in Tech package then you have to bring the cmd (in windows) path to Project i.e.
**C:\Users\Personal\Desktop\Project>**as per the above Package example. And from this path you have to enter, python -m Package_name.module_name
For example for the above Package we have to do,
C:\Users\Personal\Desktop\Project>python -m Tech.info
Imp Points
Don't use .py extension after info module i.e. python -m Tech.info.py
Enter this, where the siblings packages are in the same level.
-m is the flag, to check about it you can type from the cmd python --help

Python 3 import precedence with __init__.py

Given the following (notice test which by mistake shadows built in test package, which I should have named it tests, we will come back to this later in the question)
├── test
│   └── test_request_billing_id.py
└── requets_billing_id.py
when in Python shell, try to run from test import test_request_billing_id
it gives ImportError: cannot import name 'test_request_billing_id', which is trying to import the built-in test package instead of my own test folder.
This can be verified by running this valid import
from test import support
Question 1:
Given Python3 allows implicit namespace packages, which means my test folder is also a package, I guess the built in test package has higher priority than my own test package?
Question 2:
I created __init__.py inside my own test folder as below:
├── test
│   ├── __init__.py
│   └── test_request_billing_id.py
└── requets_billing_id.py
and ran the same import statement again, it worked fine.
from test import test_request_billing_id
Shadowing built-in test be verified by running this invalid import
from test import support gives error: ImportError: cannot import name 'support'
This seems to me that having __init__.py tells python interpreter to make my test shadows the built-in test package.
Could someone please explain this or is this documented anywhere?

Yes, packages with an __init__.py take precedence. This is explained in this section of the PEP:
During import processing, the import machinery will continue to
iterate over each directory in the parent path as it does in Python
3.2. While looking for a module or package named "foo", for each directory in the parent path:
If <directory>/foo/__init__.py is found, a regular package is imported and returned.
If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform
and whether the -O flag is specified. The list here is representative.
If not, but <directory>/foo is found and is a directory, it is recorded and the scan continues with the next directory in the parent
path.
Otherwise the scan continues with the next directory in the parent path.
If any namespace packages are encountered during the search process, they're "recorded" and the search continues. If a package containing an __init__.py is found later, the "recorded" namespace packages are discarded and the package with __init__.py is imported instead.
This is exactly what happens when your test package has no __init__.py - the test package in the standard library does have an __init__.py, so it takes precedence over your namespace test package.

Import in python [duplicate]

This question already has answers here:
Relative imports in Python 3
(31 answers)
Closed 6 years ago.
I read a lot of answers related to the question I am asking, but still I do not understand how to make possible this thing I am trying.
So let's go to the point. I will report a simplified version of my application.
Suppose I have a main folder called project and inside it a src main package containing three subpackages:
clustering (containing a file: clustering.py)
parser (containing a file: parser.py)
support_class (containing a file: myClass.py)
In each folder, except for the project one, there is a __init__.py
Now, the python scripts contained in the clustering and parser package should use both the myclass.py contained in support_class.
I tried relative imports, but they do not works because I would like to run the scripts contained in the clustering and parser package directly and I do not want to use the -m option.
Es. python parser.py [arguments]
Example of relative import I used is:
from ..supportClass import myClass
I tried to add the package path to the sys.path but something is not working because it still tells me that it can't find the module.
Es.
sys.path.insert(0, "~/project/src")
from support_class import myClass.py
Can anyone suggest the best way to do this in python 2.7?
If I could avoid the sys.path option it would be great because I do not like this solution that much.
Thanks in advance.

Let's start from your project's folder architecture:
MyProject/
└── src
├── clustering
│   ├── __init__.py
│   └── clustering.py
├── parser
│   ├── __init__.py
│   └── parser.py
├── support_class
│   ├── __init__.py
│   └── support.py
└── main.py
If I'm not mistaken, your issue is that you want to import support.py from within parser.py and clustering.py and being able to run those two independently if needed. Two words for you:
Conditional imports
(And one more, after finding a real other solution ;): PYTHONPATH)
With the assumption that your scripts have a if __name__ == "__main__": section to run your tests, you can simply have the following as their imports:
clustering.py & parser.py:
if __name__ == "__main__":
import sys
import os
PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))
from support_class.support import Support
else:
from support_class.support import Support
main.py:
from support_class.support import Support
Then, python clustering.py and python parser.py to your heart's content!
Which makes this a duplicate of https://stackoverflow.com/a/16985066/3425488

First, you have to create an __init __.py (two = "_", before and after, no spaces) file inside the directory where you have your actual package.
Second, you want to simply call your package from the python script where you are import to.
e.g.:
my_script.py #your script where you want to include/import your package
my_clustering_dir # directory containing files for package
my_clustering.py # file should be inside my_clustering_dir
"__init __".py # make sure this file is inside my_clustering_dir only (it can be empty)
Now, you can go to my_script.py. Then, you can add the following:
from my_clustering_dir import my_clustering #no extension (.py) needed

When you call a python script like this
python parser.py
That module is loaded as __main__, not parser.parser. It won't be loaded as part of any package, so it can't do relative imports. The correct way to do this is to create a separate script that is only used as the main script and not use any of your module scripts as main scripts. For example, create a script like this
main.py
from parser import parser
parser.main()
Then you can run python /path/to/main.py and it will work.
Also, a note on your package layout. In your example, parser and clustering and support_class aren't subpackages, they are top-level packages. Typically, if you have a package named project and you're using a src directory, your layout would look like this:
/project
setup.py
/src
/project
__init__.py
/clustering
__init__.py
/parser
..
Alternatively, if you're building an actual python package with a setup.py script, you can use the console_scripts entry point in setuptools to generate the script automatically.
setup(
...
entry_points = {
'console_scripts': ['myparser=project.parser:main'],
}
...
)

How do I create a namespace package in Python?

In Python, a namespace package allows you to spread Python code among several projects. This is useful when you want to release related libraries as separate downloads. For example, with the directories Package-1 and Package-2 in PYTHONPATH,
Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py
the end-user can import namespace.module1 and import namespace.module2.
What's the best way to define a namespace package so more than one Python product can define modules in that namespace?

TL;DR:
On Python 3.3 you don't have to do anything, just don't put any __init__.py in your namespace package directories and it will just work. On pre-3.3, choose the pkgutil.extend_path() solution over the pkg_resources.declare_namespace() one, because it's future-proof and already compatible with implicit namespace packages.
Python 3.3 introduces implicit namespace packages, see PEP 420.
This means there are now three types of object that can be created by an import foo:
A module represented by a foo.py file
A regular package, represented by a directory foo containing an __init__.py file
A namespace package, represented by one or more directories foo without any __init__.py files
Packages are modules too, but here I mean "non-package module" when I say "module".
First it scans sys.path for a module or regular package. If it succeeds, it stops searching and creates and initalizes the module or package. If it found no module or regular package, but it found at least one directory, it creates and initializes a namespace package.
Modules and regular packages have __file__ set to the .py file they were created from. Regular and namespace packages have __path__set to the directory or directories they were created from.
When you do import foo.bar, the above search happens first for foo, then if a package was found, the search for bar is done with foo.__path__as the search path instead of sys.path. If foo.bar is found, foo and foo.bar are created and initialized.
So how do regular packages and namespace packages mix? Normally they don't, but the old pkgutil explicit namespace package method has been extended to include implicit namespace packages.
If you have an existing regular package that has an __init__.py like this:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
... the legacy behavior is to add any other regular packages on the searched path to its __path__. But in Python 3.3, it also adds namespace packages.
So you can have the following directory structure:
├── path1
│   └── package
│   ├── __init__.py
│   └── foo.py
├── path2
│   └── package
│   └── bar.py
└── path3
└── package
├── __init__.py
└── baz.py
... and as long as the two __init__.py have the extend_path lines (and path1, path2 and path3 are in your sys.path) import package.foo, import package.bar and import package.baz will all work.
pkg_resources.declare_namespace(__name__) has not been updated to include implicit namespace packages.

There's a standard module, called pkgutil, with which you
can 'append' modules to a given namespace.
With the directory structure you've provided:
Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py
You should put those two lines in both Package-1/namespace/__init__.py and Package-2/namespace/__init__.py (*):
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
(* since -unless you state a dependency between them- you don't know which of them will be recognized first - see PEP 420 for more information)
As the documentation says:
This will add to the package's __path__ all subdirectories of directories on sys.path named after the package.
From now on, you should be able to distribute those two packages independently.

This section should be pretty self-explanatory.
In short, put the namespace code in __init__.py, update setup.py to declare a namespace, and you are free to go.

This is an old question, but someone recently commented on my blog that my posting about namespace packages was still relevant, so thought I would link to it here as it provides a practical example of how to make it go:
https://web.archive.org/web/20150425043954/http://cdent.tumblr.com/post/216241761/python-namespace-packages-for-tiddlyweb
That links to this article for the main guts of what's going on:
http://www.siafoo.net/article/77#multiple-distributions-one-virtual-package
The __import__("pkg_resources").declare_namespace(__name__) trick is pretty much drives the management of plugins in TiddlyWeb and thus far seems to be working out.

You have your Python namespace concepts back to front, it is not possible in python to put packages into modules. Packages contain modules not the other way around.
A Python package is simply a folder containing a __init__.py file. A module is any other file in a package (or directly on the PYTHONPATH) that has a .py extension. So in your example you have two packages but no modules defined. If you consider that a package is a file system folder and a module is file then you see why packages contain modules and not the other way around.
So in your example assuming Package-1 and Package-2 are folders on the file system that you have put on the Python path you can have the following:
Package-1/
namespace/
__init__.py
module1.py
Package-2/
namespace/
__init__.py
module2.py
You now have one package namespace with two modules module1 and module2. and unless you have a good reason you should probably put the modules in the folder and have only that on the python path like below:
Package-1/
namespace/
__init__.py
module1.py
module2.py

Categories