We Keep Coding

Properly doing relative imports in python

I always have the same problem and I finally want to get rid of it. My folder structure looks like this
project
├── scipts
│ └── folder
│ └── file.py
└── submodules
└── lab_devices
└── optical_devices
└── __init__.py
└── powermeter_driver.py
I now want to include the powermeter_driver.py in file.py. So what I do in the file.py is:
from submodules.lab_devices.optical_devices.powermeter_driver import PowermeterDriver
but this gives ModuleNotFoundError: No module named 'submodules'. I don't want to use
import sys
sys.path.insert(0, '../submodules')
Is there an easy workaround?

The imports will be resolved correctly if you run the script in the correct way, which is from the parent directory and using the -m switch. So you should cd in the parent folder, add __init__.py files as in:
project
├── scripts
└── __init__.py
│ └── folder
└── __init__.py
│ └── file.py
└── submodules
└── __init__.py
└── lab_devices
└── __init__.py
└── optical_devices
└── __init__.py
└── powermeter_driver.py
so that python knows these are packages then run
python -m scripts.folder.file # note no .py
In file.py you can then use the absolute import as you are cause submodules will be detected as a package. You should indeed avoid hacking the sys.path by all means.

You need to consider that if you write from submodules.... this is an absolute import. It means Python starts searching for the submodules in all directories in sys.path. Python usually adds your current working directory as first item to sys.path, so if you cd to your project directory and then run it as a module using python -m it could work.
Of course absolute imports suck if you have files in a relative location to each other. I've had similar issues and I've created an experimental, new import library ultraimport that allows to do file system based imports. It could solve your issue if you are willing to add a new library for this.
Instead of:
from submodules.lab_devices.optical_devices.powermeter_driver import PowermeterDriver
In file.py you would then write:
import ultraimport
PowermeterDriver = ultraimport('__dir__/../../submodules/lab_devices/optical_devices/powermeter_driver.py', 'PowermeterDriver')
The file path is relative to file.py and thus this will always work, no matter how you run your code or what is in sys.path.
One caveat when importing scripts like this is if they contain further relative imports. ultraimport has a builtin preprocessor to rewrite subsequent relative imports so they continue to work.

Import from subfolder python

My project structure is the following. Inside api.py i need some functions written in the upper level.
Project1
├── model.py
├── audio_utils.py
├── audio.py
└── backend
├── static
│ ├──js
│ ├──img
└── api.py
Why am I unable to import inside api.py the functions in the upper level?
When i try to do:
from audio_utils import *
I got the following:
No module named 'audio_utils'

Modules are imported from paths prefixes specified in sys.path. It usually contains '' that means that modules from current working directory are gonna be loaded.
(https://docs.python.org/3/tutorial/modules.html#packages)
I think you are starting your Python interpret while being in the backend directory. Then I think there is no way to access the modules in the upper directory -- not even with the .. (https://realpython.com/absolute-vs-relative-python-imports/#syntax-and-practical-examples_1) unless you change the sys.path which would be a really messy solution.
I suggest you create __init__.py files to indicate that the directories containing them are Python packages:
Project1
├── model.py
├── audio_utils.py
├── audio.py
└── backend
|-- __init__.py
├── static
│ ├──js
│ ├──img
└── api.py
And always start the interpret from the Project1 dir. Doing so, you should be able to import any module like this:
import model
from backed import api
import audio_utils
no matter in which module in the Project1 you are writing this in. The current directory of the interpret will be tried.
Note there is also the PYTHONPATH env variable and that you can use to your advantage.
Note that for publishing your project it is encouraged to put all the modules in a package (in other words: don't put the modues to the top level). This is to help prevent name collisions. I think this may help you to understand: https://realpython.com/pypi-publish-python-package/

You have __init__.py files in both directories right?
Try from ..audio_utils import *

If you create the dir structure this way:
$ tree
.
├── bar
│   ├── den.py
│   └── __init__.py # This indicates the bar is python package.
└── baz.py
1 directory, 3 files
$ cat bar/den.py
import baz
Then in the dir containing the bar/ and baz.py (the top level) you can start the Python interpret and use the absolute imports:
In [1]: import bar.den
In [2]: import baz
In [3]: bar.den.baz
Out[3]: <module 'baz' from '/tmp/Project1/baz.py'>
As you can see, we were able to import bar.den which also could import the baz from the top-level.

Import directory errors with Python 3.5 and above

I want to import modules from folder and subfolder with Python 3.6.
Currently I have a structure like this.
└── project
│ main.py
├── package1
│ ├── __init__.py
│ └── module1.py
└── package2
├── __init__.py
├── module2.py
When I import module1 and module2, I have no problem with that.
Within main.py having this scripts to import both modules.
from package1 import module1
from package2 import module2
It works fine!
But I want to move package2 with module2 into a subfolder under package1 as shown below:
└── project
│ main.py
└── package1
├── __init__.py
├── module1.py
└── subpackage1
├── __init__.py
├── module2.py
And want to call module2 from main.py. Having tried following path and scripts does not fix my problem I get import error, it can not find the path.
Within main.py having this scripts to import both modules.
from package1.subpackage1 import module2
or
from package1.subpackage1.module2 import Class_in_module2
or
sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
from package1.subpackage1 import module2
It does not work. Path can not be found!
Any help appreciate!

Since the from keyword accepts an hierarchy of folders and import the specific method from the file, this should work.
from MainFolder.SubFolder.SomeDotPy import foo,bar,somevalue

Airflow Relative Importing Outside /dag Directory

I haven't been able to move common code outside of the dag directory that airflow uses. I've looked in the airflow source and found imp.load_source.
Is it possible to use imp.load_source to load modules that exist outside of the dag directory? In the example below this would be importing either foo or bar from the common directory.
── airflow_home
|──── dags
│ ├── dag_1.py
│ └── dag_2.py
├── common
├── foo.py
└── bar.py

Just add __init__.py files in all 3 folders. it should work.
Infact every folder in my folder structure having __init__.py. I could run the code and see output.
Example folder structure could be as:
── airflow_home
├── __init__.py
|──── dags
│ ├── __init__.py
│ ├── dag_1.py
│ └── dag_2.py
├── common
├── __init__.py
├── foo.py
└── bar.py
and dag_1.py code can be as:
from stackoverflow.src.questions.airflow_home.common.bar import bar_test
def main():
bar_test()
main()
I'm running this piece of code from my pycharm.
Your airflow_home's folder path in my pycharm is stackoverflow/src/questions/airflow_home/
And bar.py code is
def bar_test():
print "bar hara"

Add your airflow home path to PYTHONPATH
export AIRFLOW_HOME=/usr/local/airflow
export PYTHONPATH="${PYTHONPATH}:${AIRFLOW_HOME}"
Dockerfile
ENV AIRFLOW_HOME=/usr/local/airflow
ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"

Other way instead of adding __init__.py file is to add the following include at the top of dag script:
import sys
import os
sys.path.insert(0,os.path.abspath(os.path.dirname(__file__)))

This works for me, I'm running airflow on docker by the way:
import sys
import os
sys.path.append(os.path.abspath(os.environ["AIRFLOW_HOME"]))

After running into the same problem, this fixed it for me:
import sys, os
sys.path.insert(0,os.path.abspath(os.path.join(os.path.dirname(file),os.path.pardir)))
from common.foo import *

How to define a sub-module in Python?

Python has a module named "os". It also has some other module named "os.path" which is categorized under the "os".
I can use "os.path" methods even if only import the "os" module.
import os
print(os.path.join("sdfs","x"))
I wonder how can I define a sub-module like this?

That's the __init__.py 'magic' of the os module - it imports its submodule path to its namespace, essentially giving you a way to access the latter even if you only import os.
os
|- path
|- __init.__.py # 2
|- __init__.py # 1
The first __init__.py (#1) essentially has import .path so whenever you import just os, it imports path in its namespace, and therefore you can access it as os.path.
(NOTE: This is not exactly the case with the os module, but that's how to essentially achieve it)

Use this structure:
/ Package
├── __init__.py
├── file.py
│
├─┐ subpackage
│ ├── __init__.py
│ └── file.py
│
└─┐ subpackage2
├── __init__.py
└── file.py
Note each subpackage has its own __init__.py file. This will make Package.subpackage behave like os.path, importation speaking (considering you do not import .subpackage under the main __init__ file of Package).