We Keep Coding

How to import from module_utils when running VScode Debugger

I'm trying to develop custom ansible modules using the VSCode Debugger and have run into an import problem.
My ansible role structure looks like this:
.
├── defaults
├── library
│   └── dl_script
├── meta
├── module_utils
│   ├── dl_script2
│   └── dl_script3
├── tasks
├── templates
└── vars
The way ansible works is that the library folder should be where we define custom modules, with the module_utils folder the location for supporting libraries.
When I run my role with ansible it works fine because the ansible packages is supporting these libraries and makes them available to the module, in this case dt_script.
When I try to run this in vscode it falls over.
from ansible.module_utils.dt_script2 import DTScript2
from ansible.module_utils.dt_script3 import DTScript3
Error:
No module named 'ansible.module_utils.dt_script2'
What I'd like to do is figure out a way to import the code from the module_utils folder and make it do so only when it can't do it the ansible way.
I've had a go at doing this myself, but with no success:
try:
from ansible.module_utils.dt_script2 import DTScript2
from ansible.module_utils.dt_script3 import DTScript3
except:
from ..module_utils.dt_script2 import dt_script2
from ..module_utils.dt_script3 import dt_script3
Error:
attempted relative import with no known parent package
EDIT1:
I have a solution, but I don't like it.
try:
from ansible.module_utils.dt_script2 import DTScript2
from ansible.module_utils.dt_script3 import DTScript3
except:
import sys
import os
# I don't like this, but it seems to be the only way to get VScode to look for these files.
sys.path.insert(0,os.path.dirname(os.path.abspath(__file__)).replace('library', 'module_utils'))
from dt_script2 import *
from dt_script3 import *
The reason this works is the by default python only searches the dir and subdir of the place you are running the script from, or certain package locations, like pyenv if you have it. What I'm doing here is amending the path list object to include another path to check from.
Reason this is bad in my mind is that this relies on path manipulation, which makes this very reliant on the host pathing not being really odd. I'd much prefer a safer solution for this, but this is only for when people want to use vscode to debug it.
If anyone has a better solution I'd love to see it.

Importing from above path on python for windows

I have the script I want to run in the following structure
scripts/
project/
main.py
libraries/
a.py
In main.py I need to import things from a.py. How can I import things in subfolders that are two or more folders above main.py?

The proper way to handle this would be putting everything that needs to know about each other under a shared package, then the individual sub-packages and sub-modules can be accessed through that package. But this will also require moving the application's entrypoint to either the package, or a module that's a sibling of the package in the directory and can import it. If moving the entrypoint is an issue, or something quick and dirty is required for prototyping, Python also implements a couple other methods for affecting where imports search for modules which can be found near the end of the answer.
For the package approach, let's say you have this structure and want to import something between the two modules:
.
├── bar_dir
│   └── bar.py
└── foo_dir
└── foo.py
Currently, the two packages do not know about each other because Python only adds the entrypoint file's parent (either bar_dir or foo_dir depending on which file you run) to the import search path, so we have to tell them about each other in some way, this is done through the top level package they'll both share.
.
└── top_level
├── __init__.py
├── bar_dir
│   ├── __init__.py
│   └── bar.py
└── foo_dir
├── __init__.py
└── foo.py
This is the package layout we need, but to be able to use the package in imports, the top packagehas to be initialized.
If you only need to run the one file, you can do for example python -m top_level.bar_dir.bar but a hidden entry point like that could be confusing to work with.
To avoid that, you can define the package as a runnable module by implementing a __main__.py file inside of it, next to __init__.py, which is ran when doing python -m top_level. The new __main__.py entrypoint would then contain the actual code that runs the app (e.g. the main function) while the other modules would only have definitions.
The __init__.py files are used to mark the directories as proper packages and are ran when the package is imported to initialize its namespace.
With this done the packages now can see each other and can be accessed through either absolute or relative imports, an absolute import would being with the top_level package and use the whole dotted path to the module/package we need to import, e.g. from top_level.bar_dir import bar can be used to import bar.
Packages also allow relative imports which are a special form of a from-style import that begins with one or more dots, where each dot means the import goes up one package - from the foo module from . import module would attempt to import module from the foo_dir package, from .. import module would search for it in the top_level package etc.
One thing to note is that importing a package doesn't initialize the modules under it unless it's an explicit import of that module, for example only importing top_level won't make foo_dir and bar_dir available in its namespace unless they're imported directly through import top_level.foo_dir/top_level.bar_dir or the package's __init__.py added them to the package's namespace through its own import.
If this doesn't work in your structure, an another way is to let Python know where to search for your modules by adding to its module search path, this can be done either at runtime by inserting path strings into the sys.path list, or through the PYTHONPATH environment variable.
Continuing with the above example with a scenario and importing bar from foo, an entry for the bar_dir directory (or the directory above it) can be added to the sys.path list or the aforementioned environment variable. After that import bar (or from bar_dir import bar if the parent was added) can be used to import the module, just as if they were next to each other. The inserted path can also be relative, but that is prone to breakage with a changing cwd.

Import libraries from ROOT directory in python

I have the following structure of my program:
MainDir
├── __init__.py
├── main.py
├── Routine.py
└── Folder1
├── __init__.py
└──function.py
I know that I can import a function from folder 1 to my main.py program writing this statement:
from Folder1.function import foo
But, I am looking for a way to do so in the other direction. How could I import a function defined in Routine.py to function.py?

It depends on how you are invoking your program.
Invoking as a module
Do you do something like
python -m MainDir.main
and whatever code is in your MainDir.main calls out to MainDir/Folder1/function.py?
In this case, you can simply add the following import to MainDir/Folder1/function.py:
from ..Routine import RoutineFunction
Invoking as a script
If you are instead invoking MainDir/Folder1/function.py as a script using:
python MainDir/Folder1/function.py
you will not be able to use relative imports as suggested above. There are still many options available to you, as described here: How to import other Python files?
Suggested reading
Python imports can be very confusing. Here is a great post on the topic, which could be useful to you: https://stackoverflow.com/a/14132912/4905625

Python3 importlib.util.spec_from_file_location with relative path?

There are a lot of python questions about how to import relative packages or by explicit location (linked to two popular examples).
In addition there is always the documentation
Having read this, I am still not quite sure what specs are, how they are related to modules, and why one would need to tokenize it.
So for someone who doesn't understand, could you please try to explain how one would do this (programmatically and what the means under the hood)
e.g.
if I have
proj-dir
--mod1
--|--__init__.py
--|--class1.py
--mod2
--|--__init__.py
--|--class2.py
how do I import mod2 into mod1?
import sys
sys.path.insert(0, "../mod2")
this technically works, but I fear that it may cause issues in the future if I try to pickle objects and use them elsewhere...
The explicit location suggested
import importlib.util
spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
foo.MyClass()
so in this case I just do:
import importlib.util
spec = importlib.util.spec_from_file_location("mod2.class2", "../mod2/class2.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
foo.MyClass()
??

First of all, let me clarify the differences between a Python module & a Python package so that both of us are on the same page. ✌
###A module is a single .py file (or files) that are imported under one import and used. ✔
`import aModuleName
Here 'aModuleName' is just a regular .py file.`
###Whereas, a package is a collection of modules in directories that give a package hierarchy. A package contains a distinct __init__.py file. ✔
# Here 'aPackageName` is a folder with a `__init__.py` file
# and 'aModuleName', which is just a regular .py file.
Therefore, the correct version of your proj-dir would be something like this, ⤵
proj-dir
├── __init__.py
├── package1
│ ├── __init__.py
│ ├── module1.py
└── package2
├── __init__.py
└── module2.py
🔎 Notice that I've also added an empty __init__.py into the proj-dir itself which makes it a package too.
👍 Now, if you want to import any python object from module2 of package2 into module1 of package1, then the import statement in the file module1.py would be
from proj-dir.package2.module2 import object2
# if you were to import the entire module2 then,
from proj-dir.package2 import module2

How are absolute imports possible from within a subpackage?

There's something that's bothering me about imports in packages.
Imagine I have the following directory structure:
pack
├── __init__.py
├── sub1
│   ├── __init__.py
│   └── mod1.py
└── sub2
├── __init__.py
└── mod2.py
Inside mod1.py I have the following code to import mod2.py:
# mod1.py
import pack.sub2.mod2
pack.sub2.mod2.helloworld()
I have a main.py file in the directory containing pack that imports pack/sub1/mod1.py
How does mod1.py have access to pack? pack is not in the same directory as mod1.py. Does python automatically add the topmost package to sys.path?

You can investigate this by inspecting sys.path in an interactive interpreter. What you'll find is that the first element of it is the location of the script the interpreter was told to run. This means that when you run your script at the top level (the location of the pack package), that location is added to sys.path automatically. It doesn't have anything to do with the actual package structure, so if you ran mod1.py as a script you would have things break (this is probably why you put your script at the top level!).
Note that in Python 2, you also have the issue of implicit relative imports, which doesn't impact the issue you're asking about, but might come up if you had a few more modules involved. If you added mod3.py to sub1, you could import it from mod1 with just import mod3, with the pack.sub1 prefix being figured out implicitly. This implicit behavior is generally considered a bad thing, and in Python 3 such implicit relative imports are not allowed (you can also disable them in Python 2 with from __future__ import absolute_import). To import pack.sub1.mod3 from pack.sub1.mod1 you'd need to either name it in full, or use an explicit relative import: from . import mod3
To tie this relative import business back to your question, if you wanted to avoid relying on pack being part of sys.path (or, more realistically, protect against changes to pack's name), you could modify your import of mod2 from mod1 to be an explicit relative import. Just use from .. import sub2.mod2.