We Keep Coding

Python help working with packages and modules

I need some help with working with a folder structure in python. I was given an structure like this:
/main-folder
/assets
somefiles.txt
/integrations
/module-folder
__init__.py
ingestion.py
__init__.py
models.py
Inside ingestion.py I have:
import os
from models import MyModel
PROJECT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
some_function()
some_processing()
if __name__ == "__main__":
some_function()
Both __init__.py mentioned above are empty.
So I need to process some info and use the models module to store them. when trying to execute intestion.py directly from its dir it says: No module named 'models'. So I'm guessing I have to execute the whole thing as a package. I have no idea how should I import a module located above the package and can't touch the structure.
Any help woud be appreciated.

What you have to do is to add the module's directory to the PYTHONPATH environment variable. If you don't want to do this however, You can modify the sys.path list in your program where the Python interpreter searches for the modules to import, the python documentation says:
When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:
the directory containing the input script (or the current directory).
PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
the installation-dependent default.
After initialization, Python programs can modify sys.path. The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended.
Knowing this, you can do the following in your program:
import sys
# Add the main-folder folder path to the sys.path list
sys.path.append('/path/to/main-folder/')
# Now you can import your module
from main-folder import models
# Or just
import main-folder

How do implicit relative imports work in Python?

Assume I have the following files,
pkg/
pkg/__init__.py
pkg/main.py # import string
pkg/string.py # print("Package's string module imported")
Now, if I run main.py, it says "Package's string module imported".
This makes sense and it works as per this statement in this link:
"it will first look in the package's directory"
Assume I modified the file structure slightly (added a core directory):
pkg/
pkg/__init__.py
plg/core/__init__.py
pkg/core/main.py # import string
pkg/string.py # print("Package's string module imported")
Now, if I run python core/main.py, it loads the built-in string module.
In the second case too, if it has to comply with the statement "it will first look in the package's directory" shouldn't it load the local string.py because pkg is the "package directory"?
My sense of the term "package directory" is specifically the root folder of a collection of folders with __init__.py. So in this case, pkg is the "package directory". It is applicable to main.py and also files in sub- directories like core/main.py because it is part of this "package".
Is this technically correct?
PS: What follows after # in the code snippet is the actual content of the file (with no leading spaces).

Packages are directories with a __init__.py file, yes, and are loaded as a module when found on the module search path. So pkg is only a package that you can import and treat as a package if the parent directory is on the module search path.
But by running the pkg/core/main.py file as a script, Python added the pkg/core directory to the module search path, not the parent directory of pkg. You do have a __init__.py file on your module search path now, but that's not what defines a package. You merely have a __main__ module, there is no package relationship to anything else, and you can't rely on implicit relative imports.
You have three options:
Do not run files inside packages as scripts. Put a script file outside of your package, and have that import your package as needed. You could put it next to the pkg directory, or make sure the pkg directory is first installed into a directory already on the module search path, or by having your script calculate the right path to add to sys.path.
Use the -m command line switch to run a module as if it is a script. If you use python -m pkg.core Python will look for a __main__.py file and run that as a script. The -m switch will add the current working directory to your module search path, so you can use that command when you are in the right working directory and everything will work. Or have your package installed in a directory already on the module search path.
Have your script add the right directory to the module search path (based on os.path.absolute(__file__) to get a path to the current file). Take into account that your script is always named __main__, and importing pkg.core.main would add a second, independent module object; you'd have two separate namespaces.
I also strongly advice against using implicit relative imports. You can easily mask top-level modules and packages by adding a nested package or module with the same name. pkg/time.py would be found before the standard-library time module if you tried to use import time inside the pkg package. Instead, use the Python 3 model of explicit relative module references; add from __future__ import absolute_import to all your files, and then use from . import <name> to be explicit as to where your module is being imported from.

Python test script works when using -munittest, but fails due to ModuleNotFoundError when ran directly

I've got a test script which imports modules from my application. This works when run with python -munittest, but it fails with ModuleNotFoundError if I simply use python tests/test_app.py.
This also happens with other scripts in this project which aren't unit tests.
$ python tests/test_app.py
Traceback (most recent call last):
File "tests/test_app.py", line 2, in <module>
import myapp
ModuleNotFoundError: No module named 'myapp'
$ python -munittest tests.test_app
....
----------------------------------------------------------------------
Ran 4 tests in 0.119s
OK
As you can see from the trace, it fails at the very beginning of the file, in the imports, where I use import myapp.
Project structure:
/project
/myapp
__init__.py
models.py
/otherapp
/anotherapp
/tests
test_app.py
/scripts
a_script.py
What magic is the unittest module doing to make my script load?
How do I execute my script directly?

It fails because when you run with python tests/test_app.pyyour python will look for myappin the local folder relative to the test_app.py or in PYTHONPATH and won't find it. Whereas unittest will handle importing the package while starting from project root folder.
This python documentation link explain how import statement work
When a module named spam is imported, the interpreter first searches
for a built-in module with that name. If not found, it then searches
for a file named spam.py in a list of directories given by the
variable sys.path. sys.path is initialized from these locations:
the directory containing the input script (or the current directory).
PYTHONPATH (a list of directory names, with the same syntax as the
shell variable PATH).
the installation-dependent default.
After initialization, Python programs can modify sys.path. The directory
containing the script being run is placed at the beginning of the
search path, ahead of the standard library path. This means that
scripts in that directory will be loaded instead of modules of the
same name in the library directory. This is an error unless the
replacement is intended. See section Standard Modules for more
information.
Therefore, to make your script directly executable you need to tell python to find "myapp" using one of the ways described in the documentation. Here's another SO question discussing how to use import (you can find a lot more with a quick search).

Python No module named

I have a custom module that I am trying to read from a folder under a hierarchy:
> project-source
/tests
/provider
my_provider.py
settings_mock.py
__init__.py
I am trying to call, from my_provider.py
import tests.settings_mock as settings
Example from command line:
project-source> python tests/provider/my_provider.py
Error:
... ImportError: No module named settings_mock
I keep getting No module named settings_mock as error. I have already exported project_source path to PYTHONPATH. I have made tests into a package by creating a __init__.py file in its root, but no change in the error then.
I can print the settings_mock.py attributes when cd'ing project source
>>> import tests.settings_mock as settings
>>> print settings.storage_provider
correct storage provider value
Is anyone able to point out my mistake here? Thanks!

You only have one small mistake. To use subfolders, you need __init__.py, not init.py as you stated in the question. The difference is that __init__ is a builtin function of python, whereas init is not. Having this file in each subfolder tells the pyhon interpreter that the folder is a "package" that needs to be initialized.
UPDATED: It should be noted that python usually runs from the current directory that the script is located. If your executable main script is my_provider.py, then it's not going to know what to import, since the main script is located in a lower directory than the object it is trying to import. Think of it as a hierarchy. Scripts can only import things that are beneath them. Try separating out the executable from everything else in that file, if there are things that settings_mock needs to import.

Python: is the current directory automatically included in path?

Python 3.4: From reading some other SO questions it seems that if a moduleName.py file is outside of your current directory, if you want to import it you must add it to the path with sys.path.insert(0, '/path/to/application/app/folder'), otherwise an import moduelName statement results in this error:
ImportError: No module named moduleName
Does this imply that python automatically adds all other .py files in the same directory to the path? What's going on underneath the surface that allows you to import local files without appending the Python's path? And what does an __init__.py file do under the surface?

Python adds the directory where the initial script resides as first item to sys.path:
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
So what goes on underneath the surface is that Python appends (or rather, prepends) the 'local' directory to sys.path for you.
This simply means that the directory the script lives in is the first port of call when searching for a module.
__init__.py has nothing to do with all this. __init__.py is needed to make a directory a (regular) package; any such directory that is found on the Python module search path is treated as a module.

I have faced same problem when running python script from Intellij Idea.
There is a script in a
C:\Users\user\IdeaProjects\Meshtastic-python\meshtastic
It uses
from meshtastic import portnums_pb2, channel_pb2, config_pb2
and fails.
I have realized that it looks for
C:\Users\user\IdeaProjects\Meshtastic-python\meshtastic\meshtastic
and changed working directory of this script in Run Configuration from
C:\Users\user\IdeaProjects\Meshtastic-python\meshtastic
to
C:\Users\user\IdeaProjects\Meshtastic-python
so it can find this module UNDERNEATH workdir during execution
C:\Users\user\IdeaProjects\Meshtastic-python\meshtastic