We Keep Coding

How to import modules from a script that could be called from anywhere?

I have a repo with Python code whose structure could be boiled down to this:
repo_root\
tool1\
tool1.py
tool1_aux_stuff.py
tool2\
tool2.py
tool2_aux_stuff.py
lib\
lib1\
lib1.py
lib1_aux_stuff.py
lib2\
lib2.py
lib2_aux_stuff.py
The following rules apply to the module usage:
Any tool could use the modules from any library and from its own package, but not from a different tool's one.
Any library could use the modules from any other library, and from its own package. Libraries never access the tool modules.
There must be a way to invoke any tool from any working directory, including those outside repo_root.
The question is: how do I import the lib modules from the tool ones?
I know that if I add __init__.py to each tool and lib directory and to the repo root, then I would be able to use absolute paths from the root, i.e. in tool1.py I could write
import lib.lib1, lib.lib2.lib2_aux_stuff
However, if I execute tool1.py from a random place, e.g.
machine_name: ~/random/place$ python /path/to/repo/tool1/tool1.py
I get the ModuleNotFoundError: No module named 'lib' found error.
I am aware of a workaround which could be implemented using the PYTHONPATH env variable by augmenting it with an absolute path to repo_root and supplying it to the invocation of the tool script, i.e.:
machine_name: ~/random/place$ PYTHONPATH=$PYTHONPATH:/path/to/repo python /path/to/repo/tool1/tool1.py
but I would really prefer something less clunky.
Any ideas how I could do it in a more straightforward way?

Add the path to lib to the scope using sys.path.append('/custom/path/to/modules'). It should then be callable as a module.
You do need to add __init__.py files in any directory that you want to import as if it were a module, otherwise Python doesn't treat them as modules and you'll get another ImportError

How do I find details of this python package (__main__)? [duplicate]

Currently trying to work in Python3 and use absolute imports to import one module into another but I get the error ModuleNotFoundError: No module named '__main__.moduleB'; '__main__' is not a package. Consider this project structure:
proj
__init__.py3 (empty)
moduleA.py3
moduleB.py3
moduleA.py3
from .moduleB import ModuleB
ModuleB.hello()
moduleB.py3
class ModuleB:
def hello():
print("hello world")
Then running python3 moduleA.py3 gives the error. What needs to be changed here?

.moduleB is a relative import. Relative only works when the parent module is imported or loaded first. That means you need to have proj imported somewhere in your current runtime environment. When you are are using command python3 moduleA.py3, it is getting no chance to import parent module. You can:
from proj.moduleB import moduleB OR
You can create another script, let's say run.py, to invoke from proj import moduleA
Good luck with your journey to the awesome land of Python.

Foreword
I'm developing a project which in fact is a Python package that can be installed through pip, but it also exposes a command line interface. I don't have problems running my project after installing it with pip install ., but hey, who does this every time after changing something in one of the project files? I needed to run the whole thing through simple python mypackage/main.py.
/my-project
- README.md
- setup.py
/mypackage
- __init__.py
- main.py
- common.py
The different faces of the same problem
I tried importing a few functions in main.py from my common.py module. I tried different configurations that gave different errors, and I want to share with you with my observations and leave a quick note for future me as well.
Relative import
The first what I tried was a relative import:
from .common import my_func
I ran my application with simple: python mypackage/main.py. Unfortunately this gave the following error:
ModuleNotFoundError: No module named '__main__.common'; '__main__' is not a package
The cause of this problem is that the main.py was executed directly by python command, thus becoming the main module named __main__. If we connect this information with the relative import we used, we get what we have in the error message: __main__.common. This is explained in the Python documentation:
Note that relative imports are based on the name of the current module. Since the name of the main module is always __main__, modules intended for use as the main module of a Python application must always use absolute imports.
When I installed my package with pip install . and then ran it, it worked perfectly fine. I was also able to import mypackage.main module in a Python console. So it looks like there's a problem only with running it directly.
Absolute import
Let's follow the advise from the documentation and change the import statement to something different:
from common import my_func
If we now try to run this as before: python mypackage/main.py, then it works as expected! But, there's a caveat when you, like me, develop something that need to work as a standalone command line tool after installing it with pip. I installed my package with pip install . and then tried to run it...
ModuleNotFoundError: No module named 'common'
What's worse, when I opened a Python console, and tried to import the main module manually (import mypackage.main), then I got the same error as above. The reason for that is simple: common is no longer a relative import, so Python tries to find it in installed packages. We don't have such package, that's why it fails.
The solution with an absolute import works well only when you create a typical Python app that is executed with a python command.
Import with a package name
There is also a third possibility to import the common module:
from mypackage.common import my_func
This is not very different from the relative import approach, as long as we do it from the context of mypackage. And again, trying to run this with python mypackage/main.py ends similar:
ModuleNotFoundError: No module named 'mypackage'
How irritating that could be, the interpreter is right, you don't have such package installed.
The solution
For simple Python apps
Just use absolute imports (without the dot), and everything will be fine.
For installable Python apps in development
Use relative imports, or imports with a package name on the beginning, because you need them like this when your app is installed. When it comes to running such module in development, Python can be executed with the -m option:
-m mod : run library module as a script (terminates option list)
So instead of python mypackage/main.py, do it like this: python -m mypackage.main.

In addition to md-sabuj-sarker's answer, there is a really good example in the Python modules documentation.
This is what the docs say about intra-package-references:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
If you run python3 moduleA.py3, moduleA is used as the main module, so using the absolute import looks like the right thing to do.
However, beware that this absolute import (from package.module import something) fails if, for some reason, the package contains a module file with the same name as the package (at least, on my Python 3.7). So, for example, it would fail if you have (using the OP's example):
proj/
__init__.py (empty)
proj.py (same name as package)
moduleA.py
moduleB.py
in which case you would get:
ModuleNotFoundError: No module named 'proj.moduleB'; 'proj' is not a package
Alternatively, you could remove the . in from .moduleB import, as suggested here and here, which seems to work, although my PyCharm (2018.2.4) marks this as an "Unresolved reference" and fails to autocomplete.

Maybe you can do this before importing the module：
moduleA.py3
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from moduleB import ModuleB
ModuleB.hello()
Add the current directory to the environment directory

Just rename the file from where you run the app to main.py:
from app import app
if __name__ == '__main__':
app.run()

import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
will solve the issue of import path issue.

Can't import modules - ImportError: No module named

I've created a new app called engineapp. Inside this app, there is a folder engine which is a Scrapy project.
When I try to import model from storage app inside top.py file, it returns:
from storage.models import TopItem
ImportError: No module named storage.models
Or the similar problem, when I try to import settings of scrapy project:
from engineapp.engine.engine import settings
It returns:
from engineapp.engine.engine import settings
ImportError: No module named engineapp.engine.engine
This is when I run scrapy project from command line.
Both imports created PyCharm itself.
As you can see, I've added __init__() everywhere so python would be able to recognize those files.
Do you know what should I do to be able to import those files?

PyCharm autocomplete relies on different IDE settings. You've marked your realstate_scanner as sources root, so PyCharm can resolve imports for this. From docs:
These roots contain the actual source files and resources. PyCharm uses the source roots as the starting point for resolving imports.
If you can't import some module in python, you should check PATH/PYTHONPATH variables first to make sure python interpreter knows where to find your module.

__init__ file doesn't work as expected in python

I have some folders and .py files in the following structure:
parent/
__init__.py
test.ipynb
code/
__init__.py
common.py
subcode/
__init__.py
get_data.py
In the __init__ file under the parent folder, I have import code and in the one of code, I have import subcode. But when I tried import code.subcode, I got such an error:
ImportError: No module named 'code.subcode'; 'code' is not a package
But when I just import code, no error is thrown. However, when I call code.subcode, this error happens:
AttributeError: module 'code' has no attribute 'subcode'
I try all of those mentioned above in the test.ipynb, which is at the root of the directory.
Do you know what is the reason and how can I fix it? Thanks!

The problem is that you are importing another module named code that is installed on your system rather than your own module. You can verify this by checking the module file path in code.__file__ after you import code.
The first thing to do is change the name of your module to avoid namespace collisions with the other code package on your system. If your new package name doesn't collide with something else, you should now either successfully be importing it and have it behave as expected, or it fails to import entirely.
If it fails to import, it is most likely because your parent directory is not in your PYTHONPATH environment variable.
There can potentially also be other more technical reasons that a module is not recognized by the interpreter such as old definitions being cached (in which case restarting the interpreter is often enough. Possibly after deleting any precompiled versions of the module). Another problem I have seen ended up being that a module contained a bug that made the interpreter unable to parse it. I am sure there are other odd possibilities out there.

You're on Python 3. You need to perform relative imports explicitly:
from . import code
The code module you're currently getting is the standard library code module.

Import module into python not found

I am trying to import the module
import QSTK.qstkutil.qsdateutil as du
But I get the Error
ImportError: No module named QSTK.qstkutil.qsdateutil
My current working directory is
'c:\\Python27\\Lib\\site-packages\\QSTK'
and in the path C:\Python27\Lib\site-packages\QSTK\qstkutil there are the files
qsdateutil.py
qsdateutil.pyc
qsdateutil.pyo

Does importing QSTK work?
import QSTK
How about QSTK.qstkutil?
If not this is most likely a sys.path problem. Please post the result of:
>>>import sys
>>>sys.path
It should look like:
[ [...], 'C:\Python27\Lib\site-packages', [...] ]
Another thing you can check, is if 'C:\Python27\Lib\site-packages\QSTK\qstkutil' contains a file named '__init__.py'. From the module documentation:
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.

try a fresh installation and make sure you run sudo python setup.py install , command after unpack-aging , QSTK. that process links QSTK.qstkutil.qsdateutil.