We Keep Coding

Proper ways to set the path of my app in Python

I have a question in how to properly create a path in Python (Python 3.x).
I developed a small scraping app in Python with the following directory structure.
root
├── Dockerfile
├── README.md
├── tox.ini
├── src
│   └── myapp
│   ├── __init__.py
│   ├── do_something.py
│   └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
When I want to run my code, I can go to the src directory and do with
python do_something.py
But, because do_something.py has an import statement from do_something_else.py, it fails like:
Traceback (most recent call last):
File "src/myapp/do_something.py", line 1, in <module>
from src.myapp.do_something_else import do_it
ModuleNotFoundError: No module named 'src'
So, I eventually decided to use the following command to specify the python path:
PYTHONPATH=../../ python do_something.py
to make sure that the path is seen.
But, what are the better ways to feed the path so that my app can run?
I want to know this because when I run pytest via tox, the directory that I would run the command tox would be at the root so that tox.ini is seen by tox package. If I do that, then I most likely run into a similar problem due to the Python path not properly set.
Questions I want to ask specifically are:
where should I run my main code when creating my own project like this? root as like python src/myapp/do_something.py? Or, go to the src/myapp directory and run like python do_something.py?
once, the directory where I should execute my program is determined, what is the correct way to import modules from other py file? Is it ok to use from src.myapp.do_something_else import do_it (this means I must add path from src directory)? Or, different way to import?
What are ways I can have my Python recognize the path? I am aware there are several ways to make the pass accessible as below:
a. write export PYTHONPATH=<path_of_my_choice>:$PYTHONPATH to make the
path accessible temporarily, or write that line in my .bashrc to make it permanent (but it's hard to reproduce when I want to automate creating Python environment via ansible or other automation tools)
b. write import sys; sys.path.append(<root>) to have the root as an accessible path
c. use pytest-pythonpath package (but this is not really a generic answer)
Thank you so much for your inputs!
my environment
OS: MacOS and Amazon Linux 2
Python Version: 3.7
Dependency in Python: pytest, tox

I would suggest to use setup.py to make this a python package. Then you can install it in development mode python setup.py develop. This way it will be available in your python environment w/o needing to specify the PYTHONPATH.
For testing, you can simply install the package python setup.py install.
Hope that helps.

Two simple steps should make it happen. Python experts can comment if this is a good way to do it (especially going by the concluding caution raised towards the end of this post).
I would have done it like below.
First I would have put a "__init__.py" in root so that hierarchy looks like below. This way python will treat the folder as a package.
root
├── Dockerfile
├── README.md
├── tox.ini
├── __init__.py
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
Then in "do_something.py", I would have added these lines at the top. In the second line please put the full path to the "root" directory.
import sys
sys.path += ['/home/SomeUserName/SomeFolderPath/root']
from src.myapp.do_something_else import do_it
Please note that the second line will essentially modify the sys.path by adding the root folder path (I guess until the interpreter quits). If this is not what you can afford then I am sorry.

ModuleNotFound while Dockering python app

I have a simple python app, which uses custom class I've created. The following folder structure is the following:
│ mains
| ├── run_it.py
| ├── __init__.py
│   ├── parsers
│ ├── parser.py
│ ├── __init__.py
In the run_it.py, the main program, I'm calling
from mains.parsers.parser import Parser
In local mode I've added to ~/.bashrc the line and it works good:
export PYTHONPATH="${PYTHONPATH}:/home/.../THE_FOLDER_ABOVE_MAINS"
But when I try to dockerize the app, I get the following error:
File "/app/run_it.py", line 11, in <module>
from mains.parsers.parser import Parser
ModuleNotFoundError: No module named 'mains'
My Dockerfile is:
FROM python:3
RUN mkdir /app
WORKDIR /app
ADD . /app/
RUN apt-get update
RUN pip3 install gunicorn
RUN pip3 install -r requirements.txt
EXPOSE 5000
ENV PYTHONIOENCODING=utf-8
ENV GUNICORN_CMD_ARGS="--bind 0.0.0.0:5000 --workers=2"
CMD ["gunicorn","run_it:app"]
Any idea how can I solve it?
Thanks in advance!

I didn't see you set any module path in container, then for your case, the folder app which run the top script run_it.py was automatically added to module path.
As a result, you should use next:
from parsers.parser import Parser
And another way add next to your Dockerfile(suppose it's the same folder of mains) too:
ENV PYTHONPATH=/app
Then you can still use from mains.parsers.parser import Parser

Try this its working..
from parsers.parser import Parser

AWS Chalice using custom config path or custom app path?

I am trying to use Chalice to fit into an pre-existing build folder structure, the python source file (app.py) is one level deeper than the vanilla Chalice project
├── .chalice
│   └── config.json
└── src
├── app.py
├── requirements.txt
├── requirements_test.txt
When I run chalice local in src folder it says it cannot find the config file:
Unable to load the project config file. Are you sure this is a chalice project?
When I run chalice local in the project folder, it says it cannot find the source file:
No module named 'app'
I had a look at the config file doesn't seem to have an option to specify where the source file is.

Since there were no responses on Stackoverflow, so I went to the project and opened an issue ticket for it, from the horses mouth, this is not possible at this stage.

flask cli can't locate the script manage.py

my flask app is a package named app located at /Users/gexinjie/Codes/MyProject/xinnjie_blog
the file tree is like this
xinnjie_blog
├── app
| ├── __init__.py
│   ├── config.py
│   ├── exceptions.py
│   ├── model.py
│   ├── model_sqlalchemy.py
│   ├── static
│   ├── templates
│   ├── util.py
│   └── views
├── manage.py
I export it as PATHONPATH, so manage.py can import app
echo $PATHONPATH
/Users/gexinjie/Codes/MyProject/xinnjie_blog
and export FLASK_APP
echo $FLASK_APP
manage.py
current dir is /Users/gexinjie/Codes/MyProject/xinnjie_blog
pwd
/Users/gexinjie/Codes/MyProject/xinnjie_blog
here is the manage.py
import click
from app import create_app
app = create_app('development')
#app.cli.command()
def initdb():
click.echo('Init the db...')
here is app.__init__.py
from flask import Flask
from .model_sqlalchemy import db
def create_app(config_name='default'):
app = Flask(__name__)
... # init
return app
but then if I execute flask initdb, I get this error:
Usage: flask [OPTIONS] COMMAND [ARGS]...
Error: No such command "initdb".
and if I execute flask run, I get
Usage: flask run [OPTIONS]
Error: The file/path provided (manage) does not appear to exist. Please verify the path is correct. If app is not on PYTHONPATH, ensure the extension is .py
why manage.py is not found? And how can I fix it.
(actually it worked well when manage.py have flask app in itself )
# manage.py
# this work well
app = Flask(__name__) # not app = create_app('development')
Thank you

Thank to #Adam, This problem was solved after I uninstall Anaconda.
Because all the time I tested manage.py on Pycharm command tool, and that flask was installed by Anaconda(python version 3.6), it may lack some extensions(usually I use python3.5 on terminal).So I think the problem occur during importing.
The flask command tool complain about 'can't locate the app', but the real problem is import error.So that is so confusing.
"universal" solution:
So when if you come to this kind of problem like I do, I suggest you first check the location of your app(try both relative path and absolute path), relative path may cause the locate problem when you are not at the right working directory.So absolute path is recommend.
If every thing about path went well, then make sure that all the packages required by your app is installed and can be imported. if you are using some type of virtual environment or something like that (In my case, I use the other version of python lacking some flask extensions), it may be the import error makes flask complain.
Hope this can help you.

Flask can't find the config module in parent directory

I'm basing myself on the structure of a web app I found up on github, here.
My project's structure looks like this:
~/Learning/flask-celery $ tree
.
├── config
│   ├── __init__.py
│   └── settings.py
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
└── web
├── app.py
├── __init__.py
├── static
└── templates
└── index.html
I want my Flask app in web/app.py to load the settings in the config module, as I saw in the githug project linked above.
Here's how I'm instantiating the Flask app in web/app.py:
from flask import Flask, request, render_template, session, flash, redirect, url_for, jsonify
[...]
app = Flask(__name__, instance_relative_config=True)
app.config.from_object('config.settings')
app.config.from_pyfile('settings.py')
[...]
The issue I'm getting is:
root#0e221733b3d1:/usr/src/app# python3 web/app.py
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/werkzeug/utils.py", line 427, in import_string
module = __import__(module_name, None, None, [obj_name])
ImportError: No module named 'config'
[...]
Obviously, Flask can't find the config module in the parent directory, which makes sens to me, but I don't understand how the linked project I'm basing myself on is successfully loading the module with the same tree structure and Flask config code.
In these circumstances, how can I get Flask to load the config module?

Without adding the config directory to your path your Python package will not be able to see it.
Your code can only access what is in web by the looks of it.
You can add the config directory to the package like so:
import os
import sys
import inspect
currentdir = os.path.dirname(
os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0, parentdir)
Then you should be able to import config.

In case anyone finds themselves looking at this question from years ago, it is possible to load the config file in the project structure above like this:
app.config.from_pyfile('../config/settings.py')
This will work when starting the app using flask run