We Keep Coding

Python/Flask - Cause of module not found in the docker container

The src/online_serving/main.py cannot find lib/common/util_string.py although they are in the same /app directory in which Python process is executed.
/app# python3 src/online_serving/main.py
Traceback (most recent call last):
File "/app/src/online_serving/main.py", line 18, in <module>
from lib.common import util_string
ModuleNotFoundError: No module named 'lib'
My directory layout is fairly straightforward. The Docker image is created using the Dockerfile at the BASE directory.
BASE
├── Dockerfile
├── lib
│ ├── __init__.py
│ ├── common
│ │ ├── __init__.py
│ │ └── util_string.py
├── src
│ ├── online_serving
│ │ ├── __init__.py
│ │ ├── main.py
My Dockerfile contains:
FROM python:3.9-slim
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
ENTRYPOINT gunicorn --bind :$PORT --timeout 120 src.online_serving.main:app
main.py imports the lib.common.util_string which is causing error.
from flask import (
Flask,
request,
Response
)
from lib.common import util_string
# Flask runtime
app = Flask(__name__)
As a workaround, I can set this line in the Dockerfile
ENV PYTHONPATH /app
But why isn't the PYTHONPATH automatically set to the /app directory in the container?

When the Python interpreter is started with python a/b/c.py, the current directory is not added in sys.path but the directory where the script c.py is.
$ python a/b/c.py
$ pwd
/app
$ tree
a
└── b
└── c.py
$ cat a/b/c.py
import sys
for p in sys.path:
print(p)
$ python3 a/b/c.py
/app/a/b # <--- path[0] is the directory of the script used to invoke the interpreter.
/usr/local/lib/python39.zip
/usr/local/lib/python3.9
/usr/local/lib/python3.9/lib-dynload
/usr/local/lib/python3.9/site-packages
sys.path
A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default.
As initialized upon program startup, the first item of this list,path[0], is the directory containing the script that was used to invoke the Python interpreter.
If the script directory is not available, thenpath[0] is the empty string, which directs Python to search modules in the current directory first.(e.g. if the interpreter is invoked interactively or if the script is read from standard input),
Notice that the script directory is insertedbeforethe entries inserted as a result ofPYTHONPATH.
To add the current directory to sys.path so that a module is searched in the current directory tree, need to use PYTHONPATH or start the interpreter with -m option.
-m
If this option is given, the first element of sys.argv will be the full path to the module file (while the module file is being located, the first element will be set to "-m"). As with the -c option, the current directory will be added to the start of sys.path.
$ pwd
/app
$ tree
a
└── b
└── c.py
$ export PYTHONPATH=${PYTHONPATH}:/app
$ python a/b/c.py
/app/a/b # <--- path[0] is the directory of the script used to invoke the interpreter.
/app # <--- then from the environment variable PYTHONPATH, plus an installation-dependent default.
/usr/local/lib/python39.zip
/usr/local/lib/python3.9
/usr/local/lib/python3.9/lib-dynload
/usr/local/lib/python3.9/site-packages
Or
$ pwd
/app
$ tree
a
└── b
└── c.py
$ cat a/b/c.py
import sys
for p in sys.path:
print(p)
$ python3 -m a.b.c
/app # <--- Current directory is path[0] for python -m <module> invocation
/usr/local/lib/python39.zip
/usr/local/lib/python3.9
/usr/local/lib/python3.9/lib-dynload
/usr/local/lib/python3.9/site-packages

uwisg: ModuleNotFoundError: No module named 'flask'

I am trying to make a flask application, and I am following the well known tutorials out there. I am using docker containers, so I don't need virtual environments, as the containers are already isolated environments.
that's my project folder for the flask app:
flask
├── Dockerfile
├── app
│   ├── __init__.py
│   ├── routes.py
│   ├── static
│   └── templates
│   ├── index.html
│   └── test.html
├── requirements.txt
├── run.py
└── uwsgi.ini
When starting up the container with docker-compose, I get the following error.
Traceback (most recent call last):
File "run.py", line 1, in <module>
from app import app #as application
File "./app/__init__.py", line 1, in <module>
from flask import Flask
ModuleNotFoundError: No module named 'flask'
unable to load app 0 (mountpoint='') (callable not found or import error)
I have read many other questions with such an error, but none of those solutions helped or had to do with venv (which I am not using). So I hope someone can pinpoint, where the problem might be.
I have installed flask in Dockerfile with
RUN python3 -m pip install --disable-pip-version-check --no-cache-dir -r /tmp/requirements.txt
(I also just used pip3 install... but it does not make a difference, flask is installed either way). from the build log:
Successfully installed Jinja2-3.0.3 MarkupSafe-2.0.1 Werkzeug-2.0.2 click-8.0.3 flask-2.0.2 itsdangerous-2.0.1
## run.py
from app import app
This is the import that seems to fail:
## __init__.py
from flask import Flask
app = Flask(__name__)
from app import routes
extract from uwisg.ini:
[uwsgi]
plugin = python3
wsgi-file = run.py
callable = app
After starting the container with docker-compose, I exec into the container, start a Python3 interactive shell and have no problems at all doing from flask import Flask.
I have tried running the container as root but that did not change anything.
Any ideas? thanks!

Ok, after more rinse and repeat trying, I figured it out.
In my Dockerfile I also set the PYTHONPATH to my workspace:
ENV PYTHONPATH "${PYTHONPATH}:/workspace/flask"
ENV PYTHONPATH "${PYTHONPATH}:/workspace/flask/app"
and site-packages apperently could not be accessed by Python anymore.
I also had to add this:
ENV PYTHONPATH "${PYTHONPATH}:/usr/local/lib/python3.11/site-packages"
I have no idea, why this path was not accessible for Python? Did I overwrite sys.path when using ENV PYTHONPATH? Should this not be additive to sys.path?

How to run a bash script from dockerfile

I am using a bash script (run.sh) to run a python file (calculate_ssp.py). The code is working fine. The folder structure is given below
├── dataset
└── dataset_1.csv
├── Dockerfile
├── __init__.py
├── output
├── run.sh
├── scripts
│   ├── calculate_ssp.py
│   ├── __init__.py
The bash script is
#!/bin/bash
python3 -m scripts.calculate_ssp 0.5
Now, I am trying to run the bash script (run.sh) from the Dockerfile. The content of the Dockerfile is
#Download base image ubuntu 20.04
FROM ubuntu:20.04
#Download python
FROM python:3.8.5
ADD run.sh .
RUN pip install pandas
RUN pip install rdkit-pypi
CMD ["bash", "run.sh"]
But, I am getting an error /usr/local/bin/python3: Error while finding module specification for 'scripts.calculate_ssp' (ModuleNotFoundError: No module named 'scripts')
Could you tell me why I am getting the error (as the code is working fine from the bash script)?

You need to package all the files required by your script in the image. You can drop the Ubuntu base image, since it isn't used. (The python base image is based on some other image that already provides bash.)
#Download python
FROM python:3.8.5
RUN pip install pandas
RUN pip install rdkit-pypi
ADD run.sh .
ADD scripts/ ./scripts
CMD ["bash", "run.sh"]

In the Bash script, write a program that creates a file named Dockerfile. The contents of the Dockerfile should have the following commands:
First, the base image should install python3 via the FROM command. Then the rest of the Dockerfile should look like the following:
RUN pip install {{MODULES}}
CMD ["python", {{FILENAME}}]
{{MODULES}} should be replaced with the modules: numpy, scipy, and pandas all on one line. {{FILENAME}} should be replaced with ./main.pyv

Error: the command ( (from my_script) could not be found within PATH

While working on a Python FastAPI project using Pipenv and Pytest, I was asked to write a Pipenv script to run the tests.
Project has the following structure:
.
├── app
| ├── main.py
│   ├── my_package
│   │   ├── __init__.py
│   │   ├── (project folders)
│   │   └── tests
| | └── tests.py
│   └── __pycache__
└──(deployment scripts and folders, Pipfile, Dokerfile, etc)
I'd like to run Pytest on tests.py from the top structure (./app/.../tests.py). At least for now.
As recommended here, I currently run them from the top with:
( cd app ; python -m pytest my_package/tests/tests.py )
... which works as expected.
However, when I add that my Pipfile-scripts-section:
[scripts]
my_script = "( cd app ; python -m pytest my_package/tests/tests.py )"
... run it with:
pipenv run my_script
I get the error:
Error: the command ( (from my_script) could not be found within PATH.
I've also tried:
[scripts]
my_script = "cd app && python -m pytest my_package/tests/tests.py"
... which returns another similar error:
Error: the command cd (from my_script) could not be found within PATH.
So it's clear I'm wrong to use it as bash aliases.
I've tried searching for more documentation on how the [scripts] section works as I, but I've had no luck (yet).

I'm not familiar with the tools you're using, but the error message suggests that it's looking for an executable. ( is part of shell syntax, and cd is a shell builtin, not an executable. Try this:
my_script = "bash -c 'cd app && python -m pytest my_package/tests/tests.py'"
Here bash is the executable, and -c makes it run your snippet.
BTW, keep in mind that cd can fail, so the script should bail out if it does.

running tornado python in docker: ModuleNotFoundError: No module named '__main__.config'; '__main__' is not a package

I have a tornado app with a file structure like this:
projectfolder
|_ Dockerfile
|_ src
|_ __init__.py
|_ __main__.py
|_ config.py
My dockerfile looks like this:
FROM python:3.7
EXPOSE 8888
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENTRYPOINT ["python3", "projectfolder/__main__.py"]
in __main__.py I do from .config import Config
and when doing docker run I get:
File "projectfolder/__main__.py", line 11, in <module>
from .config import Config
ModuleNotFoundError: No module named '__main__.config'; '__main__' is not a package
When I run the project inside .venv it works fine, but i cannot make it work on docker
Any ideas on how to properly import the files in the current folder? Or how should I re-structure my files in other it follows python best practices
Thanks!

1. I guess you have a typo for ENTRYPOINT ["python3", "projectfolder/__main__.py"], it should be ENTRYPOINT ["python3", "src/__main__.py"], otherwise you will get:
python3: can't open file 'projectfolder/main.py': [Errno 2] No such file or directory
I guess you made some mistake when minimal your question, because COPY . . will not copy projectfolder, instead the contents of projectfolder.
2. Back to your issue:
ModuleNotFoundError: No module named '__main__.config'; '__main__' is not a package
Here, python find __name__ is __main__, which means the script calling from .config import Config is a top level script, not in a package.
And relative import just could be used in package, not out of package. So you can just change from .config import Config to from config import Config, remove the . which means relative import then could fix your issue.