We Keep Coding

Docker-compose flask app not printing output from 'print'

I have a flask app that has one route and nothing complex going on, running in a docker container. I cannot for the life of me get print statements to show up in the logs (docker-compose logs -f <containername>). So far, I have tried various answers that supposedly have fixed this problem for others including:
Calling print("test", flush=True)
Setting PYTHONUNBUFFERED=1 and verifying it is set in the actual container with echo
Setting PYTHONUNBUFFERED=0
Running python with the -u flag
Using the logging module (logger.warning, logger.info, etc)
So far nothing has worked. The flask app is starting perfectly fine, but no output from my print statements is shown. I have sanity checked that i'm editing the correct file by adding random syntax errors and watching the app brick itself. I'm using python 3.8 and docker-compose 2

Try this:
import sys
print('It is working',file=sys.stderr)

I found this question while looking for answers to a similar problem. I was running a flask app in a conda environment in a container and wasn't getting any log output even though the flask app itself was working fine. I added the following lines to my Dockerfile and it starting logging as expected -
ENV PYTHONUNBUFFERED=1
RUN echo "source activate my_env" > ~/.bashrc
ENV PATH /opt/conda/envs/my_env/bin:$PATH
CMD ["python", "api.py"]

You can see logs with docker-compose or docker
With docker-compose you have to see SERVICE
Note: you add containername but you have to add service name
NOT $ docker-compose logs -f <containername>
USE $ docker-compose logs -f <SERVICE_NAME>)
With docker you have to add container name or container id
docker logs -f CONTAINER_ID | CONTAINER_NAME

docker and python using symfony process

I am using Laradock and want to be able to run a python script from my laravel app using Symfony Process. From inside the root on my container I can run "python3 script_name.py arg1" and it runs just fine. pip list shows all modules needed. When I run it from inside Laravel, it tells me:
"import pymysql ImportError: No module named 'pymysql'"
I have used a non-docker Laravel app to do this just fine, using:
$script = storage_path().'/app/script.py';
$process = new Process('python3 '. $script." ".session('division'));
What am I missing?

On *nix make sure that PYTHONPATH is configured correctly for all users or try to set full path to python3.
How to check
At first your php user
php -r "print shell_exec( 'whoami' );" // somebody
When run
su somebody python3 script_name.py arg1

Start django app as service

I want to create service that will be start with ubuntu and will have ability to use django models etc..
This service will create thread util.WorkerThread and wait some data in main.py
if __name__ == '__main__':
bot.polling(none_stop=True)
How I can to do this. I just don't know what I need to looking for.
If you also can say how I can create ubuntu autostart service with script like that, please tell me )
P.S. all django project run via uwsgi in emperor mode.

The easiest way in my opinion is create a script and run on crontab.
First of all create a script to start your django app.
#!/bin/bash
cd /path/to your/virtual environment #path to your virtual environment
. bin/activate #Activate your virtual environment
cd /path/to your/project directory #After that go to your project directory
python manage.py runserver #run django server
Save the script and open crontab with the command:
crontab -e
Now edit the crontab file and write on the last line:
#reboot path/to/your/script.sh
This way is not the best but the easiest, if you are not comfortable with Linux startup service creation.
I hope this help you :)

Take a look at supervisord. It is much easier than daemonizing python script.
Config it something like this:
[program:watcher]
command = /usr/bin/python /path/to/main.py
stdout_logfile = /var/log/main-stdout.log
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups = 5
stderr_logfile = /var/log/main-stderr.log
stderr_logfile_maxbytes = 10MB
stderr_logfile_backups = 5

Ok, that is answer - https://www.raspberrypi-spy.co.uk/2015/10/how-to-autorun-a-python-script-on-boot-using-systemd/
In new versions ubuntu services .conf in /etc/init fail with error Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
But services works using systemd

Running Django migrations when deploying to Elastic Beanstalk

I have my Django app set up on Elastic Beanstalk and recently made a change to the DB that I would like to have applied to the live DB now. I understand that I need to set this up as a container command, and after checking the DB I can see that the migration was run, but I can't figure out how to have more controls over the migration. For example, I only want a migration to run when necessary but from my understanding, the container will run the migration on every deploy assuming the command is still listed in the config file. Also, on occassion, I will be given options during a migration such as:
Any objects realted to these content types by a foreign key will also be deleted.
Are you sure you want to delete these content types?
If you're unsure, answer 'no'
How do I set up the container command to respond to this with a yes during the deployment phase?
This is my current config file
container_commands:
01_migrate:
command: 'source /opt/python/run/venv/bin/actiate && python app/manage.py makemigrations'
command: 'source /opt/python/run/venv/bin/activate && python app/manage.py migrate'
Is there a way to set these 2 commands to only run when necessary and to respond to the yes/no options I receive during a migration?

I'm not sure there is a specific way to answer yes or no. but you can append --noinput to your container command. Use the --noinput option to suppress all user prompting, such as “Are you sure?” confirmation messages.
try
command: 'source /opt/python/run/venv/bin/activate && python app/manage.py migrate --noinput'
OR..
You can ssh into your elasticbean instance and run your command manually.
Then you'll have more control over the migrations.
Install awsebcli with pip install awsebcli
Type eb ssh Your EnvironmentName
Navigate to your eb instance app directory with:
sudo -s
source /opt/python/run/venv/bin/activate
source /opt/python/current/env
cd /opt/python/current/app
then run your command.
./manage.py migrate
I hope this helps

Aside from the automatic migration that you can add to deploy script (which runs every time you update the environment, and may not be desirable if you have long running migration or other Django management commands), you can ssh into an EB instance to run migration manually.
Here is how to manually run migration (and any other Django management commands) while working with Amazon Linux 2 (Python 3.7, 3.8) created by Elastic Beanstalk:
First, from your EB cli: eb ssh to connect an instance.
The virtual environment can be activated by
source /var/app/venv/*/bin/activate
The manage.py can be ran by
python3 /var/app/current/manage.py
Now the only tricky bit is to get Elastic Beanstalk's environment variables. You can access them by /opt/elasticbeanstalk/bin/get-config, I'm not super familiar with bash script, but here is a little script that I use to get and set environment variables, maybe someone can improve it to make it less hard-coded:
#! /bin/bash
export DJANGO_SECRET_KEY=$(/opt/elasticbeanstalk/bin/get-config environment -k DJANGO_SECRET_KEY)
...
More info regarding Amazon Linux 2 splatform script tools: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/custom-platforms-scripts.html

Make sure that the same settings are used when migrating and running!
Thus I would recommend you change this kind of code in django.config
container_commands:
01_migrate:
command: "source /opt/python/run/venv/bin/activate && python manage.py migrate"
leader_only: true
to:
container_commands:
01_migrate:
command: "django-admin migrate"
leader_only: true
option_settings:
aws:elasticbeanstalk:application:environment:
DJANGO_SETTINGS_MODULE: fund.productionSettings
as recommended here. This will help you avoid issues with wrong settings used.
More on manage.py v.s. django-admin.py.

django-admin method not working as it was not configured properly. You can also use python manage.py migrate in
.ebextentions/django.config
container_commands:
01_migrate:
command: "python manage.py migrate"
leader_only: true

In reference to Oscar Chen answer, you can set environmental variables using eb cli with
eb setenv key1=value1 key2=valu2 ...etc

The trick is that the full output of container_commands is in /var/log/cfn-init-cmd.log (Amazon Linux 2 Elastic Beanstalk released November 2020).
To view this you would run:
eb ssh [environment-name]
sudo tail -n 50 -f /var/log/cfn-init-cmd.log
This doesn't seem to be documented anywhere obvious and it's not displayed by eb logs; I found it by hunting around in /var/log.
The Django example management command django-admin.py migrate did not work for me. Instead I had to use something like:
01_migrate:
command: "$PYTHONPATH/python manage.py migrate"
leader_only: true
02_collectstatic:
command: "$PYTHONPATH/python manage.py collectstatic --noinput --verbosity=0 --clear"
To see the values of your environment variables at deploy time, you can create a debug command like:
03_debug:
command: "env"
You can see most of these environment variable with eb ssh; sudo cat /opt/elasticbeanstalk/deployment/env, but there seem to be some subtle differences at deploy time, hence using env above to be sure.
Here you'll see that $PYTHONPATH is being in a non-typical way, pointing to the virtualenv's bin directory, not the site-packages directory.

This answer looks like it will work for you if you just want to send "yes" to a few prompts.
You might also consider the --noinput flag so that your config looks like:
container_commands:
01_migrate:
command: 'source /opt/python/run/venv/bin/actiate && python app/manage.py makemigrations'
command: 'source /opt/python/run/venv/bin/activate && python app/manage.py migrate --noinput
This takes the default setting, which is "no".
It also appears that there's an open issue/fix to solve this problem a better way.

Docker-compose and pdb

I see that I'm not the first one to ask the question but there was no clear answer to this:
How to use pdb with docker-composer in Python development?
When you ask uncle Google about django docker you get awesome docker-composer examples and tutorials and I have an environment working - I can run docker-compose up and I have a neat developer environment but the PDB is not working (which is very sad).
I can settle with running docker-compose run my-awesome-app python app.py 0.0.0.0:8000 but then I can access my application over http://127.0.0.1:8000 from the host (I can with docker-compose up) and it seems that each time I use run new containers are made like: dir_app_13 and dir_db_4 which I don't desire at all.
People of good will please aid me.
PS
I'm using pdb++ for that example and a basic docker-compose.yml from this django example. Also I experimented but nothing seems to help me. And I'm using docker-composer 1.3.0rc3 as it has Dockerfile pointing support.

Use the following steps to attach pdb on any python script.
Step 1. Add the following in your yml file
stdin_open: true
tty: true
This will enable interactive mode and will attach stdin. This is equivalent for -it mode.
Step 2.
docker attach <generated_containerid>
You'll now get the pdb shell

Try running your web container with the --service-ports option: docker-compose run --service-ports web

If after the adding of
stdin_open: true
tty: true
you started to get issues similar to that:
fd = self._input_fileno()
if fd is not None and fd in ready:
> return ord(os.read(fd, 1))
E TypeError: ord() expected a character, but string of length 0 found
You can try to add ENV LC_ALL en_US.UTF-8 at the top of your Docker file
FROM python:3.8.2-slim-buster as build_base
ENV LC_ALL en_US.UTF-8

Till my experience, docker-compose up command does not provide an interactive shell, but it starts the printing STDOUT to default read-only shell.
Or if you have specified and mapped logs directory, docker-compose up command will print nothing on the attached shell but it sends output to your mapped logs. So you have to attach the container separately once it is running.
when you do docker-compose up, make it in detached mode via -d and connect to the container via
docker exec -it your_container_name bash