We Keep Coding

How to run Python inside an expressjs Docker container

i am trying to build a container for my express.js application. The express.js-app makes use of python via the npm package PythonShell.
I have plenty of python-code, which is in a subfolder of my express-app and with npm start everything works perfectly.
However, i am new to docker and i need to containerize the app. My Dockerfile looks like this:
FROM node:18
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3001
CMD ["node", "./bin/www"]
I built the Image with:
docker build . -t blahblah-server and ran it with docker run -p 8080:3001 -d blahblah-server.
I make use of imports at the top of the python-script like this:
import datetime
from pathlib import Path # Used for easier handling of auxiliary file's local path
import pyecma376_2 # The base library for Open Packaging Specifications. We will use the OPCCoreProperties class.
from assi import model
When the pythonscript is executed (only in the container!!!) I get following error-message:
/usr/src/app/public/javascripts/service/pythonService.js:12
if (err) throw err;
^
PythonShellError: ModuleNotFoundError: No module named 'pyecma376_2'
at PythonShell.parseError (/usr/src/app/node_modules/python-shell/index.js:295:21)
at terminateIfNeeded (/usr/src/app/node_modules/python-shell/index.js:190:32)
at ChildProcess.<anonymous> (/usr/src/app/node_modules/python-shell/index.js:182:13)
at ChildProcess.emit (node:events:537:28)
at ChildProcess._handle.onexit (node:internal/child_process:291:12)
----- Python Traceback -----
File "/usr/src/app/public/pythonscripts/myPython/wtf.py", line 6, in <module>
import pyecma376_2 # The base library for Open Packaging Specifications. We will use the OPCCoreProperties class. {
traceback: 'Traceback (most recent call last):\n' +
' File "/usr/src/app/public/pythonscripts/myPython/wtf.py", line 6, in <module>\n' +
' import pyecma376_2 # The base library for Open Packaging Specifications. We will use the OPCCoreProperties class.\n' +
"ModuleNotFoundError: No module named 'pyecma376_2'\n",
executable: 'python3',
options: null,
script: 'public/pythonscripts/myPython/wtf.py',
args: null,
exitCode: 1
}
If I comment the first three imports out, I get the same error:
PythonShellError: ModuleNotFoundError: No module named 'assi'
Please notice, that assi actually is from my own python-code, which is included in the expressjs-app-directory
Python seems to be installed in the container correctly. I stepped inside the container via docker exec -it <container id> /bin/bash and there are the python packages in the #/usr/lib-directory.
I really have absolute no idea how all this works together and why python doesn't find this modules...

You are trying to use libs that are not in Standard Python Library. It seems that you are missing to run pip install , when you build the docker images.
Try adding RUN docker commands that can do this for you. Example:
RUN pip3 install pyecma376_2
RUN pip3 install /path/to/assi
Maybe, that can solve your problem. Don't forget to check if python are already installed in your container, it semms that it is. And if you have python2 and pyhton3 installed, make sure that you use pip3 instead of only pip.

Run a python script in node-red running on Docker

I'm trying to run a python script saved on my local system in node-red which is running as a docker container. I copied the python script into the docker container as the exec node was unable to locate the file using this command -
cat /local/file/path | docker exec -i <running-container-id> sh -c 'cat > /inside/docker/file/path'
But now I'm getting the following error - Traceback (most recent call last):
File "outlier.py", line 2, in
from pandas import read_csv
ModuleNotFoundError: No module named 'pandas'
I had installed pandas on my local but it's not being found by the exec node. Any help is appreciated, thanks.

When applications run inside a Docker container they only have access to the libraries/modules included inside the container. They have no access to anything in the host machine.
So if you want to run Python scripts that have dependencies on Python modules you will need to create a custom Docker container that extends the official Node-RED container and then installs those modules.
Node-RED provides doc about extending it's container here

Execute python script at launch time on Amazon Linux 2

I am trying to execute a python script on an Amazon Linux 2 instance. In my user-data section I have a script which copies the python script from an S3 bucket to the instance and executes it like so:
#!/bin/bash
# e - stops the script if there is an error
# x - output every command in /var/log/syslog
set -e -x
# set AWS region
echo "export AWS_DEFAULT_REGION=us-east-1" >> /etc/profile
source /etc/profile
# copy python script from the s3 bucket
aws s3 cp s3://${bucket_name}/ /home/ec2-user --recursive
sudo python3 my_python_script.py
The problem is that the python script doesn't seem to be getting executed at all.
Note: the python script gets copied fine from the bucket
What I am missing here?
UPDATE:
after checking /var/log/cloud-init-output.log it looks like the problem is in the python script, it cannot find the boto3 module:
+ python3 /home/ec2-user/my_python_script.py
Traceback (most recent call last):
File "/home/ec2-user/my_python_script.py", line 1, in <module>
import boto3
ModuleNotFoundError: No module named 'boto3'
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Dec 10 15:52:25 cloud-init[3697]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
The problem is that I do have boto3 module installed. I created a custom AMI image that does have all of the modules installed (I used pip3 to install them) before creating the custom AMI image
UPDATE2
I verified that the image does have boto3 package installed in the python3 library:
[ec2-user#ip-ip ~]$ python3
Python 3.7.9 (default, Aug 27 2020, 21:59:41)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>>
UPDATE3
The cause of the problem was that I installed the boto3 package for my user only (i.e. pip3 install boto3 --user) and then I created the AMI image. So after adding the bellow line to my user-data script it worked fine
#!/bin/bash
...
sudo pip3 install boto3
sudo python3 my_python_script.py

You can redirect output to a file and read it to see the error: did you have python3, did your instance have credentianl/role to access this bucket, did you script requires any third party, can you try to run the script above as root in local first, the run command should be python3 /home/ec2-user/my_python_script.py?

For anyone who is using cloudinit and terraform, this is how I managed it to work (one part of multipart cloudinit user data)-
part {
content_type = "text/x-shellscript"
filename = "run_src.sh"
content = <<-EOF
#!/bin/bash
cd /root
mkdir tmp
mkdir tmp/origin
mkdir tmp/converted
mkdir tmp/packaged
pip3 install boto3
cd src
python3 main.py
EOF
}
And it works like charm.

made a shell to run script on startup, suddently it gives me an importerror

I followed this guide: guide to create a startupfile which excecutes a python file on startup.
in step 2 it says I have to test the startupfile I just created and suddently my script says:
Traceback (most recent call last):
File "Display.py", line 1, in <module>
import pyowm
ImportError: No module named pyowm
the python file works perfect if I run it directly.
what I allready tried: run pip again to see if the lib was okay
check the /usr/local/lib/python3.4/dist-packages folder to see if it was there and it is.
I think this is a python issue and not a RaspberryPi issue thats why I uploaded it here.
runned by:
sh launcher.sh
inside is:
#!/bin/sh
# launcher.sh
# navigate to home directory, then to this directory, then execute python script, then back home
cd /
cd /home/pi/arduino/Python/Main/Master
sudo python Display.py
cd /

Simple fix: define the version of python which will be used. It used python 2.7. Yet the lib was for 3.4.

no json module when running python script in docker container

I'm trying to run a simple python script that takes a json string as an argument in a docker container. However I get the following error:
Traceback (most recent call last):
File "/root/simple.py", line 2, in <module>
import json
ImportError: No module named json
I'm running the standard ubuntu:12.04 image. Here's how I call up the container:
docker run -v $(pwd)/:/root/ ubuntu:12.04 python /root/simple.py '[{"hi":"bye"}]'
My simple.py script is just:
import sys
import json
configs = json.loads(sys.argv[1])
print configs
def read_option_keys(json_file):
json_file[0]["new"] = None
print json.dumps(json_file)
read_option_keys(configs)
Any idea why it's not returning the following as expected:
[{u'hi': u'bye'}]
[{"hi": "bye", "new": null}]

I was able to solve the issue myself. Ubuntu image is super bare-bones. I pulled the dockerfile/python image and now it works.

Docker is a containerized instance, so your python script has no knowledge of the outside world. You are currently running the Ubuntu image, but it doesn't contain the python standard module. You should run a python image instead or add python to your Ubuntu image