We Keep Coding

How to run Python inside an expressjs Docker container

i am trying to build a container for my express.js application. The express.js-app makes use of python via the npm package PythonShell.
I have plenty of python-code, which is in a subfolder of my express-app and with npm start everything works perfectly.
However, i am new to docker and i need to containerize the app. My Dockerfile looks like this:
FROM node:18
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3001
CMD ["node", "./bin/www"]
I built the Image with:
docker build . -t blahblah-server and ran it with docker run -p 8080:3001 -d blahblah-server.
I make use of imports at the top of the python-script like this:
import datetime
from pathlib import Path # Used for easier handling of auxiliary file's local path
import pyecma376_2 # The base library for Open Packaging Specifications. We will use the OPCCoreProperties class.
from assi import model
When the pythonscript is executed (only in the container!!!) I get following error-message:
/usr/src/app/public/javascripts/service/pythonService.js:12
if (err) throw err;
^
PythonShellError: ModuleNotFoundError: No module named 'pyecma376_2'
at PythonShell.parseError (/usr/src/app/node_modules/python-shell/index.js:295:21)
at terminateIfNeeded (/usr/src/app/node_modules/python-shell/index.js:190:32)
at ChildProcess.<anonymous> (/usr/src/app/node_modules/python-shell/index.js:182:13)
at ChildProcess.emit (node:events:537:28)
at ChildProcess._handle.onexit (node:internal/child_process:291:12)
----- Python Traceback -----
File "/usr/src/app/public/pythonscripts/myPython/wtf.py", line 6, in <module>
import pyecma376_2 # The base library for Open Packaging Specifications. We will use the OPCCoreProperties class. {
traceback: 'Traceback (most recent call last):\n' +
' File "/usr/src/app/public/pythonscripts/myPython/wtf.py", line 6, in <module>\n' +
' import pyecma376_2 # The base library for Open Packaging Specifications. We will use the OPCCoreProperties class.\n' +
"ModuleNotFoundError: No module named 'pyecma376_2'\n",
executable: 'python3',
options: null,
script: 'public/pythonscripts/myPython/wtf.py',
args: null,
exitCode: 1
}
If I comment the first three imports out, I get the same error:
PythonShellError: ModuleNotFoundError: No module named 'assi'
Please notice, that assi actually is from my own python-code, which is included in the expressjs-app-directory
Python seems to be installed in the container correctly. I stepped inside the container via docker exec -it <container id> /bin/bash and there are the python packages in the #/usr/lib-directory.
I really have absolute no idea how all this works together and why python doesn't find this modules...

You are trying to use libs that are not in Standard Python Library. It seems that you are missing to run pip install , when you build the docker images.
Try adding RUN docker commands that can do this for you. Example:
RUN pip3 install pyecma376_2
RUN pip3 install /path/to/assi
Maybe, that can solve your problem. Don't forget to check if python are already installed in your container, it semms that it is. And if you have python2 and pyhton3 installed, make sure that you use pip3 instead of only pip.

numpy.f2py - extension doesn't exist when source distribution is installed with pip

I'm trying to wrap my head around f2py because my organization has a lot of legacy fortran code that I would like to incorporate into some newer python-based tools I'm writing. Ideally, I would package these tools either in source packages or wheels to make it easier to distribute to the rest of the organization.
I've written a small test package based on some other examples I've seen that just sums an array of floats. The package contents are included below. If I build a source distribution tarball using py setup.py sdist, everything looks like it works. It even looks like pip successfully installs it. However, if I open a python shell and try to import the newly installed module, I get an error on the from fastadd import fadd line in the initialization script saying
AttributeError: module 'fastadd' has no attribute 'fastadd'
So it seems like it didn't actually successfully build the f2py module. Doing some troubleshooting, if I open a powershell window in the package folder and just run
py -m numpy.f2py -c fadd.pyf fadd.f90
and then open a python shell in the same folder and try to import fastadd, I get an error, ImportError: DLL load failed: The specified module could not be found. (This is after I installed the Visual Studio build tools, a fix suggested on several threads). Following the advice on this thread, changing the command to
py -m numpy.f2py -c --fcompiler=gnu95 --compiler=mingw32 fadd.pyf fadd.f90
will build a module file that I can successfully import and use. Okay, great.
However, when I change config.add_extension in the setup file to include the keyword argument f2py_options=["--fcompiler=gnu95","--compiler=mingw32"] and try to build a package distribution file with setup.py sdist command and then install using py -m pip install fastadd-1.0a1.tar.gz, I get yet a different error that says
ERROR: No .egg-info directory found in C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-c7406k03
And now I'm completely flummoxed. Other configurations of the f2py_options either result in setup.py throwing an error or fail to create the extension altogether, similar to above. Using a simple string for the options gives an error, so apparently f2py_options does in fact expect a list input. I can't seem to find any good documentation on whether I'm using f2py_options correctly, and I have no idea why just adding that option would cause pip to not know where its info directory is. That makes no sense to me. I'd really appreciate some help on this one.
I'm running Python 3.7.0 32-bit, numpy 1.20.1, and pip 21.0.1 on a Windows 10 machine.
--EDIT--
Looking in the installation directory of the test module, I found a new wrinkle to this problem: the installation directory does not actually include any files listed in MANIFEST, not even the __init__.py file. If I copy __init__.py into the directory, trying to import the module gives the same ImportError: DLL load failed error I've been getting.
Also, inspecting the output of py -m pip install, it looks like numpy.distutils doesn't recognize --fcompiler or --compiler as valid options and just ignores them, even though numpy.f2py does recognize them.
--END EDIT--
PACKAGE CONTENTS:
+-fastadd
---__init__.py
---fadd.f90
---fadd.pyf
-MANIFEST.in
-README
-setup.py
fadd.f90 has the following contents:
subroutine fadd(vals,n,mysum)
integer, intent(in) :: n
real*8, intent(out):: mysum
real*8, dimension(n), intent(in) :: vals
mysum = sum(vals)
end subroutine fadd
fadd.pyf has the following contents:
python module fastadd ! in
interface ! in :fastadd
subroutine fadd(vals,n,mysum) ! in :fastadd:fadd.f90
real*8 dimension(n),intent(in) :: vals
integer, optional,intent(in),check(len(vals)>=n),depend(vals) :: n=len(vals)
real*8 intent(out) :: mysum
end subroutine fadd
end interface
end python module fastadd
__init__.py:
"""This is the documentation!"""
from .fastadd import fadd
MANIFEST.in:
include README
recursive-include fastadd *.f90
recursive-include fastadd *.pyf
recursive-include fastadd *.py
and, finally, setup.py:
def configuration(pth=None):
from numpy.distutils.misc_util import Configuration
config = Configuration(
'fastadd',
top_path=pth,
version='1.0a1',
author='John Doe',
author_email='john.doe#fake-org.biz',
url='fake-org.biz/fastadd',
description="Testing f2py build process. Sums an arbitrary-length list of numbers.")
config.add_extension(
'fastadd',
sources=['fastadd\\fadd.pyf','fastadd\\fadd.f90']
)
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration('fastadd').todict())
If it helps at all, the final MANIFEST file looks like this after the setup script is run:
# file GENERATED by distutils, do NOT edit
README
setup.py
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.f90
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.pyf
fastadd\__init__.py
fastadd\fadd.f90
fastadd\fadd.pyf

ModuleNotFoundError: No module named 'object_detection'

i try to train.py in object_detection in under git url
https://github.com/tensorflow/models/tree/master/research/object_detection
However, the following error occurs.
ModuleNotFoundError: No module named 'object_detection'
So I tried to solve the problem by writing the following code.
import sys
sys.path.append('/home/user/Documents/imgmlreport/inception/models/research/object_detection')
from object_detection.builders import dataset_builder
This problem has not been solved yet.
The directory structure is shown below.
~/object_detection/train.py
~/object_detection/builders/dataset_bulider.py
and here is full error massage
/home/user/anaconda3/lib/python3.6/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated.
In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "train.py", line 52, in
import trainer
File"/home/user/Documents/imgmlreport/inception/models/research/object_detection/trainer.py", line 26, in
from object_detection.builders import optimizer_builder
ModuleNotFoundError: No module named 'object_detection'
how can i import modules?

Try install Tensorflow Object Detection Library Packaged
pip install tensorflow-object-detection-api

Cause of this error is installing object_detection library, So one of the solution which can work is running the below command inside models/research
sudo python setup.py install
If such solution does not work, please execute the below command one by one in the directory models/research
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
sudo python setup.py install
I hope this will work. I also faced the same problem while creating model from export_inference_graph.py. It worked for me.

You need to export the environmental variables every time you open a new terminal in that environment.
Please note that there are are back quotes on each of the pwd in the command as this might not be showing in the command below. Back quote is the same as the tilde key without pressing the shift key (US keyboard).
From tensorflow/models/research/
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

try this:
python setup.py build
python setup.py install

There are a number of modules in the object_detection folder, and I have created setup.py in the parent directory(research folder) to import all of them.
from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['Pillow>=1.0', 'Matplotlib>=2.1', 'Cython>=0.28.1']
setup(
name='object_detection',
version='0.1',
install_requires=REQUIRED_PACKAGES,
include_package_data=True,
packages=[p for p in find_packages() if p.startswith('object_detection')],
description='Tensorflow Object Detection Library',
)

You did have "sys.path.append()" before you imported the object detection, so I am surprised that you are facing this error!
Please check that the path you have used in sys.path.append() is right.
Well, the only and obvious answer for the error is that the path of the module is not added properly.
Besides the various ways mentioned here, here is a way in which you can add the "object_detection" path permanently to the PYTHONPATH variable.
If you are using a Linux system, here is how you would go about it:
Go to the Home directory. Press Ctrl + H to show hidden files. You will see a file called ".bashrc". Open this file using a code editor (I used Visual Studio).
In the last line of .bashrc file, add the line:
export PYTHONPATH=/your/module/path:/your/other/module/path:your/someother/module/path
Then press "save" in the code editor. Since ".bashrc" is a "Read-only" file the editor will throw a pop-up saying the same. Also in the pop-up there will be an option that says: "Try with sudo". Hit this button and now you are good to go.
All your modules are now permanently added to the PYTHONPATH. This means that you need not run sys.path.append every time you open your terminal and start a session!
Below is the screenshot with no error when I followed the said steps:
Try this. I hope it helps.

And finally, If you've followed all the steps here and are at your wit's end...make sure the file that you're running (the one with your source code in it ya know), isn't named object_detection.py - that would preclude it being searched for as a module.
Certainly I've never done anything like this that led me to add an embarrassing answer on Stack Overflow...

I had to do:
sudo pip3 install -e . (ref)
sudo python3 setup.py install
System:
OS: Ubuntu 16.04, Anaconda (I guess this is why I need to use pip3 and python3 even I made virtual environment with Pyehon 3.8)

How to package a linked DLL and a pyd file into one self contained pyd file?

I am building a python module with Cython that links against a DLL file. In order to succesfully import my module I need to have the DLL in the Windows search path. Otherwise, the typical error message is:
ImportError: DLL load failed: The specified module could not be found.
Is there a way to packaged the DLL directly into the produced pyd file to make the distribution easier?
One example of this is with the OpenCV distribution, where a (huge) pyd file is distributed and is the only file needed for the Python bindings to work.

Python's packaging & deployment is still a pain point for many of us. There is just not a silver bullet. Here are several methods:
1. OpenCV build method
The method is decribed here : https://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_bindings/py_bindings_basics/py_bindings_basics.html#bindings-basics
OpenCV generates these wrapper functions automatically from the C++
headers using some Python scripts which are located in
modules/python/src2.
Basically it parse the header files and generate the static PyObject keywords whenever it's needed. Once the header are created appropriately, it just call python setup. Honestly, it might work, but I would not advise this method.
2. Makefiles
If you already use Makefile, just create a rule to place your lib accordinlgy. Example, from my own code :
setup.py
from distutils.core import setup, Extension
setup(name='sha1_hmac', version='1.0', \
ext_modules=[Extension('sha1_hmac',
library_dirs=['C:\MinGW\lib'],
sources= ['../tools/sha1.c','sha1_hmac.c'])])
Makefile
# The hmac generation used by the webserver is done
# using the sha1.c implementation. There is a binding needed to
# glue the C code with the python script
libsha1_hmac:
ifeq ($(OS), Windows_NT)
$(PYTHON) setup.py build --compiler=mingw32
else
$(PYTHON) setup.py install --home=$(CURDIR)
endif
.PHONY: webserver
webserver: libsha1_hmac
ifeq ($(OS), Windows_NT)
mv $(shell find build -type f -name "sha1*.pyd") $(LIB)
else
mv -f $(shell find $(LIB)/python -type f -name "sha1*.so") $(LIB)
endif
$(PYTHON) hmac_server.py
3. Modern deployement tools
There are several new tools to deploy python applications, namely wheels which seem to gain traction. I don't use it, but it look like it can ease up your bundling problem :
How can I make a Python Wheel from an existing native library?
Once it wheeled, you can install it like this : pip install some-package.whl

Cabal installation error missingpy on Haskell Platform 2010.2.0.0 on Windows 7

I am a newbie Haskell-user and want to use Python libraries (for interaction with existing software for example using Numpy libraries). I found out that this can be done using the MissingPy package ( http://hackage.haskell.org/package/MissingPy ), however I am not able to install it via cabal on Windows 7 on the Haskell platform 2010.2.0.0.
The first error I had, was because the compiler didn't find the python2.6-libraries. This was solved by putting python2.6.dll in the correct directory (or adding it to the system path variable). However, I am unable to fix the following error at the end of the output of the "cabal install missingpy"-command:
...
[11 of 12] Compiling MissingPy.FileArchive.GZip ( MissingPy\FileArchive\GZip.hs, dist\build\MissingPy\FileArchive\GZip.p_o )
[12 of 12] Compiling MissingPy.FileArchive.BZip2 ( MissingPy\FileArchive\BZip2.hs, dist\build\MissingPy\FileArchive\BZip2.p_o )
Registering MissingPy-0.10.5...
setup.exe: MissingPy-0.10.5: library-dirs: None doesn't exist or isn't a
directory (use --force to override)
cabal: Error: some packages failed to install:
MissingPy-0.10.5 failed during the building phase. The exception was:
ExitFailure 1
I installed the full MinGW-software after reading the suggestions on the answers on the question asked on Using c2hs on Windows . However, this did not solve the issue.
Anyone having experience and solutions on this issue?

Note that the Setup.exe populates the build info using a python tool:
https://github.com/softmechanics/missingpy/blob/master/Setup.hs
The relevant code is here:
libDir <- python ["-c", "from distutils.sysconfig import *; print get_python_lib()"]
incDir <- python ["-c", "from distutils.sysconfig import *; print get_python_inc()"]
confLibDir <- python ["-c", "from distutils.sysconfig import *; print get_config_var('LIBDIR')"]
pyVersionStr <- python ["-c", "import sys; sys.stdout.write(\"%d.%d\" % (sys.version_info[0], sys.version_info[1]))"]
If the python tool doesn't exist, can' be found, or gives bad information, the setup will fail.