We Keep Coding

Why can't I deploy Python 3.9 app with zappa 0.55

I am not too familiar with AWS and Python in general so I am struggling with this issue.
I am trying to upgrade an app from Python 3.6 to 3.9 since AWS has stopped supporting 3.6
The error I get is: ModuleNotFoundError: No module named 'zappa'
Full output:
on blue-4.shared.runners-manager.gitlab.com/default J2nyww-s
Preparing the "docker+machine" executor
00:32
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:850f8694d221b7fba2358381b744c231cbe9096d3f3c888493f1889eccec25fb for python:3.9 with digest python#sha256:c1613835d7be322f98603f356b9e0c9d40f9589e94dc9f710e714a807a665700 ...
Preparing environment
00:03
Running on runner-j2nyww-s-project-2141508-concurrent-0 via runner-j2nyww-s-shared-1669887795-c20ec238...
Getting source from Git repository
00:01
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes...
Initialized empty Git repository in /builds/example/example/.git/
Created fresh repository.
Checking out d876fc09 as next...
Skipping Git submodules setup
Restoring cache
00:07
Checking cache for next-22-protected...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/2141508/next-22-protected
WARNING: backend/ve/bin/python: chmod backend/ve/bin/python: no such file or directory (suppressing repeats)
Successfully extracted cache
Executing "step_script" stage of the job script
00:07
Using docker image sha256:850f8694d221b7fba2358381b744c231cbe9096d3f3c888493f1889eccec25fb for python:3.9 with digest python#sha256:c1613835d7be322f98603f356b9e0c9d40f9589e94dc9f710e714a807a665700 ...
$ python -V
Python 3.9.15
$ cd backend
$ python3 -m pip install --user --upgrade pip
Requirement already satisfied: pip in /usr/local/lib/python3.9/site-packages (22.0.4)
Collecting pip
Downloading pip-22.3.1-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 29.0 MB/s eta 0:00:00
Installing collected packages: pip
WARNING: The scripts pip, pip3, pip3.10 and pip3.9 are installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-22.3.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python3 -m pip install --upgrade pip' command.
$ pip3.9 install virtualenv
Collecting virtualenv
Downloading virtualenv-20.17.0-py3-none-any.whl (8.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 35.4 MB/s eta 0:00:00
Collecting distlib<1,>=0.3.6
Downloading distlib-0.3.6-py2.py3-none-any.whl (468 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.5/468.5 kB 48.0 MB/s eta 0:00:00
Collecting filelock<4,>=3.4.1
Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Collecting platformdirs<3,>=2.4
Downloading platformdirs-2.5.4-py3-none-any.whl (14 kB)
Installing collected packages: distlib, platformdirs, filelock, virtualenv
Successfully installed distlib-0.3.6 filelock-3.8.0 platformdirs-2.5.4 virtualenv-20.17.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
$ source ve/bin/activate
$ zappa update dev
Traceback (most recent call last):
File "/builds/example/example/backend/ve/bin/zappa", line 5, in <module>
from zappa.cli import handle
ModuleNotFoundError: No module named 'zappa'
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
And my zappa config:
"domain": "devapi.example.com",
"django_settings": "config.dev",
"profile_name": null,
"project_name": "example-django-backend",
"runtime": "python3.9",
"s3_bucket": "example-zappa-app-dev",
"route53_enabled": false,
"cors": true,
"timeout_seconds": 300,
"events": [],
}
From .gitlab-ci.yml:
backend build:
stage: build
before_script:
- python -V
- cd backend
- pip install virtualenv
- python3 -m pip install --user --upgrade pip
script:
- if test -d ve; then echo "Virtual environment exist in cache..."; else virtualenv ve; fi
- source ve/bin/activate
- pip3.9 install -Ur requirements.txt
only:
refs:
- next
- master
- tags
except:
refs:
- schedules
backend deploy dev:
stage: deploy
environment:
name: dev
url: https://devapi.example.life
before_script:
- python -V
- cd backend
- python3 -m pip install --user --upgrade pip
- pip3.9 install virtualenv
script:
- source ve/bin/activate
- zappa update dev
- zappa manage dev "migrate"
- ./manage.py update_lambda_env example-django-backend-dev --path /app/dev/ --path /app/common/
- zappa manage dev "collectstatic --no-input"
- zappa manage dev "algolia_reindex"
only:
refs:
- next
except:
refs:
- schedules
requirements.txt:
Django==2.1.7
uwsgi
django-filter==2.0
djangorestframework==3.8.2
Markdown==2.6.11
psycopg2==2.9.5
Pillow==8.1.1
djangorestframework-jwt==1.11.0
django-cors-headers==1.2.2
django-extensions==2.0.7
django-multi-email-field==0.6.2
pyuploadcare==2.6.0
redis==2.10.5
git+https://github.com/erback/django-autofixture.git#master
django-simple-history==2.1.0
ipython==6.2.0
graphene-django==2.2.0
algoliasearch==1.20
algoliasearch-django==1.7.1
python-postmark==0.5.0
django-ipware==2.1.0
django-object-actions==1.0.0
django-model-utils==3.1.2
model-mommy==1.6.0
sentry-sdk==1.1.0
django-s3-storage==0.13.4
zappa==0.55.0
python-intercom==3.1.0
choicesenum==0.5.3
xhtml2pdf==0.2.8
Any suggestions?

pip3 offline installator complains on "no matching distribution" even when it is present

I have to prepare installation of a python3 FastAPI based service to a server without an internet conenction.
I installed all needed stuff in a minimal Debian container, tested the service and called
pip freeze > requirements.txt
I got:
asgiref==3.4.1
certifi==2020.6.20
chardet==4.0.0
click==8.0.1
fastapi==0.68.0
fastapi-utils==0.2.1
greenlet==1.1.1
h11==0.12.0
httptools==0.2.0
idna==2.10
iso8601==0.1.16
m3u8==0.9.0
pydantic==1.8.2
python-dotenv==0.19.0
PyYAML==5.4.1
requests==2.25.1
six==1.16.0
SQLAlchemy==1.4.23
starlette==0.14.2
typing-extensions==3.10.0.0
urllib3==1.26.5
uvicorn==0.15.0
uvloop==0.16.0
watchgod==0.7
websockets==9.1
Then I called on my host
mkdir dependencies
pip download -r requirements.txt -d "./dependencies"
cp requirements.txt ./dependencies/
tar cvfz dependencies.tar.gz dependencies
The approach is based on these SO questions and answers:
installing python packages without internet and using source code as .tar.gz and .whl
How to install packages offline?
I created a new fresh Debian container with access to archive made above, installed python3 and python3-pip, disconnected my host from internet and tried this:
root#3eed3ed8cafc:~/temp# pip3 install --no-index --find-links="./dependencies/" -r dependencies/requirements.txt
Looking in links: ./dependencies/
Processing ./dependencies/asgiref-3.4.1-py3-none-any.whl
Processing ./dependencies/certifi-2020.6.20-py2.py3-none-any.whl
Processing ./dependencies/chardet-4.0.0-py2.py3-none-any.whl
Processing ./dependencies/click-8.0.1-py3-none-any.whl
Processing ./dependencies/fastapi-0.68.0-py3-none-any.whl
Processing ./dependencies/fastapi_utils-0.2.1-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement greenlet==1.1.1
ERROR: No matching distribution found for greenlet==1.1.1
But it is there as greenlet-1.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl wheel:
root#3eed3ed8cafc:~/temp# ls dependencies
PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl m3u8-0.9.0-py3-none-any.whl
SQLAlchemy-1.4.23-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl pydantic-1.8.2-cp38-cp38-manylinux2014_x86_64.whl
asgiref-3.4.1-py3-none-any.whl python_dotenv-0.19.0-py2.py3-none-any.whl
certifi-2020.6.20-py2.py3-none-any.whl requests-2.25.1-py2.py3-none-any.whl
chardet-4.0.0-py2.py3-none-any.whl requirements.txt
click-8.0.1-py3-none-any.whl six-1.16.0-py2.py3-none-any.whl
fastapi-0.68.0-py3-none-any.whl starlette-0.14.2-py3-none-any.whl
fastapi_utils-0.2.1-py3-none-any.whl typing_extensions-3.10.0.0-py3-none-any.whl
greenlet-1.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl urllib3-1.26.5-py2.py3-none-any.whl
h11-0.12.0-py3-none-any.whl uvicorn-0.15.0-py3-none-any.whl
httptools-0.2.0-cp38-cp38-manylinux1_x86_64.whl uvloop-0.16.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
idna-2.10-py2.py3-none-any.whl watchgod-0.7-py3-none-any.whl
iso8601-0.1.16-py2.py3-none-any.whl websockets-9.1-cp38-cp38-manylinux2010_x86_64.whl
Not even moving the greenlet to the first line in requirements.txt helped.
What is wrong?

getting python and react component based container to work

I am attempting to dockerize this workflow of isomorphic app.
I build the container of below docker file.
FROM python:3.5-slim
RUN apt-get update && \
apt-get -y install gcc mono-mcs && \
apt-get -y install vim && \
apt-get -y install nano && \
rm -rf /var/lib/apt/lists/*
RUN mkdir -p /statics/js
VOLUME ["/statics/"]
WORKDIR /statics/js
COPY requirements.txt /opt/requirements.txt
RUN pip install -r /opt/requirements.txt
EXPOSE 8080
CMD ["python", "/statics/js/app.py"]
and this was the result:
$ docker build -t ciasto/pythonreact:v2 .
Sending build context to Docker daemon 1.327 MB
Step 1/9 : FROM python:3.5-slim
---> b27a94c44674
Step 2/9 : RUN apt-get update && apt-get -y install gcc mono-mcs && apt-get -y install vim && apt-get -y install nano && rm -rf /var/lib/apt/lists/*
---> Using cache
---> c76cb348707c
Step 3/9 : RUN mkdir -p /statics/js
---> Using cache
---> 2ef5b24f551c
Step 4/9 : VOLUME /statics/
---> Using cache
---> 5e62c6af1867
Step 5/9 : WORKDIR /statics/js
---> Using cache
---> a5a018e8c727
Step 6/9 : COPY requirements.txt /opt/requirements.txt
---> Using cache
---> 1fa4dccc6608
Step 7/9 : RUN pip install -r /opt/requirements.txt
---> Running in 8845a0efcee7
Collecting TurboGears2==2.3.10 (from -r /opt/requirements.txt (line 1))
Downloading TurboGears2-2.3.10.tar.gz (176kB)
Collecting Kajiki==0.6.3 (from -r /opt/requirements.txt (line 2))
Downloading Kajiki-0.6.3.tar.gz (174kB)
Collecting tgext.webassets==0.0.2 (from -r /opt/requirements.txt (line 3))
Downloading tgext.webassets-0.0.2.tar.gz
Collecting dukpy==0.1.0 (from -r /opt/requirements.txt (line 4))
Downloading dukpy-0.1.0.tar.gz (2.0MB)
Collecting WebOb>=1.2 (from TurboGears2==2.3.10->-r /opt/requirements.txt (line 1))
Downloading WebOb-1.7.2-py2.py3-none-any.whl (83kB)
Collecting crank<0.9.0,>=0.8.0 (from TurboGears2==2.3.10->-r /opt/requirements.txt (line 1))
Downloading crank-0.8.1.tar.gz
Collecting repoze.lru (from TurboGears2==2.3.10->-r /opt/requirements.txt (line 1))
Downloading repoze.lru-0.6.tar.gz
Collecting MarkupSafe (from TurboGears2==2.3.10->-r /opt/requirements.txt (line 1))
Downloading MarkupSafe-1.0.tar.gz
Collecting nine (from Kajiki==0.6.3->-r /opt/requirements.txt (line 2))
Downloading nine-1.0.0-py2.py3-none-any.whl
Collecting webassets (from tgext.webassets==0.0.2->-r /opt/requirements.txt (line 3))
Downloading webassets-0.12.1.tar.gz (179kB)
Collecting cssmin (from tgext.webassets==0.0.2->-r /opt/requirements.txt (line 3))
Downloading cssmin-0.2.0.tar.gz
Building wheels for collected packages: TurboGears2, Kajiki, tgext.webassets, dukpy, crank, repoze.lru, MarkupSafe, webassets, cssmin
Running setup.py bdist_wheel for TurboGears2: started
Running setup.py bdist_wheel for TurboGears2: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/51/1d/bb/c9cfdcf2a49f71955d5b66aed0dbd187e58e5d77a9fa34a4af
Running setup.py bdist_wheel for Kajiki: started
Running setup.py bdist_wheel for Kajiki: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/ad/fe/15/33e02c73fead4ea9238fcd31d273accf6fb9d922ec901e20c8
Running setup.py bdist_wheel for tgext.webassets: started
Running setup.py bdist_wheel for tgext.webassets: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/00/f2/09/0378f24bd9151b7a927093546c11685899ebec451b65eb181f
Running setup.py bdist_wheel for dukpy: started
Running setup.py bdist_wheel for dukpy: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/21/29/46/34c303b9dca370a8ccc97a84b094c8089b78edde125b0a1fcb
Running setup.py bdist_wheel for crank: started
Running setup.py bdist_wheel for crank: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/1c/00/54/4dcfd62d8268d7b34ea607bd9f8cb12aa930a7718c8c5fbc02
Running setup.py bdist_wheel for repoze.lru: started
Running setup.py bdist_wheel for repoze.lru: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/b2/cd/b3/7e24400bff83325a01d492940eff6e9579f553f33348323d79
Running setup.py bdist_wheel for MarkupSafe: started
Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57
Running setup.py bdist_wheel for webassets: started
Running setup.py bdist_wheel for webassets: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/9d/cb/c2/340b9b695822b6954840bcb6cd147b3a7cfc2bcd922296e63e
Running setup.py bdist_wheel for cssmin: started
Running setup.py bdist_wheel for cssmin: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/c3/79/88/647f59be446af4e9867362ca6e961cc7f218bd793fbdc351a6
Successfully built TurboGears2 Kajiki tgext.webassets dukpy crank repoze.lru MarkupSafe webassets cssmin
Installing collected packages: WebOb, crank, repoze.lru, MarkupSafe, TurboGears2, nine, Kajiki, webassets, cssmin, tgext.webassets, dukpy
Successfully installed Kajiki-0.6.3 MarkupSafe-1.0 TurboGears2-2.3.10 WebOb-1.7.2 crank-0.8.1 cssmin-0.2.0 dukpy-0.1.0 nine-1.0.0 repoze.lru-0.6 tgext.webassets-0.0.2 webassets-0.12.1
---> 86c189792ae7
Removing intermediate container 8845a0efcee7
Step 8/9 : EXPOSE 8080
---> Running in 9243a87c36e2
---> e7d35d54e66d
Removing intermediate container 9243a87c36e2
Step 9/9 : CMD python /statics/js/app.py
---> Running in 6e3b53cd901d
---> 0d79c4f81f3b
Removing intermediate container 6e3b53cd901d
Successfully built 0d79c4f81f3b
So my first question is what does the step 9 means ? does that mean it is attempting to run the /statics/js/app.py path even before I run the container because that would not work as I planned to mount this statics volume from the host.
Secondly, if I run the command:
$ docker run -it -v ~/Development/my-Docker-builds/pythonReact/statics/:/statics/ -d ciasto/pythonreact:v2
03d77c87651e752450e3be0aa64a0841c088b32a1db5424ad96c150c949d0366
I get the hash key but nothing works ! I do not even see the start trace or error message of app.py had any error.
So how should I run the app.py from host mounted volume when I run the container ?

You can consider CMD as the startup script for the container. Having said that at step 9 it is only marking pyhton /statics/js/app.py to be executed whenever you start the container. Also, since you are using -d flag you won't be able to see logs so you have to fetch them using docker logs command as:
docker logs 03d77c87651e752450e3be0aa64a0841c088b32a1db5424ad96c150c949d0366
The logs should be enough to help you figure out the issue. I hope it helps.

Python using requirements.txt

Im doing Google Ap Engine application,using Flask
when im add lib in requirements.txt my applcication not deploying
content requirements.txt
Flask==0.10.1
gunicorn==19.4.5
google-api-python-client==1.5.0
oauth2client==2.0.1
pandas==0.18.0
end its return error
ERROR: (gcloud.preview.app.deploy) Docker build aborted: The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 143
if im remove "google-api-python-client",error disappear

Why is my module failing to import when I try to run Travis-CI tests?

I'm trying to run Travis-CI on a module I've made. Nosetests pass on my local machine with flying colors, but for some reason the test.py file is failing to import my project. Below is the full output from Travis-CI:
Using worker: ppp3.worker.travis-ci.org:php-4
$ cd ~/builds
$ git clone --branch=master --depth=100 --quiet git://github.com/louist87/Scrappy.git louist87/Scrappy
$ cd louist87/Scrappy
$ git checkout -qf 68d1291c8a81638554d036aa01215632a6661623
$ source ~/virtualenv/python2.7/bin/activate
$ python --version
Python 2.7.3
$ pip --version
pip 1.2.1 from /home/travis/virtualenv/python2.7/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg (python 2.7)
$ pip install -r requirements.txt --use-mirrors
Downloading/unpacking guessit>=0.5.2 (from -r requirements.txt (line 1))
Downloading guessit-0.5.3.tar.gz (45kB): 45kB downloaded
Running setup.py egg_info for package guessit
Downloading/unpacking tvdb-api>=1.8.2 (from -r requirements.txt (line 2))
Downloading tvdb_api-1.8.2.tar.gz
Running setup.py egg_info for package tvdb-api
Downloading/unpacking hachoir-metadata>=1.3.3 (from -r requirements.txt (line 3))
Downloading hachoir-metadata-1.3.3.tar.gz (52kB): 52kB downloaded
Running setup.py egg_info for package hachoir-metadata
Warning: unable to recompile dialog.ui to dialog_ui.py using pyuic4
(use command "-c egg_info --egg-base pip-egg-info --disable-qt" to disable this warning)
Downloading/unpacking hachoir-core>=1.3.3 (from -r requirements.txt (line 4))
Downloading hachoir-core-1.3.3.tar.gz (91kB): 91kB downloaded
Running setup.py egg_info for package hachoir-core
Downloading/unpacking hachoir-parser>=1.3.4 (from -r requirements.txt (line 5))
Downloading hachoir-parser-1.3.4.tar.gz (359kB): 359kB downloaded
Running setup.py egg_info for package hachoir-parser
warning: no files found matching 'metadata_csv.py'
Installing collected packages: guessit, tvdb-api, hachoir-metadata, hachoir-core, hachoir-parser
Running setup.py install for guessit
Running setup.py install for tvdb-api
Running setup.py install for hachoir-metadata
Warning: unable to recompile dialog.ui to dialog_ui.py using pyuic4
(use command "-c install --record /tmp/pip-Xc_ilb-record/install-record.txt --single-version-externally-managed --install-headers /home/travis/virtualenv/python2.7/include/site/python2.7 --disable-qt" to disable this warning)
changing mode of build/scripts-2.7/hachoir-metadata from 664 to 775
changing mode of build/scripts-2.7/hachoir-metadata-gtk from 664 to 775
changing mode of build/scripts-2.7/hachoir-metadata-qt from 664 to 775
deleting hachoir_metadata.egg-info/requires.txt
changing mode of /home/travis/virtualenv/python2.7/bin/hachoir-metadata-qt to 775
changing mode of /home/travis/virtualenv/python2.7/bin/hachoir-metadata to 775
changing mode of /home/travis/virtualenv/python2.7/bin/hachoir-metadata-gtk to 775
Running setup.py install for hachoir-core
Running setup.py install for hachoir-parser
deleting hachoir_parser.egg-info/requires.txt
warning: no files found matching 'metadata_csv.py'
Successfully installed guessit tvdb-api hachoir-metadata hachoir-core hachoir-parser
Cleaning up...
$ nosetests -w tests/
E
======================================================================
ERROR: Failure: ImportError (No module named scrappy.core)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/nose/loader.py", line 390, in loadTestsFromName
addr.filename, addr.module)
File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/travis/builds/louist87/Scrappy/tests/test.py", line 8, in <module>
import scrappy.core as scrappy
ImportError: No module named scrappy.core
----------------------------------------------------------------------
Ran 1 test in 0.004s
FAILED (errors=1)
Done. Build script exited with: 1
Note that Scrappy is the name of my project and that is what's failing to import.
And here is my .travis.yml file:
language: python
python:
- "2.7"
install: "pip install -r requirements.txt --use-mirrors"
script: "nosetests -w tests/"
Any idea what could be wrong?

Ok, so it turns out that Travis-CI doesn't do an implicit develop when installing the target package.
This was fixed by editing the script line in the YAML file to:
script: "python setup.py develop && nosetests -w tests/"