Travis CI with GAE and django - python

I am having some problems when I am using google python SDK in Travis-CI. I'm always getting this exception:
Failure: ImportError (No module named google.appengine.api) ... ERROR
I think the problem is in my travis file or django settings file. Can I use the GAE SDK API in the Travis platform?
I write down my .travis.yml file:
language: python
python:
- "2.7"
before_script:
- wget https://storage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.10.zip -nv
- unzip -q google_appengine_1.9.10.zip
- mysql -e 'create database DATABASE_NAME;'
- echo "USE mysql;\nUPDATE user SET password=PASSWORD('A_PASSWORD') WHERE user='USER';\nFLUSH PRIVILEGES;\n" | mysql -u USER
- python manage.py syncdb --noinput
install:
- pip install -r requirements.txt
- pip install mysql-python
script: python manage.py test --with-coverage
branches:
only:
- testing
Thank you

After trying a lot I solved it adding this in my travis.yml file in the before_script section after the unzip order:
- export PYTHONPATH=${PYTHONPATH}:google_appengine

Related

Pyinstaller not working in Gitlab CI file

I have created a python application and I would to deploy it via Gitlab. To achieve this, I create the following gitlab-ci.yml file:
# This file is a template, and might need editing before it works on your project.
# Official language image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python/tags/
image: "python:3.10"
#commands to run in the Docker container before starting each job.
before_script:
- python --version
- pip install -r requirements.txt
# different stages in the pipeline
stages:
- Static Analysis
- Test
- Deploy
#defines the job in Static Analysis
pylint:
stage: Static Analysis
script:
- pylint -d C0301 src/*.py
#tests the code
pytest:
stage: Test
script:
- cd test/;pytest -v
#deploy
deploy:
stage: Deploy
script:
- echo "test ms deploy"
- cd src/
- pyinstaller -F gui.py --noconsole
tags:
- macos
It runs fine through the Static Analysis and Test phases, but in Deploy I get the following error:
OSError: Python library not found: .Python, libpython3.10.dylib, Python3, Python, libpython3.10m.dylib
This means your Python installation does not come with proper shared library files.
This usually happens due to missing development package, or unsuitable build parameters of the Python installation.
* On Debian/Ubuntu, you need to install Python development packages:
* apt-get install python3-dev
* apt-get install python-dev
* If you are building Python by yourself, rebuild with `--enable-shared` (or, `--enable-framework` on macOS).
As I am working on a Macbook I tried with the following addition - env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.10.5 but then I get an error that python 3.10.5 already exists.
I tried some other things, but I am a bit stuck. Any advice or suggestions?

Unable to run django test in CircleCI

I am implementing CircleCI for one of the projects. The project is built on Django 3.2.
My test cases run properly when I run using python manage.py test blog, when I run the same in CircleCI it returns ,
======================================================================
ERROR: project.blog (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: project.blog
Traceback (most recent call last):
File "/usr/local/lib/python3.8/unittest/loader.py", line 470, in _find_test_path
package = self._get_module_from_name(name)
File "/usr/local/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
__import__(name)
ModuleNotFoundError: No module named 'project.blog'
Here is my CircleCI config
version: 2
jobs:
build:
docker:
- image: circleci/python:3.8
steps:
- checkout
- run:
name: Installing dependencies
command: |
python3 -m venv venv
. venv/bin/activate
pip3 install -r requirements.txt
- run:
name: Running migrations
command: |
. venv/bin/activate
python manage.py migrate --skip-checks
- run:
name: Running tests
command: |
. venv/bin/activate
python manage.py test blog
I understand that CircelCI clones the project in project folder. Is that something that I am missing in config?
CircleCI by default checkouts our codebase to /home/circleci/project, problem was in list of installed_apps I had an app with name project.(which was conflicting)
When CircleCI ran python manage.py test the unittest module was searching the app blog inside django app project.
I fixed this problem by changing the default path to which CircleCI puts out codebase. Here is the updated CircleCI config
version: 2
jobs:
build:
working_directory: ~/platform #Here is the answer
docker:
- image: circleci/python:3.8
steps:
- checkout:
path: ~/platform #Here is the answer
- run:
name: Install dependencies
command: |
ls -l
python3 -m venv venv
. venv/bin/activate
pip3 install -r requirements.txt
- run:
name: Run migrations
command: |
. venv/bin/activate
python manage.py migrate --skip-checks
- run:
name: Run tests
command: |
. venv/bin/activate
python manage.py test website

Save a file from gitlab-ci to a gitlab repository

I made this citlab-ci.yml file but i can find the html report in my repo at the end.
Can you tell me why ?
image: python
services:
- selenium/standalone-chrome:latest
variables:
selenium_remote_url: "http://selenium__standalone-chrome:4444/wd/hub"
cucumber:
script:
- python --version
- pwd
- ls
- pip install pytest
- pip install pytest_bdd
- pip install selenium
- pip install chromedriver
- pip install pytest-html
- cd test_pytest
- ls
- python -m pytest step_defs/test_web_steps.py --html=report.html
tx
Hadrien
You can actually generate test reports in gitlab. For this, generate an XML report from Pytest that would be stored in GitLab as an artifact. On your .gitlab-ci.yml file
image: python:3.6
stages:
- test
testing:
stage: test
when: manual
script:
...
- pytest --junitxml=report.xml
artifacts:
when: always
reports:
junit: report.xml
Then, you can download this report
or visualize it under the Tests tag of your pipeline.

how to set the path of local module in python to be recognized in CircleCI?

I am building a python module. In order to define its path, a .pth file has been defined as follows:
# creation of the virtual environment
python -v venv env
# activation of the newly creation virtual environment
source env/bin/activate
To set the path of my module (my module is located in packages/regression_model/regression_model) I created this .pth file env/lib/python3.7/site-packages/regression_model.pth which contains:
# env/lib/python3.7/site-packages/regression_model.pth
../../../../packages/regression_model
Now, any where in my project, I can import my module regression_model through this command:
import regression_model
Actually my objective is to use CircleCI for the deployment of my project.
CircleCI is configured as follows:
version: 2
jobs:
test_regression_model:
working_directory: ~/project
docker:
- image: circleci/python:3.7.6
environment: # environment variables for primary container
PYTHONPATH: ~/project/packages/regression_model:~/project/packages/ml_api
steps:
- checkout
- run:
name: Runnning tests
command: |
virtualenv venv
. venv/bin/activate
pip install --upgrade pip
pip install -r packages/regression_model/requirements.txt
chmod +x ./scripts/fetch_kaggle_dataset.sh
./scripts/fetch_kaggle_dataset.sh
python packages/regression_model/regression_model/train_pipeline.py
py.test -vv packages/regression_model/tests
workflows:
version: 2
test-all:
jobs:
- test_regression_model
The problem I am facing is that CircleCI is indicating that my module can not be imported
Traceback (most recent call last):
File "packages/regression_model/regression_model/train_pipeline.py", line 4, in <module>
from regression_model import pipeline
ModuleNotFoundError: No module named 'regression_model'
To solve the problem, the path to that module regression_model has to be defined exactly as it was done locally. The question is then: how to define path in the CircleCI?
I tried to do it through the use of the environment variable PYTHONPATH but without success.
Any suggestions?
I found out the solution. Similarly to what it has been done manually on my local machine, I just define 2 command lines to get it done in CircleCI:
echo "../../../../packages/regression_model" >> env/lib/python3.7/site-packages/extra.pth
echo "../../../../packages/ml_api" >> env/lib/python3.7/site-packages/extra.pth
And below the full yml file just in case it could help others.
version: 2
jobs:
test_regression_model:
working_directory: ~/project
docker:
- image: circleci/python:3.7.6
steps:
- checkout
- run:
name: Runnning tests
command: |
virtualenv env
. env/bin/activate
pip install --upgrade pip
pip install -r packages/regression_model/requirements.txt
echo "../../../../packages/regression_model" >> env/lib/python3.7/site-packages/extra.pth
echo "../../../../packages/ml_api" >> env/lib/python3.7/site-packages/extra.pth
chmod +x ./scripts/fetch_kaggle_dataset.sh
./scripts/fetch_kaggle_dataset.sh
sudo apt-get install unzip
unzip packages/regression_model/regression_model/datasets/house-prices-advanced-regression-techniques.zip -d packages/regression_model/regression_model/datasets/
python packages/regression_model/regression_model/train_pipeline.py
py.test -vv packages/regression_model/tests
workflows:
version: 2
test-all:
jobs:
- test_regression_model

Using ansible core's pip and virtualenv on Centos or Redhat

I have created a playbook that is suppose to run a django website for local developers. These are organizational constraints
Currently the VM is Centos - http://puppet-vagrant-boxes.puppetlabs.com/centos-64-x64-vbox4210.box
The machine is being provisioned with ansible via Vagrant.
The developer will need python2.7.
I attempted to follow the software collections route in
adding a scl repo to box
installing python27 via yum
using shell modoule to enable python27
creating a virtualenv inside that shell
The newly create virtualenv and python binaries give an error after provision. Here is the pertinent part of my playbook:
main.yml
- hosts: app
sudo: yes
sudo_user: root
gather_facts: true
roles:
# insert other roles
tasks:
- name: Add SCL Repos
command: sh -c 'wget -qO- http://people.redhat.com/bkabrda/scl_python27.repo >> /etc/yum.repos.d/scl.repo'
- name: Install python dependencies
yum: pkg={{ item }} state=present
with_items:
- "python-devel"
- "scl-utils"
- "python27"
- name: Manually create virtual .env and install requirements
shell: "source /opt/rh/python27/enable && virtualenv /vagrant/.env && source /vagrant/.env/bin/activate && pip install -r /vagrant/requirements/local.txt"
Ansible - stdout
Here is the tail end of my ansible's stdout message.
pip can't proceed with requirement 'pytz (from -r /vagrant/requirements/base.txt (line 3))' due to a pre-existing build directory.\n location: /vagrant/.env/build/pytz\nThis is likely due to a previous installation that failed.\npip is being responsible and not assuming it can delete this.\nPlease delete it and try again.\n\nCleaning up...
Post Mortem Test via SSH
In an attempt to glean more information out the problem, I sshed into the box to see what feedback I could get.
$ vagrant ssh
Last login: Fri Feb 12 22:17:03 2016 from 10.0.2.2
Welcome to your Vagrant-built virtual machine.
[vagrant#localhost ~]$ cd /vagrant/
[vagrant#localhost vagrant]$ source .env/bin/activate
(.env)[vagrant#localhost vagrant]$ pip install -r requirements/local.txt
/vagrant/.env/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
In general, the approach feels like a square peg in a round hole. I'd love to hear some feedback from the community about the appropriate way to run a centos box locally using a python27 virtualenv provisioned through ansible.
You could always use ansible environment directive to manually set appropriate variables so that correct executables get called. Here's an example:
environment:
PATH: "/opt/rh/rh-python34/root/usr/bin:{{ ansible_env.PATH }}"
LD_LIBRARY_PATH: "/opt/rh/rh-python34/root/usr/lib64"
MANPATH: "/opt/rh/rh-python34/root/usr/share/man"
XDG_DATA_DIRS: "/opt/rh/rh-python34/root/usr/share"
PKG_CONFIG_PATH: "/opt/rh/rh-python34/root/usr/lib64/pkgconfig"
pip: "virtualenv={{root_dir}}/{{venvs_dir}}/{{app_name}}_{{spec}} requirements={{root_dir}}/{{spec}}_sites/{{app_name}}/requirements.txt"
In the end, I had to rebuild python from source to create a python2.7 virtual environment. I used an open source playbook.
https://github.com/Ken24/ansible-role-python
main.yml
- hosts: app
roles:
- { role: Ken24.python }
tasks:
- name: Install virtualenv
command: "/usr/local/bin/pip install virtualenv"
- name: Create virtualenv and install requirements
pip: requirements=/vagrant/requirements/local.txt virtualenv=/vagrant/cfgov-refresh virtualenv_command=/usr/local/bin/virtualenv

Categories