I'm running a docker container with alpine.And running ansible script for getting dynamic inventory from AWS and it works great with python2. But I'm changing it to python3 and this is causing me issues. Getting warnings and unable to parse it
In python2 I was able to run the python script this way ./ec2.py
Now with python3, I'm getting this error: env: can't execute 'python': No such file or directory
[WARNING]: * Failed to parse ci/ec2.py with script
plugin: Inventory script (ci/ec2.py) had an execution
error: env: can't execute 'python': No such file or directory
[WARNING]: * Failed to parse ci/ec2.py with ini plugin:
ci/ec2.py:3: Error parsing host definition ''''': No
closing quotation
[WARNING]: Unable to parse ci/ec2.py as an inventory
source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
Python3
apk --update --no-cache add python3 py3-setuptools
pip3 install --upgrade pip
pip3 install awscli ansible boto
chmod 755 ec2.py
ansible-playbook provisioning/ec2New.yml -i ec2.py --private-key ssh-key.pem -e "type_inventory=${TYPE_INVENTORY}
ansible.cfg
[defaults]
host_key_checking = False
stdout_callback = yaml
ansible_python_interpreter = /usr/bin/python3
My old configuration with python 2
apk --update --no-cache add python py-pip
pip install --upgrade pip
pip install awscli ansible botocore boto
chmod 755 ec2.py
ansible-playbook provisioning/ec2New.yml -i ec2.py --private-key ssh-key.pem -e "type_inventory=${TYPE_INVENTORY}
old ansible.cfg
defaults
host_key_checking = False
stdout_callback = yaml
I had the same issue described above, if you change the first line in your ec2.py file to be:
#!/usr/bin/env python3
Then it should parse and work as expected.
I noticed your comment and it seems python3 was replaced wrong in the shebang.
If I replace it getting this: /usr/bin/python3: can't open file 'python': [Errno 2] No such file or directory –
Diego
Apr 10, 2020 at 3:43
So, if you follow the solution above it "should" work.
Related
I am building a python module. In order to define its path, a .pth file has been defined as follows:
# creation of the virtual environment
python -v venv env
# activation of the newly creation virtual environment
source env/bin/activate
To set the path of my module (my module is located in packages/regression_model/regression_model) I created this .pth file env/lib/python3.7/site-packages/regression_model.pth which contains:
# env/lib/python3.7/site-packages/regression_model.pth
../../../../packages/regression_model
Now, any where in my project, I can import my module regression_model through this command:
import regression_model
Actually my objective is to use CircleCI for the deployment of my project.
CircleCI is configured as follows:
version: 2
jobs:
test_regression_model:
working_directory: ~/project
docker:
- image: circleci/python:3.7.6
environment: # environment variables for primary container
PYTHONPATH: ~/project/packages/regression_model:~/project/packages/ml_api
steps:
- checkout
- run:
name: Runnning tests
command: |
virtualenv venv
. venv/bin/activate
pip install --upgrade pip
pip install -r packages/regression_model/requirements.txt
chmod +x ./scripts/fetch_kaggle_dataset.sh
./scripts/fetch_kaggle_dataset.sh
python packages/regression_model/regression_model/train_pipeline.py
py.test -vv packages/regression_model/tests
workflows:
version: 2
test-all:
jobs:
- test_regression_model
The problem I am facing is that CircleCI is indicating that my module can not be imported
Traceback (most recent call last):
File "packages/regression_model/regression_model/train_pipeline.py", line 4, in <module>
from regression_model import pipeline
ModuleNotFoundError: No module named 'regression_model'
To solve the problem, the path to that module regression_model has to be defined exactly as it was done locally. The question is then: how to define path in the CircleCI?
I tried to do it through the use of the environment variable PYTHONPATH but without success.
Any suggestions?
I found out the solution. Similarly to what it has been done manually on my local machine, I just define 2 command lines to get it done in CircleCI:
echo "../../../../packages/regression_model" >> env/lib/python3.7/site-packages/extra.pth
echo "../../../../packages/ml_api" >> env/lib/python3.7/site-packages/extra.pth
And below the full yml file just in case it could help others.
version: 2
jobs:
test_regression_model:
working_directory: ~/project
docker:
- image: circleci/python:3.7.6
steps:
- checkout
- run:
name: Runnning tests
command: |
virtualenv env
. env/bin/activate
pip install --upgrade pip
pip install -r packages/regression_model/requirements.txt
echo "../../../../packages/regression_model" >> env/lib/python3.7/site-packages/extra.pth
echo "../../../../packages/ml_api" >> env/lib/python3.7/site-packages/extra.pth
chmod +x ./scripts/fetch_kaggle_dataset.sh
./scripts/fetch_kaggle_dataset.sh
sudo apt-get install unzip
unzip packages/regression_model/regression_model/datasets/house-prices-advanced-regression-techniques.zip -d packages/regression_model/regression_model/datasets/
python packages/regression_model/regression_model/train_pipeline.py
py.test -vv packages/regression_model/tests
workflows:
version: 2
test-all:
jobs:
- test_regression_model
I have created a playbook that is suppose to run a django website for local developers. These are organizational constraints
Currently the VM is Centos - http://puppet-vagrant-boxes.puppetlabs.com/centos-64-x64-vbox4210.box
The machine is being provisioned with ansible via Vagrant.
The developer will need python2.7.
I attempted to follow the software collections route in
adding a scl repo to box
installing python27 via yum
using shell modoule to enable python27
creating a virtualenv inside that shell
The newly create virtualenv and python binaries give an error after provision. Here is the pertinent part of my playbook:
main.yml
- hosts: app
sudo: yes
sudo_user: root
gather_facts: true
roles:
# insert other roles
tasks:
- name: Add SCL Repos
command: sh -c 'wget -qO- http://people.redhat.com/bkabrda/scl_python27.repo >> /etc/yum.repos.d/scl.repo'
- name: Install python dependencies
yum: pkg={{ item }} state=present
with_items:
- "python-devel"
- "scl-utils"
- "python27"
- name: Manually create virtual .env and install requirements
shell: "source /opt/rh/python27/enable && virtualenv /vagrant/.env && source /vagrant/.env/bin/activate && pip install -r /vagrant/requirements/local.txt"
Ansible - stdout
Here is the tail end of my ansible's stdout message.
pip can't proceed with requirement 'pytz (from -r /vagrant/requirements/base.txt (line 3))' due to a pre-existing build directory.\n location: /vagrant/.env/build/pytz\nThis is likely due to a previous installation that failed.\npip is being responsible and not assuming it can delete this.\nPlease delete it and try again.\n\nCleaning up...
Post Mortem Test via SSH
In an attempt to glean more information out the problem, I sshed into the box to see what feedback I could get.
$ vagrant ssh
Last login: Fri Feb 12 22:17:03 2016 from 10.0.2.2
Welcome to your Vagrant-built virtual machine.
[vagrant#localhost ~]$ cd /vagrant/
[vagrant#localhost vagrant]$ source .env/bin/activate
(.env)[vagrant#localhost vagrant]$ pip install -r requirements/local.txt
/vagrant/.env/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
In general, the approach feels like a square peg in a round hole. I'd love to hear some feedback from the community about the appropriate way to run a centos box locally using a python27 virtualenv provisioned through ansible.
You could always use ansible environment directive to manually set appropriate variables so that correct executables get called. Here's an example:
environment:
PATH: "/opt/rh/rh-python34/root/usr/bin:{{ ansible_env.PATH }}"
LD_LIBRARY_PATH: "/opt/rh/rh-python34/root/usr/lib64"
MANPATH: "/opt/rh/rh-python34/root/usr/share/man"
XDG_DATA_DIRS: "/opt/rh/rh-python34/root/usr/share"
PKG_CONFIG_PATH: "/opt/rh/rh-python34/root/usr/lib64/pkgconfig"
pip: "virtualenv={{root_dir}}/{{venvs_dir}}/{{app_name}}_{{spec}} requirements={{root_dir}}/{{spec}}_sites/{{app_name}}/requirements.txt"
In the end, I had to rebuild python from source to create a python2.7 virtual environment. I used an open source playbook.
https://github.com/Ken24/ansible-role-python
main.yml
- hosts: app
roles:
- { role: Ken24.python }
tasks:
- name: Install virtualenv
command: "/usr/local/bin/pip install virtualenv"
- name: Create virtualenv and install requirements
pip: requirements=/vagrant/requirements/local.txt virtualenv=/vagrant/cfgov-refresh virtualenv_command=/usr/local/bin/virtualenv
I am trying to install python package "M2Crypto" via requirements.txt and I receive the following error message:
/usr/include/openssl/opensslconf.h:36: Error: CPP #error ""This openssl-devel package does not work your architecture?"". Use the -cpperraswarn option to continue swig processing.
error: command 'swig' failed with exit status 1
I tried passing
option_name: SWIG_FEATURES
value: "-cpperraswarn -includeall -I/usr/include/openssl"
But the error persists. Any idea?
The following config file (placed in .ebextensions) works for me:
packages:
yum:
swig: []
container_commands:
01_m2crypto:
command: 'SWIG_FEATURES="-cpperraswarn -includeall -D`uname -m` -I/usr/include/openssl" pip install M2Crypto==0.21.1'
Make sure you don't specify M2Crypto in your requirements.txt though, Elastic Beanstalk will try to install all dependencies before running the container commands.
I have found a solution that gets M2Crypto installed on Beanstalk but it is a bit of hack and it is your responsibility to make sure that it is good for a production environment. I dropped M2Crypto from my project because this issue is ridiculous, try pycrypto if you can.
Based on (I only added python setup.py test):
#!/bin/bash
python -c "import M2Crypto" 2> /dev/null
if [ "$?" == 1 ]
then
cd /tmp/
pip install -d . --use-mirrors M2Crypto==0.21.1
tar xvfz M2Crypto-0.21.1.tar.gz
cd M2Crypto-0.21.1
./fedora_setup.sh build
./fedora_setup.sh install
python setup.py test
fi`
In the environment config file
commands:
m2crypto:
command: scripts/m2crypto.sh
ignoreErrors: True
test: echo '! python -c "import M2Crypto"' | bash
ignoreErrors is NOT a good idea but I just used it to test if the package actually gets installed and seems like it.
Again, this might seem to get the package installed but I am not sure because removing ignoreErrors causes failure. Therefore, I won't mark this as the accepted answer but it was way too much to be a comment.
When I attempt to run pyramid
[~/env/MyStore]# ../bin/pserve development.ini
It will show the following error
File "/home/vretinfo/env/lib/python3.2/site-packages/Paste-1.7.5.1-py3.2.egg/paste/fileapp.py", line 14, in <module>
from paste.httpheaders import *
File "/home/vretinfo/env/lib/python3.2/site-packages/Paste-1.7.5.1-py3.2.egg/paste/httpheaders.py", line 140, in <module>
from rfc822 import formatdate, parsedate_tz, mktime_tz
ImportError: No module named rfc822
How should I resolve this?
This is what I did to install
$ mkdir opt
$ cd opt
$ wget http://python.org/ftp/python/3.2.3/Python-3.2.3.tgz
$ tar -xzf Python-3.2.3.tgz
$ cd Python-3.2.3
./configure --prefix=$HOME/opt/Python-3.2.3
$ make;
$ make install
$ cd ~
$ wget http://python-distribute.org/distribute_setup.py
$ pico distribute_setup.py
* change first line to opt/Python-3.2.3/python
$ opt/Python-3.2.3/bin/python3.2 distribute_setup.py
$ opt/Python-3.2.3/bin/easy_install virtualenv
$ opt/Python-3.2.3/bin/virtualenv --no-site-packages env
$ cd env
$ ./bin/pip install passlib
$ ./bin/pip install pyramid_beaker
$ ./bin/pip install pyramid_mailer
$ ./bin/pip install pyramid_mongodb
$ ./bin/pip install pyramid_jinja2
$ ./bin/pip install Werkzeug
$ ./bin/pip install pyramid
$ ./bin/pcreate -s pyramid_mongodb MyShop
$ cd MyShop
$ ../bin/python setup.py develop
$ ../bin/python setup.py test -q
Ok, I've done some searching around on pyramid docs ( http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/paste.html ).
It states on the 3rd paragraph
"However, all Pyramid scaffolds render PasteDeploy configuration files, to provide new developers with a standardized way of setting deployment values, and to provide new users with a standardized way of starting, stopping, and debugging an application."
So I made changes to development.ini and replaced
[server:main]
use = egg:waitress#main
and in setup.py, I added 'waitress' into the requires array
Next step, I did to totally remove all things related to Paste, in /home/vretinfo/env/ECommerce/,
$ rm -rf Paste*;rm -rf paste*
After this, I tried running test -q again, this is the stack trace:
[~/env/ECommerce]# ../bin/python setup.py test -q
/home/vretinfo/opt/Python-3.2.3/lib/python3.2/distutils/dist.py:257: UserWarning: Unknown distribution option: 'paster_plugins'
warnings.warn(msg)
running test
Checking .pth file support in .
/home/vretinfo/env/ECommerce/../bin/python -E -c pass
Searching for Paste>=1.7.1
Reading http://pypi.python.org/simple/Paste/
Reading http://pythonpaste.org
Best match: Paste 1.7.5.1
Downloading http://pypi.python.org/packages/source/P/Paste/Paste-1.7.5.1.tar.gz#md5=7ea5fabed7dca48eb46dc613c4b6c4ed
Processing Paste-1.7.5.1.tar.gz
Writing /tmp/easy_install-q5h5rn/Paste-1.7.5.1/setup.cfg
Running Paste-1.7.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-q5h5rn/Paste-1.7.5.1/egg-dist-tmp-e3nvmj
warning: no previously-included files matching '*' found under directory 'docs/_build/_sources'
It seems like paste is needed for pyramid1.4 for some reason. Perhaps someone have some insights on this.
I have managed to solve this issue through the folks in IRC#pyramid. I'm posting the solution in here in case someone encounters in future. I've tested this out in Python3.2, works ok now.
After running ./bin/pcreate -s <...>
in the folder of the project, development.ini
Change the following:
1. in the 1st line, rename [app:<Project>] to [app:main]
2. [server:main]
If it is egg:Paste#http, change it to
use = egg:waitress#main
3. remove [pipeline:main] and its section
in the same folder, make the necessary changes to setup.py:
1. requires = [....], add in waitress into the array, remove WebError from the array
2. remove paster_plugins=['pyramid']
Then finally, run
$ ../bin/python setup.py develop
Paste will not be installed or checked if it exists.
I have Python 2.7.3 installed on RHEL 6, and when I tried to install pysvn-1.7.6, I got an error. What should I do?
/search/python/pysvn-1.7.6/Import/pycxx-6.2.4/CXX/Python2/Objects.hxx:2912: warning: deprecated conversion from string constant to 'char*'
Compile: pysvn_svnenv.cpp into pysvn_svnenv.o
Compile: pysvn_profile.cpp into pysvn_profile.o
Compile: /search/python/pysvn-1.7.6/Import/pycxx-6.2.4/Src/cxxsupport.cxx into cxxsupport.o
Compile: /search/python/pysvn-1.7.6/Import/pycxx-6.2.4/Src/cxx_extensions.cxx into cxx_extensions.o
Compile: /search/python/pysvn-1.7.6/Import/pycxx-6.2.4/Src/cxxextensions.c into cxxextensions.o
Compile: /search/python/pysvn-1.7.6/Import/pycxx-6.2.4/Src/IndirectPythonInterface.cxx into IndirectPythonInterface.o
Link pysvn/_pysvn_2_7.so
make: *** No rule to make target `egg'. Stop.
error: Not a URL, existing file, or requirement spec: 'dist/pysvn-1.7.6-py2.7-linux-x86_64.egg'
I solved this problem, the reason is that i have made a mistake.
i just executed the following command, it is not in the instruction.
python setup.py install
the installation steps are (the Source is the dir name in pysvn directory):
cd Source
python setup.py configure
make
cd ../Tests
make
cd Source
mkdir [YOUR PYTHON LIBDIR]/site-packages/pysvn
cp pysvn/__init__.py [YOUR PYTHON LIBDIR]/site-packages/pysvn
cp pysvn/_pysvn*.so [YOUR PYTHON LIBDIR]/site-packages/pysvn
I had a same problem. and I find this solution and it is working.
Download the latest epel-release rpm from
http://dl.fedoraproject.org/pub/epel/6/x86_64/
for now :
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Install epel-release rpm:
rpm -Uvh epel-release*rpm
Install pysvn rpm package:
yum install pysvn