Executing commands incorrectly in Jenkins pipeline shell script - python

I set pyenv to an enabled state. And if there is no python version corresponding to the pipfile through pipenv, I tried to install the corresponding python version automatically through pyenv.
[Reference] Automatically install required Python version when pyenv
is available. at https://pipenv.pypa.io/en/latest/
This attempt works fine when I connect to the server via ssh and run it as an interactive command.
Please refer to the example below. A case of connecting to the server with ssh and using the command
<Pipfile>
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
[requires]
python_version = "3.8.13"
I run pipenv install and get the following message.
> pipenv install
Warning: Python 3.8.13 was not found on your system…
Would you like us to install CPython 3.8.13 with pyenv? [Y/n]: Y
Installing CPython 3.8.13 with pyenv (this may take a few minutes)…
However, if Jenkins is installed on the same server and the command is executed with a shell script in the Jenkins pipeline, the server gives different results than with the interactive command.
Please refer to the example below.
This is an example of using a shell script in the jenkins pipeline.
# Before starting this pipeline, pyenv version 3.8.13 was uninstalled and the pipenv environment was also removed.
pipeline {
agent any
stages {
stage('Example') {
steps {
sh 'pipenv install'
}
}
}
}​
When the above pipeline was executed, a message asking whether to install Python 3.8.13 did not appear, and a virtualenv was created using the system Python 3.7 version on the server.
Warning: Python 3.8.13 was not found on your system…
Creating a virtualenv for this project…
Using /usr/bin/python3 (3.7.3) to create virtualenv…
I don't know why it behaves differently than when I entered an interactive command on the server.
The PATH and ENV of Jenkins and the server are the same, and the pyenv and pipenv commands in the Jenkins pipeline shell script work well.
Same as the server Is there a way to install the 3.8.13 version when the pipenv install command is executed in the Jenkins pipeline?

Related

Cannot install pipenv python version

I am trying to run the command
pipenv install python==3.7.9
and getting the error below. When I run
which python
it shows "/Users/Micky/opt/anaconda3/bin/python" and the version is 3.7.9.
Can anyone tell me how?
Micky-MBP:sportanalitica Micky$ pipenv install python==3.7.9
Installing python==3.7.9...
Error: An error occurred while installing python==3.7.9!
Error text:
ERROR: Could not find a version that satisfies the requirement python==3.7.9
ERROR: No matching distribution found for python==3.7.9
✘ Installation Failed
From what I see, I understand that you wish to use a specific version of Python in your Pipenv. It looks like the way you are passing the parameters is wrong. The parameters you are passing is more for packages and not the version of Python.
To learn more on how to specify the Python version, please refer
here. $ pipenv install --python 3.7.9
To specify the full path to your Python binary, you may
refer here. $ pipenv install --python /Users/Micky/opt/anaconda3/bin/python
Lastly, there is a trick you can use, which is to just pipenv install. Then once you have a Pipfile, modify the Python version. Then use pipenv --rm to remove the install. Next pipenv install for the second time and it should take the version specified in the Pipfile which you modified earlier.
Pipenv is not used to install Python versions. You install Python separately and then use Pipenv to create and manage virtual environments, using pipenv install <package> to install Python packages.
I think what you are trying to do is to create a virtual environment that uses your Python 3.7.9 version. To do that the correct way of specifying the Python version is to pass it as a --python=</path/to/python> option. For example, for pipenv shell:
~$ pipenv shell --help
Usage: pipenv shell [OPTIONS] [SHELL_ARGS]...
Spawns a shell within the virtualenv.
Options:
...
--python TEXT Specify which version of Python virtualenv should use.
The same option is available to pipenv install. So if you have "python3 points to /Users/Micky/opt/anaconda3/bin/python":
~$ pipenv shell --python=/Users/Micky/opt/anaconda3/bin/python
Note that you have to pass the --python option only when creating the virtual environment. Once it's created, it will "remember" it in the Pipfile to use that version.
temp$ pipenv shell --python=/usr/local/opt/python#3.7/bin/python3
...
✔ Successfully created virtual environment!
...
Creating a Pipfile for this project...
Launching subshell in virtual environment...
...
(temp) temp$ python -V
Python 3.7.9
Once your virtual environment, you use pipenv install <package> to install Python packages (not Python itself):
(temp) temp$ pipenv install somepackage
(temp) temp$ pipenv install somepackage==1.0.0
I recommend reading the Basic Usage of Pipenv docs.

Module installed returning ModuleNotFoundError

I'm using virtualenvwrapper to manage a project where I'm just running this for now
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
with beam.Pipeline(options=PipelineOptions()) as p:
pass
This is returning the following error
Traceback (most recent call last):
File "path/to/pipeline.py", line 1, in <module>
import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
I have run pip install apache-beam. Running pip list returns
➜ pip list
Package Version
------------------------------ ---------
apache-beam 2.23.0
avro-python3 1.9.2.1
...
I'll add the following outputs as suggested in a similar question.
pip freeze
➜ pip freeze
apache-beam==2.23.0
avro-python3==1.9.2.1
...
pip -V
➜ pip -V
pip 20.2.2 from /Users/miguel/.virtualenvs/myenv/lib/python3.7/site-packages/pip (python 3.7)
python -V
➜ python -V
Python 3.7.3
which python
➜ which python
/Users/miguel/.virtualenvs/myenv/bin/python
which pip
➜ which pip
/Users/miguel/.virtualenvs/myenv/bin/pip
I don't know if this is relevant but I'm using VSCode and I have selected my python interpreter according to VSCode instructions here. Additionally, I installed python following the instructions here.
Any idea why this is happening?
First test if your installed Python modules work outside of an ide like VSCode.
We can do this by opening the terminal/command prompt and activating our virtual enviornment. In this case, as you're using virtualenvwrapper you need to use the command:
workon myenv
Documentation for the virtualenvwrapper can be found here. Once activated, we can open the Python interpreter in our terminal by using the command:
python
Once running, try and import apache_beam as before using:
import apache_beam as beam
If this works we now know it's a virtual environment set up issue using VSCode. For setting up Virtual Environments for Python in VSCode, use the official documentation Using Python environments in VS Code. This should allow you to specify the virtual environment.
If this fails to work, another option is to create a new virtual environment in VSCode and install your modules. Installing modules to a environment in VSCode can be found in the VSCode instructions you linked to in your post.
I tried to re-create your issue as above using the following:
Python 3.7.7
VSCode 1.48.0
apache-beam 2.23.0
My virtual environment was named 'stack'. Once I had created it the only module I installed was apache-beam using:
pip install apache-beam
When creating the project I made the directory and launched via terminal using:
- mkdir hello
- cd hello
- code .
And then added the Python interpreter via the command:
Python: Select Interpreter
One additional step I needed (as I usually use PyCharm for Python) was to install:
Shell Command: Install 'code' command in PATH** command.
To allow the launch of code via terminal. Below is a screenshot of the project after following these steps and running the same commands as posted in your question for reference.
My .vscode/settings.json is as follows:
{
"python.pythonPath": "/Users/robertyoung/envs/stack/bin/python"
}

Poetry doesn't use the correct version of Python

I've recently installed both Pyenv and Poetry and want to create a new Python 3.8 project. I've set both the global and local versions of python to 3.8.1 using the appropriate Pyenv commands (pyenv global 3.8.1 for example). When I run pyenv version in my terminal the output is 3.8.1. as expected.
Now, the problem is that when I create a new python project with Poetry (poetry new my-project), the generated pyproject.toml file creates a project with python 2.7:
[tool.poetry]
name = "my-project"
version = "0.1.0"
description = ""
authors = ["user <user#email.com>"]
[tool.poetry.dependencies]
python = "^2.7"
[tool.poetry.dev-dependencies]
pytest = "^4.6"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
It seems that Poetry defaults back to the system version of Python. How do I change this so that it uses the version installed with Pyenv?
Edit
I'm using MacOS, which comes bundled with Python 2.7. I think that might be causing some of the issues here. I've reinstalled Python 3.8 again with Pyenv, but when I hit Poetry install I get the following error:
The currently activated Python version 2.7.16 is not supported by the project (^3.8).
Trying to find and use a compatible version.
[NoCompatiblePythonVersionFound]
Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.
Should I create an environment explicitly for the project using Pyenv or should the project be able to access the correct Python version after running pyenv local 3.8.1.? When I do the latter, nothing changes and I still get the same errors.
pyproject.toml is used to define all the dependencies for your project, including the supported python version.
The line your complaining about is just saying that the versions of python supported by the project is python2.7 or greater, this is independent of what versions of python you've installed with pyenv.
python = "^2.7"
If you want to update the versions of python supported by the project you can edit the file directly and run poetry update.
If you want to use multiple versions of python you need to make sure poetry is using the correct dependencies for the version of python you are using. To change the specific version poetry is using you should use poetry env,
poetry env list show the versions of python poetry can use
poetry env use <python> switches poetry to use that version.
For instance on my machine poetry has 3 virtual environments installed and is using the one associated with python3.6:
↪ poetry env list
sipy-a9sqc5pb-py3.6 (Activated)
sipy-a9sqc5pb-py3.7
sipy-a9sqc5pb-py3.8
I'm not sure how these virtual environments with interact with the shivs used by pyenv but their docs have a section relating to it
Managing Virtual Environments
There is a pyenv plugin named pyenv-virtualenv which comes with various features to help pyenv users to manage virtual environments created by virtualenv or Anaconda. Because the activate script of those virtual environments are relying on mutating $PATH variable of user's interactive shell, it will intercept pyenv's shim style command execution hooks. We'd recommend to install pyenv-virtualenv as well if you have some plan to play with those virtual environments.
Alright, I figured the problem. A little embarrassingly, I had not run pyenv shell 3.8.1 before running any of the other commands. Everything works now. Thank you all for your efforts.
you can specify an explicit python executable for poetry using
poetry env use <path to python executable>
This worked for me.
On my machine I was able to fix the "currently activated Python version is not supported by the project" error by reinstalling Poetry:
curl -sSL https://install.python-poetry.org | python3 - --uninstall
curl -sSL https://install.python-poetry.org | python3 -
After that,poetry was able to find the correct version installed by pyenv.
In my case, I had to delete and recreate the virtualenv used by poetry. This is because I added the python version restrictions (e.g. python = ">=3.6.2 <3.7") after creating the virtualenv.
Steps
Delete the original one: run poetry env remove myApp-XkghI9p6-py3.6
Run any poetry step, to create it, or run poetry shell, and confirm poetry run python --version is the correct version.
Even though this issue has been resolved, I am writing this for somebody who comes across this problem again.
After all attempts my python -V always resulted in 2.7 and no discussions mentioned running pyenv shell (surprising to me!)
Adding pyenv to path
$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bash_profile
In my case I had to add it to .bashrc and not bash_profile.
https://ggkbase-help.berkeley.edu/how-to/install-pyenv/
Worked!
In my case, the environment was messed up in some way that poetry failed to activate the virtualenv properly.
Try using a different shell: perhaps, sh, or zsh. If everything works in that shell, this proves that your environment is as messed up as mine was :)
Use this command in both shells:
$ env
and try to spot the difference
You can remove the python version from pyproject.toml file and then run Poetry install
What worked for me was to run python3.8 -m poetry install.

Getting command not found: rasa on using rasa commands

I have installed rasa using:
pip3 install rasa
When I try to use rasa commands like:
rasa init
I get zsh errors:
command not found: rasa error
I am on Mac OS Catalina, using:
Python version 2.7.16
Python3 version 3.7.6
pip version 19.2.3
My path variable looks like this:
/Library/Frameworks/Python.framework/Versions/3.7/bin
:/usr/local/bin
:/usr/bin
:/bin
:/usr/sbin
:/sbin
RASA is currently incompatible with python 3.9+
I'm having python 3.10, and have to switch to python 3.8 for working with Rasa.
To save yourself time:
1. Install python 3.8
2. Create and activate virtual environment:
py -3.8 -m pip install virtualenv
py -3.8 -m virtualenv venv
venv\Scripts\activate
3. Upgrade pip
py -3.8 -m pip install -U pip
4. Install Rasa
pip install rasa
5. Check installation
rasa -h
You should see something like:
usage: rasa [-h] [--version] {init,run,shell,train,interactive,telemetry,test,visualize,data,export,x,evaluate} ...
Rasa command line interface. Rasa allows you to build your own conversational assistants 🤖. The 'rasa' command allows
you to easily run most common commands like creating a new bot, training or evaluating models.
...
I also got the similar error in ubuntu, you can try these set of command:
pip install -U pip
pip install rasa
The first command will update the python package manager and the second will install rasa.
If pip install rasa is not working, then you can try pip3 install rasa
I had followed these following steps and it's working fine for me
Firstly, create a directory name of your choice and get inside of it
mkdir rasaprojects #directory name of your choice in place of rasaprojects
cd rasaprojects #get inside that directory
Now create virtual environment of python3
virtualenv rasaenv -p python3 #write your environment name instead of rasaenv
Now activate the environment
source rasaenv/bin/activate
Finally you are good to go with rasa installation
pip3 install rasa
I add ~/.local/bin to my $PATH in .bashrc, and It resolved the problem:
Go to the home directory:
cd
Find your file location (for me it was ./.bashrc):
sudo find -name .bashrc
Open .bashrc by nano:
nano ./.bashrc
Add next code at the end of the file:
export PATH="$HOME/.local/bin:$PATH"
Restart your shell.
You should activate your python env, create a folder with mkdir and then open it with cd and then: rasa init --no-prompt (ubuntu user)
I solved this problem by downgrading python specifically 3.6.8. However, you can install any version from Python 3.6.x to 3.8.x, and NOT 3.9.
Tips: Don't add variable paths while installing multiple versions of python.
In my case, I had 3.9 python and didn't want to uninstall it because I was a working project on the 3.9 version. If you have the same case, then follow the steps. Otherwise, you can alter step2 : python -m venv project_name\venv
Step 1: Install python from above given ranges. Again, please don't install python 3.9 because it is not currently compatible.
Step 2: Delete the existing virtual environment, if you have already created it while priviously installing the rasa. Make new one
> C:\Users\name\AppData\Local\Programs\Python\Python36\python -m venv project_name\venv
(venv: name of the virtual environment)
This command allows creating a new virtual environment with python just you installed it through step1. You can check python -m version
Step 3: Activate the environment
> project_name\venv\Scripts\acitvate.bat
Step 4: Finally, start installing the rasa:
> pip install rasa
Step 5:
> rasa init
Note: I wasn't using anaconda. If you are currently using an anaconda environment, please do needful with the above steps.
If you still cannot resolve it, please follow thread or you can post there your specific query.
other way to do it using docker so will be no issues.
1.make new directory example rasadocker and open terminal or command prompt
2.run following command docker run --user 1000 -v $(pwd):/app rasa/rasa:3.1.0-full init --no-prompt here --user 1000 for no error of user permission ,-v is for volume, pwd is current directory, :/app is volume in docker image , rasa/rasa is rasa docker image with tag 3.1.0-full ,important thing here to create a project we use init
3.you can see in current folder project required files are added
4.to play with bot run following command `
docker run --user 1000 -it -v $(pwd):/app rasa/rasa:3.1.0-full shell opens shell where bot asks you input
`

Unable to install Airflow even after setting SLUGIFY_USES_TEXT_UNIDECODE and AIRFLOW_GPL_UNIDECODE

I am installing airflow via command:
python3 setup.py install. It takes in the requirements file, requirements/athena.txt which is:
apache-airflow[celery,postgres,hive,password,crypto]==1.10.1
I got an error:
RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE
To remove this error, I set export SLUGIFY_USES_TEXT_UNIDECODE=yes and export AIRFLOW_GPL_UNIDECODE=yes. However, running the command python3 setup.py install still gives the same error, nothing changed. To check env variables:
➜ athena-py git:(pyspark-DataFrameStatFunctions) echo $SLUGIFY_USES_TEXT_UNIDECODE
yes
➜ athena-py git:(pyspark-DataFrameStatFunctions) echo $AIRFLOW_GPL_UNIDECODE
yes
Airflow is not officially supporting python3.7 yet.
I think pip might not be picking up your environment variables. You can try the following command on your shell to make sure it picks up the options you are providing.
AIRFLOW_GPL_UNIDECODE=yes pip3 install "apache-airflow[celery,postgres,hive,password,crypto]==1.10.1"```

Categories