I am trying to run a python program through bamboo.
How do I install python dependency modules through bamboo
I need to install some python modules like flask , xldr etc.
You have two options:
Remote or log into the Bamboo agent and manually install the modules. This is a one time install and then they will be there for the task to use in the future.
Run the job using a Docker host instead of the local agent. Then you can specify all the dependencies in the Docker image that is used to build (e.g., Python version, imports).
Related
I have a custom airflow image that has some python packages in it. I am running it in local kubernetes cluster. I have created a DAG in my airflow that uses one of the python packages and it works totally fine.
But if I use some other python package that's not in my custom base image(imageio or anyother image), it gives me module not found error.
So I added the line "RUN pip3 install imageio==2.8.0" (or any other package) in my Dockerfile, first of all, it gave me the warning: python install warning
Now the import error is no more there but if I run my DAG it fails without outputting any logs.
So next I added the line "ENV PATH="/usr/local/airflow/.local/bin:${PATH}" but still the same thing.
I am guessing my DAG is not able to find the extra python packages that are being installed, or more clearly somehow the pods being created don't have those extra python packages with them.
However, if I do "docker run -it {image_name} bash" and type import imageio inside a python shell, it works fine.
Is there some config file in which I will need to mention the extra python packages that I want so that the pods running the DAGs will register those packages?
or is there a way to specify it in the values.yaml file?
I have created a Bamboo task which runs the python code from a BitBucket Repo.
Bamboo config:
I am running the script as a file.
I have selected interpreter as Shell and given this in the Script Body to execute the script python create_issue.py -c conf.yml
After I click on 'Run Plan', the build fails with ImportError: No module named pandas. The rest of the libraries are working fine, like, requests, itertools, etc.
It sounds like you are running this Bamboo plan using the Agent Host and not a Docker Container. As such you will need to:
Remote/Log into the Bamboo server
Use pip or some other package tool to install requests, itertools, and any other missing imports
Alternatively, you could set-up an isolated Docker image that has all these dependencies and build within that.
I'm trying to work with ansible, winrm, virtualenv and Jenkins...
Currently, I have installed Ansible with Tom via epel-release.
Jenkins has only basic configuration for now.
I have then created a virtualenv inside Jenkins home named $HOME/ansible-winrm. Then inside it, I have installed winrm via pip.
What I'm trying to do is :
- create a simple job on Jenkins with only a shell script calling ansible-playbook. And it should access to the winrm library installed inside my local virtualenv.
- It should be as transparent as possible.
P.S. It seems that python binary is hard codded inside ansible-playbook script.
What are your best practices to solve this issue ?
Best way to do it is installing winrm with pip in user workspace (option --user)
Ex: pip install --user pywinrm
I am trying to run a python script with Azure functions.
I had success updating the python version and installing modules on Azure functions under the App Services plan but I need to use it under the Consumption plan as my script will only execute once everyday and for only a few minutes, so I want to pay only for the time of execution. See: https://azure.microsoft.com/en-au/services/functions/
Now I'm still new to this but from my understanding the consumption plan spins up the vm and terminates it after your script has been executed unlike the App Service plan which is always on.
I am not sure why this would mean that I can't have install anything on it. I thought that would just mean I have to install it every time I spin it up.
I have tried installing modules through the python script itself and the kudu command line with no success.
While under the app service plan it was simple, following this tutorial: https://prmadi.com/running-python-code-on-azure-functions-app/
On Functions Comsumption plan, Kudu extensions are not available. However, you can update pip to be able to install all your dependencies correctly:
Create your Python script on Functions (let's say NameOfMyFunction/run.py)
Open a Kudu console
Go to the folder of your script (should be d:/home/site/wwwroot/NameOfMyFunction)
Create a virtualenv in this folder (python -m virtualenv myvenv)
Load this venv (cd myenv/Scripts and call activate.bat)
Your shell should be now prefixed by (myvenv)
Update pip (python -m pip install -U pip)
Install what you need (python -m pip install flask)
Now in the Azure Portal, in your script, update the sys.path to add this venv:
import sys, os.path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname( __file__ ), 'myvenv/Lib/site-packages')))
You should be able to start what you want now.
(Reference: https://github.com/Azure/azure-sdk-for-python/issues/1044)
Edit: reading previous comment, it seems you need numpy. I just tested right now and I was able to install 1.12.1 with no issues.
You may upload the modules for the Python version of your choice in Consumption Plan. Kindly refer to the instructions at this link: https://github.com/Azure/azure-webjobs-sdk-script/wiki/Using-a-custom-version-of-Python
This is what worked for me:
Dislaimer: I use C# Function that includes Python script execution, using command line with System.Diagnostics.Process class.
Add relevant Python extension for the Azure Function from Azure Portal:
Platform Features -> Development Tools -> Extensions
It installed python to D:\home\python364x86 (as seen from Kudu console)
Add an application setting called WEBSITE_USE_PLACEHOLDER and set its value to 0. This is necessary to work around an Azure Functions issue that causes the Python extension to stop working after the function app is unloaded.
See: Using Python 3 in Azure Functions question.
Install the packages from Kudu CMD line console using pip install ...
(in my case it was pip install pandas)
I wish to connect to a Linux machine by ssh (from my code) and run some code that is using python libraries that are not installed on the remote machine, what would be the best way to do so?
using a call like this:
cat main.py | ssh user#server python -
will run main.py on the server, but wont help me with the dependencies, is there a way to somehow 'compile' the relevant libraries and have them sent over just for the running my code?
I wish to avoid installing the libraries on the remote machine if possible
Try virtualenv:
pip install virtualenv
then use
virtualenv venv
to create a seperated python environment in current path(in folder venv).
Instead of installing multiple packages in default python path, virtualenv needs only one package installed.