I'm currently trying to setup a Jenkins pipeline which kicks off a Python script I'm writing. In this Python script, I need to figure out how to use Azure DevOps API calls. For example, the first task I'm trying to do is download an Artifact.
The typical azure-cli command to do this would be:
az artifacts universal download --organization \"https://dev.azure.com/yourorg/\" --feed yourfeed --name your_artifact --version * --path .
Does anyone know how to write out azure-devops api calls in Python?
According to this ticket. we could invoke the Azure CLI with following way. For example:
from azure.cli.core import get_default_cli
get_default_cli().invoke(['artifacts', 'universal', 'download', '--organization', '\"https://dev.azure.com/yourorg/\"', '--feed', 'yourfeed', '--name', 'your_artifact', '--version', '*', '--path'])
If you get No module named 'azure.cli.command_modules' error, please install azure-cli.
To start using the Azure DevOps extension for Azure CLI,please refer to this.
On how to call azure-cli from python script,you can also refer to this sample on github.
Related
I would like to use a serverless lambda that will execute commands from a tool called WSO2 API CTL as I would on linux cli. I am not sure of how to mimic the downloading and calling of the commands as if I were on a linux machine using either Nodejs or Python via the lambda?
I am okay with creating and setting up the lambda and even getting it in the right VPC so that the commands will reach an application on an EC2 instance but I am stuck at how to actually execute the linux commands using either Nodejs or Python and which one would be better, if any.
After adding the following I get an error trying to download:
os.system("curl -O https://apim.docs.wso2.com/en/latest/assets/attachments/learn/api-controller/apictl-3.2.1-linux-x64.tar.gz")
Warning: Failed to create the file apictl-3.2.1-linux-x64.tar.gz: Read-only
It looks like there is no specific reason to download apictl during the initialisation of your Lambda. Therefore, I would propose to bundle it with your deployment package.
The advantage of this approach are:
Quicker initialisation
Less code in your Lambda
You could extend your CI/CD pipeline to download the application during build and then add it to your ZIP archive that you deploy.
Summary: I'm trying to create a python program to run Powershell scripts in Azure Windows VMs. But I can't find good documentation about which libraries of the SDK I can use.
Detailed: In order to automate certain administrative tasks, a python program should run, authenticate into Azure and then run Powershell scripts in certain Windows VMs. I think I have the auth part thanks to azure.identity, but I can't find any library to interact with a running VM. Sure, lots for creating or modifying a VM in Azure, but nothing to interact with it. Neither in the SDK or in the API browser. I know it can be done with Azure CLI, but id like to use the SDK, if possible at all.
My understanding is that you would like to run PowerShell remotely using the Python SDK.
AFAIK, I don't think there is a provision within the Azure Python SDK to run remote PS scripts.
Workaround :
(But please note that this outside the Azure Python SDK.)
You could create Azure Windows WinRM VM template and you could execute ps commands from the Python code using the winrm library like discussed in this thread.
I wonder is it able to create an HTTP triggered python function on Azure Function without doing any local coding? I want to do everything on Azure cloud. My python function codes are in a Github/Azure repos repository, but I do not have all the extra files of an Azure function project (for example, a init.py script file that is the HTTP trigger function of the Azure Function App). Is it possible to generate those files from Azure (without generating any Azure Function related files on my local computer)? I noticed that we cannot do in-portal editing for Python function Apps.
As far as I know, we can just deploy the python function from local to Azure cloud, but can not deploy it as you expected. And I think it will not be too difficult to us to deploy the python function from local to azure cloud.
Since you have had the main python function code "init.py", you just need to sign in to Azure in you VS code and create python function by following this tutorial. And then use your init.py code to replace the new python function code. After that, run the command below in "TERMINAL" window to generate the "requirements.txt":
pip freeze > requirements.txt
The "requirements.txt" includes all of the modules which imported in your "init.py" and when the function deployed to azure, azure will install these modules by this "requirements.txt". I saw you mentioned you don't have all the extra files of this function project, if these modules are what you care about, the "requirements.txt" will solve your problem.
Then use this command to deploy it to Azure:
func azure functionapp publish hurypyfunapp --build remote
I want to use dataflow to process in parallel a bunch of video clips I have stored in google storage. My processing algorithm has non-python dependencies and is expected to change over development iterations.
My preference would be to use a dockerized container with the logic to process the clips, but it appears that custom containers are not supported (in 2017):
use docker for google cloud data flow dependencies
Although they may be supported now - since it was being worked on:
Posthoc connect FFMPEG to opencv-python binary for Google Cloud Dataflow job
According to this issue a custom docker image may be pulled, but I couldn't find any documentation on how to do it with dataflow.
https://issues.apache.org/jira/browse/BEAM-6706?focusedCommentId=16773376&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16773376
Another option might be to use setup.py to install any dependencies as described in this dated example:
https://cloud.google.com/blog/products/gcp/how-to-do-distributed-processing-of-landsat-data-in-python
However, when running the example I get an error that there is no module named osgeo.gdal.
For pure python dependencies I have also tried to pass the --requirements_file argument, however I still get an error: Pip install failed for package: -r
I could find documentation for adding dependencies to apache_beam, but not to dataflow, and it appears the apache_beam instructions do not work, based on my tests of --requirements_file and --setup_file
This was answered in the comments, rewriting here for clarity:
In Apache Beam you can modify the setup.py file while will be run once per container on start-up. This file allows you to perform arbitrary commands before the the SDK Harness start to receive commands from the Runner Harness.
A complete example can be found in the Apache Beam repo.
As of 2020, you can use Dataflow Flex Templates, which allow you to specify a custom Docker container in which to execute your pipeline.
I am going to create search api for Android and iOS developers.
Our client have setup a lambda function in AWS.
Now we need to fetch data using jwplatform Api based on search keyword passed as parameter. For this, I have to install jwplatform module in Lambda function or upload zip file of code with dependencies. So that i want to run python script locally and after getting appropriate result i will upload zip in AWS Lambda.
I want to use the videos/list (jwplatform Api) class to search the video library using python but i don't know much about Python. So i want to know how to run python script? and where should i put the pyhton script ?
There are a handful of useful Python script examples here: https://github.com/jwplayer/jwplatform-py
I am succeed to install jwplatform module locally.
Steps are as follows:
1. Open command line
2. Type 'python' on command line
3. Type command 'pip install jwplatform'
4. Now, you can use jwplatform api.
Above command added module jwplatform in python locally
But my another challenge is to install jwplatform in AWS Lambda.
After research i am succeed to install module in AWS Lambda. I have bundled module and code in a directory then create zip of bundle and upload it in AWS Lambda. This will install module(jwplatform) in AWS Lambda.