i run the following command as a bash script.
az group create -n $1-rg -l eastus2
az deployment group create -g $1-rg -n $1-deploy \
-f ./azure/sensor/trafficmirrorstack.json \
-p #./azure/sensor/trafficmirrorstack.parameters.json \
-p CustomerName=$1
az deployment group show -g $1-rg -n $1-deploy
the following seems like it should work:
rg_name = f"{name}-rg"
deploy_name = f"{name}-deploy"
region = list(region_params.keys())[0]
# add resource group
rg_result = resource_client.resource_groups.create_or_update(
rg_name,
{
"location": region
}
)
print(f"Provisioned resource group {rg_result.name} in the {rg_result.location} region")
with open("./sensor/trafficmirrorstack.json") as template_file:
print("!!!!1")
print(f"{0}".format(template_file.read()))
#print(f"{0}".format(template_file.read()))
print("!!!!2")
template = f"{0}".format(template_file.read())
parameters = {"CustomerName": { "value": name}}
deployment_params = {
"mode": "Incremental",
"template": template,
"parameters": parameters
}
# Create deployment
deployment_create_result = resource_client.deployments.begin_create_or_update(
rg_name,
deploy_name,
{"properties": deployment_params},
# deployment_params,
)
deployment_create_result = deployment_create_result.result()
but how do you do the equivalent of "-p #./azure/sensor/trafficmirrorstack.parameters.json -p CustomerName=$1"
thanks in advance
When we use Azure python sdk to deploy arm template, we just can use provide the paramater file URL or define parameter as json with the method resource_client.deployments.begin_create_or_update. we can not use the two method at the same time. For more details, please refer to here
Besides, in azure cli, it will read multiple "parameter" you provide then define one Jobject with these "parameter" as parameters when we deploy arm template. For more details, please refer to here
Related
I am trying to submit parallel jobs in a loop on HTCondor, following is a simple example of the python script -
test_mus = np.linspace(0, 5, 10)
results = [pyhf.infer.hypotest(test_mu, data, model)
for test_mu in test_mus]
I would like to submit each job (results), over the for loop (so 10 jobs) simultaneously to 10 machines, and then combine all the results in a pickle.
I have the submission script for this job as below -
executable = CLs.sh
+JobFlavour = "testmatch"
arguments = $(ClusterId) $(ProcId)
Input = LHpruned_NEW_32.json
output = output_sigbkg/CLs.$(ClusterId).$(ProcId).out
error = output_sigbkg/CLs.$(ClusterId).$(ProcId).err
log = output_sigbkg/CLs.$(ClusterId).log
transfer_input_files = CLs_test1point.py, LHpruned_NEW_32.json
should_transfer_files = YES
queue
I would like to know how to submit 10 jobs, to parallelize the jobs.
Thank you !
Best,
Shreya
It's worth looking at the description of how to submit a HTCondor DAG via Python here. In your case, if you install the htcondor Python module, you could do something like:
import htcondor
from htcondor import dags
# create the submit script
sub = htcondor.Submit(
{
"executable": "CLs.sh",
"+JobFlavour": "testmatch",
"arguments": "$(ClusterId) $(ProcId)",
"Input": "LHpruned_NEW_32.json",
"output": "output_sigbkg/CLs.$(ClusterId).$(ProcId).out",
"error": "output_sigbkg/CLs.$(ClusterId).$(ProcId).err",
"log": "output_sigbkg/CLs.$(ClusterId).log",
"transfer_input_files": "CLs_test1point.py, LHpruned_NEW_32.json",
"should_transfer_files": "YES",
}
)
# create DAG
dag = dags.DAG()
# add layer with 10 jobs
layer = dag.layer(
name="CLs_layer",
submit_description=sub,
vars=[{} for i in range(10)],
)
# write out the DAG to current directory
dagfile = dags.write_dag(dag, ".")
You can use the vars argument to add macros giving values for each specific job if you want, e.g., if you wanted mu as one of the executable arguments you could switch this to:
import htcondor
from htcondor import dags
# create the submit script
sub = htcondor.Submit(
{
"executable": "CLs.sh",
"+JobFlavour": "testmatch",
"arguments": "$(ClusterId) $(ProcId) $(MU)",
"Input": "LHpruned_NEW_32.json",
"output": "output_sigbkg/CLs.$(ClusterId).$(ProcId).out",
"error": "output_sigbkg/CLs.$(ClusterId).$(ProcId).err",
"log": "output_sigbkg/CLs.$(ClusterId).log",
"transfer_input_files": "CLs_test1point.py, LHpruned_NEW_32.json",
"should_transfer_files": "YES",
}
)
# create DAG
dag = dags.DAG()
# add layer with 10 jobs
layer = dag.layer(
name="CLs_layer",
submit_description=sub,
vars=[{"MU": mu} for mu in np.linspace(0, 5, 10)],
)
# write out the DAG to current directory
dagfile = dags.write_dag(dag, ".")
Once the DAG is created, you can either submit it as normal with condor_submit_dag from the terminal, or submit it via Python using the instructions here.
Note: your +JobFlavour value for the submit file will actually get converted to MY.JobFlavour in the file that gets created, but that's ok and means the same thing.
I would like to use the Docker image jupyter/datascience-notebook to start a Jupyter notebook frontend, but I need to be able to control which ports it chooses to use for its communication. I understand that the server is designed to potentially provision many kernels, not just one, but what I want is for the first kernel to use the ports I specify. I have tried supply arguments like:
docker run --rm -it jupyter/datascience-notebook:latest start-notebook.sh --existing /my/connection/file.json
docker run --rm -it jupyter/datascience-notebook:latest start-notebook.sh --KernelManager.control_port=60018
And it does not seem to care, instead creating the connection file in the usual location under /home/jovyan/.local/share/jupyter/
Any assistance is appreciated.
I ended up doing as suggested in a similar question - IPython notebook: How to connect to existing kernel? this, I could not find a better way.
Subclass LocalProvisioner to override its pre_launch method
from typing import Any, Dict
from jupyter_client import LocalProvisioner, LocalPortCache, KernelProvisionerBase
from jupyter_client.localinterfaces import is_local_ip, local_ips
class PickPortsProvisioner(LocalProvisioner):
async def pre_launch(self, **kwargs: Any) -> Dict[str, Any]:
"""Perform any steps in preparation for kernel process launch.
This includes applying additional substitutions to the kernel launch command and env.
It also includes preparation of launch parameters.
Returns the updated kwargs.
"""
# This should be considered temporary until a better division of labor can be defined.
km = self.parent
if km:
if km.transport == 'tcp' and not is_local_ip(km.ip):
raise RuntimeError(
"Can only launch a kernel on a local interface. "
"This one is not: %s."
"Make sure that the '*_address' attributes are "
"configured properly. "
"Currently valid addresses are: %s" % (km.ip, local_ips())
)
# build the Popen cmd
extra_arguments = kwargs.pop('extra_arguments', [])
# write connection file / get default ports
# TODO - change when handshake pattern is adopted
if km.cache_ports and not self.ports_cached:
lpc = LocalPortCache.instance()
km.shell_port = 60000
km.iopub_port = 60001
km.stdin_port = 60002
km.hb_port = 60003
km.control_port = 60004
self.ports_cached = True
km.write_connection_file()
self.connection_info = km.get_connection_info()
kernel_cmd = km.format_kernel_cmd(
extra_arguments=extra_arguments
) # This needs to remain here for b/c
else:
extra_arguments = kwargs.pop('extra_arguments', [])
kernel_cmd = self.kernel_spec.argv + extra_arguments
return await KernelProvisionerBase.pre_launch(self, cmd=kernel_cmd, **kwargs)
Specify entry point in setup.py
entry_points = {
'jupyter_client.kernel_provisioners': [
'pickports-provisioner = mycompany.pickports_provisioner:PickPortsProvisioner',
],
},
Create kernel.json to overwrite the default one
{
"argv": [
"/opt/conda/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3 (ipykernel)",
"language": "python",
"metadata": {
"debugger": true,
"kernel_provisioner": { "provisioner_name": "pickports-provisioner" }
}
}
Dockerfile
# Start from a core stack version
FROM jupyter/datascience-notebook:latest
# Install from requirements.txt file
COPY --chown=${NB_UID}:${NB_GID} requirements.txt .
COPY --chown=${NB_UID}:${NB_GID} setup.py .
RUN pip install --quiet --no-cache-dir --requirement requirements.txt
# Copy kernel.json to default location
COPY kernel.json /opt/conda/share/jupyter/kernels/python3/
# Install from sources
COPY --chown=${NB_UID}:${NB_GID} src .
RUN pip install --quiet --no-cache-dir .
Profit???
How to use pre-existing Amazon S3 bucket for source_artifact and cloud_assembly_artifact
Artifacts docs here my code is below
source_artifact = codepipeline.Artifact()
cloud_assembly_artifact = codepipeline.Artifact()
pipeline = pipelines.CdkPipeline(self, 'Pipeline',
cloud_assembly_artifact=cloud_assembly_artifact,
pipeline_name='testPipeline',
source_action = cpactions.CodeCommitSourceAction(
repository=codecommit.Repository.from_repository_name(
self,
id="cicd_pipeline",
repository_name="repo-name"
),
branch='master',
action_name='CodeCommit',
trigger=cpactions.CodeCommitTrigger.POLL,
output=source_artifact,
),
synth_action=pipelines.SimpleSynthAction(
source_artifact=source_artifact,
cloud_assembly_artifact=cloud_assembly_artifact,
install_command='npm install -g aws-cdk && pip install -r requirements.txt',
build_command='pytest unittests',
synth_command='cdk synth'))
You should create a codepipeline Pipeline and attach a artifact bucket there.
You can specify a codepipeline in the CdkPipeline construct after you have created the pipeline.
(Typescript but should be similar as python)
this._artifactsS3Bucket = Bucket.fromBucketArn(this, 'S3ArtifactsBucket', this._props.s3ArtifactBucketARN);
new Pipeline(this, 'Pipeline', {
artifactBucket: this._artifactsS3Bucket,
pipelineName: this._pipelineName,
restartExecutionOnUpdate: true
});
Attach this codePipeline to the construct of the cdkPipeline
I am attempting to upload an additional file containing an encryption secret to AWS Lambda, and am having trouble. The file is to be read from my Python script and be processed. I have tested this functionality locally, and it works just fine.
I package and upload the .zip correctly, as AWS has no problems running the script once it uploads. However, my code fails at the line that it reads my file, even though it should be in the working directory.
Is it possible to upload a file into the AWS zip deployment, and have it be read by the script?
I was surprised that this did not work, so I did some digging for anyone interest.
I created a simple function:
import json
import os
import random
def lambda_handler(event, context):
selection = str(random.randint(1,5))
with open('mydata.csv') as dogs:
for l in dogs:
if selection == l.split(',')[0]:
random_dog = l.split(',')[2]
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!'),
'cwd': os.getcwd(),
'ls': os.listdir(),
'random_dog': random_dog
}
A data file:
akc_popularity,year,breed
1,2019,labrador Retriever
2,2019,German Shepherd Dog
3,2019,Golden Retriever
4,2019,French Bulldog
5,2019,Bulldog
Added them to a zip archive:
$ zip fileImport.zip importer.py
$ zip fileImport.zip mydata.csv
Created the function:
$ aws lambda create-function --function-name fileImport --zip-file fileb://fileImport.zip --handler importer.lambda_handler --runtime python3.8 --role arn:aws:iam::***************:role/Lambda_StackExchange
Triggered the function:
$ aws lambda invoke --function-name fileImport output.json
{
"StatusCode": 200,
"ExecutedVersion": "$LATEST"
}
$ jq . output.json
{
"statusCode": 200,
"body": "\"Hello from Lambda!\"",
"cwd": "/var/task",
"ls": [
"importer.py",
"mydata.csv"
],
"random_dog": "Bulldog\n"
}
So, please share some code so we can dive in! FYI...I would highly recommend storing secrets using AWS Secrets Manager. It is very easy to use and keeps your hardcoded secrets out of things like version control systems. Additionally, changing your secret will not require a redeployment of your function.
How to pass user_data script to Python Heat-API client.
I have the following script in a file I want to pass into an instance as user_data during creating, but I am not sure
how to go about it doing. I am using the Heat API to create the instance. The below code creates the stack with the heat template file with no user_data.
Any pointers would be appreciated.
env.yml
user_data:
#!/bin/bash
rpm install -y git vim
template_file = 'heattemplate.yaml'
template = open(template_file, 'r')
stack = heat.stacks.create(stack_name='Tutorial', template=template.read(), parameters={})
On your yaml Heat template, you should add:
parameters:
install_command:
type: string
description: Command to run from user_data
default: #!/bin/bash rpm install -y git vim
...
myserver:
type: OS::Nova::Server
properties:
...
user_data_format: RAW
user_data: { get_param: install_command }
And pass the new parameter through parameters = {}, from your create line on Python:
heat.stacks.create(stack_name='Tutorial', template=template.read(),
parameters={ 'install_command': '...' })