We Keep Coding

props.source.bindAsNotificationRuleSource is not a function in aws cdk python

I am trying to set up notification to my codepipeline in aws.
Been following this https://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_codestarnotifications/README.html
pipeline = CodePipeline(
self,
id,
pipeline_name=id,
synth=synth_step,
cross_account_keys=True,
code_build_defaults=pipelines.CodeBuildOptions(
build_environment=BuildEnvironment(
build_image=aws_codebuild.LinuxBuildImage.STANDARD_5_0,
privileged=True,
)
),
)
after creating my code pipeline with in the stack i am creating a notification rule.
rule = aws_codestarnotifications.NotificationRule(self, "NotificationRule",
source=pipeline,
events=["codepipeline-pipeline-pipeline-execution-failed", "codepipeline-pipeline-pipeline-execution-succeeded"
],
targets=[sns_topic]
)
but i am getting RuntimeError: props.source.bindAsNotificationRuleSource is not a function.
I also tried solution mentioned here, but didn't workout.
https://github.com/aws/aws-cdk/issues/9710
Does anyone has an idea on it? where am i going wrong?

The issue was with
source=pipeline,
Here the source in notification rule is expecting an pipeline ARN.
As we are working with CDK we need to make sure the pipeline is built before we set up the notification stack.
To over come this we need to build the pipeline and then set up the notification rule.
Use pipeline.build_pipeline() after pipeline and before notification rule code.
This worked for me.
also please refer active thread for detailed explanation.

Discovered resources returning a blank list even when resources exist AWS

I've been trying all the resources in AWS , using the following code :
session = boto3.Session()
client =
session.client('config',aws_access_key_id=access_key_id,aws_secret_access_key=secret_key,region_name=region)
resources =["AWS::EC2::CustomerGateway","AWS::EC2::EIP","AWS::EC2::Host","AWS::EC2::Instance","AWS::EC2::InternetGateway","AWS::EC2::NetworkAcl","AWS::EC2::NetworkInterface","AWS::EC2::RouteTable","AWS::EC2::SecurityGroup","AWS::EC2::Subnet","AWS::CloudTrail::Trail","AWS::EC2::Volume","AWS::EC2::VPC","AWS::EC2::VPNConnection","AWS::EC2::VPNGateway","AWS::EC2::RegisteredHAInstance","AWS::EC2::NatGateway","AWS::EC2::EgressOnlyInternetGateway","AWS::EC2::VPCEndpoint","AWS::EC2::VPCEndpointService","AWS::EC2::FlowLog","AWS::EC2::VPCPeeringConnection","AWS::IAM::Group","AWS::IAM::Policy","AWS::IAM::Role","AWS::IAM::User","AWS::ElasticLoadBalancingV2::LoadBalancer","AWS::ACM::Certificate","AWS::RDS::DBInstance","AWS::RDS::DBParameterGroup","AWS::RDS::DBOptionGroup","AWS::RDS::DBSubnetGroup","AWS::RDS::DBSecurityGroup","AWS::RDS::DBSnapshot","AWS::RDS::DBCluster","AWS::RDS::DBClusterParameterGroup","AWS::RDS::DBClusterSnapshot","AWS::RDS::EventSubscription","AWS::S3::Bucket","AWS::S3::AccountPublicAccessBlock","AWS::Redshift::Cluster","AWS::Redshift::ClusterSnapshot","AWS::Redshift::ClusterParameterGroup","AWS::Redshift::ClusterSecurityGroup","AWS::Redshift::ClusterSubnetGroup","AWS::Redshift::EventSubscription","AWS::SSM::ManagedInstanceInventory","AWS::CloudWatch::Alarm","AWS::CloudFormation::Stack","AWS::ElasticLoadBalancing::LoadBalancer","AWS::AutoScaling::AutoScalingGroup","AWS::AutoScaling::LaunchConfiguration","AWS::AutoScaling::ScalingPolicy","AWS::AutoScaling::ScheduledAction","AWS::DynamoDB::Table","AWS::CodeBuild::Project","AWS::WAF::RateBasedRule","AWS::WAF::Rule","AWS::WAF::RuleGroup","AWS::WAF::WebACL","AWS::WAFRegional::RateBasedRule","AWS::WAFRegional::Rule","AWS::WAFRegional::RuleGroup","AWS::WAFRegional::WebACL","AWS::CloudFront::Distribution","AWS::CloudFront::StreamingDistribution","AWS::Lambda::Alias","AWS::Lambda::Function","AWS::ElasticBeanstalk::Application","AWS::ElasticBeanstalk::ApplicationVersion","AWS::ElasticBeanstalk::Environment","AWS::MobileHub::Project","AWS::XRay::EncryptionConfig","AWS::SSM::AssociationCompliance","AWS::SSM::PatchCompliance","AWS::Shield::Protection","AWS::ShieldRegional::Protection","AWS::Config::ResourceCompliance","AWS::LicenseManager::LicenseConfiguration","AWS::ApiGateway::DomainName","AWS::ApiGateway::Method","AWS::ApiGateway::Stage","AWS::ApiGateway::RestApi","AWS::ApiGatewayV2::DomainName","AWS::ApiGatewayV2::Stage","AWS::ApiGatewayV2::Api","AWS::CodePipeline::Pipeline","AWS::ServiceCatalog::CloudFormationProvisionedProduct","AWS::ServiceCatalog::CloudFormationProduct","AWS::ServiceCatalog::Portfolio"]
for resource in resources:
response = client.list_discovered_resources(resourceType=resource)
print(response)
print('##################### {} #################'.format(resource))
for i in range(len(response['resourceIdentifiers'])-1):
print( '{} , {}'.format(response['resourceIdentifiers'][i]['resourceType'],response['resourceIdentifiers'][i]['resourceId']))
This has worked for me before, however running it now gives out the resourceidentifiers as a blank list,even though the resources exist.
The only difference is that previously I queried using Root user and now from IAM user. Does it cause any difference?

The most likely answer is that you need to appropriate set the region in your boto3 session to match where your resources are.

You have to set up AWS Config now, as it is no longer a free service.

Azure Databricks error: AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Server failed to authenticate the request

I am trying to execute this command and in a lot of scenarios, it returns me this error.
spark.sql("""drop table if exists """ + L2_STAGE + """.STERLING_INCREMENT_STAGE_DELTA""") dbutils.fs.rm("dbfs:/mnt/data/governed/l2/stage/order/dtc_mergedstage/", true) spark.sql("""drop table if exists """ + L2_STAGE + """.sterling_increment_stage_delta_ol""") dbutils.fs.rm("dbfs:/mnt/data/governed/l2/stage/order/order_line_stg/", true) spark.sql("""drop table if exists """ + L2_STAGE + """.STERLING_INCREMENT_STAGE_DELTA_str""") dbutils.fs.rm("dbfs:/mnt/data/governed/l2/stage/order/dtc_mergedstage_str/", true) spark.sql("""drop table if exists """ + L2_STAGE + """.sterling_increment_stage_delta_ol_str""") dbutils.fs.rm("dbfs:/mnt/data/governed/l2/stage/order/order_line_stg_str/", true)
This returns me this error:
AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Caused by: StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Can someone let me know what is wrong? The administrator says that the access levels for all my colleagues are same and he doesn't know how to resolve. I don't know if this is a problem to be resolved from Databricks or through Azure Portal.
I also realize that:
Error occurs when I try to read from a table, or when I try to drop a table. But I can create tables.
I can perform this command:
%fs
ls /mnt/edap/data/governed
without error.
But,
%fs
ls /mnt/data/governed
gives same error.
**'edap' is an Azure Blob container.
I hope someone can help resolve that error.

I wasn't quite aware and noticed that I had mounted a drive and only had to unmount it in order to get things working. I had to navigate in DBFS to the path where I had mounted. The mounted drive a had a different icon, like a container or a flask, while the others had a folder icon. I clicked on the drop down and unmounted it. This solved my problem.

I am assuming that the path /mnt/data/ should point to a storage container somewhere in your organization's Azure tenancy. Further, I am assuming that you intend for the data to live on containers that your organization controls.
I believe that you have not solved your problem. At present, your tables are most likely being written to a file path in dbfs: (the container set aside for your Databricks instance) rather than to your organization's Azure containers.
The command %fs ls /mnt/edap/data/governed works because it is pointed at the /mnt/edap container, which is correctly mounted. Similarly, %fs ls /mnt/data/ will now work because it is pointed at a directory in dbfs: where permissions are not an issue.
The problem is that the container for /mnt/data/ was not mounted correctly in the first place.
Here is the mistake I made that generated the same error:
I attempted to mount the container using the instructions here: https://docs.databricks.com/data/data-sources/azure/azure-storage.html
However, the container I was attempting to mount existed in a Azure Gen2 datalake. The correct method of mounting the container can be found here: https://docs.databricks.com/data/data-sources/azure/adls-gen2/azure-datalake-gen2-sp-access.html
If the container in question is truly just a blob storage instance, double check your code for mounting the container.

I am not able to pass parameter in aws batch job?

I have a Dockerfile with the following lines:
FROM python
COPY sysargs.py /
CMD python sysargs.py --date Ref::date
And my python file sysargs.py looks like this:
import sys
print('The command line arguments are:')
a = sys.argv[1]
print(a)
I just want to pass parameter date and print the date but after passing date value I am getting output as "Ref::date".
Can someone help me what I have done wrong?
I am trying to replicate as mentioned in how to retrieve aws batch parameter value in python?.

In the context of your Dockerfile Ref::date is just a string, hence that is what is printed as output in the python script.
If you want date to be a value passed in from an external source, then you can use ARG. Take 10 minutes to read through this guide.

My experience with AWS Batch "parameters"
I have been working on a project for about 4 months now. One of my tasks was to connect several AWS services together to process in the Cloud the last uploaded file that an application had placed in a S3 bucket.
What I needed
The way this works is the following. Through a website, a user uploads a file that is sent to a back-end server, and then to a S3 bucket. This event triggers a AWS Lambda function, which inside creates and runs an instance of a AWS Batch Job, that has already been defined previously (based on a Docker image) and would retrieve from the S3 bucket the file to process it and the save in a database some results. By the way, all the code I am using is done with Python.
Everything worked as charm until I found it really hard to get as a parameter the filename of the file in the S3 bucket that generated the event, inside the python script that was being executed inside the Docker container, run by the AWS Batch Job.
What I did
After a lot of research and development, I came up with a solution for my problem. The issue was based on the fact that the word "parameter", for AWS Batch Jobs, is not what a user may expect. In return, we need to use containerOverrides, the way I show below: defining an "environment" variable value inside the running container by providing a pair of name and value of that variable.
# At some point we had defined aws_batch like this:
#
#aws_batch = boto3.client(
# service_name="batch",
# region_name='<OurRegion>',
# aws_access_key_id='<AWS_ID>',
# aws_secret_access_key='<AWS_KEY>',
#)
aws_batch.submit_job(
jobName='TheJobNameYouWant',
jobQueue='NameOfThePreviouslyDefinedQueue',
jobDefinition='NameOfThePreviouslyDefinedJobDefinition',
# parameters={ #THIS DOES NOT WORK
# 'FILENAME': FILENAME #THIS DOES NOT WORK
# }, #THIS DOES NOT WORK
containerOverrides={
'environment': [
{
'name': 'filename',
'value': 'name_of_the_file.png'
},
],
},
)
This way, from my Python script, inside the Docker container, I could access the environment variable value using the well-known os.getenv('<ENV_VAR_NAME>') function.
You can also check on your AWS console, under the Batch menu, both Job configuration and Container details tabs, to make sure everything makes sense. The container that the Job is running will never see the Job parameters. In the opposite way, it will know the environment variables.
Final notes
I do not know if there is a better way to solve this. So far, I share with all the community something that does work.
I have tested it myself, and the main idea came from reading the links that I list below:
AWSBatchJobs parameters (Use just as context info)
submit_job function AWS Docs (Ideal to learn about what kind of actions we are allowed to do or configure when creating a job)
I honestly hope this helps you and wish you a happy coding!

Referencing output of a template

I have a Deployment Manager script as follows:
cluster.py creates a kubernetes cluster and when the script was run only for the k8 cluster creation, it was successful -- so it means the cluster.py had no issues in creation of a k8 cluster
cluster.py also exposes ouputs:
A small snippet of the cluster.py is as follows:
outputs.append({
'name': 'v1endpoint' ,
'value': type_name + type_suffix })
return {'resources': resources, 'outputs': outputs}
If I try to access the exposed output inside dmnginxservice resource below as $(ref.dmcluster.v1endpoint) I get an error as resource not found
imports:
- path: cluster.py
- path: nodeport.py
resources:
- name: dmcluster
type: cluster.py
properties:
zone: us-central1-a
- name: dmnginxservice
type: nodeport.py
properties:
cluster: $(ref.dmcluster.v1endpoint)
image: gcr.io/pr1/nginx:latest
port: 342
nodeport: 32123
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1519960432614-566655da89a70-a2f917ad-69eab05a]: errors:
- code: CONDITION_NOT_MET
message: Referenced resource yaml%dmcluster could not be found. At resource
gke-cluster-dmnginxservice.

I tried to reproduce a similar implementation and I have been able to deploy it with no issues making use of your very same sintax for the output.
I deployed 2 VM and a new network. I will post you my code, maybe you find some interesting hints concerning the outputs.
The first VM pass as output the name for the second VM and use a reference from the network
The second VM takes the name from the properties that have been populated from the output of the first VM
the network thanks to the references is the first one to be created.
Keep in mind that:
This can get tricky because the order of creation for resources is important; you cannot add virtual machine instances to a network that does not exist, or attach non-existent persistent disks. Furthermore, by default, Deployment Manager creates all resources in parallel, so there is no guarantee that dependent resources are created in the correct order.
I will skip that is the same. If you provide your code I could try to help you to debug it, but from the error code it seems that the DM is not aware that the first element has been created, but from the info provided is not clear why.
Moreover if I were you I would give a shot to explicitly set that dmnginxservice depends on dmcluster making use of the metadata. In this way you can double check if it is actually waiting the first resource.
UPDATE
I have been able to reproduce the bug with a simpler configuration basically depending on how I reference the variables, the behaviour is different and for some reason the property get expanded to $(ref.yaml%vm-1.paolo), it seems that the combination of project and cluster references causes troubles.
#'name': context.properties["debug"],WORKING
#'name': context.env["project"],WORKING
'name': context.properties["debug"]+context.env["project"],#NOT WORKING
You can check the configuration here, If you need it.