Here is my python cdk code which create 2 rules "AWS-AWSManagedRulesCommonRuleSet" and "AWS-AWS-ManagedRulesAmazonIpReputationList".
In each rule there are child rules that i can change their Rule Actions to Count, the question is how can i add this to my code, i didn't find any good explanation for those child rules.
Added some changes but still doesn't work, i get this error:
Resource handler returned message: "Error reason: You have used none or multiple values for a field that requires exactly one value., field: RULE, parameter: Rule (Service: Wafv2, Status Code: 400, Request ID: 248d9235-bd01-49f4-963b-109bac2776c5, Extended Request ID: null)" (RequestToken: 8bb5****-****-3e95-****-
8e336ae3eed4, HandlerErrorCode: InvalidRequest)
the code:
class PyCdkStack(core.Stack):
def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
web_acl = wafv2.CfnWebACL(
scope_=self, id='WebAcl',
default_action=wafv2.CfnWebACL.DefaultActionProperty(allow={}),
scope='REGIONAL',
visibility_config=wafv2.CfnWebACL.VisibilityConfigProperty(
cloud_watch_metrics_enabled=True,
sampled_requests_enabled=True,
metric_name='testwafmetric',
),
name='Test-Test-WebACL',
rules=[
{
'name': 'AWS-AWSManagedRulesCommonRuleSet',
'priority': 1,
'statement': {
'RuleGroupReferenceStatement': {
'vendorName': 'AWS',
'name': 'AWSManagedRulesCommonRuleSet',
'ARN': 'string',
"ExcludedRules": [
{
"Name": "CrossSiteScripting_QUERYARGUMENTS"
},
{
"Name": "GenericLFI_QUERYARGUMENTS"
},
{
"Name": "GenericRFI_QUERYARGUMENTS"
},
{
"Name": "NoUserAgent_HEADER"
},
{
"Name": "SizeRestrictions_QUERYSTRING"
}
]
}
},
'overrideAction': {
'none': {}
},
'visibilityConfig': {
'sampledRequestsEnabled': True,
'cloudWatchMetricsEnabled': True,
'metricName': "AWS-AWSManagedRulesCommonRuleSet"
}
},
]
)
The Cfn- constructs are a one to one mapping to the cloudformation resources. You can simply check the docs for aws::wafv2::webacl.
For an example on how to exclude in cloudformation, see below. Note that object keys need to start with lowercase in order for CDK to process them.
{
"name": "AWS-AWSBotControl-Example",
"priority": 5,
"statement": {
"managedRuleGroupStatement": {
"vendorName": "AWS",
"name": "AWSManagedRulesBotControlRuleSet",
"excludedRules": [
{
"name": "CategoryVerifiedSearchEngine"
},
{
"name": "CategoryVerifiedSocialMedia"
}
]
},
"visibilityConfig": {
"sampledRequestsEnabled": true,
"cloudWatchMetricsEnabled": true,
"metricName": "AWS-AWSBotControl-Example"
}
}
This actually sets the two mentioned rules to Count mode. See https://docs.aws.amazon.com/waf/latest/developerguide/web-acl-rule-group-settings.html#web-acl-rule-group-rule-to-count. Note it sais:
Rules that you alter like this are described as being excluded rules in the rule group. If you have metrics enabled, you receive COUNT metrics for each excluded rule. This change alters how the rules in the rule group are evaluated.
Related
I have an Eve app publishing a simple read-only (GET) interface. It is interfacing a MongoDB collection called centroids, which has documents like:
[
{
"name":"kachina chasmata",
"location":{
"type":"Point",
"coordinates":[-116.65,-32.6]
},
"body":"ariel"
},
{
"name":"hokusai",
"location":{
"type":"Point",
"coordinates":[16.65,57.84]
},
"body":"mercury"
},
{
"name":"cañas",
"location":{
"type":"Point",
"coordinates":[89.86,-31.188]
},
"body":"mars"
},
{
"name":"anseris cavus",
"location":{
"type":"Point",
"coordinates":[95.5,-29.708]
},
"body":"mars"
}
]
Currently, (Eve) settings declare a DOMAIN as follows:
crater = {
'hateoas': False,
'item_title': 'crater centroid',
'url': 'centroid/<regex("[\w]+"):body>/<regex("[\w ]+"):name>',
'datasource': {
'projection': {'name': 1, 'body': 1, 'location.coordinates': 1}
}
}
DOMAIN = {
'centroids': crater,
}
Which will successfully answer to requests of the form http://hostname/centroid/<body>/<name>. Inside MongoDB this represents a query like: db.centroids.find({body:<body>, name:<name>}).
What I would like to do also is to offer an endpoint for all the documents of a given body. I.e., a request to http://hostname/centroids/<body> would answer the list of all documents with body==<body>: db.centroids.find({body:<body>}).
How do I do that?
I gave a shot by including a list of rules to the DOMAIN key centroids (the name of the database collection) like below,
crater = {
...
}
body = {
'item_title': 'body craters',
'url': 'centroids/<regex("[\w]+"):body>'
}
DOMAIN = {
'centroids': [crater, body],
}
but didn't work...
AttributeError: 'list' object has no attribute 'setdefault'
Got it!
I was assuming the keys in the DOMAIN structure was directly related to the collection Eve was querying. That is true for the default settings, but it can be adjusted inside the resources datasource.
I figured that out while handling an analogous situation as that of the question: I wanted to have an endpoint hostname/bodies listing all the (unique) values for body in the centroids collection. To that, I needed to set an aggregation to it.
The following settings give me exactly that ;)
centroids = {
'item_title': 'centroid',
'url': 'centroid/<regex("[\w]+"):body>/<regex("[\w ]+"):name>',
'datasource': {
'source': 'centroids',
'projection': {'name': 1, 'body': 1, 'location.coordinates': 1}
}
}
bodies = {
'datasource': {
'source': 'centroids',
'aggregation': {
'pipeline': [
{"$group": {"_id": "$body"}},
]
},
}
}
DOMAIN = {
'centroids': centroids,
'bodies': bodies
}
The endpoint, for example, http://127.0.0.1:5000/centroid/mercury/hokusai give me the name, body, and coordinates of mercury/hokusai.
And the endpoint http://127.0.0.1:5000/bodies, the list of unique values for body in centroids.
Beautiful. Thumbs up to Eve!
I am using the Python REST API of VSTS for TFS / Azure Dev Ops (https://github.com/Microsoft/azure-devops-python-api).
I would like to add attachments to some of the steps of my Test Cases like I can do in the Web Interface.
This is how I want my step to look like:
... and when you run it, that would look like this:
However, I have not been able to find where this information is stored.
This is the JSON data for the WorkItem of my Test Case
{
id: 224,
rev: 2,
fields: {
System.AreaPath: "GM_sandbox\GM-Toto",
System.TeamProject: "GM_sandbox",
System.IterationPath: "GM_sandbox",
System.WorkItemType: "Test Case",
System.State: "Design",
System.Reason: "New",
System.AssignedTo: "Jeff",
System.CreatedDate: "2019-01-03T01:43:09.743Z",
System.CreatedBy: "Jeff",
System.ChangedDate: "2019-01-03T02:12:07.15Z",
System.ChangedBy: "Jeff",
System.Title: "Titi",
Microsoft.VSTS.Common.StateChangeDate: "2019-01-03T01:43:09.743Z",
Microsoft.VSTS.Common.ActivatedDate: "2019-01-03T01:43:09.743Z",
Microsoft.VSTS.Common.ActivatedBy: "Jeff",
Microsoft.VSTS.Common.Priority: 2,
Microsoft.VSTS.TCM.AutomationStatus: "Not Automated",
Microsoft.VSTS.TCM.Steps: "<steps id="0" last="2"><step id="2" type="ValidateStep"><parameterizedString isformatted="true"><DIV><P>Click on the rainbow button</P></DIV></parameterizedString><parameterizedString isformatted="true"><P>Screen becomes Blue (see picture)</P></parameterizedString><description/></step></steps>"
},
_links: {
self: {
href: "https://my_server.com:8443/tfs/PRODUCT/23d89bd4-8547-4be3-aa73-13a30866f176/_apis/wit/workItems/224"
},
workItemUpdates: {
href: "https://my_server.com:8443/tfs/PRODUCT/_apis/wit/workItems/224/updates"
},
workItemRevisions: {
href: "https://my_server.com:8443/tfs/PRODUCT/_apis/wit/workItems/224/revisions"
},
workItemHistory: {
href: "https://my_server.com:8443/tfs/PRODUCT/_apis/wit/workItems/224/history"
},
html: {
href: "https://my_server.com:8443/tfs/PRODUCTi.aspx?pcguid=4107d6a2-eaaa-40b9-9a8d-f8fdbb31d4b7&id=224"
},
workItemType: {
href: "https://my_server.com:8443/tfs/PRODUCT/23d89bd4-8547-4be3-aa73-13a30866f176/_apis/wit/workItemTypes/Test%20Case"
},
fields: {
href: "https://my_server.com:8443/tfs/PRODUCT/23d89bd4-8547-4be3-aa73-13a30866f176/_apis/wit/fields"
}
},
url: "https://my_server.com:8443/tfs/PRODUCT/23d89bd4-8547-4be3-aa73-13a30866f176/_apis/wit/workItems/224"
}
Any idea on where this information is stored?
And, if you are familiar with the Python REST API, how to add an attachment from a file and link it to the test step?
Thanks a lot
Here is the flow using just the azure-devops-rest-api
Create the attachment:
Request:
POST https://dev.azure.com/{organization}/_apis/wit/attachments?fileName=info.txt&api-version=4.1
Body:
{"User text content to upload"}
Response:
{
"id": "f5016cf4-4c36-4bd6-9762-b6ad60838cf7",
"url": "https://dev.azure.com/{organization}/_apis/wit/attachments/f5016cf4-4c36-4bd6-9762-b6ad60838cf7?fileName=info.txt"
}
Create the Work Item:
Request:
PATCH https://dev.azure.com/{organization}/{project}/_apis/wit/workitems/$Test Case?api-version=4.1
Body:
[
{
"op": "add",
"path": "/fields/System.Title",
"from": null,
"value": "Sample test case"
},
{
"op": "add",
"path": "/fields/Microsoft.VSTS.TCM.Steps",
"value": "<steps id=\"0\" last=\"4\"><step id=\"2\" type=\"ActionStep\"><parameterizedString isformatted=\"true\"><DIV><P>test </P></DIV></parameterizedString><parameterizedString isformatted=\"true\"><DIV><P> </P></DIV></parameterizedString><description/></step><step id=\"3\" type=\"ActionStep\"><parameterizedString isformatted=\"true\"><DIV><DIV><P>test </P></DIV></DIV></parameterizedString><parameterizedString isformatted=\"true\"><DIV><P> </P></DIV></parameterizedString><description/></step><step id=\"4\" type=\"ActionStep\"><parameterizedString isformatted=\"true\"><DIV><P>test </P></DIV></parameterizedString><parameterizedString isformatted=\"true\"><DIV><P> </P></DIV></parameterizedString><description/></step></steps>"
},
{
"op": "add",
"path": "/relations/-",
"value": {
"rel": "AttachedFile",
"url": "https://dev.azure.com/{organization}/_apis/wit/attachments/f5016cf4-4c36-4bd6-9762-b6ad60838cf7?fileName=info.txt",
"attributes": {
"comment": "[TestStep=3]:",
"name": "info.txt"
}
}
}
]
The test case that is created will look like the below. There is something off with the step numbering for the number in the comment. Looks like it need to be +1 for the actual step you want to reference.
The key is to have in the attributes of the attached file, the comment with "[TestStep=3]:" as well as a name for the attachment.
In Python, that would give something like this:
Creating of attachment with function create_attachment
Updating a Test Case with url, comment, and filename
So something like that...
from vsts.work_item_tracking.v4_1.models.json_patch_operation import JsonPatchOperation
def add_attachment(wit_id: int, project: str, url:str, comment: str, step = 0, name = ""):
"""Add attachment already uploaded to a WorkItem
"""
# For linking the attachment to a step, we need to modify the comment and add a name
if step:
attributes = {
"comment":f"[TestStep={step}]:{comment}",
"name": name if name else re.sub(r".*fileName=", "", url)
}
else:
attributes = {"comment": comment}
patch_document = [
JsonPatchOperation(
op="add",
path="/relations/-",
value={
"rel": "AttachedFile",
"url": url,
"attributes": attributes,
},
)
]
return client.wit.update_work_item(patch_document, wit_id, project)
attachment = client_wit.create_attachment(stream, project, 'smiley.png')
add_attachment(tcid, project, attachment.url, 'Attaching file to work item', step=3)
Have this section in one large JSON file
"UserDetailList": [
{
"UserName": "citrix-xendesktop-ec2-provisioning",
"GroupList": [],
"CreateDate": "2017-11-07T14:20:14Z",
"UserId": "AIDAI2YJINPRUEM3XHKXO",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "AmazonEC2FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
],
"Arn": "arn:aws:iam::279052847476:user/citrix-xendesktop-ec2-provisioning"
},
Need to extract AttachedManagedPolicy.Policy name for user
Desired output:
"citrix-xendesktop-ec2-provisioning","AmazonEC2FullAccess"
"citrix-xendesktop-ec2-provisioning","AmazonS3FullAccess"
Some users don't have any policy at all so need some checking mechanism to avoid errors
with open('1.json') as file:
data = json.load(file)
for element in data['UserDetailList']:
s = element['UserName'], element['AttachedManagedPolicies']
print s
And getting
(u'citrix-xendesktop-ec2-provisioning', [{u'PolicyName': u'AmazonEC2FullAccess', u'PolicyArn': u'arn:aws:iam::aws:policy/AmazonEC2FullAccess'}, {u'PolicyName': u'AmazonS3FullAccess', u'PolicyArn': u'arn:aws:iam::aws:policy/AmazonS3FullAccess'}])
When added element['AttachedManagedPolicies']['PolicyName']
got: TypeError: list indices must be integers, not str
You are getting error because element['AttachedManagedPolicies'] is list not dictionary you need to iterate over element['AttachedManagedPolicies'] and then access key as below:
[i['PolicyName'] for i in element['AttachedManagedPolicies']]
this will construct list of values for key PolicyName
As you said you have very large JSON structure you might have empty values or not values and for that you can proceed as below:
d = {
"UserDetailList": [
{
"UserName": "citrix-xendesktop-ec2-provisioning",
"GroupList": [],
"CreateDate": "2017-11-07T14:20:14Z",
"UserId": "AIDAI2YJINPRUEM3XHKXO",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "AmazonEC2FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
],
"Arn": "arn:aws:iam::279052847476:user/citrix-xendesktop-ec2-provisioning"
}
]
}
user_list = d.get("UserDetailList", None) # if unable to fetch key then it will return None
if user_list:
for user_detail in user_list:
username = user_detail.get("UserName", None)
policies = [i.get('PolicyName') for i in user_detail.get('AttachedManagedPolicies', []) if i.get('PolicyName', None)] # empty list constructed if no policy exist
print(username, policies)
The problem
When run kur dump mnist.yml, the output has duplicated parts: training is duplicated part of train, testing is for test, evaluation is for evaluate and etc. See example below:
{
"evaluate": {
"data": [
{
"mnist": {
"images": {
"checksum": "8d422c7b0a1c1c79245a5bcf07fe86e33eeafee792b84584aec276f5a2dbc4e6",
"path": "~/kur",
"url": "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz"
},
"labels": {
"checksum": "f7ae60f92e00ec6debd23a6088c31dbd2371eca3ffa0defaefb259924204aec6",
"path": "~/kur",
"url": "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz"
}
}
}
],
"destination": "mnist.results.pkl",
"hooks": [
"mnist"
],
"provider": {
"num_batches": null
},
"weights": "mnist.w"
},
"evaluation": {
"data": [
{
"mnist": {
"images": {
"checksum": "8d422c7b0a1c1c79245a5bcf07fe86e33eeafee792b84584aec276f5a2dbc4e6",
"path": "~/kur",
"url": "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz"
...
So, how do we remove the duplicated parts from the output?
The point of having the duplicate section names is be more user-friendly. That way, someone can use train or training to define their training-related section, rather than thinking, "Hmm... I don't remember what the key is. Is it 'train' or 'training'? I guess I'll look in the documentation."
It is true that the JSON dump can be more confusing, though. On the other hand, it is also a more advanced feature. Maybe a compromise would be to hide the duplicate keys when JSON is dumped, but this could also lead to other forms of confusion (e.g., "why did Kur accept 'training' when it isn't even a valid key, nor does it appear in a dump?").
One solution
comment out parts of the lines like below in kur/kur/kurfile.py:
builtin = {
'settings' : ('settings', ),
'train' : ('train',),# 'training'),
'validate' : ('validate',),# 'validation'),
'test' : ('test', ), #'testing'),
'evaluate' : ('evaluate', ), #'evaluation'),
'templates' : ('templates', ),
'model' : ('model', ),
'loss' : ('loss', )
}
I have experimented through kur.Kurfile.parse and kur.Kurfile._parse_section, so far I don't see any side effect of doing so.
a= [{
"data" : {
"check": true,
},
"AMI": {
"status": 1,
"firewall":{
"status": enable
},
"d_suffix": "x.y.com",
"id": 4
},
"tags": [ #Sometime tags could be like "tags": ["default","auto"]
"default"
],
"hostname": "abc.com",
}
]
How to get a hostname on the basis of tags?I am trying to implement it using
for i in a:
if i['tags'] == 'default':
output = i['hostname']
but it's failing because 'tags' is a list which is not mapping to hostname key.Is there any way i can get hostname on the basis of 'tags'?
Use in to test if something is in a list. You also need to put default in quotes to make it a string.
for i in a:
if 'default' in i['tags']:
output = i['hostname']
break
If you only need to find one match, you should break out of the loop once you find it.
If you need to find multiple matches, use #phihag's answer with the list comprehension.
To get all hostnames tagged as default, use a list comprehension:
def_hostnames = [i['hostname'] for i in a if 'default' in i['tags']]
print('Default hostnames: %s' % ','.join(def_hostnames))
If you only want the first hit, either use def_hostnames[0] or the equivalent generator expression:
print('first: %s' % next(i['hostname'] for i in a if 'default' in i['tags']))
Your current code fails because it uses default, which is a variable named default. You want to look for a string default.
Make sure that you have everything in Json format like
a= [{
"data" : {
"check": True,
},
"AMI": {
"status": 1,
"firewall":{
"status": "enable"
},
"d_suffix": "x.y.com",
"id": 4
},
"tags": [
"default"
],
"hostname": "abc.com",
}
]
and then you can easily get it by using in
for i in a:
if 'default' in i['tags']:
output = i['hostname']