We Keep Coding

How to resolve the error: "columns[11]: custom variable cannot be found" when running an SA360 API call?

Heyy!!
Hope everyone is doing well,
I'm pulling data from SA360 (or DS3, Doubleclick Search), although I receive this error when I try to download the report:
"columns[11]: A custom variable named 'DDA Product Sign Ups' with
platform source 'FLOODLIGHT' cannot be found.">
I know:
That conversion exists on the platform UI (second result)
That my script works because when I take off the conversion field I can deploy my function with no prob.
My Script (more or less):
conversion_name = "DDA Product Sign Ups"
request = ds3_manager.reports().request(body =
{
"reportScope": {
"agencyId": agency_id,
"advertiserId" : advertiser_id },
"reportType": "adGroup",
"columns": [
{ "columnName": "date"},
{ "columnName": "accountType"},
{ "columnName": "account" },
{ "columnName": "cost" },
{ "columnName": "impr" },
{ "columnName": "clicks" },
{
"customMetricName" : conversion_name,
"platformSource": "floodlight"
}
],
"timeRange": {
"startDate": start_date,
"endDate": end_date
},
"downloadFormat": "csv",
"maxRowsPerFile": 6000000,
"statisticsCurrency": "agency"
}
)
When I google the issue I land on this web result: Set up custom Floodlight metrics and dimensions but I don't understand, to me, it is already set up, as I can add to it to my reports on the UI or on my webqueries already... So I'm not why it is not picked up by the script..
If anyone has an idea that would be greatly appreciated :D.
Best,
Alex

In order to include custom columns in a report request, you first need to find out how that custom column is defined (check this link). You can then include the custom column in the request (check this link).
You will find useful information on this topic in the "Optional: Request for Specific User-defined Column" section of this article.

Unable to attach Data disk to Azure VM

I created a shared disk following Create Disk and when i try to attach it to a VM Update VM i'm getting createOption cannot be changed. Below is the full error,
Disk attachment failed, request response is - {
"error": {
"code": "PropertyChangeNotAllowed",
"message": "Changing property 'dataDisk.createOption' is not allowed.",
"target": "dataDisk.createOption"
}
Request Body for Data Disk creation (please note this is a shared disk),
{
"location": LOCATION,
"sku": {
"name": "Premium_LRS"
},
"properties": {
"creationData": {
"createOption": "empty"
},
"osType": "linux",
"diskSizeGB": SIZE,
"maxShares": 5,
"networkAccessPolicy": "AllowAll"
}
}
Request body for VM Patch request,
{
"properties": {
"storageProfile": {
"dataDisks": [
{
"caching" : "ReadOnly",
"createOption": "Attach",
"lun": 0,
"managedDisk" : {
"id": disk_id, //-> this disk_id is id of the created disk above
"storageAccountType": "Premium_LRS"
}
}
]
}
}
}
can someone please point out where im doing wrong. I haven't found much documentation about shared disk attachment, through API.

As I see, there is no problem with your request body that updates the VM. I tried it right now and it works fine. I use the same request body as yours. So you need to check the disk again such as if the lun 0 is already in use.

ElasticSearch updating documents in an automated way

I don't know how to ask questions about my current problem so I think, that is why I couldn't find the right answer. So please let me tell you what my problem is.
I am trying to do a simple internet port scanner by using Zmap and Zgrab like Shodan.io,like censys.io etc.
I need to store the data inside of the ElasticSearch(because I want to learn how to use it).
In this case;
I have created a JSON architecture to use it in ElasticSearch such as
{
"value": "192.168.0.1",
"port": [
{
"value": 80,
"HTMLbody": "BODY OF THE WEB PAGE",
"status": 200,
"headers": {
"content_type": "html/png",
"content_length": 23123,
"...": "..."
},
"certificate": {
"company_names": [
"example.com",
"acme.io"
]
}
}
]
}
There will be approximately 4 billion IP address inside of the Elasticsearch with different ports open. My problem begins here; After first initial scan, I need to update the exist IP addresses.
For example;
IP: 192.168.0.1
port: 80 open
When in the second scan, I scan port 443 and It will be probably open too. Then I need to update my Elasticsearch document depends on the open ports.
What I found so far
There is an endpoint I found which is; POST /<index>/_update/<_id> but it updates a single document. I need to update at least more than 100.000 document in one scan. And it should be automatically too. How do I know that an ip address document id and update it ?
Also secondly, I found;
POST <index>/_update_by_query
I thought about searching the ip address by using query and gathering its index number and then updating the document as follows;
{
"value": "192.168.0.1",
"port": [
{
"value": 80,
"HTMLbody": "BODY OF THE WEB PAGE",
"status": 200,
"headers": {
"content_type": "html/png",
"content_length": 23123,
"...": "..."
},
"certificate": {
"company_names": [
"example.com",
"acme.io"
]
}
},
{
"value": 443,
"HTMLbody": "BODY OF THE SSL WEB PAGE",
"status": 200
}
]
}
In theory, I could do this but in practice couldn't write the code as efficient. Because I had at least 6 GB JSON file for one scan and it takes so long to process the whole file and updating elasticsearch.
Is there any way that solve this problem efficiently ?
Please Look the image below

To fully answer to the question, first, use the ip adress as _id to make sur to update the appropriate document.
You need to add an entry in the port array, so use the _update API to perform it like this :
POST <index>/_update/192.168.0.1
{
"script": {
"source": "ctx._source.port.add(params.tag)",
"lang": "painless",
"params": {
"tag": {
"value": 443,
"HTMLbody": "BODY OF THE SSL WEB PAGE",
"status": 200
}
}
}
}
This is an update query to add an entry to an array with a little script.
To use the bulk, simply add all of your request like explained here

Azure Error: data protection system cannot create a new key because auto-generation of keys is disabled

I am trying to run an azure function on my local machine using Visual Studio Code.
My main.py looks like this:
import logging
import azure.functions as func
def main(event: func.EventHubEvent):
logging.info('Python EventHub trigger processed an event: %s', event.get_body().decode('utf-8'))
My host.json file looks like this:
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
}
}
My function.json looks something like this:
{
"scriptFile": "main.py",
"bindings": [
{
"type": "eventHubTrigger",
"name": "event",
"direction": "in",
"eventHubName": "myhubName",
"connection": "myHubConnection",
"cardinality": "many",
"consumerGroup": "$Default"
}
]
}
The problem is when I run this, it throws me the following error:
A host error has occurred at
Microsoft.AspNetCore.DataProtection: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information. Microsoft.AspNetCore.DataProtection: The key ring does not contain a valid default protection key. The data protection system cannot create a new key because auto-generation of keys is disabled.
Value cannot be null.
Parameter name: provider
I am not sure what I am I missing ? Any help is appreciated

The problem was with the Azure Storage account:
Make sure the local.settings.json has the correct credentials for the storage account
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "MyStorageKey",
"FUNCTIONS_WORKER_RUNTIME": "python",
}
}

Localhost request from angular

Hello i have created a mini endpoint in Flask in python, and it's working well. When i access it from Postman :
http://127.0.0.1:5000/retrieve_data
it retrieves the data as a json.
But when i call the endpoint from angular:
export class HomeComponent implements OnInit {
title = 'angular-demo';
data = this.getAnnounce();
getAnnounce() {
return this.http.get('http://127.0.0.1:5000/retrieve_data');
}
constructor(private http: HttpClient) {
}
The actual result is:
{
"_isScalar": false,
"source": {
"_isScalar": false,
"source": {
"_isScalar": false,
"source": {
"_isScalar": true,
"value": {
"url": "http://127.0.0.1:5000/retrieve_data",
"body": null,
"reportProgress": false,
"withCredentials": false,
"responseType": "json",
"method": "GET",
"headers": {
"normalizedNames": {},
"lazyUp
Canceldate": null,
"headers": {}
},
"params": {
"updates": null,
"cloneFrom": null,
"encoder": {},
"map": null
},
"urlWithParams": "http://127.0.0.1:5000/retrieve_data"
}
},
"operator": {
"concurrent": 1
}
},
"operator": {}
},
"operator": {}
}
PS: i am very noob at anuglar

From the document of angular's http client, the return value's type of http.get() is Observable
So if you want to get the data that you server respond, you must subscribe as this:
data = {};
getAnnounce() {
this.http.get('http://127.0.0.1:5000/retrieve_data')
.subscribe(result => this.data = result);
}

Great question, and as some others have suggested it would seem that you're missing a .subscribe() on your getAnnounce() method. Angular Docs has a guide on observables that can help you get a better understanding.
I have a few more suggestions for improving your experience with angular. Take advantage of services for grouping together similar functionality. For example, in your provided code, you could move your getAnnounce() to a new service you create; something like AnnouncementService. Then you can use this in many places of your code. it also helps test it and find bugs later on since your code is more separated.
Another thing you can do is to move your server api address to the environment variables. By default if you built your angular project using the Angular CLI you'll find in our src folder an environments folder with two files by default environment.ts and environment.prod.ts. you can access this JSON object anywhere within your .ts files and when you build your code for product vs locally it can change values to what ever you set. In your case you would put your local API address in there http://127.0.0.1:5000.
export const environment = {
production: false,
api: 'http://127.0.0.1:5000'
};
Now you can access this easily and have one spot to change if you ever change your port number or have your api in a real server.
import { environment } from './environments/environment';
/* remember to import the non prod version of the environment file, angular knows how to switch them automatically when building for production or not. */
Hopefully this helps you with your coding, good luck!

You have to subscribe to your getAnnounce function.
For instance, you can try:
data = this.getAnnounce().subscribe();
Or directly on your constructor / ngOnInit:
ngOnInit() {
this.getAnnounce().subscribe(
data => this.data = data
);
}