We Keep Coding

AWS Python StepFunctions stepfunctions.steps.Parallel [Generate Definition]

is there an example of the Python AWS Data Science SDK for stepfunctions stepfunctions.steps.Parallel class implementation?
Parallel execution requires branches, but i cant seem to find the methods or documentation about their description.
Generating a synchronous list of steps works fine, but i cant find how to define the parallel step, anyone knows?
Are there any other libraries that can do this? boto3 as far as i looked doesnt have the functionality and CDK is not suitable, as this will be a service.
I'd like to be able to generate something like this by using just code:
{
"Comment": "Parallel Example.",
"StartAt": "LookupCustomerInfo",
"States": {
"LookupCustomerInfo": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "LookupAddress",
"States": {
"LookupAddress": {
"Type": "Task",
"Resource":
"arn:aws:lambda:us-east-1:123456789012:function:AddressFinder",
"End": true
}
}
},
{
"StartAt": "LookupPhone",
"States": {
"LookupPhone": {
"Type": "Task",
"Resource":
"arn:aws:lambda:us-east-1:123456789012:function:PhoneFinder",
"End": true
}
}
}
]
}
}
}

AVRO - JSON Enconding of messages in Confluent Python SerializingProducer vs. Confluent Rest Proxy with UNIONS

attached an example AVRO-Schema
{
"type": "record",
"name": "DummySampleAvroValue",
"namespace": "de.company.dummydomain",
"fields": [
{
"name": "ID",
"type": "int"
},
{
"name": "NAME",
"type": [
"null",
"string"
]
},
{
"name": "STATE",
"type": "int"
},
{
"name": "TIMESTAMP",
"type": [
"null",
"string"
]
}
]
}
Regarding the section "JSON Encoding" of the official AVRO-Specs - see: https://avro.apache.org/docs/current/spec.html#json_encoding - a JSON Message which validates against the above AVRO-Schema should look like the following because of the UNION-Types used:
{
"ID":1,
"NAME":{
"string":"Kafka"
},
"STATE":-1,
"TIMESTAMP":{
"string":"2022-04-28T10:57:03.048413"
}
}
When producing this message via Confluent Rest Proxy (AVRO), everything works fine, the data is accepted, validated and present in Kafka.
When using the "SearializingProducer" from the confluent_kafka Python Package, the example message is not accepted and only "regular" JSON works, e. g.:
{
"ID":1,
"NAME":"Kafka",
"STATE":-1,
"TIMESTAMP":"2022-04-28T10:57:03.048413"
}
Is this intended behaviour or am I doing something wrong? Can I tell the SerializingProducer to accept this encoding?
I need to hold open both ways to produce messages but the sending system can/want´s only to provide one of the above Payloads. Is there a way to use both with the same payload?
Thanks in advance.
Best regards

Triggering Azure Function based on ServiceBus and writing back not working (Python)

I have a Python Azure function that triggers based on messages to a topic, which works fine independently. However, if I then try to also write a message to a different ServiceBus Queue it doesn't work (as in the Azure Function won't even trigger if new messages are published to the topic). Feels like the trigger conditions aren't met when I include the msg_out: func.Out[str] component. Any help would be much appreciated!
__init.py
import logging
import azure.functions as func
def main(msg: func.ServiceBusMessage, msg_out: func.Out[str]):
# Log the Service Bus Message as plaintext
# logging.info("Python ServiceBus topic trigger processed message.")
logging.info("Changes are coming through!")
msg_out.set("Send an email")
function.json
{
"scriptFile": "__init__.py",
"entryPoint": "main",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"topicName": "publish-email",
"subscriptionName": "validation-sub",
"connection": "Test_SERVICEBUS"
},
{
"type": "serviceBus",
"direction": "out",
"connection": "Test_SERVICEBUS",
"name": "msg_out",
"queueName": "email-test"
}
]
}
host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
},
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
}
}
}
}

I can reproduce your problem, it seems to be caused by the following error:
Property sessionHandlerOptions is not allowed.
Deleting sessionHandlerOptions can be triggered normally.

Django Ceilometer get resource list and resource type

I am using Openstack Mitaka.
In Django I am trying to get all the resources for tenants. (This is ok)
After that I need to understand the resource type. For example if it is an instance or floating ip, etc.
def sync_resources():
logger.info("Executing sync_resources")
sync_tenants()
tenants = Tenant.objects.all()
managers = Manager.objects.filter(is_active=True)
for manager in managers:
services = manager.services.all()
regions = manager.region_set.all()
for region in regions:
ceilometer_driver = CeilometerDriver(region_name=region.name, **manager.ceilometer_params)
if ceilometer_driver.is_authenticated:
for tenant in tenants:
queries = [ceilometer_driver.make_query("project_id", ceilometer_driver.EQUAL, tenant.tenant_id)]
resource_list = ceilometer_driver.get_resource_list(query=queries)
In this code I get all the resources for a tenant. However I do not need all of them. And I will filter them based on list of types. Such as "Instance, floating_ip, volume, snapshot, etc"
Here is a sample resource detail:
sample resource floating_ip
from the metadata I can see that it is a floating_ip. However I cannot see somewhere in the details that its type floating ip. The metadata changes for different type of resources.
I alsoe checked from terminal with
$ ceilometer resource-list
$ ceilometer resource-show id-of-resource
Again the details are not helpful.
So I need a way to easily understand the resource type and match them with "Instance, floating ip, volume, snapshot, etc."
Additional info:
After some research I realized that I can get the type from link info. This doesn't work with "ceilometerclient" that I am using in the ceilometer_driver.
I can get the additional link info when I call directly from REST API.
Here is a URL to call single project resource:
http://192.168.101.11:8777/v2/resources?q.field=project_id&q.value=63c40f56e11a4d24981710fd46285233
In results below is a sample for instance. I removed most of the links. But one link shows that it is an instance.
{
"user_id": "ec0d0110f44f42608814106a1f921a16",
"links": [
{
"href": "http://192.168.101.11:8777/v2/resources/e4d30a8e-e339-4c47-b71f-ec7061aaf113",
"rel": "self"
},
{
"href": "http://192.168.101.11:8777/v2/meters/instance?q.field=resource_id&q.value=e4d30a8e-e339-4c47-b71f-ec7061aaf113",
"rel": "instance"
},
...
],
"resource_id": "e4d30a8e-e339-4c47-b71f-ec7061aaf113",
"source": "openstack",
"last_sample_timestamp": "2018-03-15T14:18:15.368000",
"first_sample_timestamp": "2017-09-20T11:58:06.704000",
"project_id": "63c40f56e11a4d24981710fd46285233",
"metadata": {
"instance_host": "node-6.reg1.test.skyatlas.com",
"ephemeral_gb": "0",
"flavor.vcpus": "1",
"flavor.ephemeral": "0",
"instance_id": "e4d30a8e-e339-4c47-b71f-ec7061aaf113",
"display_name": "fuel service plugin",
"state": "active",
"flavor.ram": "2048",
"status": "active",
"ramdisk_id": "None",
"flavor.name": "m1.small",
"disk_gb": "20",
"kernel_id": "None",
"image.id": "4905099d-7912-4b00-b5bd-d8e1698dee92",
"flavor.id": "2",
"host": "b26c476c766c058541258e0c9275a738b889e0242be2af62fb331422",
"OS-EXT-AZ.availability_zone": "test-1",
"image.name": "Ubuntu 16.04.3 LTS (Xenial Xerus)",
"image_ref_url": "http://192.168.101.11:8774/images/4905099d-7912-4b00-b5bd-d8e1698dee92",
"name": "instance-00000037",
"flavor.disk": "20",
"root_gb": "20",
"image.links": "[{'href': 'http://192.168.101.11:8774/images/4905099d-7912-4b00-b5bd-d8e1698dee92', 'rel': 'bookmark'}]",
"memory_mb": "2048",
"instance_type": "m1.small",
"vcpus": "1",
"image_ref": "4905099d-7912-4b00-b5bd-d8e1698dee92",
"flavor.links": "[{'href': 'http://192.168.101.11:8774/flavors/2', 'rel': 'bookmark'}]"
}
}
Here is another one for floating ip:
{
"user_id": null,
"links": [
{
"href": "http://192.168.101.11:8777/v2/resources/f61e437d-dce3-4562-bd48-e14a6f0604f3",
"rel": "self"
},
{
"href": "http://192.168.101.11:8777/v2/meters/ip.floating?q.field=resource_id&q.value=f61e437d-dce3-4562-bd48-e14a6f0604f3",
"rel": "ip.floating"
},
{
"href": "http://192.168.101.11:8777/v2/meters/ip.floating.create?q.field=resource_id&q.value=f61e437d-dce3-4562-bd48-e14a6f0604f3",
"rel": "ip.floating.create"
}
],
"resource_id": "f61e437d-dce3-4562-bd48-e14a6f0604f3",
"source": "openstack",
"last_sample_timestamp": "2018-03-15T14:13:01.808000",
"first_sample_timestamp": "2018-02-14T06:33:51.663000",
"project_id": "63c40f56e11a4d24981710fd46285233",
"metadata": {
"status": "DOWN",
"router_id": "None",
"floating_network_id": "3a15b9a4-8f93-4622-80e6-d57cfee53b43",
"fixed_ip_address": "None",
"floating_ip_address": "192.168.103.132",
"port_id": "None"
}
}
so an instance has a link with
"rel": "instance"
and a floating ip has a link with
"rel": "ip.floating"
I think I can sort them out by REST API call instead of using the driver.
However this so called solution does not seem to me the right way to go.
So I still look for a better way of doing it. All the answers and inputs are welcome.

How to send pushes with python using CleverTap API?

I am in the process of migrating my Parse push notification data to CleverTap. The problem that I am having is that the documentation is not very good about it and the examples that they give don't cover push notifications.
Anyone have an example of how to send pushes with CleverTap API using Python or at least evidence that this is possible?

The documentation now covers how to migrate your Parse installations to CleverTap. For example, say we are creating a push notification to be sent to users subscribed to the "yankees" and "mets" channels.
curl -H "Content-Type:application/json"
-H "X-CleverTap-Account-ID: Your CleverTap Account ID"
-H "X-CleverTap-Passcode: Your CleverTap Account Passcode"
--data "#payload"
"https://api.clevertap.com/1/targets/create.json"
Payload
"name": "My API Campaign",
"where": {
"common_profile_prop": {
"profile_fields": [
{
"name": "channels",
"value": ["yankees", "mets"]
}
]
}
},
"content":{
"title": "Breaking News",
"body": "Yankees Win 5-4",
"platform_specific": {
"ios": {
"deep_link": "example.com",
"sound_file": "example.caf",
"category": "reactive",
"badge_count": 1,
"key": "value_ios"
},
"android": {
"background_image": "http://example.jpg",
"default_sound": true,
"deep_link": "example.com",
"key": "value_android"
}
}
},
"devices": [
"android",
"ios",
],
"when": "now"
}
Response
{
"status": "success",
"id": 1457433898,
"estimates": {
"android": 90,
"ios": 300
}
}
Hope this helps. More documentation can be found here -
https://github.com/CleverTap/parse-migrate