We Keep Coding

AVRO - JSON Enconding of messages in Confluent Python SerializingProducer vs. Confluent Rest Proxy with UNIONS

attached an example AVRO-Schema
{
"type": "record",
"name": "DummySampleAvroValue",
"namespace": "de.company.dummydomain",
"fields": [
{
"name": "ID",
"type": "int"
},
{
"name": "NAME",
"type": [
"null",
"string"
]
},
{
"name": "STATE",
"type": "int"
},
{
"name": "TIMESTAMP",
"type": [
"null",
"string"
]
}
]
}
Regarding the section "JSON Encoding" of the official AVRO-Specs - see: https://avro.apache.org/docs/current/spec.html#json_encoding - a JSON Message which validates against the above AVRO-Schema should look like the following because of the UNION-Types used:
{
"ID":1,
"NAME":{
"string":"Kafka"
},
"STATE":-1,
"TIMESTAMP":{
"string":"2022-04-28T10:57:03.048413"
}
}
When producing this message via Confluent Rest Proxy (AVRO), everything works fine, the data is accepted, validated and present in Kafka.
When using the "SearializingProducer" from the confluent_kafka Python Package, the example message is not accepted and only "regular" JSON works, e. g.:
{
"ID":1,
"NAME":"Kafka",
"STATE":-1,
"TIMESTAMP":"2022-04-28T10:57:03.048413"
}
Is this intended behaviour or am I doing something wrong? Can I tell the SerializingProducer to accept this encoding?
I need to hold open both ways to produce messages but the sending system can/want´s only to provide one of the above Payloads. Is there a way to use both with the same payload?
Thanks in advance.
Best regards

I am getting an error while using Python YouTube API

I am trying to create a Python program to get channel statisitics, but when I run it the YouTube API website gives this output (error):
{
"error": {
"code": 400,
"message": "'statisitcs'",
"errors": [
{
"message": "'statisitcs'",
"domain": "youtube.part",
"reason": "unknownPart",
"location": "part",
"locationType": "parameter"
}
]
}
}
This is my code:
class YTstats:
def __init__(self, api_key, channel_id):
self.api_key = api_key
self.channel_id = channel_id
self.channel_stats = None
def get_channel_statistics(self):
url = f'https://www.googleapis.com/youtube/v3/channels?part=statisitcs&id={self.channel_id}&key={self.api_key}'
print(url)
API_KEY = 'I cannot share my api key so I am not showing it but it is in my code'
yt = YTstats(API_KEY, 'UCbXgNpp0jedKWcQiULLbDTA')
yt.get_channel_statistics()

This problem is fixed now (it was a typo)

You have a typo here part=statisitcs - as you mentioned in your comment - and after looking closely to the code you provided in your question.
Next time, check closely your code and try to replicate the error using the try-it demo feature in the YouTube Data API documentation.
I do get the statistics of the channel_id you provided - that is: UCbXgNpp0jedKWcQiULLbDTA:
These are their statistics:
"statistics": {
"viewCount": "5642720",
"subscriberCount": "98400",
"hiddenSubscriberCount": false,
"videoCount": "158"
}
}
See the demo here.
Here is the full response:
{
"kind": "youtube#channelListResponse",
"etag": "lG-nYlbLnN81gtjVKe1zKPW6v7A",
"pageInfo": {
"totalResults": 1,
"resultsPerPage": 5
},
"items": [
{
"kind": "youtube#channel",
"etag": "j4Fo8qKbWrLHnQYB8sCI8_I4v9A",
"id": "UCbXgNpp0jedKWcQiULLbDTA",
"snippet": {
"title": "Python Engineer",
"description": "Free Python and Machine Learning Tutorials!\n\nHi, I'm Patrick. I’m a passionate Software Engineer who loves Machine Learning, Computer Vision, and Data Science. I create free content in order to help more people get into those fields. If you have any questions, feedback, or comments, just shoot me a message! I am happy to talk to you :)\n\nIf you like my content, please subscribe to the channel!\n\nPlease check out my website for more information:\nhttps://www.python-engineer.com\n\nIf you find these videos useful and would like to support my work you can find me on Patreon:\nhttps://www.patreon.com/patrickloeber\n\nLegal: https://www.python-engineer.com/legal-notice/\n",
"customUrl": "pythonengineer",
"publishedAt": "2019-05-03T11:22:33Z",
"thumbnails": {
"default": {
"url": "https://yt3.ggpht.com/ytc/AKedOLTs-Pvd4mvUi-m2rDLd8bzrKwS5a8C9HnDbkUDzHw=s88-c-k-c0x00ffffff-no-rj",
"width": 88,
"height": 88
},
"medium": {
"url": "https://yt3.ggpht.com/ytc/AKedOLTs-Pvd4mvUi-m2rDLd8bzrKwS5a8C9HnDbkUDzHw=s240-c-k-c0x00ffffff-no-rj",
"width": 240,
"height": 240
},
"high": {
"url": "https://yt3.ggpht.com/ytc/AKedOLTs-Pvd4mvUi-m2rDLd8bzrKwS5a8C9HnDbkUDzHw=s800-c-k-c0x00ffffff-no-rj",
"width": 800,
"height": 800
}
},
"localized": {
"title": "Python Engineer",
"description": "Free Python and Machine Learning Tutorials!\n\nHi, I'm Patrick. I’m a passionate Software Engineer who loves Machine Learning, Computer Vision, and Data Science. I create free content in order to help more people get into those fields. If you have any questions, feedback, or comments, just shoot me a message! I am happy to talk to you :)\n\nIf you like my content, please subscribe to the channel!\n\nPlease check out my website for more information:\nhttps://www.python-engineer.com\n\nIf you find these videos useful and would like to support my work you can find me on Patreon:\nhttps://www.patreon.com/patrickloeber\n\nLegal: https://www.python-engineer.com/legal-notice/\n"
}
},
"contentDetails": {
"relatedPlaylists": {
"likes": "",
"uploads": "UUbXgNpp0jedKWcQiULLbDTA"
}
},
"statistics": {
"viewCount": "5642720",
"subscriberCount": "98400",
"hiddenSubscriberCount": false,
"videoCount": "158"
}
}
]
}
Try to wait a few minutes after the call you made - probably, the API couldn't retrieve the data you requested due to excessive requests or the channel itself didn't have their statistics publicly available.

Triggering Azure Function based on ServiceBus and writing back not working (Python)

I have a Python Azure function that triggers based on messages to a topic, which works fine independently. However, if I then try to also write a message to a different ServiceBus Queue it doesn't work (as in the Azure Function won't even trigger if new messages are published to the topic). Feels like the trigger conditions aren't met when I include the msg_out: func.Out[str] component. Any help would be much appreciated!
__init.py
import logging
import azure.functions as func
def main(msg: func.ServiceBusMessage, msg_out: func.Out[str]):
# Log the Service Bus Message as plaintext
# logging.info("Python ServiceBus topic trigger processed message.")
logging.info("Changes are coming through!")
msg_out.set("Send an email")
function.json
{
"scriptFile": "__init__.py",
"entryPoint": "main",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"topicName": "publish-email",
"subscriptionName": "validation-sub",
"connection": "Test_SERVICEBUS"
},
{
"type": "serviceBus",
"direction": "out",
"connection": "Test_SERVICEBUS",
"name": "msg_out",
"queueName": "email-test"
}
]
}
host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
},
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
}
}
}
}

I can reproduce your problem, it seems to be caused by the following error:
Property sessionHandlerOptions is not allowed.
Deleting sessionHandlerOptions can be triggered normally.

How to send pushes with python using CleverTap API?

I am in the process of migrating my Parse push notification data to CleverTap. The problem that I am having is that the documentation is not very good about it and the examples that they give don't cover push notifications.
Anyone have an example of how to send pushes with CleverTap API using Python or at least evidence that this is possible?

The documentation now covers how to migrate your Parse installations to CleverTap. For example, say we are creating a push notification to be sent to users subscribed to the "yankees" and "mets" channels.
curl -H "Content-Type:application/json"
-H "X-CleverTap-Account-ID: Your CleverTap Account ID"
-H "X-CleverTap-Passcode: Your CleverTap Account Passcode"
--data "#payload"
"https://api.clevertap.com/1/targets/create.json"
Payload
"name": "My API Campaign",
"where": {
"common_profile_prop": {
"profile_fields": [
{
"name": "channels",
"value": ["yankees", "mets"]
}
]
}
},
"content":{
"title": "Breaking News",
"body": "Yankees Win 5-4",
"platform_specific": {
"ios": {
"deep_link": "example.com",
"sound_file": "example.caf",
"category": "reactive",
"badge_count": 1,
"key": "value_ios"
},
"android": {
"background_image": "http://example.jpg",
"default_sound": true,
"deep_link": "example.com",
"key": "value_android"
}
}
},
"devices": [
"android",
"ios",
],
"when": "now"
}
Response
{
"status": "success",
"id": 1457433898,
"estimates": {
"android": 90,
"ios": 300
}
}
Hope this helps. More documentation can be found here -
https://github.com/CleverTap/parse-migrate

Freebase Log in Python

I am new to freebase. Might be a simple answer but I cant seem to understand the freebase API. I try to run a simple query using the command queryResult = freebase.mqlread(queryFB); and I get the error below. I haven't logged in to the API. I created an account which is a google account. My question is when I call freebase.login(username,password); what do I use? Google account details? I am also attaching my query.
Query:
{
"type": "/food/dish",
"id": dish,
"name": None,
"cuisine":
[{
"id": None,
"name": None,
"region_of_origin":
[{
"id": None,
"name": None,
"/location/location/contains":
[{
"id": None,
"name": city,
"type": "/location/citytown",
"limit": 20
}]
}],
"optional": "optional"
}],
"type_of_dish1":
[{
"id": None,
"name": None,
"optional": "optional"
}]
}
Error:
{
"error":
{
"errors":
[
{
"domain": "usageLimits",
"reason": "userRateLimitExceededUnreg",
"message": "User Rate Limit Exceeded. Please sign up",
"extendedHelp": "https://code.google.com/apis/console"
}
],
"code": 403,
"message": "User Rate Limit Exceeded. Please sign up"
}
}

Sounds like you might be using an obsolete client library and/or API. The current API doesn't use a username/password pair, but rather an API key.

Try this. It will explain how to use api key for running MQL. You will need to get only the apikey for freebase. NO username or password is required.
https://developers.google.com/freebase/v1/mql-overview