We Keep Coding

Twint to Kafka -> Dont receive any events in my topic

i'm trying to load some tweets via TWINT into a kafka topic with the confluent-kafka[avro]
Producer. I don't get any errors but my topic wont receive any events from twint. I even get succes msg, when debugging(with try and except).
My Code:
import twint
import sys
import json
from time import sleep
from confluent_kafka import avro
from confluent_kafka.avro import AvroProducer
# Define Avro schema
value_schema_str = """
{
"namespace": "my.test",
"name": "value",
"type": "record",
"fields" : [
{ "name": "id", "type": "long" },
{ "name": "tweet", "type": "string" },
{ "name": "datetime", "type": "string" },
{ "name": "username", "type": "string" },
{ "name": "user_id", "type": "long" },
{ "name": "name", "type": "string" }
]
}
"""
key_schema_str = """
{
"namespace": "my.test",
"name": "key",
"type": "record",
"fields" : [
{
"name" : "name",
"type" : "string"
}
]
}
"""
kafka_broker = 'host.docker.internal:9092'
schema_registry = 'http://host.docker.internal:8081'
value_schema = avro.loads(value_schema_str)
key_schema = avro.loads(key_schema_str)
producer = AvroProducer({
'bootstrap.servers': kafka_broker,
'schema.registry.url': schema_registry
}, default_key_schema=key_schema, default_value_schema=value_schema)
module = sys.modules["twint.storage.write"]
def Json(obj, config):
tweet = obj.__dict__
tweet_new = {}
tweet_new['id'] = tweet['id']
tweet_new['tweet'] = tweet['tweet']
tweet_new['datetime'] = tweet['datetime']
tweet_new['username'] = tweet['username']
tweet_new['user_id'] = tweet['user_id']
tweet_new['name'] = tweet['name']
print(tweet_new)
try:
producer.produce(topic='tweets_test', key={"name": "Key"}, value=tweet_new)
except Exception as e:
print(f"Exception while producing record value - {tweet_new} to topic - tweets_test: {e}")
else:
print(f"Successfully producing record value - {tweet_new} to topic - tweets_test")
try:
producer.flush()
except Exception as e:
print(f"Exception while flush record value - {tweet_new} to topic - tweets_test: {e}")
else:
print(f"Successfully flushed record value - {tweet_new} to topic - tweets_test")
module.Json = Json
c2 = twint.Config()
c2.Search = "corona OR regen OR \"stark regen\" OR \"sturm\" OR überschwemmung OR landunter OR #hagel OR #regen OR #sturm OR flut"
c2.Store_json = True
c2.Custom["user"] = ["id", "tweet", "user_id", "username", "hashtags"]
c2.User_full = True
c2.Output = "tweets.json"
c2.Since = '2019-05-20'
c2.Hide_output = True
twint.run.Search(c2)
When i run it i get the following output:
{'id': 1513818741057937408, 'tweet': 'RKI: Bundesweite Sieben-Tage-Inzidenz steigt leicht auf 1087 | #corona #rki #test ', 'datetime': '2022-04-12 09:58:07 UTC', 'username': 'flashupde', 'user_id': 1179376986516606978, 'name': 'FLASH UP'}
Successfully producing record value - {'id': 1513818741057937408, 'tweet': 'RKI: Bundesweite Sieben-Tage-Inzidenz steigt leicht auf 1087 | #corona #rki #test ', 'datetime': '2022-04-12 09:58:07 UTC', 'username': 'flashupde', 'user_id': 1179376986516606978, 'name': 'FLASH UP'} to topic - tweets_test
Successfully flushed record value - {'id': 1513818741057937408, 'tweet': 'RKI: Bundesweite Sieben-Tage-Inzidenz steigt leicht auf 1087 | #corona #rki #test ', 'datetime': '2022-04-12 09:58:07 UTC', 'username': 'flashupde', 'user_id': 1179376986516606978, 'name': 'FLASH UP'} to topic - tweets_test
Any help how i can debug it better or any advice would be awesome.

TypeError when trying to set a DB value

I'm trying to get my Flask server to update to a Mongo Atlas DB on request. When a request is passed with the required arguments, it will post to a Discord webhook then attempt to update DB values.
Example: Going to https://(redacted for privacy)/(redacted for privacy)?iid=123&username=wow&price=22 with JUST the webhook code (doesn't touch DB) will do this:
(sent to discord webhook) and give a success message (outputs as 200 in the console).
But when I enable the DB code, the same link will through a 500 error. Anything I can do? The console outputs as;
balance = mycol.find_one({"UserID": uid})['balance']
TypeError: 'NoneType' object is not subscriptable
I don't understand why this won't work. Here's the code that works, but only posts to the webhook (commented stuff is DB related):
#balance = mycol.find_one({"UserID": uid})['balance']
#res = mycol.find_one({"UserID": uid})
#newvalues = { "$set": { "UserID": uid, "balance": int(balance) + int(cashback_amt)} }
#mycol.update_one(res, newvalues)
img_url = f"https://www.roblox.com/bust-thumbnail/image?userId={uid}&width=420&height=420&format=png"
data = {
"content" : "**New purchase!**",
"username" : "robate"
}
data["embeds"] = [
{
"description" : f"**ItemID**: `{item_id}`\n**Item Price**: `{price}`\n**Item Cashback**: `{cashback_amt}`",
"title" : "New purchase made",
"color": 65311,
"thumbnail": {"url": img_url}
}
]
result = requests.post(purclog, json = data)
return jsonify({"res": True})
And the non-working DB included code:
balance = mycol.find_one({"UserID": uid})['balance']
res = mycol.find_one({"UserID": uid})
newvalues = { "$set": { "UserID": uid, "balance": int(balance) + int(cashback_amt)} }
mycol.update_one(res, newvalues)
img_url = f"https://www.roblox.com/bust-thumbnail/image?userId={uid}&width=420&height=420&format=png"
data = {
"content" : "**New purchase!**",
"username" : "robate"
}
data["embeds"] = [
{
"description" : f"**ItemID**: `{item_id}`\n**Item Price**: `{price}`\n**Item Cashback**: `{cashback_amt}`",
"title" : "New purchase made",
"color": 65311,
"thumbnail": {"url": img_url}
}
]
result = requests.post(purclog, json = data)
return jsonify({"res": True})
Any help is severely appreciated. Have a good day!
EDIT: The UserID is defined here:
try:
uid = requests.get(f"https://api.roblox.com/users/get-by-username?username={username}").json()['Id']
except:
return
balance = mycol.find_one({"UserID": uid})['balance']

How to access different keys within a json file?

I'm calling an API service which returns JSON (with Czech language values) that looks like:
{
"model": "czech-morfflex-pdt-161115",
"acknowledgements": [
"http://ufal.mff.cuni.cz/morphodita#morphodita_acknowledgements",
"http://ufal.mff.cuni.cz/morphodita/users-manual#czech-morfflex-pdt_acknowledgements"
],
"result": [
[
{
"token": "Děti",
"analyses": [
{
"lemma": "dítě",
"tag": "POS=N|SubPOS=N|Gen=F|Num=P|Cas=1|Neg=A"
},
{
"lemma": "dítě",
"tag": "POS=N|SubPOS=N|Gen=F|Num=P|Cas=4|Neg=A"
},
{
"lemma": "dítě",
"tag": "POS=N|SubPOS=N|Gen=F|Num=P|Cas=5|Neg=A"
}
],
"space": " "
},
...
I want to return "lemma" value where "tag" values Cas=3
I tried:
import json
import os
import httpx
service_url = "http://lindat.mff.cuni.cz/services/morphodita/api"
output_format = "json"
model = "czech-morfflex"
text = "Děti pojedou k babičce Martě. Už se těší."
anal_service_url = "/".join([service_url, "analyze"])
params = {"output": output_format, "model": model, "data": text}
response = httpx.request("GET", anal_service_url, params=params)
response.raise_for_status()
response_dict = response.json()
result = response_dict.get("result")
print(type(result))
for res in result:
for a in res:
for b in a['analyses']:
for case in b['tag'][4]:
for i in [i for i,x in enumerate(case) if x == '3']:
print(i) # print position
But I don't know how to access "lemma" if case=3.
Help would be appreciated.

You can use an if statement to find the tag in the string:
case_tag = 'Cas=3'
for res_list in result:
for res_list_elem in res_list:
for item in res_list_elem['analyses']:
if case_tag in item['tag']:
print(item['lemma'])

Python to parse nested JSON values that can be null sometimes

I'm trying to parse the following and pull out primary_ip as a variable. Sometimes primary_ip is "null". Here is an example of the JSON, code and the most recent error I am getting.
{
"count": 67,
"next": "https://master.netbox.dev/api/dcim/devices/?limit=50&offset=50",
"previous": null,
"results": [
{
"id": 28,
"url": "https://master.netbox.dev/api/dcim/devices/28/",
"name": "q2",
"display_name": "q2",
"device_type": {
"id": 20,
"url": "https://master.netbox.dev/api/dcim/device-types/20/",
"manufacturer": {
"id": 15,
"url": "https://master.netbox.dev/api/dcim/manufacturers/15/",
"name": "Zyxel",
"slug": "zyxel"
},
"model": "GS1900",
"slug": "gs1900",
"display_name": "Zyxel GS1900"
},
"device_role": {
"id": 4,
"url": "https://master.netbox.dev/api/dcim/device-roles/4/",
"name": "Access Switch",
"slug": "access-switch"
},
"primary_ip": {
"id": 301,
"url": "https://master.netbox.dev/api/ipam/ip-addresses/301/",
"family": 4,
"address": "172.31.254.241/24"
},
Example Python
import requests
import json
headers = {
'Authorization': 'Token 63d421a5f733dd2c5070083e80df8b4d466ae525',
'Accept': 'application/json; indent=4',
}
response = requests.get('https://master.netbox.dev/api/dcim/sites/', headers=headers)
j = response.json()
for results in j['results']:
x=results.get('name')
y=results.get('physical_address')
response2 = requests.get('https://master.netbox.dev/api/dcim/devices', headers=headers)
device = response2.json()
for result in device['results']:
x=result.get('name')
z=result.get('site')['name']
# if result.get('primary_ip') != None
y=result.get('primary_ip', {}).get('address')
print(x,y,z)
I get the following error when I run it:
ubuntu#ip-172-31-39-26:~$ python3 Netbox-python
Traceback (most recent call last):
File "Netbox-python", line 22, in <module>
y=result.get('primary_ip', {}).get('address')
AttributeError: 'NoneType' object has no attribute 'get'

Which value is None? Is it the primary_ip or is it address ?
you could try the following:
y = result.get('primary_ip', {}).get('address, 'empty_address')
This will replace any None values with empty_address
Update:
I have just ran your code and got the following output:
LC1 123.123.123.123/24 site1
q1 172.31.254.254/24 COD
q2 172.31.254.241/24 COD
After running this:
import requests
import json
headers = {
"Authorization": "Token 63d421a5f733dd2c5070083e80df8b4d466ae525",
"Accept": "application/json; indent=4",
}
response = requests.get("https://master.netbox.dev/api/dcim/sites/", headers=headers)
j = response.json()
for results in j["results"]:
x = results.get("name")
y = results.get("physical_address")
response2 = requests.get("https://master.netbox.dev/api/dcim/devices", headers=headers)
device = response2.json()
for result in device["results"]:
x = result.get("name")
z = result.get("site")["name"]
if result.get("primary_ip") != None:
y = result.get("primary_ip").get("address")
print(x, y, z)
I am not sure of the expected output but the code doesn't throw any errors. From looking at the code, it seems there were a few indentation errors, where things didn't match up in terms of where they should have been indented.

AWS Lambda ERROR: AttributeError 'list' object has no attribute 'get'

I have a Lambda function that is designed to turn ON/OFF my Philip HUE lightbulbs. I am able to execute the python script & it runs (error-free) on my local machine. However, when I trigger the Lambda function (using an IoT Button) I get the following error message.
[ERROR] AttributeError: 'list' object has no attribute 'get'
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 21, in lambda_handler
bulbStatus = nested_get(data,["state","on"])
File "/var/task/lambda_function.py", line 16, in nestedDictLookup
internal_dict_value = internal_dict_value.get(k, None)
I think the error relates the the following line of code:
internal_dict_value = internal_dict_value.get(k, None)
However, I am almost certain that the "internal_dict_value" variable is a dictionary, NOT a list. To verify, I inserted the following line of code into my script:
internal_dict_value = input_dict
print (internal_dict_value)
Here is the output that I received:
{
"state": {
"on": true,
"bri": 254,
"hue": 8597,
"sat": 121,
"effect": "none",
"xy": [
0.4452,
0.4068
],
"ct": 343,
"alert": "select",
"colormode": "xy",
"mode": "homeautomation",
"reachable": false
},
"swupdate": {
"state": "noupdates",
"lastinstall": "2019-07-26T19:09:58"
},
"type": "Extended color light",
"name": "Couch Light",
"modelid": "LCT016",
"manufacturername": "Philips",
"productname": "Hue color lamp",
"capabilities": {
"certified": true,
"control": {
"mindimlevel": 1000,
"maxlumen": 800,
"colorgamuttype": "C",
"colorgamut": [
[
0.6915,
0.3083
],
[
0.1700,
0.7000
],
[
0.1532,
0.0475
]
],
"ct": {
"min": 153,
"max": 500
}
},
"streaming": {
"renderer": true,
"proxy": true
}
},
"config": {
"archetype": "sultanbulb",
"function": "mixed",
"direction": "omnidirectional",
"startup": {
"mode": "custom",
"configured": true,
"customsettings": {
"bri": 254,
"ct": 346
}
}
},
"uniqueid": "00:00:88:08:03:fd:4a:e2-0a",
"swversion": "1.46.13_r26312",
"swconfigid": "9DC82D22",
"productid": "Philips-LCT316-1-A17ECLv5"
}
Here is the script that I am working with. If you have any inspirational ideas, please share them! Thanks.
import requests,json
bridgeIP = "IP_Address_Here"
userID = "userID_here"
lightID = "4" #Represents the ID assigned to lightbulb, in the living room.
def lambda_handler(lightID, lambda_context):
url = f"http://{bridgeIP}/api/{userID}/lights/{lightID}"
r = requests.get(url)
data = json.loads(r.text)
def nested_get(input_dict, nested_key):
internal_dict_value = input_dict
for k in nested_key:
internal_dict_value = internal_dict_value.get(k, None)
if internal_dict_value is None:
return None
return internal_dict_value
bulbStatus = nested_get(data,{"state","on"})
#the nested_get() function captures the dict value, assigned to the "on" key.
#{"state":{"on":{True}} or {"state":{"on":{False}}
if bulbStatus == False:
r = requests.put(f"{url}/state", json.dumps({"on":True}))
elif bulbStatus == True:
r = requests.put(f"{url}/state", json.dumps({"on":False}))
lambda_handler(lightID, 4)
The last line in my script calls the lambda_handler() function. I'm told that I do not need this line because my Lambda calls the function when the Lambda Function is triggered. However I (believe) that I do need to manually call the function, when executing the script on my local machine.

The issue with changing the value of internal_dict_value during the loop is you dont know what type it's going to hold.
try something like if type(internal_dict_value) is dict:

With the help of #JakePearse I was able to resolve the issue.
Here is the final version of my (fully-functional) python script.
import requests,json
bridgeIP = "IP/FQDN_Here:port_here"
userID = "userHash_here"
lightID = "2"
def lambda_handler(event, lambda_context):
url = f"http://{bridgeIP}/api/{userID}/lights/{lightID}"
r = requests.get(url)
data = json.loads(r.text)
def nested_get(input_dict, nested_key):
internal_dict_value = input_dict
for k in nested_key:
if type(internal_dict_value) is dict:
internal_dict_value = internal_dict_value.get(k, None)
if internal_dict_value is None:
return None
return internal_dict_value
bulbStatus = nested_get(data,["state","on"])
if bulbStatus == False:
requests.put(f"{url}/state",json.dumps({"on":True}))
elif bulbStatus == True:
requests.put(f"{url}/state",json.dumps({"on":False}))
return {
'statusCode': 200,
'body': json.dumps('Mission accomplished!')
}