We Keep Coding

Mastodon API - /api/v2/search - Not getting expected results

My objective is to perform a search of Mastodon statuses and return the content (i.e. the text) of any status that matches. The docs suggest I can do this. Can I actually do this?
My python code is
import requests
url = 'https://<server>/api/v2/search'
auth = {'Authorization': 'Bearer <token>'}
params = {'q': '<keyword>', 'type':'statuses'}
response = requests.get(url, data=params, headers=auth)
First issue: I get the following response (no matter what keyword I choose and even when my keyword clearly appears in a recent status):
{'accounts': [], 'statuses': [], 'hashtags': []}
Second issue: If I don't restrict the search to statuses, I get results! But they are not what I expect. There's no content key :( While the example results do contain a content key, which was my goal.
{'accounts': [], 'statuses': [], 'hashtags': [{'name': 'hiring', 'url': 'https://data-folks.masto.host/tags/hiring', 'history': [{'day': '1675814400', 'accounts': '5', 'uses': '5'}, {'day': '1675728000', 'accounts': '10', 'uses': '13'}, {'day': '1675641600', 'accounts': '7', 'uses': '7'}, {'day': '1675555200', 'accounts': '6', 'uses': '6'}, {'day': '1675468800', 'accounts': '3', 'uses': '3'}, {'day': '1675382400', 'accounts': '5', 'uses': '6'}, {'day': '1675296000', 'accounts': '9', 'uses': '9'}], 'following': False}]}
Thanks for any help! I'm a beginner and truly appreciate it.

This question was answered for me by the user trwnh over on Mastodon's github discussions, so I am copying it here:
"Full-text search across all statuses is not supported. If your server
has configured the optional Elasticsearch backend, then you can
perform limited full-text search against your own posts, favourites,
and bookmarks -- basically, only posts relevant to you. To obtain
content based on a keyword, you must use hashtags."

Getting Error while Iterating over Dictionary in Python [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
My dictionary is below; I get an error while iterating over it.
mk = {'Incident': {'AlertStatus': 'SLA BREACH',
'Area': 'Test',
'Assignee': 'Incident.Coordinator',
'AssignmentGroup': 'Operating System Support (North America)',
'Category': 'incident',
'Contact': 'ALSTON, LOU',
'Description': ['Test - Request - 1 , Test - Request - 1, Test - '
'Request - 1Test - Request - 1Test - Request - '
'1Test - Request - 1Test - Request - 1Test - '
'Request - 1Test - Request - 1'],
'Impact': '2',
'IncidentID': 'IM10265',
'OpenTime': '2020-04-09T08:16:16+00:00',
'OpenedBy': 'rf',
'Phase': 'Categorization',
'Service': 'CI1001032',
'Source': '2',
'Status': 'Categorize',
'Subarea': 'Test',
'Title': 'Test - Request - 1',
'UpdatedBy': 'rf',
'UpdatedTime': '2020-04-09T08:16:25+00:00',
'Urgency': '3'},
'Messages': [],
'ReturnCode': 0}
def extract_val():
id_data = []
Assignee_data = []
id_datas = [q['Incident']['IncidentID'] for q in mk]
Assignee_datas = [t['Incident']['Assignee'] for t in mk]
print(id_datas)
print(Assignee_datas)
extract_val()
getting error as : TypeError: string indices must be integers
Though I'm using key (Incident) to then the other keys like :(Incident), (Assignee) to extract values, still getting error. Please suggest what I'm missing here

If mk is list of dicts try this.
mk = [{'Incident': {'AlertStatus': 'SLA BR', 'Area': 'Test', 'Assignee': 'Incident.Coordinator','Category': 'incident', 'Contact': 'LU', 'Impact': '2', 'IncidentID': 'IM10265', 'OpenedBy': 'rf', 'Phase': 'OP', 'Service': 'CI102', 'Urgency': '3'}, 'Messages': [], 'ReturnCode': 0}, {'Incident': {'AlertStatus': 'SLA AC', 'Area': 'Test', 'Assignee': 'Cris.Bros', 'AssignGroup': 'FCI', 'Category': 'incident', 'Contact': 'AN', 'Description': ['Test-Request-2'], 'IncidentID': 'IM10266', 'Status': 'WI', 'Subarea': '3', 'Urgency': '1'}, 'Messages': [], 'ReturnCode': 0}]
def extract_val():
id_datas , Assignee_datas = map(list, zip(*[(q['Incident']['IncidentID'], q['Incident']['Assignee']) for q in mk]))
print(id_datas)
print(Assignee_datas)
extract_val()
Output:
['IM10265', 'IM10266']
['Incident.Coordinator', 'Cris.Bros']

Let's print and you'll see your problem...
>>> for q in mk:
... print(q)
...
Incident
Messages
ReturnCode
Wen looping through dict like this, you loop through keys.
But I suppose your problem is that you are excepting to have list of that kind of dictionary objects. If so, your function works as expected.

In the following code line:
id_datas = [q['Incident']['IncidentID'] for q in mk]
You are looping through the keys of the dictionary mk. What you want to do is loop through the items. This can de done using the dict.items() method.
Try the following:
id_datas = [q['Incident']['IncidentID'] for q in mk.items()]
EDIT:
My apologies, here is a solution that works when I try it with your dictionary.
id_datas = [q[1]['IncidentID'] for q in mk.items() if q[0] == 'Incident']
So mk.items() returns a list of tuples. Each tuple is in the format (key, value)
In my example q therefore loops through mk, and if the key q[0] is 'Incident', it returns the 'IncidentID' value from the dictionary returned by q[1].

How to fix "'User' object is not subscriptable"

I am trying to write some Python code, that gets the users Gitlab Profile picture/avatar to be sent in a Discord Embed later on in the code. However, when i try to read the json that the Gitlab API returns but i receive the error "'User' object is not subscriptable" this json doesnt look like other jsons returned by the Gitlab API.
I have tried to use Attributes but i still receive the same error, i have also tried just to read it but i still receive the same error.
import gitlab
import json
# private token or personal token authentication
gl = gitlab.Gitlab('URL', private_token='')
project = gl.projects.get(13)
json_data = project.tags.list(order_by='updated', sort='desc')
newest_tagjson = (json_data[0].attributes)
latesttag = newest_tagjson["name"]
name1 = newest_tagjson["commit"]["author_name"]
projectid = newest_tagjson["project_id"]
footer1 = "Panel"
if name1 == "------":
ID = 16
user = gl.users.get(ID)
print(user)
user2 = (user['avatar_url'].attributes)
i should receive a clean json that i can read but instead i recieve this in the print
<class 'gitlab.v4.objects.User'> => {'id': 16, 'name': '', 'username': '', 'state': 'active', 'avatar_url': 'https://URL.io/uploads/-/system/user/avatar/16/avatar.png', 'web_url': '', 'created_at': '2019-01-29T18:30:53.819Z', 'bio': ' \r\n', 'location': ', United Kingdom', 'public_email': '', 'skype': '', 'linkedin': '', 'twitter': '', 'website_url': '', 'organization': ''}
and i cannot read this.

The error seems pretty clear: the result of calling gl.users.get(ID) is not a Python dictionary, so you can't access keys with subscripts as in user['avatar_url']. You can access attributes using Python's dot notation, as in user.avatar_url.
You can of course extract the information you want into a Python dictionary:
>>> user_dict = {k: getattr(user, k) for k in
... ['id', 'name', 'state', 'avatar_url', 'web_url']}
>>> user_dict
{'id': 28841, 'name': 'Lars Kellogg-Stedman', 'state': 'active', 'avatar_url': 'https://secure.gravatar.com/avatar/1c09a8d9e719f9d13b6c99f6bb2637d8?s=80&d=identicon', 'web_url': 'https://gitlab.com/larsks'}
And then you can serialize this to JSON:
>>> print(json.dumps(user_dict, indent=2))
{
"id": 28841,
"name": "Lars Kellogg-Stedman",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/1c09a8d9e719f9d13b6c99f6bb2637d8?s=80&d=identicon",
"web_url": "https://gitlab.com/larsks"
}
The Python gitlab module wraps the gitlab API in a variety of managers designed to make certain things easier, but if your goal is to serialize things to JSON it might be easier to simply call the REST API yourself:
>>> import requests
>>> session = requests.Session()
>>> session.headers['private-token'] = your_private_token
>>> res = session.get('https://gitlab.com/api/v4/users/28841')
>>> res.json()
{'id': 28841, 'name': 'Lars Kellogg-Stedman', 'username': 'larsks', 'state': 'active', 'avatar_url': 'https://secure.gravatar.com/avatar/1c09a8d9e719f9d13b6c99f6bb2637d8?s=80&d=identicon', 'web_url': 'https://gitlab.com/larsks', 'created_at': '2014-04-26T01:52:14.000Z', 'bio': '', 'location': None, 'public_email': '', 'skype': '', 'linkedin': '', 'twitter': 'larsks', 'website_url': 'http://blog.oddbit.com/', 'organization': None}

Parsing XML into a dictionary of lists Python/Django

I'm having a little issue with parsing an xml with python. I'm trying to get my dictionary to look like the following
listDict = [{'name':'Sales','id':'1','position':'1','order_by_type':'True','order_by_asc':'True;}, {'name':'Information','id':'2','position':'1','order_by_type':'True','order_by_asc':'True;}]
I'm thinking my loop after pulling data from the xml string is wrong.
xml_data = ElementTree.fromstring(self.data)
# Lets grab all the base cats info and add them to a dict containing a list
base_cats = xml_data.findall('./BaseCategory/Name')
base_cats_id = xml_data.findall('./BaseCategory/base_id')
base_postion = xml_data.findall('./BaseCategory/position')
base_order_by_type = xml_data.findall('./BaseCategory/order_by_type')
base_order_by_asc = xml_data.findall('./BaseCategory/order_by_asc')
# store all information into lists
base_cat = [t.text for t in base_cats]
base_id = [t.text for t in base_cats_id]
base_p = [t.text for t in base_postion]
base_obt = [t.text for t in base_order_by_type]
base_asc = [t.text for t in base_order_by_asc]
base_dict = defaultdict(list)
# lets put everything in the list into a dictionary
for base in range(len(base_cat)): # for each base in base_cat loop
base_dict[base].append(base_cat[base])
base_dict[base].append(base_id[base])
base_dict[base].append(base_p[base])
base_dict[base].append(base_obt[base])
base_dict[base].append(base_asc[base])
This produces the following.
instance = {0: ['Sales 2', '1', '10', 'True', 'True'], 1: ['Information 2', '2', '20', 'True', 'True'], 2: ['Listing 2', '3', '30', 'True', 'True'], 3: ['Information', '4', '40', 'True', 'True'], 4: ['Land', '5', '50', 'True', 'True'], 5: ['&', '6', '60', 'True', 'True'], 6: ['Tax', '7', '70', 'True', 'True'], 7: ['Construction', '9', '90', 'True', 'True'], 8: ['Interior/Utilites', '10', '100', 'True', 'True'], 9: ['HOA/Community', '11', '110', 'True', 'True'], 10: ['Remarks', '12', '120', 'True', 'True'], 11: ['Exterior', '8', '80', 'True', 'True']})
My end goal is to be able to do the following on my django template
{%for item in instance%}
{{ item.name }}
{% endfor %}
Any help on how I may have something wrong would help a lot. Thanks in advance for the help.
EDIT:
As asked here is the xml I have.
<?xml version="1.0" ?>
<FormInstance>
<BaseCategory>
<Name>Sales</Name>
<base_id>1</base_id>
<position>10</position>
<order_by_type>True</order_by_type>
<order_by_asc>True</order_by_asc>
</BaseCategory>
<BaseCategory>
<Name>Information</Name>
<base_id>2</base_id>
<position>20</position>
<order_by_type>True</order_by_type>
<order_by_asc>True</order_by_asc>
<MainCategory>
<main_id>1</main_id>
<Name>Address 3</Name>
<is_visible>True</is_visible>
<position>10</position>
<order_by_type>True</order_by_type>
<order_by_asc>True</order_by_asc>
<SubCategory>
<sub_id>1</sub_id>
<Name>Street Number 2</Name>
<sub_library_id>StreetNumber</sub_library_id>
<field_display_type>[u'input']</field_display_type>
<field_type>[u'varchar']</field_type>
<is_active>True</is_active>
<is_required>True</is_required>
<help_text>Street Number</help_text>
<main_category>1</main_category>
<is_visible>True</is_visible>
<position>10</position>
<order_by_type>True</order_by_type>
<order_by_asc>True</order_by_asc>
<show_seller>True</show_seller>
<Enumerations>
<enum_id>4</enum_id>
<Name>Test Enum</Name>
<library_id>test enum</library_id>
<is_active>True</is_active>
<sub_category>1</sub_category>
<is_visible>True</is_visible>
<position>10</position>
<order_by_type>True</order_by_type>
<order_by_asc>True</order_by_asc>
</Enumerations>
</SubCategory>
</MainCategory>
</BaseCategory>
</FormInstance>

So, for what I gather in the expected results, it looks like you just want to get the information about nodes that are strictly BaseCategory, right? In the XML that was provided in the edit, you have two of those.
You should see the XML as a tree of nodes. In the example, you have something like:
FormInstance # this is the root
/ \
/ \
BaseCategory BaseCategory
(name:Sales) (name:Information)
\
\
MainCategory
(name:Address 3)
\
\
Subcategory
(name:Street Number 2)
But you only need the information in the BaseCategory elements, right?
You could just position yourself in the root (which... well... is what xml.fromstring does anyway) iterate over its BaseCategory nodes, get the items you need from those BaseCategory nodes and put them in your list of dictionaries.
Something like:
import pprint
from xml.etree import ElementTree
with open("sample_xml.xml", 'r') as f:
data = f.read()
xml_data = ElementTree.fromstring(data)
base_categories = xml_data.findall("./BaseCategory")
print("Found %s base_categories." % len(base_categories))
list_dict = []
for base_category in base_categories:
list_dict.append({
"name": base_category.find("Name").text,
"id": int(base_category.find("base_id").text),
"position": int(base_category.find("position").text),
"order_by_type": (True if base_category.find("order_by_type").text.lower() == "true"
else False),
"order_by_asc": (True if base_category.find("order_by_asc").text.lower() == "true"
else False),
})
print("list_dict=%s" % (pprint.pformat(list_dict)))
Which outputs:
Found 2 base_categories.
list_dict=[{'id': 1,
'name': 'Sales',
'order_by_asc': True,
'order_by_type': True,
'position': 10},
{'id': 2,
'name': 'Information',
'order_by_asc': True,
'order_by_type': True,
'position': 20}]
The idea is that a BaseCategory item is something that can be seen as a self-contained record (like a dict, if it helps you see it) that can contain (in it) the following attributes:
A string with the name in Name
A numeric id in base_id
A numeric position
A boolean order_by_type
A boolean order_by_asc
Another object MainCategory with its own fields...
So every time you position yourself in one of these BaseCategory nodes, you just gather the interesting fields that it has and put them in dictionaries.
When you do:
base_cats = xml_data.findall('./BaseCategory/Name')
base_cats_id = xml_data.findall('./BaseCategory/base_id')
base_postion = xml_data.findall('./BaseCategory/position')
base_order_by_type = xml_data.findall('./BaseCategory/order_by_type')
base_order_by_asc = xml_data.findall('./BaseCategory/order_by_asc')
You are treating those element (base_id, position...) almost as independent elements, which is not exactly what you have in your XML.
However, if you are absolutely certain that all those lists (base_cats, base_cats_id, base_position...) do contain the same number of items, you can still re-build your dictionary, using the lenght of one of them (in the example below len(base_cats), but it could've been len(base_cats_id), len(base_position)... since all those lists have the same length) to iterate through all the lists in the same step:
base_cats = xml_data.findall('./BaseCategory/Name')
base_cats_id = xml_data.findall('./BaseCategory/base_id')
base_postion = xml_data.findall('./BaseCategory/position')
base_order_by_type = xml_data.findall('./BaseCategory/order_by_type')
base_order_by_asc = xml_data.findall('./BaseCategory/order_by_asc')
list_dict = []
for i in range(len(base_cats)):
list_dict.append({
"name": base_cats[i].text,
"id": int(base_cats_id[i].text),
"position": int(base_postion[i].text),
"order_by_type": True if base_order_by_type[i].text.lower() == "true" else False,
"order_by_asc": True if base_order_by_asc[i].text.lower() == "true" else False,
})
print("list_dict=%s" % (pprint.pformat(list_dict)))

Parse this JSON response From App Annie in Python

I am working with the request module within python to grab certain fields within the JSON response.
import json
fn = 'download.json'
data = json
response = requests.get('http://api.appannie.com/v1/accounts/1000/apps/mysuperapp/sales?break_down=application+iap&start_date=2013-10-01&end_date=2013-10-02', \
auth=('username', 'password'))
data = response.json()
print(data)
This works in python, as the response is the following:
{'prev_page': None, 'currency': 'USD', 'next_page': None, 'sales_list': [{'revenue': {'ad': '0.00', 'iap': {'refunds': '0.00', 'sales': '0.00', 'promotions': '0.00'}, 'app': {'refunds': '0.00', 'updates': '0.00', 'downloads': '0.00', 'promotions': '0.00'}},
'units': {'iap': {'refunds': 0, 'sales': 0, 'promotions': 0}, 'app': {'refunds': 0, 'updates': 0, 'downloads': 2000, 'promotions': 0}}, 'country': 'all', 'date': 'all'}], 'iap_sales': [], 'page_num': 1, 'code': 200, 'page_index': 0}
The question is how do I parse this to get my downloads number within the 'app' block - namely the "2000" value?

After the response.json() data is already a dictionary otherwise response.json() would raise an exception. Therefore you can access it just like any other dictionary.

You can use the loads() method of json -
import json
response = requests.get('http://api.appannie.com/v1/accounts/1000/apps/mysuperapp/sales?break_down=application+iap&start_date=2013-10-01&end_date=2013-10-02',
auth=('username', 'password'))
data = json.loads(response.json()) # data is a dictionary now
sales_list = data.get('sales_list')
for sales in sales_list:
print sales['revenue']['app']

You can use json.loads:
import json
import requests
response = requests.get(...)
json_data = json.loads(response.text)
This converts a given string into a dictionary which allows you to access your JSON data easily within your code.