We Keep Coding

Python iterate through JSON - dict object is not callable

I want to parse flight data from Aviationstack API.
For this example, I format the URL to get data from flight with departure at Marseille airport (France) with the parameter dep_icao=LFML (cf. documentation) :
# get data for flights from Marseille airport
url = 'http://api.aviationstack.com/v1/flights?access_key=MYAPIKEY&dep_icao=LFML'
req = requests.get(url)
response = req.json()
This is the response (shortened)
{'pagination': {'limit': 100, 'offset': 0, 'count': 100, 'total': 377},
'data': [{'flight_date': '2022-12-24',
'flight_status': 'active',
'departure': {'airport': 'Marseille Provence Airport',
'timezone': 'Europe/Paris',
'iata': 'MRS',
'icao': 'LFML',
'terminal': '1A',
'gate': None,
'delay': 10,
'scheduled': '2022-12-24T09:00:00+00:00',
'estimated': '2022-12-24T09:00:00+00:00',
'actual': '2022-12-24T09:09:00+00:00',
'estimated_runway': '2022-12-24T09:09:00+00:00',
'actual_runway': '2022-12-24T09:09:00+00:00'},
'arrival': {'airport': 'El Prat De Llobregat',
'timezone': 'Europe/Madrid',
'iata': 'BCN',
'icao': 'LEBL',
'terminal': '1',
'gate': None,
'baggage': '06',
'delay': None,
'scheduled': '2022-12-24T10:05:00+00:00',
'estimated': '2022-12-24T10:05:00+00:00',
'actual': None,
'estimated_runway': None,
'actual_runway': None},
'airline': {'name': 'Qatar Airways', 'iata': 'QR', 'icao': 'QTR'},
'flight': {'number': '3721',
'iata': 'QR3721',
'icao': 'QTR3721',
'codeshared': {'airline_name': 'vueling',
'airline_iata': 'vy',
'airline_icao': 'vlg',
'flight_number': '1509',
'flight_iata': 'vy1509',
'flight_icao': 'vlg1509'}},
'aircraft': None,
'live': None},
[...]
}]}
Then I want to iterate through this JSON response. I can get the pagination info right but I'm only able to get info from the first element in data, so I will later loop through all results.
My problem is that I would only like to get the iata item nested in flight. The way I'm trying to get it returns 'dict' object is not callable error.
# iterate through JSON response
pagination_data = response.get("pagination")
flight_data = response.get("data")[0]
total_results = pagination_data.get("total")
flight = flight_data.get('flight')('iata')
pprint(f"total results : {total_results}")
pprint(f"flight : {flight}")

The get method will return a dictionnary which is not function and then can not be called
It should be flight = flight_data.get('flight').get('iata')
Alternatives :
(but without returning None) flight_data.get('flight')['iata']
(but without returning None) flight=flight_data['flight']['iata']

How to scarpe li tag under ul?

I want to fetch the all the list tags under the ul tag with id= "demofour" from https://www.parliament.lk/en/members-of-parliament/directory-of-members/?cletter=A.
Below is the code:
print(soup.find('ul',id='demoFour'))
But the output which is being displayed is
<ul id="demoFour"></ul>

Content is served dynamically based on data of an additional XHR request, so you have to call this instead. You can inspect this by taking a look into devtools of browser on XHR tab.
Example
Instead of appending only the obvious to a list of dicts you could also iterate all detailpages while requesting them.
from bs4 import BeautifulSoup
import requests, string
data = []
for letter in list(string.ascii_uppercase):
result = requests.post(f'https://www.parliament.lk/members-of-parliament/directory-of-members/index2.php?option=com_members&task=all&tmpl=component&letter={letter}&wordfilter=&search_district=')
for e in result.json():
#result = requests.get(f"https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/{e['mem_intranet_id']}")
data.append({
'url':f"https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/{e['mem_intranet_id']}",
'id':e['mem_intranet_id'],
'name':e['member_sname_eng']
})
data
Output
[{'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3266',
'id': '3266',
'name': 'A. Aravindh Kumar'},
{'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/50',
'id': '50',
'name': 'Abdul Haleem'},
{'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3325',
'id': '3325',
'name': 'Ajith Rajapakse'},
{'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3296',
'id': '3296',
'name': 'Akila Ellawala'},
{'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3355',
'id': '3355',
'name': 'Ali Sabri Raheem'},...]

I am trying to webscrape from Zomato, however it returns with an output of "None" and Attribute Error

Whenever i try to extract the data, it returns an output of "None" which I am not sure of is it the code (I followed the rules of using bs4) or is it just the website that's different to scrape?
My code:
import requests
import bs4 as bs
url = 'https://www.zomato.com/jakarta/pondok-indah-restaurants'
req = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
html = req.text
soup = bs.BeautifulSoup(html, "html.parser")
listings = soup.find('div', class_='sc-gAmQfK fKxEbD')
rest_name = listings.find('h4', class_='sc-1hp8d8a-0 sc-eTyWNx gKsZcT').text
##Output: AttributeError: 'NoneType' object has no attribute 'find'
print(listings)
##returns None
Here is the inspected tag of the website which i try to get the h4 class showing the restaurant's name:
inspected element

What happens?
Classes are generated dynamically and may differ from your inspections via developer tools - So you won't find what you are looking for.
How to fix?
It would be a better approach to select your targets via tag or id if available, cause these are more static than css classes.
listings = soup.select('a:has(h4)')
Example
Iterating listings and scrape several infromation:
import requests
import bs4 as bs
url = 'https://www.zomato.com/jakarta/pondok-indah-restaurants'
req = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
html = req.text
soup = bs.BeautifulSoup(html, "html.parser")
data = []
for item in soup.select('a:has(h4)'):
data.append({
'title':item.h4.text,
'url':item['href'],
'etc':'...'
})
print(data)
Output
[{'title': 'Radio Dalam Diner', 'url': '/jakarta/radio-dalam-diner-pondok-indah/info', 'etc': '...'}, {'title': 'Aneka Bubur 786', 'url': '/jakarta/aneka-bubur-786-pondok-indah/info', 'etc': '...'}, {'title': "McDonald's", 'url': '/jakarta/mcdonalds-pondok-indah/info', 'etc': '...'}, {'title': 'KOPIKOBOY', 'url': '/jakarta/kopikoboy-pondok-indah/info', 'etc': '...'}, {'title': 'Kopitelu', 'url': '/jakarta/kopitelu-pondok-indah/info', 'etc': '...'}, {'title': 'KFC', 'url': '/jakarta/kfc-pondok-indah/info', 'etc': '...'}, {'title': 'HokBen Delivery', 'url': '/jakarta/hokben-delivery-pondok-indah/info', 'etc': '...'}, {'title': 'PHD', 'url': '/jakarta/phd-pondok-indah/info', 'etc': '...'}, {'title': 'Casa De Jose', 'url': '/jakarta/casa-de-jose-pondok-indah/info', 'etc': '...'}]

Sending python requests and handling JSON lists

I am sending requests to a crypto network for data on accounts. You get sent back information, but I haven't yet encountered lists being sent in JSON until now. I want to parse certain information, but am having trouble because the JSON is a list and is not as easy to parse compared to normal JSON data.
import requests
import json
url = ' https://s1.ripple.com:51234/'
payload = {
"method": "account_objects",
"params": [
{
"account": "r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59",
"ledger_index": "validated",
"type": "state",
"deletion_blockers_only": False,
"limit": 10
}
]
}
response = requests.post(url, data=json.dumps(payload))
print(response.text)
data = response.text
parsed = json.loads(data)
price = parsed['result']
price = price['account_objects']
for Balance in price:
print(Balance)
You will receive all the tokens the account holds and the value. I can not figure out how to parse this correctly and receive the correct one. This particular test account has a lot of tokens so I will only show the first tokens info.
RESULT
{'Balance': {'currency': 'ASP', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'}, 'Flags': 65536, 'HighLimit': {'currency': 'ASP', 'issuer': 'r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59', 'value': '0'}, 'HighNode': '0', 'LedgerEntryType': 'RippleState', 'LowLimit': {'currency': 'ASP', 'issuer': 'r3vi7mWxru9rJCxETCyA1CHvzL96eZWx5z', 'value': '10'}, 'LowNode': '0', 'PreviousTxnID': 'BF7555B0F018E3C5E2A3FF9437A1A5092F32903BE246202F988181B9CED0D862', 'PreviousTxnLgrSeq': 1438879, 'index': '2243B0B630EA6F7330B654EFA53E27A7609D9484E535AB11B7F946DF3D247CE9'}
I want to get the first bit of info, here. {'Balance': {'currency': 'ASP', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'},
Specifically 'value' and the number
I have tried to take parse 'Balance' but since it is a list it is not as straight forward.

You're mixing up lists and dictionaries. In order to access a dictionary by key, you need to invoke the key, as such:
for Balance in price:
print(Balance['Balance'])
Yields the following results:
{'currency': 'CHF', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '-0.3488146605801446'}
{'currency': 'BTC', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'}
{'currency': 'USD', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '-11.68225001668339'}
If you only wanted to extract the value, you simply dive one level deeper:
for Balance in price:
print(Balance['Balance']['value')
Which yields:
-0.3488146605801446
0
-11.68225001668339

I assume that under price['account_objects'] you have a list of dictionaries? And then in each dictionary you have in one of the keys: 'Balance': {'currency': 'ASP', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'. If so, why don't you iterate over the list and then access each dictionary, like:
account_objects = price['account_objects']
for account_object in price:
print(account_object['Balance'])

Python3 requests module or urllib.request module both retrieving incomplete json

I'm doing some scraping and looking at pages like this one (https://www.endomondo.com/rest/v1/users/20261627/workouts/526622897), but I have not been able to fully retrieve the JSON content.I have tried using both of the following sets of code, but each returns an incomplete JSON object:
url = 'https://www.endomondo.com/rest/v1/users/%s/workouts/%s'%(string_use_user, string_use_workout)
print(url)
response = urlopen(url)
try:
reader = codecs.getreader("utf-8")
print(reader(response))
jsonresponse = json.load(reader(response))
print(jsonresponse)
and similarly using the response library instead of urllib also fails to retrieve the full JSON
url = 'https://www.endomondo.com/rest/v1/users/%s/workouts/%s'%(string_use_user, string_use_workout)
print("using this url %s"%url)
r = requests.get(url)
try:
print(r.json())
jsonresponse = r.json()# json.loads(response.read())
In both cases I get about 1/4 of the JSON. For example, in this case:
https://www.endomondo.com/rest/v1/users/20261627/workouts/526622897
I received:
{'feed_id': 281475471235835, 'id': 526622897, 'duration': 4082.0, 'local_start_time': '2015-05-21T09:30:45.000+02:00', 'calories': 1073.0, 'tagged_users': [], 'altitude_max': 69.9523, 'sport': 0, 'distance': 11.115419387817383, 'altitud\
e_min': 14.9908, 'include_in_stats': True, 'hydration': 0.545339, 'start_time': '2015-05-21T07:30:45.000Z', 'ascent': 137.162, 'is_live': False, 'pb_count': 2, 'playlist': [], 'is_peptalk_allowed': False, 'weather': {'wind_speed': 11, '\
temperature': 12, 'wind_direction': 13, 'type': 3, 'humidity': 81}, 'speed_max': 24.8596, 'author': {'name': 'gfdgfd', 'id': 20261627, 'last_name': 'gdsgsk', 'gender': 0, 'expand': 'abs', 'picture': {'url': 'https://www.endom\
ondo.com/resources/gfx/picture/18511427/thumbnail.jpg'}, 'first_name': 'gdsgds', 'viewer_friendship': 1, 'is_premium': False}, 'sharing': [{'share_time': '2015-05-21T08:45:19.000Z', 'type': 0, 'share_id': 1635690786663532}], 'show_map':\
0, 'pictures': [], 'hashtags': [], 'descent': 150.621, 'speed_avg': 9.80291763746756, 'expand': 'full', 'show_workout': 0, 'points': {'expand': 'ref', 'id': 2199549878449}}
I am not receiving the long arrays within the data. I am also not even recovering all of the non-array data.
I ran the original page through a JSON validator, and it's fine. Similarly, I ran the JSON I do receive through a validator, and it's also fine - it doesn't show any signs of missing things unless I compare with the original.
I would appreciate any advice about how to troubleshoot this. Thanks.

Looks like this API is doing some User-Agent sniffing and only sending the complete content for what it considers to be actual web browsers.
Once you set a User-Agent header with the UA string of a common browser, you get the full response:
>>> UA = 'Mozilla/5.0 (X11; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0'
>>> url = 'https://www.endomondo.com/rest/v1/users/20261627/workouts/526622897'
>>> r = requests.get(url, headers={'User-Agent': UA})
>>>
>>> print len(r.content)
96412
See the requests docs for more details on setting custom headers.