We Keep Coding

How can I remove duplicate elements in a JSON file?

I trying to write a program that will output information to me in the form of a JSON file, but I ran into the fact that I duplicate information with the "exchange_id" parameter, but with other info. I want information with this value to be displayed to me once. How can I do this?
Example of JSON file:
{
"exchange_name": "Huobi Global",
"market_url": "https://www.huobi.com/en-us/exchange/eth_btc",
"price": 48637.97781883441,
"last_update": "2021-12-09T17:11:53.000Z",
"exchange_id": 102
},
{
"exchange_name": "Huobi Global",
"market_url": "https://www.huobi.com/en-us/exchange/xrp_btc",
"price": 48656.10607332287,
"last_update": "2021-12-09T17:10:53.000Z",
"exchange_id": 102
},
My code:
json_data = response['data']
pairs = []
out_object = {}
exchangeIds = {102,311,200,302,521,433,482,406,42,400}
for pair in json_data["marketPairs"]:
if pair['exchangeId'] in set(exchangeIds):
pairs.append({
"exchange_name": pair["exchangeName"],
"market_url": pair["marketUrl"],
"price": pair["price"],
"last_update" : pair["lastUpdated"],
"exchange_id": pair["exchangeId"]
})
out_object["name_of_coin"] = json_data["name"]
out_object["marketPairs"] = pairs
out_object["pairs"] = json_data["numMarketPairs"]
name = json_data["name"]
save_path = '.\swapMarketCap\coins'
file_name = f'{name}'
path_to_coin = os.path.join(save_path, file_name +".json")

In the absence of more detail, here's a pattern that you could use. This does not purport to be rewrite of your code and also makes certain assumptions (in the absence of detail in the original question). However, it should help you:
response = {'data':
{'marketPairs': [{
"exchange_name": "Huobi Global",
"market_url": "https://www.huobi.com/en-us/exchange/eth_btc",
"price": 48637.97781883441,
"last_update": "2021-12-09T17:11:53.000Z",
"exchange_id": 102
},
{
"exchange_name": "Huobi Global",
"market_url": "https://www.huobi.com/en-us/exchange/xrp_btc",
"price": 48656.10607332287,
"last_update": "2021-12-09T17:10:53.000Z",
"exchange_id": 102
}
]}}
exchangeIds = {102, 311, 200, 302, 521, 433, 482, 406, 42, 400}
output = {'data': {'marketPairs': []}}
for d in response['data']['marketPairs']:
if (id := d.get('exchange_id', None)):
if id in exchangeIds:
output['data']['marketPairs'].append(d)
exchangeIds.remove(id)
print(output)
So, the underlying idea is that you have a set of IDs that you're interested in. Once you find one that's in the set, take the reference to the current dictionary and add it to the new dictionary (output). Then, crucially, remove the previously processed ID from the set.

Creating Lists from Multiple Jsons with Missing Keys

I am trying to create lists from json datas by pulling one by one and append them to the lists. However, some variables does not given in all json files. For example: for the json file below, data does not have ['statistics']['aerialLost'] , so it return Key Error. My Expected solution is when json file does not have key, append 'None' value to the list and continue.
Code
s_aerialLost = []
s_aerialWon = []
s_duelLost = []
s_duelWon = []
players = ['Martin Linnes', 'Christian Luyindama', 'Marcão', 'Ömer Bayram', 'Oghenekaro Etebo', 'Muhammed Kerem Aktürkoğlu', 'Gedson Fernandes', 'Emre Kılınç', 'Ryan Babel', 'Mostafa Mohamed', 'Florent Hadergjonaj', 'Tomáš Břečka', 'Duško Tošić', 'Oussama Haddadi', 'Kristijan Bistrović', 'Aytaç Kara', 'Haris Hajradinović', 'Armin Hodžić', 'Gilbert Koomson', 'Isaac Kiese Thelin']
players_id = [109569, 867191, 840951, 68335, 839110, 903324, 862055, 202032, 1876, 873551, 354860, 152971, 14557, 867180, 796658, 128196, 254979, 138127, 341107, 178743]
for player, player_id in zip(players, players_id):
url = base_url + str(player_id)
data = requests.request("GET", url).json()
## just added 4 data for simplify
accurateLongBalls = str(data['statistics']['accurateLongBalls'])
aerialLost = str(data['statistics']['aerialLost'])
aerialWon = str(data['statistics']['aerialWon'])
duelLost = str(data['statistics']['duelLost'])
s_aerialLost.append()
s_aerialWon.append()
s_duelLost.append()
s_duelWon.append()
Json File
{
"player": {
"name": "Martin Linnes",
"slug": "martin-linnes",
"shortName": "M. Linnes",
"position": "D",
"userCount": 339,
"id": 109569,
"marketValueCurrency": "€",
"dateOfBirthTimestamp": 685324800
},
"team": {
"name": "Galatasaray",
"slug": "galatasaray",
"shortName": "Galatasaray",
"gender": "M",
"userCount": 100254,
"nameCode": "GAL",
"national": false,
"type": 0,
"id": 3061,
"teamColors": {
"primary": "#ff9900",
"secondary": "#ff0000",
"text": "#ff0000"
}
},
"statistics": {
"totalPass": 32,
"accuratePass": 22,
"totalLongBalls": 7,
"accurateLongBalls": 3,
"totalCross": 2,
"aerialWon": 1,
"duelLost": 2,
"duelWon": 7,
"totalContest": 3,
"wonContest": 2,
"totalClearance": 4,
"totalTackle": 3,
"wasFouled": 1,
"fouls": 1,
"minutesPlayed": 82,
"touches": 63,
"rating": 7.3,
"possessionLostCtrl": 18,
"keyPass": 1
},
"position": "D"
}
Error
KeyError: 'aerialLost'

Use .get(). You can specify a default value to return if the key is not found, and it defaults to None.
So you can use
aerialLost = str(data.get('statistics', {}).get('aerialLost'))
The first call defaults to an empty dictionary so that there's something to make the second .get() call on. The second call just returns the default None.

Iterating through dictionaries and Accumulation

At the moment, I am attempting to accumulate values in dictionaries nested within a dictionary for homework purposes.
a = {
"Tennis Balls": {
"User Rating": 15,
"Available Colors": 5,
"Location": "Puppy Mart"
}
}
b = {
"Squeaky Toy": {
"User Rating": 48,
"Available Colors": 3,
"Location": "Pooch Shop"
}
}
c = {
"Stick": {
"User Rating": 32,
"Available Colors": 4,
"Location": "Pooch Shop"
}
}
d = {
"Old Shoe": {
"User Rating": 13,
"Available Colors": 2,
"Location": "Puppy Mart"
}
}
e = {
"Stuffed Animal": {
"User Rating": 14,
"Available Colors": 7,
"Location": "Dog Express"
}
}
puppy_toys = {"Toys": [a, b, c, d, e]}
total_user_ratings = 0
for toy in puppy_toys["Toys"]:
# insert your own code
individual_user_rating = puppy_toys["Toys"][toy]["User Rating"]
total_user_ratings += individual_user_rating
print(total_user_ratings)
I have to fill in # your line here with a single line of code that will allow the for loop to iterate through the dictionary puppy_toys and extract the total number of user ratings for toys located at "Pooch Shop", and then store the total in the variable total_user_ratings, but my line of code, shown below, doesn't seem to be working:
if["Location"] = "Pooch Shop":
Thanks for your assistance!

Well, after playing a little, I imagine you want to achieve the sum of User rating by name of Location, so, you have a dictionary with a list actually, correct me if I'm wrong, so you just have to loop it and take the values like this:
a = {
"Tennis Balls": {
"User Rating": 15,
"Available Colors": 5,
"Location": "Puppy Mart"
}
}
b = {
"Squeaky Toy": {
"User Rating": 48,
"Available Colors": 3,
"Location": "Pooch Shop"
}
}
c = {
"Stick": {
"User Rating": 32,
"Available Colors": 4,
"Location": "Pooch Shop"
}
}
d = {
"Old Shoe": {
"User Rating": 13,
"Available Colors": 2,
"Location": "Puppy Mart"
}
}
e = {
"Stuffed Animal": {
"User Rating": 14,
"Available Colors": 7,
"Location": "Dog Express"
}
}
puppy_toys = {"Toys": [a, b, c, d, e]}
total_user_ratings = 0
for toy in puppy_toys["Toys"]:
shop = list(toy.values())[0]
if (shop["Location"] == "Pooch Shop"):
individual_user_rating = shop["User Rating"]
total_user_ratings += individual_user_rating
print(total_user_ratings)

Your line of code has two main problems. Let's go over it:
if["Location"] = "Pooch Shop":
First, you are trying to compare between two expressions. This is done in python with the equality operator which is ==. A single = usually stands for assignment.
Ex:
>>> a = 5
>>> print(a)
5
>>> print(a == 5) # "is a equal to 5?"
True
The second problem is if["Location"].
In Python the square brackets usually are used as a "getter". A way to acces elements inside an object (like a list or a dictionary).
Ex:
>>> a = ["x", "y", "z"]
>>> print(a[1])
y
>>> b = {"name": "Tomer", "age": 29}
>>> print(b["name"])
Tomer
So you would want the ["Location"] to come after some object.
A hint for what that should be is in the next line of code:
individual_user_rating = puppy_toys["Toys"][toy]["User Rating"]

Foreach loop in Python to extract value from array in json response

I've got this json response:
{
"properties": {
"basic": {
"bandwidth_class": "",
"failure_pool": "",
"max_connection_attempts": 0,
"max_idle_connections_pernode": 50,
"max_timed_out_connection_attempts": 2,
"monitors": [
"Simple HTTP"
],
"node_close_with_rst": false,
"node_connection_attempts": 3,
"node_delete_behavior": "immediate",
"node_drain_to_delete_timeout": 0,
"nodes_table": [
{
"node": "abc1.prod.local:80",
"priority": 1,
"state": "active",
"weight": 1
},
{
"node": "def1.prod.local:80",
"priority": 1,
"state": "disabled",
"weight": 1
},
{
"node": "ghi1.prod.local:80",
"priority": 1,
"state": "disabled",
"weight": 1
},
{
"node": "jkl1.prod.local:80",
"priority": 1,
"state": "active",
"weight": 1
}
],
"note": "",
"passive_monitoring": true,
"persistence_class": "",
"transparent": false
}
}
}
And this powershell script:
$nodesAarray = "abc1.prod.local:80", "jkl1.prod.local:80"
foreach($node in $nodesArray)
{
$nodes_match_and_enabled = $GetNodesResponse.properties.basic.nodes_table | Where { $_.node -eq $node -and $_.state -eq "active" }
if($nodes_match_and_enabled)
{
Write-Output "$node exists in the pool and active"
}
else
{
Write-Output "$node is either not active or the name mismatches"
$global:invalidNodeArray.Add($node)
}
}
In my powershell script I am looping to check the two nodes in my array actually match by value and the state is active. It works as I expect.
However, I am scripting the same exact logic in Python (I am a beginner) but not sure how to approach it. Any idea what the script would look like in Python???

First, filter all active nodes, then compare with node list:
data = json.loads(text)
active_nodes = {
n['node']
for n in data['properties']['basic']['nodes_table']
if n['state'] == 'active'
}
nodes = {"abc1.prod.local:80", "jkl1.prod.local:80"}
for node in nodes:
if node in active_nodes:
print('{} exists in the pool and active'.format(node))
else:
print('{} is either not active or the name mismatches'.format(node))
invalid_nodes = nodes - active_nodes

Should work in Python 2 or 3, I think:
#!/usr/bin/env python
import sys
import json
res = ""
for line in sys.stdin:
res += line.rstrip()
res_obj = json.loads(res)
nodes = [ 'abc1.prod.local:80', 'jkl1.prod.local:80' ]
invalid_nodes = []
for node in nodes:
try:
found = False
test_node_objs = res_obj['properties']['basic']['nodes_table']
for test_node_obj in test_node_objs:
test_node = test_node_obj['node']
if node == test_node:
found = True
break
if found:
sys.stdout.write("%s exists in the pool and active\n" % (node))
else:
sys.stdout.write("%s is either not active or the name mismatches\n" % (node))
invalid_nodes.append(node)
except KeyError as ke:
sys.stderr.write("malformed response? check input...\n")
pass
Example usage:
$ ./parse_response.py < response.json

Here's an implementation:
jsonObj = json.loads(jsonSrc)
expectedNodes = {"abc1.prod.local:80", "jkl1.prod.local:80"}
for node in expectedNodes:
node_table = jsonObj['properties']['basic']['nodes_table']
node_match = list(filter(lambda t_node: node == t_node['node'], node_table))
is_node_matches_and_active = len(node_match) > 0 and node_match[0]['state'] == "active"
if is_node_matches_and_active:
print('node {} exists and is active'.format(node))
else:
print('node {} not found or not active'.format(node))
Output :
node jkl1.prod.local:80 exists and is active
node abc1.prod.local:80 exists and is active

TypeError: list indices must be integers or slices, not str <encoding error>

So, my code looks like this:
import requests
import random
def load():
req = requests.get(https: // yande.re / post.json?tags = rating % 3
Asafe + -pantyshot + -panties + & ms = 1 & page = 12650 & limit = 1)
data = Posts(req.json()["id"][0], req.json()["tags"], slice(req.json()["creator_id"]), req.json()["author"],
req.json()["source"],
req.json()["score"], req.json()["md5"], req.json()["file_url"], req.json()["sample_url"],
req.json()["width"],
req.json()["height"])
all = data.tags, data.creator_id, data.author, data.source, data.score, data.md5, data.file_url, data.sample_url, data.width, data.height
return all
And, when I run the load(), I have this output:
Traceback (most recent call last): File "", line
134, in File "", line 126, in anime
TypeError: list indices must be integers or slices, not str
What could be causing it?
By the way, the data I'm fetching looks like this:
[
{
"actual_preview_height": 218,
"jpeg_url": "https://files.yande.re/image/32a001e7b5050828c9b07e62de634958/yande.re%20376617%20dress%20novelance%20see_through.jpg",
"status": "active",
"preview_url": "https://assets.yande.re/data/preview/32/a0/32a001e7b5050828c9b07e62de634958.jpg",
"has_children": false,
"source": "http://i2.pixiv.net/img-original/img/2016/12/05/00/00/10/60241721_p0.jpg",
"score": 1,
"height": 1392,
"rating": "s",
"id": 376617,
"last_commented_at": 0,
"frames": [],
"md5": "32a001e7b5050828c9b07e62de634958",
"updated_at": 1480900734,
"creator_id": 280440,
"frames_pending_string": "",
"frames_string": "",
"actual_preview_width": 300,
"is_shown_in_index": true,
"frames_pending": [],
"change": 1992459,
"last_noted_at": 0,
"approver_id": null,
"is_held": false,
"preview_width": 150,
"tags": "dress novelance see_through",
"preview_height": 109,
"created_at": 1480900721,
"file_ext": "jpg",
"sample_height": 1088,
"sample_url": "https://files.yande.re/sample/32a001e7b5050828c9b07e62de634958/yande.re%20376617%20sample%20dress%20novelance%20see_through.jpg",
"parent_id": null,
"width": 1920,
"jpeg_file_size": 0,
"sample_file_size": 478570,
"author": "LolitaJoy",
"file_size": 989513,
"file_url": "https://files.yande.re/image/32a001e7b5050828c9b07e62de634958/yande.re%20376617%20dress%20novelance%20see_through.jpg",
"is_note_locked": false,
"is_pending": false,
"sample_width": 1500,
"jpeg_width": 1920,
"jpeg_height": 1392,
"is_rating_locked": false
}
]

I found the problem actually. instead of req.json()["id"], it should have been req.json()[0]["id"]