We Keep Coding

Extract nested JSON values

I am trying to figure out on how to convert the following JSON object in to a dataframe
[
{
"id":"123",
"sources":[
{
"name":"ABC",
"first":"2020-02-26T03:19:23.247Z",
"last":"2020-02-26T03:19:23.247Z"
},
{
"name":"XYZ",
"first":"2020-02-26T03:19:23.247Z",
"last":"2020-02-26T03:19:23.247Z"
}
]
}
]
The dataframe should appear like this.
id ABC.first ABC.last XYZ.first XYZ.last
123 2020-02-26.. 2020-02-26.. 2020-02-26.. 2020-02-26T03:19:23.247Z
Thanks in advance for your help

Python includes the useful json module. You can find some documentation for it here.
To use it , you'll need to import it into your Python script using the import json command.
From there, you can use the json.loads() method and pass it a some JSON. The loads() method returns a Python dictionary which then lets you access the properties of your JSON object by name. Some example code might look like:
import json
# some JSON:
personJSON = '{"name":"John", "age":30, "city":"New York", "address": {"street": "Fake Street", "streetNumber": "123"}}'
# parse personJSON:
personDict = json.loads(personJSON)
# the result is a Python dictionary:
print(personDict["name"]) # Will print "John"
print(personDict["address"]["street"]) # Will print "Fake Street"
Good luck!

Python add new element to existing JSON File

I'm trying to append some data into a JSON file inside python.
It should add the CVE-ID into my customer JSON file.
I wasn't able to solve this by myself. If I print my dict after appending it shows the expected result.
But it seems, that the file will not be written after the append. I have also tried to use json.dump() after the "appending-line" - but also without success.
At the moment my code looks like this:
with open("test.json", "r+") as customerdata:
customers_json = json.load(customerdata)
# some other code here...
if cve["cve"]["CVE_data_meta"]["ID"] not in customer["customer"]["already-sent-cve"]:
customers_json["customers"][0]["customer"]["already-sent-cve"].append(cve["cve"]["CVE_data_meta"]["ID"])
My JSON file looks like this:
{
"customers":[
{
"customer":{
"id":"1",
"company-name":"test GmbH",
"alert-email":"xyz#googlemail.com",
"using":[
"xxx",
"xyz"
],
"already-sent-cve":[
"CVE-2013-3738"
# here new CVE
]
}
},
{
"customer":{
"id":"2",
...
}
]
}
FYI: There is another JSON file with the CVE's from which file I'm collecting the data.
Any suggestions how to solve this?
Regards.
EDIT:
Was able to solve this:
with open("test.json", "w") as customerdata:
customers_json["customers"][0]["customer"]["already-sent-cve"].append(cve["cve"]["CVE_data_meta"]["ID"])
json.dump(customers_json, customerdata, indent=2)

You need to write the data back to the file:
with open('test.json', 'w') as customerdata:
json.dump(customers_json, customerdata)

How to get a value from my JSON file, in python

I want you to ask for your help.
I need to browse folder with json files and I need to search for one attribute with specific value.
Here is my code:
def listDir():
fileNames = os.listdir(FOLDER_PATH)
for fileName in fileNames:
print(fileName)
with open('C:\\Users\\Kamčo\\Desktop\\UBLjsons\\' + fileName, 'r') as json_file:
data = json.load(json_file)
data_json = json.dumps(data, indent=2, sort_keys=True)
print(data_json)
for line in data_json:
if line['ID'] == 'Kamilko':
print("try") #just to be sure it accessed to this
I am getting this error: TypeError: string indices must be integers
I also tried to search for a solution here but it didnt help me.
Here is my JSON
{
"Invoice": {
"#xmlns": "urn:oasis:names:specification:ubl:schema:xsd:Invoice-2",
"#xmlns:cac": "urn:oasis:names:specification:ubl:schema:xsd:CommonAggregateComponents-2",
"#xmlns:cbc": "urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2",
"ID": "Kamilko",
"IssueDate": "2020-02-09",
"OrderReference": {
"ID": "22"
},
"InvoiceLine": {
"Price": {
"PriceAmount": {
"#currencyID": "EUR",
"#text": "23.50"
},
"BaseQuantity": {
"#unitCode": "C62",
"#text": "1"
}
}
}
}
}
do you have any idea how to do it?

You've loaded your file in using json.load(...). That'll convert the JSON data into a Python dictionary that you can use to access elements:
if data["Invoice"]["OrderReference"]["ID"] == 22:
print("try")
Note that you might want to check the relevant keys exist along the way, in case the structure of your file varies, or you could catch the KeyError that'll come up if the key doesn't exist, using try/except.
Some more background:
When you then call json.dumps(...), you're taking that handy python structure and converting it back into a hard-to-understand string again. I don't think you want or need to do this.
The specific error you have is because dumps has created a string. You're then trying to access an element of that string using the [ ] operator. Strings can only be indexed using integers, e.g. mystr[4], so Python doesn't understand what you're asking it to do with data_json["ID"].

JSON Parse an element inside an element in Python

I have a JSON text grabbed from an API of a website:
{"result":"true","product":{"made":{"Taiwan":"Taipei","HongKong":"KongStore","Area":"Asia"}}}
I want to capture "Taiwan" and "Taipei" but always fail.
Here is my code:
import json
weather = urllib2.urlopen('url')
wjson = weather.read()
wjdata = json.loads(wjson)
print wjdata['product']['made'][0]['Taiwan']
I always get the following error:
Keyword 0 error
Whats the correct way to parse that json?

You are indexing an array where there are none.
The JSON is the following:
{
"result":"true",
"product": {
"made": {
"Taiwan":"Taipei",
"HongKong":"KongStore",
"Area":"Asia"
}
}
}
And the above contains no arrays.
You are assuming the JSON structure to be something like this:
{
"result":"true",
"product": {
"made": [
{"Taiwan":"Taipei"},
{"HongKong":"KongStore"},
{"Area":"Asia"}
]
}
}
From a brief look at the doc pages for the json package, I found this conversion table: Conversion table using json.loads
It tells us that a JSON object translates to a dict. And a dict has a method called keys, which returns a list of the keys.
I suggest you try something like this:
#... omitted code
objectKeys = wjdata['product']['made'].keys()
# You should now have a list of the keys stored in objectKeys.
for key in objectKeys:
print key
if key == 'Taiwan':
print 'Eureka'
I haven't tested the above code, but I think you get the gist here :)

wjdata['product']['made']['Taiwan'] works

Python extract json structure from html page

in python i'm reading an html page content which contains a lot of stuff.
To do this i read the webpage as string by this way:
url = 'https://myurl.com/'
reqq = req.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
reddit_file = req.urlopen(reqq)
reddit_data = reddit_file.read().decode('utf-8')
if i print the reddit_data i can see correctly the whole html contents.
Now, inside it there's a structure like json that i would like to read and extract some fields from that.
Below the structure:
"dealDetails" : {
"f240141a" : {
"egressUrl" : "https://ccc.com",
"title" : "ZZZ",
"type" : "ghi",
},
"5f9ab246" : {
"egressUrl" : "https://www.bbb.com/",
"title" : "YYY",
"type" : "def",
},
"2bf6723b" : {
"egressUrl" : "https://www.aaa.com//",
"title" : "XXX",
"type" : "abc",
},
}
What i want to do is: find the dealDetails field and then for each f240141a 5f9ab246 2bf6723b
get the egressURL, title and type values.
Thanks

Try this,
[nested_dict['egressUrl'] for nested_dict in reddit_data['dealDetails'].keys()]
To access the values of JSON, you can consider as dictionary and use the same syntax to access values as well.
Edit-1:
Make sure your type of reddit_data is a dictionary.
if type(reddit_data) is str.
You need to do..
import ast
reddit_data = ast.literal_eval(reddit_data)
OR
import json
reddit_data = json.loads(reddit_data)

If you just wanted to know how to access the egressURL, title and the type. You might just wanna read the answer below! Be careful however, cause the following code won't work unless you converted your HTML file reddit_data in something like a dictionary ( Modified shaik moeed's answer a tiny bit to also return title and type) :
[(i['egressUrl'], i['title'], i['type']) for i in reddit_data['dealDetails'].keys()]
However, If I got it right, the part you're missing is the conversion from HTML to a JSON friendly file. What I personally use, even though it's quite unpopular, is the eval function
dictionary = eval(reddit_data)
This will convert the whole file into a dictionary, I recommend that you only use it on the part of the text that 'looks' like a dictionary! (One of the reason eval is unpopular, is because it won't convert strings like 'true'/'false' to Python's True/False, be careful with that :) )
Hope that helped!