I have two dictionaries - test1 and test2. I have to recursively compare both, if test1 contains key description$$, I have to replace second test2 of same key with the value of test1 key and then dump this in to a JSON file. I was able to get this, but the output of JSON file is not as of expected format.
sample.py
import json
test1 = {
"info" : {
"title" : "some random data",
"description$$" : "CHANGED::::",
"version" : "x.x.1"
},
"schemes" : [ "https" ],
"basePath" : "/sch/f1"
}
test2 = {
"info" : {
"title" : "some random data",
"description" : "before change",
"version" : "x.x.4"
},
"schemes" : [ "https" ],
"basePath" : "/sch/f2"
}
def walk(test1, test2):
for key, item in test1.items():
if type(item) is dict:
walk(item, test2[key])
else:
if str(key) == "description$$" or str(key) == "summary$$":
modfied_key = str(key)[:-2]
test2[modfied_key] = test1[key]
walk(test1, test2)
json.dump(test2, open('outputFile.json', "w"), indent=2)
My output is -
outpufile.json
{
"info": {
"title": "some random data",
"description": "CHANGED::::",
"version": "x.x.4"
},
"schemes": [
"https"
],
"basePath": "/sch/f2"
}
but the expected output should be -
{
"info": {
"title": "some random data",
"description": "CHANGED::::",
"version": "x.x.4"
},
"schemes": ["https"],
"basePath": "/sch/f2"
}
the schema should be printed in single line, but in my output it's taking 3 lines. how can I fix this?
Thank you
Related
trying to find out correct syntax to traverse through through the list to get all values and insert into oracle.
edit:
Below is the json structure :
[{
"publishTime" : "2021-11-02T20:18:36.223Z",
"data" : {
"DateTime" : "2021-11-01T15:10:17Z",
"Name" : "x1",
"frtb" : {
"code" : "set1"
},
"mainAccounts" : [ {
"System" : {
"identifier" : {
"domain" : "System",
"id" : "xxx"
},
"name" : "TEST1"
},
"Account" : "acc1"
}, {
"System" : {
"identifier" : {
"domain" : "System",
"id" : "xxx"
},
"name" : "TEST2"
},
"Account" : "acc2"
}, {
"System" : {
"identifier" : {
"domain" : "System",
"id" : "xxx"
},
"name" : "TEST3"
},
"Account" : "acc3"
}],
"sub" : {
"ind" : false,
"identifier" : {
"domain" : "ops",
"id" : "1",
"version" : "1"
}]
My python code :
insert_statement = """INSERT INTO mytable VALUES (:1,:2)"""
r =requests.get(url, cert=(auth_certificate, priv_key), verify=root_cert, timeout=3600)
data=json.loads(r.text)
myrows = []
for item in data:
try:
name = (item.get("data").get("Name"))
except AttributeError:
name=''
try:
account= (item.get("data").get("mainAccounts")[0].get("Account") )
except TypeError:
account=''
rows=(name,account)
myrows.append(rows)
cursor.executemany(insert_statement,myrows)
connection_target.commit()
with the above i only get first value for 'account' in list i.e. ("acc1") , how to get all the values i.e. (acc1,acc2,acc3) ?
I have tried below with no success :
try:
Account = (item.get("data").get("mainAccounts")[0].get("Account") for item in data["mainAccounts")
except TypeError:
Account= ''
please advise.Appreciate your help always.
import json
my_json = json.load(open("my_json.json", 'r'))
>>> mainAccounts = my_json['mainAccounts']
>>> for account in mainAccounts:
... account_name = account['Account']
... system_domain = account['System']['identifier']['domain']
... system_id = account['System']['identifier']['id']
... system_name = account['System']['name']
... print(f"\n======={account_name}==========")
... print(f"System domain: {system_domain}")
... print(f"System id: {system_id}")
... print(f"System name: {system_name}")
=======acc1==========
System domain: System
System id: xxx
System name: TEST1
=======acc2==========
System domain: System
System id: xxx
System name: TEST2
=======acc3==========
System domain: System
System id: xxx
System name: TEST3
I have not worked with Oracle before so I'm going to assume your SQL statement is correct and requires a tuple of (name, account_number).
data = [
{
"publishTime" : "2021-11-02T20:18:36.223Z",
"data" : {
"DateTime" : "2021-11-01T15:10:17Z",
"Name" : "x1",
"frtb" : {
"code" : "set1"
},
"mainAccounts" : [
{
"System" : {
"identifier" : {
"domain" : "System",
"id" : "xxx"
},
"name" : "TEST1"
},
"Account" : "acc1"
}, {
"System" : {
"identifier" : {
"domain" : "System",
"id" : "xxx"
},
"name" : "TEST2"
},
"Account" : "acc2"
}, {
"System" : {
"identifier" : {
"domain" : "System",
"id" : "xxx"
},
"name" : "TEST3"
},
"Account" : "acc3"
}
],
"sub" : {
"ind" : False,
"identifier" : {
"domain" : "ops",
"id" : "1",
"version" : "1"
}
}
}
}
]
myrows = []
for item in data:
accounts = [
(act["System"]["name"], act["Account"])
for act in item.get("data", {}).get("mainAccounts", [])
]
myrows.extend(accounts)
print(myrows) # [('TEST1', 'acc1'), ('TEST2', 'acc2'), ('TEST3', 'acc3')]
And then assuming it's in the correct format.
cursor.executemany(insert_statement, myrows)
How it works:
This line gets your top level dict and assigns it to item.
I used a for loop in case there are multiple dicts in the list.
for item in data:
The list comprehension is a bit more complex. But it can be translated to this.
accounts = []
for act in item.get("data", {}).get("mainAccounts", []):
name = act["System"]["name"]
account = act["Account"]
accounts.append(name, account)
Or simplified even more:
data_dict = item.get("data", {})
main_accounts_list = data_dict.get("mainAccounts", [])
for act in main_accounts_list:
# the rest of the code
I used dict.get() with an empty dict and list as a default so that even if the keys do not exist, it will still be able to process the subsequent .get and iteration without erroring.
Finally, you extend my_rows rather than append so you add don't get a list of list.
myrows.extend(accounts)
I need to create a json string like in the example below and I am thinking of using a dict that I can ultimately json.dumps to a json string. I will build this dict in a loop. This is what the json should look like
{
"big-data-list" :[
{
"indexnum": "1",
"components" :
[
{
"key": "some-key1",
"item" :"item name",
"data" :"some string",
}
]
},
{
"indexnum": "2",
"components" :
[
{
"key": "some-key2",
"item" :"item name 2",
"data" :"some string 2",
},
{
"key": "some-key3",
"item" :"item name 3",
"data" :"some string 3",
}
]
}
}
Here is what I tried without a loop to see how things work
bigdata= {}
indexnum= {}
componentList = {}
indexnum["components"] = {}
indexnum["mileage"] = 20
componentList["key"] = "some-key1"
componentList["item"] = "item name"
componentList["data"] = "some string"
indexnum["components"][0] = componentList
componentList["key"] = "some-key2"
componentList["item"] = "item name 2"
componentList["data"] = "some string 2"
indexnum["components"][1] = componentList
print(json.dumps(indexnum))
What I end up getting looks like this:
{"components": {"0": {"key": "somekey2", "item": "fuel2"}, "1": {"key": "somekey2", "item": "fuel2"}}, "mileage": 20}
How do I build the dict so I can json dump it in the way I need to? Is there a better way to come up with such a json object as represented in the example above?
You basically just need to work on your logic to put your data in the appropriate structure of dicts and lists.
Below is an example of a loop that puts some data into the specified structure:
>>> # say you originally have your data in the following list
... lists_of_components = [
... [("some-key1", "item name", "some string")],
... [("some-key2", "item name 2", "some string 2"),
... ("some-key3", "item name 3", "some string 3")],
... ]
... bigdata = {}
... bigdata["big-data-list"] = []
... for i, components in enumerate(lists_of_components, 1):
... bigdata["big-data-list"].append({
... "indexnum": str(i),
... "components": [
... {k: v for k, v in zip(["key", "item", "data"], component)}
... for component in components]
... })
... print(json.dumps(bigdata, indent=4))
{
"big-data-list": [
{
"indexnum": "1",
"components": [
{
"key": "some-key1",
"item": "item name",
"data": "some string"
}
]
},
{
"indexnum": "2",
"components": [
{
"key": "some-key2",
"item": "item name 2",
"data": "some string 2"
},
{
"key": "some-key3",
"item": "item name 3",
"data": "some string 3"
}
]
}
]
}
In Python, how do i find if a dictionary contains data from the other dictionary.
my data is assigned to a variable like this
childDict = {
"assignee" : {
"first":"myFirstName",
"last":"myLastName"
},
"status" : "alive"
}
I have another dictionary named masterDict with similar hierarchy but with some more data in it.
masterDict = {
"description": "sample description",
"assignee" : {
"first" : "myFirstName",
"last" : "myLastName"
},
"status" : "dead",
"identity": 1234
}
Now I need to read through childDict and find out if masterDict has these values in them or not.
data is nested, it can have more depth.
In the above example since the status didn't match, it should return false otherwise it should have returned true. how do i compare them. I am new to python. Thanks for your help.
Note that there were some errors in your dictionary (missing commas).
childDict1 = {
"assignee": {
"first":"myFirstName",
"last":"myLastName"
},
"status" : "alive"
}
childDict2 = {
"assignee": {
"first":"myFirstName",
"last":"myLastName"
},
"status" : "dead"
}
masterDict = {
"description": "sample description",
"assignee": {
"first":"myFirstName",
"last":"myLastName"
},
"status": "dead",
"identity": 1234
}
def contains_subdict(master, child):
if isinstance(master, dict) and isinstance(child, dict):
for key in child.keys():
if key in master:
if not contains_subdict(master[key], child[key]):
return False
return True
else:
if child == master:
return True
return False
print contains_subdict(masterDict, childDict1)
print contains_subdict(masterDict, childDict2)
Running the code produces the output:
False
True
I am having a json data as :
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"_default_":{
"_timestamp" : {
"enabled" : true,
"store" : true
}
}
}
}
I am able to write a python code to store it , my python looks like:
import json
ES = {}
settings = []
mappings = []
_default_ = []
_timestamp = []
#settings.append({"number_of_shards" : "1"})
#ES["settings"] = settings
m={}
c={}
_timestamp.append({"enabled" : "true", "store" : "true"})
m["_timestamp"]=_timestamp
_default_.append(m)
c["_default_"]=_default_
mappings.append(c)
ES["mappings"] = mappings
settings.append({"number_of_shards" : "1"})
ES["settings"] = settings
print json.dumps(ES,indent=2, separators=(',', ': '))
the above code runs fine but it prints the things as :
{
"mappings": [
{
"_default_": [
{
"_timestamp": [
{
"enabled": "true",
"store": "true"
}
]
}
]
}
],
"settings": [
{
"number_of_shards": "1"
}
]
}
I am a little hazzy on json stuff.. what am i missing ??? Any help would be highly appreciated
You should use dictionaries, not arrays, anyway all you code can be written like this (since ES is just a python object it can be mapped to JSON directly, without any manipulations):
#!/usr/bin/python
import json
ES = {
"settings": {
"number_of_shards" : "1"
},
"mappings": {
"_default_": {
"_timestamp_": {
"enabled" : "true",
"store" : "true"
}
}
}
}
print json.dumps(ES,indent=2, separators=(',', ': '))
If you want to add some keys and values to it:
ES["some_other_key"] = {"some_other_sub_key" : "whatever"}
I am wondering how do you update a nested array with PyMongo/MongoDB by selecting a document(row) and then going into the nested array and selecting a specific object.
{
"_id" : "12345",
"name" : "John Doe,
"mylist" : [
{
"nested_id" : "1",
"data1" : "lorem ipsum",
"data2" : "stackoverflow",
"data3" : "james bond"
},
{
"nested_id" : "2",
"data1" : "lorem ipsum",
"data2" : "stackoverflow",
"data3" : "james bond"
},
{
....
}
]
}
and then lets say you pass a discretionary with the elements you want to update. In this example only update data1 and data3
data = {
"data1" : "new lorem",
"data3" : "goldeneye"
}
I have tried with the following syntax, but with no success.
db.testing.find_and_modify(
query={"_id": "12345", 'mylist.nested_id' : "1"},
update={"$set": {'mylist' : data}})
what it should look like after the update
{
"_id" : "12345",
"name" : "John Doe,
"mylist" : [
{
"nested_id" : "1",
"data1" : "new lorem",
"data2" : "stackoverflow",
"data3" : "goldeneye"
},
{
"nested_id" : "2",
"data1" : "lorem ipsum",
"data2" : "stackoverflow",
"data3" : "james bond"
},
{
....
}
]
}
Use "dot notation" and the positional operator in the update portion. Also transform your input to match the "dot notation" form for the key representation:
# Transform to "dot notation" on explicit field
for key in data:
data["mylist.$." + key] = data[key]
del data[key]
# Basically makes
# {
# "mylist.$.data1": "new lorem",
# "mylist.$.data3": "goldeneye"
# }
db.testing.find_and_modify(
query = {"_id": "12345", 'mylist.nested_id' : "1"},
update = { "$set": data }
)
So that will transpose $ to the actual matched element position from the query portion of the update. The matched array element will be updated and using "dot notation" only the mentioned fields will be affected.
Have no idea what "service" is supposed to mean in this context and I am just treating it as a "transcribing error" since you are clearly trying to match an array element in position.
That could be cleaner, but this should give you the general idea.