We Keep Coding

How to parse nested JSON object?

I am working on a new project in HubSpot that returns nested JSON like the sample below. I am trying to access the associated contacts id, but am struggling to reference it correctly (the id I am looking for is the value '201' in the example below). I've put together this script, but this script only returns the entire associations portion of the JSON and I only want the id. How do I reference the id correctly?
Here is the output from the script:
{'contacts': {'paging': None, 'results': [{'id': '201', 'type': 'ticket_to_contact'}]}}
And here is the script I put together:
import hubspot
from pprint import pprint
client = hubspot.Client.create(api_key="API_KEY")
try:
api_response = client.crm.tickets.basic_api.get_page(limit=2, associations=["contacts"], archived=False)
for x in range(2):
pprint(api_response.results[x].associations)
except ApiException as e:
print("Exception when calling basic_api->get_page: %s\n" % e)
Here is what the full JSON looks like ('contacts' property shortened for readability):
{
"results": [
{
"id": "34018123",
"properties": {
"content": "Hi xxxxx,\r\n\r\nCan you clarify on how the blocking of script happens? Is it because of any CSP (or) the script will decide run time for every URL’s getting triggered from browser?\r\n\r\nRegards,\r\nLogan",
"createdate": "2019-07-03T04:20:12.366Z",
"hs_lastmodifieddate": "2020-12-09T01:16:12.974Z",
"hs_object_id": "34018123",
"hs_pipeline": "0",
"hs_pipeline_stage": "4",
"hs_ticket_category": null,
"hs_ticket_priority": null,
"subject": "RE: call followup"
},
"createdAt": "2019-07-03T04:20:12.366Z",
"updatedAt": "2020-12-09T01:16:12.974Z",
"archived": false
},
{
"id": "34018892",
"properties": {
"content": "Hi Guys,\r\n\r\nI see that we were placed back on the staging and then removed again.",
"createdate": "2019-07-03T07:59:10.606Z",
"hs_lastmodifieddate": "2021-12-17T09:04:46.316Z",
"hs_object_id": "34018892",
"hs_pipeline": "0",
"hs_pipeline_stage": "3",
"hs_ticket_category": null,
"hs_ticket_priority": null,
"subject": "Re: Issue due to server"
},
"createdAt": "2019-07-03T07:59:10.606Z",
"updatedAt": "2021-12-17T09:04:46.316Z",
"archived": false,
"associations": {
"contacts": {
"results": [
{
"id": "201",
"type": "ticket_to_contact"
}
]
}
}
}
],
"paging": {
"next": {
"after": "35406270",
"link": "https://api.hubapi.com/crm/v3/objects/tickets?associations=contacts&archived=false&hs_static_app=developer-docs-ui&limit=2&after=35406270&hs_static_app_version=1.3488"
}
}
}

You can do api_response.results[x].associations["contacts"]["results"][0]["id"].

Sorted this out, posting in case anyone else is struggling with the response from the HubSpot v3 Api. The response schema for this call is:
Response schema type: Object
String results[].id
Object results[].properties
String results[].createdAt
String results[].updatedAt
Boolean results[].archived
String results[].archivedAt
Object results[].associations
Object paging
Object paging.next
String paging.next.after
String paging.next.linkResponse schema type: Object
String results[].id
Object results[].properties
String results[].createdAt
String results[].updatedAt
Boolean results[].archived
String results[].archivedAt
Object results[].associations
Object paging
Object paging.next
String paging.next.after
String paging.next.link
So to access the id of the contact associated with the ticket, you need to reference it using this notation:
api_response.results[1].associations["contacts"].results[0].id
notes:
results[x] - reference the result in the index
associations["contacts"] -
associations is a dictionary object, you can access the contacts item
by it's name
associations["contacts"].results is a list - reference
by the index []
id - is a string

In my case type was ModelProperty or CollectionResponseProperty couldn't reach dict anyhow.
For the record this got me to go through the results.
for result in list(api_response.results):
ID = result.id

How do I insert another nested document using pymongo?

Ahoy,
I have a document that looks like this:
{"_id": "123abc456def",
"name": "John Smith",
"address": [
{"street": "First St.", "date": "yesterday", "last_updated": "two days ago"}
],
"age": 123}
I try to add another street document using $push, it errors out with:
pymongo.errors.WriteError: The field 'address' must be an array but is of type object in document {_id: ObjectId('6049e88657e43d8801197c72')}
Code I'm using:
mydb3 = myclient["catalogue"]
mycolALL = mydb3["locations"]
query = {"charID": 0}
newvalue = {"$push": {"address": {"street": "test123", "date": "test123", "last_updated": "now123"}}}
mycolALL.update_one(query, newvalue)
Not making an address book or anything, just edited it so it makes a bit more sense to anyone without context.
My desired output would be that the document would look like this:
{"_id": "123abc456def",
"name": "John Smith",
"address": [
{"street": "First St.", "date": "yesterday", "last_updated": "two days ago"},
{"street": "test123", "date": "test123", "last_updated": "now123"}
],
"age": 123}
Normally I can google my way to an answer that makes the coin drop and JACKPOT! but this time I'm outta luck.
$set = it just changes the existing document, effectively replacing it. Which is not what I want.
$addToSet = for arrays only, error message: "pymongo.errors.WriteError: Cannot apply $addToSet to non-array field. Field named 'address' has non-array type object"
Anyone that can help?

Just a guess but are you sure you're looking at the right data / database.
Based on the data you posted your update_one() won't update that record because it doesn't match your filter {"charID": 0}

Parsing through a really long text

I'm a complete beginner to python, but I am making a web scraper as a project.
I'm using a jupyter notebook, beautifulsoup, and lxml.
I managed to grab the text that contains all the information I need, but now I'm lost on what to do.
I want to obtain specific pieces of data like longitude, latitude, siteid, direction (North, South, etc.), and I want to download the photos and rename them. I need to do this for all 41 locations.
If anyone could suggest any packages or methods I would really appreciate it! Thank you!
Here's a small portion of the text I grabbed (pattern repeats 41 times):
{
"count": 41,
"message": "success",
"results": [
{
"protocol": "land_covers",
"measuredDate": "2020-06-13",
"createDate": "2020-06-13T16:35:04",
"updateDate": "2020-06-15T14:00:10",
"publishDate": "2020-07-17T21:06:31",
"organizationId": 17043304,
"organizationName": "United States of America Citizen Science",
"siteId": 202689,
"siteName": "18TWK294769",
"countryName": null,
"countryCode": null,
"latitude": xx.xxx(edited),
"longitude": xx.xxx(edited),
"elevation": 25.4,
"pid": 163672280,
"data": {
"landcoversDownwardPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682247/original.jpg",
"landcoversEastExtraData": "(source: app, (compassData.horizon: -14.32171587255965))",
"landcoversEastPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682242/original.jpg",
"landcoversMucCode": null,
"landcoversUpwardPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682246/original.jpg",
"landcoversEastCaption": "",
"landcoversMeasurementLatitude": xx.xxx(edited),
"landcoversWestClassifications": null,
"landcoversNorthCaption": "",
"landcoversNorthExtraData": "(source: app, (compassData.horizon: -10.817734330181267))",
"landcoversDataSource": "GLOBE Observer App",
"landcoversDryGround": true,
"landcoversSouthClassifications": null,
"landcoversWestCaption": "",
"landcoversNorthPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682241/original.jpg",
"landcoversUpwardCaption": "",
"landcoversDownwardExtraData": "(source: app, (compassData.horizon: -84.48900393488086))",
"landcoversEastClassifications": null,
"landcoversMucDetails": "",
"landcoversMeasuredAt": "2020-06-13T15:12:00",
"landcoversDownwardCaption": "",
"landcoversSouthPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682243/original.jpg",
"landcoversMuddy": false,
"landcoversWestPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682245/original.jpg",
"landcoversStandingWater": false,
"landcoversLeavesOnTrees": true,
"landcoversUserid": 67150810,
"landcoversSouthExtraData": "(source: app, (compassData.horizon: -14.872806403121302))",
"landcoversSouthCaption": "",
"landcoversRainingSnowing": false,
"landcoversUpwardExtraData": "(source: app, (compassData.horizon: 89.09211989270894))",
"landcoversMeasurementElevation": 24.1,
"landcoversWestExtraData": "(source: app, (compassData.horizon: -15.47334477111039))",
"landcoversLandCoverId": 32043,
"landcoversMeasurementLongitude": xx.xxx(edited),
"landcoversMucDescription": null,
"landcoversSnowIce": false,
"landcoversNorthClassifications": null,
"landcoversFieldNotes": "(none)"
}
},
{
"protocol": "land_covers",
"measuredDate": "2020-06-13",
"createDate": "2020-06-13T16:35:04",
"updateDate": "2020-06-15T14:00:10",
"publishDate": "2020-07-17T21:06:31",
"organizationId": 17043304,
"organizationName": "United States of America Citizen Science",
"siteId": 202689,
"siteName": "18TWK294769",
"countryName": null,
"countryCode": null,
"latitude": xx.xxx(edited),
"longitude": xx.xxx(edited),
"elevation": 25.4,
"pid": 163672280,
"data": {
"landcoversDownwardPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682240/original.jpg",
"landcoversEastExtraData": "(source: app, (compassData.horizon: -6.06710116543897))",
"landcoversEastPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682235/original.jpg",
"landcoversMucCode": null,
"landcoversUpwardPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682239/original.jpg",
"landcoversEastCaption": "",
"landcoversMeasurementLatitude": xx.xxx(edited),
"landcoversWestClassifications": null,
"landcoversNorthCaption": "",
"landcoversNorthExtraData": "(source: app, (compassData.horizon: -9.199031748908894))",
"landcoversDataSource": "GLOBE Observer App",
"landcoversDryGround": true,
"landcoversSouthClassifications": null,
"landcoversWestCaption": "",
"landcoversNorthPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682233/original.jpg",
"landcoversUpwardCaption": "",
"landcoversDownwardExtraData": "(source: app, (compassData.horizon: -88.86569321651771))",
"landcoversEastClassifications": null,
"landcoversMucDetails": "",
"landcoversMeasuredAt": "2020-06-13T15:07:00",
"landcoversDownwardCaption": "",
"landcoversSouthPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682236/original.jpg",
"landcoversMuddy": false,
"landcoversWestPhotoUrl": "https://data.globe.gov/system/photos/2020/06/13/1682237/original.jpg",
"landcoversStandingWater": false,
"landcoversLeavesOnTrees": true,
"landcoversUserid": 67150810,
"landcoversSouthExtraData": "(source: app, (compassData.horizon: -11.615041431350335))",
"landcoversSouthCaption": "",
"landcoversRainingSnowing": false,
"landcoversUpwardExtraData": "(source: app, (compassData.horizon: 86.6284079864236))",
"landcoversMeasurementElevation": 24,
"landcoversWestExtraData": "(source: app, (compassData.horizon: -9.251774266832626))",
"landcoversLandCoverId": 32042,
"landcoversMeasurementLongitude": xx.xxx(edited),
"landcoversMucDescription": null,
"landcoversSnowIce": false,
"landcoversNorthClassifications": null,
"landcoversFieldNotes": "(none)"
}
},

It would help to see some code. Having said that, as has been pointed out, the built-in json library would help you. This is a JSON formatted output, please see here for an introduction to this type of format.
Say for sake your output here is stored within a variable called data. You could convert this json data to a dictionary.
Coding Example
import json
data_dict = json.load(data)
What json.load does is takes a JSON object and converts this into a python dictionary. The json.load actually scans the variable to check if it's a JSON object and uses a conversion table to convert this into a dictionary. There are other json formats that get converted to other python object types. See here for that table.
Now you have a python dictionary which you can access the data from. So lets go through longitude, latitude, siteid, direction (North, South, etc.). I see there's an open '[' without the corresponding ']'. I can only assume that the list has 41 items in it from what you're describing, so I'll take the first result first. You can always loop through this quite easily to get all 41 results.
longitude = data_dict['results'][0]['longitude']
langitude = data_dict['results'][0]['langitude']
site_id = data_dict['results'][0]['siteid']
Tips
I always use jupyter notebooks as a quick way to try and grab the specific data I want from JSON objects, sometimes it can take a few goes at getting the right access the right part. That way when I write my variables I know I get the data I want back from the JSON object. Json objects can be heavily nested sometimes and can be hard to follow.

Mongodb how to update many and set profile specific to id

I have a list of ids called batch i want to update all of them to set a field called fetched to true.
Original Test Collection
[{
"user_id": 1,
},
{
"user_id": 2,
}
]
batch variable
[1, 2]
UpdateMany:
mongodb["test"].update_many({"user_id": {"$in": batch}}, {"$set": {"fetched": True}})
I can do that using the above statement.
I also have another variable called user_profiles which is a list/array of json objects. I now ALSO want to set a field profile to be the profile found in the list(user_profiles) where the id matches the user_id/batch(id) i am updating.
user_profiles
[{
"id": 1,
"name": "john"
},
{
"id": 2,
"name": "jane"
}
]
Expected Final Result
[{
"user_id": 1,
"fetched": true,
"profile": {
"id": 1,
"name": "john"
}
},
{
"user_id": 2,
"fetched": true,
"profile": {
"id": 2,
"name": "jane"
}
}
]
I have a millions of these documents so i am trying to keep performance in mind.

You'll want to use db.collection.bulkWrite, see the updateOne example in the docs
If you've got millions you'll want to batch the bulkWrites into smaller chunks that work with your database server's capabilities.
Edit:
#Kay I just re-read the second part of your question which I didn't address earlier. You may want to try the $out stage of the aggregation pipeline. Be careful though since it will overwrite the existing collection so if you don't project all fields you could lose data. Definitely worth using a temporary collection for testing first.
Finally, you could also just create a view based on the aggregation query (with $lookup) if you don't absolutely need that data physically stored in the same collection.

Update a few fields in nested array in mongodb (with pymongo)

I am trying to update a few fields within an array within array
An example document is like this:
{
id: 987654321
tweets: [
{
text: "RT #947FreshFM: A vigil will be held for #SandyHook victims at UMd. at 7pm at Nyumburu Ampitheater. #BlackTerp",
urls: [
],
id: 279718351289348100
},
{
text: "RT #WTOP: McDonnell: If you talk completely about guns, I think you're missing the point. #AskTheGov http://t.co/hbFt7t1n",
urls: [
{
status: null,
domain: null,
url: "http://t.co/hbFt7t1n",
period: null,
resolved_url: null,
annotation: null
}
],
id: 281061376275906560
}
],
}
I want to update the urls array as:
urls: [
{
status: null,
domain: "wtop.com",
url: "http://t.co/hbFt7t1n",
period: null,
resolved_url: "http://wtop.com/?nid=610&sid=3162096",
annotation: null,
annotation2: "guncontrol"
}
],
I am using something like this to do the updating:
collection.update({"id":987654321, "tweets.id":281061376275906560,"tweets.urls.url":"http://t.co/hbFt7t1n"},
{"$set":{
"tweets.urls.resolved_url":"http://wtop.com/?nid=610&sid=3162096",
"tweets.urls.domain": "wtop.com",
"tweets.urls.annotation2":"guncontrol"
}}, False,False)
However it gives me error
can't append to array using string field name [urls]
Any suggestions?

I can't be certain, but this might be what's going on. From your example document, your model has this schema:
{
id: Number,
tweets: Array
}
When you search for the instance of the model in
collection.update( {
"id": 987654321,
"tweets.id": 281061376275906560,
"tweets.urls.url": "http://t.co/hbFt7t1n"
}, ...)`
It's likely that it's not finding the instance of the model you're looking for.
I would try running this script to see if your search criteria are valid:
console.log(collection.find({
"id": 987654321,
"tweets.id": 281061376275906560,
"tweets.urls.url": "http://t.co/hbFt7t1n"
}));
Hope it helps!