We Keep Coding

How can i insert this JSON file into a MongoDB collection?

I'm trying to insert this object into a mongo DB collection. I've tried a lot of ways and haven't gotten any results. I was wondering if anyone here could help me. The main problem is when passing the array of items into the key items.
The JSON File is similar to this:
{
'code': 'iuhuilknlkn',
'description': 'nllksnd',
'currency': 'Mxn',
'items': [
{
'item': {
'_id': {
'$oid': '60065d253ef6d468ced3603f'
},
'code': '2',
'description': '22',
'currency': 'Mxn',
'MU': 'Hr',
'sellingPrice': 1,
'buyingPrice': 3,
'supplier': None,
'cid': '5fbd81b32b325e5ca15fe5c9',
'itemAmount': 1,
'itemProductPrice': 1,
'itemAmountPrice': 1
}
},
{
'item': {
'_id': {
'$oid': '6011c18883a280ae0e5b8185'
},
'code': 'prb-001',
'description': 'prueba 1 artículo',
'currency': 'Mxn',
'MU': 'Ser',
'sellingPrice': 100.59,
'buyingPrice': 12,
'supplier': None,
'cid': '5fbd81b32b325e5ca15fe5c9',
'itemAmount': 1,
'itemProductPrice': 100.59,
'itemAmountPrice': 100.59
}
}
],
'price': 101.59
}
and the code I'm using right now is the following:
productsM.insert({
'code':newP['code'],
'description':newP['description'],
'currency' :newP['currency'],
'items' : [([val for dic in newP['items'] for val in
dic.values()])],
'price' : newP['price'],
'cid': csHe })
The error I get is the next one:
key '$oid' must not start with '$'

Your error is quite clear. Simply removing the $ from the $oid keys in your dictionary/json will resolve the issue. I don't think you can have a $ in key names since they are reserved for operators such as $in or $regex.
I removed the $ from the $oid keys and it worked like a charm. All I did was the following:
data = {
"code": "iuhuilknlkn",
"description": "nllksnd",
"currency": "Mxn",
"items": [
{
"item": {
"_id": {
"oid": "60065d253ef6d468ced3603f"
},
"code": "2",
"description": "22",
"currency": "Mxn",
"MU": "Hr",
"sellingPrice": 1,
"buyingPrice": 3,
"supplier": None,
"cid": "5fbd81b32b325e5ca15fe5c9",
"itemAmount": 1,
"itemProductPrice": 1,
"itemAmountPrice": 1
}
},
{
"item": {
"_id": {
"oid": "6011c18883a280ae0e5b8185"
},
"code": "prb-001",
"description": "prueba 1 artículo",
"currency": "Mxn",
"MU": "Ser",
"sellingPrice": 100.59,
"buyingPrice": 12,
"supplier": None,
"cid": "5fbd81b32b325e5ca15fe5c9",
"itemAmount": 1,
"itemProductPrice": 100.59,
"itemAmountPrice": 100.59
}
}
],
"price": 101.59
}
db.insert(data)

Looping through JSON (Foursquare API)

I am trying to extract data from a JSON file, of which a snippet is below. I want to loop through it to get all categories>name and get , as in this case, "Convenience Store" as a result.
{
'meta': {
'code': 200,
'requestId': '5ea184baedbcad001b7a3f8c'
},
'response': {
'venues': [
{
'id': '4d03b2f6dc45a093b4b0e5c6',
'name': 'Ozbesa Market',
'location': {
'address': 'Acibadem basogretmen sokak',
'lat': 41.00622726261631,
'lng': 29.051791450375678,
'labeledLatLngs': [
{
'label': 'display',
'lat': 41.00622726261631,
'lng': 29.051791450375678
}
],
'distance': 92,
'cc': 'TR',
'country': 'Türkiye',
'formattedAddress': [
'Acibadem basogretmen sokak',
'Türkiye'
]
},
'categories': [
{
'id': '4d954b0ea243a5684a65b473',
'name': 'Convenience Store',
'pluralName': 'Convenience Stores',
'shortName': 'Convenience Store',
'icon': {
'prefix': 'https://ss3.4sqi.net/img/categories_v2/shops/conveniencestore_',
'suffix': '.png'
},
'primary': True
}
],
'referralId': 'v-1587643627',
'hasPerk': False
},
Here is my for loop, please help me fix it. It is only returning just convenience stores, but there are also others like 'shopping mall', 'residential building', etc.
for ven in json_data:
for cat in ven:
print(json_data['response']['venues'][0]['categories'][0]['name'])
Thanks in advance!

For the sake of example, I elided some of the per-venue data and added some categories... but as I mentioned in the comment, you're not using the values you loop over.
json_data = {
"meta": {"code": 200, "requestId": "5ea184baedbcad001b7a3f8c"},
"response": {
"venues": [
{
"name": "Ozbesa Market",
"categories": [
{"name": "Convenience Store", "primary": True},
{"name": "Imaginary Category", "primary": False},
],
},
{
"name": "Another Location",
"categories": [
{"name": "Bus Station", "primary": True},
{"name": "Fun Fair", "primary": False},
],
},
]
},
}
for venue in json_data["response"]["venues"]:
print(venue["name"])
for cat in venue["categories"]:
print("..", cat["name"])
will output e.g.
Ozbesa Market
.. Convenience Store
.. Imaginary Category
Another Location
.. Bus Station
.. Fun Fair

How to iterate over JSON array to find value with changing keys?

I am trying to extract many values from a JSON array so I am iterating through it to extract the values based on their keys, however one of the keys changes depending on the item and I am getting a KeyError when the loop comes across the different key.
I've tried using try and except to catch this but since I am looping through the entire array it will throw the same exception for the other key this time.
Here is my code to extract the values:
df = []
for item in json_response["items"]:
df.append({
'AccountName': item["accountName"],
'Action': item["action"],
'Application': item["application"],
'AppID': item["attributes"]["appId"],
'AppName': item["attributes"]["AppName"],
'Errors': item["attributes"]["errors"],
'ContextID': item["contextid"],
'Created': item["created"],
'HostName': item["hostname"],
'EventID': item["id"],
'Info': item["info"],
'ipaddr': item["ipaddr"],
'EventSource': item["source"],
'Stack': item["stack"],
'Target': item["target"],
'TrackingID': item["trackingId"],
'Type': item["type"]
})
Here is an example JSON from a larger array I am extracting from:
{
"accountName": null,
"action": "Disable",
"application": "Application1",
"attributes": {
"appId": "7d264050024",
"AppName": "Application1",
"errors": [
"Rule: Rule not found."
]
},
"contextid": null,
"created": 1553194821098,
"hostname": null,
"id": "ac09ea0082",
"info": null,
"ipaddr": null,
"source": "System1",
"stack": null,
"target": "TargetName1.",
"trackingId": null,
"type": null
}
This would work but sometimes the "attributes" looks like:
"attributes": {
"appId": "7d2451684288",
"cloudAppName": "Application1",
"RefreshFailure": true
}
How can I extract either the "errors" value or the "RefreshFailure" value when iterating over the entire array?

Test key existence in attributes to retrieve the different values:
df = []
for item in json_response["items"]:
errors = "NA"
if "errors" in item["attributes"]
errors = item["attributes"]["errors"]
elif "RefreshFailure" in item["attributes"]:
errors = item["attributes"]["RefreshFailure"]
df.append({
'AccountName': item["accountName"],
'Action': item["action"],
'Application': item["application"],
'AppID': item["attributes"]["appId"],
'AppName': item["attributes"]["AppName"],
'Errors': errors,
'ContextID': item["contextid"],
'Created': item["created"],
'HostName': item["hostname"],
'EventID': item["id"],
'Info': item["info"],
'ipaddr': item["ipaddr"],
'EventSource': item["source"],
'Stack': item["stack"],
'Target': item["target"],
'TrackingID': item["trackingId"],
'Type': item["type"]
})

I tried to emulate your data to make the code work.
import json
from pprint import pprint
json_data = '''
{
"items": [
{
"accountName": null,
"action": "Disable",
"application": "Application1",
"attributes": {
"appId": "7d264050024",
"AppName": "Application1",
"errors": [
"Rule: Rule not found."
]
},
"contextid": null,
"created": 1553194821098,
"hostname": null,
"id": "ac09ea0082",
"info": null,
"ipaddr": null,
"source": "System1",
"stack": null,
"target": "TargetName1.",
"trackingId": null,
"type": null
},
{
"accountName": null,
"action": "Disable",
"application": "Application1",
"attributes": {
"appId": "7d2451684288",
"cloudAppName": "Application1",
"RefreshFailure": true
},
"contextid": null,
"created": 1553194821098,
"hostname": null,
"id": "ac09ea0082",
"info": null,
"ipaddr": null,
"source": "System1",
"stack": null,
"target": "TargetName1.",
"trackingId": null,
"type": null
}
]
}'''
json_response = json.loads(json_data)
def capitalize(s):
return s[0].upper() + s[1:]
df = []
for item in json_response["items"]:
d = {}
# Iterate over the items in the dictionary/json object and add them one by one using a loop
# This will work even if the items in the json_response changes without having to change the code
for key, value in item.items():
# "attributes" is itself a dictionary/json object
# Its items have to be unpacked and added instead of adding it as a raw object
if isinstance(value, dict):
for k, v in value.items():
d[capitalize(k)] = v
else:
d[capitalize(key)] = value
df.append(d)
pprint(df)
Output:
[{'AccountName': None,
'Action': 'Disable',
'AppId': '7d264050024',
'AppName': 'Application1',
'Application': 'Application1',
'Contextid': None,
'Created': 1553194821098,
'Errors': ['Rule: Rule not found.'],
'Hostname': None,
'Id': 'ac09ea0082',
'Info': None,
'Ipaddr': None,
'Source': 'System1',
'Stack': None,
'Target': 'TargetName1.',
'TrackingId': None,
'Type': None},
{'AccountName': None,
'Action': 'Disable',
'AppId': '7d2451684288',
'Application': 'Application1',
'CloudAppName': 'Application1',
'Contextid': None,
'Created': 1553194821098,
'Hostname': None,
'Id': 'ac09ea0082',
'Info': None,
'Ipaddr': None,
'RefreshFailure': True,
'Source': 'System1',
'Stack': None,
'Target': 'TargetName1.',
'TrackingId': None,
'Type': None}]
If you want the key name to be Errors even when the actual key name is RefreshFailure, you can add these lines of code before df.append(d)
...
if 'RefreshFailure' in d:
d['Errors'] = d['RefreshFailure']
del d['RefreshFailure']
df.append(d)
With these few extra lines of code, the output would look like this:
[{'AccountName': None,
'Action': 'Disable',
'AppId': '7d264050024',
'AppName': 'Application1',
'Application': 'Application1',
'Contextid': None,
'Created': 1553194821098,
'Errors': ['Rule: Rule not found.'],
'Hostname': None,
'Id': 'ac09ea0082',
'Info': None,
'Ipaddr': None,
'Source': 'System1',
'Stack': None,
'Target': 'TargetName1.',
'TrackingId': None,
'Type': None},
{'AccountName': None,
'Action': 'Disable',
'AppId': '7d2451684288',
'Application': 'Application1',
'CloudAppName': 'Application1',
'Contextid': None,
'Created': 1553194821098,
'Errors': True,
'Hostname': None,
'Id': 'ac09ea0082',
'Info': None,
'Ipaddr': None,
'Source': 'System1',
'Stack': None,
'Target': 'TargetName1.',
'TrackingId': None,
'Type': None}]

Not able to Parse JSON Response

I am using Python2.7 and simple json module, I am able to get a response but when i want to do something with this JSON Response I am not able to do.
Python Code :
query_url = self.api_url + 'projects'
try:
req = urllib2.Request(query_url, None, {"Authorization": self._auth})
handler = self._opener.open(req)
except urllib2.HTTPError, e:
print e.headers
raise e
print simplejson.load(handler)
JSON Response:
{'start': 0, 'nextPageStart': 166, 'values': [{'description': 'This Repo is created to maintain the code versioning accordingly for My project', 'links': {'self': [{'href': 'https://bitbucket.xyz.xyz/projects/My'}]}, 'id': 757, 'key': 'MY', 'type': 'NORMAL', 'public': False, 'name': 'hub'},{'description': 'Services ', 'links': {'self': [{'href': 'https://bitbucket.xyz.xyz/projects/Hello'}]}, 'id': 1457, 'key': 'HE', 'type': 'NORMAL', 'public': False, 'name': 'Hello'}], 'limit': 25, 'isLastPage': False, 'size': 25}
Few data i have removed just kept first and last.
The error which i am observing
Error: Parse error on line 1:
..."NORMAL", "public": False, "name": "Advi
-----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined'
Can some body help me here what i am doing wrong here.

Because your json is invalid. Here it is valid version of your Json data.
First error Strings should be wrapped in double quotes.
Secondly boolean variable in javascprit is lower case. Use false instead of False
{
"start": 0,
"nextPageStart": 166,
"values": [{
"description": "This Repo is created to maintain the code versioning accordingly for My project",
"links": {
"self": [{
"href": "https://bitbucket.xyz.xyz/projects/My"
}]
},
"id": 757,
"key": "MY",
"type": "NORMAL",
"public": false,
"name": "hub"
},
{
"description": "Services ",
"links": {
"self": [{
"href": "https://bitbucket.xyz.xyz/projects/Hello"
}]
},
"id": 1457,
"key": "HE",
"type": "NORMAL",
"public": false,
"name": "Hello"
}
],
"limit": 25,
"isLastPage": false,
"size": 25
}

Json format for python

I'm rewriting a view based on what I know the final output should be in json but it's returning the dictionary as a string.
new output
{
"results":
["
{
'plot': u'',
'runtime': u'N/A',
'description': u'x',
'videos': [
{
'id': 823,
'name': u'x',
'youtube_id': u'FtcubOnXgZk'
}
],
'country': u'India',
'writer': u'Neetu Varma, Ranjeev Verma',
'name': u'Chalk N Duster',
'id': 940,
'director': u'Jayant Gilatar',
'hot': True,
'content': u'x',
'actors': u'Shabana Azmi, Arya Babbar, Gavie Chahal, Juhi Chawla',
'year': 2015,
'images': [
{'small': '/media/cache/62/fd/62fd5158d281c042e3cf1f919183e94e.jpg', 'medium': '/media/cache/5e/32/5e32ebb1a4d25bba0d0c70b4b448e948.jpg'}],
'trailer_youtube_id': u'FtcubOnXgZk',
'type': 'movie',
'slug': u'chalk-n-duster',
'categories': [{'parent_id': 2, 'id': 226, 'name': u'Drama'}],
'shows': {
'starts': '2016-01-16',
'booking_url': u'',
'venue': {
'address': u'',
'id': 854,
'name': u'Nyali Cinemax',
'area': {
'id': 52,
'parent': {
'id': 48,
'name': u'Mombasa'
},
'name': u'Nyali'
}
},
'starts_time': '18:30:00'
}
}", "{'plot': u'' ....
old output
"results": [
{
"actors": "x",
"categories": [
{
"id": 299,
"name": "Biography",
"parent_id": 2
},
],
"content": "x",
"country": "x",
"description": "x",
"director": "x",
"hot": true,
"id": 912,
"images": [
{
"medium": "/media/cache/d2/b3/d2b3a7885e7c39bfc5c2b297b66619c5.jpg",
"small": "/media/cache/e2/d0/e2d01b2c7c77d3590536666de4a7fd7d.jpg"
}
],
"name": "Bridge of Spies",
"plot": "x",
"runtime": "141 min",
"shows": [
{
"booking_url": "",
"starts": "2015-11-27",
"starts_time": "16:30:00",
"venue": {
"address": "The Junction Shopping Mall",
"area": {
"id": 68,
"name": "Ngong Road",
"parent": {
"id": 2,
"name": "Nairobi"
}
},
"id": 1631,
"name": "Century Cinemax Junction"
}
},
],
"slug": "bridge-of-spies",
"trailer_youtube_id": "",
"type": "movie",
"videos": [
{
"id": "795",
"name": "Bridge of Spies",
"youtube_id": "2-2x3r1m2I4"
}
],
"writer": "Matt Charman, Ethan Coen, Joel Coen",
"year": 2015
}, ...
]
Here's the view, I know the shows should also be a list, but in order to start testing I'll need the data to come in the right format. If it's involves too much rewriting I'm okay with links and explanation.
#memoize(timeout=60*60)
def movies_json():
today = datetime.date.today()
movies = Movie.objects.filter(shows__starts__gte=today)
results = []
number = len(movies)
for movie in movies:
print "Now Remaining: {0}".format(number)
number -= 1
medium = get_thumbnail(movie.picture(), '185x274', crop='center', quality=99).url
small = get_thumbnail(movie.picture(), '50x74', crop='center', quality=99).url
movie_details = {
'director':movie.director,
'plot':movie.plot,
'actors':movie.actors,
'content':movie.content,
'country':movie.country,
'description':movie.description,
'hot':movie.hot,
'id':movie.id,
'images':[{'medium':medium, 'small':small}],
'name':movie.name,
'plot':movie.plot,
'runtime':movie.runtime,
'slug':movie.slug,
'type':'movie',
'writer':movie.writer,
'year':movie.year,
}
youtube_details = movie.videos.filter(youtube_id__isnull=False)[0]
movie_details['trailer_youtube_id'] = youtube_details.youtube_id if youtube_details.youtube_id else ""
movie_details['videos'] = [
{
'id':youtube_details.id,
'name':movie.name,
'youtube_id':youtube_details.youtube_id,
}
]
shows = []
for show in movie.shows.all():
show_details = {
'booking_url':show.booking_url,
'starts':show.starts.isoformat(),
'starts_time':show.starts_time.isoformat(),
'venue': {
'address':show.venue.address,
'area': {
'id': show.venue.area.id,
'name': show.venue.area.name,
'parent': {
'id': show.venue.area.parent.id,
'name': show.venue.area.parent.name,
}
},
'id': show.venue.id,
'name': show.venue.name,
}
}
shows.append(show_details)
movie_details['shows'] = show_details
category_list = []
for category in movie.categories.all():
category_details = {
'id':category.id,
'name':category.name,
'parent_id':category.parent.id,
}
category_list.append(category_details)
movie_details['categories'] = category_list
results.append(movie_details)
return results
The data is returned by django rest framework 0.4.0

import json
json_obj = json.load(json_string)