{
"id": "APA91bE9N6D9Tp79gv1kUgWLhsCmbKPKJQlzgtr1iGKlL5249bzD5DxySBiaIzDmk7rOAdrWcNcP0ZxPnaj7e6Esc _iGIYJlDte-E1pMO9GME4QufgdQQOIccM2tExMd9L9RsQthR3160KbQeRmtfxW6gvuPXYN0zw",
"platform": "android",
"user": ObjectId("545b2833b21e898413de9314"),
"_id": ObjectId("545b5e76d6be01755625b284"),
"createdAt": Date(1415274102856),
"__v": 0
}{
"__v": 0,
"_id": ObjectId("545b67c4d6be01755625b2c1"),
"createdAt": Date(1415276484321),
"id": "APA91bFRxirYHIko33D1LiHODpBd77IlRhebK4tMRWecFxb5E6nfWSMFarr5mlwmY9bPQP56DGP7cnli4_jOrS8Ynn3Y9w9uaRoESoEPglqR-rA-3phsh8UtSxMC5lNoOqIrohz3hBjzzpCH_vExwo6B5yV6Mb8jyg",
"platform": "android",
"user": ObjectId("545b69a5d6be01755625b2d2")
}
Here is the content of the JSON File.
Code which i am using for importing is:
import json
with open("test.json") as json_file:
json_data = json.load(json_file)
print(json_data)
As #Puffin pointed out you need to handle ObjectId, Date etc if you are dumping a MongoDB BSON into JSON and accessing it.
If possible then use pymongo to access MongoDB directly from Python rather than dumping into JSON and accessing the data.
Your first string is not valid JSON. It is not possible to ingest this directly to JSON parser. What I would do, is to write a preprocessor, which expands the non JSON elements like ObjectId or Date to e.g. strings. Something in the line of this answer.
Related
I am looking for a way how the schema is set with a json file in Python on Big Query. The following document says I can set it with Schema field one by one, but I want to find out more efficient way.
https://cloud.google.com/bigquery/docs/schemas
Autodetect would be skeptical to make it in this case.
I will appreciate it if you helped me.
You can create a JSON file with columns/data types and use the below code to build BigQuery Schema.
JSON File (schema.json):
[
{
"name": "emp_id",
"type": "INTEGER"
},
{
"name": "emp_name",
"type": "STRING"
}
]
Python Code:
import json
from google.cloud import bigquery
bigquerySchema = []
with open('schema.json') as f:
bigqueryColumns = json.load(f)
for col in bigqueryColumns:
bigquerySchema.append(bigquery.SchemaField(col['name'], col['type']))
print(bigquerySchema)
Soumendra Mishra is already helpful, but here is a bit more general version that can optionally accept addition fields such as mode or description:
JSON File (schema.json):
[
{
"name": "emp_id",
"type": "INTEGER",
"mode": "REQUIRED"
},
{
"name": "emp_name",
"type": "STRING",
"description": "Description of this field"
}
]
Python Code:
import json
from google.cloud import bigquery
table_schema = []
# open JSON file read only
with open('schema.json', 'r') as f:
table_schema = json.load(f)
for entry in table_schema:
# rename key; bigquery.SchemaField expects `field` to be called `field_type`
entry["field_type"] = entry.pop("type")
# ** effectively provides data as argument:value pairs (e.g. name="emp_id")
table_schema.append(bigquery.SchemaField(**entry))
I am trying to figure out on how to convert the following JSON object in to a dataframe
[
{
"id":"123",
"sources":[
{
"name":"ABC",
"first":"2020-02-26T03:19:23.247Z",
"last":"2020-02-26T03:19:23.247Z"
},
{
"name":"XYZ",
"first":"2020-02-26T03:19:23.247Z",
"last":"2020-02-26T03:19:23.247Z"
}
]
}
]
The dataframe should appear like this.
id ABC.first ABC.last XYZ.first XYZ.last
123 2020-02-26.. 2020-02-26.. 2020-02-26.. 2020-02-26T03:19:23.247Z
Thanks in advance for your help
Python includes the useful json module. You can find some documentation for it here.
To use it , you'll need to import it into your Python script using the import json command.
From there, you can use the json.loads() method and pass it a some JSON. The loads() method returns a Python dictionary which then lets you access the properties of your JSON object by name. Some example code might look like:
import json
# some JSON:
personJSON = '{"name":"John", "age":30, "city":"New York", "address": {"street": "Fake Street", "streetNumber": "123"}}'
# parse personJSON:
personDict = json.loads(personJSON)
# the result is a Python dictionary:
print(personDict["name"]) # Will print "John"
print(personDict["address"]["street"]) # Will print "Fake Street"
Good luck!
I would like to grab some values within a JSON object using Python and assign them to variables for further use in the html frontend.
I tried it using the w3c resource and by googling but I can't get any successful print:
import requests
import json
response = requests.get("https://www.api-football.com/demo/api/v2/teams/team/33")
team_data = response.json()
team_name = team_data.teams.name
print(team_name)
This is the JSON Object i get from the external API:
{
"api": {
"results": 1,
"teams": [
{
"team_id": 33,
"name": "Manchester United",
"code": "MUN",
"logo": "https://media.api-football.com/teams/33.png",
"country": "England",
"founded": 1878,
"venue_name": "Old Trafford",
"venue_surface": "grass",
"venue_address": "Sir Matt Busby Way",
"venue_city": "Manchester",
"venue_capacity": 76212
}
]
}
}
The debug console tells me AttributeError: 'dict' object has no attribute 'teams'
JSONs (JavaScript Object Notation) are overall JavaScript objects used for storing serialized data. The python interpreter has an specific module import jsonthat allows to charge JSON files to Python dicts in the RAM of your machine.
In your situation:
with open('team_data.json', "r") as fobj:
dataset = json.load(fobj)
print('Teams data: ', dataset['api']['teams'])
From here you can work with it as a normal dict.
I want to insert document to the collection from json file it says bson.errors.InvalidDocument: key '$oid' must not start with '$'
How can I solve it?
example of document:
[{"name": "Company", "_id": {"$oid": "1234as123541gsdg"}, "info": {"email": "test#gmail.com"}}]
Represent ObjectIds in Python with the bson.ObjectId class:
from bson import ObjectId
_id = ObjectId("5899e0aca600741755433908")
So for a complete example:
from bson import ObjectId
collection.insert(
{"name": "Company", "_id": ObjectId("5899e0aca600741755433908"),
"info": {"email": "test#gmail.com"}})
In order to load MongoDB Extended JSON data, use PyMongo's json_util.loads:
from bson.json_util import loads
json_str = '[{"name": "Company", "_id": {"$oid": "5899e0aca600741755433908"}, "info": {"email": "test#gmail.com"}}]'
data = loads(json_str)
print(data)
for doc in data:
collection.insert(doc)
"loads()" converts from the Extended JSON syntax with "$oid" to an actual ObjectId instance.
Try removing all white space in the files (\n, spaces outside string quotes). It may work like miracle
I am working on a quick and dirty script to get Chromium's bookmarks and turn them into a pipe menu for Openbox. Chromium stores it's bookmarks in a file called Bookmarks that stores information in a dictionary form like this:
{
"checksum": "99999999999999999999",
"roots": {
"bookmark_bar": {
"children": [ {
"date_added": "9999999999999999999",
"id": "9",
"name": "Facebook",
"type": "url",
"url": "http://www.facebook.com/"
}, {
"date_added": "999999999999",
"id": "9",
"name": "Twitter",
"type": "url",
"url": "http://twitter.com/"
How would I open this dictionary in this file in Python and assign it to a variable. I know you open a file with open(), but I don't really know where to go from there. In the end, I want to be able to access the info in the dictionary from a variable like this bookmarks[bookmarks_bar][children][0][name] and have it return 'Facebook'
Do you know if this is a json file? If so, python provides a json library.
Json can be used as a data serialization/interchange format. It's nice because it's cross platform. Importing this like you ask seems fairly easy, an example from the docs:
>>> import json
>>> json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
[u'foo', {u'bar': [u'baz', None, 1.0, 2]}]
So in your case it would look something like:
import json
with open(file.txt) as f:
text = f.read()
bookmarks = json.loads(text)
print bookmarks[bookmarks_bar][children][0][name]
JSON is definitely the "right" way to do this, but for a quick-and-dirty script eval() might suffice:
with open('file.txt') as f:
bookmarks = eval(f.read())