Hi I would like to essentially use yaml data inside json
for eg.
json file:
{
"Name": "foo",
"Birthdate": "1/1/1991",
"Address": "FOO_ADDRESS",
"Note": "Please deliver package to foo at FOO_ADDRESS using COURIER service"
}
yaml file:
---
FOO_ADDRESS: "foo lane, foo state"
COURIER: "foodex"
Could someone please guide me on the most efficient way to do this? In this particular example I don't really need to use a separate yaml file (I understand that). But in my specific case I might have to do that.
Edit: sorry I didnt paste the desired output file
Should look something like this:
{
"Name": "foo",
"Birthdate": "1/1/1991",
"Address": "foo lane, foo state",
"Note": "Please deliver package to foo at foo lane, foo state using foodex service"
}
To be safe, first load the JSON and then do the replacements in the loaded strings. If you do the replacements in the JSON source, you might end up with invalid JSON output (when the replacement strings contain " or other characters that have to be escaped in JSON).
import yaml, json
def doReplacements(jsonValue, replacements):
if isinstance(jsonValue, dict):
processed = {doReplacements(key, replacements): \
doReplacements(value, replacements) for key, value in \
jsonValue.iteritems()}
# Python 3: use jsonValue.items() instead
elif isinstance(jsonValue, list):
processed = [doReplacements(item, replacements) for item in jsonValue]
elif isinstance(jsonValue, basestring):
# Python 3: use isinstance(jsonValue, str) instead
processed = jsonValue
for key, value in replacements.iteritems():
# Python 3: use replacements.items() instead
processed = processed.replace(key, value)
else:
# nothing to replace for Boolean, None or numbers
processed = jsonValue
return processed
input = json.loads("""{
"Name": "foo",
"Birthdate": "1/1/1991",
"Address": "FOO_ADDRESS",
"Note": "Please deliver package to foo at FOO_ADDRESS using COURIER service"
}
""")
replacements = yaml.safe_load("""---
FOO_ADDRESS: "foo lane, foo state"
COURIER: "foodex"
""")
print json.dumps(doReplacements(input, replacements), indent=2)
# Python 3: `(...)` around print argument
Use json.load and json.dump to read/write files instead of strings. Note that loading and writing the JSON data may change the order of the items in the object (which you should not depend on anyway).
Related
I currently have JSON in the below format.
Some of the Key values are NOT properly formatted as they are missing double quotes (")
How do I fix these key values to have double-quotes on them?
{
Name: "test",
Address: "xyz",
"Age": 40,
"Info": "test"
}
Required:
{
"Name": "test",
"Address": "xyz",
"Age": 40,
"Info": "test"
}
Using the below post, I was able to find such key values in the above INVALID JSON.
However, I could NOT find an efficient way to replace these found values with double-quotes.
s = "Example: String"
out = re.findall(r'\w+:', s)
How to Escape Double Quote inside JSON
Using Regex:
import re
data = """{ Name: "test", Address: "xyz"}"""
print( re.sub("(\w+):", r'"\1":', data) )
Output:
{ "Name": "test", "Address": "xyz"}
You can use PyYaml. Since JSON is a subset of Yaml, pyyaml may overcome the lack of quotes.
Example
import yaml
dirty_json = """
{
key: "value",
"key2": "value"
}
"""
yaml.load(dirty_json, yaml.SafeLoader)
I had few more issues that I faced in my JSON.
Thought of sharing the final solution that worked for me.
jsonStr = re.sub("((?=\D)\w+):", r'"\1":', jsonStr)
jsonStr = re.sub(": ((?=\D)\w+)", r':"\1"', jsonStr)
First Line will fix this double-quotes issue for the Key. i.e.
Name: "test"
Second Line will fix double-quotes issue for the value. i.e. "Info": test
Also, above will exclude double-quoting within date timestamp which have : (colon) in them.
You can use online formatter. I know most of them are throwing error for not having double quotes but below one seems handling it nicely!
JSON Formatter
The regex approach can be brittle. I suggest you find a library that can parse the JSON text that is missing quotes.
For example, in Kotlin 1.4, the standard way to parse a JSON string is using Json.decodeFromString. However, you can use Json { isLenient = true }.decodeFromString to relax the requirements for quotes. Here is a complete example in JUnit.
import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromString
import kotlinx.serialization.json.Json
import org.junit.jupiter.api.Assertions
import org.junit.jupiter.api.Test
#Serializable
data class Widget(val x: Int, val y: String)
class JsonTest {
#Test
fun `Parsing Json`() {
val w: Widget = Json.decodeFromString("""{"x":123, "y":"abc"}""")
Assertions.assertEquals(123, w.x)
Assertions.assertEquals("abc", w.y)
}
#Test
fun `Parsing Json missing quotes`() {
// Json.decodeFromString("{x:123, y:abc}") failed to decode due to missing quotes
val w: Widget = Json { isLenient = true }.decodeFromString("{x:123, y:abc}")
Assertions.assertEquals(123, w.x)
Assertions.assertEquals("abc", w.y)
}
}
I want to re-format a JSON file so that certain objects (dictionaries) with some specific keys are on one-line.
For example, any object with key name should appear in one line:
{
"this": "that",
"parameters": [
{ "name": "param1", "type": "string" },
{ "name": "param2" },
{ "name": "param3", "default": "#someValue" }
]
}
The JSON file is generated, and contains programming language data. One-line certain fields makes it much easier to visually inspect/review.
I tried to override python json.JSONEncoder to turn matching dict into a string before writing, only to realize quotes " within the string are escaped again in the result JSON file, defeating my purpose.
I also looked at jq but couldn't figure out a way to do it. I found similar questions and solutions based on line length, but my requirements are simpler, and I don't want other shorter lines to be changed. Only certain objects or fields.
This code recursively replaces all the appropriate dicts in the data with unique strings (UUIDs) and records those replacements, then in the indented JSON string the unique strings are replaced with the desired original single line JSON.
replace returns a pair of:
A modified version of the input argument data
A list of pairs of JSON strings where for each pair the first value should be replaced with the second value in the final pretty printed JSON.
import json
import uuid
def replace(o):
if isinstance(o, dict):
if "name" in o:
replacement = uuid.uuid4().hex
return replacement, [(f'"{replacement}"', json.dumps(o))]
replacements = []
result = {}
for key, value in o.items():
new_value, value_replacements = replace(value)
result[key] = new_value
replacements.extend(value_replacements)
return result, replacements
elif isinstance(o, list):
replacements = []
result = []
for value in o:
new_value, value_replacements = replace(value)
result.append(new_value)
replacements.extend(value_replacements)
return result, replacements
else:
return o, []
def pretty(data):
data, replacements = replace(data)
result = json.dumps(data, indent=4)
for old, new in replacements:
result = result.replace(old, new)
return result
print(pretty({
"this": "that",
"parameters": [
{"name": "param1", "type": "string"},
{"name": "param2"},
{"name": "param3", "default": "#someValue"}
]
}))
This is the structure of my json file
},
"client1": {
"description": "blabla",
"contact name": "",
"contact email": "",
"third party organisation": "",
"third party contact name": "",
"third party contact email": "",
"ranges": [
"1.1.1.1",
"2.2.2.2",
"3.3.3.3"
]
},
"client2": {
"description": "blabla",
"contact name": "",
"contact email": "",
"third party organisation": "",
"third party contact name": "",
"third party contact email": "",
"ranges": [
"4.4.4.4",
"2.2.2.2"
]
},
I've seen ways to export specific parts of this json file but not everything. Basically all I want to do is search through the file using user input.
All I'm struggling with is how I actually use the user input to search and print everything under either client1 or client2 based on the input? I am sure this is only 1 or 2 lines of code but cannot figure it out. New to python. This is my code
data = json.load(open('clients.json'))
def client():
searchq = input('Client to export: '.capitalize())
search = ('""'+searchq+'"')
a = open('Log.json', 'a+')
a.write('Client: \n')
client()
This should get you going:
# Safely open the file and load the data into a dictionary
with open('clients.json', 'rt') as dfile:
data = json.load(dfile)
# Ask for the name of the client
query = input('Client to export: ')
# Show the corresponding entry if it exists,
# otherwise show a message
print(data.get(query, 'Not found'))
I'm going to preface this by saying this is 100% a drive-by answering, but one thing you could do is have your user use a . (dot) delimited format for specifying the 'path' to the key in the dictionary/json structure, then implementing a recursive function to seek out the value under that path like so:
def get(query='', default=None, fragment=None):
"""
Recursive function which returns the value of the terminal
key of the query string supplied, or if no query
is supplied returns the whole fragment (dict).
Query string should take the form: 'each.item.is.a.key', allowing
the user to retrieve the value of a key nested within the fragment to
an arbitrary depth.
:param query: String representation of the path to the key for which
the value should be retrieved
:param default: If default is specified, returns instead of None if query is invalid
:param fragment: The dictionary to inspect
:return: value of the specified key or fragment if no query is supplied
"""
if not query:
return fragment
query = query.split('.')
try:
key = query.pop(0)
try:
if isinstance(fragment, dict) and fragment:
key = int(key) if isinstance(fragment.keys()[0], int) else key
else:
key = int(key)
except ValueError:
pass
fragment = fragment[key]
query = '.'.join(query)
except (IndexError, KeyError) as e:
return default if default is not None else None
if not fragment:
return fragment
return get(query=query, default=default, fragment=fragment)
There are going to be a million people who come by here with better suggestions than this and there are doubtless many improvements to be made to this function as well, but since I had it lying around I thought I'd put it here, at least as a starting point for you.
Note:
Fragment should probably be made a positional argument or something. IDK. Its not because I had to rip some application specific context out (it used to have a sensible default state) and I didn't want to start re-writing stuff, so I leave that up to you.
You can do some cool stuff with this function, given some data:
d = {
'woofage': 1,
'woofalot': 2,
'wooftastic': ('woof1', 'woof2', 'woof3'),
'woofgeddon': {
'woofvengers': 'infinity woof'
}
}
Try these:
get(fragment=d, query='woofage')
get(fragment=d, query='wooftastic')
get(fragment=d, query='wooftastic.0')
get(fragment=d, query='woofgeddon.woofvengers')
get(fragment=d, query='woofalistic', default='ultraWOOF')
Bon voyage!
Pass the json format into Dict then look into the topic you want and Read or write it
import json
r = {'is_claimed': True, 'rating': 3.5}
r = json.dumps(r) # Here you have json format {"is_claimed": true, "rating": 3.5}
Json to Dict:
loaded_r = json.loads(r) # {'is_claimed': True, 'rating': 3.5}
print (r)#Print json format
print (loaded_r) #Print dict
Read the Topic
Data=loaded_r['is_claimed'] #Print Topic desired
print(Data) #True
Overwrite the topic
loaded_r['is_claimed']=False
And also this would do the same
print(loaded_r['client1']['description'])
I have a nested dictionary with many items in a json file:
{
"Create Code For Animals": {
"mammals": {
"dog": {
"color": "brown",
"name": "John",
"legs": "four",
"tail": "yes"
},
"cat": {
"color": "blue",
"name": "Johnny",
"legs": "four",
"tail": "yes"
},
"donkey": {
"color": "grey",
"name": "Mickey",
"legs": "four",
"tail": "yes"
}
I want to replace the name in each one of the animals, then save it back to the file, AND keep the indent as it was (as shown).
I'm using the following 2 methods for loading and dumping the original and updated dictionary.
All is working well (for changing the value and saving it back to the file) except the indent (format) of the lines is ruined after saving the file and the file is saved as one long line (with '\n' shown after the updated value).
I've tried using 'pickle' (as seen in one of the posts here), but this didn't work, made a mess of all the data in the file.
def loadJson(self, jsonFilename):
with open(FILE_PATH + '\\' + jsonFilename, 'r') as f:
return json.load(f)
def writeJson(self, jsonFilename, jsonDict):
with open(FILE_PATH + '\\' + jsonFilename, 'w') as f:
return json.dump(jsonDict, f)
Any help will do.
json.dumps and dump have a parameter called indent
If ``indent`` is a non-negative integer, then JSON array elements and
object members will be pretty-printed with that indent level. An indent
level of 0 will only insert newlines. ``None`` is the most compact
representation. Since the default item separator is ``', '``, the
output might include trailing whitespace when ``indent`` is specified.
You can use ``separators=(',', ': ')`` to avoid this
Something like this would do:
json.dump(jsonDict,f,indent=4)
I've got a json file that I've pulled from a web service and am trying to parse it. I see that this question has been asked a whole bunch, and I've read whatever I could find, but the json data in each example appears to be very simplistic in nature. Likewise, the json example data in the python docs is very simple and does not reflect what I'm trying to work with. Here is what the json looks like:
{"RecordResponse": {
"Id": blah
"Status": {
"state": "complete",
"datetime": "2016-01-01 01:00"
},
"Results": {
"resultNumber": "500",
"Summary": [
{
"Type": "blah",
"Size": "10000000000",
"OtherStuff": {
"valueOne": "first",
"valueTwo": "second"
},
"fieldIWant": "value i want is here"
The code block in question is:
jsonFile = r'C:\Temp\results.json'
with open(jsonFile, 'w') as dataFile:
json_obj = json.load(dataFile)
for i in json_obj["Summary"]:
print(i["fieldIWant"])
Not only am I not getting into the field I want, but I'm also getting a key error on trying to suss out "Summary".
I don't know how the indices work within the array; once I even get into the "Summary" field, do I have to issue an index manually to return the value from the field I need?
The example you posted is not valid JSON (no commas after object fields), so it's hard to dig in much. If it's straight from the web service, something's messed up. If you did fix it with proper commas, the "Summary" key is within the "Results" object, so you'd need to change your loop to
with open(jsonFile, 'w') as dataFile:
json_obj = json.load(dataFile)
for i in json_obj["Results"]["Summary"]:
print(i["fieldIWant"])
If you don't know the structure at all, you could look through the resulting object recursively:
def findfieldsiwant(obj, keyname="Summary", fieldname="fieldIWant"):
try:
for key,val in obj.items():
if key == keyname:
return [ d[fieldname] for d in val ]
else:
sub = findfieldsiwant(val)
if sub:
return sub
except AttributeError: #obj is not a dict
pass
#keyname not found
return None