I have a dictionary:
d = {name : "John", age: 10}.
And a log file setup as:
logging.basicConfig(level = logging.DEBUG, filename = "sample.log")
Is it possible to log this dictionary into the "sample.log" file? If yes, how can I do it?
Simple solution that works
The logging functions will convert the '%s' to string representation (and if the object happens to be a container, then repr() will be used for the contained objects)
import logging
logging.basicConfig(level=logging.DEBUG, filename='sample.log')
logging.debug('Sample dict log: %s', {'name' : "John", 'age': 10})
How it shows in the log file:
DEBUG:root:Sample dict log: {'age': 10, 'name': 'John'}
If you have custom objects (e.g. instances of your own classes), you should implement a sensible __str__ and/or __repr__ for it, and the above will work without any hacks.
More on this here What is the difference between __str__ and __repr__?
A Note on performance
Notice that
logging.debug('%s', some_dict) is not same as
logging.debug('{0}'.format(some_dict))
In the first one, the function is passed 2 arguments '%s' and the original some_dict.
In the second one, the function is passed one argument which is the already-converted some_dict to a string.
Because a function needs the arguments evaluated, the second example will always do the conversion, even if logging configs have turned off the logging of debug messages.
That's an unnecessary performance penalty.
In the the first example, the logging.debug function can decide to do the conversion, or not.
JSON is not very good for this
For objects that aren't JSON-like (object instances, datetime, etc), json.dumps() would require a custom serializer e.g. with datetime objects you get:
TypeError: datetime.datetime(...) is not JSON serializable
This goes for any type that isn't supported by json.dumps()
Whereas the logging module would figure out a good representation for the object
Simple you can use
dict_data = {"test":"data"}
logger.info("Loging dict ---> {0}".format(dict_data))
you can convert it to string:
string_dictionary = str(d)
and then log the string dictionary variable to the file
using JSON
import json
d = {"name":"John","age":10}
json_string = json.dumps(d)
and if you want to convert the string back:
d = json.loads(json_string)
The problem with str(dictionary) and json.dumps(dictionary) is that the output can human unfriendly, especially if the dictionary is big and has nested structures.
If that's the case, you can Python's built-in pprint to pretty format the dictionary before logging:
import pprint
my_complex_dict = pprint.pformat(my_complex_dict)
logger.info(f"My complex dict:\n{my_complex_dict}")
I came to this question when I wanted to log JSON lines for Cloudwatch.
I ended up using python-json-logger.
Install it: pip install python-json-logger
Use pythonjsonlogger.jsonlogger.JsonFormatter as the formatter class.
Related
When I run the code:
import requests
import json
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140")
json_data = json.loads(catFact.text)
return json_data
print(get_fact())
The output is like
{'fact': "Cats are the world's most popular pets, outnumbering dogs by as many as three to one", 'length': 84}
However I just want the fact.
How do I get rid of the 'fact:' at the front and 'length:' at the back?
What you want is to access the key in the python dict you made with the json.loads call. We actually don't need the json library as requests can read and deserialize JSON itself.
This code also checks if the response was OK and fails with informative error message. It follows PEP 20 – The Zen of Python.
import requests
def get_fact():
# Get the facts dictionary in a JSON serialized form.
cat_fact_response = requests.get("https://catfact.ninja/fact?max_length=140")
# Let the response raise the exception if something bad happened to the cat facts server connection.
cat_fact_response.raise_for_status()
# Deserialize the json (make a Python dict from the text we got). requests can do that on it's own:
cat_fact_dict = cat_fact_response.json()
# Access the fact from the json from the dictionary
return cat_fact_dict['fact']
print(get_fact())
When called you get following output as wanted:
# python3 script.py
The cat's tail is used to maintain balance.
Short answer:
you need to use either get_fact()['fact'] or get_fact().get('fact'). The former will throw an exception if fact doesn't exist whereas the latter will return None.
Why:
In your code sample you fetch some json data, and then print out the entire bit of json. When you parse json, the output is a key/value map called a dictionary (or map or object in other languages). The dictionary in this case contains two keys: fact and length. If you only one want of the values, then you need to tell python that you want only a single value -- fact in this case.
Remember though: this wouldn't apply to every json object you read. Not every one is going to have a fact key.
What you are returning in get_fact is a complete JSON object which you are then printing.
To get just its property fact (without the length) use a reference to that key or property like:
return json_data["fact"]
Below is also a link to a tutorial on using JSON in Python:
w3schools: Python JSON
To extract fact field from the response, use:
import requests
import json
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140")
json_data = json.loads(catFact.text)
return json_data['fact'] # <- HERE
print(get_fact())
Output:
Cats have "nine lives" thanks to a flexible spine and powerful leg and back muscles
Note: you don't need json module here, use json() method of Response instance returned by requests:
import requests
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140").json()
return catFact['fact']
print(get_fact())
How to access id or nickname value with python3:
response._content = b'{"id":44564,"nickname":'Demo'}
It looks like you're trying to read in a Json string and convert it to a dict, e.g.:
import json
# response._content = b'{"id":44564,"nickname":"Demo"}'
data = json.loads(response._content.decode('utf-8'))
# data = {'id': 44564, 'nickname': 'Demo'}
This is a byte string that includes JSON as stated above. Another way to look at it is that is a dict definition (i.e python code). You can use eval for that:
foo = eval( b'{"id":44564,"nickname":"Demo"}')
foo['nickname']
Probably this is not the preferred or secure way to do it because eval is considered dangerous
https://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
I have created a DataStore, and the type of fields in the data store has been saved in json format.
Something like
{"SchemaName": "Caztor", "FieldType": {"cain": "<type 'float'>"}}
Now when i load the json using json.load , the type of the field cain is being loaded as Unicode. Is there any pre defined methods available in python which can be used to typecast this String to Type object. Ofcourse I can make a function that would handle and solve the problem. But if there is already a solution to this in the python library then it would save a bit of overhead and might be more efficient in handling the problem than I might do.
You can use the module pickle to solve the typecast. this module can serialize the instances into a string
The Documentation is present on https://docs.python.org/2/library/pickle.html
Example
import pickle
schema = {"SchemaName": "Caztor", "FieldType": {"cain": float}}
result = pickle.dumps(schema) # Will result a string
To deserialize use:
result2 = pickle.loads(result) # Will result a dictionary with typecast
There is no solution. JSON is a universal format across all languages and it doesn't make sense for it to be able to store Python classes. You need to define a fixed way to go back and forth between strings and types, which includes determining which types are acceptable. Here's an example:
class Example(object):
pass
json_types = [int, float, str, Example]
string_to_type_dict = {t.__name__: t for t in json_types}
assert len(string_to_type_dict) == len(json_types)
def string_to_type(s):
return string_to_type_dict[s]
def type_to_string(t):
return t.__name__
I want to dump this json to a file:
json.dumps(data)
This is the data:
{
"list":[
"one": { "id": "12","desc":"its 12","name":"pop"},
"two": {"id": "13","desc":"its 13","name":"kindle"}
]
}
I want id to be the first property after I dump it to file, but it is not. How can I fix this?
My guess is that it's because you're using a dictionary (hash-map). It's unsortable.
What you could do is:
from collections import OrderedDict
data = OrderedDict()
data['list'] = OrderedDict()
data['list']['one'] = OrderedDict()
data['list']['one']['id'] = '12'
data['list']['one']['idesc'] = ...
data['list']['two'] = ...
This makes it sorted by order of input.
It's "impossible" to know the output of a dict/hashmap because the nature (and speed) of a traditional dictionary makes the sort/access order vary depending on usage, items in the dictionary and a lot of other factors.
So you need to either pass your dictionary to a sort() function prior to sending it to json or use a slower version of the dictionary called OrderedDict (see above).
Many thanks goes out to #MarcoNawijn for checking the source of JSON that does not honor the sort structure of the dictionary, which means you'll have to build the JSON string yourself.
If the parser on the other end of your JSON string honors the order (which i doubt), you could pass this to a function that builds a regular text-string representation of your OrderedDict and formatting the string as per JSON standards. This will however take up more time than I have at this moment since i'm not 100% certain of the RFC for JSON strings.
You shouldnt worry about the order in which json is saved. The order will be changed when dumping. Better look at these too. JSON order mixed up
and
Is the order of elements in a JSON list maintained?
Is there a best-practice for outputting Booleans in Python? I'm generating some JSON (via Django templates), and by default all the Boolean values are output with leading character in uppercase, contrary to the JSON standard (ie, "True" as opposed to "true").
Currently, I format each Boolean string using str.lower(), but is there a better way?
Well, then serialise to JSON using json, not some custom thingy.
import json
print json.dumps({'foo': True}) # => {"foo": true}
The way to do this is to not use templates. Use the json module, as so:
import json
def my_view(request):
# ...
json_d = dict(...)
return json.dumps(json_d)
My preferred way is to write a decorator, and return a dict.
def json_view(f):
def wrapped_f(*args, **kwargs):
return json.dumps(f(*args, **kwargs))
wrapped_f.original = f # for unit testing
return wrapped_f
#json_view
my_view(request):
# ...
return dict(...)
Use the json module:
>>> import json
>>> json.dump(dict(value=True), sys.stdout)
{"value": true}
The better way would be to avoid generating JSON by hand, or via Django templates, and instead use a proper JSON library. In Python 2.6+ this is as simple as import json. In older Pythons, you'll need to pip install simplejson and import simplejson as json.
It can be tough to generate proper JSON on your own—your experience with manually serializing bool values is just the beginning. For another example, what about properly escaping strings with nested quotes?