We Keep Coding

How to convert values of a specific dictionary key to uppercase?

I have this simplified dict:
{
{
"birthPlace" : "london"
},
"hello": "hello",
"birthPlace" : "rome"
}
And I want to make the value of birthPlace uppercase: how? I tried
smallalphabetDict={}
for key, value in myjson.items():
smallalphabetDict[key.upper()] = value
It doesn't work

This changes all the values of a dict to uppercase, if the value is a string:
d = {......}
for k in d:
if type(d[k]) == str: d[k] = d[k].upper()

suggested method of encrypting via python

thanks for your time and effort but I think I probably misdeliver what I wanted, my fault.
Long story short, is there any way you can encrypt the certain string or the whole array?
{
"gender": "male",
"phone-number": "1234567890",
"job": "student",
"location": {
"county": "LA-county",
"town": "sunvalley",
"country": "USA",
"apartment-number": "13579abcdefg"
},
"item": {
"item-type": "cloth",
"item-size": "large",
"item-number": "xyz24680abc",
"item-material": "cotton"
},
"hairstyle": "long",
"alive": "true",
}
let's say that apartment-number: 13579abcdefg needs to be encrypted. Can I use fernet as below?
from cryptography.fernet import Fernet
key = Fernet.generate_key()
f = Fernet(key)
encrypt_value = f.encrypt(b"YourString")
f.decrypt(encrypt_value)
I heard some people mention about the base64... which method would you recommend when it comes for encrypting certain values?

Here is a solution that will account for nested JSON:
def mask_sensitive(payload, fields, n_front=3, n_back=3):
out = {}
for k, v in payload.items():
# if it's a dict, recurse
if isinstance(v, dict):
out[k] = mask_sensitive(v, fields, n_front, n_back)
# this assumes the field is a string, and not an iterable
# but you can always add logic to allow ints, lists, etc.
elif k in fields:
out[k] = v[:n_front] + "..." + v[-n_back:]
else:
out[k] = v
return out
There are some things you may want to write logic for, like, if the field is less than 3 characters long, how do you want to pad the sensitive information? But this gives you a good jumping off point. Example:
>>> import pprint
>>> pprint.pprint(mask_sensitive(x, ["phone-number", "apartment-number"]))
{'alive': 'true',
'gender': 'male',
'hairstyle': 'long',
'item': {'item-material': 'cotton',
'item-number': 'xyz24680abc',
'item-size': 'large',
'item-type': 'cloth'},
'job': 'student',
'location': {'apartment-number': '135...efg',
'country': 'USA',
'county': 'LA-county',
'town': 'sunvalley'},
'phone-number': '123...890'}

This code will run over the JSON and will parse the names from the keys in the JSON that in values_to_change list with the format that specified in the question
import json
with open('filename.json') as f:
data = json.load(f)
values_to_change = ["phone-number", "apartment-number", "item-number"]
for k, v in data.items():
if isinstance(v, str):
if k in values_to_change:
data[k] = "{}...{}".format(v[:3], v[-3:])
elif isinstance(v, dict):
for kv, vv in v.items():
if kv in values_to_change:
data[k][kv] = "{}...{}".format(vv[:3], vv[-3:])
with open('newfilename.json', 'w') as f:
json.dump(data, f, indent=2)
Output
{'gender': 'male',
'phone-number': '123...890',
'job': 'student',
'location': {'county': 'LA-county',
'town': 'sunvalley',
'country': 'USA',
'apartment-number': '135...efg'},
'item': {'item-type': 'cloth',
'item-size': 'large',
'item-number': 'xyz...abc',
'item-material': 'cotton'},
'hairstyle': 'long',
'alive': 'true'}

Just get the first three characters, three dots, then the last three characters.
def censor(string):
return string[:3] + "..." + string[-3:]
data["phone-number"] = censor(data["phone-number"])
data["apartment-number"] = censor(data["item-number"])
data["location"]["apartment-number"] = censor(data["location"]["apartment-number"])

Parsing dict is very important in this code. parse_dict method will parse all elements in dict. If any type of any element in data is dict, then we will use recursion, to call parse_dict again, to parse inner dict.
After this we check if key is one of ["phone-number", "apartment-number", "item-number"], if yes, then we change its value, to required format.
In this way, if in future, we want to change any other key, apart from ["phone-number", "apartment-number", "item-number"], then we simply need to append new key, in our list and this code will work.
import json
def parse_dict(data):
for key, value in data.items():
if type(value) == dict:
parse_dict(value)
if key in ["phone-number", "apartment-number", "item-number"]:
data[key] = value[:3] + "..." + value[-3:]
def main():
with open('sample.json') as f:
data = json.load(f)
parse_dict(data)
print("*****: ", data)
with open('newfilename.json', 'w') as f:
json.dump(data, f, indent=2)
main()

Using #Clinton Graham, plus some edits as you will get TypeError: unhashable type: 'slice' otherwise - something like this should work.
import json
import pandas as pd
with open('filename.json') as f:
data = json.load(f)
def reformat(data):
return data.values[:, 3] + "..." + data.valies[-3:]
data["phone-number"] = reformat(data["phone-number"])
data["item"]["item-number"] = reformat(data["item"]["item-number"])
data["location"]["apartment-number"] = reformat(data["location"]["apartment-number"])
print(reformat(data))

Flattening a dictionary of dictionaries that contain lists

I have a dictionary of dictionaries that looks like this:
data={'data': 'input',
'test':
{
'and':
{
'range': {'month': [{'start': 'Jan','end': 'July'}]},
'Student': {'Name': ['ABC'], 'Class': ['10']}
}
}
}
I need to flatten this dict into a dataframe.I tried to use json_normalize() to flatten the dictionary and the output I got looked like this:
My desired output is something like the one given below.
This can be done in R by using as.data.frame(unlist(data)) but I want to do the same flattening in Python. I am a novice in python so I dont have much idea about doing this.

I have made an attempt to normalize your json object by writing a recursive function as follows:
data={'data': 'input',
'test':
{
'and':
{
'range': {'month': [{'start': 'Jan','end': 'July'}]},
'Student': {'Name': ['ABC'], 'Class': ['10']}
}
}
}
sequence = ""
subDicts = []
def findAllSubDicts(data):
global subDicts
global sequence
for key, value in data.items():
sequence += key
#print(sequence)
if isinstance(value, str):
subDicts.append([sequence,value])
sequence = sequence[:sequence.rfind(".")+1]
#print(sequence)
elif isinstance(value, dict):
tempSequence = sequence[:sequence.rfind(".")+1]
sequence += "."
#print(sequence)
findAllSubDicts(value)
sequence = tempSequence
elif isinstance(value, list) and isinstance(value[0], dict):
sequence += "."
tempSequence = sequence[:sequence.rfind(".")+1]
#print(sequence)
findAllSubDicts(value[0])
sequence = tempSequence
elif isinstance(value, list) and len(value)==1:
tempSequence = sequence[:sequence.rfind(".")+1]
subDicts.append([sequence,value[0]])
sequence = tempSequence
return subDicts
outDict = findAllSubDicts(data)
for i in outDict:
print(i[0].ljust(40," "), end=" ")
print(i[1])
Printing the results will give you:
data input
test.and.range.month.start Jan
test.and.range.month.end July
test.and.Student.Name ABC
test.and.Student.Class 10
Notify me if you need any clarification or any modification in my code.

python json.dumps and json.loads before DynamoDB insertion

Is there any way to populate an empty string at any position (without knowing the json's structure) in a json received from a certain endpoint before inserting it into DynamoDB? As you all know it has issues with floats that you must transform them into Decimals, but can't seem to figure out an easy way to populate the empty string such as "full_name": "" with a value like "N/A".
I'm looking for something like json.loads(json.dumps(data), parse_float=Decimal), as for the parse_float thing but for empty strings. Something clean and easy to use. I've seen you can use a custom cls class for that but I don't quite get it how to do it properly especially without knowing the structure of the json which might vary.
JSON example:
{
"campaign_id": "9c1c6cd7-fd4d-480b-8c80-07091cdd4103",
"creation_date": 1530804132,
"objects": [
{
"id": 12345,
"full_name": ""
},
...
],
...
}

You can do this by defining an _object_hook_ to pass to json.loads.
From the docs:
object_hook is an optional function that will be called with the result of any object literal decoded (a dict). The return value of object_hook will be used instead of the dict.
Given this dict:
>>> pprint(d)
{'campaign_id': '9c1c6cd7-fd4d-480b-8c80-07091cdd4103',
'creation_date': 1530804132,
'float': 1.2345,
'objects': [{'full_name': '', 'id': 12345}],
'strs': ['', 'abc', {'a': ''}],
'top_str': ''}
This pair of functions will recurse over the result of json.loads and change instance of the empty string to 'N/A'.
def transform_dict(mapping=None):
if mapping is None:
mapping = {}
for k, v in mapping.items():
if v == '':
mapping[k] = 'N/A'
elif isinstance(v, dict):
mapping[k] = transform_dict(v)
elif isinstance(v, list):
mapping[k] = transform_list(v)
else:
# Make it obvious that we aren't changing other values
pass
return mapping
def transform_list(lst):
for i, x in enumerate(lst):
if x == '':
lst[i] = 'N/A'
elif isinstance(x, dict):
lst[i] = transform_dict(x)
elif isinstance(x, list):
lst[i] = transform_list(x)
else:
# Make it obvious that we aren't changing other values
pass
return lst
>>> res = json.loads(json.dumps(d), parse_float=decimal.Decimal, )
>>> pprint(res)
{'campaign_id': '9c1c6cd7-fd4d-480b-8c80-07091cdd4103',
'creation_date': 1530804132,
'float': Decimal('1.2345'),
'objects': [{'full_name': 'N/A', 'id': 12345}],
'strs': ['N/A', 'abc', {'a': 'N/A'}],
'top_str': 'N/A'}
Note that this approach depends on the input json being a json object ({...}).

Fastest way to convert a dict's keys & values from `unicode` to `str`?

I'm receiving a dict from one "layer" of code upon which some calculations/modifications are performed before passing it onto another "layer". The original dict's keys & "string" values are unicode, but the layer they're being passed onto only accepts str.
This is going to be called often, so I'd like to know what would be the fastest way to convert something like:
{ u'spam': u'eggs', u'foo': True, u'bar': { u'baz': 97 } }
...to:
{ 'spam': 'eggs', 'foo': True, 'bar': { 'baz': 97 } }
...bearing in mind the non-"string" values need to stay as their original type.
Any thoughts?

DATA = { u'spam': u'eggs', u'foo': frozenset([u'Gah!']), u'bar': { u'baz': 97 },
u'list': [u'list', (True, u'Maybe'), set([u'and', u'a', u'set', 1])]}
def convert(data):
if isinstance(data, basestring):
return str(data)
elif isinstance(data, collections.Mapping):
return dict(map(convert, data.iteritems()))
elif isinstance(data, collections.Iterable):
return type(data)(map(convert, data))
else:
return data
print DATA
print convert(DATA)
# Prints:
# {u'list': [u'list', (True, u'Maybe'), set([u'and', u'a', u'set', 1])], u'foo': frozenset([u'Gah!']), u'bar': {u'baz': 97}, u'spam': u'eggs'}
# {'bar': {'baz': 97}, 'foo': frozenset(['Gah!']), 'list': ['list', (True, 'Maybe'), set(['and', 'a', 'set', 1])], 'spam': 'eggs'}
Assumptions:
You've imported the collections module and can make use of the abstract base classes it provides
You're happy to convert using the default encoding (use data.encode('utf-8') rather than str(data) if you need an explicit encoding).
If you need to support other container types, hopefully it's obvious how to follow the pattern and add cases for them.

I know I'm late on this one:
def convert_keys_to_string(dictionary):
"""Recursively converts dictionary keys to strings."""
if not isinstance(dictionary, dict):
return dictionary
return dict((str(k), convert_keys_to_string(v))
for k, v in dictionary.items())

If you wanted to do this inline and didn't need recursive descent, this might work:
DATA = { u'spam': u'eggs', u'foo': True, u'bar': { u'baz': 97 } }
print DATA
# "{ u'spam': u'eggs', u'foo': True, u'bar': { u'baz': 97 } }"
STRING_DATA = dict([(str(k), v) for k, v in data.items()])
print STRING_DATA
# "{ 'spam': 'eggs', 'foo': True, 'bar': { u'baz': 97 } }"

for a non-nested dict (since the title does not mention that case, it might be interesting for other people)
{str(k): str(v) for k, v in my_dict.items()}

def to_str(key, value):
if isinstance(key, unicode):
key = str(key)
if isinstance(value, unicode):
value = str(value)
return key, value
pass key and value to it, and add recursion to your code to account for inner dictionary.

To make it all inline (non-recursive):
{str(k):(str(v) if isinstance(v, unicode) else v) for k,v in my_dict.items()}

>>> d = {u"a": u"b", u"c": u"d"}
>>> d
{u'a': u'b', u'c': u'd'}
>>> import json
>>> import yaml
>>> d = {u"a": u"b", u"c": u"d"}
>>> yaml.safe_load(json.dumps(d))
{'a': 'b', 'c': 'd'}

Just use print(*(dict.keys()))
The * can be used for unpacking containers e.g. lists. For more info on * check this SO answer.