We Keep Coding

suggested method of encrypting via python

thanks for your time and effort but I think I probably misdeliver what I wanted, my fault.
Long story short, is there any way you can encrypt the certain string or the whole array?
{
"gender": "male",
"phone-number": "1234567890",
"job": "student",
"location": {
"county": "LA-county",
"town": "sunvalley",
"country": "USA",
"apartment-number": "13579abcdefg"
},
"item": {
"item-type": "cloth",
"item-size": "large",
"item-number": "xyz24680abc",
"item-material": "cotton"
},
"hairstyle": "long",
"alive": "true",
}
let's say that apartment-number: 13579abcdefg needs to be encrypted. Can I use fernet as below?
from cryptography.fernet import Fernet
key = Fernet.generate_key()
f = Fernet(key)
encrypt_value = f.encrypt(b"YourString")
f.decrypt(encrypt_value)
I heard some people mention about the base64... which method would you recommend when it comes for encrypting certain values?

Here is a solution that will account for nested JSON:
def mask_sensitive(payload, fields, n_front=3, n_back=3):
out = {}
for k, v in payload.items():
# if it's a dict, recurse
if isinstance(v, dict):
out[k] = mask_sensitive(v, fields, n_front, n_back)
# this assumes the field is a string, and not an iterable
# but you can always add logic to allow ints, lists, etc.
elif k in fields:
out[k] = v[:n_front] + "..." + v[-n_back:]
else:
out[k] = v
return out
There are some things you may want to write logic for, like, if the field is less than 3 characters long, how do you want to pad the sensitive information? But this gives you a good jumping off point. Example:
>>> import pprint
>>> pprint.pprint(mask_sensitive(x, ["phone-number", "apartment-number"]))
{'alive': 'true',
'gender': 'male',
'hairstyle': 'long',
'item': {'item-material': 'cotton',
'item-number': 'xyz24680abc',
'item-size': 'large',
'item-type': 'cloth'},
'job': 'student',
'location': {'apartment-number': '135...efg',
'country': 'USA',
'county': 'LA-county',
'town': 'sunvalley'},
'phone-number': '123...890'}

This code will run over the JSON and will parse the names from the keys in the JSON that in values_to_change list with the format that specified in the question
import json
with open('filename.json') as f:
data = json.load(f)
values_to_change = ["phone-number", "apartment-number", "item-number"]
for k, v in data.items():
if isinstance(v, str):
if k in values_to_change:
data[k] = "{}...{}".format(v[:3], v[-3:])
elif isinstance(v, dict):
for kv, vv in v.items():
if kv in values_to_change:
data[k][kv] = "{}...{}".format(vv[:3], vv[-3:])
with open('newfilename.json', 'w') as f:
json.dump(data, f, indent=2)
Output
{'gender': 'male',
'phone-number': '123...890',
'job': 'student',
'location': {'county': 'LA-county',
'town': 'sunvalley',
'country': 'USA',
'apartment-number': '135...efg'},
'item': {'item-type': 'cloth',
'item-size': 'large',
'item-number': 'xyz...abc',
'item-material': 'cotton'},
'hairstyle': 'long',
'alive': 'true'}

Just get the first three characters, three dots, then the last three characters.
def censor(string):
return string[:3] + "..." + string[-3:]
data["phone-number"] = censor(data["phone-number"])
data["apartment-number"] = censor(data["item-number"])
data["location"]["apartment-number"] = censor(data["location"]["apartment-number"])

Parsing dict is very important in this code. parse_dict method will parse all elements in dict. If any type of any element in data is dict, then we will use recursion, to call parse_dict again, to parse inner dict.
After this we check if key is one of ["phone-number", "apartment-number", "item-number"], if yes, then we change its value, to required format.
In this way, if in future, we want to change any other key, apart from ["phone-number", "apartment-number", "item-number"], then we simply need to append new key, in our list and this code will work.
import json
def parse_dict(data):
for key, value in data.items():
if type(value) == dict:
parse_dict(value)
if key in ["phone-number", "apartment-number", "item-number"]:
data[key] = value[:3] + "..." + value[-3:]
def main():
with open('sample.json') as f:
data = json.load(f)
parse_dict(data)
print("*****: ", data)
with open('newfilename.json', 'w') as f:
json.dump(data, f, indent=2)
main()

Using #Clinton Graham, plus some edits as you will get TypeError: unhashable type: 'slice' otherwise - something like this should work.
import json
import pandas as pd
with open('filename.json') as f:
data = json.load(f)
def reformat(data):
return data.values[:, 3] + "..." + data.valies[-3:]
data["phone-number"] = reformat(data["phone-number"])
data["item"]["item-number"] = reformat(data["item"]["item-number"])
data["location"]["apartment-number"] = reformat(data["location"]["apartment-number"])
print(reformat(data))

Simplify dictionary in python from a Firestore Trigger Event

I'm reading data from an Update Cloud Firestore Trigger. The event is a dictionary that contains the data whithin the key ['value']['fields']. However, each of the keys contains s nested dictionary containing a key like 'integerValue', 'booleanValue' or 'stringValue', where the value of integerValue is actually a string. Is there a method to remove the 'type pointers'?
How can I convert this:
{
'fields': {
'count': {
'integerValue': '0'
},
'verified': {
'booleanValue': False
},
'user': {
'stringValue': 'Matt'
}
}
}
To this:
{
'count': 0,
'verified': False,
'user': 'Matt',
}

Recently i encountered similar problem.
We could recursively traverse the map to extract and simplify the event trigger data.
Here's python implementation, extended from previous answers.
class FirestoreTriggerConverter(object):
def __init__(self, client=None) -> None:
self.client = client if client else firestore.client()
self._action_dict = {
'geoPointValue': (lambda x: dict(x)),
'stringValue': (lambda x: str(x)),
'arrayValue': (lambda x: [self._parse_value(value_dict) for value_dict in x.get("values", [])]),
'booleanValue': (lambda x: bool(x)),
'nullValue': (lambda x: None),
'timestampValue': (lambda x: self._parse_timestamp(x)),
'referenceValue': (lambda x: self._parse_doc_ref(x)),
'mapValue': (lambda x: {key: self._parse_value(value) for key, value in x["fields"].items()}),
'integerValue': (lambda x: int(x)),
'doubleValue': (lambda x: float(x)),
}
def convert(self, data_dict: dict) -> dict:
result_dict = {}
for key, value_dict in data_dict.items():
result_dict[key] = self._parse_value(value_dict)
return result_dict
def _parse_value(self, value_dict: dict) -> Any:
data_type, value = value_dict.popitem()
return self._action_dict[data_type](value)
def _parse_timestamp(self, timestamp: str):
try:
return datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%fZ')
except ValueError as e:
return datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%SZ')
def _parse_doc_ref(self, doc_ref: str) -> DocumentReference:
path_parts = doc_ref.split('/documents/')[1].split('/')
collection_path = path_parts[0]
document_path = '/'.join(path_parts[1:])
doc_ref = self.client.collection(collection_path).document(document_path)
return doc_ref
Use this as follows
converter = FirestoreTriggerConverter(client)
simplified_data_dict = converter.convert(event_data_dict["event"]["value"]["fields"])

You can create a mapping of the known types and convert the values that way:
types = {
'integerValue': int,
'booleanValue': bool,
'stringValue': str,
}
You can replace a nested dictionary like the one you have through the magic of dict.popitem:
replacement = {}
for key, meta in event['value']['fields'].items():
typ, value = meta.popitem()
replacement[key] = types[typ](value)
event['value'] = replacement
You can reduce it to a one liner with a dictionary comprehension:
event['value'] = {k: types[t](v) for k t, v in (k, *d.popitem()) for k, d in event['value']['fields'].items())}

Use keys() in dictionary
origin_dict={
'fields': {
'count': {
'integerValue': '0'
},
'verified': {
'booleanValue': False
},
'user': {
'stringValue': 'Matt'
}
}
}
# remove first layer
b = origin_dict['fields']
new_dict = dict()
for i in b.keys():
# i will be second layer of dictionary
for j in b[i].keys():
# j will be third layer of dictionary
new_dict[i] = b[i][j]
print (new_dict)

There is no explicit method to do so. One you can do is iterate through existing dictionary picking up items you need in the new dictionary:
d = {
'fields': {
'count': {
'integerValue': '0'
},
'verified': {
'booleanValue': False
},
'user': {
'stringValue': 'Matt'
}
}
}
required = ['count', 'verified', 'user']
d1 = {}
for x in d.values():
for y in required:
if 'integerValue' in x[y].keys():
d1[y] = int(list(x[y].values())[0])
else:
d1[y] = list(x[y].values())[0]
print(d1)
# {'count': 0, 'verified': False, 'user': 'Matt'}

How to replace nested python dictionary value from a key as a string format with separated by dots?

a = {
'user': {
'username': 'mic_jack',
'name': {
'first': 'Micheal',
'last': 'Jackson'
},
'email': 'micheal#domain.com',
#...
#... Infinite level of another nested dict
}
}
str_key_1 = 'user.username=john'
str_key_2 = 'user.name.last=henry'
#...
#str_key_n = 'user.level2.level3...leveln=XXX'
Let's consider this 'str_key' string, goes with infinite number of dots/levels.
Expected Output:
a = {
'user': {
'username': 'john', # username, should be replace
'name': {
'first': 'Micheal',
'last': 'henry' # lastname, should be replace
},
'email': 'micheal#domain.com',
...
... # Infinite level of another nested dict
}
}
I'm expecting the answers for applying 'n' Level of nested key string, rather than simply replacing by a['user']['username'] = 'John' statically. Answers must be work for any number of 'dotted' string values.
Thanks in advance!

There are three steps:
Separate the key-value pair string into a fully-qualified key and
value.
Split the key into path components.
Traverse the dictionary to find the relevant value to update.
Here's an example of what the code might look like:
# Split by the delimiter, making sure to split once only
# to prevent splitting when the delimiter appears in the value
key, value = str_key_n.split("=", 1)
# Break the dot-joined key into parts that form a path
key_parts = key.split(".")
# The last part is required to update the dictionary
last_part = key_parts.pop()
# Traverse the dictionary using the parts
current = a
while key_parts:
current = current[key_parts.pop(0)]
# Update the value
current[last_part] = value

I'd go with a recursive function to accomplish this, assuming your key value strings are all valid:
def assign_value(sample_dict, str_keys, value):
access_key = str_keys[0]
if len(str_keys) == 1:
sample_dict[access_key] = value
else:
sample_dict[access_key] = assign_value(sample_dict[access_key], str_keys[1:], value)
return sample_dict
The idea is to traverse your dict until you hit the lowest key and then we assign our new value to that last key;
if __name__ == "__main__":
sample_dict = {
'user': {
'username': 'mic_jack',
'name': {
'first': 'Micheal',
'last': 'Jackson'
},
'email': 'micheal#domain.com'
}
}
str_key_1 = 'user.username=john'
str_keys_1, value_1 = str_key_1.split('=')
sample_dict = assign_value(sample_dict, str_keys_1.split('.'), value_1)
print("result: {} ".format(sample_dict))
str_key_2 = 'user.name.last=henry'
str_keys_2, value_2 = str_key_2.split('=')
sample_dict = assign_value(sample_dict, str_keys_2.split('.'), value_2)
print("result: {}".format(sample_dict))
To use the assign_value you would need to split your original key to the keys and value as seen above;

If you're okay with using exec() and modify your str_key(s), you could do something like:
def get_keys_value(string):
keys, value = string.split("=")
return keys, value
def get_exec_string(dict_name, keys):
exec_string = dict_name
for key in keys.split("."):
exec_string = exec_string + "[" + key + "]"
exec_string = exec_string + "=" + "value"
return exec_string
str_key_1 = "'user'.'username'=john"
str_key_2 = "'user'.'name'.'last'=henry"
str_key_list = [str_key_1, str_key_2]
for str_key in str_key_list:
keys, value = get_keys_value(str_key) # split into key-string and value
exec_string = get_exec_string("a", keys) # extract keys from key-string
exec(exec_string)
print(a)
# prints {'user': {'email': 'micheal#domain.com', 'name': {'last': 'henry', 'first': 'Micheal'}, 'username': 'john'}}

str_key_1 = 'user.username=john'
str_key_2 = 'user.name.last=henry'
a = {
'user': {
'username': 'mic_jack',
'name': {
'first': 'Micheal',
'last': 'Jackson'
},
'email': 'micheal#domain.com',
#...
#... Infinite level of another nested dict
}
}
def MutateDict(key):
strkey, strval = key.split('=')[0], key.split('=')[1]
strkeys = strkey.split('.')
print("strkeys = " ,strkeys)
target = a
k = ""
for k in strkeys:
print(target.keys())
if k in target.keys():
prevTarget = target
target = target[k]
else:
print ("Invalid key specified")
return
prevTarget[k] = strval
MutateDict(str_key_1)
print(a)
MutateDict(str_key_2)
print(a)

Formatting a string in required format in Python

I have a data in format:
id1 id2 value
Something like
1 234 0.2
1 235 0.1
and so on.
I want to convert it in json format:
{
"nodes": [ {"name":"1"}, #first element
{"name":"234"}, #second element
{"name":"235"} #third element
] ,
"links":[{"source":1,"target":2,"value":0.2},
{"source":1,"target":3,"value":0.1}
]
}
So, from the original data to above format.. the nodes contain all the set of (distinct) names present in the original data and the links are basically the line number of source and target in the values list returned by nodes.
For example:
1 234 0.2
1 is in the first element in the list of values holded by the key "nodes"
234 is the second element in the list of values holded by the key "nodes"
Hence the link dictionary is {"source":1,"target":2,"value":0.2}
How do i do this efficiently in python.. I am sure there should be better way than what I am doing which is so messy :(
Here is what I am doing
from collections import defaultdict
def open_file(filename,output=None):
f = open(filename,"r")
offset = 3429
data_dict = {}
node_list = []
node_dict = {}
link_list = []
num_lines = 0
line_ids = []
for line in f:
line = line.strip()
tokens = line.split()
mod_wid = int(tokens[1]) + offset
if not node_dict.has_key(tokens[0]):
d = {"name": tokens[0],"group":1}
node_list.append(d)
node_dict[tokens[0]] = True
line_ids.append(tokens[0])
if not node_dict.has_key(mod_wid):
d = {"name": str(mod_wid),"group":1}
node_list.append(d)
node_dict[mod_wid] = True
line_ids.append(mod_wid)
link_d = {"source": line_ids.index(tokens[0]),"target":line_ids.index(mod_wid),"value":tokens[2]}
link_list.append(link_d)
if num_lines > 10000:
break
num_lines +=1
data_dict = {"nodes":node_list, "links":link_list}
print "{\n"
for k,v in data_dict.items():
print '"'+k +'"' +":\n [ \n "
for each_v in v:
print each_v ,","
print "\n],"
print "}"
open_file("lda_input.tsv")

I'm assuming by "efficiently" you're talking about programmer efficiency—how easy it is to read, maintain, and code the logic—rather than runtime speed efficiency. If you're worried about the latter, you're probably worried for no reason. (But the code below will probably be faster anyway.)
The key to coming up with a better solution is to think more abstractly. Think about rows in a CSV file, not lines in a text file; create a dict that can be rendered in JSON rather than trying to generate JSON via string processing; wrap things up in functions if you want to do them repeatedly; etc. Something like this:
import csv
import json
import sys
def parse(inpath, namedict):
lastname = [0]
def lookup_name(name):
try:
print('Looking up {} in {}'.format(name, names))
return namedict[name]
except KeyError:
lastname[0] += 1
print('Adding {} as {}'.format(name, lastname[0]))
namedict[name] = lastname[0]
return lastname[0]
with open(inpath) as f:
reader = csv.reader(f, delimiter=' ', skipinitialspace=True)
for id1, id2, value in reader:
yield {'source': lookup_name(id1),
'target': lookup_name(id2),
'value': value}
for inpath in sys.argv[1:]:
names = {}
links = list(parse(inpath, names))
nodes = [{'name': name} for name in names]
outpath = inpath + '.json'
with open(outpath, 'w') as f:
json.dump({'nodes': nodes, 'links': links}, f, indent=4)

Don't construct the JSON manually. Make it out of an existing Python object with the json module:
def parse(data):
nodes = set()
links = set()
for line in data.split('\n'):
fields = line.split()
id1, id2 = map(int, fields[:2])
value = float(fields[2])
nodes.update((id1, id2))
links.add((id1, id2, value))
return {
'nodes': [{
'name': node
} for node in nodes],
'links': [{
'source': link[0],
'target': link[1],
'value': link[2]
} for link in links]
}
Now, you can use json.dumps to get a string:
>>> import json
>>> data = '1 234 0.2\n1 235 0.1'
>>> parsed = parse(data)
>>> parsed
{'links': [{'source': 1, 'target': 235, 'value': 0.1},
{'source': 1, 'target': 234, 'value': 0.2}],
'nodes': [{'name': 1}, {'name': 234}, {'name': 235}]}
>>> json.dumps(parsed)
'{"nodes": [{"name": 1}, {"name": 234}, {"name": 235}], "links": [{"source": 1, "target": 235, "value": 0.1}, {"source": 1, "target": 234, "value": 0.2}]}'

Tree style dictionary

I am stuck with the following data.
There is a list.
[{name: '/', children: [{name: 'bin'}, {name: 'sbin'}, {name: 'home'}]},
{name: 'home', children: [{name: 'user1'}, {name: 'user2'}]},
{name: 'user2', children: [{name: 'desktop'}]}]
I want to convert above list to the following dictionary.
{name: '/', children: [{name: '/bin'}, {name: '/sbin'}, {name: '/home', children: [{name: 'user1'}, {name: 'user2', children: [{name: 'desktop'}]}]}]}
I write some codes to convert data above style.
def recT(data, child, parent, collector):
dparent = dict(name=parent)
dchildren = dict()
lst = []
for c in child:
lst.append(dict(name=c['name']))
for d in data:
if c['name'] == d['name']:
if len(d) > 1:
dchildren.update(dict(children=recT(data, d['children'], d['name'], collector)))
dparent.update(dchildren)
collector.update(dparent)
return lst
Then,
myd = dict()
for d in data2:
if len(d) > 1:
recT(data2, d['children'], d['name'], myd)
NOTE: data2 is dictionary list I want to covert.
But, the output dictionary is the last record in list:
{'children': [{'name': 'desktop'}], 'name': 'user2'}
Please help.

As lazyr said, you can't duplicate keys in your dict like that. You could convert it to a format like the following to be valid python dict syntax:
{
'/': {
'bin': {},
'sbin': {},
'home': {
'user1': {},
'user2': {
'desktop': {}
}
}
}
The reason you're only getting the last record in the list is because your dict uses unique keys
mydict = {}
mydict['name'] = 1
mydict['name'] # is 1
mydict['name'] = 2
for x,y in mydict.iteritems():
print '{0}: {1}'.format(x,y)
>> name: 2 # Note only one entry

Now, I got it from #lazyr's answer of How to convert a strictly sorted list of strings into dict?.
Then, I converted into string and changed it into wanted format using myReplacer().
here:
def myReplacer(strdata):
strdata = strdata.replace("{", '{ name:')
strdata = strdata.replace(': {', ', children : [{')
strdata = strdata.replace('}', '}]')
strdata = strdata.replace(': None,', '},{ name:')
strdata = strdata.replace(': None', '')
strdata = strdata.replace(", '", "}, { name: '")
return strdata[:-1]
Thanks #lazyr and everybody helped me. It need some polish.