We Keep Coding

How to get complete dictionary data from a JSON file based on a value

I have a json file, which I will read and based on the xyz details will create excel report. Below is the sample json file I will use to extract the information which holds data in format of multiple dictionaries.
Now my requirement is to fetch xyz value one by one and based on it using certain field create a report. Below is the small snippet of the code where I am reading the file and based on key populating results. The data I am referencing after reading it from a file.
def pop_ws(dictionary,ws):
r=1
count=1
for k,v in dictionary.items():
offs=len(v['current'])
ws.cell(row=r+1,column=1).value = k
ws.cell(row=r+1,column=4).value = v['abc']
ws.cell(row=r+1,column=5).value = v['def']
wrk=read_cves(k)
count +=1
if wrk !='SAT':
ws.cell(row=r+1,column=7).value =k
ws.cell(row=r+1,column=8).value =tmp1['public_date']
if 'cvss' in list(tmp1.keys()):
.
.
.
def read_f(data):
with open(dat.json) as f:
wrk = f.read()
I am pretty much stuck on how to code in def read_f(data):, so that it read dat.json and based on value i.e data, fetch details defined as in dictionary structure one by one for all the required data and populate as defined under pop_ws in my code.
The data in def read_f(data): will be a dynamic value and based on it I need to filter the dictionary which have value (stored in data) defined against a key and then extract the whole dictionary into another json file.
Any suggestion on this will be appreciated.

Use json package to load json format data like below:
# Python program to read
# json file
import json
# Opening JSON file
f = open('data.json',)
# returns JSON object as
# a dictionary
data = json.load(f)
# Iterating through the json
# list
for i in data['emp_details']:
print(i)
# Closing file
f.close()
I got this from this link, now you can get dict from the file.
Next you can just filter the dict with specific value like below.
You should use filter() built-in function, with a function that returns True, if the dictionary contains one of the values.
def filter_func(dic, filterdic):
for k,v in filterdic.items():
if k == 'items':
if any(elemv in dic[k] for elemv in v):
return True
elif v == dic[k]:
return True
return False
def filter_cards(deck, filterdic):
return list(filter(lambda dic, filterdic=filterdic: filter_func(dic, filterdic) , deck))
You should use a dictionary as the second element.
filter_cards(deck, {'CVE': 'moderate'})
Hopefully, this could helpful for your situation.
Thanks.

Once you get your json object, you can access each value using the key like so:
print(json_obj["key"]) #prints the json value for that key
In your case
print(wrk["CVE"]) # prints CVE-2020-25624

How to iterate class objects with arguments

I'm trying to extract info from a csv file. Each line in the file contains several values for arguments that are used in a class. I want to take these values and give them to an object, but it needs to parse each line and create a new object with these argument values.
I would need each object to have a different name, e.g.: obj1 obj2 etc.
Here's how it would operate without iteration (only one object).
with open("file.csv") as csvfile:
file = csv.DictReader(csvfile)
for line in file:
name = line["name"]
cost = line["cost"]
#one object manually entered
obj = Class(name,cost)
csvfile.close()

You can create dictionary of objects and get those objects by corresponding key in dictionary.
obj_dict = {}
with open("file.csv") as csvfile:
file = csv.DictReader(csvfile)
for idx, line in enumarate(file):
name = line["name"]
cost = line["cost"]
#one object manually entered
obj = Class(name,cost)
obj_dict['obj{}'.format(idx+1)] = obj
csvfile.close()
obj_dict['obj1'] # returns first object of type Class
Also:
Another way you can create variable with your name programatically , although it is not the recommended way for instantiating variables in code (for example your IDE might show error because it will think variable is not defined):
_g = globals()
with open("file.csv") as csvfile:
file = csv.DictReader(csvfile)
for idx, line in enumarate(file):
name = line["name"]
cost = line["cost"]
#one object manually entered
obj = Class(name,cost)
_g['obj{}'.format(idx+1)] = obj
csvfile.close()

Use a list, that's what they are for:
with open("file.csv") as csvfile:
reader = csv.DictReader(csvfile)
objects = [Class(row["name"], row["cost"]) for row in reader]
# you don't need to close the file when using `with`
print(objects[0])
print(objects[1])
# etc

Populating a dictionary from a csv file with extremely large field sizes

I've received an error trying to import a .csv file from the csv module when my field size exceeded 131,072. The csv module exports files with fields exceeding 131,072. It's my value for the dictionary with the massive size. My keys are small. Do I need a different file format to store dictionaries with huge values?
I use csv throughout my program, using it consistently is convenient. If multiple data types is unavoidable, what is a good alternative? I'd like to store values which could be thousands-millions of characters in length.
Here's the error message
dictionary = e.csv_import(filename)
File "D:\Matt\Documents\Projects\Python\Project 17\e.py", line 8, in csv_import
for key, value in csv.reader(open(filename)):
_csv.Error: field larger than field limit (131072)
Here's my code
def csv_import(filename):
dictionary = {}
for key, value in csv.reader(open(filename)):
dictionary[key] = value
return dictionary
def csv_export(dictionary, filename):
csv_file = csv.writer(open(filename, "w"))
for key, value in dictionary.items():
csv_file.writerow([key, value])

If you're looking for an alternative, you should probably just use pickle. It's much faster, and much easier than converting from and to a .csv file.
eg.
with open(filename) as f:
dictionary = pickle.load(f)
and
with open(filename) as f:
pickle.dump(dictionary, f)
One downside is that it's not easily read by other languages (if that's a consideration)

You can adjust the maximum field size via:
>>> import csv
>>> csv.field_size_limit()
131072
>>> old_size = csv.field_size_limit(1024*1024)
>>> csv.field_size_limit()
1048576
For alternatives see below.
You want a persistent dictionary so you could use the shelve module.
import shelve
# open shelf and write a large value
shelf = shelve.open(filename)
shelf['a'] = 'b' * 200000
shelf.close()
# read it back in
shelf = shelve.open(filename)
>>> print len(shelf['a'])
200000
Under the hood it's using pickle so there are compatibility issues if you wanted to use the shelf file outside of Python. But if compatibility is required, you could use JSON to serialise your dictionary - I assume that the dictionary's values are strings.
import json
def dict_import(filename):
with open(filename) as f:
return json.load(f)
def dict_export(dictionary, filename):
with open(filename, "w") as f:
json.dump(dictionary, f)

Dynamically naming tuples for redis

I have a csv file in which each line contains a person's ID # and then a bunch of attributes. I want to be able to create a tuple for each person that contains all their attributes and then name the tuple some variation of their ID #.
All these tuples will then be added to a set in redis for storage.
I can't seem to figure out how to create a tuple that is named after the persons ID#.
I know its not best practice to dynamically name variables, but I would rather not put all the tuples in a list or set to then put into a redis set (which is a must); it just seems inefficient and cumbersome.
This is the code I have now:
with open('personlist.csv','rb') as f:
for line in f:
row = line.split(',')
personID = row[0]
attrb1 = row[1]
attrb2 = row[2]
attrb3 = row[3]
# Need to name tuple here and define as (attrb1, attrb2, attrb3)
r.lpush('allpersonslist',tuple)

This example needs additional code to function. I'm assuming you are using a redis API such as redis-py. The variable r is an open connection to redis.
import pickle
with open('personlist.csv', 'rb') as f:
for line in f:
row = line.split(',')
personID = row[0]
attrb1 = row[1]
attrb2 = row[2]
attrb3 = row[3]
#put the attributes in a tuple
tuple = (attrb1, attrb2, attrb3)
#serialize the tuple before adding it to the set
r.set("person/%d" %personID,pickle.dumps(tuple,-1))
def getPerson(Id):
return pickle.loads(r.get("person/%d" %Id))
You can call getPerson(5) to return the tuple associated with a person of ID 5.

If each person have max N attribute, there is language-independent solution based on hash. Here list 3 commands to save/read/delete values for a person.
HMSET 'allpersonshash' personID:0 personID:1 ......
HMGET 'allpersonshash' personID:0 personID:1 personID:2 ... personID:N
HDEL 'allpersonshash' personID:0 personID:1 personID:2 ... personID:N

A fairly general way to do it would be to use sorted sets with json blobs, eg:
ZADD userid, '{field1:value1,field2:value2}'

Dictionaries under dictionary are copied by value not copied by reference?

In the beginning I create xls_dict and xml_dict
Then wrapped the above 2 dicts into dictionary's creation. (I though it is copy be reference)
So I will load the json files into the 2 dicts.
However, I found when I exit the
for export_file, data_dict in json_files.iteritems(): block.
the xls_dict and xml_dict are not changed.
The was not my expectation.
Where did I misunderstand ? Thanks
xls = MultiLangXls(xls_path, XLS_COLUMN)
xls_dict = xls.load()
xml = MultiLangXML(xml_path)
xml_dict = xml.load()
json_files={
"xls.json": xls_dict,
"xml.json": xml_dict
}
for export_file, data_dict in json_files.iteritems():
if os.path.isfile(export_file):
pass
else: # If json file not exists, then ouput the dict into json file
with open( export_file , 'w') as f:
json.dump(data_dict, f, encoding="utf-8")
load_file = open(export_file).read().decode("utf-8-sig")
data_dict = {}
data_dict = json.loads( load_file )

The data_dict variable does refer to the same object and not to a copy. Assigning a new dict to data_dict, however, disconnects the variable from that object and assigns to it a brand new one.
To clear the existing dict, and fill it with new contents, you want to write something like this:
data_dict.clear()
data_dict.update(json.loads(load_file))

Problem here:
data_dict = {} #You create new empty dictionary
# And rewrite it. But old dictionare are not cleaned.
data_dict = json.loads( load_file )
You must clean by data_dict.clean()

You are reassigning data_dict to a new dictionary object.
See Passing values in Python