We Keep Coding

CSV to json convert

I have this data in .csv format:
I want to convert it into .json format like this :
{
"title": "view3",
"sharedWithOrganization": false,
"sharedWithUsers": [
"81241",
"81242",
"81245",
"81265"
],
"filters": [{"field":"Account ID","comparator":"==","value":"prod"}]
},
{
"title": "view3",
"sharedWithOrganization": true,
"sharedWithUsers": [],
"filters": [{"field":"Environment_AG","comparator":"=#","value":"Development"}]
}
Below these are the conversion for comparator
'equals' means '=='
'not equal' means '!='
'contains' means '=#'
'does not contain' means '!=#'
Can you please help me convert .csv to .json I am unable to convert using python .

What I would do, without giving you the proper answer (doing it yourself is better for learning).
First : Create an Object containing your informations
class View():
def __init__(self, title, field, comparator, value, sharedWithOrganization, user1, user2, user3, user4, user5, user6):
self.title = title
self.field = field
self.comparator = comparator
self.value = value
self.sharedWithOrganization = sharedWithOrganization
self.user1 = user1
...
self.user6 = user6
Then I would load the CSV and create an object for each line, and store them in a Dict with the following structure :
loadedCsv = { "Your line title (ex : view3)" : [List of all the object with the title view3] }
Yes, with this point of view, there is data redundancy of the title parameter, you can chose to remove it from the object.
When this is done, I would, for each title in my dictionary, get all the element I need and format them in JSON by using "import json" (c.f python documentation : https://docs.python.org/3/library/json.html)

Hehere I'm posting my work on your doubt.. hope u and others will find it helpful.
But I want you to try urself....
import csv
import json
def csv_to_json(csvFilePath, jsonFilePath):
jsonArray = []
jsonArray2 = []
with open(csvFilePath, encoding='utf-8') as csvf:
csvReader = csv.DictReader(csvf)
for row in csvReader:
if row["comparator"] == "equals":
row["comparator"]="=="
elif row["comparator"]=="not equal":
row["comparator"]="!#"
elif row["comparator"]=="contains":
row["comparator"]="=#"
elif row["comparator"]=="does not contain":
row["comparator"]="!=#"
final_data={
"title":row["title"],
"sharedWithOrganization":bool(row["sharedWithOrganization"]),
"sharedWithUsers": [
row["user1"],
row["user2"],
row["user3"],
row["user4"],
row["user5"],
row["user6"]
],
"filters":[ {"field":row['field'],"comparator":row["comparator"],"value":row["value"]} ]
}
jsonArray.append(final_data)
with open(jsonFilePath, 'w', encoding='utf-8') as jsonf:
jsonString = json.dumps(jsonArray, indent=4)
jsonf.write(jsonString)
csvFilePath = r'test.csv'
jsonFilePath = r'test11.json'
csv_to_json(csvFilePath, jsonFilePath)

My spreadsheet reading function constantly overwrites everything

Okay so I have a spreadsheet and I want to put all the entries into my nested dictionary lists.
I decided to use two for loops to iterate through the spreadsheet. And safe the value of the cell to the according nested dictionary.
Here is my code (I know it's shit I'm pretty inexperienced):
def SpGetLink():
global SpDic
for row in SpDicGen():
Data = dict.fromkeys(SpDic.values(), {})
Data[SpDic[row]]["Link"] = []
Data[SpDic[row]]["Title"] = []
for col in range(1000):
if ws.cell(row=row, column=col + 6).hyperlink is not None:
data = str(ws.cell(row=row, column=col + 6).hyperlink.target)
if data.startswith("http"):
if data not in Data[SpDic[row]]["Link"]:
Data[SpDic[row]]["Link"].append(data)
json.dump(Data, open("Data.json", "w+"), indent=4) # , sort_keys=True)
else:
Data[SpDic[row]]["Title"].append(data)
SpDic is a seperate Dictionary to get the corresponding Name to the row.
Now my Problem is the following.
When I open Data.json every list that should contain all links in the corresponding row contains the same 5 links which are the last 5 links the the spreadsheet. It looks something like this:
"smile": {
"Link": [
"https://media.giphy.com/media/k7J8aS3xpmhpK/giphy.gif",
"https://media.giphy.com/media/aY1HMl4E1Ju1y/giphy.gif",
"https://media.giphy.com/media/RLJxQtX8Hs7XytaoyX/giphy.gif",
"https://media.giphy.com/media/1448TKNMMg4BFu/giphy.gif",
"https://media.giphy.com/media/b7l5cvG94cqo8/giphy.gif"
],
"Title": []
},
"grin": {
"Link": [
"https://media.giphy.com/media/k7J8aS3xpmhpK/giphy.gif",
"https://media.giphy.com/media/aY1HMl4E1Ju1y/giphy.gif",
"https://media.giphy.com/media/RLJxQtX8Hs7XytaoyX/giphy.gif",
"https://media.giphy.com/media/1448TKNMMg4BFu/giphy.gif",
"https://media.giphy.com/media/b7l5cvG94cqo8/giphy.gif"
],
"Title": []
},
"laugh": {
"Link": [
"https://media.giphy.com/media/k7J8aS3xpmhpK/giphy.gif",
"https://media.giphy.com/media/aY1HMl4E1Ju1y/giphy.gif",
"https://media.giphy.com/media/RLJxQtX8Hs7XytaoyX/giphy.gif",
"https://media.giphy.com/media/1448TKNMMg4BFu/giphy.gif",
"https://media.giphy.com/media/b7l5cvG94cqo8/giphy.gif"
],
"Title": []
},
Does anyone have an Idea why this is happening and how to fix it ?

I think the reason why your data is overwritten each time is because you are dumping your json inside the for loop. I moved it outside and I think this should do the trick.
def SpGetLink():
Data = []
global SpDic
for row in SpDicGen():
Data = dict.fromkeys(SpDic.values(), {})
Data[SpDic[row]]["Link"] = []
Data[SpDic[row]]["Title"] = []
for col in range(1000):
if ws.cell(row=row, column=col + 6).hyperlink is not None:
data = str(ws.cell(row=row, column=col + 6).hyperlink.target)
if data.startswith("http"):
if data not in Data[SpDic[row]]["Link"]:
Data[SpDic[row]]["Link"].append(data)
else:
Data[SpDic[row]]["Title"].append(data)
with open("Data.json", "w") as f:
json.dump(Data, f, indent=4)

I fixed it by just making one of the for loops into a function and calling it in another for loop.

Syntax to load nested in nested keys of JSON files

I have a big tree in a JSON file and I'm searching the python syntax for loading nested in nested keys from this JSON.
Assume I have this :
{
"FireWall": {
"eth0": {
"INPUT": {
"PING": 1,
}
}
}
}
According to the man page and some questions in Stackoverflow i tried this (and some variations) :
import json
config = open('config.json', 'r')
data = json.load('config')
config.close()
if data['{"FireWall", {"eth0", {"INPUT", {"Ping"}}}}'] == 1:
print('This is working')
With no result. What is the right way to do this (as simple as possible) ? Thank you !

You are trying data = json.load('config') to load string not file object and data['{"FireWall", {"eth0", {"INPUT", {"Ping"}}}}'] it's not right way to access nested dictionary key value.
import json
with open('config.json', 'r') as f:
data = json.load(f)
if data["FireWall"]["eth0"]["INPUT"]["Ping"] == 1:
print('This is working')

data is a nested dictionary, so:
data["FireWall"]["eth0"]["INPUT"]["Ping"]
will be equal to 1; or at least it will when you fix your call to json.load.

Try this:
data["FireWall"]["eth0"]["INPUT"]["PING"]
This will give you the value in PING

How can I correctly set a nested dict path for a json when querying

I am getting the error:
Exception has occurred: KeyError
'Key not exists'
File "C:\ShopFloor\main.py", line 59, in main
res = qe.at('vehicledata.result.NUM').where('Status', '=', 1).where('Process', '=', 'SPA').get()
File "C:\ShopFloor\main.py", line 96, in <module>
main()
...in the following line:
qe = JsonQ(r"C:\ShopFloor\data.json")
res = qe.at('vehicledata.result.NUM').where('Status', '=', 1).where('Process', '=', 'SPA').get()
pprint(res)
JsonQ.reset(qe)
vehicledata is a string however result.NUM is dynamic, as such there are about 2000 different dicts for result.NUM. My query is aiming at getting all nums where status is 1 and process is SPA: how can I get this to work? I'm thinking it's due to the dynamic dict any help is appreciated
Here is a small sample data from data.json
{
"vehicledata": [
{
"12345678901234567": {
"Process": "SPA",
"Status": 0
}
},
{
"12345678901234567": {
"Process": "Road",
"Status": 0
}
},

Or you can just load it with json package and filter it that way.
import json
with open('data.json', 'r') as f:
data = json.load(f)
curr_position = data['vehicledata']
result = [key for list_item in curr_position for key in list_item.keys() if list_item[key]['Status'] == 1 and list_item[key]['Process'] == 'SPA']
print(result)
According to the jsonq docs, JsonQ does basically the same thing under the hood.

Based on the jsonq docs sample you provided you need to change your code to the following:
from pyjsonq import JsonQ
from pprint import pprint
import json
qe = JsonQ("data.json")
res = qe.at('vehicledata').get()
l = list(map(lambda r: r.values()[0], res))
d = {"vehicledata": l}
with open('extarcted_data.json', 'wb') as outfile:
json.dump(d, outfile)
q = JsonQ("extarcted_data.json")
r = q.at('vehicledata').where('Status', '=', 1).where('Process', '=', 'SPA').get()
print(r)
JsonQ.reset(q)
JsonQ.reset(qe)
The idea is the following you you take all the elements that can change between vehicledata and the object you need. Take the standard json library and wrap your previous object to the new one without extra-layer. After this use JsonQ for convenient search. The code above is the proof of concept and may be not the best solution. Based on the documentation JsonQ doesn't allow skip layers and you have to know your keys in order to retrieve data. In you case you would have to know 12345678901234567 or any number that would be instead of it.
Alternative soltuon
This solution makes usage of JsonQ doubtful. Nevertheless, I will post it here as a reference:
from pyjsonq import JsonQ
from pprint import pprint
qe = JsonQ("data.json")
objs = qe.at('vehicledata').get()
for obj in objs:
values = obj.values()[0]
if 'Status' in values and values['Status'] == 1 and 'Process' in values and values['Process'] == 'SPA':
pprint(values)
JsonQ.reset(qe)

Writing JSON data in python. Format

I have this method that writes json data to a file. The title is based on books and data is the book publisher,date,author, etc. The method works fine if I wanted to add one book.
Code
import json
def createJson(title,firstName,lastName,date,pageCount,publisher):
print "\n*** Inside createJson method for " + title + "***\n";
data = {}
data[title] = []
data[title].append({
'firstName:', firstName,
'lastName:', lastName,
'date:', date,
'pageCount:', pageCount,
'publisher:', publisher
})
with open('data.json','a') as outfile:
json.dump(data,outfile , default = set_default)
def set_default(obj):
if isinstance(obj,set):
return list(obj)
if __name__ == '__main__':
createJson("stephen-king-it","stephen","king","1971","233","Viking Press")
JSON File with one book/one method call
{
"stephen-king-it": [
["pageCount:233", "publisher:Viking Press", "firstName:stephen", "date:1971", "lastName:king"]
]
}
However if I call the method multiple times , thus adding more book data to the json file. The format is all wrong. For instance if I simply call the method twice with a main method of
if __name__ == '__main__':
createJson("stephen-king-it","stephen","king","1971","233","Viking Press")
createJson("william-golding-lord of the flies","william","golding","1944","134","Penguin Books")
My JSON file looks like
{
"stephen-king-it": [
["pageCount:233", "publisher:Viking Press", "firstName:stephen", "date:1971", "lastName:king"]
]
} {
"william-golding-lord of the flies": [
["pageCount:134", "publisher:Penguin Books", "firstName:william","lastName:golding", "date:1944"]
]
}
Which is obviously wrong. Is there a simple fix to edit my method to produce a correct JSON format? I look at many simple examples online on putting json data in python. But all of them gave me format errors when I checked on JSONLint.com . I have been racking my brain to fix this problem and editing the file to make it correct. However all my efforts were to no avail. Any help is appreciated. Thank you very much.

Simply appending new objects to your file doesn't create valid JSON. You need to add your new data inside the top-level object, then rewrite the entire file.
This should work:
def createJson(title,firstName,lastName,date,pageCount,publisher):
print "\n*** Inside createJson method for " + title + "***\n";
# Load any existing json data,
# or create an empty object if the file is not found,
# or is empty
try:
with open('data.json') as infile:
data = json.load(infile)
except FileNotFoundError:
data = {}
if not data:
data = {}
data[title] = []
data[title].append({
'firstName:', firstName,
'lastName:', lastName,
'date:', date,
'pageCount:', pageCount,
'publisher:', publisher
})
with open('data.json','w') as outfile:
json.dump(data,outfile , default = set_default)

A JSON can either be an array or a dictionary. In your case the JSON has two objects, one with the key stephen-king-it and another with william-golding-lord of the flies. Either of these on their own would be okay, but the way you combine them is invalid.
Using an array you could do this:
[
{ "stephen-king-it": [] },
{ "william-golding-lord of the flies": [] }
]
Or a dictionary style format (I would recommend this):
{
"stephen-king-it": [],
"william-golding-lord of the flies": []
}
Also the data you are appending looks like it should be formatted as key value pairs in a dictionary (which would be ideal). You need to change it to this:
data[title].append({
'firstName': firstName,
'lastName': lastName,
'date': date,
'pageCount': pageCount,
'publisher': publisher
})