I've got a data object that was created by creating what I assume is a json object -
jsonobj = {}
jsonobj["recordnum"] = callpri
and then pushing that onto a list, as there is more than one of them -
myList = []
myList.append(jsonobj)
Then that gets passed back and forth between flask subroutines and Jinja2 templates, until it ends up a few steps later coming into the function where I need to access that data, and it comes in looking something like so -
techlist: [{'recordnum': '1', 'name': 'Person 1', 'phonenumber': '123-456-7890', 'email': 'person1#company.tld', 'maxnumtechs': 'ALL'}, {'recordnum': '2', 'name': 'Person 2', 'phonenumber': '098-765-4321', 'email': 'person2#company.tld', 'maxnumtechs': 'ALL'}, {'recordnum': '3', 'name': 'Person 3', 'phonenumber': '567-890-1234', 'email': 'person3#company.tld', 'maxnumtechs': 'ALL'}]
I tried a for tech in techlist: print(tech['recordnum']) type deal and got an error, so I started printing types for everything and it's all strings. The for tech in techlist is I think just splitting it all into words even, which is obviously not what I want at all.
I tried messing around with json.loads on techlist, but it complained about expecting an entry in double quotes or something along those lines. I'm totally stumped, and would really appreciate if someone could please tell me how to turn this string back into a list of dicts or a list of json objects, or whatever it takes for me to be able to iterate through the items and access specific fields.
Response to comments about it working right:
It's coming in as a string for me, and I think for the two of you it is working for, you're creating it as a list, so it would work correctly ... sadly, that's my problem, it's a string, not a list, so it's doing this -
(env) [me#box directory]$ cat test.py
techlist = "[{'recordnum': '1', 'name': 'Person 1', 'phonenumber': '123-456-7890', 'email': 'person1#company.tld', 'maxnumtechs': 'ALL'}, {'recordnum': '2', 'name': 'Person 2', 'phonenumber': '098-765-4321', 'email': 'person2#company.tld', 'maxnumtechs': 'ALL'}, {'recordnum': '3', 'name': 'Person 3', 'phonenumber': '567-890-1234', 'email': 'person3#company.tld', 'maxnumtechs': 'ALL'}]"
print(type(techlist))
for tech in techlist:
print(type(tech))
print(str(tech))
(env) [me#box directory]$
(env) [me#box directory]$
(env) [me#box directory]$ python test.py
<class 'str'>
<class 'str'>
[
<class 'str'>
{
<class 'str'>
'
<class 'str'>
r
<class 'str'>
e
<snip>
Update:
Trenton McKinney 's comment worked PERFECTLY, THANK YOU!! If you're so inclined as to post it as a answer I'll accept it as the solution. Thank you thank you thank you!!
Convert the string back to dict:
Use ast.literal_eval to evaluate the string
from ast import literal_eval
techlist = """[{'recordnum': '1', 'name': 'Person 1', 'phonenumber': '123-456-7890', 'email': 'person1#company.tld', 'maxnumtechs': 'ALL'},
{'recordnum': '2', 'name': 'Person 2', 'phonenumber': '098-765-4321', 'email': 'person2#company.tld', 'maxnumtechs': 'ALL'},
{'recordnum': '3', 'name': 'Person 3', 'phonenumber': '567-890-1234', 'email': 'person3#company.tld', 'maxnumtechs': 'ALL'}]"""
print(type(techlist))
>>> <class 'str'>
techlist = literal_eval(techlist)
print(type(techlist))
>>> <class 'list'>
print(techlist)
# output
[{'email': 'person1#company.tld',
'maxnumtechs': 'ALL',
'name': 'Person 1',
'phonenumber': '123-456-7890',
'recordnum': '1'},
{'email': 'person2#company.tld',
'maxnumtechs': 'ALL',
'name': 'Person 2',
'phonenumber': '098-765-4321',
'recordnum': '2'},
{'email': 'person3#company.tld',
'maxnumtechs': 'ALL',
'name': 'Person 3',
'phonenumber': '567-890-1234',
'recordnum': '3'}]
Top answer are about string, but option 3 is about dealing with dictionaries with pandas
Option 1
from csv import reader
import pandas as pd
data=[str]
df=pd.DataFrame( list(reader(data)))
print(df)
results = df[col].to_list()
Option 2
Other wise just split str by a value
result = str.split(',')
Option 3 (I am pretty sure this what you want):
df = pd.DataFrame(techlist)
results = df['recordnum'].to_list()
Hopefully one of those answer is good enough, cause your question is confusing
Related
I need to parse json from a partial string I get back from a web service. I have the following snippet of code which is working fine but is extremely ugly. Is there a better or cleaner way to do this?
x = '"1":{"name":"item one","code":"1"},"2":{"name":"item two","code":"2"},"3":{"name":"item three","code":"3"}'
split = x.split('},')
index = 0
for s in split:
split[index] = '{' + s + '}}'
index += 1
joined = ','.join(split)
joined = '[' + joined[:-1] + ']'
j = json.loads(joined)
print(j)
Here is the result:
[{'1': {'name': 'item one', 'code': '1'}},
{'2': {'name': 'item two', 'code': '2'}},
{'3': {'name': 'item three', 'code': '3'}}]
You can use the following snippet:
>>> [dict([t]) for t in json.loads(f"{{{x}}}").items()]
[{'1': {'name': 'item one', 'code': '1'}},
{'2': {'name': 'item two', 'code': '2'}},
{'3': {'name': 'item three', 'code': '3'}}]
You can fix the inconsistency by hand (add the missing braces) and use json module to parse:
data = json.loads('{' + x + '}')
Then you can convert the parsed data to the desired representation:
[{item[0]: item[1]} for item in data.items()]
#[{'1': {'name': 'item one', 'code': '1'}},
# {'2': {'name': 'item two', 'code': '2'}},
# {'3': {'name': 'item three', 'code': '3'}}]
Otherwise, you will end up implementing your own JSON parser, which is not trivial.
This question already has answers here:
Convert a String representation of a Dictionary to a dictionary
(11 answers)
Closed 2 years ago.
I want to use REGEX to extract the STRING in a curly bracket inside a curly bracket, because there are millions of rows and the position of the inside curly bracket is always different in the 'larger' curly bracket.
Here's the string:
{'eventData': {'type': 'page', 'name': 'chicken 2'},
'eventId': '1993',
'deviceType': 'keroppi',
'pageUrl': '/chicken 2',
'version': '1.0.0.1999-10_7_2020__4_18_30',
'sessionGUID': 'f4123f21-31ad-4e83-ba77-41231238',
'locationid': '0601eba9-1259-4ae6-bad5-7d1231239',
'eventDescription': 'Page Load'}
It could be like this:
{'eventId': '1993',
'deviceType': 'keroppi',
'pageUrl': '/chicken 2',
'eventData': {'type': 'page', 'name': 'chicken 2'},
'version': '1.0.0.1999-10_7_2020__4_18_30',
'sessionGUID': 'f4123f21-31ad-4e83-ba77-41231238',
'locationid': '0601eba9-1259-4ae6-bad5-7d1231239',
'eventDescription': 'Page Load'}
What I want is just the interior curly bracket: {'type': 'page', 'name': 'chicken 2'}
What is the regular expression to get the string after "'eventData':" and before the ","?
Use .get() on the dict:
In [2494]: s = {'eventData': {'type': 'page', 'name': 'chicken 2'},
...: 'eventId': '1993',
...: 'deviceType': 'keroppi',
...: 'pageUrl': '/chicken 2',
...: 'version': '1.0.0.1999-10_7_2020__4_18_30',
...: 'sessionGUID': 'f4123f21-31ad-4e83-ba77-41231238',
...: 'locationid': '0601eba9-1259-4ae6-bad5-7d1231239',
...: 'eventDescription': 'Page Load'}
In [2495]: s.get('eventData')
Out[2495]: {'type': 'page', 'name': 'chicken 2'}
In [2496]: s1 = {'eventId': '1993',
...: 'deviceType': 'keroppi',
...: 'pageUrl': '/chicken 2',
...: 'eventData': {'type': 'page', 'name': 'chicken 2'},
...: 'version': '1.0.0.1999-10_7_2020__4_18_30',
...: 'sessionGUID': 'f4123f21-31ad-4e83-ba77-41231238',
...: 'locationid': '0601eba9-1259-4ae6-bad5-7d1231239',
...: 'eventDescription': 'Page Load'}
In [2497]: s1.get('eventData')
Out[2497]: {'type': 'page', 'name': 'chicken 2'}
If its a string representation of a JSON, then use:
import ast
ast.literal_eval(s).get('eventData')
I have an SQL database that I need to fetch and convert to JSON. I am thinking that the first step to do that is to fetch the data from the database and load it as a dataframe, then convert the dataframe into JSON object.
Let's say I have the following dataframe.
df_school = pd.DataFrame({'id':[1,2,3,4], 'school_code': ['ABC', 'IJK', 'QRS', 'XYZ'], 'name': ['School A','School B', 'School C', 'School D'], 'type':['private', 'public', 'public', 'private']})
print(df_school)
I want to convert it to JSON with the following code.
import collections
object_list =[]
for idx, row in df_school.iterrows():
d = collections.OrderedDict()
d['id'] = row['id']
d['school_code'] = row['school_code']
d['name'] = row['name']
d['type'] = row['type']
object_list.append(d)
j = json.dumps(object_list)
object_list = 'school_objects.js'
f = open(object_list, 'w')
print(j)
But the result is string. It only looks like a JSON, but when I try to access the item inside the so-called JSON, like j[0] it prints [, not an item inside the JSON.
I also tried another approach, by converting the result from SQL directly to JSON.
query = "Select * from school;"
df_school = pd.read_sql_query(query, connection)
json_school = df_school.head(10).to_json(orient='records')
But I also still got string.
How do I convert it to real JSON object?
Given the provided df_school variable, we can just do j=df_school.to_json(orient='records') to turn it into a JSON formatted string.
Once we have j storing the JSON formatted string, if we want to do something with it, we first have to load the JSON into Python again using json.loads(j).
So if we do:
j = df_school.to_json(orient='records')
# parse j into Python
loaded_json = json.loads(j)
print(loaded_json[0])
# print outputs: {'id': 1, 'name': 'School A', 'school_code': 'ABC', 'type': 'private'}
Hope this helps!
import pandas as pd
import json
df_school = pd.DataFrame({'id':[1,2,3,4], 'school_code': ['ABC', 'IJK', 'QRS', 'XYZ'], 'name': ['School A','School B', 'School C', 'School D'], 'type':['private', 'public', 'public', 'private']})
str_school = df_school.to_json(orient='records')
json_school = json.loads(str_school)
json_school[0]
{'id': 1, 'school_code': 'ABC', 'name': 'School A', 'type': 'private'}
JSON is a string encoding of objects.
Once you use json.dumps() or similar, you'll get a string.
Try the below code, Hope this will help :
data = [{columns:df_school.iloc[i][columns] for columns in list(df_school.columns) } for i in range(df_school.shape[0]) ]
print(data)
print("***********************")
print(type(data[0]))
Ouput will be :
[{'id': 1, 'school_code': 'ABC', 'name': 'School A', 'type': 'private'},
{'id': 2, 'school_code': 'IJK', 'name': 'School B', 'type': 'public'},
{'id': 3, 'school_code': 'QRS', 'name': 'School C', 'type': 'public'},
{'id': 4, 'school_code': 'XYZ', 'name': 'School D', 'type': 'private'}]
*************************
<class 'dict'>
data={k:list(v.values()) for k,v in df_school.to_dict().items()}
{
'id': [1, 2, 3, 4],
'school_code': ['ABC', 'IJK', 'QRS', 'XYZ'],
'name': ['School A', 'School B', 'School C', 'School D'],
'type': ['private', 'public', 'public', 'private']
}
I am trying to set a flag based on the approval type "TEST" when its value equals '1' but no "-1",am using the below but running into following error
flag = 'false'
ApprovalItem = [{'by': {'username': 'lnxbuild', 'name': 'Linux Build Service Account', 'email': 'lnxbuild#localhost'}, 'type': 'VRIF', 'description': 'Verified', 'value': '1', 'grantedOn': 1376515352}, {'by': {'username': 'c_ssugas', 'name': 'name', 'email': 'c_ssugas#company.com'}, 'type': 'TEST', 'description': 'Developer Verified', 'value': '-1', 'grantedOn': 1376532352}, {'by': {'username': 'ytkim', 'name': 'Ben Young Tae Kim', 'email': 'ytkim#company.com'}, 'type': 'CRVW', 'description': 'Code Review', 'value': '1', 'grantedOn': 1376514495}, {'by': {'username': 'ytkim', 'name': 'Ben Young Tae Kim', 'email': 'ytkim#company.com'}, 'type': 'TEST', 'description': 'Developer Verified', 'value': '1', 'grantedOn': 1376514495}]
if ApprovalItem['type'] == 'TEST' and ApprovalItem['description'] == 'Developer Verified' and ApprovalItem['value'] == '1' :
flag = True
print flag
Error:-
TypeError: list indices must be integers, not str
ApprovalItem is a list of dictionaries, not a dictionary itself.
>>> ApprovalItem = [{'by': {'username': 'lnxbuild', 'name': 'Linux Build Service Account', 'email': 'lnxbuild#localhost'}, 'type': 'VRIF', 'description': 'Verified', 'value': '1', 'grantedOn': 1376515352}, {'by': {'username': 'c_ssugas', 'name': 'name', 'email': 'c_ssugas#company.com'}, 'type': 'TEST', 'description': 'Developer Verified', 'value': '-1', 'grantedOn': 1376532352}, {'by': {'username': 'ytkim', 'name': 'Ben Young Tae Kim', 'email': 'ytkim#company.com'}, 'type': 'CRVW', 'description': 'Code Review', 'value': '1', 'grantedOn': 1376514495}, {'by': {'username': 'ytkim', 'name': 'Ben Young Tae Kim', 'email': 'ytkim#company.com'}, 'type': 'TEST', 'description': 'Developer Verified', 'value': '1', 'grantedOn': 1376514495}]
>>> print type(ApprovalItem)
<type 'list'>
>>> print type(ApprovalItem[0])
<type 'dict'>
You probably want a for-loop:
>>> for d in ApprovalItem:
... if d['type'] == 'TEST' and d['description'] == 'Developer Verified' and d['value'] == '1' :
... flag = True
... print flag
...
True
You are using list to store your several dict.
You can use for, I think, to check each dict.
I'm using python to fetch issues from Jira with xml-rpc. It works well except it is missing the 'Resolution' field in the returned dictionary. For example 'Fixed', or 'WontFix' etc.
This is how I get the issue from Jira:
import xmlrpclib
s = xmlrpclib.ServerProxy('http://myjira.com/rpc/xmlrpc')
auth = s.jira1.login('user', 'pass')
issue = s.jira1.getIssue(auth, 'PROJ-28')
print issue.keys()
And this is the list of fields that I get back:
['status', 'project', 'attachmentNames', 'votes', 'updated',
'components', 'reporter', 'customFieldValues', 'created',
'fixVersions', 'summary', 'priority', 'assignee', 'key',
'affectsVersions', 'type', 'id', 'description']
The full content is:
{'affectsVersions': [{'archived': 'false',
'id': '11314',
'name': 'v3.09',
'released': 'false',
'sequence': '7'}],
'assignee': 'myuser',
'attachmentNames': '2011-08-17_attach.tar.gz',
'components': [],
'created': '2011-06-14 12:33:54.0',
'customFieldValues': [{'customfieldId': 'customfield_10040', 'values': ''},
{'customfieldId': 'customfield_10010',
'values': 'Normal'}],
'description': "Blah blah...\r\n",
'fixVersions': [],
'id': '28322',
'key': 'PROJ-28',
'priority': '3',
'project': 'PROJ',
'reporter': 'myuser',
'status': '1',
'summary': 'blah blah...',
'type': '1',
'updated': '2011-08-18 15:41:04.0',
'votes': '0'}
When I do:
resolutions = s.jira1.getResolutions(auth )
pprint.pprint(resolutions)
I get:
[{'description': 'A fix for this issue is checked into the tree and tested.',
'id': '1',
'name': 'Fixed'},
{'description': 'The problem described is an issue which will never be fixed.',
'id': '2',
'name': "Won't Fix"},
{'description': 'The problem is a duplicate of an existing issue.',
'id': '3',
'name': 'Duplicate'},
{'description': 'The problem is not completely described.',
'id': '4',
'name': 'Incomplete'},
{'description': 'All attempts at reproducing this issue failed, or not enough information was available to reproduce the issue. Reading the code produces no clues as to why this behavior would occur. If more information appears later, please reopen the issue.',
'id': '5',
'name': 'Cannot Reproduce'},
{'description': 'Code is checked in, and is, er, ready for build.',
'id': '6',
'name': 'Ready For Build'},
{'description': 'Invalid bug', 'id': '7', 'name': 'Invalid'}]
The Jira version is v4.1.1#522 and I using Python 2.7.
Any ideas why I don't get a field called 'resolution'?
Thanks!
The answer is that the getIssue method in JiraXmlRpcService.java calls makeIssueStruct with a RemoteIssue object. The RemoteIssue object contains the Resolution field, but makeIssueStruct copies only values that are set. So if Resolution is not set, it won't appear in the Hashtable there.