Related
Input data:
data = [
['QR', ''],
['Cust', ''],
['fea', 'restroom'],
['chain', 'pa'],
['store', 'cd'],
['App', ''],
['End', 'EndnR'],
['Request', '0'],
['Sound', '15'],
['Target', '60'],
['Is', 'TRUE']
]
I want to turn this into a dictionary, and each blank value indicates the start of a new, nested sub-dictionary.
Desired output:
{
'QR': {
'Cust': {
'fea': 'restroom ',
'chain': 'pa',
'store': 'cd'
},
'App': {
'End': 'EndnR',
'Request': '0',
'Sound': '15',
'Target': '60',
'Is': 'true'
},
}
}
Here is my code so far:
from collections import defaultdict
res = defaultdict(dict)
for i in data:
res[i[0]] = i[1]
print(res)
But it only creates a flat dictionary with some blank values, not a nested dictionary.
try this:
result = {}
nbr_keys = 0
keys = [ item[0] for item in data if item[1] == "" ]
for index, item in enumerate(data):
if index == 0:
if item[1] == "":
key = item[0]
di[item[0]] = {}
else:
if item[1] == "":
di[key].update({item[0]: {}})
nbr_keys +=1
else:
di[key][keys[nbr_keys]].update({item[0]: item[1]})
which outputs this:
{'QR': {'Cust': {'fea': 'restroom', 'chain': 'pa', 'store': 'cd'},
'App': {'End': 'EndnR',
'Request': '0',
'Sound': '15',
'Target': '60',
'Is': 'TRUE'}}}
I have been trying to seed a django DB with some covid data from an api and get a KeyError for a particular data type - in the source it is a floating_timstamp ("lab_report_date" : "2014-10-13T00:00:00.000"). (edit: not sure if the type is relevant, but trying to be comprehensive here).
I tried doing a more simple API request in python but get the same keyError. Below is my code and the error message.
import requests
response = requests.get("https://data.cityofchicago.org/resource/naz8-j4nc.json")
print(response.json())
The output looks like this:
[
{
"cases_age_0_17": "1",
"cases_age_18_29": "1",
"cases_age_30_39": "0",
"cases_age_40_49": "1",
"cases_age_50_59": "0",
"cases_age_60_69": "0",
"cases_age_70_79": "1",
"cases_age_80_": "0",
"cases_age_unknown": "0",
"cases_asian_non_latinx": "1",
"cases_black_non_latinx": "0",
"cases_female": "1",
"cases_latinx": "1",
"cases_male": "3",
"cases_other_non_latinx": "0",
"cases_total": "4",
"cases_unknown_gender": "0",
"cases_unknown_race_eth": "1",
"cases_white_non_latinx": "1",
"deaths_0_17_yrs": "0",
"deaths_18_29_yrs": "0",
"deaths_30_39_yrs": "0",
"deaths_40_49_yrs": "0",
show more (open the raw output data in a text editor) ...
"hospitalizations_unknown_gender": "3",
"hospitalizations_unknown_race_ethnicity": "16",
"hospitalizations_white_non_latinx": "135"
}
]
So far so good, but if I try to extract the problem key, i get the KeyError:
report_date = []
for i in response.json():
ls = i['lab_report_date']
report_date.append(ls)
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/var/folders/h3/5wlbmz0s3jb978hyhtvf9f4h0000gn/T/ipykernel_2163/2095152945.py in <module>
1 report_date = []
2 for i in response.json():
----> 3 ls = i['lab_report_date']
4 report_date.append(ls)
KeyError: 'lab_report_date'
This issue occurs with or without using a for loop. I've gotten myself real turned around, so apologies if there are any errors or omissions in my code.
Because there's an item in the array response.json() that does not contain a key lab_report_date. That happens when the backend data is not so clean.
So what you need to do is to use try-except code block to handle this exception. The following code runs well now.
import requests
response = requests.get("https://data.cityofchicago.org/resource/naz8-j4nc.json")
print("The total length of response is %s" % len(response.json()))
report_date = []
for i in response.json():
try:
ls = i['lab_report_date']
report_date.append(ls)
except:
print("There is an item in the response containing no key lab_report_date:")
print(i)
print("The length of report_date is %s" % len(report_date))
The output of the above code is as follows.
The total length of response is 592
There is an item in the response containing no key lab_report_date:
{'cases_total': '504', 'deaths_total': '1', 'hospitalizations_total': '654', 'cases_age_0_17': '28', 'cases_age_18_29': '116', 'cases_age_30_39': '105', 'cases_age_40_49': '83', 'cases_age_50_59': '72', 'cases_age_60_69': '61', 'cases_age_70_79': '25', 'cases_age_80_': '14', 'cases_age_unknown': '0', 'cases_female': '264', 'cases_male': '233', 'cases_unknown_gender': '7', 'cases_latinx': '122', 'cases_asian_non_latinx': '15', 'cases_black_non_latinx': '116', 'cases_white_non_latinx': '122', 'cases_other_non_latinx': '30', 'cases_unknown_race_eth': '99', 'deaths_0_17_yrs': '0', 'deaths_18_29_yrs': '0', 'deaths_30_39_yrs': '0', 'deaths_40_49_yrs': '1', 'deaths_50_59_yrs': '0', 'deaths_60_69_yrs': '0', 'deaths_70_79_yrs': '0', 'deaths_80_yrs': '0', 'deaths_unknown_age': '0', 'deaths_female': '0', 'deaths_male': '1', 'deaths_unknown_gender': '0', 'deaths_latinx': '0', 'deaths_asian_non_latinx': '0', 'deaths_black_non_latinx': '0', 'deaths_white_non_latinx': '1', 'deaths_other_non_latinx': '0', 'deaths_unknown_race_eth': '0', 'hospitalizations_age_0_17': '30', 'hospitalizations_age_18_29': '78', 'hospitalizations_age_30_39': '74', 'hospitalizations_age_40_49': '96', 'hospitalizations_age_50_59': '105', 'hospitalizations_age_60_69': '111', 'hospitalizations_age_70_79': '89', 'hospitalizations_age_80_': '71', 'hospitalizations_age_unknown': '0', 'hospitalizations_female': '310', 'hospitalizations_male': '341', 'hospitalizations_unknown_gender': '3', 'hospitalizations_latinx': '216', 'hospitalizations_asian_non_latinx': '48', 'hospitalizations_black_non_latinx': '208', 'hospitalizations_white_non_latinx': '135', 'hospitalizations_other_race_non_latinx': '31', 'hospitalizations_unknown_race_ethnicity': '16'}
The length of report_date is 591
You can use the dict get method to read the data from json response like below :-
report_date = []
for i in response.json():
if type(i) == dict: # Just check the type to avoid the runtime error.
ls = i.get('lab_report_date', None)
if ls:
report_date.append(ls)
hi i have a similar issue which is sometimes the response comes empty
from the api request which cause to me a stop in the Code Execution :
i found an easy solution for it now :
let's say you have a :
requestfromapi = requests.get("https://api-server")
if requestfromapi.json()['data']['something'] != KeyError:
print(requestfromapi.json()['data']['something'])
// this will make sure that your code will not stop from executing .
I asked this question once but was very inconsistent in my wording. Here is my full code. I have a dataArray and wish to add numbers within the 5th column but only if within the same row, column 7 has a 0.
#!/usr/bin/python
#Date: 4.24.18
#importing necessary modules
import csv
import collections
import sys
from array import array
#variables for ease of use in script
fileName = 'medicaldata.tsv'
filePath = '/home/pjvaglic/Desktop/scripts/pythonScripts/final/data/'
dataURL = 'http://pages.mtu.edu/~toarney/sat3310/final/'
dataArray = []
sumBeds = 0
count = 0
countFac = 0
sumNSal = 0
sumNSalR = 0
#download file from MTU
downloadFile = urllib2.urlopen(dataURL + fileName)
#opening the file
with open(filePath + fileName, 'w') as output:
output.write(downloadFile.read())
output.close()
#count number of lines in the data file, take off the header, print results to screen
count = open(filePath + fileName).readlines()
print "There are", len(count)-1, "facilities accounted for in", filePath + fileName
#keep track of number of facilities
countFac = len(count)-1
#open data file, put everything in an array, cut everything at the tab delimiter
with open(filePath + fileName, 'rt') as inputfile:
next(inputfile)
dataArray = csv.reader(inputfile, delimiter='\t')
#sum the amount of beds are in the first column
for row in dataArray:
sumBeds += int(row[0])
print "There are ", sumBeds, "in the medical file."
print "There are about", sumBeds/countFac, "beds per facility."
#this line does not work for my purposes.
#list = [[row[4] for row in dataArray if row[6] == '1']]
#print list
Here is the dataArray. The last column has 0's and 1's. I believe they are strings. For example, in the first row it has a 0, so I want to take 5230 and add that to 6304 and then 6590, so forth and so on. Just rows that include a 0 in the last column.
['244', '128', '385', '23521', '5230', '5334', '0']
['59', '155', '203', '9160', '2459', '493', '1']
['120', '281', '392', '21900', '6304', '6115', '0']
['120', '291', '419', '22354', '6590', '6346', '0']
['120', '238', '363', '17421', '5362', '6225', '0']
['65', '180', '234', '10531', '3622', '449', '1']
['120', '306', '372', '22147', '4406', '4998', '1']
['90', '214', '305', '14025', '4173', '966', '1']
['96', '155', '169', '8812', '1955', '1260', '0']
['120', '133', '188', '11729', '3224', '6442', '1']
['62', '148', '192', '8896', '2409', '1236', '0']
['120', '274', '426', '20987', '2066', '3360', '1']
['116', '154', '321', '17655', '5946', '4231', '0']
['59', '120', '164', '7085', '1925', '1280', '1']
['80', '261', '284', '13089', '4166', '1123', '1']
['120', '338', '375', '21453', '5257', '5206', '1']
['80', '77', '133', '7790', '1988', '4443', '1']
['100', '204', '318', '18309', '4156', '4585', '1']
['60', '97', '213', '8872', '1914', '1675', '1']
['110', '178', '280', '17881', '5173', '5686', '1']
['120', '232', '336', '17004', '4630', '907', '0']
['135', '316', '442', '23829', '7489', '3351', '0']
['59', '163', '191', '9424', '2051', '1756', '1']
['60', '96', '202', '12474', '3803', '2123', '0']
['25', '74', '83', '4078', '2008', '4531', '1']
['221', '514', '776', '36029', '1288', '2543', '1']
['64', '91', '214', '8782', '4729', '4446', '1']
['62', '146', '204', '8951', '2367', '1064', '0']
['108', '255', '366', '17446', '5933', '2987', '1']
['62', '144', '220', '6164', '2782', '411', '1']
['90', '151', '286', '2853', '4651', '4197', '0']
['146', '100', '375', '21334', '6857', '1198', '0']
['62', '174', '189', '8082', '2143', '1209', '1']
['30', '54', '88', '3948', '3025', '137', '1']
['79', '213', '278', '11649', '2905', '1279', '0']
['44', '127', '158', '7850', '1498', '1273', '1']
['120', '208', '423', '29035', '6236', '3524', '0']
['100', '255', '300', '17532', '3547', '2561', '1']
['49', '110', '177', '8197', '2810', '3874', '1']
['123', '208', '336', '22555', '6059', '6402', '1']
['82', '114', '136', '8459', '1995', '1911', '1']
['58', '166', '205', '10412', '2245', '1122', '1']
['110', '228', '323', '16661', '4029', '3893', '1']
['62', '183', '222', '12406', '2784', '2212', '1']
['86', '62', '200', '11312', '3720', '2959', '1']
['102', '326', '355', '14499', '3866', '3006', '1']
['135', '157', '471', '24274', '7485', '1344', '0']
['78', '154', '203', '9327', '3672', '1242', '1']
['83', '224', '390', '12362', '3995', '1484', '1']
['60', '48', '213', '10644', '2820', '1154', '0']
['54', '119', '144', '7556', '2088', '245', '1']
['120', '217', '327', '20182', '4432', '6274', '0']
I know there is a short hand way of placing all those numbers within a list and use a sum function to add them up. I'm just not sure of how to go about it.
There are 2 ways. Below I use only an extract of your data.
Setup
We assume you begin with a list of lists of strings.
lst = [['244', '128', '385', '23521', '5230', '5334', '0'],
['59', '155', '203', '9160', '2459', '493', '1'],
['120', '281', '392', '21900', '6304', '6115', '0'],
['120', '291', '419', '22354', '6590', '6346', '0'],
['120', '238', '363', '17421', '5362', '6225', '0'],
['65', '180', '234', '10531', '3622', '449', '1'],
['120', '306', '372', '22147', '4406', '4998', '1'],
['90', '214', '305', '14025', '4173', '966', '1'],
['96', '155', '169', '8812', '1955', '1260', '0']]
Pure Python
A = [[int(i) for i in row] for row in lst]
res = sum(row[4] for row in A if row[6] == 0)
# 25441
Vectorised solution
You can use a 3rd party library such as numpy:
import numpy as np
A = np.array(lst, dtype=int)
res = A[np.where(A[:, 6] == 0), 4].sum()
# 25441
Turn your data file into an array of arrays.
['244', '128', '385', '23521', '5230', '5334', '0']
['59', '155', '203', '9160', '2459', '493', '1']
['120', '281', '392', '21900', '6304', '6115', '0']
Instead:
[['244', '128', '385', '23521', '5230', '5334', '0'],
['59', '155', '203', '9160', '2459', '493', '1'],
['120', '281', '392', '21900', '6304', '6115', '0']]
Then iterate over the elements in the array of arrays looking for the string '0' then adding the element [i][4] to your total sum. You'll need to convert the strings to a number value to add them though, otherwise you'll get one long string of numbers instead of a sum.
var sum = 0;
for (i = 0; while i < dataArray.length; i ++) {
if (dataArray[i][7] === '0') {
var sum += Number(dataArray[i][4])
}
};
At the end of the loop you'll have your total in var sum and can do with it as you please.
Just realized your working in python, my answer is in javascript. Whoops. Might not be the best answer but if you find the python version of the above solution it should get you on the right track. Cheers
I am doing serial communication. I have two lists, first is a single dimension list dataList and the other is 2D list multiList. from serial communication i am sending '101','102','103',....'109'. The sent data is stored in dataList and that dataList is stored in multiList row by row like the first dataList
should be stored in the first row of multiList, the second dataList should be stored in the second row of multiList and moreover. Currently, the multilist output is like this:
[['1', '108', '109', '100', '101'], [], [], [], []]
[['102', '103', '104', '105', '106'], ['102', '103', '104', '105', '106'], [], [], []]
[['107', '108', '109', '100', '101'], ['107', '108', '109', '100', '101'], ['107', '108', '109', '100', '101'], [], []].
But I want my output like:
[['101', '102', '103', '104', '105'], [], [], [], []]
[['101', '102', '103', '104', '105'], ['106', '107', '108', '109', '101'], [], [], []]
[['101', '102', '103', '104', '105'], ['106', '107', '108', '109', '101'], ['102', '103', '104', '105', '106'], [], []]
import serial
row=5
col=3
multiList = ([[], [], [], [], []])
ser = serial.Serial(COM4, baudrate=115200, timeout=1)
numPoints = 5
dataList = [0] * numPoints
def getValues():
ser.write(b'g')
fpgadata = ser.readline().decode('ascii').split('\n')
return fpgadata[0]
def update():
for j in range(3):
for i in range(0, numPoints):
dataList[i] = getValues()
#print(dataList)
multiList[j] = dataList
print(multiList)
update()
The problem is that in the multiList[j] = dataList step you don't copy the elements, only the dataList copy. So in fact multiList contains multiple references to the same list. To copy the elements into a new list, write multiList[j] = dataList[:], like this:
def update():
for j in range(3):
for i in range(0, numPoints):
dataList[i] = getValues()
multiList[j] = dataList[:]
print(multiList)
This will have the desired effect of assigning to multiList[j] a new list,
with the elements of dataList copied.
I'm having difficulty with iterating through the nested list table below. I understand how to iterate through the table once, but to go a level deeper and iterate through each nested list, I am stuck on the correct syntax to use. In iterating through the sublists, I am trying to cast each 'age' and 'years experience' to an integer, perform the operation 'age' - 'years experience', and append the value (as a string) to each sublist.
table = [
['first_name', 'last_name', 'age', 'years experience', 'salary'],
['James', 'Butt', '29', '8', '887174.4'],
['Josephine', 'Darakjy', '59', '39', '1051267.9'],
['Art', 'Venere', '22', '2', '47104.2'],
['Lenna', 'Paprocki', '33', '7', '343240.2'],
['Donette', 'Foller', '26', '2', '273541.4'],
['Simona', 'Morasca', '35', '15', '960967.0'],
['Mitsue', 'Tollner', '51', '31', '162776.7'],
['Leota', 'Dilliard', '64', '39', '464595.5'],
['Sage', 'Wieser', '27', '9', '819519.7'],
['Kris', 'Marrier', '59', '33', '327505.55000000005'],
['Minna', 'Amigon', '45', '23', '571227.05'],
['Abel', 'Maclead', '46', '23', '247927.25'],
['Kiley', 'Caldarera', '33', '7', '179182.8'],
['Graciela', 'Ruta', '48', '21', '136978.95'],
['Cammy', 'Albares', '29', '9', '1016378.95'],
['Mattie', 'Poquette', '39', '15', '86458.75'],
['Meaghan', 'Garufi', '21', '3', '260256.5'],
['Gladys', 'Rim', '52', '26', '827390.5'],
['Yuki', 'Whobrey', '32', '10', '652737.0'],
['Fletcher', 'Flosi', '59', '37', '954975.15']]
##Exercise 3 (rows as lists): Iterate over each row and append the following values:
#If it is the first row then extend it with the following ['Started Working', 'Salary / Experience']
#Start work age (age - years experience)
#Salary / Experience ratio = (salary / divided by experience)
for i, v in enumerate(table):
extension = ['Started Working', 'Salary/Experience']
if i == 0:
v.extend(extension)
print(i,v) #test to print out the index and nested list values
#for index, value in enumerate(v):
# age =
#exp =
#start_work = age - exp
#print(index, value) test to print out the index and each value in the nested list
Pass the argument start to enumerate, enumerate(table, 1) in your case,
table = [['first_name', 'last_name', 'age', 'years experience', 'salary'],
['James', 'Butt', '29', '8', '887174.4'],
['Josephine', 'Darakjy', '59', '39', '1051267.9'],
['Art', 'Venere', '22', '2', '47104.2']]
table[0].extend(['Started Working', 'Salary/Experience'])
for idx, row in enumerate(table[1:], 1):
start_work_age = int(row[2]) - int(row[3])
ratio = float(row[4]) / int(row[3])
table[idx].extend([str(start_work_age), str(ratio)])
print(table)
# Output
[['first_name', 'last_name', 'age', 'years experience', 'salary', 'Started Working', 'Salary/Experience'],
['James', 'Butt', '29', '8', '887174.4', '21', '110896.8'],
['Josephine', 'Darakjy', '59', '39', '1051267.9', '20', '26955.5871795'],
['Art', 'Venere', '22', '2', '47104.2', '20', '23552.1']]
If you can convert the space to an underscore in years experience you can use collections.namedtuple to make your life simpler:
from collections import namedtuple
table = [
['first_name', 'last_name', 'age', 'years_experience', 'salary'],
['James', 'Butt', '29', '8', '887174.4'],
['Josephine', 'Darakjy', '59', '39', '1051267.9'],
['Art', 'Venere', '22', '2', '47104.2'],
# ...
]
workerv1 = namedtuple('workerv1', ','.join(table[0]))
for i,v in enumerate(table):
worker = workerv1(*v)
if i == 0:
swage = 'Started Working'
sex_ratio = 'S/Ex ratio'
else:
swage = int(worker.age) - int(worker.years_experience)
sex_ratio = float(worker.salary) / float(worker.years_experience)
print("{w.first_name},{w.last_name},{w.age},{w.years_experience},{w.salary},{0},{1}".format(
swage, sex_ratio, w=worker))