I want to get weather data in spotfire as a html data table. Spotfire does provide Ironpython support but not the flexibility to add our own modules. So being a shorthand here. Using iframe and src as api doesnt help in this case.
My Script:
import clr
clr.AddReference('System.Data')
clr.AddReference('System.Web.Extensions')
import System
from System import DateTime
from System.Data import DataSet, DataTable
from System.IO import StreamReader, StreamWriter, MemoryStream, SeekOrigin
from System.Net import HttpWebRequest
from System.Web.Script.Serialization import JavaScriptSerializer
from Spotfire.Dxp.Data import DataType, DataTableSaveSettings
from Spotfire.Dxp.Data.Import import TextFileDataSource, TextDataReaderSettings
#uri = "http://api.openweathermap.org/data/2.5/weather?q=London&appid=ec0313a918fa729d4372555ada5fb1f8"
uri = "http://api.openweathermap.org/data/2.5/group?id=524901,703448,2643743&units=metric&appid=ec0313a918fa729d4372555ada5fb1f8"
webRequest = HttpWebRequest.Create(uri)
response = webRequest.GetResponse()
streamReader = StreamReader(response.GetResponseStream())
jsonData = streamReader.ReadToEnd()
js = JavaScriptSerializer()
dataDict = js.Deserialize(jsonData, object)
print dataDict
# Close the connection
response.Close()
I need to extract some common values for multiple cities like: Name, Id, Description, temperature from the dictionary dataDict.
I am hitting a stone here.
It would be great if you can help me/ guide me on how to get these values from this complex dictionary.
Example dataDict: Dictionary[str, object]({'cnt' : 3, 'list' : Array[object]((Dictionary[str, object]({'coord' : Dictionary[str, object]({'lon' : <System.Decimal object at 0x0000000000000055 [37.62]>, 'lat' : <System.Decimal object at 0x0000000000000056 [55.75]>}), 'sys' : Dictionary[str, object]({'type' : 1, 'id' : 7323, 'message' : <System.Decimal object at 0x0000000000000057 [0.2107]>, 'country' : 'RU', 'sunrise' : 1484372967, 'sunset' : 1484400490}), 'weather' : Array[object]((Dictionary[str, object]({'id' : 802, 'main' : 'Clouds', 'description' : 'scattered clouds', 'icon' : '03d'}))), 'main' : Dictionary[str, object]({'temp' : <System.Decimal object at 0x0000000000000058 [-1.5]>, 'pressure' : 1009, 'humidity' : 80, 'temp_min' : -2, 'temp_max' : -1}), 'visibility' : 10000, 'wind' : Dictionary[str, object]({'speed' : 6, 'deg' : 160, 'gust' : 12}), 'clouds' : Dictionary[str, object]({'all' : 40}), 'dt' : 1484398800, 'id' : 524901, 'name' : 'Moscow'}), Dictionary[str, object]({'coord' : Dictionary[str, object]({'lon' : <System.Decimal object at 0x0000000000000059 [30.52]>, 'lat' : <System.Decimal object at 0x000000000000005A [50.43]>}), 'sys' : Dictionary[str, object]({'type' : 1, 'id' : 7358, 'message' : <System.Decimal object at 0x000000000000005B [0.1886]>, 'country' : 'UA', 'sunrise' : 1484373141, 'sunset' : 1484403724}), 'weather' : Array[object]((Dictionary[str, object]({'id' : 804, 'main' : 'Clouds', 'description' : 'overcast clouds', 'icon' : '04d'}))), 'main' : Dictionary[str, object]({'temp' : 3, 'pressure' : 998, 'humidity' : 86, 'temp_min' : 3, 'temp_max' : 3}), 'visibility' : 10000, 'wind' : Dictionary[str, object]({'speed' : 4, 'deg' : 170, 'var_beg' : 100, 'var_end' : 210}), 'clouds' : Dictionary[str, object]({'all' : 90}), 'dt' : 1484398800, 'id' : 703448, 'name' : 'Kiev'}), Dictionary[str, object]({'coord' : Dictionary[str, object]({'lon' : <System.Decimal object at 0x000000000000005C [-0.13]>, 'lat' : <System.Decimal object at 0x000000000000005D [51.51]>}), 'sys' : Dictionary[str, object]({'type' : 1, 'id' : 5091, 'message' : <System.Decimal object at 0x000000000000005E [0.1699]>, 'country' : 'GB', 'sunrise' : 1484380764, 'sunset' : 1484410813}), 'weather' : Array[object]((Dictionary[str, object]({'id' : 501, 'main' : 'Rain', 'description' : 'moderate rain', 'icon' : '10d'}))), 'main' : Dictionary[str, object]({'temp' : <System.Decimal object at 0x000000000000005F [4.01]>, 'pressure' : 1020, 'humidity' : 80, 'temp_min' : 2, 'temp_max' : 6}), 'visibility' : 10000, 'wind' : Dictionary[str, object]({'speed' : <System.Decimal object at 0x0000000000000060 [5.7]>, 'deg' : 290}), 'clouds' : Dictionary[str, object]({'all' : 20}), 'dt' : 1484400000, 'id' : 2643743, 'name' : 'London'})))})
Related
I've exported fitbit sleep data and got a json file with nested variables + dict. I would like to convert the json file to a csv file that will display all "regular" variables, e.g. "dateOfSleep" but also the nested variables, e.g. "deep" & "wake" with all dictionary information.
I tried json_normalize; but I can only make it work for the first nested variables, e.g. "levels". Anybody has an idea?
Much appreciated.
[{
"logId" : 32072056107,
"dateOfSleep" : "2021-05-08",
"startTime" : "2021-05-07T23:22:00.000",
"endTime" : "2021-05-08T08:05:30.000",
"duration" : 31380000,
"minutesToFallAsleep" : 0,
"minutesAsleep" : 468,
"minutesAwake" : 55,
"minutesAfterWakeup" : 0,
"timeInBed" : 523,
"efficiency" : 93,
"type" : "stages",
"infoCode" : 0,
"levels" : {
"summary" : {
"deep" : {
"count" : 5,
"minutes" : 85,
"thirtyDayAvgMinutes" : 68
},
"wake" : {
"count" : 30,
"minutes" : 55,
"thirtyDayAvgMinutes" : 56
},
"light" : {
"count" : 30,
"minutes" : 267,
"thirtyDayAvgMinutes" : 235
},
"rem" : {
"count" : 10,
"minutes" : 116,
"thirtyDayAvgMinutes" : 94
}
},
.....
Use pd.json_normalize, all nested levels are flatten and join with dots (by default):
import pandas as pd
import json
with open('data.json') as fp:
data = json.load(fp)
df = pd.json_normalize(data)
Output:
>>> df
logId dateOfSleep startTime ... levels.summary.rem.count levels.summary.rem.minutes levels.summary.rem.thirtyDayAvgMinutes
0 32072056107 2021-05-08 2021-05-07T23:22:00.000 ... 10 116 94
[1 rows x 25 columns]
Content of data.json:
[{
"logId" : 32072056107,
"dateOfSleep" : "2021-05-08",
"startTime" : "2021-05-07T23:22:00.000",
"endTime" : "2021-05-08T08:05:30.000",
"duration" : 31380000,
"minutesToFallAsleep" : 0,
"minutesAsleep" : 468,
"minutesAwake" : 55,
"minutesAfterWakeup" : 0,
"timeInBed" : 523,
"efficiency" : 93,
"type" : "stages",
"infoCode" : 0,
"levels" : {
"summary" : {
"deep" : {
"count" : 5,
"minutes" : 85,
"thirtyDayAvgMinutes" : 68
},
"wake" : {
"count" : 30,
"minutes" : 55,
"thirtyDayAvgMinutes" : 56
},
"light" : {
"count" : 30,
"minutes" : 267,
"thirtyDayAvgMinutes" : 235
},
"rem" : {
"count" : 10,
"minutes" : 116,
"thirtyDayAvgMinutes" : 94
}
}
}
}]
I have been working on an educational project a small part of it requires me to convert a single line of json data into an variable in python 3 which I recieve from domoticz (an external open source software) however due to my skill level with json I have expierenced some issues and I am not exactly sure what im doing wrong. I did get the 200 response everytime so I assume from what I understood that means the connection isnt the issue but rather the python code. (I censored the addressed but they are correct.)
The code im using:
import time
import re
import requests
from ctypes import c_int, c_char_p, byref, sizeof, c_uint16, c_int32, c_byte
from ctypes import c_void_p
from datetime import datetime
import os
import urllib.request
import json
import logging
import sys
from requests.exceptions import HTTPError
logger = logging.getLogger(__name__)
domoticzserver='ip'
switchid='3'
device='5'
tempbed=str(4)
def domoticzrequest (url):
request = urllib.request.Request(url)
response = urllib.request.urlopen(request)
return response.read()
import urllib.request, json
with urllib.request.urlopen("http://domoticzip/json.htm?type=devices&rid=4") as url:
data = json.loads(url.read().decode())
print(data)
The json I get back which i can see by typing clicking the url in python:
{
"ActTime" : 1606722346,
"AstrTwilightEnd" : "18:37",
"AstrTwilightStart" : "06:23",
"CivTwilightEnd" : "17:14",
"CivTwilightStart" : "07:47",
"DayLength" : "08:08",
"NautTwilightEnd" : "17:56",
"NautTwilightStart" : "07:04",
"ServerTime" : "2020-11-30 08:45:46",
"SunAtSouth" : "12:30",
"Sunrise" : "08:26",
"Sunset" : "16:34",
"app_version" : "2020.2",
"result" :
[
{
"AddjMulti" : 1.0,
"AddjMulti2" : 1.0,
"AddjValue" : 0.0,
"AddjValue2" : 0.0,
"BatteryLevel" : 255,
"CustomImage" : 0,
"Data" : "Normal",
"Description" : "",
"Favorite" : 0,
"HardwareID" : 1,
"HardwareName" : "Domoticz Internal",
"HardwareType" : "Domoticz Internal interface",
"HardwareTypeVal" : 67,
"HaveDimmer" : false,
"HaveGroupCmd" : false,
"HaveTimeout" : false,
"ID" : "148702",
"LastUpdate" : "2020-10-19 15:10:02",
"MaxDimLevel" : 0,
"Name" : "Domoticz Security Panel",
"Notifications" : "false",
"PlanID" : "0",
"PlanIDs" :
[
0
],
"Protected" : false,
"ShowNotifications" : true,
"SignalLevel" : "-",
"Status" : "Normal",
"StrParam1" : "",
"StrParam2" : "",
"SubType" : "Security Panel",
"SwitchType" : "Security",
"SwitchTypeVal" : 0,
"Timers" : "false",
"Type" : "Security",
"TypeImg" : "security",
"Unit" : 0,
"Used" : 0,
"XOffset" : "0",
"YOffset" : "0",
"idx" : "2"
},
{
"AddjMulti" : 1.0,
"AddjMulti2" : 1.0,
"AddjValue" : 0.0,
"AddjValue2" : 0.0,
"BatteryLevel" : 255,
"CustomImage" : 0,
"Data" : "-5.0 C",
"Description" : "",
"Favorite" : 1,
"HardwareID" : 2,
"HardwareName" : "Test sensor",
"HardwareType" : "Dummy (Does nothing, use for virtual switches only)",
"HardwareTypeVal" : 15,
"HaveTimeout" : true,
"ID" : "14053",
"LastUpdate" : "2020-11-09 09:03:34",
"Name" : "Temperatuur Kachel",
"Notifications" : "false",
"PlanID" : "0",
"PlanIDs" :
[
0
],
"Protected" : false,
"ShowNotifications" : true,
"SignalLevel" : "-",
"SubType" : "LaCrosse TX3",
"Temp" : -5.0,
"Timers" : "false",
"Type" : "Temp",
"TypeImg" : "temperature",
"Unit" : 1,
"Used" : 1,
"XOffset" : "0",
"YOffset" : "0",
"idx" : "3",
"trend" : 0
},
{
"AddjMulti" : 1.0,
"AddjMulti2" : 1.0,
"AddjValue" : 0.0,
"AddjValue2" : 0.0,
"BatteryLevel" : 255,
"CustomImage" : 0,
"Data" : "17.5",
"Description" : "",
"Favorite" : 1,
"HardwareID" : 3,
"HardwareName" : "Test switch",
"HardwareType" : "Dummy (Does nothing, use for virtual switches only)",
"HardwareTypeVal" : 15,
"HaveTimeout" : true,
"ID" : "0014054",
"LastUpdate" : "2020-11-06 11:51:09",
"Name" : "Temperatuur gewenst",
"Notifications" : "false",
"PlanID" : "0",
"PlanIDs" :
[
0
],
"Protected" : false,
"SetPoint" : "17.5",
"ShowNotifications" : true,
"SignalLevel" : "-",
"SubType" : "SetPoint",
"Timers" : "false",
"Type" : "Thermostat",
"TypeImg" : "override_mini",
"Unit" : 1,
"Used" : 1,
"XOffset" : "0",
"YOffset" : "0",
"idx" : "4"
}
],
"status" : "OK",
"title" : "Devices"
}
Basicly I want SetPoint(in the last tab of text also right above this) which is in this instance 17.5 as a variable in python and I will make the python code loop so it will grab that json url everytime to update the status of setpoint. But im having issues only grabbing the 17.5 to make it into a variable. I end up getting the entire json like this code is doing. or I will end up getting everything past and including the setpoint if I change some stuff. Does anyone know what im doing wrong and possibly where I should be looking for an solution? I am a bit unexpierenced with the json part of python and I have no clue where to get started as the code's I found and have tried seem to not work or give me errors.
Thank you very much for your time!
json.loads returns a python dictionary so maybe sth like this would do:
result = json['result']
set_point = 0.0
for res in result:
if 'SetPoint' in res:
set_point = res['SetPoint']
You are getting your data stored in data ={"key": argument} as a dictionary.
If you want to access a certain value you have to call for it. in Your case:
SetPoint = float(data["result"][-1]["SetPoint"])
To break it down:
data["result"] # gives you a list which elements are dictionaries.
the [-1] # calls for the last element in you list which contains the SetPoint
["SetPoint"] # then calls for the SetPoint Value which is a String "17.5"
float(...) converts the string to a float value
i am pulling out information from this websites API:
https://financialmodelingprep.com/
to be specific i need the data from the income statements:
https://financialmodelingprep.com/developer/docs/#Company-Financial-Statements
what i get back from the API is a list, which contains 36 dictionarys with the following Data:
[ {
"date" : "2019-09-28",
"symbol" : "AAPL",
"fillingDate" : "2019-10-31 00:00:00",
"acceptedDate" : "2019-10-30 18:12:36",
"period" : "FY",
"revenue" : 260174000000,
"costOfRevenue" : 161782000000,
"grossProfit" : 98392000000,
"grossProfitRatio" : 0.378178,
"researchAndDevelopmentExpenses" : 16217000000,
"generalAndAdministrativeExpenses" : 18245000000,
"sellingAndMarketingExpenses" : 0.0,
"otherExpenses" : 1807000000,
"operatingExpenses" : 34462000000,
"costAndExpenses" : 196244000000,
"interestExpense" : 3576000000,
"depreciationAndAmortization" : 12547000000,
"ebitda" : 81860000000,
"ebitdaratio" : 0.314636,
"operatingIncome" : 63930000000,
"operatingIncomeRatio" : 0.24572,
"totalOtherIncomeExpensesNet" : 422000000,
"incomeBeforeTax" : 65737000000,
"incomeBeforeTaxRatio" : 0.252666,
"incomeTaxExpense" : 10481000000,
"netIncome" : 55256000000,
"netIncomeRatio" : 0.212381,
"eps" : 2.97145,
"epsdiluted" : 2.97145,
"weightedAverageShsOut" : 18595652000,
"weightedAverageShsOutDil" : 18595652000,
"link" : "https://www.sec.gov/Archives/edgar/data/320193/000032019319000119/0000320193-19-000119-index.html",
"finalLink" : "https://www.sec.gov/Archives/edgar/data/320193/000032019319000119/a10-k20199282019.htm"
}, ...
]
What i dont need in the dictionary are the keys:
fillingDate, acceptedDate, link, finalLink
I managed to remove them, but my problem is that now that piece of code i wrote spits out those dictionaries way too often, and i am not able to understand why...
Here is what i tried:
import requests
import json
url = "https://financialmodelingprep.com/api/v3/income-statement/AAPL?apikey=b60bb3d1967bb15bfb9daaa4426e77dc"
response = requests.get(url)
data = response.text
dataList = json.loads(data)
entriesToRemove = {
'fillingDate' : 0,
'acceptedDate' : 0,
'link' : 0,
'finalLink' : 0
}
removedEntries = []
newDict = {}
for index in range(len(dataList)):
for key in dataList[index]:
newDict[key] = dataList[index].get(key)
if key in entriesToRemove:
removedEntries = newDict.pop(key)
print(json.dumps(newDict, indent=4))
Thanks in advance
OP:
for each key in the dictionary, the dictionary gets printed a new time.
Reason:
for index in range(len(dataList)):
for key in dataList[index]:
newDict[key] = dataList[index].get(key)
if key in entriesToRemove:
removedEntries = newDict.pop(key)
print(json.dumps(newDict, indent=4)) # notice this line
The reason why the dictionary is printed for each key is because you have a print(json.dumps(newDict, indent=4)) statement inside the loop for each key-val iteration over the dictionary.
To eradicate the highlighted keys from a list of dict, you could iterate over the list and create another list of dict without the unnecessary keys:
s = [ {
"date" : "2019-09-28",
"symbol" : "AAPL",
"fillingDate" : "2019-10-31 00:00:00",
"acceptedDate" : "2019-10-30 18:12:36",
"period" : "FY",
"revenue" : 260174000000,
"costOfRevenue" : 161782000000,
"grossProfit" : 98392000000,
"grossProfitRatio" : 0.378178,
"researchAndDevelopmentExpenses" : 16217000000,
"generalAndAdministrativeExpenses" : 18245000000,
"sellingAndMarketingExpenses" : 0.0,
"otherExpenses" : 1807000000,
"operatingExpenses" : 34462000000,
"costAndExpenses" : 196244000000,
"interestExpense" : 3576000000,
"depreciationAndAmortization" : 12547000000,
"ebitda" : 81860000000,
"ebitdaratio" : 0.314636,
"operatingIncome" : 63930000000,
"operatingIncomeRatio" : 0.24572,
"totalOtherIncomeExpensesNet" : 422000000,
"incomeBeforeTax" : 65737000000,
"incomeBeforeTaxRatio" : 0.252666,
"incomeTaxExpense" : 10481000000,
"netIncome" : 55256000000,
"netIncomeRatio" : 0.212381,
"eps" : 2.97145,
"epsdiluted" : 2.97145,
"weightedAverageShsOut" : 18595652000,
"weightedAverageShsOutDil" : 18595652000,
"link" : "https://www.sec.gov/Archives/edgar/data/320193/000032019319000119/0000320193-19-000119-index.html",
"finalLink" : "https://www.sec.gov/Archives/edgar/data/320193/000032019319000119/a10-k20199282019.htm"
}
]
res = []
ignored_keys = ['fillingDate', 'acceptedDate', 'link', 'finalLink']
for dd in s:
for k,v in dd.items():
if k not in ignored_keys:
res.append({k: v})
print(res)
EDIT:
one-liner:
print({k:v for dd in s for k,v in dd.items() if k not in ignored_keys})
I'm working on character stats for a battle system and I want to input the damage by having the system multiply one of the stats by 0.5, but it doesn't accept the stat name as a reference because it's in a dictionary.
Can someone tell me how to fix this?
Mage = { 'name' : 'Mage',
'lvl' : 0,
'xp' : 0,
'lvlNext' : 25,
'stats' : {'PStrength' : 5,
'PProsperity' : 5,
'PDexterity' : 15,
'PVitality' : 5,
'PAgility' : 10,
'HP' : 30}}
PDMG = PStrength * .50
gives
NameError: name 'PStrength' is not defined
What you have to do to get the damage is :
Mage = { 'name' : 'Mage',
'lvl' : 0,
'xp' : 0,
'lvlNext' : 25,
'stats' : {'PStrength' : 5,
'PProsperity' : 5,
'PDexterity' : 15,
'PVitality' : 5,
'PAgility' : 10,
'HP' : 30}}
PDMG = Mage['stats']['PStrength'] * 0.50
I am trying to Parse a Json array, A sample of the array i get is below with my code.
I can not seem to workout what my issue is, please forgive my question if I have included too much
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import json
map = "[{'network' : 'networkA','ycoord' : '73','zcoord' : '-2612','xcoord' : '-4461','owner' : 'PlayerA','name' : 'PlaceA'}, {'network' : 'NetworkB','ycoord' : '66','zcoord' : '-1915','xcoord' : '1156','owner' : 'PlayerB','name' : 'PlaceB'}, {'network' : 'NetWorkB','ycoord' : '71','zcoord' : '3091','xcoord' : '4541','owner' : 'PlayerB','name' : 'PlaceC'}, {'network' : 'NetworkB','ycoord' : '118','zcoord' : '-66','xcoord' : '5','owner' : 'PlayerB','name' : 'PlaceD'}, {'network' : 'networkA','ycoord' : '71','zcoord' : '761','xcoord' : '-248','owner' : 'PlayerA','name' : 'PlaceE'}]"
data = json.load(map)
for item in data:
print "Network : "+ str(item['network'])
print "Name : "+ str(item['name'])
print "Owner : "+ str(item['owner'])
print "Co ords : ("+ str(item['ycoord']+", "+ str(item['xcoord']+", "+ str(item['Zcoord']+")"
I get The error
File "test.py", line 8, in <module>
data = json.load(map)
File "/usr/lib/python2.7/json/__init__.py", line 274, in load
return loads(fp.read(),
AttributeError: 'str' object has no attribute 'read'
Readable Json Data (because I know what's in the code isn't)
[{
'network' : 'networkA',
'ycoord' : '73',
'zcoord' : '-2612',
'xcoord' : '-4461',
'owner' : 'PlayerA',
'name' : 'PlaceA'
}, {
'network' : 'NetworkB',
'ycoord' : '66',
'zcoord' : '-1915',
'xcoord' : '1156',
'owner' : 'PlayerB',
'name' : 'PlaceB'
}, {
'network' : 'NetWorkB',
'ycoord' : '71',
'zcoord' : '3091',
'xcoord' : '4541',
'owner' : 'PlayerB',
'name' : 'PlaceC'
}, {
'network' : 'NetworkB',
'ycoord' : '118',
'zcoord' : '-66',
'xcoord' : '5',
'owner' : 'PlayerB',
'name' : 'PlaceD'
}, {
'network' : 'networkA',
'ycoord' : '71',
'zcoord' : '761',
'xcoord' : '-248',
'owner' : 'PlayerA',
'name' : 'PlaceE'
}]
You want loads() instead of load(). Read the documentation, load() takes a filename, loads() takes actual JSON data.
json.load() function will require filename as a parameter. In your case, You don't want filename but an actual JSON array.
Use json.loads() instead of json.load()
Also, remember The functions with an s take string parameters. The others take file
streams. This applies to json.dump() and json.dumps() too.