I am looping through a WebService in Python which returns me partially data. I am calling the WebService until i receive an "End of Data" in the response.
The returned objects are always the same structure. They are only part of a large data which the WebService returns in chunks of 1000.
I am saving the returned JSON string in a data variable. How can i copy JSON array data_next to JSON array in variable `data' or merge both JSON files.
Let say you receive something like resp1 = '{"data": [...]}' and store it in a dictionary d. Assuming you do that using json package as follows:
d = json.loads(resp1)
For the next batch of data, you should store it in a temporal dictionary td, extract the field "data_next" and append it into the original dictionary:
td = json.loads(respN)
d["data"].append(td["data_next"])
Let us consider two list A =[{"a":"apple"},{"b":"banana"}]
and list B = [{"c":"cat"}]
try below in python 3 to get merged array.
A.__add__(B)
Related
I have a data variable which contains a list of dicts of lists that i have fetched from an api, formatted as bytes. The list is variable in length.
In: data
Out: b'[{"dict1_list1":[1,2,3],"dict1_list2":[4,5,6]},{"dict2_list1":[1,2,3],"dict2_list2":[4,5,6]}]'
I want to identify when there is only one dict in the list (i.e. the api has run out of data)
In: data
Out: b'[{"onlydict_list1":[1,2,3],"onlydict_list2":[4,5,6]}]'
but because data is formatted as byte I cannot return the length of the list.
In: len(data)
Out: 53 # Not '1' as I would like
How can I identify the length of the list within the b'[....]'?
If r is response then try calling r.json() method . It will convert to a list and then you can find the length of it . You can also use json.loads(r.text) by importing json(import json).
You're getting the length of the bytes object, which is apparently the type of data, not a Python version of the JSON object to which it corresponds.
A simple way to do it would be to first convert the data into the Python object equivalent of the byte-string using json.loads(). Here's what I mean:
import json
data = (b'[{"dict1_list1":[1,2,3],"dict1_list2":[4,5,6]},'
b'{"dict2_list1":[1,2,3],"dict2_list2":[4,5,6]}]')
print(len(json.loads(data))) # -> 2
I try to use alphavantage API in my project. At the moment I am going to parse the JSON data in this way:
from alpha_vantage.timeseries import TimeSeries
def AlphaVantage(symbol):
ts = TimeSeries(key='mykey')
data = ts.get_intraday(symbol, interval='1min')
print(str(data))
AlphaVantage('MSFT')
I would like to get only the most current data.
The data returned is a tuple. There are two dictionaries inside the tuple. The first one contains the time and high/low values and the second contains the metadata. So, you can access the time:high/low values by data[0]. As you know that the keys in a dictionary are unsorted, so to only get the most recent data, you can use max of keys of this dictionary. So, the final code would be like this:
print(str(data[0][max(data[0].keys())]))
If the API does not provide it, you can just grab the first one as the data is obviously sorted by date:
print(str(data['Time Series (1min)'][0]))
Hope it helps!
I have a .json file with some information
{"items": [{"phone": "testp"}, {"phone": "Test2"}]}
I want to add another dictionary into the next available slot in the array with some more information. I have tried many ways however no of them work, has anyone got any ideas?
dictionary["items"].append(new_dictionary)
You access key "items" in your dictionary, which is a list of dictionaries, to that list you simply append your new dictionary
You can read the json into a variable using load function.
And then append the value of array using
myVar["items"].append(dictVar)
I have a Matlab (.mat, version >7.3) file that contains a structure (data) that itself contains many fields. Each field is a single column array. Each field represents an individual sensor and the array is the time series data. I am trying to open this file in Python to do some more analysis. I am using PyTables to read the data in:
import tables
impdat = tables.openFile('data_file.mat')
This reads the file in and I can enter the fileObject and get the names of each field by using:
impdat.root.data.__members__
This prints a list of the fields:
['rdg', 'freqlabels', 'freqbinsctr',... ]
Now, what I would like is a method to take each field in data and make a python variable (perhaps dictionary) with the field name as the key (if it is a dictionary) and the corresponding array as its value. I can see the size of the array by doing, for example:
impdat.root.data.rdg
which returns this:
/data/rdg (EArray(1, 1286920), zlib(3))
atom := Int32Atom(shape=(), dflt=0)
maindim := 0
flavor := 'numpy'
byteorder := 'little'
chunkshape := (1, 16290)
My question is how do I access some of the data stored in that large array (1, 1286920). How can I read that array into another Python variable (list, dictionary, numpy array, etc.)? Any thoughts or guidance would be appreciated.
I have come up with a working solution. It is not very elegant as it requires an eval. So I first create a new variable (alldata) to the data I want to access, and then I create an empty dictionary datastruct, then I loop over all the members of data and assign the arrays to the appropriate key in the dictionary:
alldata = impdat.root.data
datastruct = {}
for names in impdat.rood.data.__members___:
datastruct[names] = eval('alldata.' + names + '[0][:]')
The '[0]' could be superfluous depending on the structure of the data trying to access. In my case the data is stored in an array of an array and I just want the first one. If you come up with a better solution please feel free to share.
I can't seem to replicate your code. I get an error when trying to open the file which I made in 8.0 using tables.
How about if you took the variables within the structure and saved them to a new mat file which only contains a collection of variables. This would make it much easier to deal with and this has already been answered quite eloquently here.
Which states that mat files which are arrays are simply hdf5 files which can be read with:
import numpy as np, h5py
f = h5py.File('somefile.mat','r')
data = f.get('data/variable1')
data = np.array(data) # For converting to numpy array
Not sure the size of the data set you're working with. If it's large I'm sure I could come up with a script to pull the fields out of the structures. I did find this tool which may be helpful. It recursively gets all of the structure field names.
I'm calling a method called getNewReports() of a webservice. This method takes in a username and password and outputs xml structured data.
How can I assign this xml output to a variable in python. If I just do:
reportxml = client.service.GetNewReports(username=user,password=pw)
print reportxml
It's printing this:
(ArrayOfString){
string[] = abcd,efgh
}
I need the information in the string array. How can I create a separate array of values using the data from the String[] array ? Or how can assign an xml type variable to the output so I can parse the whole thing as an xml file?
Thanks for your help!
The datatype you recieve from calling GetNewReports is an ArrayOfString. The problem is, I don't know what methods this type supports.
Call help(reportxml) or print dir(reportxml) to see a list of it's methods and attributes or take a look into the client.service documentation (if any) to get information about this type and watch out for a way to extract the information it contains.
Hint: Actually in every languange, an array-like datatype supports indexing using [index].