What I'm trying to do isn't a huge problem in php, but I can't find much assistance for Python.
In simple terms, from a list which produces output as follows:
{"marketId":"1.130856098","totalAvailable":null,"isMarketDataDelayed":null,"lastMatchTime":null,"betDelay":0,"version":2576584033,"complete":true,"runnersVoidable":false,"totalMatched":null,"status":"OPEN","bspReconciled":false,"crossMatching":false,"inplay":false,"numberOfWinners":1,"numberOfRunners":10,"numberOfActiveRunners":8,"runners":[{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":2.8,"size":34.16},{"price":2.76,"size":200},{"price":2.5,"size":237.85}],"availableToLay":[{"price":2.94,"size":6.03},{"price":2.96,"size":10.82},{"price":3,"size":33.45}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832765}...
All I want to do is add in an extra field, containing the 'runner name' in the data set below, into each of the 'runners' sub lists from the initial data set, based on selection_id=selectionId.
So initially I iterate through the full dataset, and then create a separate list to get the runner name from the runner id (I should point out that runnerId===selectionId===selection_id, no idea why there are multiple names are used), this works fine and the code is shown below:
for market_book in market_books:
market_catalogues = trading.betting.list_market_catalogue(
market_projection=["RUNNER_DESCRIPTION", "RUNNER_METADATA", "COMPETITION", "EVENT", "EVENT_TYPE", "MARKET_DESCRIPTION", "MARKET_START_TIME"],
filter=betfairlightweight.filters.market_filter(
market_ids=[market_book.market_id],
),
max_results=100)
data = []
for market_catalogue in market_catalogues:
for runner in market_catalogue.runners:
data.append(
(runner.selection_id, runner.runner_name)
)
So as you can see I have the data in data[], but what I need to do is add it to the initial data set, based on the selection_id.
I'm more comfortable with Php or Javascript, so apologies if this seems a bit simplistic, but the code snippets I've found on-line only seem to assist with very simple Python lists and nothing 'nested' (to me the structure seems similar to a nested array).
As per the request below, here is the full list:
{"marketId":"1.130856098","totalAvailable":null,"isMarketDataDelayed":null,"lastMatchTime":null,"betDelay":0,"version":2576584033,"complete":true,"runnersVoidable":false,"totalMatched":null,"status":"OPEN","bspReconciled":false,"crossMatching":false,"inplay":false,"numberOfWinners":1,"numberOfRunners":10,"numberOfActiveRunners":8,"runners":[{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":2.8,"size":34.16},{"price":2.76,"size":200},{"price":2.5,"size":237.85}],"availableToLay":[{"price":2.94,"size":6.03},{"price":2.96,"size":10.82},{"price":3,"size":33.45}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832765},{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":20,"size":3},{"price":19.5,"size":26.36},{"price":19,"size":2}],"availableToLay":[{"price":21,"size":13},{"price":22,"size":2},{"price":23,"size":2}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832767},{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":11,"size":9.75},{"price":10.5,"size":3},{"price":10,"size":28.18}],"availableToLay":[{"price":11.5,"size":12},{"price":13.5,"size":2},{"price":14,"size":7.75}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832766},{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":48,"size":2},{"price":46,"size":5},{"price":42,"size":5}],"availableToLay":[{"price":60,"size":7},{"price":70,"size":5},{"price":75,"size":10}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832769},{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":18.5,"size":28.94},{"price":18,"size":5},{"price":17.5,"size":3}],"availableToLay":[{"price":21,"size":20},{"price":23,"size":2},{"price":24,"size":2}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832768},{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":4.3,"size":9},{"price":4.2,"size":257.98},{"price":4.1,"size":51.1}],"availableToLay":[{"price":4.4,"size":20.97},{"price":4.5,"size":30},{"price":4.6,"size":16}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832771},{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":24,"size":6.75},{"price":23,"size":2},{"price":22,"size":2}],"availableToLay":[{"price":26,"size":2},{"price":27,"size":2},{"price":28,"size":2}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":12832770},{"status":"ACTIVE","ex":{"tradedVolume":[],"availableToBack":[{"price":5.7,"size":149.33},{"price":5.5,"size":29.41},{"price":5.4,"size":5}],"availableToLay":[{"price":6,"size":85},{"price":6.6,"size":5},{"price":6.8,"size":5}]},"sp":{"nearPrice":null,"farPrice":null,"backStakeTaken":[],"layLiabilityTaken":[],"actualSP":null},"adjustmentFactor":null,"removalDate":null,"lastPriceTraded":null,"handicap":0,"totalMatched":null,"selectionId":10064909}],"publishTime":1551612312125,"priceLadderDefinition":{"type":"CLASSIC"},"keyLineDescription":null,"marketDefinition":{"bspMarket":false,"turnInPlayEnabled":false,"persistenceEnabled":false,"marketBaseRate":5,"eventId":"28180290","eventTypeId":"2378961","numberOfWinners":1,"bettingType":"ODDS","marketType":"NONSPORT","marketTime":"2019-03-29T00:00:00.000Z","suspendTime":"2019-03-29T00:00:00.000Z","bspReconciled":false,"complete":true,"inPlay":false,"crossMatching":false,"runnersVoidable":false,"numberOfActiveRunners":8,"betDelay":0,"status":"OPEN","runners":[{"status":"ACTIVE","sortPriority":1,"id":10064909},{"status":"ACTIVE","sortPriority":2,"id":12832765},{"status":"ACTIVE","sortPriority":3,"id":12832766},{"status":"ACTIVE","sortPriority":4,"id":12832767},{"status":"ACTIVE","sortPriority":5,"id":12832768},{"status":"ACTIVE","sortPriority":6,"id":12832770},{"status":"ACTIVE","sortPriority":7,"id":12832769},{"status":"ACTIVE","sortPriority":8,"id":12832771},{"status":"LOSER","sortPriority":9,"id":10317013},{"status":"LOSER","sortPriority":10,"id":10317010}],"regulators":["MR_INT"],"countryCode":"GB","discountAllowed":true,"timezone":"Europe\/London","openDate":"2019-03-29T00:00:00.000Z","version":2576584033,"priceLadderDefinition":{"type":"CLASSIC"}}}
i think i understand what you are trying to do now
first hold your data as a python object (you gave us a json object)
import json
my_data = json.loads(my_json_string)
for item in my_data['runners']:
item['selectionId'] = [item['selectionId'], my_name_here]
the thing is that my_data['runners'][i]['selectionId'] is a string, unless you want to concat the name and the id together, you should turn it into a list or even a dictionary
each item is a dicitonary so you can always also a new keys to it
item['new_key'] = my_value
So, essentially this works...with one exception...I can see from the print(...) in the loop that the attribute is updated, however what I can't seem to do is then see this update outside the loop.
mkt_runners = []
for market_catalogue in market_catalogues:
for r in market_catalogue.runners:
mkt_runners.append((r.selection_id, r.runner_name))
for market_book in market_books:
for runner in market_book.runners:
for x in mkt_runners:
if runner.selection_id in x:
setattr(runner, 'x', x[1])
print(market_book.market_id, runner.x, runner.selection_id)
print(market_book.json())
So the print(market_book.market_id.... displays as expected, but when I print the whole list it shows the un-updated version. I can't seem to find an obvious solution, which is odd, as it seems like a really simple thing (I tried messing around with indents, in case that was the problem, but it doesn't seem to be, its like its not refreshing the market_book list post update of the runners sub list)!
So the title sounds odd because perhaps my problem is odd... I have a .txt file with thousands of lines of machine output from a different program in the following format:
candidates(6,1,0,5,[ev(-1000,'C0009814','Stenosis','Acquired stenosis',[stenosis],[patf])])
Essentially we have 'candidates' marking the start of a tuple, and 'ev' marking the start of a second tuple inside of a single element list. When I read all of this into python from the file, it reads in as a string. But I need an object so I can access the nth index of the tuple. Truly, I would be happy just finding a way to consistently get the last value of the ev() tuple from this string, in this case 'patf'.
I had considered just splitting on ',' but this is not always successful because the list inside the list '[stenosis]', can sometimes have values like '[regurgitation, aortic]'. That extra ',' throws off the list index by 1 and therefore it return 'aortic]' instead of '[patf]'.
Please let me know if I can clarify anything or if I took some piece of knowledge for granted that needs to be said before this can be solved. Many thanks. I also included a second examples below that illustrates the problem of splitting on ','.
candidates(8,1,0,7,[ev(-875,'C0003501','Aortic Valve','Aortic valve structure',[aortic,valve],[bpoc])])
Edit: The object doesn't need to be a list, I guess. A tuple of the same format works well. Just as long as I can consistently reference one index for the info I need. Thanks!
Edit 2: I use python 2.7.6
You have a nested grammar that you're trying to parse. Albeit it is narrowly scoped, so regex could be constructed to deal with it, but it's going to be fragile. Like, really fragile.
Try using ast. This gets a little complex so I'll try to walk (haha) through an an example. If you want the tl;dr, skip to the middle/end.
We're looking for a name in a list node, so we can start there.
import ast
s = "candidates(6,1,0,5,[ev(-1000,'C0009814','Stenosis','Acquired stenosis',[stenosis],[patf])])"
mod = ast.parse(s)
for node in ast.walk(mod):
if isinstance(node, ast.List):
print(node, list(ast.iter_child_nodes(node)))
<_ast.List object at 0xb3f2ddec> [<_ast.Call object at 0xb3f2de0c>, <_ast.Load object at 0xb712756c>]
<_ast.List object at 0xb3f2deec> [<_ast.Name object at 0xb3f2df0c>, <_ast.Load object at 0xb712756c>]
<_ast.List object at 0xb3f2df2c> [<_ast.Name object at 0xb3f2df4c>, <_ast.Load object at 0xb712756c>]
We see that there are three ast.List nodes in our syntax tree. The first one is going to be the outer list that calls ev, and the two inner ones are going to contain those bare ast.Name nodes. That's what we want to get at - you specifically want the second one.
tl;dr skips here
We can make this all a lot more straightforward, I'm just walking through how I personally explored this syntax tree. Here's a one-ish-liner:
s = "candidates(6,1,0,5,[ev(-1000,'C0009814','Stenosis','Acquired stenosis',[stenosis],[patf])])"
mod = ast.parse(s)
[next(ast.iter_fields(node)) for node in ast.walk(mod) if isinstance(node, ast.Name)]
Out[62]: [('id', 'candidates'), ('id', 'ev'), ('id', 'stenosis'), ('id', 'patf')]
So just grab the second index of the last element of that, there's your string. This approach works for your other example too:
s = "candidates(8,1,0,7,[ev(-875,'C0003501','Aortic Valve','Aortic valve structure',[aortic,valve],[bpoc])])"
mod = ast.parse(s)
[next(ast.iter_fields(node)) for node in ast.walk(mod) if isinstance(node, ast.Name)]
Out[65]:
[('id', 'candidates'),
('id', 'ev'),
('id', 'aortic'),
('id', 'valve'),
('id', 'bpoc')]
You can use this approach to grab really any element you want out of that syntax tree. Just explore the output of ast.walk with ast.iter_fields and ast.iter_child_nodes.
If your data is always formatted the same way, the quickest way is to use regular expressions (module re), if you know how to.
Otherwise, and this is quite an unsightly hack, you can try to "parse" the data using eval. Here's an example:
eval_globals = {
"candidates": lambda *args: args,
"ev": lambda *args: args,
"aortic": "aortic",
"valve": "valve",
"bpoc": "bpoc",
# Add more of the keywords you need here
}
result = eval(line, eval_globals)
Just split on [. So you can do s.split('[')[-1].split(']')[0] where s is a line from the file ...
I use Wikipedia's API to get information about a page.
The API gives me JSON like this:
"query":{
"pages":{
"188791":{
"pageid":188791,
"ns":0,
"title":"Vanit\u00e9",
"langlinks":[
{
"lang":"bg",
"*":"Vanitas"
},
{
"lang":"ca",
"*":"Vanitas"
},
ETC.
}
}
}
}
You can see the full JSON response.
I want to obtain all entries like:
{
"lang":"ca",
"*":"Vanitas"
}
but the number key ("188791") in the pages object is the problem.
I found Find a value within nested json dictionary in python that explains me how to do enumerate the values.
Unfortunately I get the following exception:
TypeError: 'dict_values' object does not support indexing
My code is:
json["query"]["pages"].values()[0]["langlinks"]
It's probably a dumb question but I can't find a way to pass in the page id value.
One solution is to use the indexpageids parameter, e.g.: http://fr.wikipedia.org/w/api.php?action=query&titles=Vanit%C3%A9&prop=langlinks&lllimit=500&format=jsonfm&indexpageids. It will add an array of pageids to the response. You can then use that to access the dictionary.
As long as you're only querying one page at a time, Simeon Visser's answer will work. However, as a matter of good style, I'd recommend structuring your code so that you iterate over all the returned results, even if you know there should be only one:
for page in data["query"]["pages"].values():
title = page["title"]
langlinks = page["langlinks"]
# do something with langlinks...
In particular, by writing your code this way, if you ever find yourself needing to run the query for multiple pages, you can do it efficiently with a single MediaWiki API request.
You're using Python 3 and values() now returns a dict_values instead of a list. This is a view on the values of the dictionary.
Hence that's why you're getting that error because indexing fails. Indexing is possible in a list but not a view.
To fix it:
list(json["query"]["pages"].values())[0]["langlinks"]
If you really want just one page arbitrarily, do that the way Simeon Visser suggested.
But I suspect you want all langlinks in all pages, yes?
For that, you want a comprehension:
[page["langlinks"] for page in json["query"]["pages"].values()]
But of course that gives you a 2D list. If you want to iterate over each page's links, that's perfect. If you want to iterate over all of the langlinks at once, you want to flatten the list:
[langlink for page in json["query"]["pages"]
for langlink in page["langlinks"].values()]
… or…
itertools.chain.from_iterable(page["langlinks"]
for page in json["query"]["pages"].values())
(The latter gives you an iterator; if you need a list, wrap the whole thing in list. Conversely, for the first two, if you don't need a list, just any iterable, use parens instead of square brackets to get a generator expression.)