I am trying to find all the keys and their values matching a specific pattern using py-redis and lua and here is my code
rc = redis.Redis(..)
rc.set('google:',100)
rc.set('google:3',200)
rc.set('google:2',3400)
rc.set('google',200)
rc.set('fb',300)
get_script = """
local value = redis.call('GET', KEYS[1])
return value
"""
get_values = rc.register_script(get_script)
print get_values(rc.keys(pattern='google:*'))
print get_values(keys=['google:'])
print get_values(keys=['google:*'])
The output that I am getting is
100
100
None
First of all I do not get why I am getting None for the last print statement. My original purpose is to get all the keys ( and their values) matching the pattern but I am only getting the first key
I think that I have found what I was missing
Instead of GET I should have passed KEYS as first argument of the initial redis.call and then iterate over the keys to get the values
get_script = """
local keys = (redis.call('keys', ARGV[1]))
local values={}
for i,key in ipairs(keys) do
local val = redis.call('GET', key)
values[i]=val
i=i+1
end
return values
"""
get_values = rc.register_script(get_script)
print get_values(args=['google:*'])
Related
I am asking an ElasticSearch database to provide me with a list of indices and their creation dates using Python 2.7 and the Requests package. The idea is to quickly calculate which indices have exceeded the retention policy and need to be put to sleep.
The request works perfectly and the results are exactly what I want. However, when I run the code below, when I try to convert the json result to a dict, the type of theDict is correct but it reports a size of 1, when there should be at least a couple dozen entries. What am I doing wrong? I have a feeling it's something really dumb but I just can't snag it! :)
import json
import requests
esEndPoint = "https://localhost:9200"
retrieveString = "/_cat/indices?h=index,creation.date.string&format=json&s=creation.date"
# Gets the current indices and their creation dates
def retrieveIndicesAndDates():
try:
theResult = requests.get(esEndPoint+retrieveString)
print (theResult.content)
except Exception as e:
print("Unable to retrieve list of indices with creation dates.")
print("Error: "+e)
exit(3)
return theResult.content
def main():
theDict = dict(json.loads(retrieveIndicesAndDates()))
print(type(theDict)) # Reports correct type
print(len(theDict)) # Always outputs "1" ??
for index, creationdate in theDict.items():
print("Index: ",index,", Creation date: ",theDict[index])
return
The json the call returns:
[{"index":".kibana","creation.date.string":"2017-09-14T15:01:38.611Z"},{"index":"logstash-2018.07.23","creation.date.string":"2018-07-23T00:00:01.024Z"},{"index":"cwl-2018.07.23","creation.date.string":"2018-07-23T00:00:03.877Z"},{"index":"k8s-testing-internet-2018.07.23","creation.date.string":"2018-07-23T14:19:10.024Z"},{"index":"logstash-2018.07.24","creation.date.string":"2018-07-24T00:00:01.023Z"},{"index":"k8s-testing-internet-2018.07.24","creation.date.string":"2018-07-24T00:00:01.275Z"},{"index":"cwl-2018.07.24","creation.date.string":"2018-07-24T00:00:02.157Z"},{"index":"k8s-testing-internet-2018.07.25","creation.date.string":"2018-07-25T00:00:01.022Z"},{"index":"logstash-2018.07.25","creation.date.string":"2018-07-25T00:00:01.186Z"},{"index":"cwl-2018.07.25","creation.date.string":"2018-07-25T00:00:04.012Z"},{"index":"logstash-2018.07.26","creation.date.string":"2018-07-26T00:00:01.026Z"},{"index":"k8s-testing-internet-2018.07.26","creation.date.string":"2018-07-26T00:00:01.185Z"},{"index":"cwl-2018.07.26","creation.date.string":"2018-07-26T00:00:02.587Z"},{"index":"k8s-testing-internet-2018.07.27","creation.date.string":"2018-07-27T00:00:01.027Z"},{"index":"logstash-2018.07.27","creation.date.string":"2018-07-27T00:00:01.144Z"},{"index":"cwl-2018.07.27","creation.date.string":"2018-07-27T00:00:04.485Z"},{"index":"ctl-2018.07.27","creation.date.string":"2018-07-27T09:02:09.854Z"},{"index":"cfl-2018.07.27","creation.date.string":"2018-07-27T11:12:44.681Z"},{"index":"elb-2018.07.27","creation.date.string":"2018-07-27T11:13:51.340Z"},{"index":"cfl-2018.07.24","creation.date.string":"2018-07-27T11:45:23.697Z"},{"index":"cfl-2018.07.23","creation.date.string":"2018-07-27T11:45:24.646Z"},{"index":"cfl-2018.07.25","creation.date.string":"2018-07-27T11:45:25.700Z"},{"index":"cfl-2018.07.26","creation.date.string":"2018-07-27T11:45:26.341Z"},{"index":"elb-2018.07.24","creation.date.string":"2018-07-27T11:45:27.440Z"},{"index":"elb-2018.07.25","creation.date.string":"2018-07-27T11:45:29.572Z"},{"index":"elb-2018.07.26","creation.date.string":"2018-07-27T11:45:36.170Z"},{"index":"logstash-2018.07.28","creation.date.string":"2018-07-28T00:00:01.023Z"},{"index":"k8s-testing-internet-2018.07.28","creation.date.string":"2018-07-28T00:00:01.316Z"},{"index":"cwl-2018.07.28","creation.date.string":"2018-07-28T00:00:03.945Z"},{"index":"elb-2018.07.28","creation.date.string":"2018-07-28T00:00:53.992Z"},{"index":"ctl-2018.07.28","creation.date.string":"2018-07-28T00:07:19.543Z"},{"index":"k8s-testing-internet-2018.07.29","creation.date.string":"2018-07-29T00:00:01.026Z"},{"index":"logstash-2018.07.29","creation.date.string":"2018-07-29T00:00:01.378Z"},{"index":"cwl-2018.07.29","creation.date.string":"2018-07-29T00:00:04.100Z"},{"index":"elb-2018.07.29","creation.date.string":"2018-07-29T00:00:59.241Z"},{"index":"ctl-2018.07.29","creation.date.string":"2018-07-29T00:06:44.199Z"},{"index":"logstash-2018.07.30","creation.date.string":"2018-07-30T00:00:01.024Z"},{"index":"k8s-testing-internet-2018.07.30","creation.date.string":"2018-07-30T00:00:01.179Z"},{"index":"cwl-2018.07.30","creation.date.string":"2018-07-30T00:00:04.417Z"},{"index":"elb-2018.07.30","creation.date.string":"2018-07-30T00:01:01.442Z"},{"index":"ctl-2018.07.30","creation.date.string":"2018-07-30T00:08:28.936Z"},{"index":"cfl-2018.07.30","creation.date.string":"2018-07-30T06:52:16.739Z"}]
Your error is trying to convert a list of dicts to a dict:
theDict = dict(json.loads(retrieveIndicesAndDates()))
# ^^^^^ ^
That would only work for a dict of lists. It would be redundant, though.
Just use the reply directly. Each entry is a dict with the appropriate keys:
data = json.loads(retrieveIndicesAndDates())
for entry in data:
print("Index: ", entry["index"], ", Creation date: ", entry["creation.date.string"])
So what happens when you do convert that list to a dict? Why is there just one entry?
The dict understands three initialisation methods: keywords, mappings and iterables. A list fits the last one.
Initialisation from an iterable goes through it and expects key-value iterables as elements. If one were to do it manually, it would look like this:
def sequence2dict(sequence):
map = {}
for element in sequence:
key, value = element
map[key] = value
return map
Notice how each element is unpacked via iteration? In the reply each element is a dict with two entries. Iteration on that yields the two keys but ignores the values.
key, value = {"index":".kibana","creation.date.string":"2017-09-14T15:01:38.611Z"}
print(key, '=>', value) # prints "index => creation.date.string"
To the dict constructor, every element in the reply has the same key-value pair: "index" and "creation.date.string". Since keys in a dict are unique, all elements collapse to the same entry: {"index": "creation.date.string"}.
I got a nested dictionary as follows and it will return the third key while inputting the first key in the dictionary
tree = {"Luke" : {"Darth Vader" : {"The Chancellor"}},
"Neal" : {"Les" : {"Joseph"}},
"George" : {"Fred" : {"Mickey"}},
"Robert" : {"Tim" : {"Michael"}},
"Juan" : {"Hunter" : {"Thompson"}}}
check_con = input("Enter your Name")
for fi_name,fi_second in tree.items():
if check_con in fi_name:
for fi_third,fi_fourth in fi_second.items():
print(fi_fourth)
I feel that its bit more steps, is there any other way to do it?
Regard
You can use dict.get method with a default value of empty dict to get the top level dict and then convert its values to a iter, use next to get the first value
>>> check_con = 'Neal'
>>> next(iter(tree.get(check_con, {}).values()), '')
{'Joseph'}
>>>
>>> check_con = 'xxx'
>>> next(iter(tree.get(check_con, {}).values()), '')
''
>>>
You can simply use a try-excep expression in order to find out whether your name exist in the dictionary or not. If it exist there you can then return all the values of the respective value:
def get_nested_item(tree, check_on):
try:
sub_dict = tree[check_on]
except KeyError:
print("No result")
return
else:
return sub_dict.values()
Also note that about checking the existence of your name in dictionary what you're ding here is a membership checking at following line:
if check_con in fi_name:
Which will not check for equality but checks if check_con appears within the dictionary keys. However if this is what you want you have to loop over your items and find the intended one. But also note that this may have multiple answers or in other words there may be multiple keys matching your criteria which contradicts with the whole purpose of using a dictionary.
Demo:
In [11]: get_nested_item(tree, "George")
Out[11]: dict_values([{'Mickey'}])
In [12]: get_nested_item(tree, "Luke")
Out[12]: dict_values([{'The Chancellor'}])
In [13]: get_nested_item(tree, "Sarah")
No result
this is a variant where i use next(iter(...)) in order to get the 'first' element of your dict and set (note that you innermost curly brackets in your tree are sets and not dicts):
def get(tree, name):
def first(seq):
return next(iter(seq))
if name in tree:
return first(first(tree[name].values()))
else:
return None
print(get(tree=tree, name='Juan')) # 'Thompson'
print(get(tree=tree, name='Jan')) # None
as both sets and dict_values (which is the type dict(...).values() returns) are not indexable (have no __getitem__ method) i turn them into an iterator using iter and get the first element using next.
I have a variable output which can take the values #1,#2 ,
i am trying to get the value in "reasons" IF IT EXISTS,as you can see from #1 it does not exist always, can anyone suggest how this can be done?
output =
#1: {"took":42,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
#2: {"took":88,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"dispatcher","_type":"lookaheadDV","_id":"832238","_score":1.0, "_source" : {"reasons": ["12345 is associated with data in an invalid state"]}}]}}
OUTPUT:-
12345 is associated with data in an invalid state
Well, start with the logic:
hits inside hits may always be there... In additions hits looks to be a list, so you can test if hits (inside hits) is present and if it's not empty (you could test its length, for example).
Once you know that hits is there, you have to go inside that object and check if the value you're looking for is there.
When you have a dictionary in Python, you can retrieve values using the following syntax:
new_value = some_dictionary.get('some_key', None)
That value "None" at the end is the value Python gives me back if there is no value associated with that key. You can put whatever value you want in there and then check for it later on:
new_value = some_dictionary.get('some_key', 'BETTER ASK STACKOVERFLOW')
You have to just try stuff. Use the REPL.
Also, the following is not anything I've ever seen in Python. What gave you this error message:
12345 is associated with data in an invalid state
Ignore KeyErrors where necessary. There are possibly multiple reasons, so the following code gathers them all to a list, then join together using ', ' as the separator.
value = { "took":88,"timed_out":False,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"dispatcher","_type":"lookaheadDV","_id":"832238","_score":1.0, "_source" : {"reasons": ["12345 is associated with data in an invalid state"]}}]}}
reasons = []
for i in value['hits']['hits']:
try:
reasons.extend(i['_source']['reasons'])
except KeyError:
pass
reason = ', '.join(reasons)
This should do it:
adict = {"took":42,"timed_out":False,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":"null","hits":[]}}
bdict = {"took":88,"timed_out":False,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"dispatcher","_type":"lookaheadDV","_id":"832238","_score":1.0, "_source" : {"reasons": ["12345 is associated with data in an invalid state"]}}]}}
def get_reasons(some_dict):
output = ""
try:
output = some_dict.get("hits").get("hits")[0].get("_source").get("reasons")
except:
pass
return output
print get_reasons(adict)
print get_reasons(bdict)
prints nothing for the first dictionary and prints
12345 is associated with data in an invalid state
for the second dictionary.
PS: I changed false to False and null to "null" in your dictionaries.
I have a MapReduce program that returns the results of the computation as a dict that looks like the following:
{'pods': (54802L, 25417L, 59877L), 'sash': (160573L, 97199L, 178836L), ...}
I am trying to pull the items in this dictionary to make use of the values in the 3-tuples, but I keep receiving an error KeyError: 0 when trying to parse through the errors in the code shown below (specifically on this line key, indicies = results[i])
bloomFilter = [False] * 200000
for i in range(0, len(results)):
key, indicies = results[i]
bloomFilter[indicies[1]] = True
bloomFilter[indicies[2]] = True
bloomFilter[indicies[3]] = True
What is the proper way to pull the information from the dict so that I can make use of the values in the 3-tuples?
Also, can I use L values as indicies in a array, or do I need to cast them to ints?
I would iterate over the results dictionary like so:
for key in results:
indices = results[key]
bloomFilter[indices[0]] = True
bloomFilter[indices[1]] = True
bloomFilter[indices[2]] = True
Note also that the indices tuple is zero-indexed, meaning you'll need to extract the values starting with zero (not one) like the above code shows.
Here is my code:
for response in responses["result"]:
ids = {}
key = response['_id'].encode('ascii')
print key
for value in response['docs']:
ids[key].append(value)
Traceback:
File "people.py", line 47, in <module>
ids[key].append(value)
KeyError: 'deanna'
I am trying to add multiple values to a key. Throws an error like above
Check out setdefault:
ids.setdefault(key, []).append(value)
It looks to see if key is in ids, and if not, sets that to be an empty list. Then it returns that list for you to inline call append on.
Docs:
http://docs.python.org/2/library/stdtypes.html#dict.setdefault
If I'm reading this correctly your intention is to map the _id of a response to its docs. In that case you can bring down everything you have above to a dict comprehension:
ids = {response['_id'].encode('ascii'): response['docs']
for response in responses['result']}
This also assumes you meant to have id = {} outside of the outermost loop, but I can't see any other reasonable interpretation.
If the above is not correct,
You can use collections.defaultdict
import collections # at top level
#then in your loop:
ids = collections.defaultdict(list) #instead of ids = {}
A dictionary whose default value will be created by calling the init argument, in this case calling list() will produce an empty list which can then be appended to.
To traverse the dictionary you can iterate over it's items()
for key, val in ids.items():
print(key, val)
The reason you're getting a KeyError is this: In the first iteration of your for loop, you look up the key in an empty dictionary. There is no such key, hence the KeyError.
The code you gave will work, if you first insert an empty list into the dictionary under to appropriate key. Then append the values to the list. Like so:
for response in responses["result"]:
ids = {}
key = response['_id'].encode('ascii')
print key
if key not in ids: ## <-- if we haven't seen key yet
ids[key] = [] ## <-- insert an empty list into the dictionary
for value in response['docs']:
ids[key].append(value)
The previous answers are correct. Both defaultdict and dictionary.setdefault are automatic ways of inserting the empty list.