We Keep Coding

Python list to list of map

ENV: python 3.7+, boto3
I have list of private IP address ['10.0.3.11', '10.0.2.22']
response = route53_client.change_resource_record_sets(
HostedZoneId="ABDCEFGH",
ChangeBatch={
'Comment': 'Dns to ec2 instance',
'Changes': [
{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': "ts-uat",
'Type': 'A',
'TTL': 120,
'ResourceRecords': [
{
'Value': record['Value']
},
]
}
}
]
}
)
How to get it as follows
HostedZoneId="ABDCEFGH",
ChangeBatch={
'Comment': 'Dns to ec2 instance',
'Changes': [
{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': "ts-uat",
'Type': 'A',
'TTL': 120,
'ResourceRecords': [
{
'Value': 10.0.3.11
},
{
'Value': 10.0.2.22
},
]
}
}
]
}
)
I tried the following
def getListofIP(n):
return "{'value' : %s,}" % (n)
result = map(getListofIP, private_ip)
print(list(result))
output was
["{'value' : 10.0.3.11,}", "{'value' : 10.0.2.22,}"]

Thanks a lot #WillRichardson. it works for me
def getListofIP(n):
return {'Value' : '{}'.format(n)}
result = map(getListofIP, private_ip)
print(list(result))
Output
[{'Value': '10.0.3.11'}, {'Value': '10.0.2.22'}]

How to extract values from list and store it as dictionary(key-value pair)?

I need to extract 2 values from this list of dictionary and store it as a key-value pair.
Here I attached sample data..Where I need to extract "Name" and "Service" from this input and store it as a dictionary. Where "Name" is Key and corresponding "Service" is its value.
Input:
response = {
'Roles': [
{
'Path': '/',
'Name': 'Heera',
'Age': '25',
'Policy': 'Policy1',
'Start_Month': 'January',
'PolicyDocument':
{
'Date': '2012-10-17',
'Statement': [
{
'id': '',
'RoleStatus': 'New_Joinee',
'RoleType': {
'Service': 'Service1'
},
'Action': ''
}
]
},
'Duration': 3600
},
{
'Path': '/',
'Name': 'Prem',
'Age': '40',
'Policy': 'Policy2',
'Start_Month': 'April',
'PolicyDocument':
{
'Date': '2018-11-27',
'Statement': [
{
'id': '',
'RoleStatus': 'Senior',
'RoleType': {
'Service': ''
},
'Action': ''
}
]
},
'Duration': 2600
},
]
}
From this input, I need output as a dictionary type.
Output Format: { Name : Service }
Output:
{ "Heera":"Service1","Prem" : " "}
My try:
Role_name =[]
response = {#INPUT WHICH I SPECIFIED ABOVE#}
roles = response['Roles']
for role in roles:
Role_name.append(role['Name'])
print(Role_name)
I need to pair the name with its corresponding service. Any help would be really appreciable.
Thanks in advance.

You just have to write a long line which can reach till the key 'Service'.
And you a syntax error in line Start_Month': 'January') and 'Start_Month': 'April'). You can't have one unclosed brackets.
Fix it and run the following.
This is the code:
output_dict = {}
for r in response['Roles']:
output_dict[r["Name"]] = r['PolicyDocument']['Statement'][0]['RoleType']['Service']
print(output_dict)
Output:
{'Heera': 'Service1', 'Prem': ''}

You just have to do like this:
liste = []
for role in response['Roles']:
liste.append(
{
role['Name']:role['PolicyDocument']['Statement'][0]['RoleType']['Service'],
}
)
print(liste)

It seems your input data is structured kind of strange and I am not sure what the ) are doing next to the months since they make things invalid but here is a working script assuming you removed the parenthesis from your input.
response = {
'Roles': [
{
'Path': '/',
'Name': 'Heera',
'Age': '25',
'Policy': 'Policy1',
'Start_Month': 'January',
'PolicyDocument':
{
'Date': '2012-10-17',
'Statement': [
{
'id': '',
'RoleStatus': 'New_Joinee',
'RoleType': {
'Service': 'Service1'
},
'Action': ''
}
]
},
'Duration': 3600
},
{
'Path': '/',
'Name': 'Prem',
'Age': '40',
'Policy': 'Policy2',
'Start_Month': 'April',
'PolicyDocument':
{
'Date': '2018-11-27',
'Statement': [
{
'id': '',
'RoleStatus': 'Senior',
'RoleType': {
'Service': ''
},
'Action': ''
}
]
},
'Duration': 2600
},
]
}
output = {}
for i in response['Roles']:
output[i['Name']] = i['PolicyDocument']['Statement'][0]['RoleType']['Service']
print(output)

This should give you what you want in a variable called role_services:
role_services = {}
for role in response['Roles']:
for st in role['PolicyDocument']['Statement']:
role_services[role['Name']] = st['RoleType']['Service']
It will ensure you'll go through all of the statements within that data structure but be aware you'll overwrite key-value pairs as you traverse the response, if they exist in more than a single entry!
A reference on for loops which might be helpful, illustrates using if statements within them which can help you to extend this to check if items already exist!
Hope that helps

Partial word search not working in elasticsearch (elasticsearch-py) using mongo-connector

Currently I've indexed my mongoDB collection into Elasticsearch running in a docker container. I am able to query a document by it's exact name, but Elasticsearch is unable to match the query if it is only part of the name. Here is an example:
>>> es = Elasticsearch('0.0.0.0:9200')
>>> es.indices.get_alias('*')
{'mongodb_meta': {'aliases': {}}, 'sigstore': {'aliases': {}}, 'my-index': {'aliases': {}}}
>>> x = es.search(index='sigstore', body={'query': {'match': {'name': 'KEGG_GLYCOLYSIS_GLUCONEOGENESIS'}}})
>>> x
{'took': 198, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 1, 'relation': 'eq'}, 'max_score': 8.062855, 'hits': [{'_index': 'sigstore', '_type': 'sigs', '_id': '5d66c23228144432307c2c49', '_score': 8.062855, '_source': {'id': 1, 'name': 'KEGG_GLYCOLYSIS_GLUCONEOGENESIS', 'description': 'http://www.broadinstitute.org/gsea/msigdb/cards/KEGG_GLYCOLYSIS_GLUCONEOGENESIS', 'members': ['ACSS2', 'GCK', 'PGK2', 'PGK1', 'PDHB', 'PDHA1', 'PDHA2', 'PGM2', 'TPI1', 'ACSS1', 'FBP1', 'ADH1B', 'HK2', 'ADH1C', 'HK1', 'HK3', 'ADH4', 'PGAM2', 'ADH5', 'PGAM1', 'ADH1A', 'ALDOC', 'ALDH7A1', 'LDHAL6B', 'PKLR', 'LDHAL6A', 'ENO1', 'PKM2', 'PFKP', 'BPGM', 'PCK2', 'PCK1', 'ALDH1B1', 'ALDH2', 'ALDH3A1', 'AKR1A1', 'FBP2', 'PFKM', 'PFKL', 'LDHC', 'GAPDH', 'ENO3', 'ENO2', 'PGAM4', 'ADH7', 'ADH6', 'LDHB', 'ALDH1A3', 'ALDH3B1', 'ALDH3B2', 'ALDH9A1', 'ALDH3A2', 'GALM', 'ALDOA', 'DLD', 'DLAT', 'ALDOB', 'G6PC2', 'LDHA', 'G6PC', 'PGM1', 'GPI'], 'user': 'naji.taleb#medimmune.com', 'type': 'public', 'level1': 'test', 'level2': 'test2', 'time': '08-28-2019 14:03:29 EDT-0400', 'source': 'File', 'mapped': [''], 'notmapped': [''], 'organism': 'human'}}]}}
When using the full name of the document, elasticsearch is able to successfully query it. But this is what happens when I attempt to search part of the name or use a wildcard:
>>> x = es.search(index='sigstore', body={'query': {'match': {'name': 'KEGG'}}})
>>> x
{'took': 17, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 0, 'relation': 'eq'}, 'max_score': None, 'hits': []}}
>>> x = es.search(index='sigstore', body={'query': {'match': {'name': 'KEGG*'}}})
>>> x
{'took': 3, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 0, 'relation': 'eq'}, 'max_score': None, 'hits': []}}
In addition to the default index settings I also tried making an index that allows the use of the nGram tokenizer to enable me to do partial search, but that also didn't work. These are the settings I used for that index:
{
"sigstore": {
"aliases": {},
"mappings": {},
"settings": {
"index": {
"max_ngram_diff": "99",
"number_of_shards": "1",
"provided_name": "sigstore",
"creation_date": "1579200699718",
"analysis": {
"filter": {
"substring": {
"type": "nGram",
"min_gram": "1",
"max_gram": "20"
}
},
"analyzer": {
"str_index_analyzer": {
"filter": [
"lowercase",
"substring"
],
"tokenizer": "keyword"
},
"str_search_analyzer": {
"filter": [
"lowercase"
],
"tokenizer": "keyword"
}
}
},
"number_of_replicas": "1",
"uuid": "3nf915U6T9maLdSiJozvGA",
"version": {
"created": "7050199"
}
}
}
}
}
and this is the corresponding python command that created it:
es.indices.create(index='sigstore',body={"mappings": {},"settings": { 'index': { "analysis": {"analyzer": {"str_search_analyzer": {"tokenizer": "keyword","filter": ["lowercase"]},"str_index_analyzer": {"tokenizer": "keyword","filter": ["lowercase", "substring"]}},"filter": {"substring": {"type": "nGram","min_gram": 1,"max_gram": 20}}}},'max_ngram_diff': '99'}})
I use mongo-connector as the pipeline between my mongoDB collection and elasticsearch. This is the command I use to start it:
mongo-connector -m mongodb://username:password#xx.xx.xxx.xx:27017/?authSource=admin -t elasticsearch:9200 -d elastic2_doc_manager -n sigstore.sigs
I'm unsure as to why my elasticsearch is unable to get a partial match, and wondering if there is some setting I'm missing or if there's some crucial mistake I've made somewhere. Thanks for reading.
Versions
MongoDB 4.0.10
elasticsearch==7.1.0
elastic2-doc-manager[elastic5]

Updated after checked your gist:
You need to apply the mapping to your field as written in the doc, cf the first link I share in the comment.
You need to do it after applying the settings on your index according to the gist it's line 11.
Something like:
PUT /your_index/_mapping
{
"properties": {
"name": {
"type": "keyword",
"ignore_above": 256,
"fields": {
"str_search_analyzer": {
"type": "text",
"analyzer": "str_search_analyzer"
}
}
}
}
}
After you set the mapping need to apply it to your document, using update_by_query
https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-update-by-query.html
So you can continue to search with term search on your field name as it will be indexed with a keyword mapping (exact match) and on the sub_field name.str_search_analyzer with part of the word.
your_keyword = 'KEGG_GLYCOLYSIS_GLUCONEOGENESIS' OR 'KEGG*'
x = es.search(index='sigstore', body={'query': {'bool': {'should':[{'term': {'name': your_keyword}},
{'match': {'name.str_search_analyzer': your_keyword}}
]}}
})

How to seperate dictionaries that are not comma separated?

I have a text file which contains dictionaries that are not comma sepearated in the following format:
{} {} {}
Example
{
'header': 'sdf',
'meta': {
'searchId': {
'searchId': 1234
},
'timestamp': 1234,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
}
{
'header': 'sdf',
'timestamp': 14,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
}
These dictionaries may contain nested dictionaries. I want to read this file and turn it into a list of dictionaries i.e. in the format [{},{},{}]
Example
[{
'header': 'sdf',
'meta': {
'searchId': {
'searchId': 1234
},
'timestamp': 1234,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
},
{
'header': 'sdf',
'timestamp': 14,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
}]
Can someone suggest a way to do it.
Thanks

My two other answers assume that the dicts in your data file are on separate lines so that each dict can be parsed as valid Python statements. If that is not the case, however, you can use lib2to3 and modify the Python grammar in Grammar.txt so that a simple statement (denoted by simple_stmt in the grammar file) does not have to end with a newline character:
from lib2to3 import fixer_base, refactor, pygram, pgen2
from io import StringIO
from functools import partialmethod
with open(pygram._GRAMMAR_FILE) as file:
grammar = StringIO(''.join(line.replace(' NEWLINE', '') if line.startswith('simple_stmt:') else line for line in file))
pgen2.pgen.ParserGenerator.__init__ = partialmethod(pgen2.pgen.ParserGenerator.__init__, stream=grammar)
pygram.python_grammar = pgen2.pgen.generate_grammar()
and look for atom nodes at the top level (whose parent node does not have a parent) instead:
class ScrapeAtoms(fixer_base.BaseFix):
PATTERN = "atom"
def __init__(self, *args):
super().__init__(*args)
self.nodes = []
def transform(self, node, results):
if not node.parent.parent:
self.nodes.append(node)
return node
class Refactor(refactor.RefactoringTool):
def get_fixers(self):
self.scraper = ScrapeAtoms(None, None)
return [self.scraper], []
def get_result(self):
return '[%s]\n' % ',\n'.join(str(node).rstrip() for node in self.scraper.nodes)
so that:
s = '''{'a': {1: 2}}{'b': 2}{
'c': 3
}{'d': 4}'''
refactor = Refactor(None)
refactor.refactor_string(s, '')
print(refactor.get_result())
outputs:
[{'a': {1: 2}},
{'b': 2},
{
'c': 3
},
{'d': 4}]
Demo: https://repl.it/#blhsing/CompleteStarchyFactorial

Like others have stated in the comments. This isn't json data. You merely have multiple string representations of dicts pretty printed to the file in succession, and you're also missing a closing bracket in the first one.
So I suggest looping through the file and build a string for each dict then you can use ast.literal_eval to parse the string into a dict. Something like this:
from ast import literal_eval
current = ''
data = []
with open('filename.txt') as f:
for line in f:
if line.startswith('{'):
current = line
elif line.startswith('}'):
data.append(literal_eval(current + line))
else:
current += line
Results in data (using pprint):
[{'header': 'sdf',
'meta': {'attachments': ['ABC'],
'searchId': {'searchId': 1234},
'timestamp': 1234,
'xmlData': {'release': None, 'version': None}}},
{'attachments': ['ABC'],
'header': 'sdf',
'timestamp': 14,
'xmlData': {'release': None, 'version': None}}]
After this you should overwrite the data, And never use this as serialization again. This is why there's libraries for this.

Since each dict in the file is a valid Python statement, a more robust solution would be to use the lib2to3 to parse the file as Python code and extract the statement nodes so that you can enclose them in square brackets, separated by commas:
from lib2to3 import fixer_base, refactor
class ScrapeStatements(fixer_base.BaseFix):
PATTERN = "simple_stmt"
def __init__(self, *args):
super().__init__(*args)
self.nodes = []
def transform(self, node, results):
self.nodes.append(node)
return node
class Refactor(refactor.RefactoringTool):
def get_fixers(self):
self.scraper = ScrapeStatements(None, None)
return [self.scraper], []
def get_result(self):
return '[%s]\n' % ',\n'.join(str(node).rstrip() for node in self.scraper.nodes)
so that:
s = '''{
'header': 'sdf',
'meta': {
'searchId': {
'searchId': 1234
},
'timestamp': 1234,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
}
}
{
'header': 'sdf',
'timestamp': 14,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
}
'''
refactor = Refactor(None)
refactor.refactor_string(s, '')
print(refactor.get_result())
outputs:
[{
'header': 'sdf',
'meta': {
'searchId': {
'searchId': 1234
},
'timestamp': 1234,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
}
},
{
'header': 'sdf',
'timestamp': 14,
'attachments': [
'ABC'
],
'xmlData': {
'release': None,
'version': None,
}
}]

If all the dicts in the file are on separate lines as they are in your sample input, then each dict by itself is a valid Python statement, so you can use ast.parse to parse the file into an abstract syntax tree, look for the expression nodes (of type Expr), and build a new Expression node with a List node to hold all the aforementioned Expr nodes. The new Expression node can then be compiled and evaluated as an actual Python list of dicts, so that given your sample input data in variable s:
import ast
tree = ast.parse(s)
exprs = [node.value for node in ast.walk(tree) if isinstance(node, ast.Expr)]
new = ast.Expression(body=ast.List(elts=exprs, ctx=ast.Load()))
ast.fix_missing_locations(new)
lst = eval(compile(new, '', 'eval'))
lst would become:
[{'header': 'sdf',
'meta': {'searchId': {'searchId': 1234},
'timestamp': 1234,
'attachments': ['ABC'],
'xmlData': {'release': None, 'version': None}}},
{'header': 'sdf',
'timestamp': 14,
'attachments': ['ABC'],
'xmlData': {'release': None, 'version': None}}]
Demo: https://repl.it/#blhsing/FocusedCylindricalTypes

Can't access dictionary data in boto3

I'm using boto3, and I ran this loop:
for i in x["Instances"]
print(i)
Then I get:
{
'AmiLaunchIndex': 0,
'Hypervisor': 'xen',
'VpcId': 'vpc-a790ac1',
'Architecture': 'x86_64',
'InstanceId': 'i-0bab3fb8314',
'PrivateDnsName': 'ip-10-c2.internal',
'BlockDeviceMappings': [{
'Ebs': {
'DeleteOnTermination': True,
'AttachTime': datetime.datetime(2017, 4, 4, 20, 44, 27, tzinfo = tzutc()),
'VolumeId': 'vol-07fd506f45',
'Status': 'attached'
},
'DeviceName': '/dev/xvda'
}, {
'Ebs': {
'DeleteOnTermination': False,
'AttachTime': datetime.datetime(2017, 4, 6, 1, 12, 45, tzinfo = tzutc()),
'VolumeId': 'vol-01ef36c45',
'Status': 'attached'
},
'DeviceName': '/dev/sdf'
}],
'RootDeviceName': '/dev/xvda',
'InstanceType': 't2.micro',
'EnaSupport': True,
'ClientToken': 'ODrMT1465413',
'EbsOptimized': False,
'SubnetId': 'subnet-fb1a4',
'Monitoring': {
'State': 'disabled'
},
'PublicDnsName': '',
'StateTransitionReason': 'User initiated (2017-04-06 01:15:22 GMT)',
'PrivateIpAddress': '10.10.4.116',
'RootDeviceType': 'ebs',
'Tags': [{
'Value': 'wp2',
'Key': 'Name'
}, {
'Value': 'true',
'Key': 'backup'
}],
'ImageId': 'ami-0976f01f',
'StateReason': {
'Code': 'Client.UserInitiadShutdown',
'Message': 'Client.UserInitiatedShutdown: User initiated shutdown'
},
'KeyName': 'pair2',
'ProductCodes': [],
'State': {
'Name': 'stopped',
'Code': 80
},
'LaunchTime': datetime.datetime(2017, 4, 6, 1, 13, 1, tzinfo = tzutc()),
'Placement': {
'AvailabilityZone': 'us-east-1b',
'GroupName': '',
'Tenancy': 'default'
},
'SourceDestCheck': True,
'NetworkInterfaces': [{
'Description': 'Primary network interface',
'PrivateIpAddress': '10.10.4.116',
'PrivateIpAddresses': [{
'Primary': True,
'PrivateIpAddress': '10.10.4.116'
}],
'Status': 'in-use',
'SubnetId': 'subnet-ffbcba4',
'VpcId': 'vpc-a790a7c1',
'Attachment': {
'DeleteOnTermination': True,
'AttachTime': datetime.datetime(2017, 4, 4, 20, 44, 26, tzinfo = tzutc()),
'DeviceIndex': 0,
'AttachmentId': 'eni-attach-c8398',
'Status': 'attached'
},
'Ipv6Addresses': [],
'OwnerId': '895548',
'MacAddress': '0e:31:4c4:b6',
'Groups': [{
'GroupId': 'sg-26c59',
'GroupName': 'web-dmz'
}],
'NetworkInterfaceId': 'eni-5383',
'SourceDestCheck': True
}],
'SecurityGroups': [{
'GroupId': 'sg-2cab59',
'GroupName': 'web-dmz'
}],
'VirtualizationType': 'hvm'
}
I'm trying to access the 'VolumeId' using something like:
for x in ["BlockDeviceMappings"][0]["Ebs"]["VolumeId"]:
print(x)
I get TypeError: string indices must be integers
It looks like 'BlockDeviceMappings' starts as a list with a dictionary in it, but I can't get to 'VolumeId'.
I've also tried:
for x in ["BlockDeviceMappings"][0]:
for k,v in ["Ebs"]:
print(v)
And I get:
ValueError: too many values to unpack (expected 2)
And I tried:
for x in ["BlockDeviceMappings"][0]:
for v in ["Ebs"]:
print(v)
Which prints 'Ebs' several times.
Could someone please point me in the right direction?

To get VolumeId please use
print x["Instances"][0]["BlockDeviceMappings"][0]["Ebs"]["VolumeId"]
you just missed x or _.
You are getting an error because ["BlockDeviceMappings"][0] evalutates to "B".
So you trying getting "orb" from "B"
To get all volumes:
for i in x["Instances"]:
for b in i["BlockDeviceMappings"]
print b["Ebs"]["VolumeId"]
If you have to extract data from complex sturctures like that often, try some quirky search library like github.com/akesterson/dpath-python , it can extract data just using keywords