Related
I have a sufficiently large dataset that I would like to bulk index the JSON objects in AWS OpenSearch.
I cannot see how to achieve this using any of: boto3, awswrangler, opensearch-py, elasticsearch, elasticsearch-py.
Is there a way to do this without using a python request (PUT/POST) directly?
Note that this is not for: ElasticSearch, AWS ElasticSearch.
Many thanks!
I finally found a way to do it using opensearch-py, as follows.
First establish the client,
# First fetch credentials from environment defaults
# If you can get this far you probably know how to tailor them
# For your particular situation. Otherwise SO is a safe bet :)
import boto3
credentials = boto3.Session().get_credentials()
region='eu-west-2' # for example
auth = AWSV4SignerAuth(credentials, region)
# Now set up the AWS 'Signer'
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
auth = AWSV4SignerAuth(credentials, region)
# And finally the OpenSearch client
host=f"...{region}.es.amazonaws.com" # fill in your hostname (minus the https://) here
client = OpenSearch(
hosts = [{'host': host, 'port': 443}],
http_auth = auth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
Phew! Let's create the data now:
# Spot the deliberate mistake(s) :D
document1 = {
"title": "Moneyball",
"director": "Bennett Miller",
"year": "2011"
}
document2 = {
"title": "Apollo 13",
"director": "Richie Cunningham",
"year": "1994"
}
data = [document1, document2]
TIP! Create the index if you need to -
my_index = 'my_index'
try:
response = client.indices.create(my_index)
print('\nCreating index:')
print(response)
except Exception as e:
# If, for example, my_index already exists, do not much!
print(e)
This is where things go a bit nutty. I hadn't realised that every single bulk action needs an, er, action e.g. "index", "search" etc. - so let's define that now
action={
"index": {
"_index": my_index
}
}
You can read all about the bulk REST API, there.
The next quirk is that the OpenSearch bulk API requires Newline Delimited JSON (see https://www.ndjson.org), which is basically JSON serialized as strings and separated by newlines. Someone wrote on SO that this "bizarre" API looked like one designed by a data scientist - far from taking offence, I think that rocks. (I agree ndjson is weird though.)
Hideously, now let's build up the full JSON string, combining the data and actions. A helper fn is at hand!
def payload_constructor(data,action):
# "All my own work"
action_string = json.dumps(action) + "\n"
payload_string=""
for datum in data:
payload_string += action_string
this_line = json.dumps(datum) + "\n"
payload_string += this_line
return payload_string
OK so now we can finally invoke the bulk API. I suppose you could mix in all sorts of actions (out of scope here) - go for it!
response=client.bulk(body=payload_constructor(data,action),index=my_index)
That's probably the most boring punchline ever but there you have it.
You can also just get (geddit) .bulk() to just use index= and set the action to:
action={"index": {}}
Hey presto!
Now, choose your poison - the other solution looks crazily shorter and neater.
PS The well-hidden opensearch-py documentation on this are located here.
conn = wr.opensearch.connect(
host=self.hosts, # URL
port=443,
username=self.username,
password=self.password
)
def insert_index_data(data, index_name='stocks', delete_index_data=False):
""" Bulk Create
args: body [{doc1}{doc2}....]
"""
if delete_index_data:
index_name = 'symbol'
self.delete_es_index(index_name)
resp = wr.opensearch.index_documents(
self.conn,
documents=data,
index=index_name
)
print(resp)
return resp
I have used below code to bulk insert records from postgres into OpenSearch ( ES 7.2 )
import sqlalchemy as sa
from sqlalchemy import text
import pandas as pd
import numpy as np
from opensearchpy import OpenSearch
from opensearchpy.helpers import bulk
import json
engine = sa.create_engine('postgresql+psycopg2://postgres:postgres#127.0.0.1:5432/postgres')
host = 'search-xxxxxxxxxx.us-east-1.es.amazonaws.com'
port = 443
auth = ('username', 'password') # For testing only. Don't store credentials in code.
# Create the client with SSL/TLS enabled, but hostname verification disabled.
client = OpenSearch(
hosts = [{'host': host, 'port': port}],
http_compress = True,
http_auth = auth,
use_ssl = True,
verify_certs = True,
ssl_assert_hostname = False,
ssl_show_warn = False
)
with engine.connect() as connection:
result = connection.execute(text("select * from account_1_study_1.stg_pred where domain='LB'"))
records = []
for row in result:
record = dict(row)
record.update(record['item_dataset'])
del record['item_dataset']
records.append(record)
df = pd.DataFrame(records)
#df['Date'] = df['Date'].astype(str)
df = df.fillna("null")
print(df.keys)
documents = df.to_dict(orient='records')
#bulk(es ,documents, index='search-irl-poc-dump', raise_on_error=True)\
#response=client.bulk(body=documents,index='sample-index')
bulk(client, documents, index='search-irl-poc-dump', raise_on_error=True, refresh=True)
I'm trying to use the AssumeRole in such a way that i'm traversing multiple accounts and retrieving assets for those accounts. I've made it to this point:
import boto3
stsclient = boto3.client('sts')
assumedRoleObject = sts_client.assume_role(
RoleArn="arn:aws:iam::account-of-role-to-assume:role/name-of-role",
RoleSessionName="AssumeRoleSession1")
Great, i have the assumedRoleObject. But now i want to use that to list things like ELBs or something that isn't a built-in low level resource.
How does one go about doing that? If i may ask - please code out a full example, so that everyone can benefit.
Here's a code snippet from the official AWS documentation where an s3 resource is created for listing all s3 buckets. boto3 resources or clients for other services can be built in a similar fashion.
# create an STS client object that represents a live connection to the
# STS service
sts_client = boto3.client('sts')
# Call the assume_role method of the STSConnection object and pass the role
# ARN and a role session name.
assumed_role_object=sts_client.assume_role(
RoleArn="arn:aws:iam::account-of-role-to-assume:role/name-of-role",
RoleSessionName="AssumeRoleSession1"
)
# From the response that contains the assumed role, get the temporary
# credentials that can be used to make subsequent API calls
credentials=assumed_role_object['Credentials']
# Use the temporary credentials that AssumeRole returns to make a
# connection to Amazon S3
s3_resource=boto3.resource(
's3',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
)
# Use the Amazon S3 resource object that is now configured with the
# credentials to access your S3 buckets.
for bucket in s3_resource.buckets.all():
print(bucket.name)
To get a session with an assumed role:
import botocore
import boto3
import datetime
from dateutil.tz import tzlocal
assume_role_cache: dict = {}
def assumed_role_session(role_arn: str, base_session: botocore.session.Session = None):
base_session = base_session or boto3.session.Session()._session
fetcher = botocore.credentials.AssumeRoleCredentialFetcher(
client_creator = base_session.create_client,
source_credentials = base_session.get_credentials(),
role_arn = role_arn,
extra_args = {
# 'RoleSessionName': None # set this if you want something non-default
}
)
creds = botocore.credentials.DeferredRefreshableCredentials(
method = 'assume-role',
refresh_using = fetcher.fetch_credentials,
time_fetcher = lambda: datetime.datetime.now(tzlocal())
)
botocore_session = botocore.session.Session()
botocore_session._credentials = creds
return boto3.Session(botocore_session = botocore_session)
# usage:
session = assumed_role_session('arn:aws:iam::ACCOUNTID:role/ROLE_NAME')
ec2 = session.client('ec2') # ... etc.
The resulting session's credentials will be automatically refreshed when required which is quite nice.
Note: my previous answer was outright wrong but I can't delete it, so I've replaced it with a better and working answer.
You can assume role using STS token, like:
class Boto3STSService(object):
def __init__(self, arn):
sess = Session(aws_access_key_id=ARN_ACCESS_KEY,
aws_secret_access_key=ARN_SECRET_KEY)
sts_connection = sess.client('sts')
assume_role_object = sts_connection.assume_role(
RoleArn=arn, RoleSessionName=ARN_ROLE_SESSION_NAME,
DurationSeconds=3600)
self.credentials = assume_role_object['Credentials']
This will give you temporary access key and secret keys, with session token. With these temporary credentials, you can access any service. For Eg, if you want to access ELB, you can use the below code:
self.tmp_credentials = Boto3STSService(arn).credentials
def get_boto3_session(self):
tmp_access_key = self.tmp_credentials['AccessKeyId']
tmp_secret_key = self.tmp_credentials['SecretAccessKey']
security_token = self.tmp_credentials['SessionToken']
boto3_session = Session(
aws_access_key_id=tmp_access_key,
aws_secret_access_key=tmp_secret_key, aws_session_token=security_token
)
return boto3_session
def get_elb_boto3_connection(self, region):
sess = self.get_boto3_session()
elb_conn = sess.client(service_name='elb', region_name=region)
return elb_conn
with reference to the solution by #jarrad which is not working as of Feb 2021, and as a solution that does not use STS explicitly please see the following
import boto3
import botocore.session
from botocore.credentials import AssumeRoleCredentialFetcher, DeferredRefreshableCredentials
def get_boto3_session(assume_role_arn=None):
session = boto3.Session(aws_access_key_id="abc", aws_secret_access_key="def")
if not assume_role_arn:
return session
fetcher = AssumeRoleCredentialFetcher(
client_creator=_get_client_creator(session),
source_credentials=session.get_credentials(),
role_arn=assume_role_arn,
)
botocore_session = botocore.session.Session()
botocore_session._credentials = DeferredRefreshableCredentials(
method='assume-role',
refresh_using=fetcher.fetch_credentials
)
return boto3.Session(botocore_session=botocore_session)
def _get_client_creator(session):
def client_creator(service_name, **kwargs):
return session.client(service_name, **kwargs)
return client_creator
the function can be called as follows
ec2_client = get_boto3_session(role_arn='my_role_arn').client('ec2', region_name='us-east-1')
If you want a functional implementation, this is what I settled on:
def filter_none_values(kwargs: dict) -> dict:
"""Returns a new dictionary excluding items where value was None"""
return {k: v for k, v in kwargs.items() if v is not None}
def assume_session(
role_session_name: str,
role_arn: str,
duration_seconds: Optional[int] = None,
region_name: Optional[str] = None,
) -> boto3.Session:
"""
Returns a session with the given name and role.
If not specified, duration will be set by AWS, probably at 1 hour.
If not specified, region will be left unset.
Region can be overridden by each client or resource spawned from this session.
"""
assume_role_kwargs = filter_none_values(
{
"RoleSessionName": role_session_name,
"RoleArn": role_arn,
"DurationSeconds": duration_seconds,
}
)
credentials = boto3.client("sts").assume_role(**assume_role_kwargs)["Credentials"]
create_session_kwargs = filter_none_values(
{
"aws_access_key_id": credentials["AccessKeyId"],
"aws_secret_access_key": credentials["SecretAccessKey"],
"aws_session_token": credentials["SessionToken"],
"region_name": region_name,
}
)
return boto3.Session(**create_session_kwargs)
def main() -> None:
session = assume_session(
"MyCustomSessionName",
"arn:aws:iam::XXXXXXXXXXXX:role/TheRoleIWantToAssume",
region_name="us-east-1",
)
client = session.client(service_name="ec2")
print(client.describe_key_pairs())
import json
import boto3
roleARN = 'arn:aws:iam::account-of-role-to-assume:role/name-of-role'
client = boto3.client('sts')
response = client.assume_role(RoleArn=roleARN,
RoleSessionName='RoleSessionName',
DurationSeconds=900)
dynamodb_client = boto3.client('dynamodb', region_name='us-east-1',
aws_access_key_id=response['Credentials']['AccessKeyId'],
aws_secret_access_key=response['Credentials']['SecretAccessKey'],
aws_session_token = response['Credentials']['SessionToken'])
response = dynamodb_client.get_item(
Key={
'key1': {
'S': '1',
},
'key2': {
'S': '2',
},
},
TableName='TestTable')
print(response)
#!/usr/bin/env python3
import boto3
sts_client = boto3.client('sts')
assumed_role = sts_client.assume_role(RoleArn = "arn:aws:iam::123456789012:role/example_role",
RoleSessionName = "AssumeRoleSession1",
DurationSeconds = 1800)
session = boto3.Session(
aws_access_key_id = assumed_role['Credentials']['AccessKeyId'],
aws_secret_access_key = assumed_role['Credentials']['SecretAccessKey'],
aws_session_token = assumed_role['Credentials']['SessionToken'],
region_name = 'us-west-1'
)
# now we make use of the role to retrieve a parameter from SSM
client = session.client('ssm')
response = client.get_parameter(
Name = '/this/is/a/path/parameter',
WithDecryption = True
)
print(response)
Assuming that 1) the ~/.aws/config or ~/.aws/credentials file is populated with each of the roles that you wish to assume and that 2) the default role has AssumeRole defined in its IAM policy for each of those roles, then you can simply (in pseudo-code) do the following and not have to fuss with STS:
import boto3
# get all of the roles from the AWS config/credentials file using a config file parser
profiles = get_profiles()
for profile in profiles:
# this is only used to fetch the available regions
initial_session = boto3.Session(profile_name=profile)
# get the regions
regions = boto3.Session.get_available_regions('ec2')
# cycle through the regions, setting up session, resource and client objects
for region in regions:
boto3_session = boto3.Session(profile_name=profile, region_name=region)
boto3_resource = boto3_session.resource(service_name='s3', region_name=region)
boto3_client = boto3_session.client(service_name='s3', region_name=region)
[ do something interesting with your session/resource/client here ]
Credential Setup (boto3 - Shared Credentials File)
Assume Role Setup (AWS)
After a few days of searching, this is the simplest solution I have found. explained here but does not have a usage example.
import boto3
for profile in boto3.Session().available_profiles:
boto3.DEFAULT_SESSION = boto3.session.Session(profile_name=profile)
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket)
This will switch the default role you will be using. To not make the profile the default, just do not assign it to boto3.DEFAULT_SESSION. but instead, do the following.
testing_profile = boto3.session.Session(profile_name='mainTesting')
s3 = testing_profile.resource('s3')
for bucket in s3.buckets.all():
print(bucket)
Important to note that the .aws credentials need to be set in a specific way.
[default]
aws_access_key_id = default_access_id
aws_secret_access_key = default_access_key
[main]
aws_access_key_id = main_profile_access_id
aws_secret_access_key = main_profile_access_key
[mainTesting]
source_profile = main
role_arn = Testing role arn
mfa_serial = mfa_arn_for_main_role
[mainProduction]
source_profile = main
role_arn = Production role arn
mfa_serial = mfa_arn_for_main_role
I don't know why but the mfa_serial key has to be on the roles for this to work instead of the source account which would make more sense.
Here's the code snippet I used
sts_client = boto3.client('sts')
assumed_role_object = sts_client.assume_role(
RoleArn=<arn of the role to assume>,
RoleSessionName="<role session name>"
)
print(assumed_role_object)
credentials = assumed_role_object['Credentials']
session = Session(
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken']
)
self.s3 = session.client('s3')
I tried to connect to FTX using Python to get account information but it keeps giving me not logged in error; my code shown below:
from requests import Request, Session
import time
import hmac
method = 'GET'
ENDPOINT = 'https://ftx.com/api/'
# ENDPOINT = 'https://ftx.us/api/'
path = 'account'
API_KEY = '...'
API_SECRET = '...'
request = Request(method, ENDPOINT + path)
ts = int(time.time() * 1000)
prepared = request.prepare()
print('{0}{1}{2}'.format(ts, prepared.method, prepared.path_url))
signature_payload = '{0}{1}{2}'.format(ts, prepared.method, prepared.path_url).encode()
if prepared.body:
signature_payload += prepared.body
signature = hmac.new(API_SECRET.encode(), signature_payload, 'sha256').hexdigest()
request.headers['FTX-KEY'] = API_KEY
request.headers['FTX-SIGN'] = signature
request.headers['FTX-TS'] = str(ts)
# get data...
request1 = request.prepare()
print('{0}{1}{2}'.format(ts, request1.method, request1.path_url))
response = Session().send(request1)
data = response.json()
the data gives me:
{'success': False, 'error': 'Not logged in'}
I wonder where I did wrong? I locate in US is this might cause issue? and my account hasn't traded anything yet not sure if that's the issue.
fwiw, it's been a while and I haven't done much more work on this.. but I believe it's because of the location.. if the account is registered under FTX international this should work...
Please change headers to the following and url to ftx.us/api
request.headers['FTXUS-KEY'] = API_KEY
request.headers['FTXUS-SIGN'] = signature
request.headers['FTXUS-TS'] = str(ts)
I am trying to get the UserId after the creation of user, suing "account_id = boto3.client('sts').get_caller_identity().get('some_other_user')", but for output I get none, what could be the reason. I am very new to boto and python so it might be something very small.
import boto3
import sys
import json
iam = boto3.resource('iam') #resource representing IAM
group = iam.Group('group1') # Name of group
created_user = iam.create_user(
UserName='some_other_user'
)
account_id = boto3.client('sts').get_caller_identity().get('some_other_user')
print(account_id)
create_group_response = iam.create_group(GroupName = 'group1')
response = group.add_user(
UserName='some_other_user' #name of user
)
group = iam.Group('group1')
response = group.attach_policy(
PolicyArn='arn:aws:iam::196687784:policy/boto-test'
)
The get_caller_identity() function returns a dict containing:
{
'UserId': 'AIDAEXAMPLEHERE',
'Account': '123456789012',
'Arn': 'arn:aws:iam::123456789012:user/james'
}
So, to use this:
import boto3
sts = boto3.client('sts')
response = sts.get_caller_identity()
print('User ID:', response['UserId'])
Or you can use response.get('UserId') to get the user ID. The key to the user ID in the response dictionary is always the literal UserId. It doesn't vary (you cannot call response.get('james'), for example).
You cannot retrieve the identity of an arbitrary IAM principal using sts.get_caller_identity(). It only gives you the identity associated with the credentials that you implicitly used when making the call.
I have a soap 1.1/1.2 web service I'm trying to access using suds.
Unfortunately the service puts authentication token in response soap header.
Is it possible to access the header somehow?
I know one can set a custom soap header in the request, but that's not what I'm looking for.
I've been using the (still maintained) suds-jurko branch, and ran into trouble because client.last_received() was removed early after it was forked. So I had to figure out an alternative way to access the headers.
Fortunately, you can use a message plugin to store the parsed document, and then later access the headers via the plugin. For added convenience, instead of working with raw values from the xml document, you can process the headers based on the service-method, to get a correctly typed/structured value.
from suds.plugin import MessagePlugin
class HeaderPlugin(MessagePlugin):
def __init__(self):
self.document = None
def parsed(self, context):
self.document = context.reply
def get_headers(self, method):
method = method.method
binding = method.binding.output
rtypes = binding.headpart_types(method, False)
envns = ('SOAP-ENV', 'http://schemas.xmlsoap.org/soap/envelope/')
soapenv = self.document.getChild('Envelope', envns)
soapheaders = soapenv.getChild('Header', envns)
nodes = soapheaders.children
if len(nodes):
resolved = rtypes[0].resolve(nobuiltin=True)
return binding.unmarshaller().process(nodes[0], resolved)
return None
usage:
from suds.client import Client
hp = HeaderPlugin()
client = Client(wsdl, plugins=[hp])
response = client.service.LoremIpsum()
headers = hp.get_headers(client.service.LoremIpsum)
example output:
>>> headers
(AuthenticationResponseHeader){
sessionKey = "a631cd00-c6be-416f-9bd3-dbcd322e0848"
validUntil = 2030-01-01 01:01:01.123456+01:00
}
>>> headers.validUntil
datetime.datetime(2030, 1, 1, 1, 1, 1, 123456, tzinfo=<suds.sax.date.FixedOffsetTimezone object at 0x7f7347856be0>)
You can do something like
print client.last_received().getChild("soap:Envelope").getChild("soap:Header")
.getChild("ResponseHeader").getChild("resultCode").getText()
The above reads a field resultCode in the soap header. You have to do this for each field. This was a back door left to read headers as much as i know.
For details look at soap headers with suds
The Towr's class plugin works well, but it has a problem when do you have more then one obj in Soapheader Response.
His code get only the first object.
Here is the code to improvement the Towr's class:
class HeaderPlugin(MessagePlugin):
def __init__(self):
self.document = None
def parsed(self, context):
self.document = context.reply
def get_headers(self, method):
Result = {}
method = method.method
binding = method.binding.output
SHeaderElem = binding.headpart_types(method, False)
envns = ('SOAP-ENV', 'http://schemas.xmlsoap.org/soap/envelope/')
soapenv = self.document.getChild('Envelope', envns)
soapheaders = soapenv.getChild('Header', envns)
SHeaderNodes = soapheaders.children
for Elem in SHeaderElem:
for Node in SHeaderNodes:
if(Node.name == Elem.name):
ElemRes = Elem.resolve(nobuiltin=True)
NodeRes = binding.unmarshaller().process(Node, ElemRes)
Result[Elem.name] = NodeRes
return Result
#
To understand better, see the example.
If do you receive this Soap Response:
<soap-env:Envelope xmlns:eb="http://www.ebxml.org/namespaces/messageHeader" xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext">
<soap-env:Header>
<eb:MessageHeader eb:version="1.0" soap-env:mustUnderstand="1">
<!-- -->
</eb:MessageHeader>
<wsse:Security>
<!-- -->
</wsse:Security>
</soap-env:Header>
<soap-env:Body>
<!-- -->
</soap-env:Body>
</soap-env:Envelope>
The function get_headers will return a dict like this:
SoapHeadersResp = {'MessageHeader':MessageHeaderObj, 'Security':SecurityObj}
To use this class just follow the same steps that Towr said, replacing his HeaderPlugin class with this one.