We Keep Coding

The IAM authentication failed for the role postgres. Check the IAM token for this role and try again

I'm facing a hard time connecting Python Lambdas to RDS proxy.
I have rest api that has a few Javascript and python lambdas and I manage and deploy everything using CDK.
I made sure that the lambdas can connect to the RDS proxy and handled all the roles and permissions.
In both Javascript and Python I generate an auth token to be used as a password for IAM authentication with the RDS proxy.
The problem is that Python Lambdas always throw this error:
The IAM authentication failed for the role postgres. Check the IAM
token for this role and try again.
while Javascript doesn't and connect to the proxy.
I'm using psycopg2 with sqlalchemy in Python and the following is how I create the db engine.
session = boto3.Session(
aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'), aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY'),
region_name=region,
)
client = session.client('rds')
token = client.generate_db_auth_token(host, port, user_name, region)
_engine = create_engine('postgresql://{user}:{password}#{host}:{port}/{db}'.format(
user=user_name,
host=host,
port=port,
db=db_name,
password=token
), connect_args={'sslmode': 'require'},)
The role attached to the lambda:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "rds-db:connect",
"Resource": "arn:aws:rds-db:us-east-1:xxxxxxxx:dbuser:prx-xxxxxxxxxx/postgres"
}
]
}
Can anybody tell me what I'm missing here?

I had a very similar issue. Assuming the lambda function can assume the role and all the other IAM policies are in place, here are a few things to check (in no particular order).
1 - SQLModel vs SQLAlchemy
I was able to use sqlalchemy to create_engine(url) where the token was used a the password in the url string. However, the same url did not work when using SQLModel create_engine(url).I had to to:
def get_connection():
token = boto3.generate_db_token(DBHostname=host, DBUsername=dbuser, Region='us-east-1', Port=5432)
return psycopg2.connect(host=host, port=5432, database=dbname, user=dbuser, password=token)
and then create the database engine with:
url = f'postgresql://{dbuser}#{host}:5432/{dbname}?sslmode=require'
database = sqlmodel.create_engine(url, creator=get_connection)
This resolved my IAM credentials issue.
2 - Secrets and KMS keys
Ensure whatever role you've assigned to the proxy has access to the db users (e.g. "postgres" or "my_user_1") username and passwords stored as separate secrets in aws secrets manager. The secrets should be encrypted, so the same role has to be able to use the key to decrypt.
3 - Dont use admin ("postgres") user
From your rds-connect policy it looks like you are trying to log in as user "postgres", which is the default administrator username for new postgres instance. Even if that's what you want, I don't think that will work.
Create a new read-write role and another role with login on your actual RDS instance (not the proxy) aws blog on postgres roles
CREATE ROLE my_user_1 WITH LOGIN PASSWORD 'my_secret_for_secretsmanager'
Modify your `rds-db:connect' policy to include that user
{
"Version": "2012-10-17",
"Statement":
[
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "rds-db:connect",
"Resource":
[
"arn:aws:rds-db:us-east-1:xxxxxxxx:dbuser:prx-xxxxxxxxxx/postgres",
"arn:aws:rds-db:us-east-1:xxxxxxxx:dbuser:prx-xxxxxxxxxx/my_user_1"
]
}
]
}
Use 'my_user_1' in your application code to generate the token and create the database engine
4 - VPC issues
The RDS instance and the RDS proxy have to be in the same VPC and use the same subnets. And the instance must use a security group (sg-instance) that accepts TCP traffic on port 5432 with a source security group used by proxy (sg-proxy). Then sg-proxy should allow all incoming traffic and have restricted outgoing traffic to TCP on port 5432.
Furthermore, your lambda has to have the AWSLambdaVPCAccessExecutionRole (arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole)
Finally, your lambda has to be assigned to the same VPC as your instance and proxy.
NOTE on debugging You cannot connect to your proxy from the public internet. This is not obvious when you start using proxy, but it is the case aws faq. Just be sure you're code is running from inside the vpc. I think you can use the SAM CLI "https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html" or log onto a free-tier ec2 with all the requisite dependencies.

PyMongo's insert_one() method not working using PyCharm IDE

I'm trying to use mongodb but for some reason i cant put data into a collection.
Here's my code:
import pymongo
from pymongo import MongoClient
mongo_url = "mongodb+srv://<User>:<Password>#cluster0.yozx6.mongodb.net/<dbname>?retryWrites=true&w=majority"
cluster = MongoClient(mongo_url)
db = cluster["TestDatabse"]
collection = db["TestCollection"]
post = {"number": 7}
collection.insert_one(post)
for some reason the data the collection.insert_one line isn't working and it isn't giving an error message either. The program seems to get stuck on it. Can somebody please tell me what I'm doing wrong and how I can fix it.

There are different things to verify for you.
Version compatibility:
First of all insertOne was introduced inmongoDB 3.2, so make sure that you are connecting to a newer pymongo version, and also that the version of pymongo is compatible with your version
Network connection:
Make sure you have a stable connection to your DB if it is remote.
Then use:
result = cluster.admin.command("ismaster")
to check if the db is accesible, if this throws a ConnectionError there is a problem with the connection.
User permissions
Check if the user and password you are using has permissions to insert documents to the given collection.
On the mongo shell:
db.getRoles(
{
rolesInfo: 1,
showPrivileges:true,
showBuiltinRoles: true
}
)
should show:
roles: [
{
role: "readWrite",
db: "TestDatabse"
}
]
for the user User.
Result of insert
Check the result. insert_one returns inserted_id and acknowledged.
res = collection.insert_one(post)
res.acknowledged # should be True
res.inserted_id
Logs
Check your logs, you can find the log path for your server by running:
cat /etc/mongod.conf | grep log

Connecting to remote Elasticsearch server with python's Elasticsearch package

I want to use a remote Elasticsearch server for my website.
I have used elastic.co/ cloud service to create a remote Elasticsearch server.
I can connect/ping remote Elasticsearch server using the following command (it is scrubbed of sensitive info):
curl -u username:password https://55555555555bb0c30d1cba4e9e6.us-central1.gcp.cloud.es.io:9243
After tying this command into terminal, I receive the following response:
{
"name" : "instance-0000000001",
"cluster_name" : "555555555555",
"cluster_uuid" : "55555555555",
"version" : {
"number" : "7.10.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "555555555555555555555555",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
I am using the following Elasticsearch python client to integrate Elasticsearch with my website: https://elasticsearch-py.readthedocs.io/en/v7.10.1/index.html#
To initialize a local connection, I import the Elasticsearch class and initialize an instance of that class by pasting in the local url of my Elasticsearch server.
>>> from elasticsearch import Elasticsearch
>>> es = Elasticsearch('http://localhost:9200')
>>> es
<Elasticsearch([{'host': 'localhost', 'port': 9200}])>
Now I want to connect to my remote server using the same Elasticsearch class. To do this, I need to format the initialization of the Elasticsearch object with the info to connect to my remote elasticsearch server.
This is where I am having some trouble. The docstring for the Elasticsearch class is very opaque. In short, it is asking me to create a custom connection object and I have not been able to figure it out how to do it.
The documentation for the Elasticsearch object is a [little bit better][1] but still does not give an example involving a username and password.
I have the ability to create an API key, but I am not sure how to use it. An answer involving either an API key or answering how to connect using my username and password, would be very helpful.

You need to connect using TLS/SSL and Authentication as described in the documentation.
In your case you should use something like this.
from elasticsearch import Elasticsearch
es = Elasticsearch(
['5555555555bb0c30d1cba4e9e6.us-central1.gcp.cloud.es.io'],
http_auth=('username', 'password'),
scheme="https",
port=9243,
)

Authenticate After Picking the Database

I have 3 databases in my MongoDB server. I am using pymongo to do some scripting with Python3.
I want to use the latest versions and practices. Once I open the client and pick the database, the API for pymongo.MongoClient.['mydatabase'].authenticate is deprecated.
https://api.mongodb.com/python/current/api/pymongo/database.html
Authentication prior to picking the database (while dialing the client) doesn't seem to flow down toward the database. Not just for pymongo, but also when I use mongo shell. So I have a feeling this is the issue.
script.py
import pymongo
from pymongo import MongoClient
u = getUser() # function which prompts for username
p = getPassword() # getpass.getpass('Password')
uri = formatUri(u, p) # formats 'mongodb://%s:%s#%s'.format(user, password, host)
client = MongoClient(uri)
db = client['mydb']
col = db.mycollection
for doc in col.find():
print(doc)
I get the error that I am not authorized for the database. I know my account works in shell but I have to dial the client first then use the db and then auth.
Here's a mongo shell example:
$ mongo
MongoDB shell version: v3.4.10
Connecting to: mongodb://127.0.0.1:port
MongoDB server version: v3.4.10
> use mydb
switched to mydb
> db.auth("user", "pass")
1
Any idea how I can either auth after picking the database or once I use the db it remembers the context I dialed with?

You seem to be missing some concepts here so I'll basically answer as a "guide" to what you should be doing instead. So "authentication' is not really something you do "after" connection, but rather you need to be "looking in the right place" when you actually attempt to authenticate.
We can start this by essentially following the process outlined in Enable Auth from the core documentation, but specifically altered because you want to be running this "test" under your own user account and local directory.
Revision Steps - Straight from Documentation
So first would would want to pick a local working directory and make a path for the database storage files underneath that. On *nix based systems you can do something like:
mkdir -p scratch/data/db
cd scratch
Then we want to startup a separate MongoDB instance without any other options. Making sure the port does not conflict with any other running instance:
mongod --port 37017 --dbpath data/db
In a new terminal or command line window, you can then connect to the shell:
mongo --port 37017
You always want at least one account with administrative privileges to at least "create accounts" and alter them in case you get in trouble, so create one:
use admin
db.createUser(
{
user: "admin",
pwd: "admin",
roles: [{ role: "userAdminAnyDatabase", db: "admin" }]
}
)
Now exit the shell and close the existing mongod instance running in the other terminal or command prompt and then start it again using --auth:
mongod --auth --port 37017 --dbpath data/db
Specific User - Make sure you follow these
Now you actually want to create a user that will be "used by your application". So these steps are important to ensure you get it right.
Log into a shell using your "adminstrative user":
mongo -u admin -p admin --port 37017 --authenticationDatabase 'admin'
You can alternately do the db.auth() method as shown in the question, but as noted this must be authorised on the "admin" namespace.
The next thing you want to do is create a user with access to "mydb" as a namespace with the readWrite role. For kicks, we are also going to let this user have the readAnyDatabase allowing them to "list" all databases namespaces, if not actually being able to do anything else with them.
IMPORTANT: You create ALL your users in the "admin" namespace. And this will be very important in future releases:
use admin
db.createUser(
{
"user": "myuser",
"pwd": "password",
"roles": [
{ "role": "readWrite", "db": "mydb" },
"readAnyDatabase"
]
}
)
Just for additional output, let's look at the current created users:
db.getUsers()
[
{
"_id" : "admin.admin",
"user" : "admin",
"db" : "admin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
}
]
},
{
"_id" : "admin.myuser",
"user" : "myuser",
"db" : "admin",
"roles" : [
{
"role" : "readWrite",
"db" : "mydb"
},
{
"role" : "readAnyDatabase",
"db" : "admin"
}
]
}
]
See how these have expanded in naming, and particularly the values assigned to the various "db" keys on each user. This should give you a little more insight into how MongoDB looks this up and why.
Python Connection
Finally we just want to connect from python. So presuming you have python and pymongo installed already, then it's just a simple listing to verify:
import pymongo
from pymongo import MongoClient
client = MongoClient('mongodb://myuser:password#localhost:37017');
db = client['mydb']
col = db.test
col.remove()
col.insert_one({ "a": 1 })
for doc in col.find():
print(doc)
Which shows the document created and listed without problem:
{u'a': 1, u'_id': ObjectId('5a08e5e0760108251722a737')}
Note that we don't actually need to make any mention of "admin" here, because this is the default where the driver "expects the accounts to be" and where you really "should" be doing it.
But I did it the wrong way
So let's say you originally got all confused and created the user under "mydb" instead:
use mydb
db.createUser({ "user": "bert", "pwd": "password", "roles": ["readWrite"] })
If you go look in "admin" that user is not there. But if you look on "mydb":
use mydb
db.getUsers()
[
{
"_id" : "mydb.bert",
"user" : "bert",
"db" : "mydb",
"roles" : [
{
"role" : "readWrite",
"db" : "mydb"
}
]
}
]
So you can see where the actual user data is now kept and how it has been recorded.
The simple case here is you "must" tell MongoDB where to obtain the authentication from for this user:
client = MongoClient('mongodb://bert:password#localhost:37017/mydb');
See how we add "mydb" on to the connection string. This is how it's done.
This is actually "in progress" to be made consistent with ALL drivers in how connections are made and where authentication happens as well as where you select the database. But there are basic rules:
If no other database namespace is provided with connection details for authentication credentials, then "admin" is taken to be the default.
Where there is a database namespace provided on the connection string, this will be used for authentication and this is the actual intent of the database namespace on the connection string.
Though other drivers "presently" differ in the role of the database namespace on the connection string, the usage is being changed to be consistent with all drivers that "using" a database namespace is in fact an API call, rather than being assigned from the connection string.
So where you need to authenticate depends on "where you created the user". But you should really be noting that "admin" is the place where you "should" be doing this instead of anywhere else.
Deprecation of Authenticate after connect
Whilst all drivers actually do have a similar method to authenticate(), which is used much like the shell example in the question, this method is now considered DEPRECATED as is mentioned throughout the content of the answer it is "intended" that you actually store your users in the "admin" namespace:
"Changed in version 3.5: Deprecated. Authenticating multiple users conflicts with support for logical sessions in MongoDB 3.6. To authenticate as multiple users, create multiple instances of MongoClient."
This is why the whole answer here is based on NOT using that method as you are meant to creating new connection instances, or using the "sessions" functionality available from MongoDB 3.6 instead.

Not autorized to execute any command using Service Connector on MongoDB

I am using the MongoDB on my app and when I try to access the database directly using the service connector, I am able to connect but then I am getting :
Error: error: {
"ok" : 0,
"errmsg" : "not authorized on admin to execute command { *any command*}",
"code" : 13
}
and this on any query or command.
Is there a way to change authorization or accessing the data of my MongoDB
P.S: My MongoDB was bind as in the tutorial: https://docs.developer.swisscom.com/tutorial-python/bind-service.html

It looks like you're trying to execute commands on the admin database on which your user is not authorized. You can find the correct database which your user is authorized on in the credentials (key mongodb.credentials.database) but ideally you connect using the provided URI (mongodb.credentials.uri) which will connect you to the correct database automatically.
You can have a look at the Python example in the tutorial you linked to find out how to access and use those credentials correctly.

The answer from Sandro Mathys is correct and helpful, I wish to clarify/simplyfy a little bit.
The service broker grants you the Role dbOwner and creates a database with random name for you. This is done during cf create-service process.
The database owner can perform any administrative action on the
database. This role combines the privileges granted by the readWrite,
dbAdmin and userAdmin roles.
You have no privileges on admin database. The admin database is only for Swisscom operators. Please use for login with mongo shell the parameter --authenticationDatabase with the random database name from cf env.
Specifies the database in which the user is created. See Authentication Database.
If you do not specify a value for --authenticationDatabase, mongo uses the database specified in the connection string.