Python example to connect to Aurora MySQL with IAM User Authentication - python

I am connecting to an Aurora database using python.Everything works when I have a static userid and password;
I am trying to use IAM Authenticated user and the user has been setup and it works well using MySQL Workbench
I am trying to use SQLAlchemy to connect using python - this is supposed to be a no brainier but it does not work
from _collections import OrderedDict
import datetime
from sqlalchemy import create_engine
from boto3 import client
# import pymysql
db_end_point = "yyyyyyyyyyyy.cluster-xxxxxxxxxxxx.eu-west-1.rds.amazonaws.com"
def get_db_password(cluster_end_point, user_name):
rds_client = client('rds', region_name='eu-west-1')
rds_token = rds_client.generate_db_auth_token(DBHostname=cluster_end_point, Port=3306, DBUsername=user_name, Region='eu-west-1')
return rds_token
user_password = get_db_password(db_end_point, "svc_payment_data_write")
print(user_password)
url = "mysql://{}:3306/payment_data_store".format(db_end_point)
rds_credentials = {
'user' : 'svc_payment_data_write', 'passwd' : user_password
}
kw = dict()
kw.update(rds_credentials)
print(kw)
engine = create_engine(url, connect_args=kw)
connection = engine.connect()
I have also tried creating the url with the password string quoted properly. The same thing works in JAVA properly

Related

flask_mysqldb - MYSQL cursor is throwing an error - cursor is not a known member of "None"

I am writing a simple auth service in python using flask and flask_mysqldb. There is an error with the cursor.
import jwt
from flask import Flask, request
from flask_mysqldb import MySQL
server = Flask(__name__)
mysql = MySQL(server)
# server configuration
server.config["MYSQL_HOST"] = os.environ.get("MYSQL_HOST")
server.config["MYSQL_USER"] = os.environ.get("MYSQL_USER")
server.config["MYSQL_PASSWORD"] = os.environ.get("MYSQL_PASSWORD")
server.config["MYSQL_DB"] = os.environ.get("MYSQL_DB")
server.config["MYSQL_PORT"] = os.environ.get("MYSQL_PORT")
# print(server.config["MYSQL_HOST"])
# print(server.config["MYSQL_PORT"])
#server.route("/login", methods=["POST"])
def login():
auth = request.authorization
if not auth:
return "missing credentials",401
#check db for username and password
cur = mysql.connection.cursor()
res = cur.execute(
"SELECT email,password FROM user WHERE email=%s, (auth.username)"
)
This works on a virtual environment. All the specified packages are correctly installed.
Please try
cur = mysql.connect.cursor()
if you use a connection it will not suggest cursor(). once you use connect.cursor() it will not show the error. Thanks

How to connect SQLAlchemy to Snowflake database using OAuth2?

I need to connect to Snowflake using SQLAlchemy but the trick is, I need to authenticate using OAuth2. Snowflake documentation only describes connecting using username and password and this cannot be used in the solution I'm building. I can authenticate using Snowflake's python connector but I see no simple path how to glue it with SQLAlchemy. I'd like to know if there is a ready solution before I write a custom interface for this.
Use snowflake.connector.connect to create a PEP-249 Connection to the database - see documentation. Then use param creator of create_engine (docs) - it takes a callable that returns PEP-249 Connection. If you use it then URL param is ignored.
Example code:
def get_connection():
return snowflake.connector.connect(
user="<username>",
host="<hostname>",
account="<account_identifier>",
authenticator="oauth",
token="<oauth_access_token>",
warehouse="test_warehouse",
database="test_db",
schema="test_schema"
)
engine = create_engine("snowflake://not#used/db", creator=get_connection)
I got this working but just adding more params in the connection URL:
from sqlalchemy.engine import create_engine
import urllib.parse
connection_url = f"snowflake://{user}:#{account}/{database}/{schema}?warehouse={warehouse}&authenticator=oauth&token={urllib.parse.quote(access_token)}"
engine = create_engine(connection_url)
with engine.begin() as connection:
print(connection.execute('select count(*) from lineitem').fetchone())
If you don't want to be constructing the URL on your own, you can use snowflake.sqlalchemy.URL like this:
from snowflake.sqlalchemy import URL
connection_url = URL(
user=user,
authenticator="oauth",
token=access_token,
host=host,
account=account,
warehouse=warehouse,
database=database,
schema=schema
)

Use Ipython-sql with snowflake and externalbrowser authenticator

in my jupyter notebook I connect to snowflake with an externalbrowser auth like so:
conn = snowflake.connector.connect(
user='<my user>',
authenticator='externalbrowser',
account='<my account>',
warehouse='<the warehouse>')
this opens an external browser to auth and after that works fine with pandas read sql:
pd.read_sql('<a query>', conn)
want to use it with ipython sql, but when I try:
%sql snowflake://conn.user#conn.account
I get:
snowflake.connector.errors.ProgrammingError) Password is empty
well I don't have one :)
any ideas how to pass this?
IPython-sql connection strings are SQLAlchemy URL standard, therefore you can do the following:
%load_ext sql
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(URL(
account = '<account>',
user = '<user>',
database = 'testdb',
schema = 'public',
warehouse = '<wh>',
role='public',
authenticator='externalbrowser'
))
connection = engine.connect()
This would open the external browser for authentication.

OperationalError: 250003: Failed to get the response. Hanging? method: post

I am trying to connect to snowflake using my login credentials. I'm using the following code:
snowflake.connector.connect(
user="<my_user_name>",
password="<my_password>",
account="<my_account_name_with_region_and_cloud>"
)
When I try to run the above code, I'm getting the following error:
OperationalError: 250003: Failed to get the response. Hanging? method: post, url: https://hm53485.us-east-2.aws.snowflakecomputing.com:443/session/v1/login-request?request_id=fcfdd77a-11ff-4956-9ed8-bcc332c5989a&databaseName=S3_DB&schemaName=PUBLIC&warehouse=COMPUTE_WH&request_guid=b9fdb5c9-81cb-4ecb-8d20-abef44249bbf
I'm sure that all my packages are up to date. I'm using python 3.6.4 and the latest snowflake_connector_python.
I'm currently on us-east-2 location in aws.
Can someone please help me out on this????
Just Give your account name in the account .We dont need the region and full URL.
Please check below .
----------------------------------------------------------------------
import snowflake.connector
PASSWORD = '*******'
USER = '<USERNAME>'
ACCOUNT = 'SFCSUPPORT'
WAREHOUSE = '<WHNAME>'
DATABASE = '<DBNAME>'
SCHEMA = 'PUBLIC'
print("Connecting...")
# -- (> ------------------- SECTION=connect_to_snowflake --------------------
con = snowflake.connector.connect(
user=USER,
password=PASSWORD,
account=ACCOUNT,
warehouse=WAREHOUSE,
database=DATABASE,
schema=SCHEMA
)
con.cursor().execute("USE WAREHOUSE " + WAREHOUSE)
con.cursor().execute("USE DATABASE " + DATABASE)
#con.cursor().execute("USE SCHEMA INFORMATION_SCHEMA")
try:
result = con.cursor().execute("Select * from <TABLE>")
result_list = result.fetchall()
print(result_list)
finally:
con.cursor().close()
con.cursor().close()
I'm using sqlalchemy, which you can install via pip:
pip install SQLAlchemy
https://docs.snowflake.net/manuals/user-guide/sqlalchemy.html
Here's what I have at the beginning of my notebook:
import snowflake.connector
import pandas as pd
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
url = URL(
account = 'xxxxxxxx.east-us-2.azure',
user = 'xxxxxxxx',
password = 'xxxxxxxx',
database = 'xxxxxxxx',
schema = 'xxxxxxxx',
warehouse = 'xxxxxxxx',
role='xxxxxxxx'
)
engine = create_engine(url)
connection = engine.connect()
query = '''
select 1 AS VAL;
'''
df = pd.read_sql(query, connection)
df
I was getting a similar error. Tried few things like making sure the account name is correct as per https://docs.snowflake.com/en/user-guide/admin-account-identifier.html. The account name depends on the region in which your snowflake account is located. Note that some of the cloud regions need a cloud provider name at the end and some do it.
But it didn't help fix the issue I was facing. For me, it turned out to be a proxy issue. I was trying to connect from a corporate network with a proxy and it was blocking the connection to Snowflake. Whitelisting the snowflake URL in proxy fixed the issue for me.

Having Trouble Connecting to Cloud SQL (PostgreSQL) using Python's SQLALCHEMY

I set up the Cloud SQL instance on Google Cloud Platform and followed the official instructions, but don't seem to be able to connect to the Cloud SQL instance. When I try to do a sanity check and access the PostgreSQL db through Cloud Shell, I'm able to connect successfully though.
Could someone please help - I would be much obliged.
Code:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://<user>:<pass>#<public IP Address/<table>')
engine.connect()
Error:
Is the server running on host "XX.XX.XXX.XX" and accepting
TCP/IP connections on port XXXX?
I found another way to connect to a PostgreSQL GCP instance without using the Cloud SQL Proxy.
Code:
import sqlalchemy
username = '' # DB username
password = '' # DB password
host = '' # Public IP address for your instance
port = '5432'
database = '' # Name of database ('postgres' by default)
db_url = 'postgresql+psycopg2://{}:{}#{}:{}/{}'.format(
username, password, host, port, database)
engine = sqlalchemy.create_engine(db_url)
conn = engine.connect()
I whitelisted my IP address before trying to connect. (https://cloud.google.com/sql/docs/postgres/connect-external-app#appaccessIP)
Use the Cloud SQL proxy to connect to Cloud SQL from external applications.
In order to achieve this please follow the relevant documentation.
The steps described would consist of:
Enabling the Cloud SQL Admin API on your Cloud Console.
Installing the relevant proxy client according to your OS.
Use any of the available methods to authenticate the Cloud SQL Proxy.
Invoke the proxy with ./cloud_sql_proxy -instances=INSTANCE_CONNECTION_NAME=tcp:5432 & ond your terminal and connect the proxy by changing your code and using SQLALCHEMY:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://DATABASE_USER:PASSWORD#localhost:5432/')
NOTE: the code above assumes you are not trying to connect to the proxy in a production environment and are using an authenticated Cloud SDK client in order to connect to the proxy.
This worked to me using the Cloud SQL Proxy on my personal computer and uploading the code to Google App Engine standard.
db_user = os.environ.get('CLOUD_SQL_USERNAME')
db_pass = os.environ.get('CLOUD_SQL_PASSWORD')
db_name = os.environ.get('CLOUD_SQL_DATABASE_NAME')
db_connection_name = os.environ.get('CLOUD_SQL_CONNECTION_NAME')
if os.environ.get('GAE_ENV') == 'standard':
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#/{db_name}?host=/cloudsql/{db_connection_name}'
else:
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#127.0.0.1:1234/{db_name}'
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = db_uri
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
Depending on the database client library, the socket (/cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432) needs to be specified.
The docs have this example for SQLAlchemy:
db_user = os.environ["DB_USER"]
db_pass = os.environ["DB_PASS"]
db_name = os.environ["DB_NAME"]
db_socket_dir = os.environ.get("DB_SOCKET_DIR", "/cloudsql")
cloud_sql_connection_name = os.environ["CLOUD_SQL_CONNECTION_NAME"]
pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgresql+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<socket_path>/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL.create(
drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
database=db_name, # e.g. "my-database-name"
query={
"unix_sock": "{}/{}/.s.PGSQL.5432".format(
db_socket_dir, # e.g. "/cloudsql"
cloud_sql_connection_name) # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
}
),
**db_config
)
Be aware that this example is with pg8000 that uses unix_sock instead of unix_socket as socket identifier.

Categories