I recently had to enable SSO with Okta and had a few python projects I was running in Google Colab.
I am trying to redesign the connection string but can't seem to get it right.
This was my initial connection string before SSO:
from snowflake.sqlalchemy import URL
from sqlalchemy import create_engine
engine = create_engine(URL(
account = acc,
user = usr,
password = psw,
warehouse = whs,
role = rol
))
engine.connect()
This is what I found from research it should be with SSO:
from snowflake.sqlalchemy import URL
from sqlalchemy import create_engine
engine = create_engine(URL(
account = acc,
user = usr,
password = psw,
warehouse = whs,
role = rol
),
connect_args={
'authenticator': 'https://myokta.okta.com/',
}
)
engine.connect()
I tried that but I am getting this error:
I also tried using {'authenticator': 'externalbrowser'} but because I am in Google Colab I get an error stating Unable to open a browser in this environment..
The Web UI is working for the same user so it's just in Colab that I am having this issue.
How should I go about to connect?
EDIT:
So after doing some research I found that because we have MFA enabled this would not work. Is it possible to then use:
engine = create_engine(URL(
account = acc,
user = usr,
warehouse = whs,
role = rol,
authenticator = 'externalbrowser'
))
engine = engine.connect()
And have the externalbrowser be an iframe in the same notebook?
I managed to find the solution. When running engine.connect() with authenticator='externalbrowser' and Google Collab cannot open a separate tab, it will provide a manual link, which, when clicked, opens another tab pointing to localhost URL with a token as a param. I then copy this URL and when going back to the notebook, I paste this URL to the input box opened in the cell.
Related
in my jupyter notebook I connect to snowflake with an externalbrowser auth like so:
conn = snowflake.connector.connect(
user='<my user>',
authenticator='externalbrowser',
account='<my account>',
warehouse='<the warehouse>')
this opens an external browser to auth and after that works fine with pandas read sql:
pd.read_sql('<a query>', conn)
want to use it with ipython sql, but when I try:
%sql snowflake://conn.user#conn.account
I get:
snowflake.connector.errors.ProgrammingError) Password is empty
well I don't have one :)
any ideas how to pass this?
IPython-sql connection strings are SQLAlchemy URL standard, therefore you can do the following:
%load_ext sql
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(URL(
account = '<account>',
user = '<user>',
database = 'testdb',
schema = 'public',
warehouse = '<wh>',
role='public',
authenticator='externalbrowser'
))
connection = engine.connect()
This would open the external browser for authentication.
I am trying to connect a Python Flask app running in Azure App Service Web App to an Azure SQL Database.
The works just fine when I use SQL authentication with username and password.
Now I want to move to using the Web Apps managed identity.
I have activated the system-assigned managed identity, created a user for it in SQL and added it to the db_datareader role.
I am connecting with SqlAlchemy using a connection string like this
params = urllib.parse.quote_plus(os.environ['SQL_CONNECTION_STRING'])
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine_azure = db.create_engine(conn_str,echo=True)
The connection string is stored as an application setting, and its value is
"Driver={ODBC Driver 17 for SQL Server};Server=tcp:<server>.database.windows.net,1433;Database=<database>;Authentication=ActiveDirectoryMsi;"
I expected this to be all I need to do, but now my app is not starting.
The logs report a timeout when connecting to the database.
How can I fix this?
I know this is quite an old post, but it may help people like me who are looking for a solution.
You could modify the connection string by adding "Authentication" parameters as "ActiveDirectoryMsi", no need to use endpoint and headers.
(Works with Azure SQL, for other databases like Postgress you may need to use the struct token)
import pyodbc
pyodbc.connect(
"Driver="
+ driver
+ ";Server="
+ server
+ ";PORT=1433;Database="
+ database
+ ";Authentication=ActiveDirectoryMsi")
I wrote a quick article for those who are interested in Azure MSI:
https://hedihargam.medium.com/python-sql-database-access-with-managed-identity-from-azure-web-app-functions-14566e5a0f1a
If you want to connect Azure SQL database with Azure MSI in python application, we can use the SDK pyodbc to implement it.
For example
Enable system-assigned identity for your Azure app service
Add the MSi as contained database users in your database
a. Connect your SQL database with Azure SQL AD admin (I use SSMS to do it)
b. run the following the script in your database
CREATE USER <your app service name> FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER <your app service name>
ALTER ROLE db_datawriter ADD MEMBER <your app service name>
ALTER ROLE db_ddladmin ADD MEMBER <your app service name>
Code
import os
import pyodbc
import requests
import struct
#get access token
identity_endpoint = os.environ["IDENTITY_ENDPOINT"]
identity_header = os.environ["IDENTITY_HEADER"]
resource_uri="https://database.windows.net/"
token_auth_uri = f"{identity_endpoint}?resource={resource_uri}&api-version=2019-08-01"
head_msi = {'X-IDENTITY-HEADER':identity_header}
resp = requests.get(token_auth_uri, headers=head_msi)
access_token = resp.json()['access_token']
accessToken = bytes(access_token, 'utf-8');
exptoken = b"";
for i in accessToken:
exptoken += bytes({i});
exptoken += bytes(1);
tokenstruct = struct.pack("=i", len(exptoken)) + exptoken;
conn = pyodbc.connect("Driver={ODBC Driver 17 for SQL Server};Server=tcp:andyserver.database.windows.net,1433;Database=database2", attrs_before = { 1256:bytearray(tokenstruct) });
cursor = conn.cursor()
cursor.execute("select ##version")
row = cursor.fetchall()
For more details, please refer to the
https://github.com/AzureAD/azure-activedirectory-library-for-python/wiki/Connect-to-Azure-SQL-Database
https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-aad-authentication-configure
I am using a Python script to connect to a SQL Server database:
import pyodbc
import pandas
server = 'SQL'
database = 'DB_TEST'
username = 'USER'
password = 'My password'
sql='''
SELECT *
FROM [DB_TEST].[dbo].[test]
'''
cnxn = pyodbc.connect('DRIVER=SQL Server;SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
data = pandas.read_sql(sql,cnxn)
cnxn.close()
The script is launched everyday by an automatisation tools so there is no physical user.
The issue is how to replace the password field by a secure method?
The automated script is still ran by a windows user. Add this windows user to the SQL-Server users and give it the appropriate permissions, so you can use:
import pyodbc
import pandas
server = 'SQL'
database = 'DB_TEST'
sql='''
SELECT *
FROM [DB_TEST].[dbo].[test]
'''
cnxn = pyodbc.connect(
f'DRIVER=SQL Server;SERVER={server};DATABASE={database};Trusted_Connection=True;')
data = pandas.read_sql(sql,cnxn)
cnxn.close()
I am also interested in secure coding using Python .I did my own research to figure out available options, I would recommend reviewing this post as it summarize it all. Check on the listed options, and apply the one suits you better.
I am trying to connect to snowflake using my login credentials. I'm using the following code:
snowflake.connector.connect(
user="<my_user_name>",
password="<my_password>",
account="<my_account_name_with_region_and_cloud>"
)
When I try to run the above code, I'm getting the following error:
OperationalError: 250003: Failed to get the response. Hanging? method: post, url: https://hm53485.us-east-2.aws.snowflakecomputing.com:443/session/v1/login-request?request_id=fcfdd77a-11ff-4956-9ed8-bcc332c5989a&databaseName=S3_DB&schemaName=PUBLIC&warehouse=COMPUTE_WH&request_guid=b9fdb5c9-81cb-4ecb-8d20-abef44249bbf
I'm sure that all my packages are up to date. I'm using python 3.6.4 and the latest snowflake_connector_python.
I'm currently on us-east-2 location in aws.
Can someone please help me out on this????
Just Give your account name in the account .We dont need the region and full URL.
Please check below .
----------------------------------------------------------------------
import snowflake.connector
PASSWORD = '*******'
USER = '<USERNAME>'
ACCOUNT = 'SFCSUPPORT'
WAREHOUSE = '<WHNAME>'
DATABASE = '<DBNAME>'
SCHEMA = 'PUBLIC'
print("Connecting...")
# -- (> ------------------- SECTION=connect_to_snowflake --------------------
con = snowflake.connector.connect(
user=USER,
password=PASSWORD,
account=ACCOUNT,
warehouse=WAREHOUSE,
database=DATABASE,
schema=SCHEMA
)
con.cursor().execute("USE WAREHOUSE " + WAREHOUSE)
con.cursor().execute("USE DATABASE " + DATABASE)
#con.cursor().execute("USE SCHEMA INFORMATION_SCHEMA")
try:
result = con.cursor().execute("Select * from <TABLE>")
result_list = result.fetchall()
print(result_list)
finally:
con.cursor().close()
con.cursor().close()
I'm using sqlalchemy, which you can install via pip:
pip install SQLAlchemy
https://docs.snowflake.net/manuals/user-guide/sqlalchemy.html
Here's what I have at the beginning of my notebook:
import snowflake.connector
import pandas as pd
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
url = URL(
account = 'xxxxxxxx.east-us-2.azure',
user = 'xxxxxxxx',
password = 'xxxxxxxx',
database = 'xxxxxxxx',
schema = 'xxxxxxxx',
warehouse = 'xxxxxxxx',
role='xxxxxxxx'
)
engine = create_engine(url)
connection = engine.connect()
query = '''
select 1 AS VAL;
'''
df = pd.read_sql(query, connection)
df
I was getting a similar error. Tried few things like making sure the account name is correct as per https://docs.snowflake.com/en/user-guide/admin-account-identifier.html. The account name depends on the region in which your snowflake account is located. Note that some of the cloud regions need a cloud provider name at the end and some do it.
But it didn't help fix the issue I was facing. For me, it turned out to be a proxy issue. I was trying to connect from a corporate network with a proxy and it was blocking the connection to Snowflake. Whitelisting the snowflake URL in proxy fixed the issue for me.
I set up the Cloud SQL instance on Google Cloud Platform and followed the official instructions, but don't seem to be able to connect to the Cloud SQL instance. When I try to do a sanity check and access the PostgreSQL db through Cloud Shell, I'm able to connect successfully though.
Could someone please help - I would be much obliged.
Code:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://<user>:<pass>#<public IP Address/<table>')
engine.connect()
Error:
Is the server running on host "XX.XX.XXX.XX" and accepting
TCP/IP connections on port XXXX?
I found another way to connect to a PostgreSQL GCP instance without using the Cloud SQL Proxy.
Code:
import sqlalchemy
username = '' # DB username
password = '' # DB password
host = '' # Public IP address for your instance
port = '5432'
database = '' # Name of database ('postgres' by default)
db_url = 'postgresql+psycopg2://{}:{}#{}:{}/{}'.format(
username, password, host, port, database)
engine = sqlalchemy.create_engine(db_url)
conn = engine.connect()
I whitelisted my IP address before trying to connect. (https://cloud.google.com/sql/docs/postgres/connect-external-app#appaccessIP)
Use the Cloud SQL proxy to connect to Cloud SQL from external applications.
In order to achieve this please follow the relevant documentation.
The steps described would consist of:
Enabling the Cloud SQL Admin API on your Cloud Console.
Installing the relevant proxy client according to your OS.
Use any of the available methods to authenticate the Cloud SQL Proxy.
Invoke the proxy with ./cloud_sql_proxy -instances=INSTANCE_CONNECTION_NAME=tcp:5432 & ond your terminal and connect the proxy by changing your code and using SQLALCHEMY:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://DATABASE_USER:PASSWORD#localhost:5432/')
NOTE: the code above assumes you are not trying to connect to the proxy in a production environment and are using an authenticated Cloud SDK client in order to connect to the proxy.
This worked to me using the Cloud SQL Proxy on my personal computer and uploading the code to Google App Engine standard.
db_user = os.environ.get('CLOUD_SQL_USERNAME')
db_pass = os.environ.get('CLOUD_SQL_PASSWORD')
db_name = os.environ.get('CLOUD_SQL_DATABASE_NAME')
db_connection_name = os.environ.get('CLOUD_SQL_CONNECTION_NAME')
if os.environ.get('GAE_ENV') == 'standard':
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#/{db_name}?host=/cloudsql/{db_connection_name}'
else:
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#127.0.0.1:1234/{db_name}'
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = db_uri
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
Depending on the database client library, the socket (/cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432) needs to be specified.
The docs have this example for SQLAlchemy:
db_user = os.environ["DB_USER"]
db_pass = os.environ["DB_PASS"]
db_name = os.environ["DB_NAME"]
db_socket_dir = os.environ.get("DB_SOCKET_DIR", "/cloudsql")
cloud_sql_connection_name = os.environ["CLOUD_SQL_CONNECTION_NAME"]
pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgresql+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<socket_path>/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL.create(
drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
database=db_name, # e.g. "my-database-name"
query={
"unix_sock": "{}/{}/.s.PGSQL.5432".format(
db_socket_dir, # e.g. "/cloudsql"
cloud_sql_connection_name) # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
}
),
**db_config
)
Be aware that this example is with pg8000 that uses unix_sock instead of unix_socket as socket identifier.