We Keep Coding

Issue Connecting to Mssql Instance From Google Cloud Function

I am trying to connect to a mssql instance in cloud sql in a cloud function. I have gone through the necessary steps of setting up a private IP, serverless VPC connector, and connecting my function to the VPC. I have been able to connect to the instance in nodejs but python suits my current needs more. The error I'm getting in the logs is:
pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server'
From all the examples I have read it does not appear that you need to import them or anything.
This is my process of connecting and executing a simple request.
import sqlalchemy
import pyodbc
def hello_world(request):
# connect_simple()
db = connect_tcp_socket()
a = execute_request(db)
return a
def connect_tcp_socket() -> sqlalchemy.engine.base.Engine:
db_host = 'my_private_ip'
db_user = 'my_db_user'
db_pass = 'my_db_pass'
db_name = 'my_db_name'
db_port = 'my_db_port'
connection_string = 'DRIVER={ODBC Driver 17 for SQL Server};SERVER='+db_host+';PORT='+db_port+'DATABASE='+db_name+';UID='+db_user+';PWD='+ db_pass+';Encrypt=no'
connection_url = sqlalchemy.engine.url.URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = sqlalchemy.create_engine(
connection_url
)
def execute_request(db: sqlalchemy.engine.base.Engine):
print('ok')
with db.connect() as conn:
result = conn.execute('SELECT ##VERSION')
barray= []
for row in result:
barray.append(row)
return barray

I'd recommend using the Cloud SQL Python Connector to connect to Cloud SQL from Python as it will not require the ODBC driver and is much easier to use within Cloud Functions/Cloud Run.
Just replace your connect_tcp_socket with the below connect_with_connector function.
from google.cloud.sql.connector import Connector, IPTypes
import pytds
import sqlalchemy
def connect_with_connector() -> sqlalchemy.engine.base.Engine:
def getconn() -> pytds.Connection:
with Connector() as connector:
conn = connector.connect(
"project-id:region:instance-name", # Cloud SQL connection name
"pytds",
user="my-user",
password="my-password",
db="my-database",
ip_type=IPTypes.PRIVATE
)
return conn
engine = sqlalchemy.create_engine(
"mssql+pytds://localhost",
creator=getconn,
)
return engine
You can find a code sample for the Python Connector similar to the one you are using for establishing a TCP connection.
Note: Pytds driver is not super great with error handling. If you see the OSError: [Errno 9] Bad file descriptor error it usually means your database user is missing proper permissions and should grant them the necessary grants from a root user.
Your requirements.txt should include the following:
cloud-sql-python-connector
SQLAlchemy
python-tds
sqlalchemy-pytds
There is also an interactive getting started Colab Notebook that will walk you through using the Python Connector without you needing to change a single line of code!
It makes connecting to Cloud SQL both easy and secure from Cloud Functions.

There is another way to access Azure SQL with Python in an Azure Function?

I have an Azure function witch python that connect a database azure SQL, I’m using the package pyodbc to connect database.
On my computer it's working, but when I deploy via vscode, azure does all the python installations based on the requirements and he tries to do the download through website
https://github.com/pypa/pip/issues/8368./n , which is out
There is another way to access Azure SQL with Python in an Azure Function??
I’m using the above tools
Developer in VSCODE in Windows 10
Azure Functions:
Linux
Type azure function is “HTTP”
Local is East 2
My code
Import pyodbc
password = *********
server = 'prd-xx-xxx-xxx-01.database.windows.net'
database = 'prd-xx-xxx-xxx-db-02'
username = 'adm'
driver= '{ODBC Driver 17 for SQL Server}'
conn = pyodbc.connect('DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password)
df = pd.read_sql(select_cidades,conn)
conn.close()

The Python worker comes with an ODBC 17 driver. It's not documented at this time, but you can use it by adding pyodbc to your requirements.txt file in VS Code. See: https://github.com/MicrosoftDocs/azure-docs/issues/54423
The documented way to solve this is to use an environment where you can provide the package. You can create a Docker custom image with the Azure Function runtime and publish in the Azure Cloud (or in an on-premise infrastructure).
https://medium.com/globant/serverless-applications-with-azure-functions-python-and-docker-b594fb90fd4f

You also could use pymssql to connect to the Azure SQL database.
Here's the example code:
import pymssql
conn = pymssql.connect(server='yourserver.database.windows.net', user='yourusername#yourserver', password='yourpassword', database='AdventureWorks')
cursor = conn.cursor()
cursor.execute('SELECT c.CustomerID, c.CompanyName,COUNT(soh.SalesOrderID) AS OrderCount FROM SalesLT.Customer AS c LEFT OUTER JOIN SalesLT.SalesOrderHeader AS soh ON c.CustomerID = soh.CustomerID GROUP BY c.CustomerID, c.CompanyName ORDER BY OrderCount DESC;')
row = cursor.fetchone()
while row:
print str(row[0]) + " " + str(row[1]) + " " + str(row[2])
row = cursor.fetchone()
Please ref: concept connecting to SQL using pymssql

Use Managed Identity to authenticate Azure App Service to SQL Database

I am trying to connect a Python Flask app running in Azure App Service Web App to an Azure SQL Database.
The works just fine when I use SQL authentication with username and password.
Now I want to move to using the Web Apps managed identity.
I have activated the system-assigned managed identity, created a user for it in SQL and added it to the db_datareader role.
I am connecting with SqlAlchemy using a connection string like this
params = urllib.parse.quote_plus(os.environ['SQL_CONNECTION_STRING'])
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine_azure = db.create_engine(conn_str,echo=True)
The connection string is stored as an application setting, and its value is
"Driver={ODBC Driver 17 for SQL Server};Server=tcp:<server>.database.windows.net,1433;Database=<database>;Authentication=ActiveDirectoryMsi;"
I expected this to be all I need to do, but now my app is not starting.
The logs report a timeout when connecting to the database.
How can I fix this?

I know this is quite an old post, but it may help people like me who are looking for a solution.
You could modify the connection string by adding "Authentication" parameters as "ActiveDirectoryMsi", no need to use endpoint and headers.
(Works with Azure SQL, for other databases like Postgress you may need to use the struct token)
import pyodbc
pyodbc.connect(
"Driver="
+ driver
+ ";Server="
+ server
+ ";PORT=1433;Database="
+ database
+ ";Authentication=ActiveDirectoryMsi")
I wrote a quick article for those who are interested in Azure MSI:
https://hedihargam.medium.com/python-sql-database-access-with-managed-identity-from-azure-web-app-functions-14566e5a0f1a

If you want to connect Azure SQL database with Azure MSI in python application, we can use the SDK pyodbc to implement it.
For example
Enable system-assigned identity for your Azure app service
Add the MSi as contained database users in your database
a. Connect your SQL database with Azure SQL AD admin (I use SSMS to do it)
b. run the following the script in your database
CREATE USER <your app service name> FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER <your app service name>
ALTER ROLE db_datawriter ADD MEMBER <your app service name>
ALTER ROLE db_ddladmin ADD MEMBER <your app service name>
Code
import os
import pyodbc
import requests
import struct
#get access token
identity_endpoint = os.environ["IDENTITY_ENDPOINT"]
identity_header = os.environ["IDENTITY_HEADER"]
resource_uri="https://database.windows.net/"
token_auth_uri = f"{identity_endpoint}?resource={resource_uri}&api-version=2019-08-01"
head_msi = {'X-IDENTITY-HEADER':identity_header}
resp = requests.get(token_auth_uri, headers=head_msi)
access_token = resp.json()['access_token']
accessToken = bytes(access_token, 'utf-8');
exptoken = b"";
for i in accessToken:
exptoken += bytes({i});
exptoken += bytes(1);
tokenstruct = struct.pack("=i", len(exptoken)) + exptoken;
conn = pyodbc.connect("Driver={ODBC Driver 17 for SQL Server};Server=tcp:andyserver.database.windows.net,1433;Database=database2", attrs_before = { 1256:bytearray(tokenstruct) });
cursor = conn.cursor()
cursor.execute("select ##version")
row = cursor.fetchall()
For more details, please refer to the
https://github.com/AzureAD/azure-activedirectory-library-for-python/wiki/Connect-to-Azure-SQL-Database
https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-aad-authentication-configure

SQL Connect error in Python(Windows): severity 9:\nAdaptive Server connection failed

Not able to connect to Azure DB. I get the following error while connecting via Python.
I'm able to connect to my usual SQL environment
import pandas as pd
import pymssql
connPDW = pymssql.connect(host=r'dwprd01.database.windows.net', user=r'internal\admaaron',password='',database='')
connPDW.autocommit(True)
cursor = connPDW.cursor()
conn.autocommit(True)
cursor = conn.cursor()
sql = """
select Top (10) * from TableName
"""
cursor.execute(sql);
Run without errors.

Just according to your code, there is an obvious issue of connecting Azure SQL Database by pymssql package in Python which use the incorrect user format and lack of the values of password and database parameters.
Please follow the offical document Step 3: Proof of concept connecting to SQL using pymssql carefully to change your code correctly.
If you have an instance of Azure SQL Database with the connection string of ODBC, such as Driver={ODBC Driver 13 for SQL Server};Server=tcp:<your hostname>.database.windows.net,1433;Database=<your database name>;Uid=<username>#<host>;Pwd=<your_password>;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30; show in the Connection strings tab of your SQL Database on Azure portal.
Then, your code should be like below
hostname = '<your hostname>'
server = f"{hostname}.database.windows.net"
username = '<your username>'
user = f"{username}#{hostname}"
password = '<your password>'
database = '<your database name>'
import pymssql
conn = pymssql.connect(server=server, user=user, password=password, database=database)
Meanwhile, just additional note for the version of Azure SQL Database and MS SQL Server are 2008+ like the latest Azure SQL Database, you should use the ODBC Driver connection string which be started with DRIVER={ODBC Driver 17 for SQL Server};, not 13 show in the connection string of Azure portal if using ODBC driver for Python with pyodbc, please refer to the offical document Step 3: Proof of concept connecting to SQL using pyodbc.

Connect Azure SQL Server using Service Principal

I want to connect the Azure SQL Database using Azure service principal through Python.
Please help me
I am able to connect it through ADF using service principal

There is a library Microsoft Azure Active Directory Authentication Library (ADAL) for Python to connect sql server.You could get it from here.
And in the wiki doc, you could find a tutorial about connecting to Azure SQL Database.
Also you could refer to this article, it has detailed steps to connect server.

It took me some time to figure this out, so I'll leave some code samples here in case it helps someone.
In my case, I had to connect to Synapse SQL Serverless from Databricks. Previously, I installed the driver "msodbcsql17" with this script:
%sh
#!/bin/bash
apt install unixodbc-dev
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install -y msodbcsql17
Then:
Get token:
from msal import ConfidentialClientApplication
creds = ConfidentialClientApplication(
client_id='<your_client_id>',
authority='https://login.microsoftonline.com/<your_tenant_id>',
client_credential= 'your_secret')
token = creds.acquire_token_for_client(scopes='https://database.windows.net//.default')
Encode token (more info here: https://www.linkedin.com/pulse/using-azure-ad-service-principals-connect-sql-from-python-andrade/):
import struct
SQL_COPT_SS_ACCESS_TOKEN = 1256
tokenb = bytes(token["access_token"], "UTF-8")
exptoken = b'';
for i in tokenb:
exptoken += bytes({i});
exptoken += bytes(1);
tokenstruct = struct.pack("=i", len(exptoken)) + exptoken;
With pyodbc, open a connection to the database using the token and execute a SQL statement:
import pyodbc
connString = 'DRIVER={ODBC Driver 17 for SQL Server};' \
+ 'SERVER=<your_server>;' \
+ 'DATABASE=<your_database>;'
conn = pyodbc.connect(connString, attrs_before = { SQL_COPT_SS_ACCESS_TOKEN:tokenstruct});
cursor = conn.cursor()
query="select name from sys.databases"
cursor.execute(query)
row = cursor.fetchall()

Look at this tutorial:Lesson Learned #49: Does Azure SQL Database support Azure Active Directory connections using Service Principals?
This tutorial teaches us connect the Azure SQL Database through AAD using Azure service principle, and it provides example code in Powershell and C#.
I didn't find the example code in Python. I think this tutorial may be helpful for you, so I want to share with you.
Hope this helps.