I am trying to connect to a mssql instance in cloud sql in a cloud function. I have gone through the necessary steps of setting up a private IP, serverless VPC connector, and connecting my function to the VPC. I have been able to connect to the instance in nodejs but python suits my current needs more. The error I'm getting in the logs is:
pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server'
From all the examples I have read it does not appear that you need to import them or anything.
This is my process of connecting and executing a simple request.
import sqlalchemy
import pyodbc
def hello_world(request):
# connect_simple()
db = connect_tcp_socket()
a = execute_request(db)
return a
def connect_tcp_socket() -> sqlalchemy.engine.base.Engine:
db_host = 'my_private_ip'
db_user = 'my_db_user'
db_pass = 'my_db_pass'
db_name = 'my_db_name'
db_port = 'my_db_port'
connection_string = 'DRIVER={ODBC Driver 17 for SQL Server};SERVER='+db_host+';PORT='+db_port+'DATABASE='+db_name+';UID='+db_user+';PWD='+ db_pass+';Encrypt=no'
connection_url = sqlalchemy.engine.url.URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = sqlalchemy.create_engine(
connection_url
)
def execute_request(db: sqlalchemy.engine.base.Engine):
print('ok')
with db.connect() as conn:
result = conn.execute('SELECT ##VERSION')
barray= []
for row in result:
barray.append(row)
return barray
I'd recommend using the Cloud SQL Python Connector to connect to Cloud SQL from Python as it will not require the ODBC driver and is much easier to use within Cloud Functions/Cloud Run.
Just replace your connect_tcp_socket with the below connect_with_connector function.
from google.cloud.sql.connector import Connector, IPTypes
import pytds
import sqlalchemy
def connect_with_connector() -> sqlalchemy.engine.base.Engine:
def getconn() -> pytds.Connection:
with Connector() as connector:
conn = connector.connect(
"project-id:region:instance-name", # Cloud SQL connection name
"pytds",
user="my-user",
password="my-password",
db="my-database",
ip_type=IPTypes.PRIVATE
)
return conn
engine = sqlalchemy.create_engine(
"mssql+pytds://localhost",
creator=getconn,
)
return engine
You can find a code sample for the Python Connector similar to the one you are using for establishing a TCP connection.
Note: Pytds driver is not super great with error handling. If you see the OSError: [Errno 9] Bad file descriptor error it usually means your database user is missing proper permissions and should grant them the necessary grants from a root user.
Your requirements.txt should include the following:
cloud-sql-python-connector
SQLAlchemy
python-tds
sqlalchemy-pytds
There is also an interactive getting started Colab Notebook that will walk you through using the Python Connector without you needing to change a single line of code!
It makes connecting to Cloud SQL both easy and secure from Cloud Functions.
Related
So I am trying to communicate to a Google Cloud SQL Server that I have created with an external python program that I have written in VS Code but I don't know where to begin. Any help will be useful.
I'd recommend using the Cloud SQL Python Connector to manage your connections to Cloud SQL. It supports the pytds driver and should help resolve your troubles for connecting to a SQL Server instance from a Python application.
from google.cloud.sql.connector import connector
import sqlalchemy
# configure Cloud SQL Python Connector properties
def getconn() ->:
conn = connector.connect(
"PROJECT:REGION:INSTANCE",
"pytds",
user="YOUR_USER",
password="YOUR_PASSWORD",
db="YOUR_DB"
)
return conn
# create connection pool to re-use connections
pool = sqlalchemy.create_engine(
"mssql+pytds://localhost",
creator=getconn,
)
# query or insert into Cloud SQL database
with pool.connect() as db_conn:
# query database
result = db_conn.execute("SELECT * from my_table").fetchall()
# Do something with the results
for row in result:
print(row)
For more detailed examples and additional params refer to the README of the repository.
I think you can be inspired by this :Python django
"Run the app on your local computer"
I am using the connection string as below
params=parse.quote_plus("Driver={ODBC Driver 17 For SQL server};Server=tcp:server name,1433;database=database name;Encrypt=yes;TrustServerCertificate=no;Authentication=ActiveDirectoryIntegrated'
engine=sqlalchemy.create_engine("mssql:///?odbc_connect=%s" %params)
using the above connection string it is giving me the error
[Microsoft][ODBC Driver 17 for SQL server][SQL server]111214 an attempt to an attempt to complete the transaction has failed no corresponding transaction found
UPDATE
You can add connect_args, then try.
Please make sure you have same account login your windows pc and sql server.
engine = create_engine('mssql+pyodbc:///?odbc_connect=%s' % params, echo=True, connect_args={'autocommit': True})
PREVIOUS
You can consider to use Authentication=ActiveDirectoryPassword which be easier than Authentication=ActiveDirectoryIntegrated, and the code as below which is works for me.
Thank for Peter Pan's answer, for more details, you can refer his description. His answer has detailed usage of Authentication=ActiveDirectoryIntegrated in his description, I prefer Authentication=ActiveDirectoryPassword, so I posted my answer, you can refer to it.
How to connect to Azure sql database with python SQL alchemy using Active directory integrated authentication
from urllib import parse
from sqlalchemy import create_engine
your_user_name = 'pa**i#**a.onmicrosoft.com'
your_password_here = 'J***20'
connecting_string = 'Driver={ODBC Driver 17 for SQL Server};Server=tcp:yoursqlserver.database.windows.net,1433;Database=yoursqldb;Uid='+your_user_name+';Pwd='+your_password_here+';Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;Authentication=ActiveDirectoryPassword'
params = parse.quote_plus(connecting_string)
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection = engine.connect()
result = connection.execute("select 1+1 as res")
for row in result:
print("res:", row['res'])
connection.close()
Not able to connect to Azure DB. I get the following error while connecting via Python.
I'm able to connect to my usual SQL environment
import pandas as pd
import pymssql
connPDW = pymssql.connect(host=r'dwprd01.database.windows.net', user=r'internal\admaaron',password='',database='')
connPDW.autocommit(True)
cursor = connPDW.cursor()
conn.autocommit(True)
cursor = conn.cursor()
sql = """
select Top (10) * from TableName
"""
cursor.execute(sql);
Run without errors.
Just according to your code, there is an obvious issue of connecting Azure SQL Database by pymssql package in Python which use the incorrect user format and lack of the values of password and database parameters.
Please follow the offical document Step 3: Proof of concept connecting to SQL using pymssql carefully to change your code correctly.
If you have an instance of Azure SQL Database with the connection string of ODBC, such as Driver={ODBC Driver 13 for SQL Server};Server=tcp:<your hostname>.database.windows.net,1433;Database=<your database name>;Uid=<username>#<host>;Pwd=<your_password>;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30; show in the Connection strings tab of your SQL Database on Azure portal.
Then, your code should be like below
hostname = '<your hostname>'
server = f"{hostname}.database.windows.net"
username = '<your username>'
user = f"{username}#{hostname}"
password = '<your password>'
database = '<your database name>'
import pymssql
conn = pymssql.connect(server=server, user=user, password=password, database=database)
Meanwhile, just additional note for the version of Azure SQL Database and MS SQL Server are 2008+ like the latest Azure SQL Database, you should use the ODBC Driver connection string which be started with DRIVER={ODBC Driver 17 for SQL Server};, not 13 show in the connection string of Azure portal if using ODBC driver for Python with pyodbc, please refer to the offical document Step 3: Proof of concept connecting to SQL using pyodbc.
I am trying to connect to the SQL Server database using Python3.4
This is the code that works for me
cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER=DESKTOP-GDM2HQ17\SQLEXPRESS;DATABASE=pyconnect;Trusted_Connection=yes')
and I login into my Management studio - database using Windows connection.
Here is the code, which is not working for me :
cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER=DESKTOP-GDM2HQ17\SQLEXPRESS;DATABASE=pyconnect;UID=DESKTOP-GDM2HQ17\sid;PWD=123')
Kindly share your thoughts on where I am going wrong.
There are two SQL Server Authentication modes:
1, Connecting Through Windows Authentication:
When a user connects through a Windows user account, SQL Server validates the account name and password using the Windows principal token in the operating system.
2, Connecting Through SQL Server Authentication:
When using SQL Server Authentication, logins are created in SQL Server that are not based on Windows user accounts. Both the user name and the password are created by using SQL Server and stored in SQL Server.
Your first code is working as it is Connecting Through Windows Authentication.
Your second code is not working as it is trying to find the credentials (login and password) which are stored in SQL Server, but the credentials is not created in SQL server.
Moreover, you could refer official doc to know how to Change Server Authentication Mode.
Hope it will help you.
DRIVER='{SQL Server}' works
below code is tested.....
import pyodbc
con = pyodbc.connect('Driver={SQL Server};'
'Server=LAPTOP-PPDS6BPG;'
'Database=training;'
'Trusted_Connection=yes;')
cursor = con.cursor()
sql_query = 'SELECT * FROM Students'
cursor.execute(sql_query)
for row in cursor:
print(row)
This works fine for me better than any other I could find
import pyodbc
import pandas as pd
conn = pyodbc.connect('Driver={SQL Server};'
'Server=10.****;'
'Database=Ma**;'
'UID=sql**;'
'PWD=sql**;')
cursor = conn.cursor()
sql = """\
EXEC [dbo].[GetNewPayment] #Login=?, #PasswordMD5=?, #RevokeTimeFrom=?, #RevokeTimeTo=? Status=?
"""
params = ('a***', 'c2ca***', '2021-05-01','2021-05-02', '3')
cursor.execute(sql, params)
I'm trying to connect to a SQL Server 2012 database using SQLAlchemy (with pyodbc) on Python 3.3 (Windows 7-64-bit). I am able to connect using straight pyodbc but have been unsuccessful at connecting using SQLAlchemy. I have dsn file setup for the database access.
I successfully connect using straight pyodbc like this:
con = pyodbc.connect('FILEDSN=c:\\users\\me\\mydbserver.dsn')
For sqlalchemy I have tried:
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://c/users/me/mydbserver.dsn/mydbname')
The create_engine method doesn't actually set up the connection and succeeds, but
iIf I try something that causes sqlalchemy to actually setup the connection (like engine.table_names()), it takes a while but then returns this error:
DBAPIError: (Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect)') None None
I'm not sure where thing are going wrong are how to see what connection string is actually being passed to pyodbc by sqlalchemy. I have successfully using the same sqlalchemy classes with SQLite and MySQL.
The file-based DSN string is being interpreted by SQLAlchemy as server name = c, database name = users.
I prefer connecting without using DSNs, it's one less configuration task to deal with during code migrations.
This syntax works using Windows Authentication:
engine = sa.create_engine('mssql+pyodbc://server/database')
Or with SQL Authentication:
engine = sa.create_engine('mssql+pyodbc://user:password#server/database')
SQLAlchemy has a thorough explanation of the different connection string options here.
In Python 3 you can use function quote_plus from module urllib.parse to create parameters for connection:
import urllib
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
In order to use Windows Authentication, you want to use Trusted_Connection as parameter:
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"Trusted_Connection=yes")
In Python 2 you should use function quote_plus from library urllib instead:
params = urllib.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")
I have an update info about the connection to MSSQL Server without using DSNs and using Windows Authentication. In my example I have next options:
My local server name is "(localdb)\ProjectsV12". Local server name I see from database properties (I am using Windows 10 / Visual Studio 2015).
My db name is "MainTest1"
engine = create_engine('mssql+pyodbc://(localdb)\ProjectsV12/MainTest1?driver=SQL+Server+Native+Client+11.0', echo=True)
It is needed to specify driver in connection.
You may find your client version in:
control panel>Systems and Security>Administrative Tools.>ODBC Data
Sources>System DSN tab>Add
Look on SQL Native client version from the list.
Just want to add some latest information here:
If you are connecting using DSN connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#SOME_DSN")
If you are connecting using Hostname connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#HOST_IP:PORT/DATABASENAME?driver=SQL+Server+Native+Client+11.0")
For more details, please refer to the "Official Document"
import pyodbc
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://ServerName/DatabaseName?driver=SQL+Server+Native+Client+11.0',echo = True)
This works with Windows Authentication.
I did different and worked like a charm.
First you import the library:
import pandas as pd
from sqlalchemy import create_engine
import pyodbc
Create a function to create the engine
def mssql_engine(user = os.getenv('user'), password = os.getenv('password')
,host = os.getenv('SERVER_ADDRESS'),db = os.getenv('DATABASE')):
engine = create_engine(f'mssql+pyodbc://{user}:{password}#{host}/{db}?driver=SQL+Server')
return engine
Create a variable with your query
query = 'SELECT * FROM [Orders]'
Execute the Pandas command to create a Dataframe from a MSSQL Table
df = pd.read_sql(query, mssql_engine())