I'm trying to test a class that loads data from an SQL server given a query. To do this, I was instructed to use sqlite3. Now, the problem is that while the class manages to connect to the real database with ease, I'm struggling to connect with the temporary sqlite3 server that I create, as I cannot figure out what the connection string should look like. I'm using pyodbc in the class to connect with databases. So, has anyone got an idea on what the connection string should look like?
The class looks as follows:
import petl as etl
import pyodbc
class Loader:
"""
This is a class from which one can load data from an SQL server.
"""
def __init__(self, connection_string):
"""
This is the initialization file, and it requires the connection_string.
:param connection_string:
:type connection_string: str
:return:
"""
self.connection = pyodbc.connect(connection_string)
def loadFromSQL(self, query):
"""
This function loads the data according to the query passed in query.
:param query:
:type query: str
"""
self.originalTableETL = etl.fromdb(self.connection, query)
self.originalTablePD = etl.todataframe(self.originalTableETL)
And the temporary sqlite3 server is as follows
import sqlite3 as lite
con = lite.connect('test.db')
with con:
cur = con.cursor()
cur.execute("DROP TABLE IF EXISTS test_table")
cur.execute("CREATE TABLE test_table(col1 TEXT, col2 TEXT)")
cur.execute("INSERT INTO test_table VALUES('Hello', 'world!')")
So, what I wish to input is something like
tester = Loader('connection_string_goes_here')
tester.loadFromSQL("SELECT * FROM test_table")
EDIT
Okay, I've scoured the web a bit and found that a possible connection string is
"DRIVER={SQL Server};SERVER=localhost;DATABASE=test.db;Trusted_connection=yes". However, the connection times out after a while and returns the following error message:
pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect)')
Which I found strange as it's local and as I haven't specified any password. I've also tried specifying the exact path name to no avail.
Best,
Victor
Solved the problem! Downloaded an ODBC driver for SQLite from http://www.ch-werner.de/sqliteodbc/, and defined the connection string such as
"DRIVER={SQLite3 ODBC Driver};SERVER=localhost;DATABASE=test.db;Trusted_connection=yes"
And it worked, hope this helps people!
Related
I am using the connection string as below
params=parse.quote_plus("Driver={ODBC Driver 17 For SQL server};Server=tcp:server name,1433;database=database name;Encrypt=yes;TrustServerCertificate=no;Authentication=ActiveDirectoryIntegrated'
engine=sqlalchemy.create_engine("mssql:///?odbc_connect=%s" %params)
using the above connection string it is giving me the error
[Microsoft][ODBC Driver 17 for SQL server][SQL server]111214 an attempt to an attempt to complete the transaction has failed no corresponding transaction found
UPDATE
You can add connect_args, then try.
Please make sure you have same account login your windows pc and sql server.
engine = create_engine('mssql+pyodbc:///?odbc_connect=%s' % params, echo=True, connect_args={'autocommit': True})
PREVIOUS
You can consider to use Authentication=ActiveDirectoryPassword which be easier than Authentication=ActiveDirectoryIntegrated, and the code as below which is works for me.
Thank for Peter Pan's answer, for more details, you can refer his description. His answer has detailed usage of Authentication=ActiveDirectoryIntegrated in his description, I prefer Authentication=ActiveDirectoryPassword, so I posted my answer, you can refer to it.
How to connect to Azure sql database with python SQL alchemy using Active directory integrated authentication
from urllib import parse
from sqlalchemy import create_engine
your_user_name = 'pa**i#**a.onmicrosoft.com'
your_password_here = 'J***20'
connecting_string = 'Driver={ODBC Driver 17 for SQL Server};Server=tcp:yoursqlserver.database.windows.net,1433;Database=yoursqldb;Uid='+your_user_name+';Pwd='+your_password_here+';Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;Authentication=ActiveDirectoryPassword'
params = parse.quote_plus(connecting_string)
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection = engine.connect()
result = connection.execute("select 1+1 as res")
for row in result:
print("res:", row['res'])
connection.close()
I'm using sqlalchemy trying to connect to Teradata via ODBC as I need to be able to read/write to Teradata.
from sqlalchemy import create_engine
import sqlalchemy_teradata
user = 'user'
pasw='pasw'
host = 'host'
# connect
td_engine = create_engine("teradata://"+user+":"+pasw+"#"+host+"/?authentication=ODBC?driver=Teradata")
#execute sql
sql = "select * from table"
result = td_engine.execute(sql)
However I get the following error.
(teradata.api.DatabaseError) (0, '[28000] [Teradata][ODBC Teradata Driver] User Specified Mechanism for Logon is Not Available')
(Background on this error at: http://sqlalche.me/e/4xp6)
The link provided is not very informative unless I'm missing something. The error is from Teradata but I'm not sure what it actually means. It looks like it saying that I can't use ODBC? Any suggestions or alternatives?
I am trying to connect Azure SQL Database from Azure Machine Learning service, but I got the below error.
Please check Error: -
**('IM002', '[IM002] [unixODBC][Driver Manager]Data source name not found and no default driver specified (0) (SQLDriverConnect)')**
Please Check the below code that I have used for database connection: -
import pyodbc
class DbConnect:
# This class is used for azure database connection using pyodbc
def __init__(self):
try:
self.sql_db = pyodbc.connect(SERVER=<servername>;PORT=1433;DATABASE=<databasename>;UID=<username>;PWD=<password>')
get_name_query = "select name from contacts"
names = self.sql_db.execute(get_name_query)
for name in names:
print(name)
except Exception as e:
print("Error in azure sql server database connection : ", e)
sys.exit()
if __name__ == "__main__":
class_obj = DbConnect()
Is there any way to solve the above error? Please let me know if there is any way.
I'd consider using azureml.dataprep over pyodbc for this task (the API may change, but this worked last time I tried):
import azureml.dataprep as dprep
ds = dprep.MSSQLDataSource(server_name=<server-name,port>,
database_name=<database-name>,
user_name=<username>,
password=<password>)
You should then be able to collect the result of an SQL query in pandas e.g. via
dataflow = dprep.read_sql(ds, "SELECT top 100 * FROM [dbo].[MYTABLE]")
dataflow.to_pandas_dataframe()
Alternatively you can create SQL datastore and create a dataset from the SQL datastore.
Learn how:
https://learn.microsoft.com/en-us/azure/machine-learning/service/how-to-create-register-datasets#create-tabulardatasets
Sample code:
from azureml.core import Dataset, Datastore
# create tabular dataset from a SQL database in datastore
sql_datastore = Datastore.get(workspace, 'mssql')
sql_ds = Dataset.Tabular.from_sql_query((sql_datastore, 'SELECT * FROM my_table'))
#AkshayGodase Any particular reason that you want to use pyodbc?
I cannot get Python Turbodbc to connect to a Sql Server table, simple as that seems, to read or write user tables. However I have established ODBC connection, and can print a list of objects from it.
1 List objects from server to test connection. Seems to work:
from turbodbc import connect, make_options
options = make_options(prefer_unicode=True)
connection = connect(dsn='FPA', turbodbc_options=options)
cursor = connection.cursor()
cursor.execute('''SELECT * FROM sys.objects WHERE schema_id = SCHEMA_ID('dbo');''')
2 Simple Select: Does not work
cursor.execute('''SELECT * from [dbo].[Kits_Rec];''')
From #1 I get
From # 2 message: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid object name 'dbo.Kits_Rec'.
enter image description here
A SQL Server contains multiple databases, your dsn "FPA" probably doesn't specify the database name, so you are connecting to the master database instead of the database containing your Kits_Rec table.
Fix that dsn entry to specify the correct database, or use this syntax instead :
connection = connect(driver="MSSQL Driver",
server="hostname",
port="1433",
database="myDataBase",
uid="myUsername",
pwd="myPassword")
I'm trying to query an RDS (Postgres) database through Python, more specifically a Jupyter Notebook. Overall, what I've been trying for now is:
import boto3
client = boto3.client('rds-data')
response = client.execute_sql(
awsSecretStoreArn='string',
database='string',
dbClusterOrInstanceArn='string',
schema='string',
sqlStatements='string'
)
The error I've been receiving is:
BadRequestException: An error occurred (BadRequestException) when calling the ExecuteSql operation: ERROR: invalid cluster id: arn:aws:rds:us-east-1:839600708595:db:zprime
In the end, it was much simpler than I thought, nothing fancy or specific. It was basically a solution I had used before when accessing one of my local DBs. Simply import a specific library for your database type (Postgres, MySQL, etc) and then connect to it in order to execute queries through python.
I don't know if it will be the best solution since making queries through python will probably be much slower than doing them directly, but it's what works for now.
import psycopg2
conn = psycopg2.connect(database = 'database_name',
user = 'user',
password = 'password',
host = 'host',
port = 'port')
cur = conn.cursor()
cur.execute('''
SELECT *
FROM table;
''')
cur.fetchall()