Trying to connect a SQL Synapse database with read-only access, Followed the advice re autocommit here https://docs.sqlalchemy.org/en/14/dialects/mssql.html
However receive the following error...
SAWarning: Could not fetch transaction isolation level, tried views:
('sys.dm_exec_sessions', 'sys.dm_pdw_nodes_exec_sessions'); final
error was: ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL
Server][SQL Server]User does not have permission to perform this
action. (6004) (SQLExecDirectW)')
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
conn_string = "DRIVER={ODBC Driver 17 for SQL Server};" \
"SERVER=mydatabase.database.windows.net;" \
"DATABASE=somedbname;" \
"AUTHENTICATION=ActiveDirectoryInteractive;" \
"UID=gary#test.com;"
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": conn_string, "autocommit": "true"})
print(connection_url)
engine = create_engine(connection_url).execution_options(
isolation_level="AUTOCOMMIT"
)
with engine.connect() as con:
con.execute("SELECT 1")
Must the read-only uses still require access to these views?
Related
I am trying to connect a database using DSN, when I setup the database it did not asked for any username and password, instead i provided the DSN name and selected the security authentication as "Azure Active Directory Integrated Authentication".
I can connect like this
conn = pyodbc.connect(DSN='EAP') # EAP is my DSN name, and it works fine
df = pd.read_sql(myquery, conn)
conn.close()
# this query does not fail, just gives the warnings.
# I would like to use engine method, instead of this connection method.
warning:
pandas only support SQLAlchemy connectable(engine/connection)
ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2
objects are not tested, please consider using SQLAlchemy
Question: How to get the same query using engine, instead of conn?
My Attempts
import sqlalchemy
from sqlalchemy.engine import URL
from sqlalchemy import create_engine
#conn_str = "DRIVER={ODBC Driver 17 for SQL Server};DSN=EAP"
conn_str = "DRIVER={msodbcsql17.dll};DSN='EAP'"
conn_url = URL.create("mssql+pyodbc", query={"odbc_connect": conn_str})
engine = create_engine(conn_url)
# engine = sqlalchemy.create_engine('mysql+pyodbc://EAP')
parent = pd.read_sql(myquery, engine)
Gives ERROR:
InterfaceError: (pyodbc.InterfaceError) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
(Background on this error at: https://sqlalche.me/e/14/rvf5)
References:
There are literally infinite materials similar to this question, but none of them could solve my case.
https://docs.sqlalchemy.org/en/14/dialects/mssql.html#module-sqlalchemy.dialects.mssql.pyodbc
Pandas to_sql not working with SQL Alchemy connection
sqlalchemy, specify database name with DSN
I am trying to use sqlalchemy-access library to insert a data frame into an access database:
The code that I have is:
import msaccessdb
import pyodbc
import pandas as pd
print(pyodbc.version)
db_file = r'database\sampledatabase.accdb'
input_csv_file= r'database\sample_data.csv'
print(input_csv_file)
msaccessdb.create(db_file)
cnxn_str = (
r'DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};'
r'DBQ={};'
r'ExtendedAnsiSQL=1;'.format(db_file)
)
print(cnxn_str)
cnxn = pyodbc.connect(cnxn_str,autocommit=True)
input_data=pd.read_csv(input_csv_file)
input_data.to_sql('sample_table', cnxn, index=False, if_exists='replace')
cnxn.close()
but when I run this code, I am getting this error:
Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC Microsoft Access Driver] The Microsoft Access database engine cannot find the input table or query 'sqlite_master'. Make sure it exists and that its name is spelled correctly. (-1305) (SQLExecDirectW)")
the error is generated when I am trying to run this line:
input_data.to_sql('sample_table', cnxn, index=False, if_exists='replace')
what is wrong with this code and how can I fix it?
Edit 1
Based on the comments and this post, I changed the code to this:
import msaccessdb
import pandas as pd
from sqlalchemy import create_engine
import urllib
db_file = r'database\sampledatabase.accdb'
input_csv_file= r'database\sample_data.csv'
msaccessdb.create(db_file)
cnxn_str = (
r'DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};'
r'DBQ={};'
r'ExtendedAnsiSQL=1;'.format(db_file)
)
params = urllib.parse.quote_plus(cnxn_str)
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
print(cnxn_str)
create_engine("access+pyodbc://#your_dsn")
input_data=pd.read_csv(input_csv_file)
input_data.to_sql('sample_table', engine, index=False, if_exists='replace')
but I am still getting error:
(pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Microsoft Access Driver] Undefined function 'schema_name' in expression. (-3102) (SQLExecDirectW)")
[SQL: SELECT schema_name()]
(Background on this error at: http://sqlalche.me/e/14/f405)
Edit 2
copied what was stated here: https://github.com/gordthompson/sqlalchemy-access/wiki/Getting-Connected#connecting-with-an-odbc-connection-string
so the code looks this now:
import msaccessdb
import pandas as pd
from sqlalchemy import create_engine
import urllib
db_file = r'database\sampledatabase.accdb'
input_csv_file= r'database\sample_data.csv'
msaccessdb.create(db_file)
connection_string = (
r'DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};'
r'DBQ=database\sample_data.csv;'
r'ExtendedAnsiSQL=1;'
)
connection_uri = f"access+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string)}"
engine = create_engine(connection_uri)
input_data=pd.read_csv(input_csv_file)
input_data.to_sql('sample_table', engine, index=False, if_exists='replace')
but I am still getting this error:
(pyodbc.Error) ('IM012', '[IM012] [Microsoft][ODBC Driver Manager] DRIVER keyword syntax error (0) (SQLDriverConnect)')
(Background on this error at: http://sqlalche.me/e/14/dbapi)
it should be noted that I have installed Access on my system.
You have two sets of curly brackets surrounding the driver name …
connection_string = (
r'DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};'
r'DBQ=database\sampledatabase.accdb;'
r'ExtendedAnsiSQL=1;'
)
connection_uri = f"access+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string)}"
engine = create_engine(connection_uri)
… but because you are using an r'string' (not an f'string') you should only have one pair of curly brackets:
connection_string = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=database\sampledatabase.accdb;'
r'ExtendedAnsiSQL=1;'
)
connection_uri = f"access+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string)}"
engine = create_engine(connection_uri)
I have a huge table (147 columns) and I would like to know if it is possible to create the table in SQL server from my pandas dataframe so I don't have to CREATE TABLE for 147 columns.
I am trying to follow what is answered in most of related questions:
params = urllib.parse.quote_plus("DRIVER={ODBC Driver 17 for SQL Server};SERVER=DESKTOP-LFOSSEF;DATABASE=test;UID=xxxx;PWD=xxx")
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection = engine.raw_connection()
df.to_sql("table_name", connection,index=False)
The user and password work because that's what I am using to sign in into sqlserver.
When I run this, I get:
Execution failed on sql 'SELECT name FROM sqlite_master WHERE
type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC Driver
17 for SQL Server][SQL Server]Invalid object name 'sqlite_master'.
(208) (SQLExecDirectW); [42S02] [Microsoft][ODBC Driver 17 for SQL
Server][SQL Server]Statement(s) could not be prepared. (8180)")
If I remove the connection = engine.raw_connection() and use engine directly, I get:
AttributeError: 'Engine' object has no attribute 'cursor'
Any idea what is wrong? Do I need to create an empty table first? I've been troubleshooting for hours and can't find any reason why this isnt working.
Do it like this.
import pyodbc
engine = "mssql+pyodbc://Your_Server_Name/Your_DB_Name?driver=SQL Server Native Client 11.0?trusted_connection=yes"
df.to_sql(x, engine, if_exists='append', index=True)
df = name of your dataframe & x = name of your table in SQL Server
I have a Synapse database from which i am reading some tables to create a dataframe. I am using the below to connect and read the database tables:
server = 'gbs-marcus-dev-dw.database.windows.net'
database = 'romeo'
username = 'romeouser'
password = 'romeopass'
driver = 'ODBC Driver 17 for SQL Server'
cnxn = pyodbc.connect('DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
# select command
query = ''' SELECT * FROM dbo.Masterdata''';
df_input_orig = pd.read_sql(query, cnxn)
pyodbc.drivers()
['SQL Server', 'ODBC Driver 17 for SQL Server']
Post creating the dataframes, i am trying to write the resultant dataframes back to the DB and i am following the below two approches both of which give different errors.
Approch 1:
engine = sqlalchemy.create_engine("mssql+pyodbc://romeouser:romeopass#gbs-marcus-dev-dw.database.windows.net/romeo?driver=ODBC+Driver+17+for+SQL+Server",echo=False)
Insight_instance.to_sql(name='Insight_Instance',con=engine, index=False, if_exists='append')
cursor.execute('commit')
Error for approch 1:
ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]111214;An attempt to complete a transaction has failed. No corresponding transaction found. (111214) (SQLEndTran)')
(Background on this error at: http://sqlalche.me/e/13/f405)
Approch 2:
import pyodbc
import sqlalchemy
import urllib
params = urllib.parse.quote_plus("DRIVER=driver;SERVER=gbs-marcus-dev-dw.database.windows.net;DATABASE=romeo;UID=romeouser;PWD=romeopass")
engine = sqlalchemy.create_engine("mssql+pyodbc://romeouser:romeopass#gbs-marcus-dev-dw.database.windows.net/romeo",echo=False)
engine.connect()
Insight_instance.to_sql(name='Insight_Instance',con=engine, index=False, if_exists='append')
Error from approch 2:
InterfaceError: (pyodbc.InterfaceError) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
(Background on this error at: http://sqlalche.me/e/13/rvf5)
I am quite new to this and have no clue where or what i am doing wrong.
I want to copy tables from MS SQL server to MS access using sqlalchemy.
But I don't know how to do it. Please help. Here is the code.
from sqlalchemy import create_engine
from sqlalchemy import select, Table, MetaData
import sqlalchemy_access
import urllib
# =============================================================================
# # Create tables from mssql
# =============================================================================
connection_string_ms = (
r'Driver={SQL Server};'
r'Server=myserver;'
r'Database=mydb;'
r'Trusted_Connection=yes;'
)
connection_url_ms = f'mssql+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string_ms)}'
engine_ms = create_engine(connection_url_ms)
# Create a MetaData instance
metadata_ms = MetaData(bind = engine_ms)
address = Table('ADDRESS', metadata_ms, autoload = True, autoload_with = engine_ms)
# =============================================================================
# # Paste table into access database
# =============================================================================
connection_string_access = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=myaccessdb.accdb;'
r'ExtendedAnsiSQL=1;'
)
connection_url = f'access+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string_access)}'
engine = create_engine(connection_url)
address.create(engine)
Then error pops up
ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][ODBC Microsoft Access Driver] Syntax error in field definition. (-3553) (SQLExecDirectW)')
Thank you.