I cannot get Python Turbodbc to connect to a Sql Server table, simple as that seems, to read or write user tables. However I have established ODBC connection, and can print a list of objects from it.
1 List objects from server to test connection. Seems to work:
from turbodbc import connect, make_options
options = make_options(prefer_unicode=True)
connection = connect(dsn='FPA', turbodbc_options=options)
cursor = connection.cursor()
cursor.execute('''SELECT * FROM sys.objects WHERE schema_id = SCHEMA_ID('dbo');''')
2 Simple Select: Does not work
cursor.execute('''SELECT * from [dbo].[Kits_Rec];''')
From #1 I get
From # 2 message: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid object name 'dbo.Kits_Rec'.
enter image description here
A SQL Server contains multiple databases, your dsn "FPA" probably doesn't specify the database name, so you are connecting to the master database instead of the database containing your Kits_Rec table.
Fix that dsn entry to specify the correct database, or use this syntax instead :
connection = connect(driver="MSSQL Driver",
server="hostname",
port="1433",
database="myDataBase",
uid="myUsername",
pwd="myPassword")
Related
I have a SQL script that joins two CTE's together, one CTE is from database1 and another CTE is from database2. It can be run successfully in SQL Server.
However, I'd like to establish a connection between the SQL Server to Python using pyodbc package (like below) so that I can read-in the results directly. Since we can only specify one database in the following code, how do I establish the connection if my SQL script contains two different databases?
conn = pyodbc.connect('Driver= {SQL Server Native Client 11.0};'
'Server= server;'
'Database = database1;'
'InitialCatalog=dbo;'
'Trusted_Connection=yes;')
query = open(file_path, 'r')
df = pd.read_sql_query(query.read(), conn)
query.close()
I am trying to create pyspark jdbc dataframe using LDAP connections from Oracle.
Below code is working for normal connection string based JDBC dataframe creation which worked.
creds = {"user":"USER_NAME",
"password":"PASSWORD",
"driver": "oracle.jdbc.OracleDriver"}
connection_string = "jdbc:oracle:thin:#//hostname.com:1521/myint.domain.com"
df = spark_session.read.jdbc(url=connection_string, table=query, properties=creds)
Now, our database configurations changed and we are supposed to use LDAP based authentication only.
Hence I tried changing connection_string as below.
connection_string = "myint"
But, it is throwing below issue.
py4j.protocol.Py4JJavaError: An error occurred while calling o51.jdbc.
: java.lang.NullPointerException
Without Spark, I tried connecting using cx_Oracle module(python module for connecting to Oracle) for testing and it worked.
Before:
host = 'myint.domain.com'
ip = 'jdbc:oracle:thin:#//hostname.com'
port = 1521
conn = cx_Oracle.makedsn(ip, port, service_name=host)
db = cx_Oracle.connect('USER_NAME', 'PASSWORD', conn) # giving the conn object
cursor = db.cursor()
cursor.execute("""select * from myschema.mytable fetch first 5 rows only""")
for row in cursor:
print(row)
After:
db = cx_Oracle.connect('USER_NAME', 'PASSWORD', "myint") # Now, just giving only the database
cursor = db.cursor()
cursor.execute("""select * from myschema.mytable fetch first 5 rows only""")
for row in cursor:
print(row)
I need to achieve the same in Pyspark JDBC dataframe by just passing only the database name. Please suggest what needs to be done.
I hope this question can be applicable for scala spark as well.
I'm getting this error when trying to update a db2 database that is a linked server on our SQL Server db.
ERROR:root:('42000', '[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]The requested operation could not be performed because OLE DB provider "IBMDA400" for linked server "iSeries" does not support the required transaction interface. (7390) (SQLExecDirectW)')
I am connecting to sql server via pyodbc and can run sql scripts with no issues. Here is the sql I get the error with
sql3 = " exec ('UPDATE SVCEN2DEV.SRVMAST SET SVRMVD = ? WHERE svtype != ''*DCS-'' AND svcid = ? and svacct = ? ') AT [iSeries]"
db.execute(sql3, (row[2],srvid,row[0]))
db.commit()
And just in case here is my connection string using pyodbc:
conn = pyodbc.connect("DRIVER={SQL Server};SERVER="+ Config_Main.dbServer +";DATABASE="+ Config_Main.encludeName +";UID="+ Config_Main.encludeUser +";PWD=" + Config_Main.encludePass)
db = conn.cursor()
Also note that this query runs just fine in SSMS. I have also tried the openquery method but had no luck. Any ideas?
Python's DB API 2.0 specifies that, by default, connections should open with autocommit "off". This results in all database operations being performed in a transaction that must be explicitly committed (or rolled back) in the Python code.
When a pyodbc connection with autocommit = False (the default) sends an UPDATE to the SQL Server, that UPDATE is enclosed in a Local Transaction managed by SQL Server. When the SQL Server determines that the target table is on a Linked Server it tries to promote the transaction to a Distributed Transaction managed by MSDTC. If the connection technology used to manage the Linked Server does not support Distributed Transactions then the operation will fail.
This issue can often be avoided by ensuring that the pyodbc connection has autocommit enabled, either by
cnxn = pyodbc.connect(conn_str, autocommit=True)
or
cnxn = pyodbc.connect(conn_str)
cnxn.autocommit = True
That will send each SQL statement individually, without being wrapped in an implicit transaction.
I'm trying to test a class that loads data from an SQL server given a query. To do this, I was instructed to use sqlite3. Now, the problem is that while the class manages to connect to the real database with ease, I'm struggling to connect with the temporary sqlite3 server that I create, as I cannot figure out what the connection string should look like. I'm using pyodbc in the class to connect with databases. So, has anyone got an idea on what the connection string should look like?
The class looks as follows:
import petl as etl
import pyodbc
class Loader:
"""
This is a class from which one can load data from an SQL server.
"""
def __init__(self, connection_string):
"""
This is the initialization file, and it requires the connection_string.
:param connection_string:
:type connection_string: str
:return:
"""
self.connection = pyodbc.connect(connection_string)
def loadFromSQL(self, query):
"""
This function loads the data according to the query passed in query.
:param query:
:type query: str
"""
self.originalTableETL = etl.fromdb(self.connection, query)
self.originalTablePD = etl.todataframe(self.originalTableETL)
And the temporary sqlite3 server is as follows
import sqlite3 as lite
con = lite.connect('test.db')
with con:
cur = con.cursor()
cur.execute("DROP TABLE IF EXISTS test_table")
cur.execute("CREATE TABLE test_table(col1 TEXT, col2 TEXT)")
cur.execute("INSERT INTO test_table VALUES('Hello', 'world!')")
So, what I wish to input is something like
tester = Loader('connection_string_goes_here')
tester.loadFromSQL("SELECT * FROM test_table")
EDIT
Okay, I've scoured the web a bit and found that a possible connection string is
"DRIVER={SQL Server};SERVER=localhost;DATABASE=test.db;Trusted_connection=yes". However, the connection times out after a while and returns the following error message:
pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect)')
Which I found strange as it's local and as I haven't specified any password. I've also tried specifying the exact path name to no avail.
Best,
Victor
Solved the problem! Downloaded an ODBC driver for SQLite from http://www.ch-werner.de/sqliteodbc/, and defined the connection string such as
"DRIVER={SQLite3 ODBC Driver};SERVER=localhost;DATABASE=test.db;Trusted_connection=yes"
And it worked, hope this helps people!
I am using PyODBC to back up my database, using following code:
SQL_command = """
BACKUP DATABASE [MyDatabase]
TO DISK = N'D:\MSSQL\BACKUP\MyDatabase_20141212.bak' WITH
NOFORMAT
, NOINIT
, NAME = N'MyDatabase_20141212'
, SKIP
, REWIND
, NOUNLOAD
, STATS = 10
"""
conn.cursor.execute(SQL_command)
conn.cursor.commit()
The above code give me an error message:
pyodbc.ProgrammingError:
('42000', '[42000] [Microsoft][ODBC SQL
Server Driver][SQL Server]Cannot perform a backup or restore operation
within a transaction. (3021) (SQLExecDirectW);
[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]BACKUP DATABASE is
terminating abnormally. (3013)')
I tried to run the SQL code in SQL Server, and it works fine.
May I know what is wrong with my code?
Many thanks.
Your error says
Cannot perform a backup or restore operation within a transaction.
Transactions are started by default in pyodbc, so how do you execute a query without creating a transaction? Simply turn on autocommit:
conn.autocommit = true
// do stuff
conn.autocommit = false
The pyodbc FAQ has an entry about this.
By default all pyodbc connections start with a transaction.
You need to turn autocommit on - using the autocommit keyword to the connect function:
conn.autocommit = true
The other answers are correct. You do need to set autocommit. However, the transaction will complete but the backup won't actually be taken because of a quirk in SQL Server and the way it returns status messages for backup and restore operations.
To work around this you need to loop through these return messages until none remain:
SQL_command = """
BACKUP DATABASE [MyDatabase]
TO DISK = N'D:\MSSQL\BACKUP\MyDatabase_20141212.bak' WITH
NOFORMAT
, NOINIT
, NAME = N'MyDatabase_20141212'
, SKIP
, REWIND
, NOUNLOAD
, STATS = 10
"""
conn.cursor.execute(SQL_command)
while conn.cursor.nextset():
pass
conn.cursor.close()