We Keep Coding

Python Exception No Description with Pandas read_sql

Using Python, I'm trying to read a table from SQL Server and then insert the data into a table in Access. The best way I've found to do this is using the pandas dataframe. I wrote up a program that reads a SQL Server table into a dataframe like so:
dataframe = pandas.read_sql(selectSql, srcConn)
And it works great on a ~209MB table. When I try it on a ~1,116MB table it throws an exception with no description. I'm guessing it has to do with the size of the table it's reading in (it would be nice if it said that). I know Access can only hold 2GB but there is plenty of room left in it and it doesn't even get to the part where it writes to Access before throwing the error.
Is there any way to fix this for larger tables? Is there a better way I should be copying tables from SQL Server 2008 R2 to Access 2016 using Python? I have 16GB of RAM on Win10 64-bit so that shouldn't be a problem. I've tried 32-bit Python 3.7 and 64-bit Python 3.6 to no avail. I tried SSIS first but it crashes my entire Visual Studio whenever I try to open a package with a connection to Access.
UPDATE:
I followed Gord's advice below and now my code looks like this:
access_cnxn_str = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=' + access_db + ';'
)
sqls_cnxn_str = (
r'DRIVER=ODBC Driver 13 for SQL Server;'
r'SERVER=' + sqls_server + ';'
r'DATABASE=' + sqls_db + ';'
r'UID=' + sqls_username + ';'
r'PWD=' + sqls_password + ';'
)
This connection works by itself:
sqls_cnxn = pyodbc.connect(sqls_cnxn_str)
And this connection works by itself:
pyodbc.pooling = False
access_cnxn = pyodbc.connect(access_cnxn_str, autocommit = True)
But this is throwing an error:
access_cnxn.execute(f"SELECT * INTO {access_table} FROM [ODBC;{sqls_cnxn_str}].{sqls_table}")
The error thrown:
Message=('HY000', "[HY000] [Microsoft][ODBC Microsoft Access Driver]
ODBC--connection to 'ODBC Driver 13 for SQL ServerSERVERNAME' failed.
(-2001) (SQLExecDirectW)")
Source=C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py
StackTrace: File
"C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
57, in sync_table
dest_cnxn.execute(f"SELECT * INTO {access_table} FROM [ODBC;{sqls_cnxn_str}].{sqls_table}") File
"C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
121, in main
sync_table('', sqls_table, get_access_cnxn(), access_table) File "C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
124, in
main()
SERVERNAME in the error is the name of the server on which SQL Server resides. Not sure why it jammed it up against the driver name in the error.
Any ideas?
UPDATE 2:
It turns out my Access is 32-bit. This still doesn't explain why it won't connect as I was originally using Python 3.7 32-bit.
Thanks.

I was able to get the Access Database Engine to pull a table from SQL Server and create a copy in the Access database by simply doing
pyodbc.pooling = False # required
cnxn = pyodbc.connect("DSN=myAccessDb", autocommit=True)
cnxn.execute("SELECT * INTO access_tbl FROM [ODBC;DSN=SQLmyDb].sql_server_tbl")
where SQLmyDb is the ODBC DSN for my SQL Server instance.
Update
Just tested to confirm that DSN-less connection strings also work:
pyodbc.pooling = False # required
access_cnxn_str = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=C:\__tmp\test.accdb;'
)
cnxn = pyodbc.connect(access_cnxn_str, autocommit=True)
sql_cnxn_str = (
r'DRIVER=ODBC Driver 17 for SQL Server;'
r'SERVER=(local)\SQLEXPRESS;'
r'DATABASE=myDb;'
r'Trusted_Connection=Yes;'
)
cnxn.execute(f"SELECT * INTO access_tbl FROM [ODBC;{sql_cnxn_str}].sql_server_tbl")

How to place range/table lock at 2 tables at MS SQL server via pyodbc

I am now processing 2 tables at the same time by pyodbc.
Before finishing processing, I would like to place locks on the 2 tables so that nobody can change the 2 tables before I finish.
How can I do that?
I tried the below, but failed with an error.
import pyodbc
conn = pyodbc.connect("conn_str")
conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;")
conn.execute("begin transaction trans;")
cur1 = conn.execute("select * from tbl1;")
cur2 = conn.execute("select * from tbl2;")
... some processing at cur1 and cur2 at python ...
conn.execute("commit transaction trans;")
However, the following error comes up when execution "cur2 = conn.execute("select * from tbl2;")"
[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt (0) (SQLExecDirectW)
Is there anything wrong with my code? Highly appreciated for any help.

Originally, SQL Server ODBC was limited to one active hstmt (the ODBC equivalent of a pyodbc.cursor) per connection. Later on, Microsoft added the MARS (Multiple Active Result Sets) feature to SQL Server ODBC, but that feature is "off" by default.
So this code
import pyodbc
import sys
print(f"Python version {sys.version}") # Python version 3.6.4 ...
print(f"pyodbc version {pyodbc.version}") # pyodbc version 4.0.24
conn_str = (
r'DRIVER=ODBC Driver 17 for SQL Server;'
r'SERVER=.\SQLEXPRESS;'
r'DATABASE=myDb;'
r'Trusted_Connection=yes;'
)
cnxn = pyodbc.connect(conn_str, autocommit=True)
cnxn.set_attr(pyodbc.SQL_ATTR_TXN_ISOLATION, pyodbc.SQL_TXN_SERIALIZABLE)
cnxn.autocommit = False # enable transactions
cur1 = cnxn.execute("SELECT 1 AS x UNION ALL SELECT 2 AS x")
cur2 = cnxn.execute("SELECT 'foo' AS y UNION ALL SELECT 'bar' AS y")
print(cur1.fetchone())
print(cur2.fetchone())
print(cur1.fetchone())
print(cur2.fetchone())
fails with
Traceback (most recent call last):
File "C:/Users/Gord/PycharmProjects/py3pyodbc_demo/main.py", line 18, in <module>
cur2 = cnxn.execute("SELECT 'foo' AS y UNION ALL SELECT 'bar' AS y")
pyodbc.Error: ('HY000', '[HY000] [Microsoft][ODBC Driver 17 for SQL Server]Connection is busy with results for another command (0) (SQLExecDirectW)')
However, if we add MARS_Connection=yes to the connection string
conn_str = (
r'DRIVER=ODBC Driver 17 for SQL Server;'
r'SERVER=.\SQLEXPRESS;'
r'DATABASE=myDb;'
r'Trusted_Connection=yes;'
r'MARS_Connection=yes;'
)
then the code works.
Unfortunately in your case you are using the ancient DRIVER=SQL Server which is too old to support MARS_Connection=yes so your options are
use a newer version of the SQL Server ODBC driver, or
open two separate connections, one for each cursor.

Connect python to Sybase IQ

First of all thank you for your help.
I was trying to retrieve some data from a sybase IQ database using python, but I can not make it.
I've tried with the following code( from https://github.com/sqlanywhere/sqlanydb):
import sqlanydb
conn = sqlanydb.connect(uid='dba', pwd='sql', eng='demo', dbn='demo' )
curs = conn.cursor()
curs.execute("select 'Hello, world!'")
print( "SQL Anywhere says: %s" % curs.fetchone() )
curs.close()
conn.close()
Unfotunately it gives me the following error:
InterfaceError: ('Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
Does anyone know how to fix it?
Thanks in advance
Jessica

On windows, first you need to add the data source name (DSN).
You do this by searching for 'odbc data source administrator' on windows and creating a DSN for 'SQL Anywhere 12'. Fill in the necessary information like username,password,host,port,server name and database name. Finally test the connection as shown.
Once finished you can call the code as follows:
import sqlanydb
conn = sqlanydb.connect(dsn='SYBASE_IQ')
curs = conn.cursor()
curs.execute("select 'Hello, world!'")
print( "SQL Anywhere says: %s" % curs.fetchone())
curs.close()
conn.close()

Get and install the SYBASE ODBC DRIVER.
Configure the DSN on your PC.
On Windows, search for the Microsoft ODBC Administrator. Then create a DSN.
Python code:
using SQLAchemy
import sqlalchemy as sa
from sqlalchemy import create_engine, event
from sqlalchemy.engine.url import URL
import urllib
params = urllib.parse.quote_plus('DSN=dsn_name;PWD=user_pwd')
engine = sa.create_engine("sybase+pyodbc:///?odbc_connect={}".format(params))
with engine.connect() as cursor:
cursor.execute(""" SELECT * FROM database """)
Using PyODBC
import pyodbc
conn = pyodbc.connect('DSN=dsn_name;PWD=user_pwd')
with conn:
cursor = conn.cursor()
cursor.execute(""" SELECT * FROM database """)

Python - Pyodbc Connection error

I am trying to connect to the SQL Server database using Python3.4
This is the code that works for me
cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER=DESKTOP-GDM2HQ17\SQLEXPRESS;DATABASE=pyconnect;Trusted_Connection=yes')
and I login into my Management studio - database using Windows connection.
Here is the code, which is not working for me :
cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER=DESKTOP-GDM2HQ17\SQLEXPRESS;DATABASE=pyconnect;UID=DESKTOP-GDM2HQ17\sid;PWD=123')
Kindly share your thoughts on where I am going wrong.

There are two SQL Server Authentication modes:
1, Connecting Through Windows Authentication:
When a user connects through a Windows user account, SQL Server validates the account name and password using the Windows principal token in the operating system.
2, Connecting Through SQL Server Authentication:
When using SQL Server Authentication, logins are created in SQL Server that are not based on Windows user accounts. Both the user name and the password are created by using SQL Server and stored in SQL Server.
Your first code is working as it is Connecting Through Windows Authentication.
Your second code is not working as it is trying to find the credentials (login and password) which are stored in SQL Server, but the credentials is not created in SQL server.
Moreover, you could refer official doc to know how to Change Server Authentication Mode.
Hope it will help you.

DRIVER='{SQL Server}' works
below code is tested.....
import pyodbc
con = pyodbc.connect('Driver={SQL Server};'
'Server=LAPTOP-PPDS6BPG;'
'Database=training;'
'Trusted_Connection=yes;')
cursor = con.cursor()
sql_query = 'SELECT * FROM Students'
cursor.execute(sql_query)
for row in cursor:
print(row)

This works fine for me better than any other I could find
import pyodbc
import pandas as pd
conn = pyodbc.connect('Driver={SQL Server};'
'Server=10.****;'
'Database=Ma**;'
'UID=sql**;'
'PWD=sql**;')
cursor = conn.cursor()
sql = """\
EXEC [dbo].[GetNewPayment] #Login=?, #PasswordMD5=?, #RevokeTimeFrom=?, #RevokeTimeTo=? Status=?
"""
params = ('a***', 'c2ca***', '2021-05-01','2021-05-02', '3')
cursor.execute(sql, params)

Connect Python with SQL Server Database

When I am trying to connect python with SQL Server, following error occurred.
"pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server
Driver][DBNETLIB]SQL Server does not exist or access denied. (17)
(SQLDriverConnect)')"
Following is the my code.
import pyodbc
connection = pyodbc.connect("Driver={SQL Server}; Server=localhost;
Database=emotionDetection; uid=uname ;pwd=pw;Trusted_Connection=yes")
cursor = connection.cursor()
SQLCommand = ("INSERT INTO emotion" "(happy, sad, angry) "
"VALUES (?,?,?)")
Values = ['smile','cry','blame']
cursor.execute(SQLCommand,Values)
connection.commit()
connection.close()
This is my first attempt to connect Python with sql server. I don't have an idea what would be the driver name, server name, username and password.Do you have any idea of what should be my configuration. Please help me.

CONNECTION FROM WINDOWS TO MS SQL SERVER DATABASE:
Here you have an example I use myself to connect to MS SQL database table with a Python script:
import pyodbc
server = 'ip_database_server'
database = 'database_name'
username = 'user_name'
password = 'user_password'
driver = '{SQL Server}' # Driver you need to connect to the database
port = '1433'
cnn = pyodbc.connect('DRIVER='+driver+';PORT=port;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+
';PWD='+password)
cursor = cnn.cursor()
'User' and 'password' and 'table_name' are attibutes defined by the DB administrator, and he should give them to you. The port to connect to is also defined by the admin. If you are trying to connect from a Windows device to the DB, go to ODBC Data Source Administrator from Windows, and check if you have installed the driver:
Where is the ODBC data source administrator in a Windows machine.
The image is in spanish, but you only have to click on 'Drivers' tab, and check if the driver is there as in the image.
CONNECTION FROM LINUX/UNIX TO MS SQL SERVER DATABASE:
If you are working in Linux/Unix, then you shoud install a ODBC manager like 'FreeTDS' and 'unixODBC'. To configure them, you have some examples in the following links:
Example: Connecting to Microsoft SQL Server from Linux/Unix
Example: Installing and Configuring ODBC

I think you should check out this.
stackoverflow answer about odbc
Also, what sql server do you use?

The library pymssql doesnot require any drivers and works on both Windows as well as Ubunutu.
import pymssql
import pandas as pd
server = 'yourusername'
username = 'yourusername'
password = 'yourpassword'
database = 'yourdatabase'
table_name = 'yourtablename'
conn = pymssql.connect(host=server,user=username,password=password,database=database)
dat = pd.read_sql("select * from table_name,conn)

Try pyodbc with SQLalchemy
try this:
import sqlalchemy
import pyodbc
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://user:password#host:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
cnxn = engine.connect()
Use your corresponding driver
It works for me
Luck!

Working Examples Work Best For Me:
Need Mac ODBC Drivers?
If you need the mac driver I used homebrew and found the commands here
Detail
I personally learn best by reverse enginerring, with that said I am sharing one of my examples, it may be a bit crude but I'm growing my Python skills.
My script I created allows me to connect my Mac OS to a AWS RDS instance.
The whole script is a copy paste with a little modification for you about your server info, and you are off and running.
just modify these lines to connect.
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
Then Run the file: python3 ~/Your/path/pyodbc_mssqldbtest.py and you should be set.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# =============================================================================
# Created By : Jeromie Kirchoff
# Created Date: Mon July 31 22:32:00 PDT 2018
# FILENAME: pyodbc_mssqldbtest.py
# =============================================================================
"""The Module Has Been Build for Interaction with MSSQL DBs To Test the con."""
# =============================================================================
# Thanks to this post for headers https://stackoverflow.com/q/12704305/1896134
# Answer to an SO question: https://stackoverflow.com/q/42433408/1896134
# =============================================================================
import pyodbc
def runningwithqueries(query):
"""The Module Has Been Build to {Open, Run & Close} query connection."""
print("\nRunning Query: " + str(query) + "\nResult :\n")
crsr = cnxn.execute(query)
columns = [column[0] for column in crsr.description]
print(columns)
for row in crsr.fetchall():
print(row)
crsr.close()
# =============================================================================
# SET VARIABLES NEEDED FOR SERVER CONNECTION
# =============================================================================
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
connStr = (r'DRIVER={ODBC Driver 17 for SQL Server};' +
r"Integrated Security=True;" +
r'SERVER=' + server +
r';UID=' + username +
r';PWD=' + password +
r';DSN=MSSQL-PYTHON' +
r';DATABASE=' + database + ';'
)
print("Your Connection String:\n" + str(connStr) + "\n\n")
# =============================================================================
# CONNECT TO THE DB
# =============================================================================
cnxn = pyodbc.connect(connStr, autocommit=True)
# =============================================================================
# SET QUERIES TO VARIABLES
# =============================================================================
SQLQUERY1 = ("SELECT ##VERSION;")
SQLQUERY2 = ("SELECT * FROM sys.schemas;")
SQLQUERY3 = ("SELECT * FROM INFORMATION_SCHEMA.TABLES;")
SQLQUERY4 = ("SELECT * FROM INFORMATION_SCHEMA.COLUMNS;")
SQLQUERY5 = ("SELECT * FROM INFORMATION_SCHEMA.CHECK_CONSTRAINTS;")
SQLQUERY6 = ("EXEC sp_databases;")
SQLQUERY7 = ("EXEC sp_who2 'active';")
# =============================================================================
# RUN QUERIES
# YOU CAN RUN AS MANY QUERIES AS LONG AS THE CONNECTION IS OPEN TO THE DB
# =============================================================================
runningwithqueries(SQLQUERY1)
runningwithqueries(SQLQUERY2)
runningwithqueries(SQLQUERY3)
runningwithqueries(SQLQUERY4)
runningwithqueries(SQLQUERY5)
runningwithqueries(SQLQUERY6)
runningwithqueries(SQLQUERY7)
# =============================================================================
# CLOSE THE CONNECTION TO THE DB
# =============================================================================
cnxn.close()

import pyodbc
conn = pyodbc.connect('Driver={SQL Server};' 'Server=**SERVER NAME**;' 'Database=**DATABASE NAME**;' 'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT * FROM Output3')
This works just check you specify the Respective Driver, Server and the Database names correctively!