Using Python, I'm trying to read a table from SQL Server and then insert the data into a table in Access. The best way I've found to do this is using the pandas dataframe. I wrote up a program that reads a SQL Server table into a dataframe like so:
dataframe = pandas.read_sql(selectSql, srcConn)
And it works great on a ~209MB table. When I try it on a ~1,116MB table it throws an exception with no description. I'm guessing it has to do with the size of the table it's reading in (it would be nice if it said that). I know Access can only hold 2GB but there is plenty of room left in it and it doesn't even get to the part where it writes to Access before throwing the error.
Is there any way to fix this for larger tables? Is there a better way I should be copying tables from SQL Server 2008 R2 to Access 2016 using Python? I have 16GB of RAM on Win10 64-bit so that shouldn't be a problem. I've tried 32-bit Python 3.7 and 64-bit Python 3.6 to no avail. I tried SSIS first but it crashes my entire Visual Studio whenever I try to open a package with a connection to Access.
UPDATE:
I followed Gord's advice below and now my code looks like this:
access_cnxn_str = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=' + access_db + ';'
)
sqls_cnxn_str = (
r'DRIVER=ODBC Driver 13 for SQL Server;'
r'SERVER=' + sqls_server + ';'
r'DATABASE=' + sqls_db + ';'
r'UID=' + sqls_username + ';'
r'PWD=' + sqls_password + ';'
)
This connection works by itself:
sqls_cnxn = pyodbc.connect(sqls_cnxn_str)
And this connection works by itself:
pyodbc.pooling = False
access_cnxn = pyodbc.connect(access_cnxn_str, autocommit = True)
But this is throwing an error:
access_cnxn.execute(f"SELECT * INTO {access_table} FROM [ODBC;{sqls_cnxn_str}].{sqls_table}")
The error thrown:
Message=('HY000', "[HY000] [Microsoft][ODBC Microsoft Access Driver]
ODBC--connection to 'ODBC Driver 13 for SQL ServerSERVERNAME' failed.
(-2001) (SQLExecDirectW)")
Source=C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py
StackTrace: File
"C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
57, in sync_table
dest_cnxn.execute(f"SELECT * INTO {access_table} FROM [ODBC;{sqls_cnxn_str}].{sqls_table}") File
"C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
121, in main
sync_table('', sqls_table, get_access_cnxn(), access_table) File "C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
124, in
main()
SERVERNAME in the error is the name of the server on which SQL Server resides. Not sure why it jammed it up against the driver name in the error.
Any ideas?
UPDATE 2:
It turns out my Access is 32-bit. This still doesn't explain why it won't connect as I was originally using Python 3.7 32-bit.
Thanks.
I was able to get the Access Database Engine to pull a table from SQL Server and create a copy in the Access database by simply doing
pyodbc.pooling = False # required
cnxn = pyodbc.connect("DSN=myAccessDb", autocommit=True)
cnxn.execute("SELECT * INTO access_tbl FROM [ODBC;DSN=SQLmyDb].sql_server_tbl")
where SQLmyDb is the ODBC DSN for my SQL Server instance.
Update
Just tested to confirm that DSN-less connection strings also work:
pyodbc.pooling = False # required
access_cnxn_str = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=C:\__tmp\test.accdb;'
)
cnxn = pyodbc.connect(access_cnxn_str, autocommit=True)
sql_cnxn_str = (
r'DRIVER=ODBC Driver 17 for SQL Server;'
r'SERVER=(local)\SQLEXPRESS;'
r'DATABASE=myDb;'
r'Trusted_Connection=Yes;'
)
cnxn.execute(f"SELECT * INTO access_tbl FROM [ODBC;{sql_cnxn_str}].sql_server_tbl")
Related
im trying to make a connection to an as400 with db2 using pyodbc and the ibm db2 odbc driver.
import pyodbc
connection = pyodbc.connect(
driver='{IBM DB2 ODBC DRIVER}',
system='192.168.1.100',
uid='user',
pwd='pass')
c1 = connection.cursor()
#this is meaningless sql, i just want the connection
c1.execute('select * from libname.filename')
for row in c1:
print (row)
Running this gives me this error
python pydata.py
Traceback (most recent call last):
File "C:\Users\tca\Desktop\ScriptingSTuff\pydata.py", line 3, in <module>
connection = pyodbc.connect(
pyodbc.OperationalError: ('08001', '[08001] [IBM][CLI Driver] SQL1013N The database alias name or database name "" could not be found. SQLSTATE=42705\r\n (-1013) (SQLDriverConnect)')
Any ideas?
This is all under win10
EDIT:
Adding this "database='s10c38ft',"
import pyodbc
connection = pyodbc.connect(
driver='{IBM DB2 ODBC DRIVER}',
system='192.168.1.100,8471',
database='s10c38ft',
uid='user',
pwd='pass')
c1 = connection.cursor()
c1.execute('select * from libname.filename')
for row in c1:
print (row)
Makes it hang on a blinking cursor, I cant even CTRL+C to end it, I have to close cmd.
The proper driver name should be IBM i Access ODBC Driver (but see notes below). Other than that, your first example was correct:
connection = pyodbc.connect(
driver='{IBM i Access ODBC Driver}',
system='192.168.1.100',
uid='user',
pwd='pass')
If that doesn't work, there are two main possibilities:
You are using an old ODBC driver. This would happen if you are using the old iSeries Access (in which case the driver name is iSeries Access ODBC Driver) or even older Client Access (driver name Client Access ODBC Driver (32-bit)). If you choose the appropriate name for your driver, it will work.
You are using an ODBC driver that is not for IBM i. The most commonly used member of the Db2 family is Db2 for LUW (Linux, Unix, Windows), but there are others. None of these will work for you.
You can find out the list of exact ODBC driver names you have installed by calling pyodbc.drivers(). If you don't have any of the ones I mentioned above by name, then you don't have the right driver. The ODBC driver you want is the one described here.
I am trying to connect to SQL Server using pyodbc to query some tables for work. I am able to connect to one of our servers and run a SQL query, but the same code does not work for a different server. The error message I get is this:
Error: ('008001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]
SQL Server does not exist or access denied. (17) (SQLDriverConnect)')
This is the code that I'm using:
import pyodbc
import pandas as pd
conn = pyodbc.connect('Driver={SQL Server};'
'Server=servername;'
'Database=databasename;'
'Trusted_Connection=yes;')
[rest of code]
The code works for one server, but when I change the server & database names to the other, I get the error message (there are no spelling errors in the server/database names). I am able to access this server through SQL. Additionally, I am able to access this server and to query/update its tables using Excel VBA (even though it will not connect in Python).
Based on my research, there are many possible issues that I do not fully understand. I have tried changing the driver name in the code, and that failed. Since this code is for work, there are security permissions that prevent me from trying out certain solutions.
What could be causing this issue? If somebody could point me in the right direction, that would help me ask better questions of our IT team, if I'm not able to solve it myself.
(I'm using Microsoft SQL Server Management Studio 17 and Python 3.6 with Spyder.)
Try with this driver:
SQLserver_settings = {'server': 'servername', 'database': 'database', 'username': 'username', 'password': 'password', 'driver': '{ODBC Driver 17 for SQL Server}'}
cnxn = pyodbc.connect('DRIVER=' + SQLserver_settings['driver'] + ';SERVER=' + SQLserver_settings['server'] + ';PORT=1433;DATABASE=' + SQLserver_settings['database'] + ';UID=' + SQLserver_settings['username'] +';PWD=' + SQLserver_settings['password'])
Whenever I attempt to query a file using Python script, I receive the following error
pyodbc.InterfaceError: ('IM002', u'[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
Image of Error Message
My code is as followings:
conn = pyodbc.connect ("DRIVER={ODBCDataFile [Microsoft Access Driver
(*.mdb, *.accdb)]};DBQ=C:\Users\jmtr\Documents\IRST_old.accdb;")
cur = conn.cursor()
cur.execute("SELECT Name, CAI, Email, SSPLocation, BUNUM from Tbl_SSP")
My Access database is "Microsoft Access 2016 32-bit". I am also using "32-bit" python 2.7.13 and 32-bit PYODBC. And, I have 32-bit drivers set-up in the ODBC Data Source Administrator:
Image of ODBC 32-bit
I don't understand why I am still receiving this error message?
Connection string is incorrect. There is no ODBCDataFile keyword with brackets [...]. Simply remove them and assign DRIVER to installed ODBC driver as your screenshot shows:
conn = pyodbc.connect("DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};" + \
"DBQ=C:\\Users\\jmtr\\Documents\\IRST_old.accdb;")
I think you misspelled ODBCDataFile instead of ODBCFile.
But in python the character \ is the escape character in the strings. You need to append the prefix r"DRIVER..." to force raw strings, so without missinterpreting the escape character.
First download and install Microsoft Access Database Engine 2010 Redistributable if you haven’t.
Then you should install pyodbc module.
Now you can connect to access database :
ConFileName=(r'c:\mydb\myaccess.mdb')
conn = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=' + ConFileName + ';')
cursor = conn.cursor()
To select from any table in the database please use this simple code :
ConFileName=(r'c:\mydb\myaccess.mdb')
conn = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=' + ConFileName + ';')
cursor = conn.cursor()
cursor.execute('select * from table1')
for row in cursor.fetchall():
Table1Array.append((row[0],row[1],row[2])
print(str(len(Table1Array))+" records in Table1 loaded successfully.")
You can follow this link to get more information about working with MS Access by Pyton :
https://elvand.com/python-and-ms-access/
I am trying to retrieve SQL data on a Linux Server (Ubuntu 16.04.2) in a Python 3.5 script using the ODBC Driver 13 for SQL Server. Running the script on SQL Server and Python in windows goes fine. Running the script in Python in Linux throws a SQL Server Syntax error:
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Incorrect syntax near '0x107c'. (102) (SQLExecDirectW)")
As I add or remove columns it changes '0x107c' to a different character suggesting that it is not a single illegal character, but several. With a more limited amount of columns the script even runs (excluded [Order Type text] and [Order Nr]).This leads me to suspect something goes wrong in character set conversion. What am I doing wrong and how do I fix it?
Python3.5:
import pandas as pd
import pyodbc
#Set parameters
sql_file = 'file.sql'
#Define methods
def SQLDataToDataframe(filename):
fd = open('file.sql','r')
content = fd.read()
fd.close()
df = pd.read_sql(content, connection)
return df
#Import Data from SQL
connection = pyodbc.connect('Driver={ODBC Driver 13 for SQL Server};'
'Server=Server;'
'Database=DB;'
'uid=User;pwd=Password')
dataframe = SQLDataToDataframe(sql_file)
file.sql:
SELECT [ID]
,[Company Code]
,[Description]
,[Order Category]
,[Order Category Text]
,[Order Type]
,[Order Type text]
,[Order Nr]
FROM [TABLE]
Solution for future reference. Don't use pyodbc on Linux to connect to MS SQL Server. Use pymssql instead. Instructions here: https://learn.microsoft.com/en-us/sql/connect/python/pymssql/step-1-configure-development-environment-for-pymssql-python-development.
I downloaded Python 2.7 (python-2.7.1.amd64.msi) and pyodbc, the python extension module for connecting to DB2 database (i.e. pyodbc-2.1.8.win-amd64-py2.7.exe).
I wrote sample script as shown below.
import csv
import pyodbc
conn = pyodbc.connectpyodbc.connect('DRIVER={DB2};SERVER=localhost;DATABASE=DBT1;UID=scott;PWD=tiger;')
curs = conn.cursor()
curs.execute('select count(edokimp_id) from edokimp')
print curs.fetchall()
The script throws following error
pyodbc.Error: ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnectW)')
As I am a newbie to Python, I realized from the error that I need to download the IBM DB2 driver for pyodbc and hence searched extensively on Google but couldn't find any.
I would greatly appreciate if you could point me to the site where I can download the driver and later explain me how to configure/load the driver.
In case of Java
the driver will be shipped in the form of ojdbc.jar which will be copied to the lib directory which will be on classpath
make changes to configuration file
reference the DataSource from Java Class
I am newbie to Python so I would greatly appreciate if you could let me know cooresponding steps with an example in Python.
You can get the PyDB2 driver on the project homepage.
If you run into compilation issues with the official Python, ActivePython is a good alternate distribution of Python on Windows.
Edit: If it asks you for DB2 headers, you need to get the IBM Data Server Client for ODBC and CLI.
It does work using pyodbc. I think you have a wrong connection string. After some research and tests I solved with this code:
con = pyodbc.connect('DRIVER=iSeries Access ODBC Driver;SYSTEM=10.0.0.1;UID=bubi;PWD=xyz;DBQ=DEFAULTSCHEMA;EXTCOLINFO=1')
cur = con.cursor()
cur.execute('select * from MYTABLE')
row = cur.fetchone()
if row:
field1 = row[0]
field2 = row[1]
# etc...
As you see it doesn't need a DSN to be configured on your system.
This connection string for pyodbc, work for me:
conexion_str = 'SYSTEM=%s;db2:DSN=%s;UID=%s;PWD=%s;DRIVER=%s;' % (self._SYSTEM, self._DSN, self._UID, self._PWD, self._DRIVER)
self._cnn = pyodbc.connect(conexion_str)