I am trying to retrieve SQL data on a Linux Server (Ubuntu 16.04.2) in a Python 3.5 script using the ODBC Driver 13 for SQL Server. Running the script on SQL Server and Python in windows goes fine. Running the script in Python in Linux throws a SQL Server Syntax error:
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Incorrect syntax near '0x107c'. (102) (SQLExecDirectW)")
As I add or remove columns it changes '0x107c' to a different character suggesting that it is not a single illegal character, but several. With a more limited amount of columns the script even runs (excluded [Order Type text] and [Order Nr]).This leads me to suspect something goes wrong in character set conversion. What am I doing wrong and how do I fix it?
Python3.5:
import pandas as pd
import pyodbc
#Set parameters
sql_file = 'file.sql'
#Define methods
def SQLDataToDataframe(filename):
fd = open('file.sql','r')
content = fd.read()
fd.close()
df = pd.read_sql(content, connection)
return df
#Import Data from SQL
connection = pyodbc.connect('Driver={ODBC Driver 13 for SQL Server};'
'Server=Server;'
'Database=DB;'
'uid=User;pwd=Password')
dataframe = SQLDataToDataframe(sql_file)
file.sql:
SELECT [ID]
,[Company Code]
,[Description]
,[Order Category]
,[Order Category Text]
,[Order Type]
,[Order Type text]
,[Order Nr]
FROM [TABLE]
Solution for future reference. Don't use pyodbc on Linux to connect to MS SQL Server. Use pymssql instead. Instructions here: https://learn.microsoft.com/en-us/sql/connect/python/pymssql/step-1-configure-development-environment-for-pymssql-python-development.
Related
im trying to make a connection to an as400 with db2 using pyodbc and the ibm db2 odbc driver.
import pyodbc
connection = pyodbc.connect(
driver='{IBM DB2 ODBC DRIVER}',
system='192.168.1.100',
uid='user',
pwd='pass')
c1 = connection.cursor()
#this is meaningless sql, i just want the connection
c1.execute('select * from libname.filename')
for row in c1:
print (row)
Running this gives me this error
python pydata.py
Traceback (most recent call last):
File "C:\Users\tca\Desktop\ScriptingSTuff\pydata.py", line 3, in <module>
connection = pyodbc.connect(
pyodbc.OperationalError: ('08001', '[08001] [IBM][CLI Driver] SQL1013N The database alias name or database name "" could not be found. SQLSTATE=42705\r\n (-1013) (SQLDriverConnect)')
Any ideas?
This is all under win10
EDIT:
Adding this "database='s10c38ft',"
import pyodbc
connection = pyodbc.connect(
driver='{IBM DB2 ODBC DRIVER}',
system='192.168.1.100,8471',
database='s10c38ft',
uid='user',
pwd='pass')
c1 = connection.cursor()
c1.execute('select * from libname.filename')
for row in c1:
print (row)
Makes it hang on a blinking cursor, I cant even CTRL+C to end it, I have to close cmd.
The proper driver name should be IBM i Access ODBC Driver (but see notes below). Other than that, your first example was correct:
connection = pyodbc.connect(
driver='{IBM i Access ODBC Driver}',
system='192.168.1.100',
uid='user',
pwd='pass')
If that doesn't work, there are two main possibilities:
You are using an old ODBC driver. This would happen if you are using the old iSeries Access (in which case the driver name is iSeries Access ODBC Driver) or even older Client Access (driver name Client Access ODBC Driver (32-bit)). If you choose the appropriate name for your driver, it will work.
You are using an ODBC driver that is not for IBM i. The most commonly used member of the Db2 family is Db2 for LUW (Linux, Unix, Windows), but there are others. None of these will work for you.
You can find out the list of exact ODBC driver names you have installed by calling pyodbc.drivers(). If you don't have any of the ones I mentioned above by name, then you don't have the right driver. The ODBC driver you want is the one described here.
I have a pandas data frame called : data
I am trying to read this pandas dataframe into a table in sql server.
I am able to read data into python from sql but I am expiering problems loading the dataframe into a table.
I have tried a few examples but keep on getting the same error:
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
At the moment I have the following code:
PASSW = 'test'
SERVER = '111.111.11'
DB = 'Database'
Table = 'TableName'
data = pd.read_csv('2019_Prediction_24_10.csv')
cnxn = pyodbc.connect(DRIVER='{ODBC Driver 13 for SQL Server}', SERVER= SERVER,
DATABASE= DB,User = 'User', Password = PASSW)
data.to_sql(con=cnxn, name='Predictions',schema = 'PA' ,if_exists='replace')
I am new to pyodbc and using python together with sql server, Am not quite sure what is going wrong let alone fix it.
Can please someone assist me, or point me in the right direction
As noted in the to_sql documentation:
con : sqlalchemy.engine.Engine or sqlite3.Connection
You have supplied to_sql with a (pyodbc) Connection object, so pandas is treating it like a SQLite connection. To use to_sql with SQL Server you'll need to install SQLAlchemy, create an Engine object, and pass that to to_sql.
Using Python, I'm trying to read a table from SQL Server and then insert the data into a table in Access. The best way I've found to do this is using the pandas dataframe. I wrote up a program that reads a SQL Server table into a dataframe like so:
dataframe = pandas.read_sql(selectSql, srcConn)
And it works great on a ~209MB table. When I try it on a ~1,116MB table it throws an exception with no description. I'm guessing it has to do with the size of the table it's reading in (it would be nice if it said that). I know Access can only hold 2GB but there is plenty of room left in it and it doesn't even get to the part where it writes to Access before throwing the error.
Is there any way to fix this for larger tables? Is there a better way I should be copying tables from SQL Server 2008 R2 to Access 2016 using Python? I have 16GB of RAM on Win10 64-bit so that shouldn't be a problem. I've tried 32-bit Python 3.7 and 64-bit Python 3.6 to no avail. I tried SSIS first but it crashes my entire Visual Studio whenever I try to open a package with a connection to Access.
UPDATE:
I followed Gord's advice below and now my code looks like this:
access_cnxn_str = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=' + access_db + ';'
)
sqls_cnxn_str = (
r'DRIVER=ODBC Driver 13 for SQL Server;'
r'SERVER=' + sqls_server + ';'
r'DATABASE=' + sqls_db + ';'
r'UID=' + sqls_username + ';'
r'PWD=' + sqls_password + ';'
)
This connection works by itself:
sqls_cnxn = pyodbc.connect(sqls_cnxn_str)
And this connection works by itself:
pyodbc.pooling = False
access_cnxn = pyodbc.connect(access_cnxn_str, autocommit = True)
But this is throwing an error:
access_cnxn.execute(f"SELECT * INTO {access_table} FROM [ODBC;{sqls_cnxn_str}].{sqls_table}")
The error thrown:
Message=('HY000', "[HY000] [Microsoft][ODBC Microsoft Access Driver]
ODBC--connection to 'ODBC Driver 13 for SQL ServerSERVERNAME' failed.
(-2001) (SQLExecDirectW)")
Source=C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py
StackTrace: File
"C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
57, in sync_table
dest_cnxn.execute(f"SELECT * INTO {access_table} FROM [ODBC;{sqls_cnxn_str}].{sqls_table}") File
"C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
121, in main
sync_table('', sqls_table, get_access_cnxn(), access_table) File "C:\Users\bruescm\source\repos\DB_Test\DB_Test\SyncAllTests.py", line
124, in
main()
SERVERNAME in the error is the name of the server on which SQL Server resides. Not sure why it jammed it up against the driver name in the error.
Any ideas?
UPDATE 2:
It turns out my Access is 32-bit. This still doesn't explain why it won't connect as I was originally using Python 3.7 32-bit.
Thanks.
I was able to get the Access Database Engine to pull a table from SQL Server and create a copy in the Access database by simply doing
pyodbc.pooling = False # required
cnxn = pyodbc.connect("DSN=myAccessDb", autocommit=True)
cnxn.execute("SELECT * INTO access_tbl FROM [ODBC;DSN=SQLmyDb].sql_server_tbl")
where SQLmyDb is the ODBC DSN for my SQL Server instance.
Update
Just tested to confirm that DSN-less connection strings also work:
pyodbc.pooling = False # required
access_cnxn_str = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=C:\__tmp\test.accdb;'
)
cnxn = pyodbc.connect(access_cnxn_str, autocommit=True)
sql_cnxn_str = (
r'DRIVER=ODBC Driver 17 for SQL Server;'
r'SERVER=(local)\SQLEXPRESS;'
r'DATABASE=myDb;'
r'Trusted_Connection=Yes;'
)
cnxn.execute(f"SELECT * INTO access_tbl FROM [ODBC;{sql_cnxn_str}].sql_server_tbl")
I am now processing 2 tables at the same time by pyodbc.
Before finishing processing, I would like to place locks on the 2 tables so that nobody can change the 2 tables before I finish.
How can I do that?
I tried the below, but failed with an error.
import pyodbc
conn = pyodbc.connect("conn_str")
conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;")
conn.execute("begin transaction trans;")
cur1 = conn.execute("select * from tbl1;")
cur2 = conn.execute("select * from tbl2;")
... some processing at cur1 and cur2 at python ...
conn.execute("commit transaction trans;")
However, the following error comes up when execution "cur2 = conn.execute("select * from tbl2;")"
[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt (0) (SQLExecDirectW)
Is there anything wrong with my code? Highly appreciated for any help.
Originally, SQL Server ODBC was limited to one active hstmt (the ODBC equivalent of a pyodbc.cursor) per connection. Later on, Microsoft added the MARS (Multiple Active Result Sets) feature to SQL Server ODBC, but that feature is "off" by default.
So this code
import pyodbc
import sys
print(f"Python version {sys.version}") # Python version 3.6.4 ...
print(f"pyodbc version {pyodbc.version}") # pyodbc version 4.0.24
conn_str = (
r'DRIVER=ODBC Driver 17 for SQL Server;'
r'SERVER=.\SQLEXPRESS;'
r'DATABASE=myDb;'
r'Trusted_Connection=yes;'
)
cnxn = pyodbc.connect(conn_str, autocommit=True)
cnxn.set_attr(pyodbc.SQL_ATTR_TXN_ISOLATION, pyodbc.SQL_TXN_SERIALIZABLE)
cnxn.autocommit = False # enable transactions
cur1 = cnxn.execute("SELECT 1 AS x UNION ALL SELECT 2 AS x")
cur2 = cnxn.execute("SELECT 'foo' AS y UNION ALL SELECT 'bar' AS y")
print(cur1.fetchone())
print(cur2.fetchone())
print(cur1.fetchone())
print(cur2.fetchone())
fails with
Traceback (most recent call last):
File "C:/Users/Gord/PycharmProjects/py3pyodbc_demo/main.py", line 18, in <module>
cur2 = cnxn.execute("SELECT 'foo' AS y UNION ALL SELECT 'bar' AS y")
pyodbc.Error: ('HY000', '[HY000] [Microsoft][ODBC Driver 17 for SQL Server]Connection is busy with results for another command (0) (SQLExecDirectW)')
However, if we add MARS_Connection=yes to the connection string
conn_str = (
r'DRIVER=ODBC Driver 17 for SQL Server;'
r'SERVER=.\SQLEXPRESS;'
r'DATABASE=myDb;'
r'Trusted_Connection=yes;'
r'MARS_Connection=yes;'
)
then the code works.
Unfortunately in your case you are using the ancient DRIVER=SQL Server which is too old to support MARS_Connection=yes so your options are
use a newer version of the SQL Server ODBC driver, or
open two separate connections, one for each cursor.
I have a problem using pyodbc with a TeraData server -- My perl connection which uses the same ODBC drivers work without issue.
It appears pyodbc is connecting because I can fetch table names, however, the table names are not encoded properly (example below). Here is a list of things I've tried:
Setting CHARSET/ENCODING in my DSN as 'ASCII', 'UTF-8', and many others
Setting the locale:
locale.setlocale(locale.LC_ALL, 'en_US.utf8')
Changing autocommit and ANSI=True/False' in pyodbc.connect()
python==3.4.3, pyodbc==3.0.10
.odbc.ini:
[ODBC]
InstallDir=/opt/teradata/client/15.10/odbc_64
Trace=0
TraceDll=/opt/teradata/client/15.10/lib/odbctrac.so
TraceFile=/home/solberg/teratrace
TraceAutoStop=0
[ODBC Data Sources]
tera01=Teradata ODBC Driver 15.10
[tera01]
Driver=/opt/teradata/client/15.10/lib64/tdata.so
DBCName=tera01
Username=solberg
Authentication=LDAP
.odbcinst.ini:
[ODBC DRIVERS]
Teradata=Installed
[ODBC Translators]
OEB to ANSI=Installed
python code:
import pyodbc
pyodbc.pooling = False
conn = pyodbc.connect('DSN=tera01;', password=pw)
cursor = conn.cursor()
cursor.tables()
rows = cursor.fetchall()
print(row[0].table_name)
扁牯䱴獩却獥楳湯
print(row[0].table_name.encode('utf_16_le'))
b'AbortListSession'
conn.execute("SELECT DISTINCT column FROM table;").fetchall()
Error: ('HY000', '[HY000] [unixODBC][Driver Manager]Driver returned SQL_ERROR or SQL_SUCCESS_WITH_INFO but no error reporting API found (0) (SQLExecDirectW)')
not using pyodbc, but two things:
1) sometimes frameworks struggle if not all columns are explicitly named, so could you try
SELECT DISTINCT column as d_col FROM table;
SELECT column FROM table group by column;
2) did you try the python teradata module? - At the end of the day it also uses ODBC (or REST), but it encapsulates lot of nasty stuff. (https://github.com/Teradata/PyTd)
[sudo] pip install teradata