I am using the following code to extract data from SQL into a Data Frame and it works fine.
import pyodbc
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=DESKTOP-5422GFU;"
"Database=Python_Data;"
"Trusted_Connection=yes;"
"uid=User;pwd=password")
df = pd.read_sql_query('select * from Persons', cnxn)
df
But when I add this line
df.to_sql('test', schema = 'public', con = cnxn, index = False,
if_exists = 'replace')
to send data back to the Server I get an error that says
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][SQL Server Native Client 11.0][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][SQL Server Native Client 11.0][SQL Server]Statement(s) could not be prepared. (8180)")
I have tried a variety of solutions and just can't get it to work.
I believe the issue is in your "con = cnxn" statement. Try this:
import pandas as pd
import sqlalchemy
from sqlalchemy import create_engine
OBDC_cnxn='Python_Data_ODBC' #this will be the name of the ODBC connection to this database in your ODBC manager.
engine = create_engine('mssql+pyodbc://'+ODBC_cnxn)
df.to_sql('test', schema = 'public', con = engine, index = False,
if_exists = 'replace')
Related
I am trying to pull data from SQL server, I am able to pull the sample data ( up to top 1000 records) but when I run it for all records I get following error:
('01000', '[01000] [Microsoft][ODBC SQL Server
Driver][DBNETLIB]ConnectionWrite (send()). (10054) (SQLGetData);
[01000] [Microsoft][ODBC SQL Server Driver][DBNETLIB]General network
error. Check your network documentation. (11)')
I am using following syntax
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=g8w00761s.inc.hpicorp.net;'
'Database=ContraDigital2.0;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
sql_query = pd.read_sql_query("""SELECT top 1000 * FROM [ContraDigital2.0].[dbo].[contra_anomaly_upfront_backend_1]""",conn)
print(sql_query)
I am trying to connect Python to our remote SQL Server but I am not getting it. Following is a code that I used.
server = 'server,1433'
database = 'db'
username = 'username'
password = 'pw'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute('SELECT top 1 * FROM db.dbo.t_location')
for row in cursor:
print(row)
We have 2 servers. One is database server but I use application server for SQL which connects to database server. This is the error I'm getting. I am trying for a week but I'm not sure what am I missing here.
Any help would be appreciated
OperationalError: ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: No such host is known.\r\n (11001) (SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 17 for SQL Server]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. (11001)')
ADDED:
connection_str = ("Driver={SQL Server Native Client 11.0};"
"Server= 10.174.124.12,1433;"
#"Port= 1433;"
"Database=AAD;"
"UID=dom\user;"
"PWD=password;"
)
connection = pyodbc.connect(connection_str)
data = pd.read_sql("select top 1 * from dbo.t_location with (nolock);",connection)
I used the above code and now I see this error. Seems like it worked but failed to login. Usually I have to use Windows authentication in SSMS once I put my credentials to login in remote desktop.
('28000', "[28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'dom\user'. (18456) (SQLDriverConnect); [28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'dom\user'. (18456)")
Answer:
I am excited that I finally found a solution using pymssql. I don't know pyodbc wasn't working but I am sure I must have had done something wrong. I used below code to get the data from remote SQL server using Python.
import pymssql
conn = pymssql.connect(
host=r'10.174.124.12',
user=r'dom\user',
password=r'password',
database='db'
)
cursor = conn.cursor(as_dict=True)
cursor.execute('Select top 4 location_id, description from t_location with (nolock)')
data = cursor.fetchall()
data_df = pd.DataFrame(data)
cursor.close()
Ignore my code at this moment. I still have to do some cleaning but this code will work.
Finally to answer my question, I had to use pymssql which worked. I did not have to put the port number which was making me confused. Thanks everyone for taking out time to answer.
import pymssql
conn = pymssql.connect(
host=r'10.174.124.12',
user=r'dom\user',
password=r'password',
database='db'
)
cursor = conn.cursor(as_dict=True)
cursor.execute('Select top 4 location_id, description from t_location with (nolock)')
data = cursor.fetchall()
data_df = pd.DataFrame(data)
cursor.close()
you can use this function :
def connectSqlServer(Server , Database , Port , User , Password):
try:
conn = pyodbc.connect('Driver={SQL Server}; Server='+Server+';
Database='+Database+'; Port='+Port+'; UID='+User+'; PWD='+Password+';')
cursor = conn.cursor()
except Exception as e:
print("An error occurred when connecting to DB, error details: {}".format(e))
return False, None
else:
return True, cursor
With the pandas DataFrame called 'data' (see code), I want to put it into a table in SQL Server. How should I do this?
I read something on the internet with data.to_sql, so I tried a little with this function. However, it gives me an error message (see error message). It probably has to do with the argument con = conn. I don't know how to fix this. The error message says something about SQLite. I am not very advanced in using SQL, so I do not understand all errors.
Do you have any ideas? I think the conn is the only problem. The rest works fine.
Using SQL Server Management Studio 2018.
Ty in advance.
Error message:
Exception has occurred: DatabaseError
Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;':
('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]
Invalid object name 'sqlite_master'. (208) (SQLExecDirectW);
[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]
Statement(s) could not be prepared. (8180)")
def connection():
try:
#connect to network database
conn = None
conn = db.connect('Driver={SQL Server};'
'Server=xxsql.database.windows.net;'
'Database=xx;'
'uid=xx;'
'pwd=xx;')
#if connected do
if conn is not None:
print("The connection is established")
sql_query = 'select top 20 * from xx'
#read table into dataframe
#make a dataframe with records from table
dfObj = pd.read_sql_query(sql_query, conn)
#inspect result
print(dfObj)
#read into list
cursor = conn.cursor()
cursor.execute(sql_query)
#put cursor to list
cursor_list = list(cursor.fetchall())
#make copy so that we wont change original data
table = cursor_list.copy()
bibliography = Extract(table)
data = pd.DataFrame(bibliography)
#convert it to SQL
data.to_sql('ExtractedBib', con = conn, if_exists = 'append') #????
except db.Error as e:
print(e)
finally:
if conn is not None:
cursor.close()
conn.close()
print("The connection is closed")
else:
print("No connection established")
def main():
connection()
main()
Try using sqlalchemy instead.
import sqlalchemy
engine = sqlalchemy.create_engine(
"xxsql+pyodbc://uid:pwd#Server/Database",
echo=False)
...
data.to_sql('ExtractedBib', con=engine, if_exists = 'append')
Just change the template values into your database values (uid, pwd etc.)
Hi I'm using sql server V17.3 on Micorsoft Azure platform. I'm trying to upload data from python 3.7 data frame to my table test on sql server. So I wrote following code
import time
start_time = time.time()
import pyodbc
from sqlalchemy import create_engine
import urllib
dataToUpload=pd.read_csv("intermediate.csv")
params = urllib.parse.quote_plus(r'DRIVER=SQLServer};SERVER=nesbaexplsql001.database.windows.net;DATABASE=mydatabase;Trusted_Connection=False;Encrypt=True;uid=myuid;pwd=my password')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str)
dataToUpload.to_sql(name='test',con=engine, if_exists='append',index=False)
But I'm getting error message
DBAPIError: (pyodbc.Error) ('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver][SQL Server]Windows logins are not supported in this version of SQL Server. (40607) (SQLDriverConnect); [HY000] [Microsoft][ODBC SQL Server Driver][SQL Server]Windows logins are not supported in this version of SQL Server. (40607)') (Background on this error at: http://sqlalche.me/e/dbapi)
While executing to_sql. I also tried by putting Trusted_Connection=yes & removing Encryption=True.But I got same error. Can you guide me to resolve this issue?
Please refer to my sample code:
import pyodbc
import csv
server = 'tcp:***.database.windows.net'
database = '***'
username = '***'
password = '***'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute("select * from ***")
row = cursor.fetchone()
while row:
print(row[0])
row = cursor.fetchone()
mycsv = r'D:\insert.csv' # SET YOUR FILEPATH
with open (mycsv, 'r') as f:
reader = csv.reader(f)
columns = next(reader)
query = 'insert into <TABLE NAME>({0}) values ({1})'
query = query.format(','.join(columns), ','.join('?' * len(columns)))
cursor = cnxn.cursor()
for data in reader:
cursor.execute(query, data)
cursor.commit()
Add one piece to your connection string:
Integrated Security = False;
Hope this helps.
I am new to Python and SQL server. I have been trying to insert a pandas df into our database for the past 2 days without any luck. Can anyone please help me debugging the errors.
I have tried the following
import pyodbc
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc:///?odbc_connect=DRIVER={SQL Server};SERVER=bidept;DATABASE=BIDB;UID=sdcc\neils;PWD=neil!pass')
engine.connect()
df.to_sql(name='[BIDB].[dbo].[Test]',con=engine, if_exists='append')
However at the engine.connect() line I am getting the following error
sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver]Neither DSN nor SERVER keyword supplied (0) (SQLDriverConnect)')
Can anyone tell me what I am missing. I am using Microsoft SQL Server Management Studio - 14.0.17177.0
I connect to the SQL server through the following
Server type: Database Engine
Server name: bidept
Authentication: Windows Authentication
for which I log into my windows using username : sdcc\neils
and password : neil!pass
I have also tried this
import pyodbc
conn_str = (
r'Driver={SQL Server Native Client 11.0};'
r'Server=bidept;'
r'Database=BIDB;'
r'Trusted_Connection=yes;'
)
cnxn = pyodbc.connect(conn_str)
df.to_sql(name='Test',con=cnxn, if_exists='append')
for which I got this error
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][SQL Server Native Client 11.0][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Statement(s) could not be prepared. (8180)")
Any help would be greatly appreciated as I am clueless as what to do.
As stated in the SQLAlchemy documentation, "The delimeters must be URL escaped" when using a pass-through exact pyodbc string.
So, this will fail ...
import pyodbc
from sqlalchemy import create_engine
params = r'DRIVER={SQL Server};SERVER=.\SQLEXPRESS;DATABASE=myDb;Trusted_Connection=yes'
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str)
... but this will work:
import pyodbc
from sqlalchemy import create_engine
import urllib
params = urllib.parse.quote_plus(r'DRIVER={SQL Server};SERVER=.\SQLEXPRESS;DATABASE=myDb;Trusted_Connection=yes')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str)