How to backup a database by pyodbc - python

The backup statement can't be used in a transaction when it execute with pyodbc cursor. It seems that the pyodbc execute the query inside a default transaction.
I have also tried to use the autocommit mode or add the commit statement before the backup statement. Both of these are not working.
#can't execute the backup statement in transaction
cur.execute("backup database database_name to disk = 'backup_path'")
#not working too
cur.execute("commit;backup database database_name to disk = 'backup_path'")
Is it possible to execute the backup statement by pyodbc? Thanks in advance!
-----Added aditional info-----------------------------------------------------------------------
The backup operation is encapsulate in a function such as:
def backupdb(con, name, save_path):
# with autocommit mode, should be pyodbc.connect(con, autocommit=True)
con = pyodbc.connect(con)
query = "backup database %s to disk = '%s'" % (name, save_path)
cur = con.cursor()
cur.execute(query)
cur.commit()
con.close()
If the function is called by following code,
backupdb('DRIVER={SQL Server};SERVER=.\sqlexpress;DATABASE=master;Trusted_Connection=yes',
'DatabaseName',
'd:\\DatabaseName.bak')
then the exception will be:
File "C:/Documents and Settings/Administrator/Desktop/bakdb.py", line 14, in <module>'d:\\DatabaseName.bak')
File "C:/Documents and Settings/Administrator/Desktop/bakdb.py", line 7, in backupdb cur.execute(query)
ProgrammingError: ('42000', '[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Cannot perform a backup or restore operation within a transaction. (3021) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]BACKUP DATABASE is terminating abnormally. (3013)')
With open the keyword autocommit=True, the function will run silently but there is no backup file generated in the backup folder.


Assuming you are using SQL Server, specify autocommit=True when the connection is built:
>>> import pyodbc
>>> connection = pyodbc.connect(driver='{SQL Server Native Client 11.0}',
server='InstanceName', database='master',
trusted_connection='yes', autocommit=True)
>>> backup = "BACKUP DATABASE [AdventureWorks] TO DISK = N'AdventureWorks.bak'"
>>> cursor = connection.cursor().execute(backup)
>>> connection.close()
This is using pyodbc 3.0.7 with Python 3.3.2. I believe with older versions of pyodbc you needed to use Cursor.nextset() for the backup file to be created. For example:
>>> import pyodbc
>>> connection = pyodbc.connect(driver='{SQL Server Native Client 11.0}',
server='InstanceName', database='master',
trusted_connection='yes', autocommit=True)
>>> backup = "E:\AdventureWorks.bak"
>>> sql = "BACKUP DATABASE [AdventureWorks] TO DISK = N'{0}'".format(backup)
>>> cursor = connection.cursor().execute(sql)
>>> while cursor.nextset():
>>> pass
>>> connection.close()
It's worth noting that I didn't have to use Cursor.nextset() for the backup file to be created with the current version of pyodbc and SQL Server 2008 R2.


After hours I found solution. It must be performed no MASTER, other sessions must be terminated, DB must be set to OFFLINE, then RESTORE and then set to ONLINE again.
def backup_and_restore():
server = 'localhost,1433'
database = 'myDB'
username = 'SA'
password = 'password'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE=MASTER;UID='+username+';PWD='+ password)
cnxn.autocommit = True
def execute(cmd):
cursor = cnxn.cursor()
cursor.execute(cmd)
while cursor.nextset():
pass
cursor.close()
execute("BACKUP DATABASE [myDB] TO DISK = N'/usr/src/app/myDB.bak'")
# do something .......
execute("ALTER DATABASE [myDB] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;")
execute("ALTER DATABASE [myDB] SET OFFLINE;")
execute("RESTORE DATABASE [myDB] FROM DISK = N'/usr/src/app/myDB.bak' WITH REPLACE")
execute("ALTER DATABASE [myDB] SET ONLINE;")
execute("ALTER DATABASE [myDB] SET MULTI_USER;")

Related

Cannot connect to MySQL database using SQLAlchemy

I have a Python program which is trying to connect to a MySQL 8.31 database. This is the code I am using.
connection_string = "DRIVER={ODBC Driver 17 for SQL Server};SERVER=localhost;DATABASE=%s;UID=root;PWD=password" % db_name
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = create_engine(connection_url)
pd.read_sql('select * from apps_list', engine)
I get this exception
sqlalchemy.exc.OperationalError: (pyodbc.OperationalError) ('08001',
'[08001] [Microsoft][ODBC Driver 17 for SQL Server]Named Pipes
Provider: Could not open a connection to SQL Server 1. (2)
(SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL
Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 17
for SQL Server]A network-related or instance-specific error has
occurred while establishing a connection to SQL Server. Server is not
found or not accessible. Check if instance name is correct and if SQL
Server is configured to allow remote connections. For more information
see SQL Server Books Online. (2)') (Background on this error at:
https://sqlalche.me/e/14/e3q8)
I have researched this exception and most people recommend using SQL Server Configuration manager which I do not seem to have on my system.
Not in Computer Management
Not in C:\Windows\SysWOW64
Others have said to ensure Named Pipes are allowed which they are
And finally I've ensured Ports 3306 and 1433 are open
Yet I still get this exception.
For the record I am able to connect using this code
mydb = pymysql.connect(
host="localhost",
database=db_name,
user="root",
password="****",
autocommit=True
)
mycursor = mydb.cursor()
engine = create_engine('mysql+pymysql://****#localhost/%s' % db_name )
However this gives me a warning and tells me to use SQLAlchemy:
userwarning: pandas only support SQLAlchemy
connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2
connectionother DBAPI2 objects are not tested, please consider using
SQLAlchemy warnings.warn(
I am slightly confused at this because I am using SQLAlchemy to create the connection on this line
engine = create_engine('mysql+pymysql://****#localhost/%s' % db_name )
Is it fine the way I am currently connecting? Please let me know how I should connect.

Connecting Python to Remote SQL Server

I am trying to connect Python to our remote SQL Server but I am not getting it. Following is a code that I used.
server = 'server,1433'
database = 'db'
username = 'username'
password = 'pw'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute('SELECT top 1 * FROM db.dbo.t_location')
for row in cursor:
print(row)
We have 2 servers. One is database server but I use application server for SQL which connects to database server. This is the error I'm getting. I am trying for a week but I'm not sure what am I missing here.
Any help would be appreciated
OperationalError: ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: No such host is known.\r\n (11001) (SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 17 for SQL Server]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. (11001)')
ADDED:
connection_str = ("Driver={SQL Server Native Client 11.0};"
"Server= 10.174.124.12,1433;"
#"Port= 1433;"
"Database=AAD;"
"UID=dom\user;"
"PWD=password;"
)
connection = pyodbc.connect(connection_str)
data = pd.read_sql("select top 1 * from dbo.t_location with (nolock);",connection)
I used the above code and now I see this error. Seems like it worked but failed to login. Usually I have to use Windows authentication in SSMS once I put my credentials to login in remote desktop.
('28000', "[28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'dom\user'. (18456) (SQLDriverConnect); [28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'dom\user'. (18456)")
Answer:
I am excited that I finally found a solution using pymssql. I don't know pyodbc wasn't working but I am sure I must have had done something wrong. I used below code to get the data from remote SQL server using Python.
import pymssql
conn = pymssql.connect(
host=r'10.174.124.12',
user=r'dom\user',
password=r'password',
database='db'
)
cursor = conn.cursor(as_dict=True)
cursor.execute('Select top 4 location_id, description from t_location with (nolock)')
data = cursor.fetchall()
data_df = pd.DataFrame(data)
cursor.close()
Ignore my code at this moment. I still have to do some cleaning but this code will work.

Finally to answer my question, I had to use pymssql which worked. I did not have to put the port number which was making me confused. Thanks everyone for taking out time to answer.
import pymssql
conn = pymssql.connect(
host=r'10.174.124.12',
user=r'dom\user',
password=r'password',
database='db'
)
cursor = conn.cursor(as_dict=True)
cursor.execute('Select top 4 location_id, description from t_location with (nolock)')
data = cursor.fetchall()
data_df = pd.DataFrame(data)
cursor.close()

you can use this function :
def connectSqlServer(Server , Database , Port , User , Password):
try:
conn = pyodbc.connect('Driver={SQL Server}; Server='+Server+';
Database='+Database+'; Port='+Port+'; UID='+User+'; PWD='+Password+';')
cursor = conn.cursor()
except Exception as e:
print("An error occurred when connecting to DB, error details: {}".format(e))
return False, None
else:
return True, cursor

How to put pandas DataFrame using pyodbc in SQL Server Management Studio

With the pandas DataFrame called 'data' (see code), I want to put it into a table in SQL Server. How should I do this?
I read something on the internet with data.to_sql, so I tried a little with this function. However, it gives me an error message (see error message). It probably has to do with the argument con = conn. I don't know how to fix this. The error message says something about SQLite. I am not very advanced in using SQL, so I do not understand all errors.
Do you have any ideas? I think the conn is the only problem. The rest works fine.
Using SQL Server Management Studio 2018.
Ty in advance.
Error message:
Exception has occurred: DatabaseError
Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;':
('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]
Invalid object name 'sqlite_master'. (208) (SQLExecDirectW);
[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]
Statement(s) could not be prepared. (8180)")
def connection():
try:
#connect to network database
conn = None
conn = db.connect('Driver={SQL Server};'
'Server=xxsql.database.windows.net;'
'Database=xx;'
'uid=xx;'
'pwd=xx;')
#if connected do
if conn is not None:
print("The connection is established")
sql_query = 'select top 20 * from xx'
#read table into dataframe
#make a dataframe with records from table
dfObj = pd.read_sql_query(sql_query, conn)
#inspect result
print(dfObj)
#read into list
cursor = conn.cursor()
cursor.execute(sql_query)
#put cursor to list
cursor_list = list(cursor.fetchall())
#make copy so that we wont change original data
table = cursor_list.copy()
bibliography = Extract(table)
data = pd.DataFrame(bibliography)
#convert it to SQL
data.to_sql('ExtractedBib', con = conn, if_exists = 'append') #????
except db.Error as e:
print(e)
finally:
if conn is not None:
cursor.close()
conn.close()
print("The connection is closed")
else:
print("No connection established")
def main():
connection()
main()

Try using sqlalchemy instead.
import sqlalchemy
engine = sqlalchemy.create_engine(
"xxsql+pyodbc://uid:pwd#Server/Database",
echo=False)
...
data.to_sql('ExtractedBib', con=engine, if_exists = 'append')
Just change the template values into your database values (uid, pwd etc.)

Bulk load failed due to invalid column value in CSV data file

I am trying to import data from a .csv file to a SQL server table. This works fine with running the SQL in Microsoft SQL Server Management Studio (SSMS). However, when I try to do it from Python with pyodbc, it gives me the following error:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0]
[SQL Server]Bulk load failed due to invalid column value in CSV data file
C:/~pathToFile~/file.csv in row 2, column 38. (4879) (SQLExecDirectW);
[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]The OLE DB
provider "BULK" for linked server "(null)" reported an error.
The provider did not give any information about the error. (7399);
[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)". (7330)')
My code so far is:
import pyodbc
# ------------------------------------------------------
# DEFINE FUNCTIONS
# ------------------------------------------------------
# Define a database command
def DB(SQL):
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=~myServer~;"
"Database=~myDB~;"
"Trusted_Connection=yes;")
try:
with cnxn.cursor() as cursor:
cursor.execute(SQL)
cnxn.commit()
finally:
cnxn.close()
# ------------------------------------------------------
# Import from .csv file to table
# ------------------------------------------------------
sql = '''BULK INSERT dbo.~myDB~
FROM 'C:/~pathToFile~/file.csv'
WITH (
FORMAT='CSV',
FIRSTROW = 2,
ROWTERMINATOR = '\n',
FIELDQUOTE= '"',
TABLOCK
)'''
DB(sql)
And here is the first few lines of the .csv file I'm trying to import:
SITE_ID,FIELD_SAMPLE_ID,LOCATION_ID,SAMPLE_DATE,PARAMETER_NAME,REPORT_RESULT,REPORT_UNITS,LAB_QUALIFIER,DETECTED,SAMPLE_MATRIX,SAMPLE_PURPOSE,SAMPLE_TYPE,SAMPLE_TIME,LATITUDE_(DECIMAL),LONGITUDE_(DECIMAL),FILTERED,FIELD_SAMPLE_COMMENTS,LAB_MATRIX,COC_#,LAB_METHOD,REPORT_DETECTION_LIMIT,SOURCE_FILENAME,WTR_SOURCE_FLOW,VALIDATION_QUALIFIER,VALIDATION_REASON_CODES,ANALYSIS_DATE,RESULT_TYPE,PARAMETER_CODE,LAB_RESULT,DILUTION_FACTOR,METHOD_DETECTION_LIMIT,INSTRUMENT_DETECTION_LIMIT,ANALYSIS_TYPE_CODE,ANALYSIS_TIME,QC_BATCH_SEQUENCE_#,SAMPLE_RESULT_COMMENTS,LAB_SAMPLE_ID,FIELD_SAMPLE_RESULT_RECORD_ID
"N3B","CAPA-08-11017","03-B-10","03-17-2008","RDX","0.325","ug/L","U","N","W","REG","WG","10:40","35.873716600000","-106.330115800000","N",,"W","08-824","SW-846:8321A_MOD","0.33",,"N","U","U_LAB","03-26-2008","TRG","121-82-4","0.325","2","0.13",,"INIT","00:00",,,"204935003","638"
"N3B","CAPA-08-13138","03-B-10","06-12-2008","RDX","0.325","ug/L","U","N","W","REG","WG","10:35","35.873716600000","-106.330115800000","N",,"W","08-1350","SW-846:8321A_MOD","0.33",,"N","U","U_LAB","06-24-2008","TRG","121-82-4","0.325","2","0.13",,"INIT","00:00",,,"210389014","638"
"N3B","CAPA-08-13139","03-B-10","06-12-2008","RDX","0.325","ug/L","U","N","W","FB","WG","10:35","35.873716600000","-106.330115800000","N",,"W","08-1350","SW-846:8321A_MOD","0.33",,"N","U","U_LAB","06-24-2008","TRG","121-82-4","0.325","2","0.13",,"INIT","00:00",,,"210389017","638"
Any idea why this will not work? Again, it works fine from SSMS, just not Python/pyodbc.

In some cases, you need to add a carriage return to the row terminator along with the new line.
Try adding \r to the row terminator parameter.
ROWTERMINATOR = '\r\n'

Azure Windows logins are not supported in this version of SQL Server while connecting sql server using python

Hi I'm using sql server V17.3 on Micorsoft Azure platform. I'm trying to upload data from python 3.7 data frame to my table test on sql server. So I wrote following code
import time
start_time = time.time()
import pyodbc
from sqlalchemy import create_engine
import urllib
dataToUpload=pd.read_csv("intermediate.csv")
params = urllib.parse.quote_plus(r'DRIVER=SQLServer};SERVER=nesbaexplsql001.database.windows.net;DATABASE=mydatabase;Trusted_Connection=False;Encrypt=True;uid=myuid;pwd=my password')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str)
dataToUpload.to_sql(name='test',con=engine, if_exists='append',index=False)
But I'm getting error message
DBAPIError: (pyodbc.Error) ('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver][SQL Server]Windows logins are not supported in this version of SQL Server. (40607) (SQLDriverConnect); [HY000] [Microsoft][ODBC SQL Server Driver][SQL Server]Windows logins are not supported in this version of SQL Server. (40607)') (Background on this error at: http://sqlalche.me/e/dbapi)
While executing to_sql. I also tried by putting Trusted_Connection=yes & removing Encryption=True.But I got same error. Can you guide me to resolve this issue?

Please refer to my sample code:
import pyodbc
import csv
server = 'tcp:***.database.windows.net'
database = '***'
username = '***'
password = '***'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute("select * from ***")
row = cursor.fetchone()
while row:
print(row[0])
row = cursor.fetchone()
mycsv = r'D:\insert.csv' # SET YOUR FILEPATH
with open (mycsv, 'r') as f:
reader = csv.reader(f)
columns = next(reader)
query = 'insert into <TABLE NAME>({0}) values ({1})'
query = query.format(','.join(columns), ','.join('?' * len(columns)))
cursor = cnxn.cursor()
for data in reader:
cursor.execute(query, data)
cursor.commit()

Add one piece to your connection string:
Integrated Security = False;
Hope this helps.

Categories