my stored procedure works fine on its own but my python script fails to fully execute the stored procedure with my downloaded files. the purpose of the python script is to download files using ftp and store the files locally.It first compares the remote location and the local location to find new files and then download the new files to the local location. And then executes the stored procedure on each new file.
python script:
import os
import ftplib
import pyodbc
connection to sql server*
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=localhost;DATABASE=Development;UID=myid;PWD=mypassword')
cursor = conn.cursor()
ftp = ftplib.FTP("myftpaddress.com")
ftp.login("loginname", "password")
print 'ftp on'
#directory listing
rfiles = ftp.nlst()
print 'remote listing'
#save local directory listing to files
lfiles = os.listdir(r"D:\Raw_Data\myFiles")
print 'local listing'
#compare and find files in rfiles but not in lfiles
nfiles = set(rfiles) - set(lfiles)
nfiles = list(nfiles)
print 'compared listings'
#loop through the new files
#download the new files and open each file and run stored proc
#close files and disconnect to sql server
for n in nfiles:
local_filename = os.path.join(r"D:\Raw_Data\myFiles",n)
lf = open(local_filename, "wb")
ftp.retrbinary("RETR " + n, lf.write, 1024)
lf.close()
print 'file written'
cursor.execute("exec SP_my_Dailyfiles('n')")
conn.close()
lf.close()
print 'sql executed'
ftp.quit()
stored proc:
ALTER PROCEDURE [dbo].[SP_my_Dailyfiles]
-- Add the parameters for the stored procedure here
#file VARCHAR(255)
-- Add the parameters for the stored procedure here
AS
BEGIN
IF EXISTS(SELECT * FROM sysobjects WHERE name = 'myinvoice')
DROP TABLE dbo.myinvoice
----------------------------------------------------------------------------------------------------
CREATE TABLE myinvoice(
[Billing] varchar(255)
,[Order] varchar(45)
,[Item] varchar(255)
,[Quantity in pack] varchar(255)
,[Invoice] varchar(255)
,[Date] varchar(255)
,[Cost] varchar(255)
,[Quantity of pack] varchar(255)
,[Extended] varchar(255)
,[Type] varchar(25)
,[Date Due] varchar(255)
)
----------------------------------------------------------------------------------------------------
DECLARE #SourceDirectory VARCHAR(255)
DECLARE #SourceFile VARCHAR(255)
EXEC (' BULK
INSERT dbo.myinvoice
FROM ''D:\Raw_Data\myfile\'+#file+'''
WITH
(
FIRSTROW = 1,
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''0x0a''
)'
)
-------------------------------------------------------------------------------------------------------------
INSERT INTO [Development].[dbo].[my_Dailyfiles](
[Billing]
,[Order]
,[Item]
,[Quantity in pack]
,[Invoice]
,[Date]
,[Cost]
,[Quantity of pack]
,[Extended]
,[Type]
,[Date Due]
,[FileName]
,[IMPORTEDDATE]
)
SELECT
replace([Billing], '"', '')
,replace([Order], '"', '')
,replace([Item], '"','')
,replace([Quantity in pack],'"','')
,replace([Invoice],'"','')
,cast(replace([Date],'"','') as varchar(255)) as date
,replace([Cost],'"','')
,replace([Quantity of pack],'"','')
,replace([Extended],'"','')
,replace([Type],'"','')
,cast(replace([Date Due],'"','') as varchar(255)) as date
,#file,
GetDate()
FROM [myinvoice] WHERE [Bill to] <> ' ' and ndc != '"***********"'
I think the problem may be that you are closing the DB connection immediately after you execute the stored procedure, whilst still in the loop.
This means the second time around the loop, the DB connection is closed when you try to execute the SP. I would actually expect an error to be thrown the second around the loop.
The way I would structure this is something like:
conn = pyodbc.connect(...)
for n in nfiles:
...
cursor = conn.cursor()
cursor.execute("exec SP_my_Dailyfiles('n')")
conn.commit()
Related
After some data manipulation I store two columns in a txt file in a csv format as following:
result.txt ->
id,avg
0,38.0
1,56.5
3,66.5
4,48.666666666666664
then I store the data in a table, which is where i find trouble, i tried running a .sql query that stores the data successfully, but executing the same query from python doesnt seem to work for some reason.
python code->
.
.
.
open('C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/result.txt', 'w').write(res)
print(res)
try:
with mysql.connector.connect(
host="localhost",
user='root',
password='tt',
database="dp",
) as connection:
clear_table_query = "drop table if exists test_db.marks;"
create_table_query = '''
create table test_db.marks (
id varchar(255) not null,
avg varchar(255) not null,
primary key (id)
);
'''
# droping the table and recreating it works fine
add_csv_query = "LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/result.txt' INTO TABLE marks FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\\n' IGNORE 1 LINES;"
print(add_csv_query) # query is printed correctly
with connection.cursor() as cursor:
cursor.execute(clear_table_query)
cursor.execute(create_table_query)
cursor.execute(add_csv_query)
cursor.execute("SELECT * FROM test_db.marks;") # this produces -> Unread result found
except mysql.connector.Error as e:
print(e)
connection.close()
The following procedure works fine from mysql client but not running from Python.
Stored Procedure
CREATE DEFINER=`music-cnv`#`%` PROCEDURE `StoreFileStats`(FNAME VARCHAR(200), FEXT varchar(4), FBDIR VARCHAR(100), FRDIR VARCHAR(250), FSIZE bigint(8), FMDATE bigint(8), FCDATE bigint(8), CONVERTED tinyint(1))
BEGIN
DECLARE FCount int DEFAULT 0;
SELECT COUNT(FileName) INTO FCount FROM FileList where (FleRelativeDir LIKE FRDIR) AND (FileName LIKE FNAME);
IF FCount = 0 THEN
INSERT INTO FileList (FileName,FileBaseDir,FleRelativeDir,FileExt,FileSize,FileModDate,FileCDate,Con#verted) VALUES (FNAME,FBDir,FRDir,FEXT,FSize,FMDate,FCDate,CONVERTED);
END IF;
END
Data
'In the Light', 'FLAC', '/var/data/Music_FLAC', 'Led Zeppelin/Physical Graffiti, Disc 2', 51472669, 1289282499, 1458631127, False
Python Code
The connection and cursor give no errors
try:
myargs = [fnamesub, self.type.strip(), self.directory,
subdirname, fpathstat[2], fpathstat[3],
fpathstat[4], False]
result_args = mycur.callproc('StoreFileStats', myargs)
except mysql.connector.Error as Err:
errno = 51
print('Error ' + str(errno) + ' !!!, Cannot Update MySQL Data with Name ' + fnamesub)
print(Err)
The code runs without error but does not update database
Thank you for any help
Just needed to commit. When running procedure from shell auto commit is enabled. But when run from Python must manually run commit.
I have written a small script to create a MySQL database, create a table (previously erase it if already exists), and insert many entries. When I execute my script, it works creating the database and table, but does not write any entry to the table:
from warnings import filterwarnings
import MySQLdb as db
filterwarnings('ignore', category = db.Warning)
try:
db_name = 'chom'
con = db.connect(user='user', passwd='pass')
cur = con.cursor()
# Create new database
cur.execute('CREATE DATABASE IF NOT EXISTS ' + db_name + ';')
# Create PARAMETERS table
cur.execute('DROP TABLE IF EXISTS ' + db_name + '.PARAMETERS;')
query = ('CREATE TABLE ' + db_name + '.PARAMETERS ('
'idPARAMETERS INT(10) NOT NULL AUTO_INCREMENT, '
'Param_name VARCHAR(30) NULL DEFAULT NULL, '
'Param_value VARCHAR(255) NULL DEFAULT NULL, '
'Timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP '
'ON UPDATE CURRENT_TIMESTAMP, '
'User_id VARCHAR(20) NULL DEFAULT NULL, '
'PRIMARY KEY (idPARAMETERS) );'
)
cur.execute(query)
# Insert entries
parameters = ['param1', 'param2', 'param3',
'param4']
for i, param_name in enumerate(parameters, start=1):
cur.execute('INSERT INTO ' + db_name + '.PARAMETERS '
'(idPARAMETERS, Param_name, Param_value, User_id) '
'VALUES (' + str(i) + ', %s, %s, %s);',
(param_name, '', 'user2#localhost'))
cur.close()
con.commit()
except Exception, e:
print 'Error. Last query: ' + str(cur._last_executed)
print e
print 'DB installation script finished'
I can't see where the problem is. Any idea?
The code worked correctly, it was mysql-workbench fault, which was not showing the correct database content (while mysql console client did).
I am trying to access SQL server 2008 R2 from Eclipse pydev ( python 3.2 ) on win7 .
I need to create a table on database.
The code can be run well. But, I cannot create tables in the database.
If I print the sql string and run the query from SQL server management studio, no problems.
import pyodbc
sql_strc = " IF OBJECT_ID(\'[my_db].[dbo].[my_table]\') IS NOT NULL \n"
sql_strc1 = " DROP TABLE [my_db].[dbo].[my_table] \n"
sql_stra = " CREATE TABLE [my_db].[dbo].[my_table] \n"
sql_stra1 = "(\n"
sql_stra1a = " person_id INT NOT NULL PRIMARY KEY, \n"
sql_stra1b = " value float NULL, \n"
sql_stra1r = "); \n"
sql_str_create_table = sql_strc + sql_strc1 + sql_stra + sql_stra1 + sql_stra1a + sql_stra1b + sql_stra1r
# create table
sql_str_connect_db = "DRIVER={SQL server};SERVER={my_db};DATABASE={my_table};UID=my_id; PWD=my_password"
cnxn = pyodbc.connect(sql_str_connect_db)
cursor = cnxn.cursor()
cursor.execute( sql_str_create_table)
Any help would be appreciated.
Thanks
Autocommit is off by default, add the following to commit your change:
cnxn.commit()
Some unsolicited advice for making your code more readable:
Remove unnecessary escape characters from SQL strings
Use triple-quote (""") syntax when defining multiline strings. Newline characters are preserved and don't need to be explicitly added.
Use keywords in the connect function call (this is trivial, but I think it makes formatting easier)
With these changes, your final code looks something like:
import pyodbc
sql = """
IF OBJECT_ID('[my_db].[dbo].[my_table]') IS NOT NULL
DROP TABLE [my_db].[dbo].[my_table]
CREATE TABLE [my_db].[dbo].[my_table]
(
person_id INT NOT NULL PRIMARY KEY,
value FLOAT NULL
)
"""
cnxn = pyodbc.connect(driver='{SQL Server}', server='server_name',
database='database_name', uid='uid', pwd='pwd')
cursor = cnxn.cursor()
# create table
cursor = cursor.execute(sql)
cnxn.commit()
I've got some problems with my SQLite database. I've got some sql script like this:
CREATE TABLE Workers(
Id_worker INT NOT NULL PRIMARY KEY,
Name VARCHAR(20) NOT NULL,
Surname VARCHAR(30) NOT NULL,
Id_job INT NOT NULL, -- REFERENCES Job(Id_job),
Adress VARCHAR(30) NOT NULL,
Start_date SMALLDATETIME NOT NULL
);
CREATE TABLE OldWorkers(
Id_arch INT NOT NULL PRIMARY KEY,
Name VARCHAR(20) NOT NULL,
Surname VARCHAR(30) NOT NULL,
Id_job INT NOT NULL, -- REFERENCES Job(Id_job),
Adress VARCHAR(30) NOT NULL,
Start_date SMALLDATETIME NOT NULL,
Delete_date SMALLDATETIME NOT NULL
);
CREATE TRIGGER OldWorkersTrigger
AFTER DELETE ON Workers
FOR EACH ROW
BEGIN
INSERT INTO OldWorkers (Id_arch, Name, Surname, Id_job, Adress, Start_date) VALUES (old.Id_arch, old.Name, old.Surname, old.Id_job, old.Adress, old.Start_date,datatime('now'));
END;
I try to do it in Python 2.7.4 by sqlite3 like this:
conn = sqlite3.connect('Company.db')
c = conn.cursor()
fd = open('MyScript.sql', 'r')
sqlFile = fd.read()
fd.close()
# all SQL commands (split on ';')
sqlCommands = sqlFile.split(';')
i = 1
# Execute every command from the input file
for command in sqlCommands:
# This will skip and report errors
# For example, if the tables do not yet exist, this will skip over
# the DROP TABLE commands
print i," : ",command
i = i + 1
try:
c.execute(command)
except OperationalError, msg:
print "Command skipped: ", msg
But in command wiht my trigget it returns: Command skipped: near ")": syntax error.
And after END; it returns: Command skipped: cannot commit - no transaction is active
You are splitting the file at every semicolon, but the CREATE TRIGGER statement has an embedded semicolon.
To check whether a statement is complete, try the sqlite3.complete_statement function.
SQLite can ingest and execute arbitrary SQL statements.
Why not consider the following (untested):
conn = sqlite3.connect('Company.db')
c = conn.cursor()
with open('MyScript.sql', 'r') as fd:
try:
c.execute(fd.read())
except OperationalError, msg:
print "Command skipped: ", msg