Problem
I am connecting with JayDeBeApi to SQL Server 2017 and running a script like:
SELECT ... INTO #a-temp-table
DELETE FROM a-table
INSERT INTO a-table SELECT FROM #a-temp-table
DELETE #a-temp-table
During step 3 i get the following error:
Cannot insert duplicate key row in object 'dbo.a-table' with unique index 'UQ_a-table'. The duplicate key value is (11, 0001, 3751191, T70206CAT, 0000).
Instead of ~360k records, only ~180k get inserted. So step 3 aborts.
The temp table however gets deleted. So step 4 completes.
I am able to fix the error. But with JayDeBeApi, I am not seeing the error.
It seems like everything went fine from the Python point of view.
My goal is to capture those errors to handle them appropriately.
Any idea how to achieve that?
What I've tried
My Python code looks like.
try:
localCursor = dbConnection.cursor()
x = localCursor.execute(query)
logInfo("Run script %s... done" % (scriptNameAndPath), "run script", diagnosticLog)
except Exception as e:
logError("Error running sql statement " + scriptNameAndPath + ". Skipping rest of row.",
"run script", e, diagnosticLog)
myrow = skipRowAndLogRecord(startRowTime, cursor, recordLog)
continue
x = localCursor.execute(myqrystm) completes successfully, so no exception is thrown. x is None and while inspecting localCursor, I see no sign of any error message(s)/code(s)
Step 3 should be all-or-none so the a-table should be empty following the duplicate key error unless your actual code has a WHERE clause.
Regarding the undetected exception, add SET NOCOUNT ON as the first statement in the script. That will suppress DONE_IN_PROC messages that will interfere with script execution unless your code handles multiple result sets.
https://learn.microsoft.com/en-us/sql/t-sql/language-elements/try-catch-transact-sql?view=sql-server-2017
-- Create procedure to retrieve error information.
CREATE PROCEDURE usp_GetErrorInfo
AS
SELECT
ERROR_NUMBER() AS ErrorNumber
,ERROR_SEVERITY() AS ErrorSeverity
,ERROR_STATE() AS ErrorState
,ERROR_PROCEDURE() AS ErrorProcedure
,ERROR_LINE() AS ErrorLine
,ERROR_MESSAGE() AS ErrorMessage;
GO
BEGIN TRY
-- Generate divide-by-zero error.
SELECT 1/0;
END TRY
BEGIN CATCH
-- Execute error retrieval routine.
EXECUTE usp_GetErrorInfo;
END CATCH;
Related
My colleagues and I are working on some code to produce SQL merge strings for users of a library we're building in Python to be run in the Azure Databricks environment. These functions provide the SQL string through a custom exception that we've written called DebugMode. The issue that we've encountered and I can't find a satisfactory answer to is why when the DebugMode string is printed do the <=> characters get removed? This can be replicated with a simpler example below where I've tossed various items into the Exception string to see what would get printed and what wouldn't.
raise Exception('this is a string with the dreaded spaceship <=> < > <= >= `<=>` - = + / <random> \<rand2>')
This snippet results in the following:
What I don't understand is why the <=> character is missing in the Exception printout at the top but is present when you expand the Exception. Is there a way to get the first string to include the <=> character?
I've also included the custom DebugMode class we're using.
class DebugMode(Exception):
'''
Exception raised in the event of debug mode being enabled on any of the merge functions. It is intended to halt the merge and provide
the SQL merge string for manual review.
Attributes:
sql_string (str): The sql merge string produced by the merge function.
Methods:
None
'''
def __init__(self, sql_string, message='Debug mode was enabled, the SQL operation has halted to allow manual review of the SQL string below.'):
self.sql_string = sql_string
self.message = message
super().__init__(self.message) # overwrite the Exception base classe's message
def __str__(self):
return f'{self.message}\n{self.sql_string}'
Just providing a follow up for anyone who runs into the same thing. This was a Databricks notebook specific issue. #user2357112 had indicated, this was a problem where Databricks was parsing the html and had reserved the notation <some key work> for specific purposes (you can see some of these keywords and how they're using them here: https://docs.databricks.com/error-messages/index.html).
As #tdelaney noted, this isn't an issue in Jupyter notebooks or the python shell.
I written the below Try-Except code in a For Loop for Python - PostgreSQL database input.
I have a list in .csv file to input into the PostgreSQL database.
1 of the column is primary data enabled.
Hence, if the data in the list is duplicated, the Python will throw error.
I written below code and my imagination is the code will run "try" 1st, if "try" error then only jump to the "except" line, for every rows in the list (loop).
But, when I execute my code, once the program go to "except" line, then the program would not go to "try" line again.
Ex.
If my list of data all fresh, the program run and input all rows to database correctly.
If my duplicated data is in the 1st line of the list, the program run to "except" line and the fresh data at the bottom are not input to the database.
As long as there is duplicated data on top, the program will just run into "except" line without go back to "try" line.
Any idea how to improve my code here? Thank you.
My intention is all fresh data need to capture into the database, while there is duplicated, the data row shall be skipped and continue for next line.
for data in df.iterrows():
vreceivingdate = str(data[1][0])
vreceivingtime = str(data[1][1])
vscanresult = str(data[1][2])
vinvoicenbr = str(vscanresult[0:6])
vlotnbr = str(vscanresult[6:9])
vcasenbr = str(vscanresult[9:12])
try:
rec.insertRec(vreceivingdate, vreceivingtime, vscanresult, vinvoicenbr, vlotnbr, vcasenbr)
except:
dupDataCounter += 1
Finally I found the solution for my question.
I shall not use Try-Except for this case.
To do what I want to do should use the PostgreSQL function:
"ON CONFLICT DO NOTHING".
The answer I'm looking for is a reference to documentation. I'm still debugging, but can't find a reference for the error code.
The full error comment is:
snowflake.connector.errors.OperationalError: 255005: Failed to read next arrow batch: b'Array length did not match record batch length'
Some background, if it helps:
The error is in response to the call to fetchall as shown here (python):
cs.execute (f'SELECT {specific_column} FROM {table};')
all_starts = cs.fetchall()
The code context is: when running from cron (i.e., timed job), successful connect, then for a list of tables, the third time through, there's an error. (I.e., two tables are "successful".) When the same script is run at other times (via command line, not cron), there's no error (all tables are "successful").
I have a python script that I want to perform an action for each row found in a table or report if no rows were found.
This is some dummy code for troubleshooting
cur_users = lv_pgsql.cursor()
cur_users.execute("select * from users where us_idno > 10")
for lr_users in cur_users:
print("ping")
I know I will be able to check if I attempt to store the results in an array like bellow
la_users = cur_users.fetchall()
if len(la_users) != 0:
print("ping")
But is it possible to check with the above code as well without using fetchall?
You can wrap your code in a try except block, thus if the cursor is None it will throw an error. Also by doing this you can capture other errors and handle them.
if cur_users.execute("select * from users where us_idno > 10").rowcount:
print "ping!"
should do what you want?
I have the following code that works fine when I run it from my localhost, but it fails when I run it from Amazon. There is no differences in the code, its using the same database connection.
I use pymysql and I use dictCursor to fetch the data by column name. This is how I create the link:
self.link = self.db.cursor(pymysql.cursors.DictCursor)
My problem is when I try to fetch the data. The following simple code snippet fails:
try:
self.link.execute('SELECT * FROM crawler_data WHERE id="%d" LIMIT 1' % id_crawl);
row = self.link.fetchone()
except pymysql.err.Error as ex:
print "Can not fetch crawler data from database: %s" % ex
print "Before"
try:
if row["address"]:
print "Filter Address: %s" % row["address"]
except Exception as ex:
print "Could not filter the data: %s" % ex.message
print "After"
It does not even print "Filter Address". I guess that means that it fails on if row["address"]:. It does print both "Before" and "After".
I have also tried row[u"address"]: without any success.
When I print the row. I get the following data printed:
{u'address': u'Ekhammargr\\xe4nd 7'}
So questions:
My except is done, and it prints "Could not filter the data:". It does not print ex.message. How do I get the exception or error that is causing the try to fail?
I'm using the same code and the same database connection, fetching the same data. It works fine on my localhost Windows machine, but not on my Amazon Web Services Elastic Beanstalk Linux App. How could this be?
EDIT:
I tried changing the if statement to:
if True:
print "Filter Address"
It still failed on Amazon. If it was indentation error it should'nt work on localhost either... Any ideas?
Here is the complete function: http://pastebin.com/haJGFuj3
For some unknown reason, when I rewrote the code exactly the same, it started working. I was only using tabs as indentations and no spaces, I tripple checked and I didnt get any indentation error. So I'm not sure what could be causing these issues.
But when I rewrote the code from scratch, with same code. It worked.
EDIT: I was using Sublime Text 2 as editor and utf8 as encoding.