We Keep Coding

How to catch exception clickhouse_driver dbapi?

I want to catch exception while executing scipts/connecting to base using clickhouse_driver-drive dbapi.
Can I catch errors codes and errors message like
errorcodes.lookup(e.pgcode)
and
e.diag.message_primary
from
psycopg2.import errorcodes?

Assuming you're using the most well known clickhouse-driver from here: https://pypi.org/project/clickhouse-driver (GitHub here: https://github.com/mymarilyn/clickhouse-driver), you must catch standard exceptions/errors. Most errors are defined in the clickhouse_driver.connection module, and they include socket errors, EOF errors, and other lower level errors.
Even though the dbapi for that project defines exception classes, none of them are actually used in the code. The driver does not in any way use the errors or error codes from the PostgreSQL psycopg2 project.

How to handle syntax Errors

So, we had instance in the past where code were broken in IOT devices because of syntax errors.
While there is exception handling in the code. I wanted to create a script to check and make sure that the codes compiles and run without syntax error, else the script replace the broken code by an earlier version.
I tried this
from delta_script import get_update
def test_function():
try:
get_update()
except SyntaxError as syntaxError:
replace_script("utility.py", syntaxError)
except Exception as ignored:
pass
However the problem it when it hit a SyntaxError, it just throw it on the screen and replace_script
because the exception happens on delta_script.py from which get_update() was imported.
So what's the solution in this case?
I have also another function
def compile():
try:
for file in compile_list:
py_compile.compile(file)
except Exception as exception:
script_heal(file, exception)
however in this one, it never report any exception, because I go and introduce syntaxError and the code still compile without reporting an error
Any one could help me figure out a better way to solve those two problems?
thanks,

SyntaxErrors occur at compile time, not run time, so you generally can't catch them. There are exceptions, involving run time compilation using eval/exec, but in general, except SyntaxError: is nonsensical; something goes wrong compiling the code before it can run the code that sets up the try/except to catch the error.
The solution is to not write syntactically invalid code, or if you must write it (e.g. to allow newer Python syntax only when supported) to evaluate strings of said code dynamically with eval (often wrapping compile if you need something more complicated than a single expression) or exec.

How to find out where a Python Warning is from

I'm still kinda new with Python, using Pandas, and I've got some issues debugging my Python script.
I've got the following warning message :
[...]\pandas\core\index.py:756: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
return self._engine.get_loc(key)
And can't find where it's from.
After some research, I tried to do that in the Pandas lib file (index.py):
try:
return self._engine.get_loc(key)
except UnicodeWarning:
warnings.warn('Oh Non', stacklevel=2)
But that didn't change anything about the warning message.

You can filter the warnings to raise which will enable you to debug (e.g. using pdb):
import warnings
warnings.filterwarnings('error')
*The warnings filter can be managed more finely (which is probably more appropriate) e.g.:
warnings.filterwarnings('error', category=UnicodeWarning)
warnings.filterwarnings('error', message='*equal comparison failed*')
Multiple filters will be looked up sequentially. ("Entries closer to the front of the list override entries later in the list, if both match a particular warning.")

You can also use the commandline to control the warnings:
python -W error::UnicodeWarning your_code.py
From the man page:
-W argument
[...] error to raise an exception instead of printing a warning message.
This will have the same effect as putting the following in your code:
import warnings
warnings.filterwarnings('error', category=UnicodeWarning)
As was already said in Andy's answer.

The most informative way to investigate a warning is to convert it into an error (Exception) so you can see its full stacktrace:
import warnings
warnings.simplefilter("error")
See warnings.

If you enable logging in python, then when an exception is received you can use the method logging.exception to log when an exception has been caught - this method will print out a nicely formatted stack trace that shows you exactly in the code where the exception originated. See the python document on logging for more information.
import logging
log = logging.getLogger('my.module.logger')
try:
return self._engine.get_loc(key)
except UnicodeWarning:
log.exception('A log message of your choosing')
Alternatively, you can get a tuple that contains details of the exception in your code by calling sys.exc_info() (this requires you to import the sys module).

Properly handling IOError thrown by logging.config.fileConfig?

This may be an open ended or awkward question, but I find myself running into more and more exception handling concerns where I do not know the "best" approach in handling them.
Python's logging module raises an IOError if you try to configure a FileHandler with a file that does not exist. The module does not handle this exception, but simply raises it. Often times, it is that the path to the file does not exist (and therefore the file does not exist), so we must create the directories along the path if we want to handle the exception and continue.
I want my application to properly handle this error, as every user has asked why we don't make the proper directory for them.
The way I have decided to handle this can be seen below.
done = False
while not done:
try:
# Configure logging based on a config file
# if a filehandler's full path to file does not exist, it raises an IOError
logging.config.fileConfig(filename)
except IOError as e:
if e.args[0] == 2 and e.filename:
# If we catch the IOError, we can see if it is a "does not exist" error
# and try to recover by making the directories
print "Most likely the full path to the file does not exist, so we can try and make it"
fp = e.filename[:e.rfind("/")]
# See http://stackoverflow.com/questions/273192/python-best-way-to-create-directory-if-it-doesnt-exist-for-file-write#273208 for why I don't just leap
if not os.path.exists(fp):
os.makedirs(fp)
else:
print "Most likely some other error...let's just reraise for now"
raise
else:
done = True
I need to loop (or recurse I suppose) since there is N FileHandlers that need to be configured and therefore N IOErrors that need to be raised and corrected for this scenario.
Is this the proper way to do this? Is there a better, more Pythonic way, that I don't know of or may not understand?

This is not something specific to the logging module: in general, Python code does not automatically create intermediate directories for you automatically; you need to do this explicitly using os.makedirs(), typically like this:
if not os.path.exists(dirname):
os.makedirs(dirname)
You can replace the standard FileHandler provided by logging with a subclass which does the checks you need and, when necessary, creates the directory for the logging file using os.makedirs(). Then you can specify this handler in the config file instead of the standard handler.

Assuming it only needs to be done once at the beginning of your app's execution, I would just os.makedirs() all the needed directories without checking for their existence first or even waiting for the logging module to raise an error. If you then you get an error trying to start a logger, you can just handle it the way you likely already did: print an error, disable the logger. You went above and beyond just by trying to create the directory. If the user gave you bogus information, you're no worse off than you are now, and you're better in the vast majority of cases.

Which Python exception should I throw?

I'm writing some code to manipulate the Windows clipboard. The first thing I do is to try and open the clipboard with OpenClipboard() function from the Windows API:
if OpenClipboard(None):
# Access the clipboard here
else:
# Handle failure
This function can fail. So if it does, I would like to raise an exception. My question is, which of the standard Python exceptions should I raise? I'm thinking WindowsError would be the right one, but not sure. Could someone please give me a suggestion?

It is better to avoid raising standard exceptions directly. Create your own exception class, inherit it from the most appropriate one (WindowsError is ok) and raise it. This way you'll avoid confusion between your own errors and system errors.

Raise the windows error and give it some extra infomation, for example
raise WindowsError("Clipboard can't be opened")
Then when its being debugged they can tell what your windows error means rather than just a random windowserror over nothing.

WindowsError seems a reasonable choice, and it will record extra error information for you. From the docs:
exception WindowsError
Raised when a Windows-specific error occurs or when the error number does not correspond to an errno value. The winerror and strerror values are created from the return values of the GetLastError() and FormatMessage() functions from the Windows Platform API. The errno value maps the winerror value to corresponding errno.h values. ...