We Keep Coding

Unestablished connection and content manager

The following function allows to use the with-statment and the a pop connection. But if the connection is not established, the quit() in finally will raise an exception. How can this be fixed?
#contextmanager
def pop_connect(server, user, password, timeout, use_SSL=False):
try:
pop = poplib.POP3_SSL if use_SSL else poplib.POP3
pop_conn = pop(server, timeout=timeout)
pop_conn.pass_(password)
yield pop_conn
except poplib.error_proto as pop_error:
print('Authentication for receiving emails failed:{}'.format(pop_error))
except OSError as os_error:
print('Name resolution or connection failed:{}'.format(os_error))
finally:
pop_conn.quit()

I suppose you can put your pop_conn.quit() in a try: with pass as corresponding except action:
finally:
try:
pop_conn.quit()
except <WhaterverException>:
pass

The solution is to rethrow the exceptions in the handlers. The contextmanager than does not except the yield:
#contextmanager
def pop_connect(server, user, password, timeout, use_SSL=False):
try:
pop_conn = poplib.POP3_SSL(server,timeout=timeout) if use_SSL else poplib.POP3_SSL(server,timeout=timeout)
pop_conn.user(user)
pop_conn.pass_(password)
yield pop_conn
except poplib.error_proto as pop_error:
print('Receiving mail failed:{}'.format(pop_error))
raise
except OSError as os_error:
print('Name resolution or connection failed:{}'.format(os_error))
raise
finally:
try:
pop_conn.quit()
except UnboundLocalError:
pass

Managing a database connection with a global, global name disappears

class MyAddon(pyxbmct.AddonDialogWindow):
def __init__(self, title=''):
super(MyAddon, self).__init__(title)
self.mysql_connect()
self.populate()
def populate(self):
categories = self.read_data()
def read_data(self):
query = ("SELECT category FROM test")
cursor = connection.cursor()
categories = cursor.execute(query)
return categories
def mysql_connect(self):
global connection
try:
connection = mysql.connector.connect(**config).cursor()
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
xbmc.executebuiltin('Notification(Error!, Bad user name of password)')
elif err.errno == errorcode.ER_BAD_DB_ERROR:
xbmc.executebuiltin('Notification(Error!, Database does not exist)')
else:
xbmc.executebuiltin('Notification(Error!, {0})'.format(err))
I'm developing a Python add-on for Kodi. I get a Global name 'connection' is not defined error when trying to use a global variable for the database connection. I cannot read the global variable connection from the read_data function. I'm sure this is not a forward reference problem because I tested it that way.
The purpose of using a global variable for the connection is to reuse the connection in all the functions without creating a new connection every time.

It may be that Kodi does something funky with namespaces or your instances are pickled; when unpickled the global will be gone. Another problem with a global like this is that the connection might be lost at some point.
I'd restructure the code to have a method that returns a connection, and use that in all methods that require a connection. Make the connection method a classmethod and allow for the connection to be gone:
class MyAddonConnectionFailed(Exception): pass
def read_data(self):
query = ("SELECT category FROM test")
try:
conn = self.connect()
except MyAddonConnectionFailed:
# connection failed; error message already displayed
return []
cursor = conn.cursor()
categories = cursor.execute(query)
return categories
_connection = None
#classmethod
def connect(cls):
if cls._connection and cls._connection.open:
return cls._connection
try:
cls._connection = mysql.connector.connect(**config).cursor()
return cls._connection
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
xbmc.executebuiltin('Notification(Error!, Bad user name of password)')
elif err.errno == errorcode.ER_BAD_DB_ERROR:
xbmc.executebuiltin('Notification(Error!, Database does not exist)')
else:
xbmc.executebuiltin('Notification(Error!, {0})'.format(err))
raise MyAddonConnectionFailed
I'm raising an exception in the connect classmethod; you'll need to decide how you want to handle the case where your database is misconfigured or can't connect. Displaying an error message is not enough. You could still call self.connect() from the __init__ method to signal this problem early of course.

How to properly use try/except in Python

I have a function that returns the DB connection handler from MongoDB. I have various other functions that makes a call to the DB, I figure let's throw the connection handler into a function so I don't have to define it in every function.
Does this look right? I guess my question is, if it can't make a connection to the DB server, it will print both messages Could not connect to server and No hosts found How can I go about only printing "Could not connect to the server."
def mongodb_conn():
try:
conn = pymongo.MongoClient()
except pymongo.errors.ConnectionFailure, e:
print "Could not connect to server: %s" % e
return conn
def get_hosts()
try:
conn = mongodb_conn()
mongodb = conn.dbname.collection
b = []
hosts_obj = mongodb.find({'_id': 'PR'})
for x in hosts_obj:
print x
except:
print "No hosts found"
get_hosts()

Move your conn = mongodb_conn() call out of the try .. except handler, and test if None was returned:
def get_hosts()
conn = mongodb_conn()
if conn is None:
# no connection, exit early
return
try:
mongodb = conn.dbname.collection
b = []
hosts_obj = mongodb.find({'_id': 'PR'})
for x in hosts_obj:
print x
except:
print "No hosts found"
You should, at all cost, avoid using a blanket except however; you are catching everything now, including memory errors and keyboard interrupts, see Why is "except: pass" a bad programming practice?
Use specific exceptions only; you can use one except statement to catch multiple exception types:
except (AttributeError, pymongo.errors.OperationFailure):
or you can use multiple except statements handle different exceptions in different ways.
Limit the exception handler to just those parts of the code where the exception can be thrown. The for x in hosts_obj: loop for example is probably not going to throw an AttributeError exception, so it should probably not be part of the try block.
Note that you'll need to adjust your mongodb_conn() function to not try and use the conn local if it has never been set; you'll get an UnboundLocal error if you do:
def mongodb_conn():
try:
return pymongo.MongoClient()
except pymongo.errors.ConnectionFailure, e:
print "Could not connect to server: %s" % e
Now the function returns the connection if successful, None if the connection failed.

You can also check if the server is available
like this:
from pymongo.errors import ConnectionFailure
client = MongoClient()
try:
# The ismaster command is cheap and does not require auth.
client.admin.command('ismaster')
except ConnectionFailure:
print("Server not available")

When using multiprocessing to access MySQL, it always throws the following error, how to fix?

When I used Python multiprocessing to access MySQL database, always got the errors:
OperationalError: (2006, 'MySQL server has gone away')
Lost connection to MySQL server during query
I would greatly appreciate if somebody could explain this to me.
Here is my code:
class MetricSource:
def __init__(self, task):
self.task = task
def get_next_task_time(self):
try:
task_id = self.task.id
next_task = Task.objects.get(id=task_id)
next_time = next_task.last_success_time
except Task.DoesNotExist as e:
print 'Find Exception: %d' % self.task.id
def access_queue(input_queue, output_queue):
while True:
try:
metric_source = input_queue.get(timeout=0.5)
metric_source.get_next_task_time()
output_queue.put(metric_source)
except Queue.Empty:
print "Queue Empty Error"
continue
class Command(BaseCommand):
def handle(self, *args, **options):
self.manager = multiprocessing.Manager()
self.input_queue = self.manager.Queue()
self.output_queue = self.manager.Queue()
self.init_data()
for i in range(PROCESS_NUM):
Process(target=access_queue, args=(self.input_queue, self.output_queue)).start()
def init_data(self):
for i in range(200):
try:
task = Task.objects.get(id=i+1)
self.input_queue.put(MetricSource(task))
except Exception as e:
print 'find task_id %d' % i
continue
except IOError as e:
print "find IOError: %r" % e
continue
And I have doubt that it's my MySQL configuration's problem, but I think it's not the problem.
Here is my.cnf:
[mysqld]
default-character-set=utf8
collation_server = utf8_general_ci
character_set_server = utf8
max_allowed_packet = 100M
datadir=/home/work/data1/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
slow_query_log
slow_query_log_file=/home/work/data1/mysql/mysql-slow.log
max_allowed_packet=100M
log-error=/home/work/data1/mysql/error.log
general_log
general_log_file=/home/work/data1/mysql/mysql.log
tmp_table_size=2G
max_heap_table_size=2G
wait_timeout=2880000
interactive_timeout=2880000
innodb_data_home_dir=/home/work/data1/mysql/ibdata/
[mysqld_safe]
default-character-set=utf8
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

i have found the reason why the db connection crash, share some points here.
The problem is caused by the order of 'init_data' and 'fork subProcess':
self.init_data()
for i in range(PROCESS_NUM):
Process(target=access_queue, args=(self.input_queue, self.output_queue)).start()
in this order, when executing init_data(), the db connection has been built and then fork the subProcess, then the connection is copyed into the subprocess. So all the subprocess are using the same connection actually, it will cause some indeterminate problems definitely.
When i change the order to:
for i in range(PROCESS_NUM):
Process(target=access_queue, args=(self.input_queue, self.output_queue)).start()
self.init_data()
and add sleep for subprocess:
def access_queue(id, input_queue, output_queue):
time.sleep(5)
while True:
...
it works. Because under this change, the subProcesses will be constructed before the connection built and then the subProcesses will use their separate connection to db.
so i have a question:
Is there any graceful solutions to resolve/prevent this kind of problems in multiprocessing to access db by orm ?
Anyone to share some points?
thanks
Have found a good post: Django multiprocessing and database connections

Python: MySQL connection is open, but can't create cursor

I'm trying to open a cursor to a MySQL-DB. But I'm getting this error:
'NoneType' object has no attribute 'cursor'
Here is a small sourcecode:
class Sample:
def __init__(self):
self.conn = None
self.value = self.setValue()
def connect(self):
self.conn = MySQLdb.connect(...)
#cursor = self.conn.cursor()
#cursor.execute("SELECT ...")
#value = str(cursor.fetchone()[0])
#raise Exception(value)
#cursor.close() <- here everything is working fine
def setValue(self):
if (self.conn == None):
self.connect()
#raise Exception(self.conn.open)
cursor = self.conn.cursor() # ERROR: 'NoneType' object has no attribute 'cursor'
...
If I use the exception I get a 1 ... connection is open.
And if I do the cursor creation and the SQL statement in the 'connect' function everything is working well.
The strange this is, everything looks correct and for some other connections with the same functions everything is working well, too. I don't know how to solve this error. I hope someone can point me in the right direction.

I would change the statement that checks if the connection is open to both check if conn is none as well as if the connection is open. And because you always execute the setValue function I would recommend that you call the connect inside the__init__ function.
class Sample:
conn = None
def __init__(self):
self.connect()
self.value = self.setValue()
self.close()
def connect(self):
self.conn = MySQLdb.connect(...)
def close(self):
if self.conn:
self.conn.close()
def setValue(self):
if not self.conn and not self.conn.open:
self.connect()
cursor = self.conn.cursor()
Also, remember that with the Python MySQL Connector you need to call commit after you execute a insert or update statement.
cur = self.conn.cursor()
cur.execute("...")
self.conn.commit()