I want to create socket errors (By doing things, obviously) but I've no idea how I should test if my script handles errors properly (If it dectes them.)
Currently, my code is this:
except socket.error as err:
print "Connection lost, waiting..."
time.sleep(5)
In theory, it should handle all the socket errors, print and then sleep (It's a part of a while loop.).
Any idea of how can I test it to see how it handles errors?
Use the raise statement:
try:
raise socket.error
except socket.error as err:
print "Connection lost, waiting..."
time.sleep(5)
Yet another example:
try:
raise AttributeError
except AttributeError:
print 'Sorry'
#Sorry
Also take a look at here and here
I am trying to have a client connect to my server, and have a stream of communication between them. The only reason the connection should break is due to network errors, or unless the client wants to stop talking.
The issue I am running into is keeping the handler in a tight loop, and parsing the JSON.
My server code is :
#!/usr/bin/env python
import SocketServer
import socket
import json
import time
class MyTCPServer(SocketServer.ThreadingTCPServer):
allow_reuse_address = True
class MyTCPServerHandler(SocketServer.BaseRequestHandler):
def handle(self):
while 1:
try:
networkData = (self.request.recv(1024).strip())
try:
jsonInputData = json.loads(networkData)
print jsonInputData
try:
if jsonInputData['type'] == 'SAY_HI':
print "HI"
except Exception, e:
print "no hi"
pass
try:
if jsonInputData['type'] == 'GO_AWAY':
print "Going away!"
except Exception, e:
print "no go away"
pass
except Exception, e:
pass
#time.sleep(0.001)
#print "JSON Error", e
except Exception, e:
#time.sleep(0.001)
pass
#print "No message", e
server = MyTCPServer(('192.168.1.115', 13373), MyTCPServerHandler)
server.serve_forever()
My client code is simple :
#!/usr/bin/env python
import socket
import json
import time
import sys
hostname = '192.168.1.103'
port = 13373
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((hostname,port))
except Exception, e:
print "Error, could not open socket: ", e
data = {'type':'SAY_HI'}
sock.send(json.dumps(data))
data = {'type':'SAY_BYE'}
sock.send(json.dumps(data))
Sometimes I'll see the messages being sent, "SAY_HI" and "SAY_BYE", but most of the times, no data is being displayed on the server side.
This question is really not clear, but calling self.request.recv(1024) is very likely not what you want to do. You're eliminating all of the nice application-level handling that TCP will happily do for you. If you change that to self.request.recv(8) or a similarly very small number (such that recv() returns whenever it receives data, and doesn't try to fill your buffer), you may get better results.
Ultimately this is super-simplistic change, even if it works, that will not work in a larger context. You will need to be handling exceptions from your json parser on the server side and waiting for more data until an entire well-formed message is received.
This is a hopelessly more complex subject than will be handled generally in any SO answer. If you're going to be doing any amount of raw sockets programming, you absolutely must own a copy of Unix Network Programming, Volume 1.
I'm trying to catch an error in memcached when this one is down with python:
import memcache
import socket
mc = memcache.Client(['127.0.0.1:11211'], debug=1)
try:
print mc.get('gfdsgf')
except socket.error:
print 'error'
But I still have this error in my console:
MemCached: MemCache: inet:127.0.0.1:11211: connect: Connection refused. Marking dead.
None
this is not really an error that you can catch this is just a log, and it's displayed because you have the debug parameter to 1, so turn off the debug parameter. And as you can see you still get None from your print that mean your key doesn't exist
Try something like that:
import memcache
mc = memcache.Client(['127.0.0.1:11211'], debug=0)
try:
print mc.get('gfdsgf')
except (mc.MemcachedKeyTypeError, mc.MemcachedKeyNoneError,
TypeError, mc.MemcachedKeyCharacterError,
mc.MemcachedKeyError, mc.MemcachedKeyLengthError,
mc.MemcachedStringEncodingError):
print 'error'
I need use port forwarding in Python to communicate with a remote MySQL DB through an SSH tunnel. I downloaded the paramiko package and tried out the port forwarding demo (forward.py). It works very nicely, but I am having trouble integrating it into my own scripts (similar to the one below). When the main forwarding function is called it enters an infinite loop and the rest of my code does not execute. How can I use the forward.py demo and get past the infinite loop?
My script:
import paramiko
import forward
import MySQLdb
import cfg
import sys
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.WarningPolicy())
try:
client.connect(cfg.remhost, cfg.remport, username=cfg.user, password=cfg.password)
except Exception, e:
print '*** Failed to connect to %s:%d: %r' % (cfg.remhost, cfg.remport, e)
sys.exit(1)
try:
forward.forward_tunnel(3306, cfg.remhost, 3306, client.get_transport())
except KeyboardInterrupt:
print 'C-c: Port forwarding stopped.'
sys.exit(0)
try:
db = MySQLdb.connect('127.0.0.1', cfg.dbuser, cfg.dbpass, cfg.dbname)
except Exception, e:
print 'Failed to connect to database'
sys.exit(1)
try:
cursor = self.db.cursor(MySQLdb.cursors.DictCursor)
sql = 'SELECT * FROM ' + cfg.dbtable
cursor.execute(sql)
results = cursor.fetchall()
print str(len(results))
except Exception, e:
print 'Failed to query database'
sys.exit(1)
Here is the main chunk of the forward.py demo code:
class ForwardServer (SocketServer.ThreadingTCPServer):
daemon_threads = True
allow_reuse_address = True
class Handler (SocketServer.BaseRequestHandler):
def handle(self):
try:
chan = self.ssh_transport.open_channel('direct-tcpip',
(self.chain_host, self.chain_port),
self.request.getpeername())
except Exception, e:
verbose('Incoming request to %s:%d failed: %s' % (self.chain_host,
self.chain_port,
repr(e)))
return
if chan is None:
verbose('Incoming request to %s:%d was rejected by the SSH server.' %
(self.chain_host, self.chain_port))
return
verbose('Connected! Tunnel open %r -> %r -> %r' % (self.request.getpeername(),
chan.getpeername(), (self.chain_host, self.chain_port)))
while True:
r, w, x = select.select([self.request, chan], [], [])
if self.request in r:
data = self.request.recv(1024)
if len(data) == 0:
break
chan.send(data)
if chan in r:
data = chan.recv(1024)
if len(data) == 0:
break
self.request.send(data)
chan.close()
self.request.close()
verbose('Tunnel closed from %r' % (self.request.getpeername(),))
def forward_tunnel(local_port, remote_host, remote_port, transport):
# this is a little convoluted, but lets me configure things for the Handler
# object. (SocketServer doesn't give Handlers any way to access the outer
# server normally.)
class SubHander (Handler):
chain_host = remote_host
chain_port = remote_port
ssh_transport = transport
ForwardServer(('', local_port), SubHander).serve_forever()
def verbose(s):
if g_verbose:
print s
I think you need the handler forward code to run in its own thread and use queues to communicate to it.
Or pull out the relevant calls and stuff it is in your own loop. Hmmm, I am not sure how this works exactly.
Do you want your mysql calls to work with it in your program or forward from a different program?
I am the upstream author of a Pyhon module called Python X2Go that heavily uses Paramiko for SSH session management.
Please take a look at the forward.py file in the code:
http://code.x2go.org/gitweb?p=python-x2go.git;a=blob;f=x2go/forward.py
The code of Python X2Go heavily uses Python gevent for server-client communication, port forwarding etc.
Greets,
Mike
I have searched for other posts, as I felt this is a rather common problem, but all other Python exception questions I have found didn't reflect my problem.
I will try to be as specific here as I can, so I will give a direct example. And pleeeeease do not post any workarounds for this specific problem. I am not specifically interested how you can send an email much nicer with xyz.
I want to know how you generally deal with dependent, error prone statements.
My question is, how to handle exceptions nicely, ones that depend on one another, meaning:
Only if the first step was successful, try the next, and so on. One more criterion is: All exceptions have to be caught, this code has to be robust.
For your consideration, an example:
try:
server = smtplib.SMTP(host) #can throw an exception
except smtplib.socket.gaierror:
#actually it can throw a lot more, this is just an example
pass
else: #only if no exception was thrown we may continue
try:
server.login(username, password)
except SMTPAuthenticationError:
pass # do some stuff here
finally:
#we can only run this when the first try...except was successful
#else this throws an exception itself!
server.quit()
else:
try:
# this is already the 3rd nested try...except
# for such a simple procedure! horrible
server.sendmail(addr, [to], msg.as_string())
return True
except Exception:
return False
finally:
server.quit()
return False
This looks extremely unpythonic to me, and the error handling code is triple the real business code, but on the other hand how can I handle several statements that are dependent on one another, meaning statement1 is prerequisite for statement2 and so on?
I am also interested in proper resource cleanup, even Python can manage that for itself.
Thanks, Tom
Instead of using the try/except's else block, you could simply return when it errors:
def send_message(addr, to, msg):
## Connect to host
try:
server = smtplib.SMTP(host) #can throw an exception
except smtplib.socket.gaierror:
return False
## Login
try:
server.login(username, password)
except SMTPAuthenticationError:
server.quit()
return False
## Send message
try:
server.sendmail(addr, [to], msg.as_string())
return True
except Exception: # try to avoid catching Exception unless you have too
return False
finally:
server.quit()
That's perfectly readable and Pythonic..
Another way of doing this is, rather than worry about the specific implementation, decide how you want your code to look, for example..
sender = MyMailer("username", "password") # the except SocketError/AuthError could go here
try:
sender.message("addr..", ["to.."], "message...")
except SocketError:
print "Couldn't connect to server"
except AuthError:
print "Invalid username and/or password!"
else:
print "Message sent!"
Then write the code for the message() method, catching any errors you expect, and raising your own custom one, and handle that where it's relevant. Your class may look something like..
class ConnectionError(Exception): pass
class AuthError(Exception): pass
class SendError(Exception): pass
class MyMailer:
def __init__(self, host, username, password):
self.host = host
self.username = username
self.password = password
def connect(self):
try:
self.server = smtp.SMTP(self.host)
except smtplib.socket.gaierror:
raise ConnectionError("Error connecting to %s" % (self.host))
def auth(self):
try:
self.server.login(self.username, self.password)
except SMTPAuthenticationError:
raise AuthError("Invalid username (%s) and/or password" % (self.username))
def message(self, addr, to, msg):
try:
server.sendmail(addr, [to], msg.as_string())
except smtplib.something.senderror, errormsg:
raise SendError("Couldn't send message: %s" % (errormsg))
except smtp.socket.timeout:
raise ConnectionError("Socket error while sending message")
In general, you want to use as few try blocks as possible, distinguishing failure conditions by the kinds of exceptions they throw. For instance, here's my refactoring of the code you posted:
try:
server = smtplib.SMTP(host)
server.login(username, password) # Only runs if the previous line didn't throw
server.sendmail(addr, [to], msg.as_string())
return True
except smtplib.socket.gaierror:
pass # Couldn't contact the host
except SMTPAuthenticationError:
pass # Login failed
except SomeSendMailError:
pass # Couldn't send mail
finally:
if server:
server.quit()
return False
Here, we use the fact that smtplib.SMTP(), server.login(), and server.sendmail() all throw different exceptions to flatten the tree of try-catch blocks. In the finally block we test server explicitly to avoid invoking quit() on the nil object.
We could also use three sequential try-catch blocks, returning False in the exception conditions, if there are overlapping exception cases that need to be handled separately:
try:
server = smtplib.SMTP(host)
except smtplib.socket.gaierror:
return False # Couldn't contact the host
try:
server.login(username, password)
except SMTPAuthenticationError:
server.quit()
return False # Login failed
try:
server.sendmail(addr, [to], msg.as_string())
except SomeSendMailError:
server.quit()
return False # Couldn't send mail
return True
This isn't quite as nice, as you have to kill the server in more than one place, but now we can handle specific exception types different ways in different places without maintaining any extra state.
If it was me I would probably do something like the following:
try:
server = smtplib.SMTP(host)
try:
server.login(username, password)
server.sendmail(addr, [to], str(msg))
finally:
server.quit()
except:
debug("sendmail", traceback.format_exc().splitlines()[-1])
return True
All errors are caught and debugged, the return value == True on success, and the server connection is properly cleaned up if the initial connection is made.
Just using one try-block is the way to go. This is exactly what they
are designed for: only execute the next statement if the previous
statement did not throw an exception. As for the resource clean-ups,
maybe you can check the resource if it needs to be cleaned up
(e.g. myfile.is_open(), ...) This does add some extra conditions, but
they will only be executed in the exceptional case. To handle the case
that the same Exception can be raised for different reasons, you
should be able to retrieve the reason from the Exception.
I suggest code like this:
server = None
try:
server = smtplib.SMTP(host) #can throw an exception
server.login(username, password)
server.sendmail(addr, [to], msg.as_string())
server.quit()
return True
except smtplib.socket.gaierror:
pass # do some stuff here
except SMTPAuthenticationError:
pass # do some stuff here
except Exception, msg:
# Exception can have several reasons
if msg=='xxx':
pass # do some stuff here
elif:
pass # do some other stuff here
if server:
server.quit()
return False
It is no uncommon, that error handling code exceeds business code. Correct error handling can be complex.
But to increase maintainability it helps to separate the business code from the error handling code.
I would try something like this:
class Mailer():
def send_message(self):
exception = None
for method in [self.connect,
self.authenticate,
self.send,
self.quit]:
try:
if not method(): break
except Exception, ex:
exception = ex
break
if method == quit and exception == None:
return True
if exception:
self.handle_exception(method, exception)
else:
self.handle_failure(method)
def connect(self):
return True
def authenticate(self):
return True
def send(self):
return True
def quit(self):
return True
def handle_exception(self, method, exception):
print "{name} ({msg}) in {method}.".format(
name=exception.__class__.__name__,
msg=exception,
method=method.__name__)
def handle_failure(self, method):
print "Failure in {0}.".format(method.__name__)
All of the methods (including send_message, really) follow the same protocol: they return True if they succeeded, and unless they actually handle an exception, they don't trap it. This protocol also makes it possible to handle the case where a method needs to indicate that it failed without raising an exception. (If the only way your methods fail is by raising an exception, that simplifies the protocol. If you're having to deal with a lot of non-exception failure states outside of the method that failed, you probably have a design problem that you haven't worked out yet.)
The downside of this approach is that all of the methods have to use the same arguments. I've opted for none, with the expectation that the methods I've stubbed out will end up manipulating class members.
The upside of this approach is considerable, though. First, you can add dozens of methods to the process without send_message getting any more complex.
You can also go crazy and do something like this:
def handle_exception(self, method, exception):
custom_handler_name = "handle_{0}_in_{1}".format(\
exception.__class__.__name__,
method.__name__)
try:
custom_handler = self.__dict__[custom_handler_name]
except KeyError:
print "{name} ({msg}) in {method}.".format(
name=exception.__class__.__name__,
msg=exception,
method=method.__name__)
return
custom_handler()
def handle_AuthenticationError_in_authenticate(self):
print "Your login credentials are questionable."
...though at that point, I might say to myself, "self, you're working the Command pattern pretty hard without creating a Command class. Maybe now is the time."
Why not one big try: block? This way, if any exception is caught, you'll go all the way to the except. And as long as all the exceptions for the different steps are different, you can always tell which part it was that fired the exception.
I like David's answer but if you are stuck on the server exceptions you can also check for server if is None or states. I flattened out the method a bit bit it is still a but unpythonic looking but more readable in the logic at the bottom.
server = None
def server_obtained(host):
try:
server = smtplib.SMTP(host) #can throw an exception
return True
except smtplib.socket.gaierror:
#actually it can throw a lot more, this is just an example
return False
def server_login(username, password):
loggedin = False
try:
server.login(username, password)
loggedin = True
except SMTPAuthenticationError:
pass # do some stuff here
finally:
#we can only run this when the first try...except was successful
#else this throws an exception itself!
if(server is not None):
server.quit()
return loggedin
def send_mail(addr, to, msg):
sent = False
try:
server.sendmail(addr, to, msg)
sent = True
except Exception:
return False
finally:
server.quit()
return sent
def do_msg_send():
if(server_obtained(host)):
if(server_login(username, password)):
if(send_mail(addr, [to], msg.as_string())):
return True
return False