to demonstrate problem I've prepared simple code:
from multiprocessing import Pool
class MyError(Exception):
def __str__(self):
return repr("Error msg: " + self.args[0])
def pool_function(msg):
print msg
raise MyError(msg)
return 0
def some_function():
my_pool = Pool(2)
msg = ['first', 'second']
my_pool.map(pool_function, msg)
if __name__ == '__main__':
try:
some_function()
except MyError, msg:
print msg
In this particular example as output I get:
first
second
'Error msg: first'
but I need rather:
first
second
'Error msg: first'
'Error msg: second'
Problem is that on the level of the main function I try to get access to all error messages which are unique, rather than handling it on level of pool_function but I get only first error msg. Unfortunately the real code with which I work is much more complex so is rather hard to do something with the structure of this code which is demonstrated in example. I need some clean and straight forward solution to get all error messages and process it on the level of main function.
Thanks in advice for any solutions.
You have to put try~ except in your pool_function not __main__. If no, __main__ will stop after the first except raised and left no chance for the second one to run. This is following what you are trying:
def pool_function(msg):
print msg
try:
raise MyError(msg)
except:
return MyError(msg)
def some_function():
my_pool = Pool(2)
msg = ['first', 'second']
return my_pool.map(pool_function, msg)
if __name__ == '__main__':
try:
msg= some_function()
except MyError, msg:
print msg
It works, but seem not a good way, so:
def pool_function(msg):
print msg
try:
# do something
raise MyError(msg)
except:
return 0,MyError(msg)
else:
return 1,# some result
def some_function():
my_pool = Pool(2)
msg = ['first', 'second']
return return my_pool.map(pool_function, msg)
if __name__ == '__main__':
msg = some_function()
for result in msg:
if result[0]:
# do something when it run successfully
elif not result[0]:
print result[1]
# do something when it got errors
Related
I'm trying to catch the exceptions I raise in a function called callback.
Here's a simplification of the code.
try:
logger.info("This message gets logged!!!")
with Pool() as executor:
results = executor.map(callback, made_up_list)
except Exception as e:
pass
This doesn't seem to work, since any exception raised by callback shows on the terminal.
One interesting thing is if we use the debug mode in an IDE (my case vscode), and put a breakpoint on the except line, we never get there, since we remain stuck due to the raised exception. However, if I do CTRL+c, I stop the process, and then I pass to the except line.
It may be worth knowing the callback itself already uses multithreading when doing some computations which will result in raising an exception.
If I change the line order a bit, to:
with Pool() as executor::
logger.info("This message gets logged!!!")
try:
results = executor.map(callback, made_up_list)
except Exception as e:
pass
I get a similar undesired result of not catching the exception.
I would suggest you to wrap your callback inside decorator to catch exception and then pass your decorated function inside executor like this
import logging
from multiprocessing import Pool
from functools import update_wrapper
class my_decorator(object):
def __init__(self, target):
self.target = target
try:
update_wrapper(self, target)
except:
pass
def __call__(self, args):
self.target(args)
def func(a):
print(int(a))
callback = my_decorator(func)
if __name__ == "__main__":
with Pool(4) as executor:
logging.info("This message gets logged!!!")
results = executor.map(callback,[1,2,3, '4', 'a'])
I am not sure why you are not catching an exception (assuming one occurs in your worker function, callback. Perhaps this is due to some peculiarity of your IDE. The following code when run from a command line (not an IDE) works as expected:
def callback(x):
if x == 2:
raise ValueError(f'Invalid value {x}')
return x ** 2
if __name__ == '__main__':
from multiprocessing import Pool
made_up_list = [1, 2, 3]
with Pool() as executor:
try:
results = executor.map(callback, made_up_list)
print(results)
except Exception as e:
print('Got exception:', e)
Prints:
Got exception: Invalid value 2
You might see if the following works, which has the additional benefit of getting a results list for those tasks that have completed normally, i.e. if you want to catch individual exceptions as well as the results of successful completions that occur in your worker function callback in your main process, then you could use the Pool.imap method instead of Pool.map:
def callback(x):
if x == 2:
raise ValueError(f'Invalid value {x}')
return x ** 2
if __name__ == '__main__':
from multiprocessing import Pool
made_up_list = [1, 2, 3]
with Pool() as executor:
results = []
it = executor.imap(callback, made_up_list)
while True:
try:
result = next(it)
except StopIteration:
break
except Exception as e:
result = e
print('Got exception:', e)
results.append(result) # Include Exception instances
print(results)
Prints:
Got exception: Invalid value 2
[1, ValueError('Invalid value 2'), 9]
If this makes no difference, then you need to post a minimal, reproducible example.
I have a problem when doing an exception handling in a Python class.
My class structure is like:
class base():
def func():
try:
# some codes to deal with requests headers in here
requests.get('...', timeout=0.1)
return something
except:
# So when timeout in request occurs, func() will return 'Error'
return 'Error'
def A():
func()
def B():
func()
# there are about 10 functions that have called func().
def index():
reply = A()
reply = B()
# and A() B() functions are called here.
return reply
My question is, is there a way to return an 'Error' to index function directly, instead of doing exception handling every time when calling it? That is, change func() only, and it has to return 2 times(func() -> A() -> index()), so reply in index function will be 'Error'.
def test(a = 1):
try:
if a:
raise Exception
else:
return a+10
except:
return "error"
You can try something like this:
def func():
try:
# the area may raise excetion
pass
except Exception1:
# anything you like
return 'error'
except Exception2:
# anything you like
return 'error'
Using requests.Timeout
def func():
try:
# some codes to deal with requests headers in here
rq = requests.get('...', timeout=0.1)
return 'something'
except requests.Timeout as err:
# So when timeout in request occurs, func() will return 'Error'
return ('Error {}'.format(err))
I am trying to call a function inside another class function in Python. However, I am getting the error that global name is not defined. I don't know what to do as I can't figure out what's the issue.
The error is:
NameError: global name 'sendCmdWaitForRaffleResponse' is not defined
Here is the code I have written:
Class T:
def connect(self,ipAddr,portNum):
try:
print "Check1"
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ipAddr,portNum))
s.settimeout(1000*30)
response=s.recv(8192)
print response
cmd="some command"
s.send(cmd)
print 'check'
response=s.recv(1024)
print response
#time.sleep(5)
except Exception as e:
print e
def Send(self,command):
result ="ok"
try:
print "start"
if command.startswith("#"):
#print result
return result
elif not command:
return result
print 'yes'
result=sendCmdWaitResponse(command)
except Exception as e:
print (e)
return result
def sendCmdWaitResponse(self,cmd):
print "Running"
s.send(cmd)
while true:
response=s.recv(1024)
response=+ '\n'
print response
print "Leaving"
return sb.getvalue()
if __name__ == "__main__":
test=T()
a='some command'
test.Connect(ipaddr,port)
test.Send(a)
You need self.
result=self.sendCmdWaitResponse(command)
Change
result=sendCmdWaitResponse(command)
to
result=self.sendCmdWaitResponse(command)
Your code seems to have many typo: This is the fix I have done. I don't know what is your expected output. Please add the valid code:
import socket
class T():
def __init__(self, ipAddr, portNum):
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self,ipAddr,portNum):
try:
print "Check1"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ipAddr,portNum))
s.settimeout(1000*30)
response=s.recv(8192)
print response
cmd="some command"
s.send(cmd)
print 'check'
response=s.recv(1024)
print response
#time.sleep(5)
except Exception as e:
print e
def Send(self, command):
result ="ok"
try:
print "start"
if command.startswith("#"):
#print result
return result
elif not command:
return result
print 'yes'
result= self.sendCmdWaitResponse(command)
except Exception as e:
print (e)
return result
def sendCmdWaitResponse(self,cmd):
print "Running"
self.s.send(cmd)
while True:
response= self.s.recv(1024)
response=+ '\n'
print response
print "Leaving"
return sb.getvalue()
if __name__ == "__main__":
a='some command'
ipAddr = ""
port = 1000
test = T(ipAddr, port)
test.connect(ipAddr,port)
test.Send(a)
I want to collect time and failure messages in pool.map function calls and print them when all jobs are done. Is that possible in python?
Perhaps there is some event.. or smth else...
failures = []
def test():
sleep(1)
global failures
failures.append('test message')
def main():
data = [...]
pool = ThreadPool(45)
results = pool.map(test, data)
pool.close()
pool.join()
# after all calls are finished i want to perform this
print failures
Example (note that you can return anything, you could return [not raise] exceptions instead of normal results):
import random
import pprint
import multiprocessing
def test(value):
if random.random() > 0.5:
return 'Noes, we got an error for %d!' % value
else:
return 'All is ok for %d :)' % value
def main():
data = range(10)
pool = multiprocessing.Pool()
results = pool.map(test, data)
pool.close()
pool.join()
pprint.pprint(results)
if __name__ == '__main__':
main()
Example output:
['All is ok for 0 :)',
'Noes, we got an error for 1!',
'Noes, we got an error for 2!',
'Noes, we got an error for 3!',
'All is ok for 4 :)',
'Noes, we got an error for 5!',
'All is ok for 6 :)',
'Noes, we got an error for 7!',
'All is ok for 8 :)',
'Noes, we got an error for 9!']
This seems like a remedial topic, but I'm a bit unsure of how to deal with this. Every solution I think of seems messy.
I'm working with some code that builds up a message while performing several actions, then ultimately returns that msg with an http response. Currently it looks somewhat like this:
try:
pdict = parser.parseProtocol(id)
msg = "parsing worked"
except:
msg = "parsing failed"
try:
file = parser.getFile(pdict['filePath'])
msg += "file retrieved"
except:
msg += "file not found"
Say I want to encapsulate the code into functions. How could do I have a message that gets updated throughout? Strings are immutable, so I can't just pass them to a function and modify them. A super ugly solution would be:
(pdict, msg) = parseSomething()
if pdict:
(file, msg) = retrieveFile(pdict, msg)
def parseSomething():
try:
pdict = parser.parseProtocol(id)
return (pdict, "parsing worked")
except:
return (None, "parsing failed")
def retrieveFile(pdict, msg)
try:
file = parser.getFile(pdict['filePath'])
return (file, msg + "file retrieved")
except:
return (None, msg + "file not found")
Super ugly.
I could create a message class, or use a list of length 1, and that would be prettier, but still not very pythonic, right? I think I just want these functions to take a message string and modify it, without having to return it, but strings are immutable so that's not the default behavior.
There's gotta be a smooth way to do this that I'm just blanking on. Help?
Consider putting your messages in a list and appending to it as you go?
messages = []
try:
pdict = parser.parseProtocol(id)
messages.append("parsing worked")
except:
messages.append("parsing failed")
try:
file = parser.getFile(pdict['filePath'])
messages.append("file retrieved")
except:
messages.append("file not found")
print '\n'.join(messages)
If your codepath is particularly convuluted, consider embedding them in a class:
class Tests(object):
def __init__(self):
messages = []
self.pdict = None
def parsetest(self):
try:
self.pdict = parser.parseProtocol(id)
except:
self.messages.append("parsing failed")
else:
self.messages.append("parsing worked")
def retrievetest(self):
if self.pdict is None:
raise Exception("called retrievetest() before successfully parsing")
try:
file = parser.getFile(self.pdict['filePath'])
except:
self.messages.append("file not found")
else:
self.messages.append("file retrieved")
And then later:
tests = Tests()
tests.parsetest()
if condition:
tests.retrievetest()
print '\n'.join(tests.messages)
put your message in an array, pass it around, and just append each part to it.
just before sending it, do a ''.join(msg).
Make your message a member of a class, and pass around an instance of the class.
Better yet, make all these function methods on a class, and keep the message as an attribute of the object.