I am really new to python's multiple processing, and I have some notions about async call, yield and etc...the most basic stuff. And I came to this snippet, in which multiprocessing.Process wraps around tornado.ioloop.IOLoop.instance
# Set up the tornado web app
app = make_app(predicted_model_queue)
app.listen(8080)
server_process = Process(target=tornado.ioloop.IOLoop.instance().start)
# Start up the server to expose the metrics.
server_process.start()
It intends to start a tornado server as a server_process, but the code does not work. I got the error,
OSError: [Errno 9] Bad file descriptor
I have no experiences with both lib, and have no idea how to fix it. Can anyone please help me?
This is an unusual pattern - if you're writing a new app, I wouldn't recommend copying it.
If you're just trying to run an app that does this (looks like it came from here), the problem is that IOLoops cannot safely cross process boundaries (on some platforms it can sometimes work, but not always). To rewrite this code to correctly create the app and IOLoop in the child process, you could do this:
def run_server():
app = make_app(predicted_model_queue)
app.listen(8080)
tornado.ioloop.IOLoop.current().start()
server_process = Process(target=run_server)
server_process.start()
This way only the predicted_model_queue is shared between the two processes.
Related
I'm doing some unit testing for a flask application. A part of this includes restarting the flask application for each test. To do this, I'm creating my flask application in the setUp() function of my unitest.TestCase, so that I get the application in its fresh state for each run. Also, I'm starting the application in a separate thread so the tests can run without the flask application blocking.
Example below:
import requests
import unittest
from threading import Thread
class MyTest(unittest.TestCase):
def setUp(self):
test_port = 8000
self.test_url = f"http://0.0.0.0:{str(test_port)}"
self.app_thread = Thread(target=app.run, kwargs={"host": "0.0.0.0", "port": test_port, "debug": False})
self.app_thread.start()
def test_a_test_that_contacts_the_server(self):
response = requests.post(
f"{self.test_url}/dosomething",
json={"foo": "bar"},
headers=foo_bar
)
is_successful = json.loads(response.text)["isSuccessful"]
self.assertTrue(is_successful, msg=json.loads(response.text)["message"])
def tearDown(self):
# what should I do here???
pass
This becomes problematic because when the tests that come after the initial test run, they run into an issue with port 8000 being used. This raises OSError: [Errno 98] Address already in use.
(For now, I've built a workaround, where I generate a list of high ranged ports, and another list of ports used per test, so that I never select a port used by a previous test. This work around works, but I'd really like to know the proper way to shut down this flask application, ultimately closing the connection and releasing/freeing that port.)
I'm hopeful that there is a specific way to shutdown this flask application in the tearDown() function.
How should I go about shutting down the flask application in my tearDown() method?
I found the solution to my own question while writing it, and since it's encouraged to answer your own question on Stack Overflow, I'd like to still share this for anyone else with the same issue.
The solution to this problem is to treat the flask application as another process instead of a thread. This is accomplished using Process from the multiprocessing module en lieu of Thread from the threading module.
I came to this conclusion after reading this Stack Overflow answer regarding stopping flask without using CTRL + C. Reading that answer then lead me to read about the differences between multiprocessing and threading in this Stack Overflow answer. Of course, after that, I moved on to the official documentation on the multiprocessing module, found here. More specifically, this link will take you straight to the Process class.
I'm not able to fully articulate why the multiprocessing module serves this purpose better than threading, but I do feel that it makes more sense for this application. After all, the flask application is acting as its own API server that is separate from my test, and my test is testing the calls to it/responses it gets back. For this reason, I think it makes the most sense for my flask application to be its own process.
tl;dr
Use multiprocessing.Process en lieu of threading.Thread, and then call Process.terminate() to kill the process, followed by Process.join() to block until the process is terminated.
example:
import requests
import unittest
from multiprocessing import Process
class MyTest(unittest.TestCase):
def setUp(self):
test_port = 8000
self.test_url = f"http://0.0.0.0:{str(test_port)}"
self.app_process = Process(target=app.run, kwargs={"host": "0.0.0.0", "port": test_port, "debug": False})
self.app_process.start()
def test_a_test_that_contacts_the_server(self):
response = requests.post(
f"{self.test_url}/dosomething",
json={"foo": "bar"},
headers=foo_bar
)
is_successful = json.loads(response.text)["isSuccessful"]
self.assertTrue(is_successful, msg=json.loads(response.text)["message"])
def tearDown(self):
self.app_process.terminate()
self.app_process.join()
Test early, and test often!
I have a Tornado app which is using python firebase_admin SDK.
When I run in single process:
console_server = tornado.httpserver.HTTPServer(ConsoleApplication())
console_server.listen(options.console_port, options.bind_addr)
tornado.ioloop.IOLoop.instance().start()
firebase_admin works fine. But when I change it to run in multiprocess:
console_server = tornado.httpserver.HTTPServer(ConsoleApplication())
console_server.bind(options.console_port, options.bind_addr)
console_server.start(4)
tornado.ioloop.IOLoop.instance().start()
The last line here is getting stuck:
if (not len(firebase_admin._apps)):
cred = ...
self.app = firebase_admin.initialize_app(cred)
self.app = firebase_admin.get_app()
self.db = firestore.client()
...
ref = self.db.document(USER_DOC.format(org, value))
user_ref = ref.get()
Seems like get() is not getting resolved since I don't get any exception.
Does anyone has an idea why it's happening or at least how can I debug it?
The multiprocess fork (i.e. the start(4) call) must come very early in the life of your application. In particular, most things that touch the network must come after the fork (bind() is one of the few exceptions, and must come before the fork in this case).
You (probably) need to reorganize things so that you're creating the firebase app after the fork. This can be annoying if you're using the HTTPServer.start method, so you may want to switch to calling tornado.process.fork_processes() directly instead (this is documented as the "advanced multi-process" pattern).
I know it's an old question, but I want to share my experience regarding this issue to help future visitors.
I recently developed a script with multiprocessing that uses Firebase Admin Python SDK, everything worked fine in my local Windows machine, but when I deployed it for production in Linux server, I noticed the script is getting stuck in get() function.
After hours of searching, I found out that the default start method of a python process is different in Windows and Unix environments: Windows uses spawn as default start method, whereas Unix uses fork. You can learn more about start methods in the documentation.
So to make it work in my Linux server, I just changed the start method to spawn:
if __name__ == '__main__':
multiprocessing.set_start_method('spawn') # <-- Set spawn as start_method
# The rest of your script here
# ...
I'm getting some very strange behaviour in a tornado application.
I'm running a lot of processes each which have their own HTTP servers and running on a different port.
I added a new process to the system which is another TCPServer class that listens on an entirely different port and doesn't have any interaction with the other processes.
I bring the new server up as follows:
def runSimService(port):
sim=SimService()
sim.listen(port)
currentIOLoop = tornado.ioloop.IOLoop.current()
currentIOLoop.start()
class SimService(TCPServer):
def __init__(self,host='localhost',motorport=27017):
TCPServer.__init__(self)
self.log=logging.getLogger("tornado.access")
# Needs to contain a User class log.
self.con=motor.MotorClient(host,motorport)
self.db=self.con.pDB
self.col=self.db.pCol
Basically that's the only code I left when debugging. The crash i get isn't a normal python exception crash which is worrying me.
I'm developing on a mac the moment. Can someone please explain is this crash is something wrong with my code or is there something else happening here?'
UPDATE:
Ok this is really bizare it only seems to happen when I import the following:
from nltk.stem.snowball import SnowballStemmer
OR
from nltk import word_tokenize, pos_tag
or nltk in general...
Could there be some weird interaction between libraries happening? I'm stuck
Code to create processes
if __name__ =='__main__':
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient",max_clients=2000)
processes=[]
processes.append(Process(target=runSimServer,args=(...,)))
processes.append(Process(target=runServer,args=(...)))
processes.append(Process(target=runServer,args=(...)))
processes.append(Process(target=runServer,args=(...)))
processes.append(Process(target=runServer,args=(...)))
processes.append(Process(target=runServer,args=(...)))
# write pids to pid/ directory for use with the shutdown program
with open("pid/"+__file__.split(".")[0]+".pid","w") as f:
for p in processes:
p.start()
f.write(str(p.pid)+"\n")
Thanks
The key part of that crash message is "multi-threaded process forked". If you are going to use both threads and processes, you must fork all your processes before creating any threads. It looks like nltk is creating some threads when you import it. If you are also using multiple processes (it doesn't look like you are from the code you quoted, but that is obviously incomplete), you must not import nltk until after all processes have been started.
I am new to web app programming in general, and flask in particular. I am trying to create a "graphical" interface to some Python code I wrote a while ago. It was suggested to me that instead of writing a graphical interface I should make it a web app with flask. The code I want it to run may take a while and I also want the user to be able to run multiple instances concurrently. I was told I should use celery for the concurrency.
Another major requirement is that feedback about the progress should be given to the user. Not knowing much about web programming from the start I did a bit of research to figure out how this can be done on the web and discovered server-sent events which seemed perfect to me. Trying to figure out how to do this in Flask, I found http://flask.pocoo.org/snippets/116/ which I got to work as described. After learning about celery and getting the example at http://flask.pocoo.org/docs/patterns/celery/ to work, I tried creating a simple example of a task that would have some sort of state that would update over time and feed that back to the user. The problem I ran into was that my celery tasks didn't seem to start.
I then found an example where server-sent events were used in a chat program that used redis, so I thought I would try that, but I had the same result. Below is what I wrote to implement the subscribe functionality:
#flask_app.route("/subscribe", methods=['GET'])
def subscribe():
targetname = request.args['target']
print 'The subscribe function was called with target: ' + targetname
return Response(buildSubscription(targetname), mimetype="text/event-stream")
I then have a celery task:
#celery.task()
def testPrint(printfunc,initvalue):
print "In testPrint"
x=hashlib.md5(initvalue)
i=0
while True:
time.sleep(5)
printfunc(x.hexdigest())
x.update(str(i))
i+=1
where celery is set by the make_celery function. I then have a function:
def ssePrint(mystring,sub):
print "In ssePrint"
red.pubsub(sub,mystring)
and a wrapper for that function:
def appPrint(subscription):
sub = subscription
def myprint(myinput):
ssePrint(myinput, sub)
return myprint
which is then called in
#flask_app.route('/start', methods=['POST'])
def start():
myobject = request.get_json()
testPrint.delay(appPrint(myobject['target']), myobject['target'])
When I then look at the shell that invoked celery, I do not see any tasks running. This happens when I also use gevent.Queue as described in the SSE Flask example.
My suspicion is that there might be an issue with communicating from a Celery task to something running outside of the task. Is there a standard way of extracting the kind of information I want from a Celery task? Should I consider another way of doing concurrency?
Thanks.
I have simple Twisted webserver application serving my math requests. Everything working fine (I hide big code pieces which is not related to my question):
#import section ...
class PlsPage(Resource):
isLeaf = True
def render_POST(self, request):
reactor.callLater(0, self._delayedRender, request)
return NOT_DONE_YET
def _delayedRender(self, request):
#some actions before
crossval_scores = cross_validation.cross_val_score(pls1, X, y=numpy.asarray(Y), scoring=my_custom_scorer, cv=KFold(700, n_folds=700))
#some actions after
request.finish()
reactor.listenTCP(12000, server.Site(PlsPage()))
reactor.run()
When I try to speed up cross_validation calculation by setting n_jobs for example to 3.
crossval_scores = cross_validation.cross_val_score(pls1, X, y=numpy.asarray(Y), scoring=my_custom_scorer, cv=KFold(700, n_folds=700), n_jobs=3)
and after that I got exactly 3 exceptions:
twisted.internet.error.CannotListenError: Couldn't listen on any:12000: [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted.
For some reasons I can't call cross_val_score with n_jobs > 1 inside _delayedRender.
Here is a traceback of exception, for some reasons reactor.listenTCP trying to start 3 times too.
Any ideas how to get it work?
UPD1. I create file PLS.py and moved all the code here, except last 2 lines:
from twisted.web import server
from twisted.internet import reactor, threads
import PLS
reactor.listenTCP(12000, server.Site(PLS.PlsPage()))
reactor.run()
But the problem still persists. I also found that this problem persists only on Windows. My Linux machine run this scripts well.
scikit_learn apparently uses the multiprocessing module in order to achieve concurrency. The multiprocessing transmits data between processes using pickle, which, among other... idiosyncratic problems that it causes, will cause some of the modules imported in your parent process to be imported in your worker processes.
Your PLS_web.py "module", however, is not actually a module, it's a script; since you have put reactor.listenTCP and reactor.run at the bottom of it, it actually does stuff when you import it rather than just loading its code.
This particular error is because since your web server is being run 4 times (once for the controller process, once for each of the three jobs), each of the 3 times beyond the first encounter an error because the first server is already listening on port 12000.
You should remove the reactor.run/reactor.listenTCP lines elsewhere, into a top level script. A good rule of thumb is that these lines should never appear in the same file as a class or def statement; define your code in one place and start it up in another. Once you've moved it to a file that doesn't get imported (and you might want to even put it in a file whose name isn't a legal module identifier, like run-my-server.py) then multiprocessing might be able to import all the code it needs and do its job.
Better yet, don't write those lines at all, write a twisted application plugin and run your program with twistd. If you don't have to put the reactor.run statement in any place, you can't put it in the wrong place :).