I have long running file I/O tasks which I'd like to be able to move into a daemon/server process. A CLI tool would be used to queue new jobs to run, query the status of running jobs, and wait for individual jobs. Python's multiprocessing.managers looks like a nice simple way to handle the IPC. I'd like to be able to construct a SyncManager.Event for the client to wait on without blocking the server, but attempting to do so results in triggers a "server not yet started" assertion. Ironically this assertion gets sent from the server to the client, so obviously the server is started, somewhere.
Here's the minimal example:
#!/usr/bin/env python3
import time
import sys
import concurrent.futures
from multiprocessing.managers import SyncManager
def do_work(files):
"""Simulate doing some work on a set of files."""
print(f"Starting work for {files}.")
time.sleep(2)
print(f"Finished work for {files}.")
# Thread pool to do work in.
pool = concurrent.futures.ProcessPoolExecutor(max_workers=1)
class Job:
job_counter = 1
def __init__(self, files):
"""Setup a job and queue work for files on our thread pool."""
self._job_number = self.job_counter
Job.job_counter += 1
print(f"manager._state.value = {manager._state.value}")
self._finished_event = manager.Event()
print(f"Queued job {self.number()}.")
future = pool.submit(do_work, files)
future.add_done_callback(lambda f : self._finished_event.set())
def number(self):
return self._job_number
def event(self):
"""Get an event which can be waited on for the job to complete."""
return self._finished_event
class MyManager(SyncManager):
pass
MyManager.register("Job", Job)
manager = MyManager(address=("localhost", 16000), authkey=b"qca-authkey")
if len(sys.argv) > 1 and sys.argv[1] == "server":
manager.start()
print(f"Manager listening at {manager.address}.")
while True:
time.sleep(1)
else:
manager.connect()
print(f"Connected to {manager.address}.")
job = manager.Job(["a", "b", "c"])
job.event().wait()
print("Done")
If I run the client I see:
$ ./mp-manager.py
Connected to ('localhost', 16000).
Traceback (most recent call last):
File "./mp-manager.py", line 54, in <module>
job = manager.Job(["a", "b", "c"])
File "/usr/lib/python3.8/multiprocessing/managers.py", line 740, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 625, in _create
id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 91, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/managers.py", line 210, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 403, in create
obj = callable(*args, **kwds)
File "./mp-manager.py", line 24, in __init__
self._finished_event = manager.Event()
File "/usr/lib/python3.8/multiprocessing/managers.py", line 740, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 622, in _create
assert self._state.value == State.STARTED, 'server not yet started'
AssertionError: server not yet started
---------------------------------------------------------------------------
The server output is:
$ ./mp-manager.py server
Manager listening at ('127.0.0.1', 16000).
manager._state.value = 0
Related
I'm following an example from the official python documentation here:
I'm trying to make it so that I spin up a BaseManager at localhost:50000 which registers a queue, then a bunch of workers that read from that queue. I can get it to work if I use the method in the official python docs that has three files (one server, one put client, one get client), but I can't get it to work all in one file where I spawn the clients via multiprocessing.Process(target=...).
Here is my full code. The issue is that when the clients attempt to connect they get a ConnectionRefused (stack trace below)
from typing import Dict, Optional, Any, List
from multiprocessing.managers import BaseManager, SyncManager
import time
import multiprocessing as mp
import argparse
import queue
q = queue.Queue()
def parse_args() -> argparse.Namespace:
a = argparse.ArgumentParser()
a.add_argument("--n-workers", type=int, default=2)
return a.parse_args()
def run_queue_server(args: argparse.Namespace) -> None:
class QueueManager(BaseManager): pass
QueueManager.register("get_queue", lambda: q)
m = QueueManager(address=('', 50000), authkey=b'abracadabra')
m.start()
def _worker_process(worker_uid: str) -> None:
class QueueManager(BaseManager): pass
QueueManager.register("get_queue")
m = QueueManager(address=('', 50000), authkey=b'abracadabra')
# <-- This line fails with ConnectionRefused -->
m.connect()
queue: queue.Queue = m.get_queue()
def spawn_workers(args: argparse.Namespace) -> None:
time.sleep(2)
worker_procs = dict()
for i in range(args.n_workers):
print(f"Spawning worker process {i}..")
p = mp.Process(target=_worker_process, args=[str(i)])
p.start()
worker_procs[str(i)] = p
def main():
args = parse_args()
run_queue_server(args)
spawn_workers(args)
while True:
time.sleep(1)
if __name__ == '__main__':
main()
The error is here
$ python minimal.py
Spawning worker process 0..
Spawning worker process 1..
Process Process-2:
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "minimal.py", line 26, in _worker_process
m.connect()
File "/usr/lib/python3.8/multiprocessing/managers.py", line 548, in connect
conn = Client(self._address, authkey=self._authkey)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 502, in Client
c = SocketClient(address)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 629, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "minimal.py", line 26, in _worker_process
m.connect()
File "/usr/lib/python3.8/multiprocessing/managers.py", line 548, in connect
conn = Client(self._address, authkey=self._authkey)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 502, in Client
c = SocketClient(address)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 629, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
However, If I spawn another process that targets the manager creatoin step and run m.get_server().serve_forever() then I do not get the connection-refused error, see this code below which works
from typing import Dict, Optional, Any, List
from multiprocessing.managers import BaseManager, SyncManager
import time
import multiprocessing as mp
import argparse
import queue
q = queue.Queue()
def parse_args() -> argparse.Namespace:
a = argparse.ArgumentParser()
a.add_argument("--n-workers", type=int, default=2)
return a.parse_args()
def run_queue_server(args: argparse.Namespace) -> None:
class QueueManager(BaseManager): pass
QueueManager.register("get_queue", lambda: q)
m = QueueManager(address=('', 50000), authkey=b'abracadabra')
#m.start()
# This works!!
m.get_server().serve_forever()
def _worker_process(worker_uid: str) -> None:
class QueueManager(BaseManager): pass
QueueManager.register("get_queue")
m = QueueManager(address=('', 50000), authkey=b'abracadabra')
m.connect()
queue: queue.Queue = m.get_queue()
print(f"Gotten queue: {queue}")
def spawn_workers(args: argparse.Namespace) -> None:
time.sleep(2)
worker_procs = dict()
for i in range(args.n_workers):
print(f"Spawning worker process {i}..")
p = mp.Process(target=_worker_process, args=[str(i)])
p.start()
worker_procs[str(i)] = p
def main():
args = parse_args()
#run_queue_server(args)
# I don't want to run this in another process?
mp.Process(target=run_queue_server, args=(args,)).start()
spawn_workers(args)
while True:
time.sleep(1)
if __name__ == '__main__':
main()
The thing is, I don't want to have to start another process to be the manager.. why can't it just be this process?
Edit - I'm an idiot who was programming too late into the night. The issue is that my run_queue_server when calling m.start() and returning was... then losing the reference to the QueueManager which I'm sure caused it to be garbage collected.
All I did was change
def run_queue_server(args: argparse.Namespace) -> None:
class QueueManager(BaseManager): pass
QueueManager.register("get_queue", lambda: q)
m = QueueManager(address=('', 50000), authkey=b'abracadabra')
m.start()
return m
and change the caller to accept the return value and everything works..
I'm trying to control a robot with ROS and flask. The problem is that when i kill ROS with ctrl-c (SIGINT) flask is slowing this process down because it is not closing right away. I have implemented a signal_handler to handle the ctrl-c and close flask.
The problem is that when i run this and press
ctrl-c i closes everything right away but i get the following error:
RuntimeError: Working outside of request context.
How can i fix this error?
#!/usr/bin/env python
import rospy
from raspimouse_ros.msg import MotorFreqs
from time import sleep
from flask import Flask, request
from os.path import join, dirname
from signal import signal, SIGINT
cwd = dirname(__file__)
open(join(cwd, "file.html"))
app = Flask(__name__)
deltaX = 0
deltaY = 0
pub = rospy.Publisher('/motor_raw', MotorFreqs, queue_size=10)
rospy.init_node('control')
msg = MotorFreqs()
def signal_handler(signal_received, frame):
msg.left = 0
msg.right = 0
pub.publish(msg)
print("Quitting .......")
func = request.environ.get('werkzeug.server.shutdown')
func()
signal(SIGINT,signal_handler)
#app.route("/")
def main():
with open(join(cwd, "file.html"), 'r') as f:
program = f.read()
return program
#app.route("/SetSpeed")
def SetSpeed():
global deltaX
global deltaY
deltaX = int(float(request.args.get('x')) * 4)
deltaY = int(float(request.args.get('y')) * 10)
publisher()
return ""
def publisher():
msg.left = int(-deltaY+deltaX)
msg.right = int(-deltaY-deltaX)
rospy.loginfo(msg)
pub.publish(msg)
app.run(host="0.0.0.0")
[control-1] killing on exit
Quitting .......
Traceback (most recent call last):
File "/home/pi/workspace/src/manual_control/scripts/control.py", line 54, in <module>
app.run(host="0.0.0.0")
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 841, in run
run_simple(host, port, self, **options)
File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 708, in run_simple
inner()
File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 673, in inner
srv.serve_forever()
File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 511, in serve_forever
HTTPServer.serve_forever(self)
File "/usr/lib/python2.7/SocketServer.py", line 231, in serve_forever
poll_interval)
File "/usr/lib/python2.7/SocketServer.py", line 150, in _eintr_retry
return func(*args)
File "/home/pi/workspace/src/manual_control/scripts/control.py", line 28, in signal_handler
func = request.environ.get('werkzeug.server.shutdown')
File "/usr/lib/python2.7/dist-packages/werkzeug/local.py", line 343, in __getattr__
return getattr(self._get_current_object(), name)
File "/usr/lib/python2.7/dist-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/usr/lib/python2.7/dist-packages/flask/globals.py", line 37, in _lookup_req_object
raise RuntimeError(_request_ctx_err_msg)
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.
shutting down processing monitor...
... shutting down processing monitor complete
done
The request context is active when a request arrives and is destroyed after it finishes, so it is not availale in the signal_handler function, you can, however, access app_context.
What I want to do is set up an XMPP receiver using the Slixmpp library which listens for messages throughout the script execution. I think the threading library might provide a solution so tried that.
This script is for an online PvP game and I need to receive information from the opponent continuously.
This is the code I have tried:
import time
import threading as td
import slixmpp as slix
import logging
import asyncio
my_jid = "test#xmpp.jp"
my_pass = "test"
class receiver(slix.ClientXMPP):
def __init__(self, jid, password):
slix.ClientXMPP.__init__(self, jid, password)
self.add_event_handler("session_start", self.start)
self.add_event_handler("message", self.message)
def start(self, event):
self.send_presence()
self.get_roster()
def message(self, msg):
if msg['type'] in ('chat', 'normal'):
msg.reply("Thanks for sending\n%(body)s" % msg).send()
print(msg['body'])
def function():
recv = receiver(my_jid, my_pass)
recv.connect()
recv.process()
newthread = threading.Thread(target=function)
logging.basicConfig(level="DEBUG", format='%(levelname)-8s %(message)s')
newthread.start()
input("Press Enter to continue")
This returns the following error:
Exception in thread Thread-1:
Press Enter to continueTraceback (most recent call last):
File "C:\Program Files (x86)\Python37-32\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Program Files (x86)\Python37-32\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/dps/Documents/GitHub/Python-Problems/UltimateTicTacToe-scripts/xmppp.py", line 46, in good
recv = receiver(my_jid, my_pass)
File "C:/Users/dps/Documents/GitHub/Python-Problems/UltimateTicTacToe-scripts/xmppp.py", line 32, in __init__
slix.ClientXMPP.__init__(self, jid, password)
File "C:\Users\dps\venv\lib\site-packages\slixmpp\clientxmpp.py", line 70, in __init__
BaseXMPP.__init__(self, jid, 'jabber:client', **kwargs)
File "C:\Users\dps\venv\lib\site-packages\slixmpp\basexmpp.py", line 48, in __init__
XMLStream.__init__(self, **kwargs)
File "C:\Users\dps\venv\lib\site-packages\slixmpp\xmlstream\xmlstream.py", line 219, in __init__
self.disconnected = asyncio.Future()
File "C:\Program Files (x86)\Python37-32\lib\asyncio\events.py", line 644, in get_event_loop
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-1'.
In my case executing:
newthread.daemon = True
Before calling:
newthread.start()
Works as expected creating a non-blocking thread.
I want to run a method I know this method doesn't work and I want to get the error returned by the method.
This is my code :
def is_connect(s):
print("ok connection")
print(s)
ioloop.stop()
try:
current_job_ready = 0
print("ok1")
beanstalk = beanstalkt.Client(host='host', port=port)
print("ok1")
beanstalk.connect(callback=is_connect)
ioloop = tornado.ioloop.IOLoop.instance()
ioloop.start()
print("ok2")
except IOError as e:
print(e)
And this is the error I have when I run my program with wring port :
WARNING:tornado.general:Connect error on fd 7: ECONNREFUSED
ERROR:tornado.application:Exception in callback <functools.partial object at 0x7f5a0eac6f18>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 604, in _run_callback
ret = callback()
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 619, in <lambda>
self.add_future(ret, lambda f: f.result())
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 270, in wrapper
result = func(*args, **kwargs)
TypeError: connect() takes exactly 1 argument (2 given)
I want to have e when I enter a false port or host.
How can I do this?
I tired to add raise IOError("connection error") after beanstalk = beanstalkt.Client(host='host', port=port) But this force the error, and I just want to have error when it exist.
Here's where reading the code helps. In beanstalkt 0.6's connect, it creates an IOStream to connect to the server:
https://github.com/nephics/beanstalkt/blob/v0.6.0/beanstalkt/beanstalkt.py#L108
It registers your callback to be executed on success, but if the connection fails it'll just call Client._reconnect once per second forever. I think you should open a feature request in their GitHub project asking for an error-notification system for connect. With the current beanstalkt implementation, you just have to decide how long you're willing to wait for success:
import sys
from datetime import timedelta
from tornado.ioloop import IOLoop
def is_connect(s):
print("ok connection")
print(s)
loop.remove_timeout(timeout)
# Do something with Beanstalkd....
def connection_failed():
print(sys.stderr, "Connection failed!")
# Could call IOLoop.stop() or just quit.
sys.exit(1)
loop = IOLoop.current()
timeout = loop.add_timeout(timedelta(seconds=1), connection_failed)
beanstalk.connect(callback=is_connect)
loop.start()
Consider the following code :
Server :
import sys
from multiprocessing.managers import BaseManager, BaseProxy, Process
def baz(aa) :
l = []
for i in range(3) :
l.append(aa)
return l
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
manager = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
manager.register('solver', callable=baz, proxytype=MyProxy)
def serve_forever(server):
try :
server.serve_forever()
except KeyboardInterrupt:
pass
def runpool(n):
server = manager.get_server()
workers = []
for i in range(int(n)):
Process(target=serve_forever, args=(server,)).start()
if __name__ == '__main__':
runpool(sys.argv[1])
Client :
import sys
from multiprocessing.managers import BaseManager, BaseProxy
import multiprocessing, logging
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
def main(args) :
SolverManager.register('solver')
m = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
m.connect()
print m.solver(args[1])._getvalue()
if __name__ == '__main__':
sys.exit(main(sys.argv))
If I run the server using only one process as python server.py 1
then the client works as expected. But if I spawn two processes (python server.py 2) listening for connections, I get a nasty error :
$python client.py ping
Traceback (most recent call last):
File "client.py", line 24, in <module>
sys.exit(main(sys.argv))
File "client.py", line 21, in main
print m.solver(args[1])._getvalue()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 637, in temp
authkey=self._authkey, exposed=exp
File "/usr/lib/python2.6/multiprocessing/managers.py", line 894, in AutoProxy
incref=incref)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 700, in __init__
self._incref()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 750, in _incref
dispatch(conn, None, 'incref', (self._id,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 79, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 181, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 402, in incref
self.id_to_refcount[ident] += 1
KeyError: '7fb51084c518'
---------------------------------------------------------------------------
My idea is pretty simple. I want to create a server that will spawn a number of workers that will share the same socket and handle requests independently. Maybe I'm using the wrong tool here ?
The goal is to build a 3-tier structure where all requests are handled via an http server and then dispatched to nodes sitting in a cluster and from nodes to workers via the multiprocessing managers...
There is one public server, one node per machine and x number of workers on each machine depending on the number of cores... I know I can use a more sophisticated library, but for such a simple task (I'm just prototyping here) I would just use the multiprocessing library... Is this possible or I should explore directly other solutions ? I feel I'm very close to have something working here ... thanks.
You're trying to invent a wheel, many have invented before.
It sounds to me that you're looking for task queue where your server dispatches tasks to, and your workers execute this tasks.
I would recommend you to have a look at Celery.