Consider the following code :
Server :
import sys
from multiprocessing.managers import BaseManager, BaseProxy, Process
def baz(aa) :
l = []
for i in range(3) :
l.append(aa)
return l
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
manager = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
manager.register('solver', callable=baz, proxytype=MyProxy)
def serve_forever(server):
try :
server.serve_forever()
except KeyboardInterrupt:
pass
def runpool(n):
server = manager.get_server()
workers = []
for i in range(int(n)):
Process(target=serve_forever, args=(server,)).start()
if __name__ == '__main__':
runpool(sys.argv[1])
Client :
import sys
from multiprocessing.managers import BaseManager, BaseProxy
import multiprocessing, logging
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
def main(args) :
SolverManager.register('solver')
m = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
m.connect()
print m.solver(args[1])._getvalue()
if __name__ == '__main__':
sys.exit(main(sys.argv))
If I run the server using only one process as python server.py 1
then the client works as expected. But if I spawn two processes (python server.py 2) listening for connections, I get a nasty error :
$python client.py ping
Traceback (most recent call last):
File "client.py", line 24, in <module>
sys.exit(main(sys.argv))
File "client.py", line 21, in main
print m.solver(args[1])._getvalue()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 637, in temp
authkey=self._authkey, exposed=exp
File "/usr/lib/python2.6/multiprocessing/managers.py", line 894, in AutoProxy
incref=incref)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 700, in __init__
self._incref()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 750, in _incref
dispatch(conn, None, 'incref', (self._id,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 79, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 181, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 402, in incref
self.id_to_refcount[ident] += 1
KeyError: '7fb51084c518'
---------------------------------------------------------------------------
My idea is pretty simple. I want to create a server that will spawn a number of workers that will share the same socket and handle requests independently. Maybe I'm using the wrong tool here ?
The goal is to build a 3-tier structure where all requests are handled via an http server and then dispatched to nodes sitting in a cluster and from nodes to workers via the multiprocessing managers...
There is one public server, one node per machine and x number of workers on each machine depending on the number of cores... I know I can use a more sophisticated library, but for such a simple task (I'm just prototyping here) I would just use the multiprocessing library... Is this possible or I should explore directly other solutions ? I feel I'm very close to have something working here ... thanks.
You're trying to invent a wheel, many have invented before.
It sounds to me that you're looking for task queue where your server dispatches tasks to, and your workers execute this tasks.
I would recommend you to have a look at Celery.
Related
I have long running file I/O tasks which I'd like to be able to move into a daemon/server process. A CLI tool would be used to queue new jobs to run, query the status of running jobs, and wait for individual jobs. Python's multiprocessing.managers looks like a nice simple way to handle the IPC. I'd like to be able to construct a SyncManager.Event for the client to wait on without blocking the server, but attempting to do so results in triggers a "server not yet started" assertion. Ironically this assertion gets sent from the server to the client, so obviously the server is started, somewhere.
Here's the minimal example:
#!/usr/bin/env python3
import time
import sys
import concurrent.futures
from multiprocessing.managers import SyncManager
def do_work(files):
"""Simulate doing some work on a set of files."""
print(f"Starting work for {files}.")
time.sleep(2)
print(f"Finished work for {files}.")
# Thread pool to do work in.
pool = concurrent.futures.ProcessPoolExecutor(max_workers=1)
class Job:
job_counter = 1
def __init__(self, files):
"""Setup a job and queue work for files on our thread pool."""
self._job_number = self.job_counter
Job.job_counter += 1
print(f"manager._state.value = {manager._state.value}")
self._finished_event = manager.Event()
print(f"Queued job {self.number()}.")
future = pool.submit(do_work, files)
future.add_done_callback(lambda f : self._finished_event.set())
def number(self):
return self._job_number
def event(self):
"""Get an event which can be waited on for the job to complete."""
return self._finished_event
class MyManager(SyncManager):
pass
MyManager.register("Job", Job)
manager = MyManager(address=("localhost", 16000), authkey=b"qca-authkey")
if len(sys.argv) > 1 and sys.argv[1] == "server":
manager.start()
print(f"Manager listening at {manager.address}.")
while True:
time.sleep(1)
else:
manager.connect()
print(f"Connected to {manager.address}.")
job = manager.Job(["a", "b", "c"])
job.event().wait()
print("Done")
If I run the client I see:
$ ./mp-manager.py
Connected to ('localhost', 16000).
Traceback (most recent call last):
File "./mp-manager.py", line 54, in <module>
job = manager.Job(["a", "b", "c"])
File "/usr/lib/python3.8/multiprocessing/managers.py", line 740, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 625, in _create
id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 91, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/managers.py", line 210, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 403, in create
obj = callable(*args, **kwds)
File "./mp-manager.py", line 24, in __init__
self._finished_event = manager.Event()
File "/usr/lib/python3.8/multiprocessing/managers.py", line 740, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 622, in _create
assert self._state.value == State.STARTED, 'server not yet started'
AssertionError: server not yet started
---------------------------------------------------------------------------
The server output is:
$ ./mp-manager.py server
Manager listening at ('127.0.0.1', 16000).
manager._state.value = 0
I am using nameko to build an ETL pipeline with a micro-service architecture, and I do not want to wait for a reply after making a RPC request.
from nameko.rpc import rpc, RpcProxy
class Scheduler(object):
name = "scheduler"
task_runner = RpcProxy('task_runner')
#rpc
def schedule(self, task_type, group_id, time):
return self.task_runner.start.async(task_type, group_id)
This code throws an error:
Traceback (most recent call last):
File "/home/satnam-sandhu/.anaconda3/envs/etl/bin/nameko", line 8, in <module>
sys.exit(main())
File "/home/satnam-sandhu/.anaconda3/envs/etl/lib/python3.8/site-packages/nameko/cli/main.py", line 112, in main
args.main(args)
File "/home/satnam-sandhu/.anaconda3/envs/etl/lib/python3.8/site-packages/nameko/cli/commands.py", line 110, in main
main(args)
File "/home/satnam-sandhu/.anaconda3/envs/etl/lib/python3.8/site-packages/nameko/cli/run.py", line 181, in main
import_service(path)
File "/home/satnam-sandhu/.anaconda3/envs/etl/lib/python3.8/site-packages/nameko/cli/run.py", line 46, in import_service
__import__(module_name)
File "./scheduler/service.py", line 15
return self.task_runner.start.async(task_type, group_id)
^
SyntaxError: invalid syntax
I am new with microservices and Nameko, and also I am using RabbitMQ as the queuing service.
I had the same problem; you need to replace the async method with the call_async one, and retrieve the data with result().
Documentation
GitHub issue
use call_async instead async or for better result use event
from nameko.events import EventDispatcher, event_handler
#event_handler("service_a", "event_emit_name")
def get_result(self, payload):
#do_something...
and in other service
from nameko.events import EventDispatcher, event_handler
#event_handler("service_a", "event_emit_name")
def return_result(self, payload):
#get payload and work over there
I am trying to implement a generic "timeout" function which allows me to send a function to be run, and if it doesn't complete after a certain amount of time, kill it. Here is the current implementation:
from multiprocessing import Process, Queue
import multiprocessing as mp
mp.allow_connection_pickling()
from fn.monad import Full, Empty
import traceback
def timeout(timeout, func, args=()):
'''
Calls function, and if it times out returns an Empty()
:param timeout: int | The amount of time to wait for the function
:param func: () => Any | The function to call (must take no arguments)
:param queue: Queue | The multiprocessing queue to put the result into
:return Option | Full(Result) if we get the result, Empty() if it times out
'''
queue = Queue()
p = Process(target=_helper_func, args=(func, queue, args,))
p.daemon = True
p.start()
p.join(timeout)
if p.is_alive() or queue.empty():
p.terminate()
return Empty()
else:
out = queue.get()
# if 'rebuild_handle' in dir(out):
# out.rebuild_handle()
return Full(out)
def _helper_func(func, queue, args):
try:
func(*args, queue)
except Exception as e:
pass
The function must put the "return value" into the multiprocessing queue. However, when timeout is run, and a socket is put into the queue, I get the following error.
Traceback (most recent call last):
File "socket_test.py", line 27, in <module>
print(q.get())
File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 239, in _rebuild_socket
fd = df.detach()
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
I have tried various previous stack overflow posts, such as the following: Python3 Windows multiprocessing passing socket to process
Please let me know if you know of a solution to this issue, as it is throwing a giant wrench into my code. Thanks!
I am trying to connect to JMS queue by spring-python. I take example of receiver from official spring python site
enter code herefrom springpython.jms.core import JmsTemplate
from springpython.jms.factory import WebSphereMQConnectionFactory
qm_name = "my_qmname"
channel = "my_channel"
host = 'my_host'
listener_port = 'my_port'
queue1 = "MY_QUEUE"
# The connection factory we're going to use.
factory = WebSphereMQConnectionFactory(qm_name, channel, host, listener_port)
# Every JmsTemplate uses a connection factory for actually communicating with a JMS provider.
jms_template = JmsTemplate(factory)
# Get a message off the queue. The call to receive will by default time out
# after 1000ms and raise springpython.jms.NoMessageAvailableException then.
jms_template.receive(queue1)
# We're not using an IoC so we need to shut down the connection factory ourselves.
factory.destroy()
And i get next error:
Traceback (most recent call last):
File "E:/Repos/provisioning/test.py", line 30, in <module>
jms_template.receive('MY_QUEUE')
File "c:\Python27\Lib\site-packages\springpython\jms\core.py", line 108, in receive
return self.factory.receive(dest, timeout)
File "c:\Python27\Lib\site-packages\springpython\jms\factory.py", line 327, in receive
self._connect()
File "c:\Python27\Lib\site-packages\springpython\util.py", line 45, in lockedfunc
return f(*args, **kwargs)
File "c:\Python27\Lib\site-packages\springpython\jms\factory.py", line 211, in _connect
raise exc
springpython.jms.WebSphereMQJMSException: MQSCO wrong size. Given: 540, expected 560
As i found MQSCO is something related to SSL, how can i fix the problem?
Thank you
I'm trying to set speed limits on downloading/uploading files and found that twisted provides twisted.protocols.policies.ThrottlingFactory to handle this job, but I can't get it right. I set readLimit and writeLimit, but file is still downloading on a maximum speed. What am I doing wrong?
from twisted.protocols.basic import FileSender
from twisted.protocols.policies import ThrottlingFactory
from twisted.web import server, resource
from twisted.internet import reactor
import os
class DownloadPage(resource.Resource):
isLeaf = True
def __init__(self, producer):
self.producer = producer
def render(self, request):
size = os.stat(somefile).st_size
request.setHeader('Content-Type', 'application/octet-stream')
request.setHeader('Content-Length', size)
request.setHeader('Content-Disposition', 'attachment; filename="' + somefile + '"')
request.setHeader('Accept-Ranges', 'bytes')
fp = open(somefile, 'rb')
d = self.producer.beginFileTransfer(fp, request)
def err(error):
print "error %s", error
def cbFinished(ignored):
fp.close()
request.finish()
d.addErrback(err).addCallback(cbFinished)
return server.NOT_DONE_YET
producer = FileSender()
root_resource = resource.Resource()
root_resource.putChild('download', DownloadPage(producer))
site = server.Site(root_resource)
tsite = ThrottlingFactory(site, readLimit=10000, writeLimit=10000)
tsite.protocol.producer = producer
reactor.listenTCP(8080, tsite)
reactor.run()
UPDATE
So sometime after I run it:
2012-10-25 09:17:03+0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 402, in startReactor
self.config, oldstdout, oldstderr, self.profiler, reactor)
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 323, in runReactorWithLogging
reactor.run()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 334, in unthrottleWrites
p.unthrottleWrites()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 225, in unthrottleWrites
self.producer.resumeProducing()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 919, in resumeProducing
self.consumer.unregisterProducer()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/web/http.py", line 811, in unregisterProducer
self.transport.unregisterProducer()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 209, in unregisterProducer
del self.producer
exceptions.AttributeError: ThrottlingProtocol instance has no attribute 'producer'
I see that I'm not supposed to assign producer like I do know tsite.protocol.producer = producer, I'm new to Twisted and I don't know how to do that another way.
Every producer needs (eventually) to be registered with whatever you want to consume the data. I don't see registration happening anywhere here. Maybe that is the issue you are having?
Twisted has been used on some big-time projects like Friendster, but all the callbacks do not sit well with the usual way I write in python (and I have some experience with functional programming). I switched to gevent.
If you are working with gevent libraries, many of the details (callbacks/generators that provide the asynchronous functionality) are abstracted out, so that you can typically get away with just monkey patching your code and writing it in the usual object-oriented style you are used to. If you are working on a project with anyone unfamiliar with a callback-heavy language like js/lisp, I bet they will appreciate gevent over twisted.
As egbutter said, you have to register a producer. So instead of this:
tsite.protocol.producer = producer
you have to call a registerProducer method explicitly:
tsite.protocol.registerProducer( ... )
or, if you're using FileSender as a producer, call its beginFileTransfer method, in our case:
file_to_send = open( ... )
producer.beginFileTransfer(file_to_send, tsite.protocol)