We Keep Coding

How to separately start and stop multiprocessing processes in Python?

I use a dedicated Python (3.8) library to control a motor drive via a USB port.
The Python library provided by the motor control drive manufacturers (ODrive) allows a single Python process to control one or more drives.
However, I would like to run 3 processes, each controlling 1 drive.
After researching options (I first considered virtual machines, Docker containers, and multi-threading) I began believing that the easiest way to do so would be to use multiprocessing.
My problem is that I would then need a way to manage (i.e., start, monitor, and stop independently) multiple processes. The practical reason behind it is that motors are connected to different setups. Each setup must be able to be stopped and restarted separate if malfunctioning, for instance, but other running setups should not be affected by this action.
After reading around the internet and Stack Overflow, I now understand how to create a Pool of processing, how to associate processes with processor cores, how to start a pool of processes, and queuing/joining them (the latter not being needed for me).
What I don't know is how to manage them independently.
How can I separately start/stop different processes without affecting the execution of the others?
Are there libraries to manage them (perhaps even with a GUI)?

I'd probably do something like this:
import random
import time
from multiprocessing import Process, Queue
class MotorProcess:
def __init__(self, name, com_related_params):
self.name = name
# Made up some parameters relating to communication
self._params = com_related_params
self._command_queue = Queue()
self._status_queue = Queue()
self._process = None
def start(self):
if self._process and self._process.is_alive():
return
self._process = Process(target=self.run_processing,
args=(self._command_queue, self._status_queue,
self._params))
self._process.start()
#staticmethod
def run_processing(command_queue, status_queue, params):
while True:
# Check for commands
if not command_queue.empty():
msg = command_queue.get(block=True, timeout=0.05)
if msg == "stop motor":
status_queue.put("Stopping motor")
elif msg == "exit":
return
elif msg.startswith("move"):
status_queue.put("moving motor to blah")
# TODO: msg parsing and move motor
else:
status_queue.put("unknown command")
# Update status
# TODO: query motor status
status_queue.put(f"Motor is {random.randint(0, 100)}")
time.sleep(0.5)
def is_alive(self):
if self._process and self._process.is_alive():
return True
return False
def get_status(self):
if not self.is_alive():
return ["not running"]
# Empty the queue
recent = []
while not self._status_queue.empty():
recent.append(self._status_queue.get(False))
return recent
def stop_process(self):
if not self.is_alive():
return
self._command_queue.put("exit")
# Empty the stats queue otherwise it could potentially stop
# the process from closing.
while not self._status_queue.empty():
self._status_queue.get()
self._process.join()
def send_command(self, command):
self._command_queue.put(command)
if __name__ == "__main__":
processes = [MotorProcess("1", None), MotorProcess("2", None)]
while True:
cmd = input()
if cmd == "start 1":
processes[0].start()
elif cmd == "move 1 to 100":
processes[0].send_command("move to 100")
elif cmd == "exit 1":
processes[0].stop_process()
else:
for n, p in enumerate(processes):
print(f"motor {n + 1}", end="\n\t")
print("\n\t".join(p.get_status()))
Not production ready (e.g. no exception handling, no proper command parsing, etc.) but shows the idea.
Shout if there are any problems :D

You can create multiple multriprocessing.Process instances manually like this:
def my_func(a, b):
pass
p = multiprocessing.Process(target=my_func, args=(100, 200)
p.start()
and manage them using multiprocessing primitives Queue, Event, Condition etc. Please refer to the official documentation for details: https://docs.python.org/3/library/multiprocessing.html
In the following example multiple processes are started and stopped independently. Event is used to determine when to stop a process. Queue is used for results passing from the child processes to the main process.
import multiprocessing
import queue
import random
import time
def worker_process(
process_id: int,
results_queue: multiprocessing.Queue,
to_stop: multiprocessing.Event,
):
print(f"Process {process_id} is started")
while not to_stop.is_set():
print(f"Process {process_id} is working")
time.sleep(0.5)
result = random.random()
results_queue.put((process_id, result))
print(f"Process {process_id} exited")
process_pool = []
result_queue = multiprocessing.Queue()
while True:
if random.random() < 0.3:
# staring a new process
process_id = random.randint(0, 10_000)
to_stop = multiprocessing.Event()
p = multiprocessing.Process(
target=worker_process, args=(process_id, result_queue, to_stop)
)
p.start()
process_pool.append((p, to_stop))
if random.random() < 0.2:
# closing a random process
if process_pool:
process, to_stop = process_pool.pop(
random.randint(0, len(process_pool) - 1)
)
to_stop.set()
process.join()
try:
p_id, result = result_queue.get_nowait()
print(f"Completed: process_id={p_id} result={result}")
except queue.Empty:
pass
time.sleep(1)

How to avoid classmethod side effect using Celery?

I am running a class based app using celery, but I am noting that when two processes run simultaneously, certain staticmethods in the class are not acting independently. Here is the app invocation
import os
from PriceOptimization.celery import app
from .Tasks_Sim.sim import Sim, final_report
#app.task(name='Simulations.tasks.scoring')
def simulation(clients, deciles):
s = Sim(**sim_params)
market_by_year = s.control_flow(my_save_path)
report = final_report(market_by_year)
return report
Within my Sim app, I have a class method that creates id's for my instance as follows
class Company:
company_id = 0
#classmethod
def set_company_no(cls):
cls.company_id += 1
return cls.company_id-1
def __init__(self, companies, year):
self._company_id = Company.set_company_no()
self._company_year = year
Usually the first task instantiated will complete successfully, but on the next invocation, I am getting a list index out of range error that suggests to me that my workers are not independent and that my company_id object is not commencing from zero with the next invocation. How can I prevent this side effect and have each app run independently?

For now, I have elected to make my process run sequentially using a redis lock:
from settings import REDIS_INSTANCE
REDIS_LOCK_KEY = 'ABC'
#app.task(name='Simulations.tasks.scoring')
def simulation(clients, deciles):
timeout = (60 * 5)
have_lock = False
my_lock = REDIS_INSTANCE.lock(REDIS_LOCK_KEY, timeout=timeout)
while have_lock == False:
have_lock = my_lock.acquire(blocking=False)
if have_lock:
print('unique process commencing...')
s = Sim(**sim_params)
market_by_year = s.control_flow(my_save_path)
report = final_report(market_by_year)
else:
print('waiting for lock to commence...')
time.sleep(10)
my_lock.release()
return report

Queue-based communication: is it a good idea to send a return-queue?

I was playing with the following approach to wait for the result of an asynchronous operation runnig in another thread:
def client(worker_queue, message):
answer_queue = queue.Queue(maxsize=1)
worker_queue.put((answer_queue, message))
result = answer_queue.get(timeout=10)
def worker():
while True:
answer_queue, message = worker_queue.get()
result = do_someting_with(message)
answer_queue.put(result)
worker_queue.task_done()
(The primitiv worker is only an example. In other cases it could be necessary to pass the "answer_queue" between multiple callbacks)
Is this a good idea, or will I run into problems (for example with memory management)?
Are there better ways to do this?
I know that asyncio has things like futures to handle problems like this, but for the moment I'm looking for something that (also) works multithreading.

Python uses memory addresses to access class variables and their methods. Using a queue simply ensures that each worker has a unique memory address location to place its answers.
Make sure, if the variable remains a local variable, that when you no longer need the variable, you reassign it, so garbage collection can release memory the queue was using, or clear memory using the del keyword to manually do it, such as del answer_queue.
You could use any data type to fulfill the role of transmitting the data between worker and client because all class methods are accessed by memory address; however, the most common ways of transmitting data are:
Using Queues
queue.Queue() is already quite optimized and very versitile, so it is a reliable way of performing communication between clients and workers.
import queue
import datetime
from threading import Thread
def queue_client(worker_queue, message):
answer_queue = queue.Queue(maxsize=1)
worker_queue.put((answer_queue, message))
result = answer_queue.get(timeout=60)
print('Bytes received from worker:', len(result))
def queue_worker(worker_queue):
answer_queue, message = worker_queue.get()
do_something = lambda x: x[::-1]
result = do_something(message)
answer_queue.put(result)
if __name__ == '__main__':
large_message = 'X' * (2<<30) #2GB
worker_queue = queue.Queue()
client = Thread(target=queue_client, args=(worker_queue, large_message,))
worker = Thread(target=queue_worker, args=(worker_queue,))
start_time = datetime.datetime.now()
client.start()
worker.start()
client.join()
worker.join()
dt = datetime.datetime.now() - start_time
print('Time elapsed using Queue:', dt.total_seconds()) #~1.47 secs on my machine
Shared memory types
Since you have direct memory access to the variable addresses when multithreading, you can use shared memory; this is generally the fastest way to access the data, but you would need some kind of container object for the data.
An very basic example (worker spawning, answer submission, etc. could be added to the manager):
import datetime
from threading import Thread
class DataManager:
def __init__(self,):
self.tasks = {
#worker_id: message
}
self.answers = {
#worker_id: answer
}
def shared_client(datamanager, worker_id, message):
datamanager.tasks[worker_id] = message
while not datamanager.answers.get(worker_id, None): #Wait for answer
pass
result = datamanager.answers[worker_id]
print('Bytes received from worker:', len(result))
def shared_worker(data_manager, worker_id):
while not data_manager.tasks.get(worker_id, None): #Wait for task to get assigned
pass
message = data_manager.tasks[worker_id]
do_something = lambda x: x[::-1]
result = do_something(message)
data_manager.answers[worker_id] = result
if __name__ == '__main__':
large_message = 'X' * (2<<30) #2GB
data_manager = DataManager()
worker_id = 0
client = Thread(
target=shared_client,
args=(data_manager, worker_id, large_message,)
)
worker = Thread(target=shared_worker, args=(data_manager, worker_id))
start_time = datetime.datetime.now()
client.start()
worker.start()
client.join()
worker.join()
dt = datetime.datetime.now() - start_time
print('Time elapsed using Shared Memory:', dt.total_seconds()) #~1.44 secs on my machine
More graceful (and potentially even faster) solutions may be available if you use shared memory types/ctypes such as Multiprocessing.Value and Multiprocessing.Array (link).
Pipes
Pipes act as a connection with one end listening while the other is sending data.
Importantly, this behaves more favorably than queues while multiprocessing when sending packets larger than ~32MB. Additionally, it is serializable, unlike queue.Queue().
A sample of using a Pipe:
import datetime
from threading import Thread
from multiprocessing import Pipe
def client(conn, message):
conn.send(message) #Send message to worker
result = conn.recv() #Receive result
print('Bytes received from worker:', len(result))
def worker(conn):
message = conn.recv() #Get message from client
do_something = lambda x: x[::-1]
result = do_something(message)
conn.send(result) #Send result to client
if __name__ == '__main__':
large_message = 'X' * (2<<30) #2GB
client_conn, worker_conn = Pipe(duplex=True) #Bidirectional pipe
client_process = Thread(target=client, args=(client_conn, large_message))
worker_process = Thread(target=worker, args=(worker_conn,))
start_time = datetime.datetime.now()
client_process.start()
worker_process.start()
client_process.join()
worker_process.join()
dt = datetime.datetime.now() - start_time
print('Time elapsed using Pipe:', dt.total_seconds()) #~9.07 secs on my machine
Proxy
Proxies register functions on a client and allow for their execution via a worker calling the underlying exposed function. This is a lot trickier, but necessary for cluster computing, when your workers are on different computers than your clients.

Autocompletion of function parameters in (I)Python for XMLRPC-Client

I am using xmlrpclib to interactively remote control some lab equipment with IPython. I love IPython's autocompletion features and I would also like to have it via xmlrpclib. So far I managed to accomplish method name completion and method help with the following approach.
A little test server simulating a motorized stage (this is only useful, if you want to test my client code):
import time # needed for StageSimulation
from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
class StageSimulation:
""" A very simple simulation of a motorized linear stage """
v = 5 # mm/s -- speed of stage
goalPos = 0 # mm -- goal Position in mm
goalTime = 0 # sec -- time when goal position should be reached
def getPos(self):
""" Return actual Position of stage """
delta_t = self.goalTime - time.time() # remaining moving time
if delta_t <= 0 : # stage is not moving
return self.goalPos
else: # stage is moving
return self.goalPos - self.v*delta_t
def move(self, goalPos, v=5):
""" Move stage to position ``goalPos`` with speed ``v`` """
p0 = self.getPos()
delta_p = goalPos - p0
if v*delta_p < 0: # sign of v wrong
v *= -1
self.goalTime = time.time() + delta_p/v
self.goalPos, self.v = goalPos, v
# Restrict to a particular path (see python Docs)
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)
if __name__=='__main__':
""" Instaniate Server """
host, hport = "localhost", 8787
LogXMLRPCRequests = False
server = SimpleXMLRPCServer((host, hport), allow_none=True,
requestHandler=RequestHandler)
server.register_introspection_functions()
StS = StageSimulation()
server.register_instance(StS)
try:
server.serve_forever()
except KeyboardInterrupt:
print("Terminated server.")
My client instantiates an object in which all known methods are registered:
import xmlrpclib
class XMLRPCClientObject(object):
"""XMLRPC Client which allows Tab completion on server instances
This is achieved by reading all methods names from the server and
using them to generate local wrappers to them.
"""
def __init__(self, url):
""" Connect to server with ``url`` and generate methods """
self.SP = xmlrpclib.ServerProxy(url)
self.generateMethods()
def generateMethods(self):
""" Read names of server methods and use them for local wrappers """
SP = self.SP
for n in SP.system.listMethods():
f = getattr(SP, n)
f.__doc__ = SP.system.methodHelp(n) # add doc string
f.__name__ = n # needed to make help() work
setattr(self, n, f) # register as local method
if __name__ == "__main__":
""" main function connects to Test Server """
S = XMLRPCClientObject("http://localhost:8787")
In addition to the method name completion, I would also like to have parameter name completion as in S.move(goal<TAB>. An approach would be to utilize xmlrpc.ServerProxy.system.methodSignature(), but system_methodSignature() is not supported by SimpleXMLRPCServer. Does anybody have an idea how to retrieve the signatures of the server methods?

I tend to think that python inspect module can help, it provides the basics you can use for your desired features

Python threads/processes and class vars

The question may be really stupid but I'm working on this code since this morning and now even stupid things are hard :\
I've got this code and I call it by making 8 processes and run them.
Then there's another thread that has to print infos about this 8 processes. (code is below).
import MSCHAPV2
import threading
import binascii
import multiprocessing
class CrackerThread(multiprocessing.Process):
password_header = "s."
current_pin = ""
username = ""
server_challenge = ""
peer_challenge = ""
nt_response = ""
starting_pin = 0
limit = 0
testing_pin = 0
event = None
def __init__(self, username, server_challenge, peer_challenge, nt_response, starting_pin, limit, event):
#threading.Thread.__init__(self)
super(CrackerThread, self).__init__()
self.username = username
self.server_challenge = server_challenge
self.peer_challenge = peer_challenge
self.nt_response = nt_response
self.starting_pin = starting_pin
self.limit = limit
self.event = event
self.testing_pin = starting_pin
#self.setDaemon(True)
def run(self):
mschap = MSCHAPV2.MSCHAPV2()
pin_range = self.starting_pin+self.limit
while self.testing_pin <= pin_range and not self.event.isSet():
self.current_pin = "%s%08d" % (self.password_header, self.testing_pin)
if(mschap.CheckPassword(self.server_challenge, self.peer_challenge, self.username, self.current_pin.encode("utf-16-le"), self.nt_response)):
self.event.set()
print "Found valid password!"
print "user =", self.username
print "password =", self.current_pin
self.testing_pin+=1
print "Thread for range (%d, %d) ended with no success." % (self.starting_pin, pin_range)
def getCurrentPin(self):
return self.testing_pin
def printCrackingState(threads):
info_string = '''
++++++++++++++++++++++++++++++++++
+ Starting password = s.%08d +
+--------------------------------+
+ Current pin = s.%08d +
++++++++++++++++++++++++++++++++++
+ Missing pins = %08d +
++++++++++++++++++++++++++++++++++
'''
while 1:
for t in threads:
printed_string = info_string % (t.starting_pin, t.getCurrentPin(), t.getMissingPinsCount())
sys.stdout.write(printed_string)
sys.stdout.write("--------------------------------------------------------------------")
time.sleep(30)
printCrackingState is called by these lines in my "main":
infoThread = threading.Thread(target = utils.printCrackingState, args=([processes]))
#infoThread = cursesTest.CursesPrinter(threads, processes, event)
infoThread.setDaemon(True)
infoThread.start()
Now the quesion is: why t.starting_pin and t.getCurrentPin() print the SAME value?
It's like the t.getCurrentPin() returns the value set in the __init__() method and is not aware that I'm incrementing it!
Suggestions?

Your problem here is that you're trying to update a variable in one process, and read it in another process. You can't do that. The whole point of multiprocessing, as opposed to multithreading, is that variables are not shared by default.
Read the docs, especially Exchanging objects between processes and Sharing state between processes, and it will explain the various ways around this. But really, there's two: either you need some kind of channel/API to let the parent process ask the child process for its current state, or you need some kind of shared memory to store the data in. And you may need a lock to protect either the channel/shared memory.
While shared memory may seem like the "obvious" answer here, you may want to time the following:
val = 0
for i in range(10000):
val += 1
val = Value('i', 0)
lock = Lock()
for i in range(10000):
with lock:
val.value += 1
It's worth noting that your code would also be incorrect with threads—although it would probably work, in CPython. If you don't do any synchronization, there is no guaranteed ordering. If you write a value in one thread and read it "later" in another thread, you can still read the older value. How much later? Well, if thread 0 runs on core 0, and thread 1 on core 1, and they both have the variable in their cache, and nobody tells the CPUs to flush the cache, thread 1 will go on reading the old value forever. In practice, CPython's Global Interpreter Lock eventually synchronizes everything implicitly (so we're talking milliseconds rather than infinity), and all variables have explicit memory locations rather than being, say, optimized into registers, and so on, so you can usually get away with writing unprotected races. But, thanks to Murphy's Law, you should read "usually" as "every time until the first demo to the investors" or "until we attach the live nuclear reactor".