We Keep Coding

Why Does my Python Multiprocessing Process Subclass Return as None When Instantiated?

my goal is to create a subclass of multiprocessing.Process to execute tasks based on instructions that are fed into a multiprocessing.Queue as objects and after the task is completed, set a task attribute to indicate the success of the process and pass the task-object to another queue handling responses. Later I plan to instantiate this custom class multiple times to have the different tasks completed faster.
My code is as follows:
import multiprocessing as mp
import time
# Define task-object that should be passed through queues
class Task(object):
def __init__(self, task_type, detail=None, error=None):
self.type = task_type
self.detail = detail
self.error = error
# Define how to handle a task - simulating for now, to see that everything is passed around properly
def handle_task(task):
if task.type == 'UPDATE':
task.detail = 'updating data'
elif task.type == 'ACTUALIZE':
task.detail = 'actualizing knowledge'
else:
task.detail = 'UNKNOWN TASK TYPE'
task.error = True
# if no errors by now, assume success
if task.error == None:
task.error = False
return task
# Define worker process that executes task handling
class Task_handler(mp.Process):
def __init__(self, task_queue, response_queue):
mp.Process.__init__(self)
self.task_queue = task_queue
self.response_queue = response_queue
self.keep_going = True
def run(self):
while self.keep_going:
task = self.task_queue.get()
if task.type == 'TERMINATE':
self.keep_going = False
self.detail = self.name
self.error = False
elif task.type == 'STATUS':
task.detail = self.name
task.error = False
else:
task = handle_task(task)
self.response_queue.put(task)
if __name__ == '__main__':
task_queue = mp.Queue()
response_queue = mp.Queue()
t = Task_handler(task_queue, response_queue)
t.start()
task_queue.put(Task('STATUS'))
task_queue.put(Task('TERMINATE'))
t.join()
while not response_queue.empty():
task = response_queue.get()
print('{} {}, error {}'.format(task.type, task.detail, task.error))
When I run my code in python 3.7.3 on windows 10 it runs fine, but when I run it in python 3.6.9 in linux it gets stuck and I don't understand why this is the case. Also I would appreciate hints on how to do this most efficiently, as I have received no formal training in programming and likely am not aware of all the "dos and don'ts".
Thank you in advance.

Python Keyboard module - Exit blocking read event function

Hello,
I have the below code that corrects user input and I want to exit the blocking function keyboard.read_event when the control is returned from the correction thread.
The whole program works well but I cannot exit immediately after the corrector thread is finished (the program waits for key press).
I tried using a custom Exception for interrupting the keyboard.read_event function, but I didn't manage to make it work.
import keyboard
import threading
import time
class Interrupt_Custom_Exception(Exception):
"""Base class for other exceptions"""
pass
#########################################################
def delete_and_write(times_to_delete, word_to_write):
print("------------Deleting & Rewrite Started---")
time.sleep(2)
print("------------Deleting & Rewrite Ended---")
# simulate deletion and rewrite
#**here I tried the raise Interrupt_Custom_Exception and tried to catch it at the code in the class, but didn't work**
def write_the_suppressed_string(string):
keyboard.write(string)
#########################################################
class keyboard_monitor(threading.Thread):
def __init__(self,thread_name, threadID, word_typed, keyboard_suppress, counter_for_key_pressed):
threading.Thread.__init__(self)
self.name = thread_name
self.threaID = threadID
self.fstring = word_typed
self.counter_for_key_presses = counter_for_key_pressed
self.suppressed = keyboard_suppress
self.temp = ""
def stop(self):
self._is_running = False
def run(self):
if (self.suppressed is False):
while(True):
event = keyboard.read_event(suppress = self.suppressed)
if (event.event_type == keyboard.KEY_DOWN):
if (event.name == "space"):
suppressed_monitor = keyboard_monitor("suppressed_monitor", 2, self.fstring, True, self.counter_for_key_presses)
suppressed_monitor.start()
suppressed_monitor.join()
print("RETURNED TO MAIN MONITOR")
self.counter_for_key_presses = 0
self.fstring = ""
elif (event.name in "abcdefghijklmnopqrstuvwxyz"):
self.fstring = ''.join([self.fstring, event.name])
self.counter_for_key_presses += 1
elif (self.suppressed is True):
def listen_to_keyboard():
event = keyboard.read_event(suppress=self.suppressed)
# **here is where the program waits and don't continue when the correction thread is finished.**
if (event.event_type == keyboard.KEY_DOWN):
print("---KEYS PRESSED WHILE SUPPRESSED = {}---".format(event.name))
if (event.name in "abcdefghijklmnopqrstuvwxyz"):
self.fstring = ''.join([self.fstring, event.name])
self.counter_for_key_presses += 1
try:
#########################################################
# INITIALY CORRECTING THE WORD PASSED FROM THE NORMAL KEY MONITOR
self.temp = self.fstring
self.fstring = ""
thread_delete_and_rewrite = threading.Thread(
target = delete_and_write, args=(self.counter_for_key_presses, self.temp))
thread_delete_and_rewrite.start()
# raise Interrupt_Custom_Exception
#########################################################
print("-BEFORE WHILE LOOP-")
while(thread_delete_and_rewrite.is_alive() is True): # **this works ok but if the control enters the listen_to_keyboard function waits there until a key is pressed. I want somehow to stop this manually and continue the code after this while**
print("--ENTERING THE WHILE LOOP--")
listen_to_keyboard()
print("----EXITING THE WHILE LOOP----\n")
except Interrupt_Custom_Exception:
print("!!!!!!!!!!!!!!!!!CAUGHT IT!!!!!!!!!!!!!!!!!!!")
print("----EXITING THE WHILE LOOP----\n")
print("------BEFORE FINAL WRITE------")
if (self.fstring != ""):
thread_write = threading.Thread(
target = write_the_suppressed_string, args=(self.fstring, ))
thread_write.start()
thread_write.join()
print("SUPPRESSED ENDED")
self._is_running = False
if __name__ == "__main__":
kb_not_suppressed = keyboard_monitor("not_suppressed_monitor", 1, "", False, 0)
kb_not_suppressed.start()
kb_not_suppressed.join()
Any idea on how to exit this blocking function would be very very useful.
Thanks in advance.

It's not possible unless you find some keyboard.read_event that has a timeout, or does a non-blocking check if there's a event. I haven't found any of those in keyboard module ;/
A big workaround would be to keyboard.press in case you want to exit. Not sure if you can detect if it's not from the user. It's up to you if it's acceptable.

Python Threading Flask

I'm creating a platform using flask+python where each loggedin user can add several social media accounts to be used for analysis. Each account added starts a new thread with the account name, which then is saved into a dic (key=account name and value=the thread). HOWEVER, when I do a hard refresh/reload on the homepage, the dic that references all the threads gets reset and returns None. I can make modifications, like get variable values from the threads or call the kill function to end the threads.I've been looking everywhere here on this site and I can't find the solution.
I've simplified my code:
class threads_manager(threading.Thread):
def __init__(self):
super(threads_manager, self).__init__()
self.cancelled = False
self.accounts = {}
def is_loggedin(self, username):
if not self.accounts:
return self.accounts[username].logged_in()
def add_account(self, usname, uspass, PROXY):
# AddAccount runs on a seperate thread that is managed by this thread
acc = AddAccount(usname, uspass, '')
acc.start()
self.accounts.update({usname: acc})
def kill_account(self, username):
if self.accounts[username] != None:
self.accounts[username].cancel()
return True
else:
return False
def run(self):
# make sure that the thread kill it self after 1 min as I cant
#get access to it later
while not self.cancelled:
if self.timeout > 0:
time.sleep(2)
self.timeout -= 1
else:
self.cancelled = True
# create new instance as global variable
manager = threads_manager()
#app.route('/addAccounts/')
def addAccounts():
global manager
# for displaying the accounts that are already stored in the server
#shows the names of the running threads (ajax call)
if request.args.get('command') == 'GETACCOUNTS':
tnames = []
for t in threading.enumerate():
tnames.append(t.name)
# prints the names as a flash/toast
return jsonify({'success' : tname})
if request.args.get('command') == 'ADDACCOUNT':
#ajax call
manager.add_account('username', 'password', 'proxy')
else:
#when reloading the website
return render_template('addAccounts.html')
Any help, ideas?

Unable to stop running Python thread

I have an application listening on a specific TCP port to handle received requests (listen.py). After that, I have another one (trigger.py) that depending on the requested parameters triggers the respective operation.
Now, lets say the operation A was triggered (opA.py). Operation A uses a worker thread to start (worker.py). When the user request listen.py to stop operation A, the started thread is supposed to stop.
UPDATED:
The problem is that the thread is never stopped since the problem lies in trigger.py. The OperationA instance is lost once the code exits. So, I can never call stopOperation since it show me AttributeError: 'NoneType' object has no attribute 'stopOperation'
Any ideas of How to solve this?
listen.py
from trigger import Trigger
'''
code to handle requests here:
1st: param -> 'start'
2nd: param -> 'stop'
'''
t = Trigger()
t.execute(param)
trigger.py
from opA import OperationA
class Trigger():
def execute(param):
opA = OperationA()
if param == 'start':
opA.startOperation()
elif param == 'stop':
opA.stopOperation()
opA.py
from worker import ThreadParam
class OperationThread(ThreadParam):
def run(self):
while (self.running == False):
'''
do something here
'''
class OperationA():
def _init__(self):
listenThread = OperationThread(self)
def startOperation(self):
self.listenThread.start()
def stopOperation(self):
if self.listenThread.isAlive() == True:
print 'Thread is alive'
self.listenThread.killSignal()
else:
print 'Thread is dead'
worker.py
from threading import Thread
class ThreadParam(Thread):
def __init__(self, _parent):
Thread.__init__(self)
self.parent = _parent
self.running = False;
def killSignal(self):
self.running = True;

A minimal useful Trigger might look like this:
class Trigger(object):
def __init__(self):
self.operation = None
def execute(self, command):
if command == 'start':
assert self.operation is None
self.operation = OperationA()
self.operation.start_operation()
elif command == 'stop':
self.operation.stop_operation()
self.operation = None
else:
print 'Unknown command', repr(command)

Python send variables to thread

I'm trying to create my own threading class in Python2.7. I want it to be able to stop that thread with my own class function. Currently I have something like this:
class loop(threading.Thread):
def __init__(self, myvar):
super(loop, self).__init__()
self.terminate = False
self.myvar = myvar
def run(self):
while not self.terminate:
do.smthng.useful(self.myvar)
def change(self, newvar):
self.myvar = newvar #Doesnt work, in run() my old var is still being used
def stoploop(self):
self.terminate = True #Also not working
l = loop(1)
l.start()
time.sleep(1)
l.change(2) #thread still using "1"
time.sleep(1)
l.stoploop() #doesnt stop
I've read some posts here about this, but it wasnt what I needed.
Any help would be appreciated.
Thank you.
EDIT:
As some of the commenters already stated, this part of code looks like to be really working! Problem is in another place of my project. I've found it, but can't solve it. Maybe some of you could help.
So, my project uses Apache Thrift library and the server is in python.
Server.py:
loo = loop(0)
handler = ServHandler(loo)
processor = serv.Processor(handler)
transport = TSocket.TServerSocket('0.0.0.0', port=9090)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()
server = TProcessPoolServer.TProcessPoolServer(processor, transport, tfactory, pfactory)
print 'Starting the server...'
server.serve()
ServHandler.py:
class ServHandler:
def __init__(self, loo):
self.loo = loo
def terminate(self): #Function that can be called remotely
self.loo.stoploop() #Doesn't work
In above case thread isn't terminated and I don't why. There's no error, object exists, but it sets self.terminate value somewhere else. The object id seems to be the same as well as memory address, but it just looks like object is different although loop init function is called only once...
Below is the example, when the loop is terminated successfully.
ServHandler.py:
class ServHandler:
def __init__(self, loo):
self.loo = None
def terminate(self): #Function that can be called remotely
self.loo.stoploop() #Does work!!!!!!
def create(self):
self.loo = loop(0) #Function that can be called remotely
When I create loop object remotely, I can terminate it remotely. But it doesn't fit me. There should be a thread created before thrift server is served and multiple users have to be able to change vars/terminate/etc of that thread. How can I achieve this?
Thank you!

Not a answer per sae, but a useful debug code for the OP
from time import sleep
from threading import Thread
class loop(Thread):
def __init__(self, myvar):
Thread.__init__(self)
self.terminate = False
self.myvar = myvar
def run(self):
while self.terminate is False:
print('Run says myvar is:',self.myvar)
sleep(0.5)
def change(self, newvar):
self.myvar = newvar
def stoploop(self):
self.terminate = True
l = loop(1)
l.start()
sleep(1)
l.change(2)
sleep(1)
l.stoploop()
print('Final product:',l.myvar)
sleep(2)
print('Is the thread alive:',l.isAlive())
Tried your code with some debugging prints, and it's working?
Following code produced:
[torxed#archie ~]$ python test.py
Run says myvar is: 1
Run says myvar is: 1
Run says myvar is: 2 <-- Proves that change() does change `myvar`
Run says myvar is: 2
Final product: 2 <-- Also the global scope knows about the change
Is the thread alive: False <-- And the thread got terminated as intended
However, these are not bulletproof ideas when fetching data or dealing with thread-returns for a number of reasons (even tho i use this method myself from time to time), you should consider using thread.join which should be used in combination with l.toplooop() like so:
l = loop(1)
l.start()
l.change(2)
l.stoploop()
ret = l.join()
Also when updating data you should aquire locks on your data so collisions don't occur, have a look at semaphore objects.

Is it what you need?
import threading
import time
class Worker(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.state = threading.Condition()
self.variable = 10
self.paused = False
def run(self):
while True:
with self.state:
if self.paused:
self.state.wait()
self.do_stuff()
def do_stuff(self):
time.sleep(.1)
print self.variable
def resume(self):
with self.state:
self.paused = False
self.state.notify()
def pause(self):
with self.state:
self.paused = True
loop = Worker()
loop.start()
time.sleep(1)
loop.pause()
loop.variable = 11
print 'CHANGED!'
loop.resume()
time.sleep(1)