I'm new to Python so please bear with my question.
Let's say my application has a module named message_printer which simply defines a print_message function to print the message. Now in my main file, I create two threads which calls print function in message_printer.
My question is: How can I set a different message per thread and access it in message_printer?
message_printer:
import threading
threadLocal = threading.local()
def print_message():
name = getattr(threadLocal, 'name', None);
print name
return
main:
import threading
import message_printer
threadLocal = threading.local()
class Executor (threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
threadLocal.name = name
def run(self):
message_printer.print_message();
A = Executor("A");
A.start();
B = Executor("B");
B.start();
This just outputs None and None while I expect A and B. I also tried accessing threadLocal object inside the print_message function directly but doesn't work.
Note that this is just an example. In my application, the exact use case is for logging. Main launches a bunch of thread which call other modules. I want to have a different logger per thread (each thread should log to its own file) and each logger needs to be configured in Main. So I'm trying to instantiate logger per thread and set in thread local storage which can then be accessed in other modules.
What am I doing wrong? I'm following this question as an example Thread local storage in Python
The problem with your code, is that you are not assigning your name to the correct local() context. Your __init__() method is run in the main thread, before you start your A and B threads by calling .start().
Your first thread creation A = Executor("A"); will create a new thread A but update the local context of the main thread. Then, when you start A by calling A.start(); you will enter A:s context, with a separate local context. Here name is not defined and you end up with None as output. The same then happens for B.
In other words, to access the thread local variables you should be running the current thread, which you are when running .start() (which will call your .run() method), but not when creating the objects (running __init__()).
To get your current code working, you could store the data in each object (using self references) and then, when each thread is running, copy the content to the thread local context:
import threading
threadLocal = threading.local()
def print_message():
name = getattr(threadLocal, 'name', None);
print name
return
class Executor (threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
# Store name in object using self reference
self.name = name
def run(self):
# Here we copy from object to local context,
# since the thread is running
threadLocal.name = self.name
print_message();
A = Executor("A")
A.start()
B = Executor("B")
B.start()
Note, though, in this situation, it is somewhat of an overkill to use the thread local context, since we already store the separate data values in the different objects. To use it directly from the objects, would require a small rewrite of print_message() though.
I think this may be helpful for your use case. Another way on how thread storage can been done across files/modules.
Related
I am trying to create a global state variable, which is written in a callback method (event handler).
However, the callback creates a copy (deep) on another memory location, which is not being seen (of course) by the other methods.
Here is the situation
class Server:
def __init__(self):
self.callbacks=[]
#create a web server instance to listen to requests
self.app=Flask("test")
def add_callback(self, func):
self.calbacks.append(func)
self.app.add_url_rule("/test", "test", self.handle_http_request)
def handle_http_request(self):
content = request.get_json(silent=True)
for ca in self.callbacks:
ca(content)
def start_server(self):
#some stuff starting flask here...
class SomeModule:
def __init__(self):
self.ws=Server()
self.ws.add_callback(self.callback)
self.callback_called=False
def callback(self, content):
print "callback executing---"
print "var addr before callback assign: "+str(hex(id(self.callback_called)))
self.callback_called=True
print "var addr after callback assign: "+str(hex(id(self.callback_called)))
def start(self):
self.ws.start()
#send a request to the server using the request library, which invokes all the trigger
#check the state variable:
print "var addr before check: "+str(hex(id(self.callback_called)))
if (not self.callback_called):
raise Exception("error...")
if __name__ == '__main__':
sm=SomeModule()
sm.start()
The output is then:
callback executing---
var addr before callback assign: 0x927910
var addr before callback assign: 0x927930
var addr before check: 0x927910
Can anyone suggest me a way how to avoid this?
In c++ its clear how to access pointer and mutex. Here however, I did not manage to find any ways to do a secure write on the variable...
Thanks a lot in advance!
Since you're using multithreading, and not multiprocessing in your tags, I'll still go ahead and post this answer.. Might be helpful for some.
Some objects are immutable, copied as you said. Other variables, such as dictionaries tend not to be and can be manipulated from functions or threats (not sure if threads only apply to certain cases).
If you pass a dict as a parameter to a thread for instance, that variable can be manipulated and that affects the original version of your variable.
However, doing this is risky. There might be update collisions, access violations and in general just hard to keep track of where things happen.
But here's an example of how to pass a dict into a thread and present the change. It's crude, but gives you a working example.
from threading import *
class server(Thread):
def __init__(self, o):
self.o = o
Thread.__init__(self)
self.start()
def run(self):
for i in range(3):
self.o['test'] = i
test_var = {'test' : 0}
server(test_var)
while len(enumerate()) > 1: # Stupid and oversimplified wait for threads to end.
pass
print(test_var)
The problem was actually somewhere else. :( The reason was, that Flask (the Webserver) was started in a separate thread and there some instances are copied for some reason. I did not dig any further, Just avoided to use Flask and switched to cherrypy and started it in a non-blocking mode.
Thinkgs started to work there as expected.
#torxed, thank you very much for your help. It indeed pointed me into the right direction!
Imagine the following classes:
Class Object(threading.Thread):
# some initialisation blabla
def run(self):
while True:
# do something
sleep(1)
class Checker():
def check_if_thread_is_alive(self):
o = Object()
o.start()
while True:
if not o.is_alive():
o.start()
I want to restart the thread in case it is dead. This doens't work. Because the threads can only be started once. First question. Why is this?
For as far as I know I have to recreate each instance of Object and call start() to start the thread again. In case of complex Objects this is not very practical. I've to read the current values of the old Object, create a new one and set the parameters in the new object with the old values. Second question: Can this be done in a smarter, easier way?
The reason why threading.Thread is implemented that way is to keep correspondence between a thread object and operating system's thread. In major OSs threads can not be restarted, but you may create another thread with another thread id.
If recreation is a problem, there is no need to inherit your class from threading.Thread, just pass a target parameter to Thread's constructor like this:
class MyObj(object):
def __init__(self):
self.thread = threading.Thread(target=self.run)
def run(self):
...
Then you may access thread member to control your thread execution, and recreate it as needed. No MyObj recreation is required.
See here:
http://docs.python.org/2/library/threading.html#threading.Thread.start
It must be called at most once per thread object. It arranges for the
object’s run() method to be invoked in a separate thread of control.
This method will raise a RuntimeError if called more than once on the
same thread object.
A thread isn't intended to run more than once. You might want to use a Thread Pool
I believe, that has to do with how Thread class is implemented. It wraps a real OS thread, so that restarting the thread would actually change its identity, which might be confusing.
A better way to deal with threads is actually through target functions/callables:
class Worker(object):
""" Implements the logic to be run in separate threads """
def __call__(self):
# do useful stuff and change the state
class Supervisor():
def run(self, worker):
thr = None
while True:
if not thr or not thr.is_alive():
thr = Thread(target=worker)
thr.daemon = True
thr.start()
thr.join(1) # give it some time
I am trying to create a class than can run a separate process to go do some work that takes a long time, launch a bunch of these from a main module and then wait for them all to finish. I want to launch the processes once and then keep feeding them things to do rather than creating and destroying processes. For example, maybe I have 10 servers running the dd command, then I want them all to scp a file, etc.
My ultimate goal is to create a class for each system that keeps track of the information for the system in which it is tied to like IP address, logs, runtime, etc. But that class must be able to launch a system command and then return execution back to the caller while that system command runs, to followup with the result of the system command later.
My attempt is failing because I cannot send an instance method of a class over the pipe to the subprocess via pickle. Those are not pickleable. I therefore tried to fix it various ways but I can't figure it out. How can my code be patched to do this? What good is multiprocessing if you can't send over anything useful?
Is there any good documentation of multiprocessing being used with class instances? The only way I can get the multiprocessing module to work is on simple functions. Every attempt to use it within a class instance has failed. Maybe I should pass events instead? I don't understand how to do that yet.
import multiprocessing
import sys
import re
class ProcessWorker(multiprocessing.Process):
"""
This class runs as a separate process to execute worker's commands in parallel
Once launched, it remains running, monitoring the task queue, until "None" is sent
"""
def __init__(self, task_q, result_q):
multiprocessing.Process.__init__(self)
self.task_q = task_q
self.result_q = result_q
return
def run(self):
"""
Overloaded function provided by multiprocessing.Process. Called upon start() signal
"""
proc_name = self.name
print '%s: Launched' % (proc_name)
while True:
next_task_list = self.task_q.get()
if next_task is None:
# Poison pill means shutdown
print '%s: Exiting' % (proc_name)
self.task_q.task_done()
break
next_task = next_task_list[0]
print '%s: %s' % (proc_name, next_task)
args = next_task_list[1]
kwargs = next_task_list[2]
answer = next_task(*args, **kwargs)
self.task_q.task_done()
self.result_q.put(answer)
return
# End of ProcessWorker class
class Worker(object):
"""
Launches a child process to run commands from derived classes in separate processes,
which sit and listen for something to do
This base class is called by each derived worker
"""
def __init__(self, config, index=None):
self.config = config
self.index = index
# Launce the ProcessWorker for anything that has an index value
if self.index is not None:
self.task_q = multiprocessing.JoinableQueue()
self.result_q = multiprocessing.Queue()
self.process_worker = ProcessWorker(self.task_q, self.result_q)
self.process_worker.start()
print "Got here"
# Process should be running and listening for functions to execute
return
def enqueue_process(target): # No self, since it is a decorator
"""
Used to place an command target from this class object into the task_q
NOTE: Any function decorated with this must use fetch_results() to get the
target task's result value
"""
def wrapper(self, *args, **kwargs):
self.task_q.put([target, args, kwargs]) # FAIL: target is a class instance method and can't be pickled!
return wrapper
def fetch_results(self):
"""
After all processes have been spawned by multiple modules, this command
is called on each one to retreive the results of the call.
This blocks until the execution of the item in the queue is complete
"""
self.task_q.join() # Wait for it to to finish
return self.result_q.get() # Return the result
#enqueue_process
def run_long_command(self, command):
print "I am running number % as process "%number, self.name
# In here, I will launch a subprocess to run a long-running system command
# p = Popen(command), etc
# p.wait(), etc
return
def close(self):
self.task_q.put(None)
self.task_q.join()
if __name__ == '__main__':
config = ["some value", "something else"]
index = 7
workers = []
for i in range(5):
worker = Worker(config, index)
worker.run_long_command("ls /")
workers.append(worker)
for worker in workers:
worker.fetch_results()
# Do more work... (this would actually be done in a distributor in another class)
for worker in workers:
worker.close()
Edit: I tried to move the ProcessWorker class and the creation of the multiprocessing queues outside of the Worker class and then tried to manually pickle the worker instance. Even that doesn't work and I get an error
RuntimeError: Queue objects should only be shared between processes
through inheritance
. But I am only passing references of those queues into the worker instance?? I am missing something fundamental. Here is the modified code from the main section:
if __name__ == '__main__':
config = ["some value", "something else"]
index = 7
workers = []
for i in range(1):
task_q = multiprocessing.JoinableQueue()
result_q = multiprocessing.Queue()
process_worker = ProcessWorker(task_q, result_q)
worker = Worker(config, index, process_worker, task_q, result_q)
something_to_look_at = pickle.dumps(worker) # FAIL: Doesn't like queues??
process_worker.start()
worker.run_long_command("ls /")
So, the problem was that I was assuming that Python was doing some sort of magic that is somehow different from the way that C++/fork() works. I somehow thought that Python only copied the class, not the whole program into a separate process. I seriously wasted days trying to get this to work because all of the talk about pickle serialization made me think that it actually sent everything over the pipe. I knew that certain things could not be sent over the pipe, but I thought my problem was that I was not packaging things up properly.
This all could have been avoided if the Python docs gave me a 10,000 ft view of what happens when this module is used. Sure, it tells me what the methods of multiprocess module does and gives me some basic examples, but what I want to know is what is the "Theory of Operation" behind the scenes! Here is the kind of information I could have used. Please chime in if my answer is off. It will help me learn.
When you run start a process using this module, the whole program is copied into another process. But since it is not the "__main__" process and my code was checking for that, it doesn't fire off yet another process infinitely. It just stops and sits out there waiting for something to do, like a zombie. Everything that was initialized in the parent at the time of calling multiprocess.Process() is all set up and ready to go. Once you put something in the multiprocess.Queue or shared memory, or pipe, etc. (however you are communicating), then the separate process receives it and gets to work. It can draw upon all imported modules and setup just as if it was the parent. However, once some internal state variables change in the parent or separate process, those changes are isolated. Once the process is spawned, it now becomes your job to keep them in sync if necessary, either through a queue, pipe, shared memory, etc.
I threw out the code and started over, but now I am only putting one extra function out in the ProcessWorker, an "execute" method that runs a command line. Pretty simple. I don't have to worry about launching and then closing a bunch of processes this way, which has caused me all kinds of instability and performance issues in the past in C++. When I switched to launching processes at the beginning and then passing messages to those waiting processes, my performance improved and it was very stable.
BTW, I looked at this link to get help, which threw me off because the example made me think that methods were being transported across the queues: http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html
The second example of the first section used "next_task()" that appeared (to me) to be executing a task received via the queue.
Instead of attempting to send a method itself (which is impractical), try sending a name of a method to execute.
Provided that each worker runs the same code, it's a matter of a simple getattr(self, task_name).
I'd pass tuples (task_name, task_args), where task_args were a dict to be directly fed to the task method:
next_task_name, next_task_args = self.task_q.get()
if next_task_name:
task = getattr(self, next_task_name)
answer = task(**next_task_args)
...
else:
# poison pill, shut down
break
REF: https://stackoverflow.com/a/14179779
Answer on Jan 6 at 6:03 by David Lynch is not factually correct when he says that he was misled by
http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html.
The code and examples provided are correct and work as advertised. next_task() is executing a task received via the queue -- try and understand what the Task.__call__() method is doing.
In my case what, tripped me up was syntax errors in my implementation of run(). It seems that the sub-process will not report this and just fails silently -- leaving things stuck in weird loops! Make sure you have some kind of syntax checker running e.g. Flymake/Pyflakes in Emacs.
Debugging via multiprocessing.log_to_stderr()F helped me narrow down the problem.
Is there any multiprocessing type module for Python 2.3? I am stuck using 2.3 for the programs I interface with and would like to be able to setup some multiprocessing as the tasks I do only use one CPU and are really inefficient.
I would like each thread/process to handle its own global variables and each thread/process should not share any variables with any other thread/process. Basically I would just like to have a queue of files that need be run through a function and each run would be an entirely new thread.
I have tried using thread.start_new_thread, but it just turned into a mess with my global variables.
A thought just occurred to me, can I do a os.popen('python C:\function_dir\function.py vars...') from each new thread? Sounds rather ugly, but I don't see why it wouldn't work. The master program wouldn't continue until the os.popen "thread" finishes correct?
Any thoughts or modules I may be overlooking?
None that I ever found anywhere, I have since moved on to python 2.5
Use threading. You simply need to build a class based on Thread:
import threading
class myThread(threading.Thread):
#
# Constructor.
#
def __init__(self, ...):
#
# Call threading constructor.
#
threading.Thread.__init__(self)
#
# Your constructor code.
#
...
#
# The code executed when starting the thread.
#
def run(self):
...
#
# Create an instance and start the thread.
#
myThread(...).start()
Make sure to keep all your variables local. If you need to access global variables use the global statement:
counter = 0
class myThread(threading.Thread):
...
def run(self):
global counter
...
counter = 17
...
For locking, etc. have a look at the Python documentation as well: http://docs.python.org/release/2.3.5/lib/module-threading.html
I Have run into a few examples of managing threads with the threading module (using Python 2.6).
What I am trying to understand is how is this example calling the "run" method and where. I do not see it anywhere. The ThreadUrl class gets instantiated in the main() function as "t" and this is where I would normally expect the code to start the "run" method.
Maybe this is not the preferred way of working with threads? Please enlighten me:
#!/usr/bin/env python
import Queue
import time
import urllib2
import threading
import datetime
hosts = ["http://example.com/", "http://www.google.com"]
queue = Queue.Queue()
class ThreadUrl(threading.Thread):
"""Threaded Url Grab"""
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
def run(self):
while True:
#grabs host from queue
host = self.queue.get()
#grabs urls of hosts and prints first 1024 bytes of page
url = urllib2.urlopen(host)
print url.read(10)
#signals to queue job is done
self.queue.task_done()
start = time.time()
def main():
#spawn a pool of threads, and pass them queue instance
for i in range(1):
t = ThreadUrl(queue)
t.setDaemon(True)
t.start()
for host in hosts:
queue.put(host)
queue.join()
main()
print "Elapsed time: %s" % (time.time() - start)
Per the pydoc:
Thread.start()
Start the thread’s activity.
It must be called at most once per thread object. It arranges for the
object’s run() method to be invoked in
a separate thread of control.
This method will raise a RuntimeException if called more than
once on the same thread object.
The way to think of python Thread objects is that they take some chunk of python code that is written synchronously (either in the run method or via the target argument) and wrap it up in C code that knows how to make it run asynchronously. The beauty of this is that you get to treat start like an opaque method: you don't have any business overriding it unless you're rewriting the class in C, but you get to treat run very concretely. This can be useful if, for example, you want to test your thread's logic synchronously. All you need is to call t.run() and it will execute just as any other method would.
The method run() is called behind the scene by "threading.Thread" (Google inheritance and polymorphism concepts of OOP). The invocation will be done just after t.start() has called.
If you have an access to threading.py (find it in python folder). You will see a class name Thread. In that class, there is a method called "start()". start() called '_start_new_thread(self.__bootstrap, ())' a low-level thread start-up which will run a wrapper method called '__bootstrap()' by a new thread. '__bootstrap()', then, called '__bootstrap_inner()' which do some more preparation before, finally, call 'run()'.
Read the source, you can learn a lot. :D
t.start() creates a new thread in the OS and when this thread begins it will call the thread's run() method (or a different function if you provide a target in the Thread constructor)