I am trying to develop a downloader app in pygtk
So when a user adds a url following actions happen
addUrl()
which calls
validateUrl()
getUrldetails()
So it took a little while to add the url to the list because of urllib.urlopen delay
so i tried to implement threads. I added the following code to main window
thread.start_new_thread(addUrl, (self,url, ))
I passed a reference to the main window so that i can access the list from thread
but nothing seems to happen
I think that you check this thread first How to use threading in Python?.
for example:
import Queue
import threading
import urllib2
# called by each thread
def get_url(q, url):
q.put(urllib2.urlopen(url).read())
theurls = '''http://google.com http://yahoo.com'''.split()
q = Queue.Queue()
for u in theurls:
t = threading.Thread(target=get_url, args = (q,u))
t.daemon = True
t.start()
s = q.get()
print s
Hope this helps you.
Related
I am trying to use a Semaphore in Python to block the execution of a Thread, and release the Semaphore in another Thread depending on some conditions (in my case receiving some content over tcp).
But it is not working, my main Thread is not blocked like I though it would. My minimal example:
import time
from threading import Thread, Semaphore
class ServerManager():
sem = Semaphore()
def run(self):
time.sleep(15)
self.sem.release()
manager = ServerManager()
thread = Thread(target = manager.run)
thread.start()
manager.sem.acquire()
print("I am here immediately but should not")
The message is printed immediately in the console but should only be printed after 15 seconds in my example. I'm sure I did something horribly wrong, but what?
You need to read the documentation about threading.Semaphore. A semaphore blocks when a thread tries to acquire and the counter is 0. When you create a Semaphore, the default count is 1, so your acquire is able to succeed immediately. I suspect you want
sem = Semaphore(0)
so the resource is immediately unavailable.
The answer from Tim Roberts is right.I read the Python doc, but I did not understand it. I thought (wrongly) that the default value had the behavior I wanted. The full code is:
I should do:
import time
from threading import Thread, Semaphore
class ServerManager():
sem = Semaphore(0)
def run(self):
time.sleep(15)
self.sem.release()
manager = ServerManager()
thread = Thread(target = manager.run)
thread.start()
manager.sem.acquire()
print("I am here immediately but should not")
The message is printed after 15 seconds.
So I have been struggling with this one error of pickle which is driving me crazy. I have the following master Engine class with the following code :
import eventlet
import socketio
import multiprocessing
from multiprocessing import Queue
from multi import SIOSerever
class masterEngine:
if __name__ == '__main__':
serverObj = SIOSerever()
try:
receiveData = multiprocessing.Process(target=serverObj.run)
receiveData.start()
receiveProcess = multiprocessing.Process(target=serverObj.fetchFromQueue)
receiveProcess.start()
receiveData.join()
receiveProcess.join()
except Exception as error:
print(error)
and I have another file called multi which runs like the following :
import multiprocessing
from multiprocessing import Queue
import eventlet
import socketio
class SIOSerever:
def __init__(self):
self.cycletimeQueue = Queue()
self.sio = socketio.Server(cors_allowed_origins='*',logger=False)
self.app = socketio.WSGIApp(self.sio, static_files={'/': 'index.html',})
self.ws_server = eventlet.listen(('0.0.0.0', 5000))
#self.sio.on('production')
def p_message(sid, message):
self.cycletimeQueue.put(message)
print("I logged : "+str(message))
def run(self):
eventlet.wsgi.server(self.ws_server, self.app)
def fetchFromQueue(self):
while True:
cycle = self.cycletimeQueue.get()
print(cycle)
As you can see I can trying to create two processes of def run and fetchFromQueue which i want to run independently.
My run function starts the python-socket server to which im sending some data from a html web page ( This runs perfectly without multiprocessing). I am then trying to push the data received to a Queue so that my other function can retrieve it and play with the data received.
I have a set of time taking operations that I need to carry out with the data received from the socket which is why im pushing it all into a Queue.
On running the master Engine class I receive the following :
Can't pickle <class 'threading.Thread'>: it's not the same object as threading.Thread
I ended!
[Finished in 0.5s]
Can you please help with what I am doing wrong?
From multiprocessing programming guidelines:
Explicitly pass resources to child processes
On Unix using the fork start method, a child process can make use of a shared resource created in a parent process using a global resource. However, it is better to pass the object as an argument to the constructor for the child process.
Apart from making the code (potentially) compatible with Windows and the other start methods this also ensures that as long as the child process is still alive the object will not be garbage collected in the parent process. This might be important if some resource is freed when the object is garbage collected in the parent process.
Therefore, I slightly modified your example by removing everything unnecessary, but showing an approach where the shared queue is explicitly passed to all processes that use it:
import multiprocessing
MAX = 5
class SIOSerever:
def __init__(self, queue):
self.cycletimeQueue = queue
def run(self):
for i in range(MAX):
self.cycletimeQueue.put(i)
#staticmethod
def fetchFromQueue(cycletimeQueue):
while True:
cycle = cycletimeQueue.get()
print(cycle)
if cycle >= MAX - 1:
break
def start_server(queue):
server = SIOSerever(queue)
server.run()
if __name__ == '__main__':
try:
queue = multiprocessing.Queue()
receiveData = multiprocessing.Process(target=start_server, args=(queue,))
receiveData.start()
receiveProcess = multiprocessing.Process(target=SIOSerever.fetchFromQueue, args=(queue,))
receiveProcess.start()
receiveData.join()
receiveProcess.join()
except Exception as error:
print(error)
0
1
...
I'm trying to read a very slow sensor (1-wire) while still operating other functions. I've replaced the read sensors with a sleep of 3 sec. I'd like to understand if I can have below time concurrently print accurate time every second while the sensor (wait 3sec.) occurs. This threading concept is new to me.
import time
import threading
from tkinter import Tk
def events_every_second(): #update clock every second
right_now = time.strftime("%I:%M:%S %p")#("%H:%M:%S")
print("time is now",right_now)
root.after(1000, events_every_second)
def one_wire():
time.sleep(3)
print("one_wire loop")
root.after(3010, one_wire)
root = Tk()
t_one_wire = one_wire()
thread_one_wire = threading.Thread(target = t_one_wire)
thread_one_wire.start()
t_ees = events_every_second
thread_ees = threading.Thread(target = t_ees)
thread_ees.start()
root.mainloop()
Just the function that does the lenghty sensor read needs to be
in a separate thread.
You can use a queue.Queue to obtain data from it -
in the example bellow I inserted the value fetching in the
every_one_sec function.
The every_one_sec function is handled by Tkinter scheduling events -
no need to creat a thread for it.
Other than that, the most incorrect part in your code was doing a
full call to one_wire before creating the thread in
t_one_wire = one_wire() - and the fact that it would run also require
tkinter to call it back. Tkinter wants to run all its events in the same
thread - so this would lead to problems.
I hope the comments bellow are enough for a better comprehension
import time
import threading
from queue import Queue, Empty
from tkinter import Tk
def some_ui_code():
global stop_reading
stop_reading = True
def events_every_second(): #update clock every second
right_now = time.strftime("%I:%M:%S %p")#("%H:%M:%S")
print("time is now",right_now)
try:
result = queue.get(block=False)
except Empty:
result = None
if result is not None:
# code to display the result read from the sensor in tkinter interface goes here
...
# tkinter, not a separate thread, handles this
root.after(1000, events_every_second)
def one_wire(queue):
# this is handled in a separate thread, and "does not know" about tkinter at all
while not stop_reading:
result = call_to_read_sensor() # takes 3 seconds
queue.put(result)
print("one_wire loop")
time.sleep(0.1) # actual call to time.sleep to space sensor readings, if needed.
root = Tk()
stop_reading = False
queue = Queue()
# start events_every_second - tkinter will handle the subsequent calls with the `root.after` scheduling
events_every_second()
thread_one_wire = threading.Thread(target=t_one_wire, args=(queue,))
thread_one_wire.start()
root.mainloop()
You should not use root.after(3010,one_wire) this will cause your Tk GUI to hang for 3 seconds and don't update.
You want to create a new Thread for this function so it does not stop your Tk app.
def one_wire():
time.sleep(3)
print("one_wire_loop")
create a new thread for this function.
import threading
one_wire_thread = threading.Thread(target = one_wire, args = ())
one_wire_thread.start()
Note that the code above will only run your function once. you could create a wrapper function with a while True: in there to keep it running, in the wrapper function you could also check for condition to stop and break the function and thus stop the one_wire function. you could also do this in the one_wire function:
def one_wire():
while True:
time.sleep(3)
print("one_wire_loop")
if {"condition to stop reading sensor"}:
break
Is there a way to create a callback that executes whenever something is sent to the main process from a child process initiated via multiprocessing? The best I can think of thus far is:
import multiprocessing as mp
import threading
import time
class SomeProcess(mp.Process):
def run(self):
while True
time.sleep(1)
self.queue.put(time.time())
class ProcessListener(threading.Thread):
def run(self):
while True:
value = self.queue.get()
do_something(value)
if __name__ = '__main__':
queue = mp.Queue()
sp = SomeProcess()
sp.queue = queue
pl = ProcessListener()
pl.queue = queue
sp.start()
pl.start()
No there is no other clean way to do so than the one you already posted.
This is how concurrent.fututes.ProcessPoolExecutor and multiprocessing.Pool are actually implemented. They have a dedicated thread which drains the tasks/results queue and run any associated callback.
If you want to save some resource, you can use a SimpleQueue in this case.
I have a python program that I have written. This python program calls a function within a module I have also written and passes it some data.
program:
def Response(Response):
Resp = Response
def main():
myModule.process_this("hello") #Send string to myModule Process_this function
#Should wait around here for Resp to contain the Response
print Resp
That function processes it and passes it back as a response to function Response in the main program.
myModule:
def process_this(data)
#process data
program.Response(data)
I checked and all the data is being passed correctly. I have left out all the imports and the data processing to keep this question as concise as possible.
I need to find some way of having Python wait for resp to actually contain the response before proceeding with the program. I've been looking threading and using semaphores or using the Queue module, but i'm not 100% sure how I would incorporate either into my program.
Here's a working solution with queues and the threading module. Note: if your tasks are CPU bound rather than IO bound, you should use multiprocessing instead
import threading
import Queue
def worker(in_q, out_q):
""" threadsafe worker """
abort = False
while not abort:
try:
# make sure we don't wait forever
task = in_q.get(True, .5)
except Queue.Empty:
abort = True
else:
# process task
response = task
# return result
out_q.put(response)
in_q.task_done()
# one queue to pass tasks, one to get results
task_q = Queue.Queue()
result_q = Queue.Queue()
# start threads
t = threading.Thread(target=worker, args=(task_q, result_q))
t.start()
# submit some work
task_q.put("hello")
# wait for results
task_q.join()
print "result", result_q.get()