We are using python multithreading(I/O bound) for our application deployments. From main/parent thread we are creating multiple child threads (multiple hosts [application/web] for deployment within each thread) and waiting for them to finish using .join() in main thread. We are also using some Fabric package commands to ask user input whether to proceed with deployment or not in-case of an error for whatsoever reason.
Problem statement:
Since we are using multithreading, all the child threads are writing the messages to terminal (stdout)output and thus the errors from one thread on the screen are overlapping by other thread messages and it is difficult to comb through the console/terminal to see what thread has failures.
What could be the best way to gather all the failure steps in the deployment from all the threads collectively and display all of them at once on the console/terminal output ?
The difficulty I'm facing is how to interact with child threads while in main thread we are just waiting to finish them (if there are any fabric user interactive questions then will type y/n based on our decision about the failure).
Related
Is there a difference in thread operation functionality between an application with 3 daemon threads that all pull from a multiprocessing Queue and 4 separate applications: a multiprocessing Queue/Pipe and 3 daemon thread applications that read from the Queue/Pipe application?
Neither application uses blocking/synchronisation. At the end of the day the operating system will decide when to allow a thread to run and for how long. Are there any other differences in functionality here or are they essentially the same?
Generic Application (no synchronisation or blocking):
'Stock Market Feed' Queue: StockTrade messages (dictonaries)
'TradingStrategy' 1 Daemon Thread: Pull from queue, inspect messages and perform trades
'TradingStrategy' 1 Daemon Thread: Pull from queue, inspect messages and perform trades
'TradingStrategy' 1 Daemon Thread: Pull from queue, inspect messages and perform trades
Alternate architecture:
Feed Application (no multi-threading):
'Stock Market Feed' Queue or Pipe: StockTrade messages (dictonaries). Can a Queue be accessed from another outside process? I know a named pipe can but can a queue?
Trading Application (no multi-threading):
'TradingStrategy': Interacts with feed (pipe?/queue), inspect messages and perform trades
Trading Application (no multi-threading):
'TradingStrategy': Interacts with feed (pipe?/queue), inspect messages and perform trades
Trading Application (no multi-threading):
'TradingStrategy': Interacts with feed (pipe?/queue), inspect messages and perform trades
Yes, the two options are quite different. But it gets complicated fast trying to explain the difference. You should research and read up on the differences between a thread and a process. Get that in your head straight first.
Now, given your specific scenario, assuming by "multiprocessing queue" you actually mean an instance of a python Queue in one thread of a process, since the queue is inside the same process as all the worker threads, the workers will be able to access, and share that same instance of the Queue.
However when the workers are all separate processes then they cannot access the Queue by shared memory and will need some form of interprocess communication to gain access to that queue.
In practice, I'd be thinking something like redis or zeromq to be your queue, then build a python program to talk to it, then scale up as few, or as many copies of it, as you need.
I am creating a program in Python that listens to varios user interactions and logs them. I have these requirements/restrictions:
I need a separate process that sends those logs to a remote database every hour
I can't do it in the current process because it blocks the UI.
If the main process stops, the background process should also stop.
I've been reading about subprocess but I can't seem to find anything on how to stop both simultaneously. I need the equivalent of spawn_link if anybody know some Erlang/Elixir.
Thanks!
To answer the question in the title (for visitors from google): there are robust solutions on Linux, Windows using OS-specific APIs and less robust but more portable psutil-based solutions.
To fix your specific problem (it is XY problem): use a daemon thread instead of a process.
A thread would allow to perform I/O without blocking GUI, code example (even if GUI you've chosen doesn't provide async. I/O API such as tkinter's createfilehandler() or gtk's io_add_watch()).
I built a web-scraper application with Python. It consists of three main parts:
The GUI (built on tkinter)
A Client (controls interface between front- and back-end)
Back-end code (various threaded processes).
The problem I have is that when the user hits X to exit the program instead of quitting through the interface, it seems like root.destroy() never gets called and the application runs forever, even though the window does disappear. This ends up consuming vast amounts of system resources.
I have tried setting all threads to Daemon without much success. Is there any other reason the program would keep eating up CPU after exit?
You don't want to set all threads to daemon. You want to set the client thread and the back-end thread to daemon. That way, when the GUI thread dies, the threads with daemon set to True end as well.
From the documentation:
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left.
The Situation
I am using python 2.7. I am working on an application that has a GUI utilizing PyQt4.
I have a button in a widget and when you click it it will request a range of IDs specified by the user. The user can specify how many threads the program should allocate for the "job".
My Original Idea
The button spawns a job thread and passes it the range of IDs and the thread count. The job thread then can create child threads and pass them IDs from the range as necessary.
When all the IDs have been fetched the job thread can stop all its worker threads and then stop itself.
I would utilize signals / slots to update GUI elements with data as it is fetched by the worker threads within the job.
I am using threads so the GUI remains interact-able during the fetching process.
The Problem
The "Dreaded PyQt Multi-threading Issue" occurred when I tried to spawn a thread from within a thread.
QObject: Cannot create children for a parent that is in a different thread.
My Workaround
I simply spawned the job thread and the worker threads from the widget that contains the button so no thread is spawned from within a thread, and then "connected" the worker threads to the job thread.
A la aThread = ScrapeThread(self, aJob) where aJob is the master Job thread.
This seems to work, but feels like a workaround more than the proper way to do this.
The Question
What I did seems like a workaround rather than the proper way to multi-thread in PyQt.
Any recommendations?
I'm running Django, and I'm creating threads that run in parallel while Django runs. Those threads sometimes run external processes that block while waiting for external input.
When I restart Django, those threads that are blocking while awaiting external input sometimes persist through the restart, and further they have and keep open Port 8080 so Django can't restart.
If I knew when Django was restarting, I could kill those threads. How can I tell when Django is restarting so that I can kill those threads (and their spawn).
It wasn't obvious from django.utils.autoreload where any hooks may be to tell when a restart is occurring.
Is there an alternative way to kill these threads when Django starts up?
Thanks for reading.
Brian
It's not easy for a Python process to kill its own threads -- even harder (nearly impossible) to kill the threads of another process, and I suspect the latter is the case you have... the "restart" is presumably happening on a different process, so those threads are more or less out of bounds for you!
What I suggest instead is "a stitch in time saves nine": when you create those threads, make sure you set their daemon property to True (see the docs -- it's the setDaemon method in Python <= 2.5). This way, when the main thread finishes, e.g. to restart in another process, so will the entire process (which should take all the daemon threads down, too, automatically!-)
What are you using to restart django? I'd put something in that script to look for process id's in the socket file(s) and kill those before starting django.
Alternatively, you could be very heavy handed and just run something like 'pkill -9 *django*' before your django startup sequence.