Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I have a Flask application that schedules long running jobs to run using python-rq. One of my requirements is that the user can specify the number of jobs running at any given time.
The app doesn't need to kill any job in case the user use a smaller value than currently running jobs, but it needs to spawn another one in case the user increases the limit.
To run a job, the rq worker takes some time, but it doesn't need to babysit the job, it can safely run it and move on to the next one.
My issue is, sometimes the initial setup phase can take some time, so using only one worker might not be ideal. Another issue, which bugs me more, is that using this scheme my rq workers have to poll the database to know when they can go on and launch another job. Is there a better way to architect this?
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 12 months ago.
Improve this question
I recently heard of this feature in Python3.7+ where the asyncio brought a thing called "tasks" which people refer to as background tasks. So that's may first question:
Do these tasks really run in background?
Also, when comparing asyncio tasks to threads in Python, we know that python has a GIL. So, there's nothing like parallel. I know the difference in core structure i.e. asyncio tasks run in an event loop inside the same thread, while python threads are simply forked threads. But when it comes to speed, none of these are parallel.
We can call them concurrent instead. So the second question is:
Which of these two would be faster?
A few things I got to know about memory consumption is:
Threads consume a fair amount of data since each thread needs to have its own stack. With async code, all the code shares the same stack and the stack is kept small due to continuously unwinding the stack between tasks.
Threads are OS structures and therefore require more memory for the platform to support. There is no such problem with asynchronous tasks.
References:
What does asyncio.create_task() do?
How does asyncio actually work?
Coming to my last question:
When should you use asyncio tasks compared to threads? (This question has came in my mind as we can even fire async task from sync code)
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
Hi I have a Server/client model using SocketServer module. The server job is to receive test name from the clients and launch the test.
the test is launched using subprocess module.
I would like the server to keep answering clients and any new jobs to be stacked on a list or queue and launch one after the other, the only restriction I have is the server should not launch the test unless currently running one is completed.
Thanks
You can use the module multiprocessing for starting new processes. On the server-side, you would have a variable which refers to the current running process. You can still have your SocketServer running and accepting requests and storing them in a list. Every second (or whatever you want), in another thread, you would check if the current process is dead or not by calling isAlive(). If it is dead, then just simply run the next test on the list.
Another way to do it (better), is that on the third thread (the one that checks), you call .join() from the process so that it will only call the next line of code once the current process is dead. That way you don't have to keep checking every second or whatever and it is more efficient.
What you might want to do is:
Get test name in server socket, put it in a Queue
In a separate thread, read test names from the Queue one by one
Execute the process and wait for it to end using communicate()
Keep polling Queue for new tests, repeat steps 2, 3 if test names are available
Meanwhile server continues receiving and putting test names in Queue
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I'm creating a program in python that auto-runs for an embedded system and want to handle program interruptions gracefully. That is if possible close resources and signal child processes to also exit gracefully before actually exiting. At which point my watchdog should notice this and respawn everything.
What signals can/should I expect to receive in a non-interactive program from linux? I'm using try/except blocks to handle i/o errors.
Is the system shutting down an event that is signaled? In addition to my watchdog I will also have an independent process monitoring a hardware line that gets set when my power supply detects a brownout. I have a supercap to provide some run-time to allow a proper shutdown.
Trap sigint, sigterm and make sure to clean up anything like sockets, files, locks, etc.
Trap other signals based on what you are doing. For instance if you have open pipes you might trap sigpipe.
Just remember signal handling opens you to race conditions. You probably want to use sigprocmask to disable signals while handling them.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
Recently I've started working on Python socket server which handles raw UTF input from Java's streams and sends the result back on all of the currently connected servers, and that works fine, but I'm so pumped and worried about thread usage: you see, I'm using about 2 threads per each connection and I'm worried that CPU will die out that way soon, so, I need a better solution now so that my server could handle hundreds of connections.
I have two ideas for that:
Using a non-blocking IO
Having a fixed amount of thread pools (i.e. FixedThreadPool as it called in Java)
I have no idea which one is gonna work better, so I'd appreciate your advice and ideas.
Thanks!
I would advise not to invent a bicycle and to use some framework for async/streaming processing. For example Tornado.
Also if you can consider using Go language - a lot of developers (including me) are switching from Python to Go for this kind of tasks. It's designed from ground up to support async processing.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I need to develop the performance review based python script , here is the scenario.
I need to send the logs to ElK (Elasticsearch, logstash , Kibana)
from yocto linux but only when system resources are free enough
So what I need here a python script which continuously monitor the
system performance and when system resources like CPU is less then 50%
start sending the logs and if CPU again goes above 50% PAUSE the logging
Now I am don't have idea we can pause any process with python
or not? This is because I want this for logs so when its start
again send the logs from where it stops last time
Yes, all your requirements are possible in Python.
In fact it's possible in basically any language because you're not asking for cutting edge stuff, this is basic scripting.
Sending logs to ES/Kibana
It's possible, Kibana, ES and Splunk all have public API's with good documentation on how to do it, so yes it's possible.
Pausing a process in Linux
Yes, also possible. If it's a external process simply find the PID of your process and send kill -STOP <PID> which would stop the process, to resume the process, do run kill -CONT <PID>. If it's your own process that you want to pause, simply enter a sleep cycle in your code (simple example while PAUSED: time.sleep(0.5).