We Keep Coding

Python script freezing program

I am new here but I recently have been messing around with python and Qt. My situation is that one of the scripts I call does a lot of OS commands and basically waits for a response. When I call this script it runs fine and acts accordingly except in my main program the screen is frozen until I exit out of the cmd. I think this is because mt script just waits there for a response, is there anyway to make it so that even though the script is running and executing(waiting for response with cmd) the user can still use other aspects of the main program?

As mentioned in comments, you will need to use threading. Threading allows multiple functions to executed at the same time. Check out this link Python threading.
You'll just have to run your side script on a different thread.

Python interactive subprocess reading

I have an interactive Windows console application that I communicate through subprocess module in my program. So far, the console commands(of the application) that I called from my script did not made any problems because they gave instant results(pipe outputs).
However there are other functionalities I need to use that keeps printing output to the console until it recieves "stop" command. I need to read those outputs, but my application is a GUI and I cannot let any hanging occur, and I don't want to get all the output after stop command is issued, I need to get it realtime, when it is served. I made some research and it appears this is a difficult problem to overcome(especially in Windows), but there are no recent questions about this topic.
I did find this,
Non-blocking read on a subprocess.PIPE in python, looks like what I am looking for.
But the answer seems really low-level for a Python solution and it is old. Are there any better-newer solutions for this problem ?
Thanks in advance.

Threads not being executed under supervisord

I am working on a basic crawler which crawls 5 websites concurrently using threads.
For each site it creates a new thread. When I run the program from the shell then the output log indicates that all the 5 threads run as expected.
But when I run this program as a supervisord program then the log indicates that only 2 threads are being run everytime! The log indicates that the all the 5 threads have started but only the same two of them are being executed and the rest get stuck.
I cannot understand why this inconsistency is happening when it is run from a shell and when it run from supervisor. Is there something I am not taking into account?
Here is the code which creates the threads:
for sid in entries:
url = entries[sid]
threading.Thread(target=self.crawl_loop, \
args=(sid, url)).start()
UPDATES:
As suggested by tdelaney in the comments, I changed the working directory in the supervisord configuration and now all the threads are being run as expected. Though I still don't understand that why setting the working directory to the crawler file directory rectifies the issue. Perhaps some one who knows about how supervisor manages processes can explain?

AFAIK python threads can't do threads properly because it is not thread safe. It just gives you a facility to simulate simultaneous run of the code. Your code will still use 1 core only.
https://wiki.python.org/moin/GlobalInterpreterLock
https://en.wikibooks.org/wiki/Python_Programming/Threading
Therefore it is possible that it does not spawn more processes/threads.
You should use multiprocessing I think?
https://docs.python.org/2/library/multiprocessing.html

I was having the same silent problem, but then realised that I was setting daemon to true, which was causing supervisor problems.
https://docs.python.org/2/library/threading.html#threading.Thread.daemon
So the answer is, daemon = true when running the script yourself, false when running under supervisor.

Just to say, I was just experiencing a very similar problem.
In my case, I was working on a low powered machine (RaspberryPi), with threads that were dedicated to listening to a serial device (an Arduino nano on /dev/ttyUSB0). Code worked perfectly on the command line - but the serial reading thread stalled under supervisor.
After a bit of hacking around (and trying all of the options here), I tried running python in unbuffered mode and managed to solve the issue! I got the idea from https://stackoverflow.com/a/17961520/741316.
In essence, I simply invoked python with the -u flag.

Running same Pexpect program simultaneously on the same server but differnt terminals

I have a python pexpect code lets say Program1.py which logs-in to one router from each program using pexpect.spawn and does required operations using sendline and expect modules in pexpect.
If I try to run this same program, from multiple prompts on my server with each program login to two diffrerent routers only one program seems to get the expect() input while the other instance of the programs times out atchild.expect() -> at read non_blocing()
Example:
In prompt-1 on my RHEL Server I execute the program to login to router X with ARGs
bash$ python program1.py 10.11.12.13/2001 configure_MGMT
In Prompt-2 on my RHEL Server I execute the program to login to router Y with ARGs
bash$ python program1.py 20.20.12.13/2020 configure_MGMT
one of the Programs runs successfully, while the other hits TIMEOUT at the first child.expect() call.
Is this due to the GIL?
Is there a workaround for this?
(I wish to avoid multiprocessing here, because my webserver handles the multiprocessing aspect and executes the same program multiple times.)

The GIL has nothing to do with this, because independent processes do not share a GIL. The most likely cause of this is that your router only supports one log-in at a time. The easiest way to verify that this is the problem is to remove Python from the equation by manually logging in to the router from two different terminal sessions at the same time.

Found the answer to this. Since python programs will always get executed synchronously and not in parallel, the pexpect will timeout waiting for input while the process is waiting to get scheduled.
to run all of them as background processes and in parallel, we need to use the '&' at the end of execution CLI.
Example:
bash$ python program1.py 20.20.12.13/2020 configure_MGMT &
thanks

Python: How to Run multiple programs on same interpreter

How to start an always on Python Interpreter on a server?
If bash starts multiple python programs, how can I run it on just one interpreter?
And how can I start a new interpreter after tracking number of bash requests, say after X requests to python programs, a new interpreter should start.
EDIT: Not a copy of https://stackoverflow.com/questions/16372590/should-i-run-1000-python-scripts-at-once?rq=1
Requests may come pouring in sequentially

You cannot have new Python programs started through bash run on the same interpreter, each program will always have its own. If you want to limit the number of Python programs running the best approach would be to have a Python daemon process running on your server and instead of creating a new program through bash on each request you would signal the daemon process to create a thread to handle the task.

To run a program forever in python:
while True :
do_work()
You could look at spawning threads for incoming request. Look at threading.Thread class.
from threading import Thread
task = new Thread(target=do_work, args={})
task.start()

You probably want to take a look at http://docs.python.org/3/library/threading.html and http://docs.python.org/3/library/multiprocessing.html; threading would be more lightweight but only allows one thread to execute at a time (meaning it won't take advantage of multicore/hyperthreaded systems), while multiprocessing allows for true simultaneous execution but can be a bit less lightweight than threading if you're on a system that doesn't utilize lightweight subprocesses and may not be as necessary if the threads/processes spend lots of time doing I/O requests.