what I'm trying do to is:
start Screen session
inside this screen session, execute an alias (which, for completeness, goes into a directory and runs a .sh) inside that screen
and I want to do this task with my python program using the library ParallelSSH, hence to many remote hosts.
I had many problems during my work, in particular I experienced that:
I can't run an all-in-one command to create a screen and run the alias inside that
I need some trick to send the "enter" key, otherwise the alias (or any command) will just be written and not executed in the screen
There is some tricky problem in terms of process handling, given that (to debug my code problems) I tried to run just the commands through ssh from command line, as follows:
ssh mymachine screen -dmS myscreen
ssh mymachine screen -r myscreen -X stuff "myalias ; wait"
ssh mymachine screen -r myscreen -X eval "stuff \015"
What I obtain is that the screen is create, then I noticed that without the wait in the second command it doesn't write the command inside the screen, or, probably, it writes it but then it disappear, hence the wait in some way does help. The last command is to send the Enter key to run the alias.
The most strange thing is that this command sequence doesn't work 100/100: sometimes it runs correctly, sometimes it doesn't, and I don't understand why.
First of all I would like to understand these commands from command line, because then I want to execute them through ParallelSSH in my python program. What I do at the moment is the following:
start = client.run_command('screen -dmS XXX')
xxx_run = client.run_command("screen -r XXX -X stuff 'myalias' ; wait")
I didn't wrote the third command since the second one yet doesn't work, while the first one runs correctly (the screen session with that name is created).
I can add that I'm already using ParallelSSH library to execute some other tasks, hence I have no problem with set up of hosts or client, it always worked good.
Can someone help me please? I suspect it has something to do with process handling, but I'm not so good with it.
Thank you all
Related
I have a linux system server using ssh to connect. Now I have a python script.I want it run always.I using this commond
ubuntu:~$ nohup python3 -u ~/test/main.py > test.outs 2>&1 &
but I exit ssh connect,That python script exit at the same time.
What should I do?
You could run the script regularly / check it is running with a cronjob and this would also allow you to run the script at system start-up so it would keep running in the event of a reboot.
There are a few suggestions here;
https://superuser.com/questions/448445/run-bash-script-in-background-and-exit-terminal
Although this also suggests that nohub should stop the child process being killed when you exit the session. How are you aware that the script stops running upon exit?
There are multiple ways to do this and they depend on your use-case.
One way to do it is to install 'screen' on your server.
sudo apt-get install screen
Now, whenever you connect to your server with ssh, you can type 'screen' and then start your code there. (first time you do this, you get an explanation on screen, press space to skip it)
With this code executing, you type Ctrl + A and then Ctrl + D. This 'detaches' the screen. You can now disconnect and this 'screen' will keep existing and the code will still run.
When reconnecting, you might want to go back to this screen. Type
screen -ls
to get an overview of screens that you have running. (in this case, there'll only be one) They can be identified by the 5 numbers they all start with. So go back to this screen by typing
screen -r XXXXX
and now you're back. More information on this here:
https://www.howtogeek.com/662422/how-to-use-linuxs-screen-command/
Again, I don't know what your script does so it might not be the best solution.
You may use screen
After installation, you can start screen by sending screen to the console, and then run your script. When you type CTRL+A+D, the screen will disappear and you can exit your ssh connection.
If you re-open the screen you opened before, just type screen -r
If you have multiple screen instances, the app will show you numbers of screen when you type screen -r. You just need to find the id of the screen and type screen -r id i.e. screen -r 2643
So, I have a python script which outputs some data into terminal from time to time. Im trying to run in on the Ubuntu VPS even after I close the SSH connection and still keep the logs somewhere.
Im saving the logs by using:
python3 my_script.py >>file.txt
and it works perfect, however when I try to run this process using
nohup python3 my_script.py >>file.txt &
so it runs in the background and after the ssh connection is closed it seems to save only the first log outputted from my_script.py. I've also tried running this in crontab but the result is similar - only the first log is saved.
Any tips? What am I doing wrong?
I could not understand what you mean "the first log". Maybe the first line of logs?
To run something in the background when SSH connection is closed, I prefer Linux screen, a terminal simulation tool that help you run your command in a sub-process. With it, you could choose to view your output any time in the foreground, or leave your process run in the background.
Usage (short)
screen is not included in most Linux distributions. Install it (Ubuntu):
$ sudo apt-get install screen
Run your script in the foreground:
$ screen python3 my_script.py
You'll see it running. Now detach from this screen: Press keys Ctrl-A followed by Ctrl-D. You'll be back to your shell where you run previous screen command. If you need to switch back to the running context, use screen -r command.
This tool supports multiple parallel running process too.
Something weird
I've tried to redirect stdout or stderr to a file with > or >> symbol. It turned out in failure. I am not an expert of this either, and maybe you need to see its manual page. However, I tend to directly write to a file in Python scripts, with some essential output lines on the console.
Not able to send commands to shell I logged into
Originally, I wrote a Python script. It was able to send commands like
subprocess.run(['kubectl', 'config', 'get-context'], shell=True)
but when it came time to get to the child shell, in this case bash, the command wouldn't run until I exited that shell and it would say things like it couldn't find the command.
I then tried to do it with the module "sh," but was also unsuccessful
I thought maybe using Python was problem and also realized my ultimate goal was to use a different shell (cypher-shell) and so skipped immediately to that with bash as the parent shell. In there I have a line that is sometimes successful, sometimes not
kubectl run -it --rm cypher-shell --image=gcr.io/cloud-marketplace/neo4j-public/causal-cluster-k8s:3.4 --restart=Never --namespace=default --command -- ./bin/cypher-shell -u neo4j -p "password" -a "domain.name"
But even when it successfully logs in it, it just hangs until I manually exit and then it runs the next commands
Note: I saw this and so, perhaps, it's not a child shell? Run shell command from child shell
I can't say I know exactly what you are doing, but if I understand your objective correctly you want the Python program to continue to log while the script continues to run? The problem is that the logger continues to run and holds up your program. The way I would deal with that would be to run the logger as a background process.
With bash, that would be ./script.sh & which would make it run without holding the rest of the program back from running.
Hopefully that may give you an idea! Good luck.
I've created a script for my school project that works with data. I'm quite new to working remotely on a server, so this might seem like a dumb question, but how do I execute my script named
stats.py
so that it continues executing even after I log off PuTTy? The script file is located on the server. It has to work with a lot of data, so I don't want to just try something and then few days later find out that it has exited right after I logged off.
Thank you for any help!
There are many ways you can run a python program after you disconnect from an SSH session.
1) Tmux or Screen
Tmux is a "terminal multiplexer" which enables a number of terminals to be accessed by a single one.
You start by sshing as you do, run it by typing tmux and executing it. Once you are done you can disconnect from putty and when you login back you can relog to the tmux session you left
Screen also does that you just type screen instead of tmux
2) nohup
"nohup is a POSIX command to ignore the HUP signal. The HUP signal is, by convention, the way a terminal warns dependent processes of logout."
You can run it by typing nohup <pythonprogram> &
I’m writing a web app that uses Selenium to screen-scrape another website. This screen-scraping only happens once a day, so I’d rather not leave Selenium and Xvfb running all the time.
I’m trying to figure out how to start Xvfb and Selenium from Python, and then stop them once the screen-scraping’s done.
If I was doing it manually, I’d start them at the command line, and hit CTRL C to stop them. I’m trying to do the same thing from Python.
I seem to be able to successfully start Xvfb like this:
xvfb = Popen('Xvfb :99 -nolisten tcp', shell=True)
But when I’ve tried to terminate it:
xvfb.terminate()
and then tried to start it again (by repeating my initial command), it tells me it’s already running.
I don't know why you want to run Xvfb as root. Your usual X server only needs to run as root (on many but not all unices) only so that it can access the video hardware; that's not an issue for Xvfb by definition.
tempdir = tempfile.mkdtemp()
xvfb = subprocess.Popen(['Xvfb', ':99', '-nolisten', 'tcp', '-fbdir', tempdir])
When you terminate the X server, you may see a zombie process. This is in fact not a process (it's dead), just an entry in the process table that goes away when the parent process either reads the child's exit status or itself dies. Zombies are mostly harmless, but it's cleaner to call wait to read the exit status.
xvfb.terminate()
# At this point, `ps -C Xvfb` may still show a running process
# (because signal delivery is asynchronous) or a zombie.
xvfb.wait()
# Now the child is dead and reaped (assuming it didn't catch SIGTERM).
I assume you can parametrize your system to allow any user to launch Xvfb as explained here solving all your problems
EDIT
the correct command line is
sudo chmod u+s `which Xvfb`