We Keep Coding

coreutils timeout in a bash script not transparent for the application

I have an issue with executing application via /usr/bin/timeout in a bash script.
In this specific case this is a simple python fabric script (fabric version 1.14)
In order to install this version of fabric library run: pip install "fabric<2"
There is no reproduction with new fabric 2.x.
Shell script causing issue:
[root#testhost:~ ] $ cat testNOK.sh
#!/bin/bash
timeout 10 ./test.py
echo "RETCODE=$?"
[root#testhost:~ ] $ ./testNOK.sh
[localhost] run: echo Hello!
RETCODE=124
[root#testhost:~ ] $
Similar script (without timeout) working fine
[root#testhost:~ ] $ cat testOK.sh
#!/bin/bash
./test.py
echo "RETCODE=$?"
[root#testhost:~ ] $ ./testOK.sh
[localhost] run: echo Hello!
[localhost] out: Hello!
[localhost] out:
RETCODE=0
[root#testhost:~ ] $
Manual execution from bash commandline with timeout working fine:
[root#testhost:~ ] $ timeout 10 ./test.py && echo "RETCODE=$?"
[localhost] run: echo Hello!
[localhost] out: Hello!
[localhost] out:
RETCODE=0
[root#testhost:~ ] $
Python2.7 test.py script
[root#testhost:~ ] $ cat test.py
#!/usr/bin/python
from fabric.api import run, settings
with settings(host_string='localhost', user='root', password='XXXXX'):
run('echo Hello!')
[root#testhost:~ ] $
I have observed the same behavior on different Linux distributions.
Now the question is why application executed via timeout within bash script behaves in a different way and what would be the best solution to this issue?

You need to invoke timeout with the --foreground option:
timeout --foreground ./test.py
This is only required if the timeout command is not executed from an interactive shell (that is, if it's executed from a script file).
Quoting from the timeout info page:
‘--foreground’
Don’t create a separate background program group, so that the
managed COMMAND can use the foreground TTY normally. This is
needed to support timing out commands not started directly from an
interactive shell, in two situations.
1. COMMAND is interactive and needs to read from the terminal for
example
2. the user wants to support sending signals directly to COMMAND
from the terminal (like Ctrl-C for example)
What's actually going on in this case is that fabric (or something invokes) is calling tcsetattr to turn terminal echo off. I don't know why, but I suppose it has something to do with the process used to (not) collect the user password. (I just saw it in an strace; I made no attempt to find the call.) Attempting to change tty configuration from a background process will cause the process to block until it regains control of the tty, and that's what's happening.
It doesn't happen when timeout is not used because bash doesn't create a background program group. I suppose that fabric 2 avoids the call to tcsetattr.
You could probably also avoid the issue by avoiding password-based SSH authentication but I didn't try that.
You can also avoid the problem by redirecting stdin to /dev/null (either in the timeout command or in the invocation of the shell script.) If you don't need to forward stdin to the remote command (and you probably don't), that might also be useful.

You Can Use time out without using bash Just by using the time model in python
import time
time.sleep(5)
#change the 5 by the seconds that you need to set a timeout

Run python script from rc.local does not execute

I want to run a python script on boot of ubuntu 14.04LTS.
My rc.local file is as follows:
sudo /home/hduser/morey/zookeeper-3.3.6/bin/zkServer.sh start
echo "test" > /home/hduser/test3
sudo /home/hduser/morey/kafka/bin/kafka-server-start.sh /home/hduser/morey/kafka/config/server.properties &
echo "test" > /home/hduser/test1
/usr/bin/python /home/hduser/morey/kafka/automate.py &
echo "test" > /home/hduser/test2
exit 0
everything except my python script is working fine even the echo statement after running the python script, but the python script doesnt seem to run.
My python script is as follows
import sys
from subprocess import Popen, PIPE, STDOUT
cmd = ["sudo", "./sbt", "project java-examples", "run"]
proc = Popen(cmd, shell=False, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
proc.communicate(input='1\n')
proc.stdin.close()
which works perfectly fine if executed individually.
I went through the following questions , link
I did a lot of research but couldn't find a solution
Edit : echo statements are for testing purpose only, and the second actual command (not considering the echo statements) is starting a server which keeps on running, and even the python script starts a listener which runs on an infinite loop, if this is any help

The Python script tries to launch ./sbt. Are you sure of what if the current directory when rc.local runs? The rule is always use absolute paths in system scripts

Do not run the Python script in background, run it in foreground. Do not exit from its parent script. Better call another script from "rc.local" that does all the job of "echo" and script launching.
Run that script from "rc.local"; not in background (no &).
You do not need "sudo" as "rc.local" is run as root.

If you want to run python script at system boot there is an alternate solution which i have used.
1:Create sh file like sample.sh and copy paste following content
#!/bin/bash
clear
python yourscript.py
2:Now add a cron job at reboot.If you are using linux you can use as following
a:Run crontab -e(Install sudo apt-get install cron)
b:#reboot /full path to sh file > /home/path/error.log 2>&1
And restart your device

Python fabric unable to start process

I'm using python fabric to deploy binaries to an ec2 server and am attempting to run them in background (a subshell).
All the fabric commands for performing local actions, putting files, and executing remote commands w/o elevated privileges work fine. The issue I run into is when I attempt to run the binary.
with cd("deploy"):
run('mkdir log')
sudo('iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080', user="root")
result = sudo('./dbserver &', user="root") # <---- This line
print result
if result.failed:
print "Running dbserver failed"
else:
print "DBServer now running server" # this gets printed despite the binary not running
After I login to the server and ps aux | grep dbserver nothing shows up. How can I get fabric to execute the binary? The same command ./dbserver & executed from the shell does exactly what I want it to. Thanks.

This is likey reated to TTY issues, and/or that you're attempting to background a process.
Both of these are discussed in the FAQ under these two headings:
http://www.fabfile.org/faq.html#init-scripts-don-t-work
http://www.fabfile.org/faq.html#why-can-t-i-run-programs-in-the-background-with-it-makes-fabric-hang
Try making the sudo like this:
sudo('nohup ./dbserver &', user=root, pty=False)

python subprocess output on nohup

Trying to monitor the available physical disc space of a remote machine using a python script, which executes the df -h . command using subprocess.popen.
import subprocess
import time
command = 'ssh remoteserver "df -h ."'
while True:
proc = subprocess.Popen(command,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output,err=proc.communicate()
print output
print err
time.sleep(60)
The script runs fine and prints the output to the terminal when run from command line
$> python2.7 script.py
Filesystem Size Used Avail Use% Mounted on
remoteserver:/home/user
555G 447G 109G 81% /home
The scripts does not produce any output or seems to be blocking when the script is started with nohup command.
$> nohup python2.7 script.py &
Would like the script to work and fetch the disc space of remote machine using the above script when started in nohup.

I'm not 100% sure of the underlying issue here, but when you invoke NOHUP in the shell, it's disconnected some of the STDIN/STDOUT from the terminal process, which I suspect it causing some of this interactions you're seeing.
Given that you're doing this from a remote machine, I'd actually recommend you look at using something like Fabric as a library to do what you're after. It's pretty straightforward, and does most of the handling of terminal sessions as well as closing things down nicely for you when you're complete.
something like:
from fabric import api
from fabric.api import env
import fabric
env.host_string = '%s#%s' % (username, remote_host)
env.disable_known_hosts = True
env.password = password
fabric.state.output['stdout'] = False
fabric.state.output['stderr'] = False
results = api.run('df -h')

You might try sending stdin=subprocess.PIPE to the subprocess command, then calling proc.stdin.close() on the next line, before the communicate() call. Or you can try changing the command to 'ssh remoteserver "df -h ." </dev/null'. Others report using FNULL = open(os.devnull, 'r') and passing in FNULL to the stdin= argument, but I'm not sure if you need to call FNULL.close() after or not.
SSH is most likely waiting for input for some reason when it is run from nohup. Perhaps it is unable to authenticate in the nohup environment and is asking for password input?
To make sure SSH is not waiting for input, try adding -o "BatchMode yes" to the ssh command and see if there are some clues in the output/error from the subprocess communicate call.

How to execute remote pysftp commands under a specific shell

I use python module pysftp to connect to remote server. Below you can see python code :
import pysftp
import sys
import sqr_common
srv = pysftp.Connection(host="xxxxxx", username="xxxx",
password="xxxxx")
command = "/usr/bin/bash"
command2="APSHOME=/all/aps/msc_2012; export APSHOME; "
srv.execute(command)
srv.execute(command2)
srv.close()
Problem is that command /usr/bin/bash is an infinite process , so my script will never be executed. Can anyone help me how to choose shell on remote server for example bash and execute command in bash on remote server?? Is there any pysftp function that allows me chosing shell??

try this
/usr/bin/bash -c "APSHOME=/all/aps/msc_2012; export APSHOME; "

This problem is not specific to Python, but more like how to execute commands under specific shell.
If you need to run only single command you can run using bash -c switch
bash -c "echo 123"
You can run multiple commands ; separated
bash -c "echo 123 ; echo 246"
If you need to many commands under a specific shell, remotely create a shell script file (.bash file) an execute it
bash myscript.bash