I would like to submit Python scripts directly on a computing cluster using SGE. I have a test script:
#!/usr/bin/python
print "hello"
I have run chmod +x hello.py such that on the front end:
./hello.py
>> hello
However, when I run qsub -o out.log hello.py, the following files are generated:
out.log:
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
hello.py.e707589:
print: Command not found.
It seems to me that qsub is nonetheless trying to interpret the script as a bash script, even despite the header. How can I fix this?
Related
I'm trying to execute a sh file from script python.
My script python
os.system('sh run.sh')
My sh file
echo 'The house is blue' | /opt/palavras/por.pl > output.txt
Error:
sh: 0: Cant' open run.sh
How can I fix it?
Make sure that your bash script has the right permissions (i.e it is executable). In a terminal run:
chmod +x run.sh
and then try (assuming that run.sh is in the same directory as your python script)
import os
os.system('./run.sh')
I do not believe this would run from the terminal either because you must run the file in order. Try:
os.system('sh chmod +x run.sh|./run.sh')
instead.
See: https://askubuntu.com/questions/38661/how-do-i-run-sh-files for details on running sh files and how to use os.system() in python for running an shell order for the use of | running in an order in a shell.
I have a bash script that I'm using to execute a python file with a specific version of Python (3.6). The Bash script is currently located on my desktop (/home/pi/Desktop/go.sh)
#!/bin/bash
python3.6 /home/pi/scriptDir/myScript.py
Here is my crontab entry, when I do crontab -l (note, I've deleted my other jobs)
* * * * * bash /home/pi/Desktop/go.sh # JOB_ID_3
When I run this file using the command line or from the GUI it executes properly.
When I have crontab do it, nothing happens.
Both my python file and the bash script are executable. chmod +x
Is there something obvious I'm missing?
**my python script does depend on other files in the same script directory, could that be the issue?
Here's what got it working for me. I was not using the full path to my python install. Unless you log the bash file there's no indication that you have an issue.
This is my bash file now. echo's were just to determine that I was indeed running the bash file.
#!/bin/bash
echo started
/home/pi/Python-3.6.0/python home/pi/myScriptFolder/myScript.py
echo finished
To break down the line executing the script:
/home/pi/Python-3.6.0/python - is where python 3.6.0 is installed on my Pi, it could be different for you. home/pi/myScriptFolder/myScript.py is the script I want to run.
And here is my cron statement:
*/15 * * * * bash /home/pi/Desktop/go.sh > /home/pi/Desktop/clog.log 2>&1 -q -f
Breaking down this line:
*/15 * * * * is the cron time, in this case every 15 mins. bash /home/pi/Desktop/go.sh specifies to run a bash file and the directory of that file. > /home/pi/Desktop/clog.log 2>&1 -q -f this last section creates a log file named clog.log so you can see what's going on.
The key here was not just logging the go.sh bash file execution, but adding the 2>&1 -q -f to the end of the log request. Before I did that there was no indication of a problem, afterwards I was getting the python files error returned into the log file.
Cron jobs are surprisingly tricky. Aside from working directory ( which you'd have to set somewhere), you also need to handle environment setup ( $PATH, for example).
Start by redirecting your shell script's standard output and error to a log file so you can get feedback.
From this stackoverflow thread https://stackoverflow.com/questions/4443...mmand-line, I have extracted this command line:
gimp-console -idf --batch-interpreter python-fu-eval -b "import sys;sys.path=['.']+sys.path;import batch;batch.run('./images')" -b "pdb.gimp_quit(1)"
It works perfectly well.
Now, I would like to run this command from a Python script, usually I use subprocess.Popen but this time it does not work and I get this message:
"batch command experienced an execution error"
How can I launch the GIMP command line from a Python script?
One easy way to resolve this is to just put your GIMP startup script into a bash script, say startgimp.sh
#!/bin/bash
#set your path to GIMP or cd into the folder where you installed GIMP
gimp-console -idf --batch-interpreter python-fu-eval -b "import sys;sys.path=['.']+sys.path;import batch;batch.run('./images')" -b "pdb.gimp_quit(1)"
then from Python simply call the bash script like so
import subprocess
subprocess.call(["bash","/path/to/your/script/startgimp.sh"])
If you are able to make the .sh script executable, e.g. chmod +x startgimp.sh then you can skip the bash part and just do subprocess.call("/path/to/your/script/startgimp.sh")
Some caveats
This is assuming you're on a UNIX based system
I used subprocess.call so this WILL block while waiting for GIMP to complete. Use Popen like you've used if you don't want this
I don't have GIMP to try this out, but you could also try splitting your GIMP command into elements in the list and pass it to subprocess and see if that works.
e.g. subprocess.call(["gimp-console","-idf","--batch-interpreter","python-fu-eval" and so on)
I want to run a python script on boot of ubuntu 14.04LTS.
My rc.local file is as follows:
sudo /home/hduser/morey/zookeeper-3.3.6/bin/zkServer.sh start
echo "test" > /home/hduser/test3
sudo /home/hduser/morey/kafka/bin/kafka-server-start.sh /home/hduser/morey/kafka/config/server.properties &
echo "test" > /home/hduser/test1
/usr/bin/python /home/hduser/morey/kafka/automate.py &
echo "test" > /home/hduser/test2
exit 0
everything except my python script is working fine even the echo statement after running the python script, but the python script doesnt seem to run.
My python script is as follows
import sys
from subprocess import Popen, PIPE, STDOUT
cmd = ["sudo", "./sbt", "project java-examples", "run"]
proc = Popen(cmd, shell=False, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
proc.communicate(input='1\n')
proc.stdin.close()
which works perfectly fine if executed individually.
I went through the following questions , link
I did a lot of research but couldn't find a solution
Edit : echo statements are for testing purpose only, and the second actual command (not considering the echo statements) is starting a server which keeps on running, and even the python script starts a listener which runs on an infinite loop, if this is any help
The Python script tries to launch ./sbt. Are you sure of what if the current directory when rc.local runs? The rule is always use absolute paths in system scripts
Do not run the Python script in background, run it in foreground. Do not exit from its parent script. Better call another script from "rc.local" that does all the job of "echo" and script launching.
Run that script from "rc.local"; not in background (no &).
You do not need "sudo" as "rc.local" is run as root.
If you want to run python script at system boot there is an alternate solution which i have used.
1:Create sh file like sample.sh and copy paste following content
#!/bin/bash
clear
python yourscript.py
2:Now add a cron job at reboot.If you are using linux you can use as following
a:Run crontab -e(Install sudo apt-get install cron)
b:#reboot /full path to sh file > /home/path/error.log 2>&1
And restart your device
I am building an interactive installer using a nifty command line:
curl -L http://install.example.com | bash
The bash script then rapidly delegates to a python script:
# file: install.sh
[...]
echo "-=- Welcome -=-"
[...]
/usr/bin/env python3 deploy_p3k.py
And the python script itself prompts the user for input:
# file: deploy_py3k.py
[...]
input('====> Confirm or enter installation directory [/srv/vhosts/project]: ')
[...]
input('====> Confirm installation [y/n]: ')
[...]
PROBLEM: Because the python script is ran from a bash script itself being piped from curl, when the prompt comes up, it is automatically "skipped" and everything ends like so:
$ curl -L http://install.example.com | bash
-=- Welcome ! -=-
We have detected you have python3 installed.
====> Confirm or enter installation directory [/srv/vhosts/project]: ====> Confirm installation [y/n]: Installation aborted.
As you can see, the script doesn't wait for user input, because of the pipe which ties the input to the curl output. Thus, we have the following problem:
curl [STDOUT]=>[STDIN] BASH (which executes python script)
= the [STDIN] of the python script is the [STDOUT] of curl (which contains at a EOF) !
How can I keep this very useful and short command line (curl -L http://install.example.com | bash) and still be able to prompt the user for input ? I should somehow detach the stdin of python from curl but I didn't find how to do it.
Thanks very much for your help !
Things I have also tried:
Starting the python script in a subshell: $(/usr/bin/env python3 deploy.py)
You can always redirect standard input from the controlling tty, assuming there is one:
/usr/bin/env python3 deploy_p3k.py < /dev/tty
or
/usr/bin/env python3 deploy_p3k.py <&1