I have a python program that uses the ThreadPool for multithreading. The program is one step in a shell script. When I execute the shell script manually on the command line, the entire flow works as expected. However, when I execute the shell script as a cronjob, it appears that the flow goes to the next steps before the python multithreading steps are completely finished.
Inside the python program, I do call AsyncResult.get(timeout) to wait for all the results to come back before moving on.
Run your program via batch(1) (see the output of the command man batch) as well. If that works OK, but the cron version does not, then it is almost certainly a problem with your environment variable setup. To verify that, run printenv from your interactive shell to inspect your environment there. Then do the same thing inside the crontab (you will just need to temporarily set up an extra cron entry for it). Try setting the variables in your shell script before invoking Python.
On the other hand, if it doesn't work via batch(1) either, it could be something to do with the files that your code has open. Try running your shell script with input redirected from /dev/null and output going to a file:
$ /usr/local/bin/myscript </dev/null >|/tmp/outfile.txt 2>&1
Try setting "TERM=xterm" (or whatever env variable you have, figure out by command 'env' on your terminal) in your crontab.
Related
My python script generate a proper command that user need to run in the same console. My scenario is that user is running a script and then as a result see the command that must to run. Is there any way to exit python script and send that command to console, so user do not need to copy/paste?
A solution would be to have your python script (let's call it script.py) just print the command: print('ls -l') and use it in a terminal like so: $(python3 script.py). This makes bash run the output of your script as a command, and would basically run a ls -l in the terminal.
You can even go a step beyond and create an alias in ~/.bashrc so that you no longer need to call the whole line. You can write at the end of the file something like alias printls=$(python3 /path/to/script.py). After starting a new terminal, you can type printls and the script will run.
A drawback of this method is that you have no proper way of handling exceptions or errors in your code, since everything it prints will be run as a command. One way (though ugly) would be to print('echo "An error occured!"') so that the user who runs the command can see that something malfunctioned.
However, I'd suggest going for the "traditional" way and running the command directly from python. Here's a link to how you can achieve this: Calling an external command in Python.
Python can run system commands in new subshells. The proper way of doing this is via the subprocess module, but for simple tasks it's easier to just use os.system. Example Python script (assuming a Unix-like system):
import os
os.system('ls')
I'm trying to run a python script on my raspberrypi using cron.
I did the following:
crontab -e # To edit a crontab job
After the cron file opened, I added the following line:
#reboot /usr/bin/python /home/pi/path/to/file/example.py > /home/pi/cronlogs/mylog.log # JOB_ID_!
If I understand the documentation correctly, this cron job should be executed every time after the system booted up. However in my case, when I reboot the computer, the script will not be executed.
What's strange:
I checked the log file and it's empty, so it seems like everything goes fine
If I run the given command manually (so basically write the following code to the terminal) it executes and works correctly: /usr/bin/python /home/pi/path/to/file/example.py > /home/pi/cronlogs/mylog.log
I guess I missed something really obvious but I can't see it. Please can I ask for any advise how to debug this. Thanks!
The cron definition looks correct; I just checked this on my Pi running Debian stretch and it works OK:
#reboot /usr/bin/python /home/pi/example.py > /home/pi/mylog.log
Some other possible reasons it might not work:
working directory issue (if you're using relative paths)
a long running script (being a scraping script it might take a while to complete) - you can check if it's still running using ps aux | grep python
the script does not output anything (would need some more details about the script)
Just to be sure you catch any errors from the script, redirect stderr to stdout by using 2>&1
I am using dryscrape in a python script. The python script is called in a bash script, which is run by cron. For those who may not be aware, dryscrape is a headless browser (use QtWebkit in the background - so requires an xsession).
Here are the main points concerning the issue I'm having
When I run the python script from the command line, it works
When I run the bash script from the command line, it works too
I figured out that this may have something to do with different environments between my command prompt and when the cron job is running, so I modified my bash script to source my .profile as follows:
#/bin/bash
. /full/path/to/my/home/directory/.profile
python script_to_run.py
This is what my cronjob crontab entry looks like:
0,55 14-22 * * 1-5 /path/to/script.sh >> $(date "+/path/to/logs/\%Y\%m\%d.mydownload.log" )
By the way, I know that the job is being run (I can see entries in /var/log/syslog, and the script also writes to a log file - which is where I get the error message below):
In all cases, I got the following error message:
Could not connect to X server. Try calling dryscrape.start_xvfb()
before creating a session
I have installed the prerequisites, on my machine (obviously - since it runs at the command line). At the moment, I have run out of ideas.
What is causing the script to run fine at the console, and then fail when run by cron?
[[Relevant Details]]
OS: Linux 16.0.4 LTS
bash: version 4.3.46(1)
cron user: myself (i.e. same user at the command prompt)
dryscrape: version 1.0.1
The solution to this was to call the dryscrape.start_xvfb() method before starting the dryscrape session.
Cron user does not have display, so you cannot run any command which requires a display.
You need to modify the python script to do not use any type of display (check carefully, because some python commands, even though they do not open any display , they internally check for this variable).
The best way to test is to ssh into the machine without Display, and check if you can run it from there without erros.
recently, I want to use python script to set environment in linux.This is one line of my code:
p = subprocess.call(['/bin/csh', '-c', "source setup.csh"])
My setup.csh file is below:
add questa10.2b
add ds5-2013.06
setenv MODELSIM modelsim.ini
But when I run my python, it shows that the files have sourced on screen, but it turns out I have to type myself on command line.
How could I solve these problem? Can any one please help me with this?
You're creating a new csh shell as a subprocess and then running your commands inside that shell, which then terminates. The commands do not run in, or affect, the parent shell within which Python is running. When you just run the commands yourself, they affect the current shell.
If you need these settings to persist in your current shell after Python terminates, your best bet in general is to source setup.csh rather than putting it in a Python script. If other child processes of the Python script need your environment variables, you can alter os.environ.
I am writing a python script (Linux) that is adding some shell aliases (writes them to HOME/.bash_aliases).
In order to make an alias available immediately after it was written I should issue the following bash built-in:
source HOME/.bashrc
source is a bash built-in so I cannot just:
os.system(source HOME/.bashrc)
If i try something like:
os.system('/bin/bash -c source HOME/.bashrc')
...will freeze the script (just like is waiting for something).
Any suggestions ?
What you want is not possible. A program (your script) cannot modify the environment of the caller (the shell you run it from).
Another approach which would allow you to do something close is to write it in terms of a bash function, which is run in the same process and can modify the caller. Note that sourcing during runtime can have possible negative side-effects depending on what the user has in their bashrc.
what you are trying to do is impossible. or better: how you are trying to do it is impossible.
your bash command is wrong. bash -s command does not execute command. it just stores the string "command" in the variable $1 and then drops you to the prompt. that is why the python script seems to freeze. what you meant to do is bash -c command.
why do you source .bashrc? would it not be enough to just source .bash_aliases?
even if you got your bash command right, the changes will only take effect in the bash session started from python. once that bash session is closed, and your python script is done, you are back at your original bash session. all changes in the bash session started from python is lost.
everytime you want to change something in the current bash session, you have to do it from inside the current bash session. most of the commands you run from bash (system commands, python scripts, even bash scripts) will spawn another process, and everything you do in that other process will not affect your first bash session.
source is a bash builtin which allows you to execute commands inside the currently running bash session, instead of spawning another process and running the commands there. defining a bash function is another way to execute commands inside the currently running bash session.
see this answer for more information about sourcing and executing.
what you can do to achieve what you want
modify your python script to just do the changes necessary to .bash_aliases.
prepare a bash script to run your python script and then source .bash_aliases.
#i am a bash script, but you have to source me, do not execute me.
modify_bash_aliases.py "$#"
source ~/.bash_aliases
add an alias to your .bashrc to source that script
alias add_alias='source modify_bash_aliases.sh'
now when you type add_alias some_alias in your bash prompt it will be replaced with source modify_bash_aliases.sh and then executed. since source is a bash builtin, the commands inside the script will be executed inside the currently running bash session. the python script will still run in another process, but the subsequent source command will run inside your currently running bash session.
another way
modify your python script to just do the changes necessary to .bash_aliases.
prepare a bash function to run your python script and then source .bash_aliases.
add_alias() {
modify_bash_aliases.py "$#"
source ~/.bash_aliases
}
now you can call the function like this: add_alias some_alias
I had an interesting issue where I needed to source an RC file to get the correct output in my python script.
I eventually used this inside my function to bring over the same variables from the bash file I needed to source. Be sure to have os imported.
with open('overcloudrc') as data:
lines = data.readlines()
for line in lines:
var = line.split(' ')[1].split('=')[0].strip()
val = line.split(' ')[1].split('=')[1].strip()
os.environ[var] = val
Working solution from Can I use an alias to execute a program from a python script :
import subprocess
sp = subprocess.Popen(["/bin/bash", "-i", "-c", "nuke -x scriptpath"])
sp.communicate()