We Keep Coding

Python scripts couldn't work with crontab

I'm new to stackoverflow and really hope to get answers to my question about crontab.
I'm using MacbookAir in BigSur(don't know if it affects)And my crontab just couldn't run python scripts below.
f = open("test_crontab.txt",'w+')
f.write("Success")
f.close()
But I could run this script with Spyder.
I could edit crontab with crontab -e, and my code in vi crontab was
13 01 * * * /bin/zsh /Users/myusername/Documents/Lynn\'s\ Python\ work/shopee/test.py >>/Users/myusername/Desktop/out.log 2>>/Users/tzulingkuo/Desktop/err.log
But it failed to generate txt file.
The messages in err.log were like below:
/Users/*myusername*/Documents/Lynn's Python work/shopee/test.py:5: missing end of string
/Users/*myusername*/Documents/Lynn's Python work/shopee/test.py:6: unknown username ''
I've searched stackoverflow for two nights and couldn't find the solution. I major in finance and I have no friends studying computer science.
Could anyone help me 🥺
Any help is reaaaaally appreciated!

Since that's a python script, you need to run it in python, not /bin/zsh. zsh is trying to interpret it as a shell script, but the syntax is all wrong, so you get errors.
The simplest way to fix this to replace /bin/zsh in your crontab entry with /usr/bin/python (assuming you're using the built-in Python version 2; if you have installed Python version 3 and want to use that, run which python at the command line to find its path).
But it's generally better to add a shebang line to your python script, and let that control how it's run. Add this as the very first line of the script:
#!/usr/bin/python
(Again, if you want to use Python 3, replace the path part with the path for the Python 3 interpreter.) Then make the script executable with chmod +x /Users/myusername/Documents/Lynn\'s\ Python\ work/shopee/test.py, and just remove the /bin/zsh part from the crontab entry.
BTW, you're probably going to have trouble with paths. By default, cron jobs run with their working directory set to your home directory, so when the script opens test_crontab.txt, that's going to be interpreted as /Users/myusername/test_crontab.txt. If you want it down in the Documents/Lynn's Python work/shopee directory, you'll have to specify that explicitly.
You may also run into trouble with missing environment variables. If you're setting any environment variables in your shell initialization scripts that configure Python, add libraries, etc, those won't be set up in the cron job's environment.

Unexpected behaviour when running python from crontab

I have a python script (script A) that runs multiple sub-processes (running another python script - script B). When I run the script manually, it works fine, but when it is being run using crontab it behaves differently.
To be more specific, my script is limited to 100TB of quota, so script B runs over a tree and produce a text file of size-per-dir. This script is being run multiple times on several dir-trees as a subprocess, and the results are being further analyzed by script A.
When I run manually, both scripts run fine. But when I run it using Cron, script A runs fine but script B produce odd resutls.
Any ideas of why using Cron would affect a python script?

Make sure the crontab you are using is with full path of the .sh file.
And in beginning of your script try to add the following:
. /home/gemapp/.bash_profile
to load all needed items in bash_profile.

The default directory for any scheduled job is user's home dir, not the location of the shell script/program itself.
To mimic the crontab job, you must simulate running your script from user's home dir using "cd ~" command. Run your job using the exact same syntax as in your "crontab -l" output to simulate the error.
In other word, you must simulate running your schedule job from home dir. Once it worked, then you put it into the crontab.

Running python file from inside a bash script through cron

I have created a bash script which first activates a python virtual environment, and then runs a python file. When I execute the bash script manually, the python file runs as intended. Bash script code:
sample.sh
#!/usr/bin/env bash
source ./project/bin/activate
python3 /home/abc/project/server/sample.py
However, when I try to run this bash script using cron, the python file does not execute.
cron:
16 12 * * * /home/abc/sample.sh > /home/abc/bulkcat.log 2>&1
When this cron triggers at the specified time, the python file inside my bash script does not run and the log file is empty.
What seems to be wrong with my code?

It might well be the relative path you're using in the source command. Cron will run your script from a different directory, so
source ./project/bin/activate
will likely not be a valid path.
Try
source /home/abc/project/bin/activate
... guessed path based on the full path in your python3 ... line.

Cron writes logs and you can find the error which occured when it tried to execute the task - an answer to this question mentions usual locations where to look for these logs.
Most common issues are:
cron is using sh and not bash ignoring shebang in your script - you can try configuring your cron job like 6 12 * * * /bin/bash /home/abc/sample.sh > /home/abc/bulkcat.log 2>&1
the script not having permission to be executable set - this can be fixed by running chmod 700 /home/abc/sample.sh or chmod 755 /home/abc/sample.sh - the latter should be used only if you want to allow other users to read and execute your script
as mentioned already in another answer, always use absolute paths in cron job as cron might execute your script from other directory than you expect - I'm also using wrapper bash script in such scenarios - I give absolute path to the script in cron job and the first thing the bash script does is cd /desired/work/directory

Cron job can't run bash script (that shells python script requiring xsession) that runs perfectly well from console

I am using dryscrape in a python script. The python script is called in a bash script, which is run by cron. For those who may not be aware, dryscrape is a headless browser (use QtWebkit in the background - so requires an xsession).
Here are the main points concerning the issue I'm having
When I run the python script from the command line, it works
When I run the bash script from the command line, it works too
I figured out that this may have something to do with different environments between my command prompt and when the cron job is running, so I modified my bash script to source my .profile as follows:
#/bin/bash
. /full/path/to/my/home/directory/.profile
python script_to_run.py
This is what my cronjob crontab entry looks like:
0,55 14-22 * * 1-5 /path/to/script.sh >> $(date "+/path/to/logs/\%Y\%m\%d.mydownload.log" )
By the way, I know that the job is being run (I can see entries in /var/log/syslog, and the script also writes to a log file - which is where I get the error message below):
In all cases, I got the following error message:
Could not connect to X server. Try calling dryscrape.start_xvfb()
before creating a session
I have installed the prerequisites, on my machine (obviously - since it runs at the command line). At the moment, I have run out of ideas.
What is causing the script to run fine at the console, and then fail when run by cron?
[[Relevant Details]]
OS: Linux 16.0.4 LTS
bash: version 4.3.46(1)
cron user: myself (i.e. same user at the command prompt)
dryscrape: version 1.0.1

The solution to this was to call the dryscrape.start_xvfb() method before starting the dryscrape session.

Cron user does not have display, so you cannot run any command which requires a display.
You need to modify the python script to do not use any type of display (check carefully, because some python commands, even though they do not open any display , they internally check for this variable).
The best way to test is to ssh into the machine without Display, and check if you can run it from there without erros.

Cron job calling bash script and python scripts

Okay so I have a bash script that simply downloads a web page, and then I use python to pulls some data out of the downloaded page.
So my bash script is along the lines of
#!/bin/bash
html_file="web_page.html"
wget -O /home/michael/Documents/CS288/homework7/web_page.html http://markets.usatoday.com/custom/usatoday-com/html-mktscreener.asp?exchange=13\&screen=1
python hw_7_2.py $html_file
Now, when I just execute this bash script from the command line it runs fine, the wget runs and then my python script executes, however when I set it up as a cron job the wget will run but the python script never executes.
I have not really set up cron jobs so this I think may be the issue. This is basically what my crontab file looks like
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
* * * * * michael /home/michael/Documents/CS288/homework7/usatoday_runner.sh

try to replace the cron line with :
* * * * * michael /home/michael/Documents/CS288/homework7/usatoday_runner.sh > /tmp/why_is_this_failing.log 2>&1
the answer may be in the /tmp/why_is_this_failing.log

It's possible that your script doesn't have some environment variables set. When a cron job runs it doesn't have your normal profile information - it doesn't load your .profile/.bashprofile (simpler path, JAVA_HOME, etc) one possible option is to have the script source your .profile etc.

Cron frequently fails because of $PATH/working directory sorts of problems. You're setting the $PATH, but I wouldn't be surprised if neither your bash script nor your python script work if you aren't in the right directory.
Try using more absolute paths and see if that clears things up. Similarly, try running your cron command yourself from / or someplace and see if it works for you.