We Keep Coding

python don't extract zipfile when ran through crontab

#!/usr/bin/python
import requests, zipfile, StringIO, sys
extractDir = "myfolder"
zip_file_url = "download url"
response = requests.get(zip_file_url)
zipDocument = zipfile.ZipFile(StringIO.StringIO(response.content))
zipinfos = zipDocument.infolist()
for zipinfo in zipinfos:
extrat = zipDocument.extract(zipinfo,path=extractDir)
System configuration
Ubuntu OS 16.04
Python 2.7.12
$ python extract.py
when I run the code on Terminal with above command, it works properly and create the folder and extract the file into it.
Similarly, when I create a cron job using sodu rights the code executes but don't create any folder or extracts the files.
crontab command:-
40 10 * * * /usr/bin/sudo /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
also tried
40 10 * * * /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
Notes :
I check the syslog, it says the cron is running successfully
The above code gives no errors
also made the python program executable by chmod +x filename.py
Please help where am I going wrong.

Oups, there is nothing really wrong in running a Python script in crontab, but many bad things can happen because the environment is not the one you are used to.
When you type in an interactive shell python directory.py, the PATH and all required PYTHON environment variable have been set as part of login and interactive shell initialization, and the current directory is your home directory by default or anywhere you currently are.
When the same command is run from crontab, the current directory is not specified (but may not be what you expect), PATH is only /bin:/usr/bin and python environment variables are not set. That means that you will have to tweak environment variables in crontab file until you get a correct Python environment, and set the current directory.

I had a very similar problem and it turned out cron didn’t like importing matplotlib, I ended up having to specify Agg backend. I figured it out by putting log statements after each line to see how far the program got before it crapped out. Of course, my log was empty which tipped me off that it crashed on imports.
TLDR: log each line inside the script

Run python from bash script that is run by another bash script

I currently have a folder structure that will contain a few python scripts which need to be fired from a certain folder but I would like to write a global script that runs each python script via a seperate script in each folder.
-Obtainer
--Persona
---Arthur
----start.sh
--Initialise.sh
-Persona
--Arthur
---lib
----pybot
-----pybot.py
When I run initialise I am aiming to make initialise run "start.sh" Arthur is the bot and there will be more folders with different names and initialise with find and fire each start.sh.
In initialise.sh I have:
#!/bin/bash
. ./Persona/Arthur/start.sh
In start.sh I have:
#!/bin/bash
python ../../../Persona/Arthur/lib/pybot/pybot.py
I get this error:
python: can't open file '../../../Persona/Arthur/lib/pybot/pybot.py': [Errno 2] No such file or directory
However if I run the start.sh itself from its directory it runs fine. This is because I assume it's running it from the proper shell and consequently directory. Is there a way to make the main script run the start.sh in it's own shell like it is being run by itself? The reason why is because the pybot.py saves a bunch of files to where the start script is and because there will be more than one bot I need them to save in each seperate folder.

In the first place, do not source when you mean calling it,
#!/bin/bash
. ./Persona/Arthur/start.sh
Don't do this.
Your script has a number of issue. It won't work because of your current working directory is uncertain. You'd better have your script derive the path to relieve yourself from the hustle of abs paths or relative paths.
The general code could be
script_dir=`dirname "${BASH_SOURCE[0]}"`
then you can use this to derive the path of your target file,
#!/bin/bash
script_dir=`dirname "${BASH_SOURCE[0]}"`
"$script_dir/Persona/Arthur/start.sh"
Your python invocation becomes:
#!/bin/bash
script_dir=`dirname "${BASH_SOURCE[0]}"`
python "$script_dir/../../../Persona/Arthur/lib/pybot/pybot.py"
This should work out properly.
Regarding BASH_SOURCE, check out https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html
If you want the directory of start.sh to be cwd, you should call cd:
#!/bin/bash
script_dir=`dirname "${BASH_SOURCE[0]}"`
cd "$script_dir"
python "$script_dir/../../../Persona/Arthur/lib/pybot/pybot.py"

Trouble with os.chroot()

I tried running the following script and it works fine but I am not getting the expected result:
import os,sys
os.system("mount /dev/sdb3 /mnt")
os.system("lsblk")
os.system("mount --bind /proc /mnt/proc")
os.system("mount --bind /home /mnt/home")
os.system("mount --bind /dev /mnt/dev")
os.system("mount --bind /sys /mnt/sys")
os.chroot("/mnt")
os.system("pwd")
os.system("lsblk")
This is because when I do pwd after os.chroot("/mnt"), I am not inside the "/mnt" directory. However, when I run the following code directly into Linux terminal with shell commands(without using a python script), I am getting my expected result.
Now my question is, why can't I go into the /mnt directory using os.chroot("/mnt"). How can I achieve this using os.chroot(using python script) ?

Python's os.chroot is a raw wrapper around the chroot(2) system call. As stated in the system call documentation:
This call does not change the current working directory, so that after the call '.' can be outside the tree rooted at '/'. In particular, the superuser can escape from a "chroot jail" by doing:
mkdir foo; chroot foo; cd ..
This call does not close open file descriptors, and such file descriptors may allow access to files outside the chroot tree.
You need to explicitly change the current working directory yourself, e.g. os.chdir('/') to move into the new root.
You're seeing different behavior than when running the commands from a shell because the chroot(1) executable explicitly changes the current working directory to the new root (source).

The method chroot() changes the root directory of the current process
to the given path.To use this method, you would need super user
privilege.
Source: http://www.tutorialspoint.com/python/os_chroot.htm
Could it be the root-cause?
As a workaround, you could call os.system("cd /mnt; mycommand")

Why can't bash see my files?

I have a simple bash script that allows cron to execute a series of Python scripts in a virtualenv. The script keeps raising No such file or directory errors.
~/nightly.sh works fine:
#!/bin/bash
source virt_env/myproject/bin/activate
cd virt_env/myproject/main
python script1.py
python script2.py
I want to keep everything in ~/virt_env/myproject/main/ to simplify deployment. I thought I could call bash virt_env/myproject/main/nightly.sh on this:
#!/bin/bash
MAINDIR=`dirname $0`
cd $MAINDIR
source ../bin/activate
python script1.py
python script2.py
but I get the No such file or directory. If I manually cd to ~/virt_env/myproject/main/, then I can run the main commands no problem. Clearly I'm missing something about how dirname and cd work in this context.
How can I point bash at the right place?
Solution
As proposed in the accepted answer, it's best to avoid calling cd from within the script and use an explicit path variable instead. Here's the working version of virt_env/myproject/main/nightly.sh:
#!/bin/bash
MAINDIR=`dirname $0`
echo "The main directory is" $MAINDIR
# Activate virtual environment
source $MAINDIR/../bin/activate
# Run Python scripts
python $MAINDIR/python1.py
python $MAINDIR/python2.py
Because the Python scripts are now called from an arbitrary path, I needed to update the python scripts to be smarter about path awareness as well.
This code fails because os.path.basename omits path information:
# works when called with "python python1.py"
# fails when called with "python $MAINDIR/python1.py"
CONFIG_FILE = os.path.basename(__file__)[:-3] + ".config"
f = open(CONFIG_FILE,"r")
Updating it to use os.path.abspath fixes the problem:
# works regardless of how it is called
CONFIG_FILE = os.path.abspath(__file__)[:-3] + ".config"
f = open(CONFIG_FILE,"r")

Perhaps it would simply be better to eliminate the 'cd' command. Invoke everything from a full path specification. In your example add $MAINDIR/ to the executables.
Your bash script can then be in any directory where the executables are reachable. You are not exposed to the problems of what happens when cd fails.
Example:
cd yourdir
rm -f yourglob # oops things got removed from where you started if yourdir did not exist.

Two things:
Are you sure you know what the dirname command is doing? It's going to remove the top-level directory as well as the leading slash on whatever you call it on. I would make absolutely sure that the output of dirname is exactly what you think it is.
For example, /home/user/ will output /home.
You're using ~, which references the $HOME variable in your environment. You didn't mention where the cron is listed, but make sure it's not being run as a different user. Root's ~ and your ~ will be two completely different directories.
That's all I can think of. I hope that helps!

Add echo $MAINDIR after
MAINDIR=`dirname $0`
cd $MAINDIR
So you can see, if the contents of MAINDIR is correct.
Also you can run sh with -x or add set -x to the beginning of the script to see what happens.

Crontab Python Script Execution (Can't find settings config file)

My Crontab -l
# m h dom mon dow command
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
00 8,20 * * * python /home/tomi/amaer/controller.py >>/tmp/out.txt 2>&1
My controller.py has config file settings.cfg also it uses other script in the folder it's located (I chmoded only controller.py)
The error
1;31mIOError^[[0m: [Errno 2] No such file or directory: 'settings.cfg'
I have no idea how to fix this? Please help me?
Edit: The part that read the config file
def main():
config=ConfigParser.ConfigParser()
config.readfp(open("settings.cfg"),"r")

As I initially wrote in my comment, this is because you are using relative path to the current working directory. However, that is not going to be the same when running all this via the cron executable rather than the python interpreter directly via the shebang.
Your current code would look for the "settings.cfg" in the current working directory which is where the cron executable resides, and not your script. Hence, you would need to change your code logic to using absolute paths by the help of the "os" built-in standard module.
Try to following line:
import os
...
def main():
config = ConfigParser.ConfigParser()
scriptDirectory = os.path.dirname(os.path.realpath(__file__))
settingsFilePath = os.path.join(scriptDirectory, "settings.cfg")
config.readfp(open(settingsFilePath,"r"))
This will get your the path of your script and then appends the "settings.cfg" with the appropriate dir separator for your operating system which is Linux in this particular case.
If the location of the config file changes any time in the future, you could use the argparse module for processing a command line argument to handle the config location properly, or even without it simply just using the first argument after the script name like sys.argv[1].

Your code is looking for settings.cfg in its current working directory.
This working directory will not be the same when cron executes the job, hence the error
You have two "easy" solutions:
Use an absolute path to the config file in your script (/home/tomi/amaer/config.cfg)
CD to the appropriate directory first in your crontab (cd /home/tomi/amaer/ && python /home/tomi/amaer/controller.py)
The "right" solution, though, would be to pass your script a parameter (or environment variable) that tells it where to look for the config file.
It's not exactly good practice to assume your config file will always be lying just next to your script.
You might want to have alook at this question: https://unix.stackexchange.com/questions/38951/what-is-the-working-directory-when-cron-executes-a-job