We Keep Coding

How to run bash script at startup without root access?

I have a bash file, it works fine when executed from terminal.
#!/bin/bash
source activate tensorflow_p36
python /home/ec2-user/abc/wsgi.py
Note: tensorflow_p36 being an in-built conda environment, does not require to be called from specific /env/bin directory. It can be activated from any directory. I think it's a feature of Amazon Deep Learning AMIs.
If I run this bash script with sudo it doesnt activate the virtual environment and works in default python environment. The python file can run in that virtual environment only.
I have tried all 3 alternatives (rc.local, .conf file, init.d config)here, also tried to use crontab as suggested here. I have also tried using supervisord to add this bash script as a program.
When the program runs from these methods, I always get the same import errors because it is using default python 3 environment which doesn't have the required dependencies.
I am working on Amazon CentOS (Deep learning AMI). Can someone please suggest a method to run this script every time system restarts?

In the rc.local, instruct root to run it as you:
su --command /path/to/bash/file --login grimlock

You can run it from your personal Crontab.
( crontab -l; printf '#reboot /path/to/bash/file\n' ) | crontab -
If you don't have a crontab there will be an error message from crontab -l but it's harmless.
crontab: no crontab for ec2-user
You just need to do this once, and the job will execute as yourself once the system comes up.

try to change source by .
. activate tensorflow_p36
python /home/ec2-user/abc/wsgi.py
also check chmod +x your path file.

Run Flask using python3 not python

How can you run Flask app which uses a specific version of python?
On my env "python" = python2.7 and "python3" = python3.6. I have installed Flask using pip3. I start the app with FLASK_APP=app.py flask run. It uses python2.7 to execute, I would like it to use python3.6.
Also tried adding #!flask/bin/python3 to app.py, but it did not have an effect.

There are several ways:
Virtualenv lets you create isolated python environments with different versions (this is the way I would recommend)
You can put #!/usr/bin/python3 on top of your python file (see here)
Or you can start your script with python3 script.py
As mentioned in the comments above you can also start your script inside a Docker container that hosts a python3 installation

What I did was:
which flask, to find flask binary executable path (in my case /home/myuser/.local/bin/flask
edit /home/myuser/.local/bin/flask, changing the first line from #!/usr/bin/python to #!/usr/bin/python3
In summary, making flask use python3, regardless of which shebang was specified in other scripts, since those scripts are not the entrypoint of execution, but flask is. I didn't have to change the shebang in any of my scripts, just the flask executable.

Cron does not execute a python script that needs a python3 module in AWS-ec2

I tried to make a cron job on crontab which runs a python script on AWS ec2. My python script includes a module that is only available for python3.
Using the following command I changed the ec2 default python interpreter from python2.7 to python3.4
Source /home/ec-2user/venv/python34/bin/activate
and then using pip install, I installed the required module for python3.4. So now the default interpreter is python3.4 and when I run the script on ec2-user directory using the following command:
python test.py
the program runs without any problem (so I am sure the module is installed correctly).
But when I assign python file to a cronjob
* * * * * python test.py
It does not work. Checking the mail, the error is:
“No module found named “xxxxx” “
But as I said it worked fine outside of the cron.
I was wondering if you can help me with this problem. I appreciate your time and information.

You have to make a shell script which will do the steps of changing to script directory, activating virtual environment and then running it.
Example:
#!/bin/bash
cd $YOUR_DIR
. venv/bin/activate
python3.4 test.py
Then you call this script in cron with
/bin/bash /.../script.sh
What you could do additionally is
chmod +x test.py
and add/update first line to:
#!/usr/bin/env python3.4
This way you can just run Python script with ./test.py

Create file as 'user_cron.sh'
#!/bin/bash
cd '/root/my_new_project_python'
. my_project_venv/bin/activate
python3 main.py
set the cron using crontab -e

How do you iteratively develop with docker?

How does one iteratively develop their app using Docker? I have only just started using it and my workflow is very slow, so I'm pretty sure I'm using it wrong.
I'm following along with a python machine learning course on Youtube, and so I am using Docker to work with python 3. I know I can use virtualenv or a VM, but I want to learn Docker as well so bear with me.
My root directory looks like so:
Dockerfile main.py*
My docker file:
FROM python
COPY . /src
RUN pip install quandl
RUN pip install pandas
CMD ["python", "/src/main.py"]
And the Python file:
#!/usr/bin/env python
import pandas as pd
import quandl
print("Hello world from main.py")
df = quandl.get("WIKI/GOOGL")
print("getting data frame for WIKI/GOOGL")
print(df.head())
My workflow has been:
Learn something new from the tutorial
Update python file
Build the docker image: docker build -t myapp .
Run the app: docker run my app python /src/main.py
Questions:
How can I speed this all up? For every change I want to try, I end up rebuilding. This causes pip to get dependencies each time which takes way too long.
Instead of editing a python file and running it, how might a get an interactive shell from the python version running in the container?
If I wanted my program to write out a file, how could I get this file back to my local system from the container after the program has finished?
Thanks for the help!
Edit:
I should add, this was the tutorial I was following in general to run some python code in Docker: https://www.civisanalytics.com/blog/using-docker-to-run-python/

Speeding up the rebuild process
The simplest thing you can do is reorder your Dockerfile.
FROM python
RUN pip install quandl
RUN pip install pandas
COPY . /src
CMD ["python", "/src/main.py"]
The reason this helps is that Docker will re-use the cached build for commands it has already run. Now when you rebuild after modifying your source code, it will re-use the build results for the pip commands, as they do not need to be run again. It will only run the COPY step.
Getting a python shell
You can exec a shell in the running container and run your python command.
docker exec -it <container-id> bash
python <...>
Or, you can run a container with just a shell, and skip running your app entirely (then, run it however you want).
docker run -it <image> bash
python <...>
Writing outside the container
Mount an external directory into the container. Then write to the mounted path.
docker run -v /local/path:/path <.. rest of command ..>
Then when you write in the container to /path/file, the file will show up outside the container at /local/path/file.

How to run python script in directory in Cron (Raspberry Pi)

I just started using Cron to automate this one python script I have. I understand how to use all the time parameters in nano, but I'm confused with how you would run the script. Normally just to run it right from the console, I would do:
cd /pi/home/weather/Adafruit_Python_BMP/examples
and then from there I would run the script with:
python weatherFINAL.py
Now that I'm trying to automate this in Cron, I can't do the multiple commands to cd into the directory, and then run the program. I know this is probably a really easy problem to fix, but I've been stuck on this for a while. Any help is appreciated

It is quite simple:
Write a shebang line on the top of your script in order to make it executable:
!/usr/bin/env python
Make sure you can execute that script with chmod command:
$ chmod +x
Programn a crontab task with contrab command:
$contrab -e