I've seen a lot of scripts for daemonizing a python script in linux, but not much information about how to use them. Could anyone guide me on this?
I currently have a lengthy python script that listens on a socket for an incoming message, if it's the correct format accepts it and then stores it into the database. The script itself just opens the socket and then listens on a while true (which does the job!) and does all the work in there.
To daemonize it, would I have to modify my current script or call it from a separate script? I've seen examples of both but got neither to work.
Also, I'm using virtualenv which might the root of my problems, any hints on using this with daemonized scripts?
Create a shell-script that activates the virtual environment, and runs your Python script in the background.
Also, there should by a python module in the virtual environment that you can import and activate the environment from too. I don't have virtualenv working at the moment, so I can not check where it is, but search for activate (or something similar) in the virtual environment and you should find it.
Edit: Added a minimal Debian init.d script
The absolute minimal script needed to start a daemon when the computer boots, is this:
#!/bin/sh
/path/to/program &
The & makes the program run in the background, so it wont stop the rest of the boot process.
For a more complete script, copy /etc/init.d/skeleton and edit the new file. The important part to edit is the block at the beginning (between ### BEGIN INIT INFO and ### END INIT INFO, which is used by the update-rc.d program), and the NAME, DAEMON and DAEMON_ARGS variables. Hopefully that should be all that's needed for making a startup-script.
Activate the script as this:
sudo update-rc.d <name of script> defaults
sudo update-rc.d <name of script> enable
And to start it:
sudo update-rc.d <name of script> start
The <name of script> is just the name, not the full path.
script
export PYTHONPATH=.:/home/ubuntu/.local/lib/python2.7/site-packages/:/home/ubuntu/python/lib/python2.7/site-packages/
exec start-stop-daemon --start --chuid ubuntu --exec /home/ubuntu/python_envs/MyProj/bin/python /home/ubuntu/www/MyProj/MyProj.py -- --config-file-dir=/home/ubuntu/www/MyProj/config/ >> /home/ubuntu/startup.log 2>&1 &
end script
When you need to run an application in a python virtualenv, you can either 'activate' the virtualenv, or use that environment's unique python command.
As per the website "If you directly run a script or the python interpreter from the virtualenv's bin/ directory (e.g. path/to/env/bin/pip or /path/to/env/bin/python script.py) there's no need for activation." - http://pypi.python.org/pypi/virtualenv
I also have some python modules that were compiled from source. Those need to be in the PYTHONPATH environment variable. That could be part of your virtualenv activation, done with virtualwrapper, or explicitly called (as I do below.)
Calling the program from an UPSTART job works as well. My example is above.
On an Ubuntu 10.10 instance on Amazon EC2, I had better luck with the start-stop-daemon command.
I also struggled with some of the other upstart 'stanzas.' I am calling a python application with a specific virtualenv and some parameters to my executed program.
Related
I have a bash file, it works fine when executed from terminal.
#!/bin/bash
source activate tensorflow_p36
python /home/ec2-user/abc/wsgi.py
Note: tensorflow_p36 being an in-built conda environment, does not require to be called from specific /env/bin directory. It can be activated from any directory. I think it's a feature of Amazon Deep Learning AMIs.
If I run this bash script with sudo it doesnt activate the virtual environment and works in default python environment. The python file can run in that virtual environment only.
I have tried all 3 alternatives (rc.local, .conf file, init.d config)here, also tried to use crontab as suggested here. I have also tried using supervisord to add this bash script as a program.
When the program runs from these methods, I always get the same import errors because it is using default python 3 environment which doesn't have the required dependencies.
I am working on Amazon CentOS (Deep learning AMI). Can someone please suggest a method to run this script every time system restarts?
In the rc.local, instruct root to run it as you:
su --command /path/to/bash/file --login grimlock
You can run it from your personal Crontab.
( crontab -l; printf '#reboot /path/to/bash/file\n' ) | crontab -
If you don't have a crontab there will be an error message from crontab -l but it's harmless.
crontab: no crontab for ec2-user
You just need to do this once, and the job will execute as yourself once the system comes up.
try to change source by .
. activate tensorflow_p36
python /home/ec2-user/abc/wsgi.py
also check chmod +x your path file.
I use python-mode.el and have just discovered Pipenv. I can start Python within a project environment from a terminal by first changing to the project directory and entering "pipenv run python". But python-mode wants to execute a process, not a command string.
I tried creating a shell script like this:
#!/bin/bash
cd ~/myprojectdir
pipenv run python
but got this complaint
Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
This variable is set in my terminal environment: how do i ensure it's set for pipenv? But the bigger question is, how do i run this pipenv virtual environment python inside emacs to get a buffer with a Python process?
pipenv.el helps here, by setting the appropriate variables and providing a porcelain around Pipenv inside Emacs.
Looks like i have to explicitly source my .bashrc file (or equivalent). Changing the contents of my runpipenv.sh script to this
#!/bin/bash
source ~/.bashrc
cd ~/myprojectdir
pipenv run python
(where my .bashrc file sets LANG) and then calling
(setq py-shell-name "runpipenv.sh")
in Emacs seems to work.
I'm using PythonAnywhere. I've used virtualenvwrapper to create an environment where I have the required packages.
When in bash I do:
activate virtualenv
python my_subfolder/my_script.py
and it works.
However, I want to run this scheduled using PythonAnywhere's feature for scheduling. But when it runs, it reverts (expectedly) to the default Python interpreter.
I'm not so good with paths and navigating, so I tried shebanging in various ways. I know for certain that my venv is in this path:
https://www.pythonanywhere.com/user/my_username/files/home/my_username/.virtualenvs/my_venv/bin/python
I'm omitting the domain of course, but trying to append any of the following parts of the URL to #! like so never works:
#!user/my_username/files/home/my_username/.virtualenvs/my_venv/bin/python
Please note that the script is in a subfolder my_subfolder/my_script.py. I've also chmod u+x the Python script. Very confused.
If one defines which version of python to use in a bash script, it would be
export PYTHON = "/path/python/python-3.5.1/bin/python"
But for Python virtualenv's, one executes these commands in the command line
cd /path/pathto/virtualenv
source activate
cd another_directory
How does one "enter" a Python virtualenv in a bash script? What is the standard approach here?
We have to distinguish two cases here:
You want to use/call python (or python-based tools) in your bash script, but python or those tools should be taken from and run in a virtualenv
You want a script that, amongst other things, lets the shell from which you call it enter the virtualenv, so that you can interactively call python (or python-based tools) inside the virtualenv
Case 1: Using a virtualenv inside a script
How does one "enter" a Python virtualenv in a bash script?
Just like on the interactive bash command line:
source /path/to/the/virtual_env/bin/activate
What is the standard approach here?
The standard approach is not to enter the virtualenv in a bash script. Instead, call python and/or the python-based commands you want to use by their full path. To make this easier and less repetitive, you can use aliases and variables.
Case 2: Activating a virtualenv in an interactive bash session by calling a script
There already is such a script. It's called activate and it's located in the bin directory of the virtualenv. You have to source it rather than calling it like a normal command. Only then will it run in the same session instead of in a subshell, and thus only then can it make modifications to the session that won't be lost due to the subshell terminating at the end of the script.
So just do:
source /path/to/the/virtual_env/bin/activate
in your interactive shell session.
But what if you want to do more than the activate script does? You can put
source /path/to/the/virtual_env/bin/activate
into a shell script. But, due to the reason mentioned above, it won't have much effect when you call your script normally. Instead, source your script to use it from an interactive session.
Thus:
Content of my_activate.sh
#!/bin/bash
# Do something
# ...
# then
source /path/to/the/virtual_env/bin/activate
# Do more stuff
# ...
and in your interactive session
source my_activate.sh
I recommend using virtualenvwrapper. It provides some useful tools for managing your virtual environments.
pip install --user virtualenvwrapper
When you create the virtual environment, you specify which version of python should be used in the environment.
mkvirtualenv -p /usr/local/bin/python2.6 myproject.2.6
mkvirtualenv -p /usr/local/bin/python3.3 myproject.3.3
Then, "enter" the environment with the workon command.
workon myproject.2.6
Here are few steps to follow, one thing you can do is
export PYTHON = "/path/pathto/virtualenv/python"
Use this path in bashrc to use. Or you can do something like:-
vim ~/.bashrc
Go to end and set
alias python=/path/pathto/virtualenv/python
source ~/.bashrc
I am trying to automate running a python script every minute on my mac, from a virtual environment. I am convinced that I don't properly understand permissions, paths, and environment variables in some crucial way that is preventing me from figuring this out.
I am an admin user with root rights enabled. I run HomeBrew, PIP and Virtualenv to manage python packages and virtual environments for different projects.
I would like to do the following every 60 seconds:
$ source /.virtualenvs/myenvironment/bin/activate
$ cd ~/desktop/python/
$ python myscript.py
$ deactivate
I have tried:
(a) writing my own plist for Launchd - and I believe these documents were well formed.
(b) programs which manage Launchd daemons and agents for you (both Launch Control and Lingon).
(c) I have tried simply editing the crontab (only lets me if I use the sudo command).
The python script, which works on command, pulls data from an online source and stores it in a sqlite table. I can tell the cron isn't running because the sqlite db isn't being touched.
Any thoughts would be enormously appreciated.
You don't say exactly what you tried with launchd and cron, but I'd bet you weren't using either of them correctly. Both are oriented toward running single, isolated commands (/programs), not sequences of shell commands. While it's possible to do this with a single cron job or launchd item, it's going to be messy. The simplest thing would be to write a shell script that does the sequence you want (be sure to include a shebang at the beginning, and enable execute permission on the script with chmod +x /path/to/script), and run that from either cron or launchd:
#!/bin/bash
source /.virtualenvs/myenvironment/bin/activate
cd ~/desktop/python/
python myscript.py
deactivate
I would not recommend using Automator to wrap the command sequence; it's designed for GUI-based scripting, and may not work right in a background-only job.
I have had this exact same problem and have recently solved it. Look here for the steps I took. Basically it deals with the shell needing the PYTHONPATH, not just the PATH.
So I'm still not sure why I can't configure Launchd or the Crontab to work for me: I still suspect it has to do with the environment variables.
If anyone else is having a tough time with this issue, I suggest creating an Automator application that runs the shell script you want to run, then use Lingon or Launch Control to run the Automator application file at whatever frequency you desire.