I am trying to automate running a python script every minute on my mac, from a virtual environment. I am convinced that I don't properly understand permissions, paths, and environment variables in some crucial way that is preventing me from figuring this out.
I am an admin user with root rights enabled. I run HomeBrew, PIP and Virtualenv to manage python packages and virtual environments for different projects.
I would like to do the following every 60 seconds:
$ source /.virtualenvs/myenvironment/bin/activate
$ cd ~/desktop/python/
$ python myscript.py
$ deactivate
I have tried:
(a) writing my own plist for Launchd - and I believe these documents were well formed.
(b) programs which manage Launchd daemons and agents for you (both Launch Control and Lingon).
(c) I have tried simply editing the crontab (only lets me if I use the sudo command).
The python script, which works on command, pulls data from an online source and stores it in a sqlite table. I can tell the cron isn't running because the sqlite db isn't being touched.
Any thoughts would be enormously appreciated.
You don't say exactly what you tried with launchd and cron, but I'd bet you weren't using either of them correctly. Both are oriented toward running single, isolated commands (/programs), not sequences of shell commands. While it's possible to do this with a single cron job or launchd item, it's going to be messy. The simplest thing would be to write a shell script that does the sequence you want (be sure to include a shebang at the beginning, and enable execute permission on the script with chmod +x /path/to/script), and run that from either cron or launchd:
#!/bin/bash
source /.virtualenvs/myenvironment/bin/activate
cd ~/desktop/python/
python myscript.py
deactivate
I would not recommend using Automator to wrap the command sequence; it's designed for GUI-based scripting, and may not work right in a background-only job.
I have had this exact same problem and have recently solved it. Look here for the steps I took. Basically it deals with the shell needing the PYTHONPATH, not just the PATH.
So I'm still not sure why I can't configure Launchd or the Crontab to work for me: I still suspect it has to do with the environment variables.
If anyone else is having a tough time with this issue, I suggest creating an Automator application that runs the shell script you want to run, then use Lingon or Launch Control to run the Automator application file at whatever frequency you desire.
Related
I have been trying to run a python script on start-up (on a Pi). I initially did this via an .sh script triggered by cron.
Posting with a problem on StackExchange Pi (https://raspberrypi.stackexchange.com/questions/110868/parts-of-code-not-running-when-autostarting-script-in-crontab) the suggestion is to use systemd.
The person helping me there has suggested not using a virtual environment when executing the Python script (they note their limited familiarity with Python), and using the real environment instead. But other resources strongly suggest the use of a virtual environment (e.g. https://docs.python.org/3/tutorial/venv.html).
In the hope of setting this up correctly could anyone weigh in on the correct approach?
Use the virtual environment. I don't see any reason not to. At some point you might want to have multiple Python applications run at the same time on that system, and these applications might require different versions of the same dependency and then you would be back to square one, so... Use the virtual environment.
When configuring systemd, crontab, or whatever, make sure to use the python binary that is placed inside the virtual environment's bin directory, so that there is no need to activate the virtual environment:
/path/to/venv/bin/python -m my_executable_module
/path/to/venv/bin/python /path/to/my_script.py
/path/to/venv/bin/my_executable_script
systemd is going to try to run your script on startup so your virtual environment will not have been activated yet. You can (maybe) avoid that issue by telling systemd to use the python in the virtualenv's bin, with the appropriate environment variables. Or you can activate as a pre-run step for that script's launch in systemd. Maybe.
But on balance I'd make it easy on systemd and your OS and ignore the virtualenv absolutists. Get the script to work on your dev machine using virtualenv all you want, but then prep systemd to use the global python, with suitable packages installed. You can always use virtualenvs on that pi, for scripts that don't have to work with systemd. Systemd doesn't always have the clearest error messages.
(If you need to import custom modules you could inject directories into sys.path in your script. This could even avoid installing packages, for the global Python, entirely.)
This answer is certainly opinion-based.
I am looking to schedule my python script runs with jenkins. The issue is, my scripts use a lot of libs like pandas etc that are installed on my mac terminal.
Is there a way to allow Jenkins to pick up these modules (or run the scripts as if it was terminal)? Also is there a way to run Python3 in jenkins?
I have already configured Jenkins to execute from custom workspace and have tried both shell and plugin executions.
The answer is yes, but it is detailed so I can only give you high level steps here. Jenkins can execute command line statements, and python modules can be run from the command line.
I would start by using the begins library to create a python file to run from the command line with arguments. Get it working on your local machine that way.
You will want to use either virtualenv or venv, and do all your pip installs using that virtual environment. Then you can copy the virtual environment to your Jenkins machine, or create a new one. Look into the freeze tool.
When calling your python from jenkins, you must first activate your virtual environment just as if you were working on it yourself.
You have a lot of research to do, but is very doable. I can help with follow up questions if needed.
I'm using PythonAnywhere. I've used virtualenvwrapper to create an environment where I have the required packages.
When in bash I do:
activate virtualenv
python my_subfolder/my_script.py
and it works.
However, I want to run this scheduled using PythonAnywhere's feature for scheduling. But when it runs, it reverts (expectedly) to the default Python interpreter.
I'm not so good with paths and navigating, so I tried shebanging in various ways. I know for certain that my venv is in this path:
https://www.pythonanywhere.com/user/my_username/files/home/my_username/.virtualenvs/my_venv/bin/python
I'm omitting the domain of course, but trying to append any of the following parts of the URL to #! like so never works:
#!user/my_username/files/home/my_username/.virtualenvs/my_venv/bin/python
Please note that the script is in a subfolder my_subfolder/my_script.py. I've also chmod u+x the Python script. Very confused.
I have an account to a computing cluster that uses Scientific Linux. Of course I only have user access. I'm working with python and I need to run python scripts, so I need to import some python modules. Since I don't have root access, I installed a local python copy on my $HOME with all the required modules. When I run the scripts on my account (hosting node), they run correctly. But in order to submit jobs to the computing queues (to process on much faster machines), I need to submit a bash script that has a line that executes the scripts. The computing cluster uses SunGrid Engine. However when I submit the bash script, I get an error that the modules I installed can't be found!
So my understanding to the problem is that the modules are not sent or something to the machine that executes the script. My question is: is it possible to include all the modules in the script or so?
EDIT: I just created a bash script that runs which python and I noticed that the output was NOT my python copy. But when I run 'which python' on my ssh account, I get my python copy correctly..
The submitted script is most likely using the system Python installation and not your own. Try submitting a shell script with only one command, which python, to confirm.
The fix is to prepend the path to your Python interpreter to your system path. On my machine, the right Python is installed at /Users/mbatchkarov/anaconda/bin/python. I added export PATH="/Users/mbatchkarov/anaconda/bin:$PATH" to ~/.bash_profile
EDIT Add the same line to ~/.bashrc.
I've seen a lot of scripts for daemonizing a python script in linux, but not much information about how to use them. Could anyone guide me on this?
I currently have a lengthy python script that listens on a socket for an incoming message, if it's the correct format accepts it and then stores it into the database. The script itself just opens the socket and then listens on a while true (which does the job!) and does all the work in there.
To daemonize it, would I have to modify my current script or call it from a separate script? I've seen examples of both but got neither to work.
Also, I'm using virtualenv which might the root of my problems, any hints on using this with daemonized scripts?
Create a shell-script that activates the virtual environment, and runs your Python script in the background.
Also, there should by a python module in the virtual environment that you can import and activate the environment from too. I don't have virtualenv working at the moment, so I can not check where it is, but search for activate (or something similar) in the virtual environment and you should find it.
Edit: Added a minimal Debian init.d script
The absolute minimal script needed to start a daemon when the computer boots, is this:
#!/bin/sh
/path/to/program &
The & makes the program run in the background, so it wont stop the rest of the boot process.
For a more complete script, copy /etc/init.d/skeleton and edit the new file. The important part to edit is the block at the beginning (between ### BEGIN INIT INFO and ### END INIT INFO, which is used by the update-rc.d program), and the NAME, DAEMON and DAEMON_ARGS variables. Hopefully that should be all that's needed for making a startup-script.
Activate the script as this:
sudo update-rc.d <name of script> defaults
sudo update-rc.d <name of script> enable
And to start it:
sudo update-rc.d <name of script> start
The <name of script> is just the name, not the full path.
script
export PYTHONPATH=.:/home/ubuntu/.local/lib/python2.7/site-packages/:/home/ubuntu/python/lib/python2.7/site-packages/
exec start-stop-daemon --start --chuid ubuntu --exec /home/ubuntu/python_envs/MyProj/bin/python /home/ubuntu/www/MyProj/MyProj.py -- --config-file-dir=/home/ubuntu/www/MyProj/config/ >> /home/ubuntu/startup.log 2>&1 &
end script
When you need to run an application in a python virtualenv, you can either 'activate' the virtualenv, or use that environment's unique python command.
As per the website "If you directly run a script or the python interpreter from the virtualenv's bin/ directory (e.g. path/to/env/bin/pip or /path/to/env/bin/python script.py) there's no need for activation." - http://pypi.python.org/pypi/virtualenv
I also have some python modules that were compiled from source. Those need to be in the PYTHONPATH environment variable. That could be part of your virtualenv activation, done with virtualwrapper, or explicitly called (as I do below.)
Calling the program from an UPSTART job works as well. My example is above.
On an Ubuntu 10.10 instance on Amazon EC2, I had better luck with the start-stop-daemon command.
I also struggled with some of the other upstart 'stanzas.' I am calling a python application with a specific virtualenv and some parameters to my executed program.