I am currently struggling with how to run a method from a python file in a virtual environment via crontab.
I have a directory that looks as follows: /home/ubuntu/project has the file file.py
and the folder venv in it. In file.py there is a method() that I want to execute regularly via crontab, using the python and dependencies of the virtual environment.
I have already figured out that I need to use the python from inside the virtual environment, so instead of
python3
I use
/home/ubuntu/project/venv/bin/python3.
Now, I have also found answers to the question how to run a method from the command line, namely via
python3 -c 'import foo; print foo.hello()'.
I have tried to combine the two, but unfortunately
/home/ubuntu/project/venv/bin/python3 -c 'import /home/ubuntu/project/file; print(file.method())'
is invalid syntax. Also
/home/ubuntu/project/venv/bin/python3 -c 'from /home/ubuntu/project/ import file; print(file.method())'
only results in errors. On the other hand,
/home/ubuntu/project/venv/bin/python3 -c 'import file; print(file.method())'
results in file not being found.
How do I do this properly?
Thank you very much for considering this question.
The argument to import is not a file name. The simplest workaround is probably to cd into the directory, then run the script with the virtual environment's Python interpreter.
42 17 * * * cd project && ./venv/bin/python3 -c 'import file; file.method()'
from the crontab of the user whose home directory is /home/ubuntu.
More generally, the directory you want to import from needs to be on your PYTHONPATH,so you could equivalently set that instead of cd into the directory. A third alternative is to make the code in file.py into an installable module, and install it in the virtual environment. For a one-off, this may be an unnecessary chore, but it is definitely the most robust and sustainable solution.
Related
I'm trying to make a batch script (called run_windows) that check if the python virtual environment exists and if not, create it then activate it, install the requirements and finally run some python code.
set "VIRTUAL_ENV=mat_visualizer_env"
:read_mat
%VIRTUAL_ENV%\Scripts\activate
pip install -r app_files/requirements.txt
python -c "import sys; sys.path.insert(1,'app_files'); from main import visualize_mat_eeg; visualize_mat_eeg('%1')"
pause
EXIT /B 0
IF EXIST "%VIRTUAL_ENV%\Scripts\activate.bat" (
CALL :read_mat
) ELSE (
pip install virtualenv
python -m venv mat_visualizer_env
CALL :read_mat
)
However, when I run my script, the code exits at line 4: %VIRTUAL_ENV%\Scripts\activate with no errors:
Few things:
Running the .bat or bash version of activate in powershell is not predictable, Activate.ps1 is found in newer releases. It could probably be used with older versions if copied from a newer release.
Scripts in windows (.bat or .cmd in windows) processes from top down, :<ANCHOR> is not skipped like a Sub or Function would be.
I have only worked with the Conda version of activate.ps1 and it adds some nice commands. Had a quick look in standard Python 3.10.5 and it does not add commands but should work fine.
Edit: Added that while working, is not the best option to use the bash version of activate
Here is a quick example of your batch file rearranged, and using the Call command as previously instructed.
#Echo Off
Set "VIRTUAL_ENV=mat_visualizer_env"
If Not Exist "%VIRTUAL_ENV%\Scripts\activate.bat" (
pip.exe install virtualenv
python.exe -m venv %VIRTUAL_ENV%
)
If Not Exist "%VIRTUAL_ENV%\Scripts\activate.bat" Exit /B 1
Call "%VIRTUAL_ENV%\Scripts\activate.bat"
pip.exe install -r app_files/requirements.txt
python.exe -c "import sys; sys.path.insert(1,'app_files'); from main import visualize_mat_eeg; visualize_mat_eeg('%~1')"
Pause
Exit /B 0
Please note, that there is no working directory defined in this script, as was yours, and therefore no way to guarantee that when run, the relative paths point to the intended locations. Additionally, you have not made it clear what %1 was supposed to be, or where it was coming from, so I cannot guarantee whether that is correct.
Hi I am trying to run a python script as sudo from inside my virtualenvironment.
When I have activated my virtualenvironment I would normally use python somescript.py and my script starts up with the correct version of python and everything
When I use sudo python somescript.py I load up the wrong python install, which is not the one from my environment.
How do I resove this?
The activate script sets some environment variables (defines some functions, ...), which facilitate invoking Python (and tools). One way (more like a workaround) of achieving your goal, would be the variables to be carried across the [man7]: sudo(8) session. For that, you need to:
Pass the -E flag to sudo
PATH needs to be carried manually ([StackExchange.Unix]: How to make `sudo` preserve $PATH?)
All in all:
sudo -E env PATH=${PATH} python somescript.py
Output (works for simple commands):
(py_venv_pc064_03.05.02_test0) [cfati#cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q061715573]> python3 -c "import sys, os; print(\"EXE: {0:s}\nPATH: {1:s}\n\".format(sys.executable, os.environ[\"PATH\"]))"
EXE: /home/cfati/Work/Dev/VEnvs/py_venv_pc064_03.05.02_test0/bin/python3
PATH: /home/cfati/Work/Dev/VEnvs/py_venv_pc064_03.05.02_test0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
(py_venv_pc064_03.05.02_test0) [cfati#cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q061715573]> sudo python3 -c "import sys, os; print(\"EXE: {0:s}\nPATH: {1:s}\n\".format(sys.executable, os.environ[\"PATH\"]))"
EXE: /usr/bin/python3
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
(py_venv_pc064_03.05.02_test0) [cfati#cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q061715573]> sudo -E env PATH=${PATH} python3 -c "import sys, os; print(\"EXE: {0:s}\nPATH: {1:s}\n\".format(sys.executable, os.environ[\"PATH\"]))"
EXE: /home/cfati/Work/Dev/VEnvs/py_venv_pc064_03.05.02_test0/bin/python3
PATH: /home/cfati/Work/Dev/VEnvs/py_venv_pc064_03.05.02_test0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
The one way that never fails in this kind of situations, is using (Python's) executable full path. But since that's just a symlink, you'd probably want to preserve the environment anyway:
sudo -E env PATH=${PATH} /somePath/someFolder/myEnvironment/bin/python somescript.py
I think this is answered in here: https://askubuntu.com/questions/234758/how-to-use-a-python-virtualenv-with-sudo
The issue is almost certainly that when you run sudo, the virtualenv
environment variables, aliases, functions, etc aren't being carried
over.
The solution would be to explicitly run the virtual environment's
Python executable with sudo. For example if your virtualenv is
./AwesomeProject, then you could run sudo ./AwesomeProject/bin/python
to use the script with the virtualenv with root privileges.
I created a python script ExeMain.py that use virtualenv for dependence. So to launch this script using .desktop I made this command
Exec=sh -c 'source ~/PycharmProjects/ProBlog/venv/bin/activate; python ~/PycharmProjects/ProBlog/ExeMain.py;'
Which resulted with this error "sh: 1: source: not found" and followed by "ImportError"
Without the source command in .desktop file with all necessary modules installed in direct python environment it works fine (in other Linux machine). As it seams not to be a good practice i tried using venv. And as of the errors i could see than the import error is because source command is not executed.
When i launch the .desktop file it should execute the ExeMain.py file.
Standard versions of sh does not have the source command. Perhaps use /bin/bash instead? So it'd be written something along the lines of this:
Exec=/bin/bash -c 'source /home/username/PycharmProjects/ProBlog/venv/bin/activate && /home/username/PycharmProjects/ProBlog/venv/bin/python /home/username/PycharmProjects/ProBlog/ExeMain.py'
I also suggest three other changes: 1) the && operator so that it won't try to run python code until the activate is finished, 2) replace the ~ tilde with the full path in case the desktop environment doesn't properly expand it, and 3) put in the full path to the venv python so that it doesn't default to the system python. Some of these may not be necessary (depending on the system setup), but it doesn't hurt to be careful.
I feel like this is incredibly easy to fix, but for some reason it isn't.
I want to run a program in linux that opens python file filename.py by writing:
python3 filename arg
but it only works if i write:
python3 filename.py arg
Is there an easy way to run it without adding the extension? And without removing the extension completely? I wouldn't have imagined this to be a problem at all, but here we are.
Thankful for help!
Your file is called filename.py, therefore you call it with python3 filename.py. If you want to call it with python3 filename, you'll need to rename the file.
The one thing you can do is call Python with the -m switch, which will try to import a module with that name, i.e. python3 -m filename. That should work without renaming the file.
Firstly, if you start your python script with the shebang, then you won't need to type 'python3' before the filename each time you wish to run the code.
`#!/usr/bin/env python3'
Secondly, if you create a setup.py file is the same directory. Then you can install your script locally using:
$ sudo pip3 install -e .
from within your directory.
You will then be able to run your script from anywhere within terminal using:
$ myscript args
More info on creating a setup file here: https://python-packaging-user-guide.readthedocs.io/tutorials/distributing-packages/#setup-py
and example setup file here: https://github.com/pypa/sampleproject/blob/master/setup.py
I have recently installed kmos, a python package using pip in my user account on my institute cluster. How to create a module for kmos and set the path to the directory such that python accesses the library. Currently, I am giving the path to the kmos binary while running the program.
Linux distro: Cent OS
Module support: Lua-based Lmod environmental modules system
I am not exactly sure what you mean by "How to create a module for kmos". You didn't mention which terminal you are using. However it will be definitely helpful to understand the mechanism behind finding executables and python import.
If you want to execute the kmos command-line interface (e.g. kmos export ...) you need to make sure that wherever the kmos shell client is is in your $PATH variable. When you installed kmos (pip install --user --upgrade kmos) it should tell where it went. That directory needs to show up when you run echo $PATH. Most likely something like ~/.local/bin. If it doesn't show up you may want to put export PATH=${PATH}:~/.local/bin into your ~/.bashrc or the corresponding syntax in your echo $SHELL configuration file.
The other location is where the python module gets copied to. When you do the pip installation it should print this out as well. Most likely something like ~/.local/lib/pythonXY/site-packages. When you run python -c "import sys; print(sys.path)" it should include given directory. You can automatically add this directory again using your echo ${SHELL} configuration file like export PYTHONPATH=${PYTHONPATH}:~/.local/lib/pythonXY/site-packages.
If you can already import kmos from python then python -c "import kmos; print(kmos.__file__)" will tell you where it found it.