I'm on Ubuntu 20.04, and I am having issues setting up the crontab correctly using pyenv + pipenv together. Simply adding pipenv run python script.py to the cronjob does not work; I think it may be due to:
the environment required by pyenv
the non-interactive shell of cronjob
UPDATED BIG QUESTION
How do I use /home/jennings/.pyenv/shims/pipenv correctly in crontab??
I've checked $? for pipenv -v in run.sh scheduled in crontab, and it fails.
https://github.com/pyenv/pyenv#advanced-configuration
00. (for reference) Bash startup
pyenv requires these entries in these startup files for interactive/login shells I don't understand how to translate this to a non-interactive cronjob call. How do I set up my BASH_ENV to emmulate these environents below?
https://stackoverflow.com/a/9954208/9335288
# ~/.profile
eval "$(pyenv init --path)"
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init --path)"
fi
# ~/.bashrc:
export PYENV_ROOT="$HOME/.pyenv"
I. Crontab
I am trying to effectively write a .profile for the cronjob... but if you a better solution, please let me know.
# CRONTAB
SHELL=/bin/bash
BASH_ENV="home/jennings/.custom_bash_env
# BASH_ENV="home/jennings/.profile"
# BASH_ENV="home/jennings/.bashrc"
* * * * * cd $PROJECT_DIR; ./run.sh
# BASH_ENV:
Should I point to .profile or .bashrc instead?*
# PYENV
#eval "$(pyenv init --path)"
#if command -v pyenv 1>/dev/null 2>&1; then
# eval "$(pyenv init --path)"
#fi
# ENV VARIABLES
PYENV_ROOT="/home/jennings/.pyenv"
PYTHONPATH=$SOMEPLACE
PATH=all:of:them:paths
II. Project Folder
# run.sh:
#!/usr/bin/bash
# PYENV
eval "$(pyenv init --path)"
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init --path)"
fi
# actual pipenv command I'm trying to run
pipenv run python main.py
# main.py:
import this
# Does some python and logging
What I've tried and know
Pointing directly at the .profile and .bashrc doesn't work
run.sh will run okay; it's thepipenv step that fails
Scheduling other non-pipenv commands work fine
pyenv init block
Placed in the BASH_ENV file, the cronjob doesn't run at all
Placed in the run.sh, the cronjob now runs, but the pipenv run still fails
pipenv related
I've tried pipenv shell; which python to use that one in the crontab enjoy -- no cigar
Cron has a limited environment. You have to declare the full path to the pipenv executable. Instead of using pipenv run python script.py you should use /usr/local/bin/pipenv run python script.py.
In your case the full path can be different which pipenv will give you the proper path.
This is explained in this post how to run pipenv in cronjob in ubuntu?
Related
Goal: I am trying to install the "pyenv" on my Linux machine with the help of this article "https://realpython.com/intro-to-pyenv/"
Expected Result: according the article pyenv should add to the "load path" of my Linux machine.
Actual Result: pyenv doesnt add to the load path and I have to manually add to the .bashrc file.
Error Messages: Actually I am not seeing any error messages for now. When I close my terminal window and open new one ; each time my terminal window opens up with the below message.
WARNING: `pyenv init -` no longer sets PATH.
Run `pyenv init` to see the necessary changes to make to your configuration.
[centos#localhost ~]$
What I have done: I have searched through the SO for answers and I found my issue with some resolutions. but they didn't work for me as expected.
I have added the below lines to the .bashrc file of my user.
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Mind you that, even though this warning comes every time when I add 'pyenv' command to the terminal it shows me with the valid result.
for the references below is my .bash_profile and .bashrc files contents.
.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
#JAVA_HOME=/data/programs/jdk1.8.0_221
#JAVA_HOME=/data/programs/jdk-13.0.2
#JAVA_HOME=/data/programs/jdk1.7.0_80
JAVA_HOME=/data/programs/jdk-11.0.8
#JAVA_HOME=/data/programs/jdk-14.0.1
M2_HOME=/data/programs/apache-maven-3.6.1
JRE_HOME=$JAVA_HOME
export PATH JAVA_HOME M2_HOME JRE_HOME
export PATH=$JAVA_HOME/bin:$M2_HOME/bin::$PATH
export PATH
Also below is my .bashrc file contents.
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Environment: Centos 7
I want remove this error when I'm opening a terminal session every time. How do I resolve this ?
pyenv is upgraded to 2.0 and the init script is changed. Read from the official homepage but not some random article on the web.
Check README.md in pyenv on GitHub
Fix for your problem.
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
NOTE: code above is not the whole pyenv init script, it only answers this very question. If you're setting up pyenv from the beginning, check pyenv's repo on Github for detail.
Just replace the line eval "$(pyenv init -)" from your ~/.bashrc file to eval "$(pyenv init --path)".
I created a new Conda environment on an x86-64 Linux mainframe, using the command
conda create --name myenv --file somefile.txt --python=3.8.
I double checked my Python version in this environment using conda list, which returns
...
python 3.8.3 hcff3b4d_0
...
However, after activating this environment, Python 3 scripts doesn't run, and running which python reveals that the environment defaults to using the default system Python 2 interpreter:
$ which python
/usr/bin/python
My Efforts So Far
First, I added the line export PATH=$PATH:/home/miniconda3/envs/myenv/bin to my ~/.bashrc and ~/.profile files, to no effect. which python still returns /usr/bin/python.
I then checked my alias file to see if python is aliased to Python 2. But there is no entry in the alias file about python.
For your reference
my ~/.bashrc looks like this:
$ cat ~/.bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/miniconda3/etc/profile.d/conda.sh" ]; then
. "/home/miniconda3/etc/profile.d/conda.sh"
else
export PATH="/home/miniconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
alias julia="/home/julia-1.4.0/bin/julia"
export PATH=$PATH:/home/miniconda3/envs/myenv/bin
$ cat ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
$ cat ~/.profile
export PATH=$PATH:/home/miniconda3/envs/myenv/bin
I think you need to add your env path to the front of the PATH variable so it will find python there first. The OS will look for a file or application in the PATH list, and use the first match that it finds.
Change
export PATH=$PATH:/home/qingyanz/miniconda3/envs/myenv/bin
to this:
export PATH=/home/qingyanz/miniconda3/envs/myenv/bin:$PATH
As an example, here are the settings on my conda environment:
(ds_tensorflow) C:\Users\me>which python
/c/Users/me/miniconda3/envs/ds_tensorflow/python
(ds_tensorflow) C:\Users\me>env | grep PATH
...
PATH=/c/Users/me/miniconda3/envs/ds_tensorflow:/mingw-w64/bin:/usr/bin:/bin:...
I would like to set permanently a conda environment in my docker image in order that the functions of the conda package could be used by the script given as argument to the entrypoint.
This is the dockerfile that I created.
FROM continuumio/anaconda3
RUN conda create -n myenv
RUN echo "source activate myenv" > ~/.bashrc
ENV PATH:="/opt/conda/envs/myenv/bin:$PATH"
SHELL ["/bin/bash", "-c"]
ENTRYPOINT ["python3"]
It seems that the ~/.bashrc file is not sourced when I run the docker container. Am I doing something wrong?
Thank you
As a work around either use 'SHELL ["/bin/bash", "-i", "--login", "-c"]'
-or-
edit the .bashrc file in the image to not exit if not in interactive mode by changing "*) return;;" to read "*) ;;"
Using the first option bash will complain about job control and ttys, but the error can be ignored.
cause of the issue:
the .bashrc file contains the following command:
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
which causes bash to stop sourcing the file if not in interactive mode. (the -i flag)
Unfortunately, I haven't found a way for the conda stanza to be inserted into .bash_profile or .profile automatically instead of (or in addition to) .bashrc, as there doesn't seem to be an option to override or add to the list of what files conda init examines for modification.
I have an environment.yml in my applications folder
I have this in my dockerfile:
RUN conda env create
RUN source activate myenvfromymlfile
When I run the container though the env is not activated. If I do conda env list Is see /opt/conda is activated:
root#9c7181cf86aa:/app# conda env list
# conda environments:
#
myenvfromymlfile /opt/conda/envs/myenvfromymlfile
root * /opt/conda
If I attach to the container I can manually run source activate myenvfromymlfile and it works, but why doesn't that work in the RUN directive??
In examples, I see this often in dockerfiles that require conda:
CMD [ "source activate your-environment && exec python application.py" ]
Can someone explain why it is necessary to use && to make it a single command? And why running "source activate" in a RUN directive does not work? I want to have my dockerfile look like this:
RUN conda env create
RUN source activate myenvfromymlfile
ENTRYPOINT ["python"]
CMD ["application.py"]
Consider the below Dockerfile
RUN conda env create
RUN source activate myenvfromymlfile
ENTRYPOINT ["python"]
CMD ["application.py"]
Statement #1 conda env create. Create the environment and changes files on the disk.
Statement #2 source activate myenvfromymlfile. Loads some stuff in the bash sessions. No disk changes done here
Statement #3 and #4 specifies what happens when you run the container
ENTRYPOINT ["python"]
CMD ["application.py"]
So now when you run the container. Anything that you did in step#2 is not there, because a shell was launched to run step #2, when it completed the shell was closed. Now when you run the image a new shell is started and it is brand new shell with now no knowledge that in past inside your dockerfile you ran source activate myenvfromymlfile
Now you want to run this application.py in the environment you created. The default shell of docker is sh -c. So when you set CMD as below
CMD [ "source activate your-environment && exec python application.py" ]
The final command executed at start of container becomes
sh -c "source activate your-environment && exec python application.py"
Which activates the environment in current shell and then runs your program.
I'm using ZSH/oh-my-zsh and the regular OSX terminal, though the same problem occurs in iTerm. I'd been using rbenv and nvm without issue, but recently started working with python and pyenv, and have run into the following issue. On loading a new terminal window, I get the following message at the prompt:
Last login: Sat Apr 1 11:56:46 on ttys001
/Users/jackfuller/.zshenv:3: command not found: pyenv
Since installing pyenv, my machine seems noticeably slower. Obviously loading pyenv will slow things down but load times have dropped way off.
The catch is that pyenv works perfectly after the terminal is loaded, and as far as I can tell my .zshrc is configured properly:
alias dev="cd ~/development"
alias gow="cd ~/goworkspace"
alias dl="cd ~/downloads"
export PATH=/usr/local/bin:$HOME/bin:$PATH
export EDITOR='atom -n'
export PAGER='less -f'
export PATH=$HOME/.rbenv/shims:$PATH
RBENV
if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi
# NVM
export NVM_DIR="/Users/jackfuller/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm
# PYENV
export PYENV_ROOT=/usr/local/opt/pyenv
eval "$(pyenv init - --no-rehash)"
# if which pyenv > /dev/null; then eval "$(pyenv init - --no-rehash)";
fi
# For go.
export GOPATH="$HOME/goworkspace"
export PATH=$PATH:/usr/local/go/bin
export GOROOT="usr/local/go"
If anyone can offer any advice/solutions, it would be much appreciated. Environment variables and shell config files seem more temperamental /confusing than they should be.
/Users/jackfuller/.zshenv:3: command not found: pyenv indicates that your error is in .zshenv at line 3. Maybe you could post your .zshenv. Is there a reason the fi after #PYENV is not commented? Also you could do export PATH=... once and not thrice. Try which pyenv to find the path to pyenv and look if its in your $PATH. For Future uses i'd put my aliases into ~/.zsh_aliases and do a source ~/.zsh_aliases in the .zshrc, otherwise it gets pretty ugly once you have some more aliases. Sorry for the bad structured answer ;)