We Keep Coding

Python Poetry, install project into shell's namespace

What is a / the proper way to install Python programs into the shell's namespace via Poetry? With setuptools, you can use $ pip install -e path/to/project and then you can invoke $ project. However I have not found a way to do that with Poetry. Instead I need to use $ poetry run project.
To clarify, I want the following behavior:
$ poetry <some command> path/to/project
$ project
<output of project>

I guess your goal is to install your cli and make changes to it without having to reinstall it?
As you've probably noticed pip install -e doesn't yet support PEP 518 based projects like you get with poetry.
Poetry's solution for this is of course to let you execute installed libs or cli tools with poetry run as you described, or to activate the virtualenv that poetry creates containing your project in a new shell with poetry shell, which is similar to an interactive installation as long as your stay inside that shell.
Of course you can still install the project as usual with pip install ., and mess around with the files as the installed location, though this isn't a recommended workflow.

Why does clear Python virtualenv use system Python library?

I am newbie in Automation (Python + Pytest).
I try to use clear virtualenv on my mac, but somehow Pytest runs from a local directory.
So, what I try to do:
I have a project with many pip packages (pytest, selene etc.).
I do :
$ pip3 install --user virtualenv
$ virtualenv --no-site-packages venv
$ cd venv/
$ source venv/bin/activate
$ echo $PATH
> /Users/.../venv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/puppetlabs/bin
$ which python
> /Users/.../venv/bin/python
$ which pytest
> /usr/local/bin/pytest
And when I try to run my pytest - it does! But shouldn't!
If I delete Pytest from my mac as:
$ pip3 uninstall pytest
it works, and I got clear venv.
Python 3.7.4
This is pytest version 5.2.1, imported from /usr/local/lib/python3.7/site-packages/pytest.py
setuptools registered plugins: pytest-reportportal-1.0.4 at /usr/local/lib/python3.7/site-packages/pytest_reportportal/plugin.py
Can someone say what I do wrong? Or what is the problem with it?
How to create clear venv without dependencies from local PATH libs?

The behavior seems perfectly normal, working as intended.
Install pytest in your virtual environment, it will then take priority over the one globally installed, since it will be located higher up in your $PATH. Consider uninstalling the global one if you don't intend to use it.

How to install a Python package inside a virtual environment with Pip (OS X)

Edit:
I'm going to close this question as the reason its happening is different from my original assumption, and it's clearer to ask the question anew:
Pip installs packages in the wrong directory with virtualenv
The accepted answer doesn't directly answer the original question, but is a very useful overview.
Based on discussion below the issue is that even after
$ source ~/PycharmProjects/Practice/venv/bin/activate
$ pip install numpy
numpy is installed in /usr/local/lib/python2.7/site-packages
What could the reason for this be?
Original:
Using Python on OS X via Homebrew:
I've been trying much of the day to sort this out but either I get a must supply either home or prefix/exec-prefix -- not both error, or the package I try to install goes into totally the wrong place:
$ pip3 --version
pip 18.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
$ cd venv
$ pip3 install numpy
..... [snip with following error:]
"must supply either home or prefix/exec-prefix -- not both")
Using this hint
$ pip3 install numpy -t .
Then I get a new error,
`Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/.../pip-install-0fvveq3v/package/'
Searching around SO gives various possibilities involving pip install setuptools. but pip install throws the above error or installs in the wrong place. i.e. the solution involves something that's causing the error in the first place.
I tried to use the Python.org installer but it didn't install pip at all. (The custom installer showed the option checked but with size zero).

An introductory overview is available in this nice tutorial. Here is a good summary with more detail. But, if you renamed or moved the virtual env dir after its creation, it could break it. Create a new one from scratch: $ cd ~/PycharmProjects; python3 -mvenv newenv ; Activate: $ source newenv/bin/activate ; Install something: $ pip install colorama (same as pip3 install only if venv activated); Check: ls ~/PycharmProjects/newenv/lib/python3*/site-packages ; Deactivate: $ deactivate
Then you could try this solution for Pycharm: how to associate a virtual environment with a python project in pycharm. PyCharm indeed comes bundled with virtualenv which could have been customized, please look into Pycharm-specific resources: creating virtual environments and installing packages in Pycharm.
If you have installed PyPI's mainstream virtualenv, by default it will create new environments with the python interpreter that virtualenv was installed with. But it's possible to specify an alternate Python Interpreter upon a new env creation: $ virtualenv -p python3.7 newenvname
Regarding the error DistutilsOptionError: must supply either home or prefix - please check this and this for solutions. Homebrew'ed mappings between python and pip are described here. The normal pip install --user is disabled in homebrewed Python, but there are workarounds. MacOS system Python doesn't provide pip, but it can installed, reinstalled or upgraded for any specific python version manually. Original non-brewed installers are also available for all Python versions: https://www.python.org/downloads/mac-osx/
By default there's no pip.conf, but it can be created by hand to customize things. All possible pip.conf locations (per-user, per-venv, and global/system-wide, and how they override each other) are listed here. If someone faces an issue, they could use pip config list command to see their active configuration, or locate pip.conf and find it.
Finally, you may want to ensure you aren't using pip against macOS' system python. Shell commands such as $ brew info python, which pip, which pip3, pip3 -V, which python3 can help you see what you actually use. Since the macOS default $PATH used to be /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin, stock macOS binaries (including python) may take precedence over some homebrew'ed installations (including python). If so, a custom PATH could be exported via the ~/.bashrc if required.

Why can't I find ansible when I install it using setup.py?

Because I had some trouble with Ansible (I'm on mac) which seemed to be fixed in the latest dev version today I uninstalled ansible through pip (sudo pip uninstall ansible) and reinstalled the latest dev version from the github repo using the classic setup.py method, which seemed to end successfully (full output here.
So then I tried using it:
$ ansible --version
-bash: ansible: command not found
$ which ansible
$
I checked where it is installed. From the full output I linked to above I found that it is installed in /usr/local/lib/python2.7/site-packages, and indeed in there I find an egg:
$ ls -l /usr/local/lib/python2.7/site-packages | grep ansible
drwxr-xr-x 4 root admin 136 Aug 22 16:33 ansible-2.4.0-py2.7.egg
When I start Python and check the site-packages folder I find a different folder:
>>> import site; print site.getsitepackages()[0]
/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
but that is a symlink to the same folder:
$ ls -l /usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
lrwxr-xr-x 1 hielke admin 54 Aug 13 22:36 /usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages -> ../../../../../../../../../lib/python2.7/site-packages
So I guess the problem is that no symlink is created to the ansible package in /usr/local/bin/. But I'm unsure how I could create such a symlink and why it wouldn't appear in the first place.
Does anybody know how I can move forward from here? All tips are welcome!

When you invoke ansible from the shell, bash will search in your $PATH for a file named ansible that is executable. This may not be the only issue, but this is the immediate cause for the error you're seeing. The .egg file itself is not an executable, it's just a file used for distributing the code.
If ansible has been installed correctly, you should be able to find it by using locate or the OSX Finder GUI. The name should match exactly, with no file extensions. You will probably also find ansible-connection, ansible-console, etc. in the same place where you find the ansible executable. If you find it, great! Test it out and add that directory to your $PATH in a terminal like so:
export PATH=$PATH:/path/to/ansible
Where /path/to/ansible is the directory where you found the executables. This change to the $PATH variable is temporary, and will go away when you close your shell. If you can now run ansible from bash, then you can make the change permanent by adding that export to the end of your $HOME/.bash_profile file, or by adding a rule in /etc/paths.d (recommended by Apple). See more on how exactly to do those here if you are unfamiliar with them.
Now, if that's not the problem and you can't find the ansible executable, then the installation itself is your problem. You might also try using a virtual environment (if you have it installed) to make sure that the version you're pulling from github isn't broken:
git clone https://github.com/ansible/ansible.git
cd ansible
virtualenv venv
source venv/bin/activate
pip install .
which ansible
As of this writing, the above gives me a working ansible install.

For those using the Windows 10 Ubuntu terminal, running this command should fix the issue:
export PATH=$PATH:~/.local/bin

Find where ansible reside on your Mac. Most times its /Users/<yourusername>/Library/Python/3.7/bin
or /Users/<yourusername>/Library/Python/2.7/bin. Then ...
export PATH=$PATH:/Users/<yourusername>/Library/Python/3.7/bin
You can store this in your .bashrc file.

Well, I think you just need to create a soft link
ln -s /Users/${yourname}/Library/Python/${python version}/bin/ansible /usr/local/bin/ansible

pip3 install ansible --user
this installs in ~/.local. just include this in PATH, it will work
example: export PATH="$PATH:~/.local/bin"

I faced same issue when I installed ansible. Run the below commands to solve the issue. But we have to active each time when we open a bash session if we want to use ansible.
$ python -m virtualenv ansible
$ source ansible/bin/activate
$ pip install ansible

I'm using zsh so in my /Users/arojas/.zshrc I add this line that's where my Ansible got installed by Python
export PATH="$PATH:$HOME/Library/Python/3.7/bin"

I faced the same issue and after have installed it used pip3 install ansible it all works now.

I suggest uninstalling Ansible and re-installing it using pip according to the method suggested in the Ansible docs:
Or if you are looking for the latest development version:
pip install git+https://github.com/ansible/ansible.git#devel
If you are installing on OS X Mavericks, you may encounter some noise from your compiler. A workaround is to do the following:
$ sudo CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install ansible
Readers that use virtualenv can also install Ansible under virtualenv, though we’d recommend to not worry about it and just install Ansible globally. Do not use easy_install to install ansible directly.

system/instance used: ec2 RH8, as root
# pip3 install ansible (not recommended - should be by a user)
# ansible --version ( not found - wtf?!)
# yum install mlocate
# update
# locate ansible (long output; scroll to where you input command)
# export PATH=$PATH:/usr/local/bin
# ansible --version (success, Yes!)

system/instance used: ec2 RH8, as root
# pip3 install ansible (not recommended - should be by a user)
# ansible --version ( not found - :( )
# yum install mlocate
# updatedb (updatedb creates or updates a database used by locate)
# locate ansible (TL;DR)
# export PATH=$PATH:/usr/local/bin (Add this line to .bashrc)
# source .bashrc (To reflect the changes in the bash)
# ansible --version (success, Yes!)

How to use Python virtualenv

The scenario is: I am on Ubuntu 11 which comes with Python 2.7, I want to run Mozilla JetPack which supports Python 2.5/2.6 and Google App Engine which only supports Python 2.5.
Read that its not a good idea to remove Python 2.7 as Ubuntu maybe using it. So the correct way is to use virtualenv. But I am quite lost using it. I installed Python 2.5 in /usr/local/python25 following this guide
I tried
jiewmeng#JM:/usr/local/python25/bin$ ./python --version
Python 2.5.5
jiewmeng#JM:/usr/local/python25/bin$ ./python virtualenv /works/tmp/test
./python: can't open file 'virtualenv': [Errno 2] No such file or directory
then the below works but I will be using Python 2.7
jiewmeng#JM:/usr/local/python25/bin$ virtualenv /works/tmp/test
New python executable in /works/tmp/test/bin/python
Installing distribute.................................................................................................................................................................................done.
jiewmeng#JM:/usr/local/python25/bin$ cd /works/tmp/test/bin
jiewmeng#JM:/works/tmp/test/bin$ ls
activate activate_this.py easy_install easy_install-2.7 pip python
jiewmeng#JM:/works/tmp/test/bin$ ./python --version
Python 2.7.1+
Also, how do I then run Mozilla JetPack or Google App Engine with this version of Python? Sorry I am new to Python (and Linux/Ubuntu)

Outline:
First cd to /usr/local/python25/bin
Download setuptools for Python2.5 (setuptools-0.6c11-py2.5.egg)
Install it (sh setuptools-0.6c11-py2.5.egg).
Now install pip (easy_install pip).
Install virtualenv and virtualenvwrapper using pip (pip install v... etc.).
Configure WORKON_HOME for virtualenv wrapper to work (export WORKON_HOME = $HOME/.virtualenvs). You can use any other directory you want (not just $HOME/.virtualenvs). Just make sure to use the full path.
Now create a virtualenv (mkvirtualenv foobar).
Switch to the new virtualenv (workon foobar).
Now install GAE, JetPack and whatever you want using pip install blah
Why did your install not work?
Looks like you did not install virtualenv for Python2.5. Hence this will not work.
jiewmeng#JM:/usr/local/python25/bin$ ./python virtualenv /works/tmp/test
You can check by running ls command in that directory. I suspect you won't find virtualenv file there.
However this worked for you.
jiewmeng#JM:/usr/local/python25/bin$ virtualenv /works/tmp/test
Because it is using the virtualenv file for system default Python2.7. You can check which virtualenv and opening the virtualenv script. You'll see that the #! will point to system default python.
So you need to install the easy_install and pip for Python 2.5 before you can create virtualenv for Python 2.5. Just follow the steps outlined above.

You don't need to do anything fancy outside the virtualenv wrapper. Just use the --python=python2.5 flag (check out the man page for virtualenv form more). It does not matter what version you install it with, you just have to select the right executable for python in the virtual environment.
e.g. mkvirtualenv --python=python2.5 --distribute python25 if the python flag fails, either add a symlink (ln -s) to python25 in your $PATH or use the full path name on the python flag.
Also, default for multiple python installations is to have, for all 'altinstall' versions, a separate python and easy_install. So, for example: python2.5 ,easy_install-2.5 ,python2.6, easy_install-2.6 etc.