We Keep Coding

How to install python library to specific environment (without conda)

Update update: json_lines is not supported by python versions < 3 - my issue had pretty much nothing to do with environments. I am now using 3.9.1 and all is gucci.
Update: After using which python in my jupyter notebook and in my Terminal, I see that they are both using the same environment. As such I am still at a loss as to why my notebook cannot find json_lines.
I have two python environments on my computer, a default one and one I have for running my jupyter notebook on. I am trying to install the library, json_lines to the latter environment. I am not used the Anaconda environment manager.
On my Mac's Terminal I used the general pip install command pip install json-lines, but when I try to execute the following line of Python import json_lines in my notebook, I still receive the following error ImportError: No module named json_lines.
As I suspect I am not installing to the correct environment, I tried installing the library from inside my notebook with the following, import sys; !{sys.executable} -m pip install jsonlines.
However, this has not changed my dilemma.
Is there some way I can specify from my Terminal which environment to install to? or is it likely I am encountering a different issue to what I suspect?

The package for json_lines in pip in json-lines. Hence you could install it as:
$ pip install json-lines
It may be appropriate to use an isolated python environment for your particular project if you want to use particular conda libraries but without the whole package. In this instance, you would be able to use virtualenv. This will allow you to create an isolated python environment.
$ pip3 install virtualenv
You can call virtualenv to create a virtual python environment with the working name e.g. myvenv.
$ virtualenv myvenv
From here, you can set your terminal to use this python version. If you are on *nix:
$ which python
/usr/bin/python
$ source myvenv/bin/activate
(myvenv)$ which python
/.../myvenv/bin/python

This article can help you out.
https://janakiev.com/blog/jupyter-virtual-envs/
You need to create a virtualenv which will be used by your notebooks.

Module installed returning ModuleNotFoundError

I'm using virtualenvwrapper to manage a project where I'm just running this for now
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
with beam.Pipeline(options=PipelineOptions()) as p:
pass
This is returning the following error
Traceback (most recent call last):
File "path/to/pipeline.py", line 1, in <module>
import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
I have run pip install apache-beam. Running pip list returns
➜ pip list
Package Version
------------------------------ ---------
apache-beam 2.23.0
avro-python3 1.9.2.1
...
I'll add the following outputs as suggested in a similar question.
pip freeze
➜ pip freeze
apache-beam==2.23.0
avro-python3==1.9.2.1
...
pip -V
➜ pip -V
pip 20.2.2 from /Users/miguel/.virtualenvs/myenv/lib/python3.7/site-packages/pip (python 3.7)
python -V
➜ python -V
Python 3.7.3
which python
➜ which python
/Users/miguel/.virtualenvs/myenv/bin/python
which pip
➜ which pip
/Users/miguel/.virtualenvs/myenv/bin/pip
I don't know if this is relevant but I'm using VSCode and I have selected my python interpreter according to VSCode instructions here. Additionally, I installed python following the instructions here.
Any idea why this is happening?

First test if your installed Python modules work outside of an ide like VSCode.
We can do this by opening the terminal/command prompt and activating our virtual enviornment. In this case, as you're using virtualenvwrapper you need to use the command:
workon myenv
Documentation for the virtualenvwrapper can be found here. Once activated, we can open the Python interpreter in our terminal by using the command:
python
Once running, try and import apache_beam as before using:
import apache_beam as beam
If this works we now know it's a virtual environment set up issue using VSCode. For setting up Virtual Environments for Python in VSCode, use the official documentation Using Python environments in VS Code. This should allow you to specify the virtual environment.
If this fails to work, another option is to create a new virtual environment in VSCode and install your modules. Installing modules to a environment in VSCode can be found in the VSCode instructions you linked to in your post.
I tried to re-create your issue as above using the following:
Python 3.7.7
VSCode 1.48.0
apache-beam 2.23.0
My virtual environment was named 'stack'. Once I had created it the only module I installed was apache-beam using:
pip install apache-beam
When creating the project I made the directory and launched via terminal using:
- mkdir hello
- cd hello
- code .
And then added the Python interpreter via the command:
Python: Select Interpreter
One additional step I needed (as I usually use PyCharm for Python) was to install:
Shell Command: Install 'code' command in PATH** command.
To allow the launch of code via terminal. Below is a screenshot of the project after following these steps and running the same commands as posted in your question for reference.
My .vscode/settings.json is as follows:
{
"python.pythonPath": "/Users/robertyoung/envs/stack/bin/python"
}

How to install Python packages in a specific environment?

I installed Anaconda3 so I can create environments and install different packages in each environment. But I fail to understand the difference between the Python in
/usr/bin/python
and
/opt/anaconda3/bin/python
I can seem to access Python 3.6.5 Anaconda from both, why is that? And, what is the difference between both?
Furthermore, I would like to install packages to a single Python environment only.

When you are running python in the terminal, it is looking up your default path to your the python command. In this case, anaconda probably put a line in your shell profile specify the path to the anaconda version, which is why you are seeing it in the interpreter when you run python from either directory.
Secondly, you can set up a conda environment to download app specific dependencies without interfering with your default set up by
conda create --name myenv
source activate myenv
conda install packagename
This will install it in the myenv environment only. To deactivate the environment just run
source deactivate
Here is the documentation on that https://conda.io/docs/user-guide/tasks/manage-environments.html

Judging by your path, you are using Linux which comes with python installed. So /usr/bin/python is default and you have installed the other one later.
For the environments use https://conda.io/docs/user-guide/tasks/manage-environments.html to activate the desired environment, then you can pip install or conda install the packages and it will be places safely only in that environment. Note that spyder icon runs the root environment by default and you have to run it from terminal after activating one of the environments.
Edit:
I'm not sure why you want to use cd to change the python version. I suggest use aliases. I guess you are just changing the path but running the same version of the python anyway. Take a look at this question:
Two versions of python on linux. how to make 2.7 the default

I wanted to create a new virtual environment to install new packages. Following worked for me:
Commands are executed in Jupyter Notebook (OS: Ubuntu 16.04 LTS)
Upgrade pip:
!pip install --upgrade pip
Install virtual environment:
!pip install virtualenv
Select version of Python you want to use in new environment:
I wanted to create an environment with Python version 3. Naming it as Python3_xyz:
!virtualenv -p python3 Python3_xyz
After execution, this will create a folder with the same name in the current working directory (i.e. the location where Jupyter notebook is present)
Create a new option with the name of the created environment
And finally, run the following command:
!python -m ipykernel install --user --name=Python3_xyz
This will create a new option with the name Python3_xyz in the menu from where we create a new notebook.
NOTE: One can run above commands from the terminal as well just don't use '!' before the commands.

This question is bit dated, but since I faced a similar issue, what worked for me might help someone!
I did pip install requests from within my conda environment, but failed to import requests even after trying out everything.
What worked for me: run python -m pip install requests or python3 -m pip install requests within you environment. This installed requests successfully for me.

How do I use pytest with virtualenv?

I installed pytest into a virtual environment (using virtualenv) and am running it from that virtual environment, but it is not using the packages that I installed in that virtual environment. Instead, it is using the main system packages. (Using python -m unittest discover, I can actually run my tests with the right python and packages, but I want to use the py.test framework.)
Is it possible that py.test is actually not running the pytest inside the virtual environment and I have to specify which pytest to run?
How to I get py.test to use only the python and packages that are in my virtualenv?
Also, since I have several version of Python on my system, how do I tell which Python that Pytest is using? Will it automatically use the Python within my virtual environment, or do I have to specify somehow?

There is a bit of a dance to get this to work:
activate your venv : source venv/bin/activate
install pytest : pip install pytest
re-activate your venv: deactivate && source venv/bin/activate
The reason is that the path to pytest is set by the sourceing the activate file only after pytest is actually installed in the venv. You can't set the path to something before it is installed.
Re-activateing is required for any console entry points installed within your virtual environment.

Inside your environment, you may try
python -m pytest

In my case I was obliged to leave the venv (deactivate), remove pytest (pip uninstall pytest), enter the venv (source /my/path/to/venv), and then reinstall pytest (pip install pytest). I don't known exacttly why pip refuse to install pytest in venv (it says it already present).
I hope this helps

you have to activate your python env every time you want to run your python script, you have several ways to activate it, we assume that your virtualenv is installed under /home/venv :
1- the based one is to run the python with one command line
>>> /home/venv/bin/python <your python file.py>
2- add this line on the top of python script file
#! /home/venv/bin/python and then run python <you python file.py>
3- activate your python env source /home/venv/bin/activate and then run you script like python <you python file.py>
4- use virtualenvwrapper to manager and activate your python environments

Scrapy appears to be installed but can't be imported when in the python shell

I have used the command: sudo pip install scrapy to install scrapy inside of a virtual environment. After having done so, I used the command: scrapy --version
and got the following screen:
But when I try to import scrapy in the python shell, I get an import error. Is there a specific way to install scrapy within Ubuntu 14.04 besides the way provided in the docs?

Following this guide: http://docs.python-guide.org/en/latest/dev/virtualenvs/
pip install virtualenv
cd PROJECT_FOLDER
virtualenv venv
source venv/bin/activate
pip install scrapy
deactivate #when ready to leave
If you are in your virtual environment you should see (venv) next to your command prompt. (I got that in OSX Terminal bash)
As mentioned by others, running:
which scrapy
will tell you where the binary is. It should be in under the venv folder.
tested on my 2013 Macbook Pro