I'm using virtualenvwrapper to manage a project where I'm just running this for now
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
with beam.Pipeline(options=PipelineOptions()) as p:
pass
This is returning the following error
Traceback (most recent call last):
File "path/to/pipeline.py", line 1, in <module>
import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
I have run pip install apache-beam. Running pip list returns
➜ pip list
Package Version
------------------------------ ---------
apache-beam 2.23.0
avro-python3 1.9.2.1
...
I'll add the following outputs as suggested in a similar question.
pip freeze
➜ pip freeze
apache-beam==2.23.0
avro-python3==1.9.2.1
...
pip -V
➜ pip -V
pip 20.2.2 from /Users/miguel/.virtualenvs/myenv/lib/python3.7/site-packages/pip (python 3.7)
python -V
➜ python -V
Python 3.7.3
which python
➜ which python
/Users/miguel/.virtualenvs/myenv/bin/python
which pip
➜ which pip
/Users/miguel/.virtualenvs/myenv/bin/pip
I don't know if this is relevant but I'm using VSCode and I have selected my python interpreter according to VSCode instructions here. Additionally, I installed python following the instructions here.
Any idea why this is happening?
First test if your installed Python modules work outside of an ide like VSCode.
We can do this by opening the terminal/command prompt and activating our virtual enviornment. In this case, as you're using virtualenvwrapper you need to use the command:
workon myenv
Documentation for the virtualenvwrapper can be found here. Once activated, we can open the Python interpreter in our terminal by using the command:
python
Once running, try and import apache_beam as before using:
import apache_beam as beam
If this works we now know it's a virtual environment set up issue using VSCode. For setting up Virtual Environments for Python in VSCode, use the official documentation Using Python environments in VS Code. This should allow you to specify the virtual environment.
If this fails to work, another option is to create a new virtual environment in VSCode and install your modules. Installing modules to a environment in VSCode can be found in the VSCode instructions you linked to in your post.
I tried to re-create your issue as above using the following:
Python 3.7.7
VSCode 1.48.0
apache-beam 2.23.0
My virtual environment was named 'stack'. Once I had created it the only module I installed was apache-beam using:
pip install apache-beam
When creating the project I made the directory and launched via terminal using:
- mkdir hello
- cd hello
- code .
And then added the Python interpreter via the command:
Python: Select Interpreter
One additional step I needed (as I usually use PyCharm for Python) was to install:
Shell Command: Install 'code' command in PATH** command.
To allow the launch of code via terminal. Below is a screenshot of the project after following these steps and running the same commands as posted in your question for reference.
My .vscode/settings.json is as follows:
{
"python.pythonPath": "/Users/robertyoung/envs/stack/bin/python"
}
Related
I am trying to import the module mtcnn in VSCode.
I have run the following commands in the terminal:
pip3 install MTCNN
and
python3.8 -m pip install mtcnn
Which downloads MTCNN
Terminal showing its already installed
But when I try run my python file in VSCode, I run into this error:
Error
I am using python version 3.8.5 in VSCode. There is no red line error under the import line in VSCode so I'm confused why it's not working.
Open an integrated terminal in VS Code, run python --version to check if it's as the same as the one you selected for python interpreter, which is shown in status bar.
Run pip show mtcnn. If you get detailed information like name, version and location, reload window should solve your question.
If you get WARNING: Package(s) not found: mtcnn, it means no such module in your current selected environment, install it.
Besides, to avoid messing up the global python environment, you may create a virtual environment. See Create a Virtual Environment.
I am trying to use a python script to run another python project which needs some specific conda environment settings. I am using the code below to do that, but everytime I run this, it outputs ImportErrors and it seems that it is not using any of the modules that my conda environment has already installed
import os
os.system('source ~/anaconda3/etc/profile.d/conda.sh; conda activate tf-2; python demo.py --image /Users/mingaioh/Desktop/car.png')
This may help... try python3 -m pip install os or python -m pip install os
Update update: json_lines is not supported by python versions < 3 - my issue had pretty much nothing to do with environments. I am now using 3.9.1 and all is gucci.
Update: After using which python in my jupyter notebook and in my Terminal, I see that they are both using the same environment. As such I am still at a loss as to why my notebook cannot find json_lines.
I have two python environments on my computer, a default one and one I have for running my jupyter notebook on. I am trying to install the library, json_lines to the latter environment. I am not used the Anaconda environment manager.
On my Mac's Terminal I used the general pip install command pip install json-lines, but when I try to execute the following line of Python import json_lines in my notebook, I still receive the following error ImportError: No module named json_lines.
As I suspect I am not installing to the correct environment, I tried installing the library from inside my notebook with the following, import sys; !{sys.executable} -m pip install jsonlines.
However, this has not changed my dilemma.
Is there some way I can specify from my Terminal which environment to install to? or is it likely I am encountering a different issue to what I suspect?
The package for json_lines in pip in json-lines. Hence you could install it as:
$ pip install json-lines
It may be appropriate to use an isolated python environment for your particular project if you want to use particular conda libraries but without the whole package. In this instance, you would be able to use virtualenv. This will allow you to create an isolated python environment.
$ pip3 install virtualenv
You can call virtualenv to create a virtual python environment with the working name e.g. myvenv.
$ virtualenv myvenv
From here, you can set your terminal to use this python version. If you are on *nix:
$ which python
/usr/bin/python
$ source myvenv/bin/activate
(myvenv)$ which python
/.../myvenv/bin/python
This article can help you out.
https://janakiev.com/blog/jupyter-virtual-envs/
You need to create a virtualenv which will be used by your notebooks.
I see a ModuleNotFoundError: No module named 'xlwt'
I created a virtual environment and installed the packages using the following commands:
py -3 -m venv .venv
.venv\scripts\activate
python -m pip install xlwt
This error is not specific to this module. Even matplotlib, seaborn, spacy and other libraries are resulting in the same issue.
I have also checked for the interpreter using the "Select Interpreter" command. It seems like the right interpreter is being used - the one I just created.
Typing pip list in the terminal also lists these libraries as expected.
Running the code (ctrl+Alt+N) is when I see the ModuleNotFoundError.
Any idea if I'm missing something here?
When started, Jupyter notebook encounters a problem with module import
import findspark
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-ff073c74b5db> in <module>
----> 1 import findspark
ModuleNotFoundError: No module named 'findspark'
Conda list shows that module is here
filelock 3.0.8 py37_0
findspark 1.3.0 py_1 conda-forge
flask 1.0.2 py37_1
Python version
(myenv) mm#mm-HP-EliteBook-8560p:~$ python -V
Python 3.6.8
It seems that my installation is not clean.
Three Python lines from .bash_profile
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
Why do I get import error?
I'd suggest a slightly different route.
Download spark on your local. Let's say you've unzipped in /Users/me/spark-2.4.0-bin-hadoop2.7 location.
Assuming you're on mac, update your ~/.bash_profile to contain these entries:
export SPARK_HOME=/Users/me/spark-2.4.0-bin-hadoop2.7
export PYTHONPATH=${SPARK_HOME}/python:$PYTHONPATH
export PYTHONPATH=${SPARK_HOME}/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export PYSPARK_PYTHON=<path to your python location>
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
PATH=$PATH:$SPARK_HOME/bin
Execute a source ~/.bash_profile.
From your bash shell, just run pyspark and it'll open the jupyter notebook. Now your notebook will be tied to this spark installation.
If you're using linux, I think the only change is in the syntax for appending stuffs to path, and instead of changing bash_profile you probably need to change bashrc file.
Make sure you are using the correct virtualenv.
Create a fresh virtualenv for your work (eg. using 3.7.4 as an example here. Use a version you have installed):
pyenv virtualenv 3.7.4 myenv
You can see which python versions you have installed with:
pyenv versions
And which versions are available for installation with:
pyenv install -l
You can either activate the virtualenv shell with:
pyenv shell myenv
With the virtualenv active, you should see the virtualenv name before your prompt. Something like "(myenv)~$: "
Now install all the python packages as you normally would. Make sure you are in the right virutalenv before you run your packages. You can also set the PYENV_VERSION environment variable to specify the virtualenv to use. Something like:
PYENV_VERSION=myenv python -m pip install findspark
Then
PYENV_VERSION=myenv python -m pip show findspark
Should give you something like:
Name: findspark
Version: 1.3.0
Summary: Find pyspark to make it importable.
Home-page: https://github.com/minrk/findspark
Author: Min RK
Author-email: benjaminrk#gmail.com
License: BSD (3-clause)
Location: /home/tzhuang/.pyenv/versions/3.7.4/envs/myenv/lib/python3.7/site-packages
Requires:
Required-by: