I'm trying to install a module called Scrapy. I installed it using
pip install Scrapy
I see the 'scrapy' folder in my /usr/local/lib/python2.7/site-packages, but when I try to import it in a Python program, is says there is no module by that name. Any ideas as to why this might be happening?
EDIT: Here is the output of the pip command:
Downloading/unpacking Scrapy
Downloading Scrapy-0.20.0.tar.gz (745kB): 745kB downloaded
Running setup.py egg_info for package Scrapy
no previously-included directories found matching 'docs/build'
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/local/lib/python2.7/site-packages (from w3lib>=1.2->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: Scrapy
Running setup.py install for Scrapy
changing mode of build/scripts-2.7/scrapy from 644 to 755
no previously-included directories found matching 'docs/build'
changing mode of /usr/local/bin/scrapy to 755
Successfully installed Scrapy
Cleaning up...
When I run /usr/local/bin/scrapy I get the usage for the command and the available commands. I noticed that I have a python2.7 and python2.7-32 in my /usr/local/bin, and I remember installing the 32 bit version because of a problem with Mavericks.
Here is the output of python /usr/local/bin/scrapy:
Traceback (most recent call last): File "/usr/local/bin/scrapy", line 3, in <module> from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline
And head /usr/local/bin/scrapy:
#!/usr/local/opt/python/bin/python2.7 from scrapy.cmdline import execute execute()
Are you using Homebrew or MacPorts or something? As #J.F.Sebastian said, it sounds like you are having issues mixing the default python that comes with OS X, and one that is installed via a package manager... Try /usr/local/opt/python/bin/python2.7 -m scrapy and see if that throws an ImportError.
If that works, then you may want to consider making that python executable your default. Something like alias python2.7=/usr/local/opt/python/bin/python2.7 and then always use python2.7 instead of the default python. You can likewise just point python to the /urs/local... bit, but then you won't have easy access to the system (OS X-supplied) python if you ever needed it for some reason.
EDIT: You can force pip to install to an alternate location. The details are here: Install a Python package into a different directory using pip?. If you do indeed have extra Python folders on your system, maybe you can try directing scrapy to those, even if just for a temporary solution.
Can you post the output of the pip command? Perhaps it is failing somewhere?
Also, is it possible you have two versions of Python on your machine? Pip only installs to one location, but perhaps the version of Python on your path is different.
Finally, sometimes package names given to pip are not exactly the same as the name used to import. Check the documentation of the package. I took a quick look and the import should be lowercase:
import scrapy
When all else fails you can always set the environment variable PYTHONPATH (see Permanently add a directory to PYTHONPATH for help) to the path where you installed Scrapy. (pending you're not using virtualenv -- and if you are please specify so we can help, it's generally a good idea to provide OS too)
if you run on Ubuntu:
use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.
Optionally, even if it solves your problem, it is always better to install python libraries on a virtual environment, using virtualenvwrapper to keep the libraries separated, try to examine the apt-get installation log to find out what tools where added, then remove scrapy python library and reinstall it in the virtual env. using pip
It appears that the scrapy module that is installed on the Python path is an executable file that will bootstrap a Scrapy project directory for you.
The Python code in the scrapy executable looks like this:
#!/usr/bin/env python
from scrapy.cmdline import execute
execute()
That's intended to be run from the command line rather than imported into your own Python project module.
According to the documentation for the project, running the scrapy executable with this syntax:
scrapy startproject <your-project-name>
will bootstrap a Scrapy project that has the following directory structure:
your-project-name/
scrapy.cfg
tutorial/
__init__.py
items.py
pipelines.py
settings.py
spiders/
__init__.py
...
There are a number of examples in the documentation that demonstrate how you create and run your own spiders, link extractors, etc., and how to manipulate the data that you retrieve with the application. They each demonstrate the appropriate Python imports from subdirectories in the scrapy package to get you up and running.
Hope that this helps.
Related
i'm unable to import flask_sqlalchemy even after installing flask-sqlachemy
from flask import Flask, render_template
from flask_sqlalchemy import SQLAlchemy
# ^this line shows error ImportError: "flask_sqlalchemy" could not be resolved
app = Flask(__name__)
#app.route('/')
def index():
return render_template('index.html')
i'm not using any virtual environment and flask-sqlalchemy shows already installed
(base) sachin#sachin:~/Coding/flask/blog$ pip install Flask-SQLAlchemy
Requirement already satisfied: Flask-SQLAlchemy in /home/sachin/anaconda3/lib/python3.8/site-packages (2.4.4)
Requirement already satisfied: SQLAlchemy>=0.8.0 in /home/sachin/anaconda3/lib/python3.8/site-packages (from Flask-SQLAlchemy) (1.3.20)
Requirement already satisfied: Flask>=0.10 in /home/sachin/anaconda3/lib/python3.8/site-packages (from Flask-SQLAlchemy) (1.1.2)
Requirement already satisfied: Jinja2>=2.10.1 in /home/sachin/anaconda3/lib/python3.8/site-packages (from Flask>=0.10->Flask-SQLAlchemy) (2.11.2)
Requirement already satisfied: itsdangerous>=0.24 in /home/sachin/anaconda3/lib/python3.8/site-packages (from Flask>=0.10->Flask-SQLAlchemy) (1.1.0)
Requirement already satisfied: click>=5.1 in /home/sachin/anaconda3/lib/python3.8/site-packages (from Flask>=0.10->Flask-SQLAlchemy) (7.1.2)
Requirement already satisfied: Werkzeug>=0.15 in /home/sachin/anaconda3/lib/python3.8/site-packages (from Flask>=0.10->Flask-SQLAlchemy) (1.0.1)
Requirement already satisfied: MarkupSafe>=0.23 in /home/sachin/anaconda3/lib/python3.8/site-packages (from Jinja2>=2.10.1->Flask>=0.10->Flask-SQLAlchemy) (1.1.1)
pip freeze shows me its already installed -
Flask==1.1.2
Flask-SQLAlchemy==2.4.4
I had the same error. Changed my system python by doing the following in VS Code:
Under the view menu select 'show command pallet'. One of the commands you can then select from the command palette is 'Python: Select Interpreter'.
Selecting this option will allow you to choose which version of python to use.
This post is pretty old but I found a different solution that might help others out. For me, the issue was my IDE/Python (venv). I am using Visual Studio ver. 1.57.1 as of now on Windows 10.
Open Visual Code.
Go to 'View' on the toolbar.
Click on 'Command Palette...' (or press CTRL + SHIFT + P on Windows).
Search for: 'Python Select Interpreter'.
Click on: 'Enter interpreter path...' followed by 'Find...'.
Go to your (venv) scripts folder: project_folder/venv/Scripts.
Select python.exe or pythonw.exe inside of your project_folder/venv/Scripts folder.
That's all.
The Flask-SQLAlchemy was installed in my virtual environment (venv). But my VSCode was using the system/PC Python interpreter to find the packages instead of the venv.
Hope this helps someone else out.
I had the same issue, this worked for me:
In cmd, cd to your project folder, (for me i had to activate venv, so it showed (env) C:\ etc\etc>)
Then install it here with:
pip3 install flask-SQLAlchemy
for some reason this took this noob 12 tries, but the give away was there was no sqla files in my site folder, and it kept showing the install had taken place in my python39 folder.
This is for people who do not use Anaconda (I had this issue and I solved it this way)
Your venv might be incorrect. Did you copy paste your venv from another directory?
venv saves path information, so when you run env\Scripts\activate, the venv believes it is in a different directory.
It will install the packages in the OLD directory. Make sure you generate a new py -m venv env for every app you create rather than copy pasting.
The solution is to delete your current venv directory, and run the py -m venv env command again. Then the packages will be installed in the current venv.
Had the same issue, resolved using:
activate the venv
use pip3 to install flask and flask_sqlalchemy by running:
pip3 install flask_sqlalchemy flask
And lastly checking if correct python interpreter was selected in vscode from:
CMD + Shift + P --> Python: Select Interpreter --> selecting appropriate python version for the venv (preferably python10)
I had the same problem, but it worked when I changed to Python Interpreter under the command palette of the view tab. If you believe you have installed all required packages and it does not work then definitely try this.
One thing I have noticed is that if I change the python interpreter from command palette(ctrl+shift+p) to my virtual environment one it does not work. First I tried the command,
py -3 -m pip install flask-SQLALchemy
Instead of the above command, I upgrade pip by using,
python -m pip install --upgrade pip
then I tried using the following to import SQLAlchemy.
python -m pip install flask-SQLALchemy
and it worked for me.
When you have done changes, I mean change your python interpreter, delete your env or re-create install packages again. Please close your app.py file and re-open it. Some times it still gives your error.
For anyone looking for a more recent answer. I couldn't find a scripts folder. On my ubuntu machine I did this (in vscode):
ctrl + shft + p
search select python interpreter
enter interpreter path
navigate to your-app/venv/python3
select python3
This resolved the import issue.
I have been through the various Stack Overflow questions and non of the answers resolve the issue for me.
It is likely a path 'thing' that I am not seeing.
Have installed pandas, it says its there, but numpy will import and pandas won't.
Any ideas? What am I missing?
If I enter:
!pip3 install pandas
I get:
Requirement already satisfied: pandas in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (1.0.3)
Requirement already satisfied: pytz>=2017.2 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from pandas) (2020.1)
Requirement already satisfied: numpy>=1.13.3 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from pandas) (1.18.4)
Requirement already satisfied: python-dateutil>=2.6.1 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from pandas) (2.8.1)
Requirement already satisfied: six>=1.5 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from python-dateutil>=2.6.1->pandas) (1.14.0)
Okay, I'm going to give a completely new, opinionated, answer.
It looks like you've got at least two installations of Python, one in ..python\python38, and one in ..\python\python38-32. pandas is installed on the first, but jupyter is running on the second. You've got a version of pip for each, plus a third one in Microsoft\WindowsApps.
This happens.
On Windows, you are probably safe to just uninstall all the versions of Python on your computer. Then download Miniconda for Windows. Miniconda is, like pip, a package manager for Python; but unlike pip, is able to also install non-Python files, which makes all sorts of things much easier. It's called 'Miniconda' to distinguish it from 'Anaconda', which is conda plus almost all the packages a data scientist might need -- Miniconda just gives you the package manager conda, and you can then choose to install what you actually want.
Once conda is installed, create a new conda environment (conda create -n MYNEWENV -- if you have twenty minutes then read the quickstart explanation) and then try to download everything with conda install...
You can specify which 'channel' you download packages from, but you probably don't need to. Just try conda install jupyter numpy pandas; or search the Conda cloud for the specifics of whatever you want.
Make sure that the python installation which is running jupyter is the one that has pandas. Run where jupyter (which on linux/mac), and make sure it matches the paths above. Also try where pip to make sure the installations of pip, python, and jupyter match.
To be sure you're using a specific version of python+pip to install a package, you can call pip the module rather than the script, e.g. with c:\users\ron mcmillan\appdata\local\programs\python\python38\python -m pip install [pkg]
I'd consider using an environment manager such as virtualenv or conda to simplify environments & installation.
Try this:
# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install pandas
There is a lengthy explanation here.
Going through the lengthy explanation above - will need to go through it again.
import sys
!{sys.executable} -m pip install numpy
returns:
'c:\users\ron' is not recognized as an internal or external command,
operable program or batch file.
import sys
sys.path
returns:
['C:\\Users\\Ron McMillan',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-
32\\python38.zip',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\DLLs',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32',
'',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\win32',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\win32\\lib',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\Pythonwin',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\IPython\\extensions',
'C:\\Users\\Ron McMillan\\.ipython']
!type python
returns
The system cannot find the file specified.
And in the Command Prompt:
where jupyter
C:\users\ron mcmillan\appdata\local\programs\python\python38-32\Scripts\jupyter.exe
and
where pip
C:\users\ron mcmillan\appdata\local\programs\python\python38\Scripts\pip.exe
C:\users\ron mcmillan\appdata\local\programs\python\python38-32\Scripts\pip.exe
C:\users\ron mcmillan\appdata\local\Microsoft\WindowsApps\pip.exe
So, Jupyter note finding Python?
Ok, thank you.
Yes, I knew it was messy - to the point I din't get it. The picture makes sense.
So I am trying to install pylint using pip, as my work machine is offline I have downloaded pylint using pip and transferred this using a CD. As part of pylint download it also brought down asteroid, colorama, isort, lazy_object_proxy, McCabe, six, typed ast and wrapt.
However when running the install for pylint using the following command inside the directory with all the above files in:
python -m pip install --no-index --find-links . -r requirements.txt
This starts to work with it collecting pylint, isort and a couple of others, however after collecting asteroid it goes to collect lazy object proxy (which is in the directory) and gives the following error:
Could not find a version that satisfies the requirement lazy-object-proxy (from asteroid<3,>=2.2.0->pylint->-r requirements.txt (line 1)) (from versions: ) No matching distribution found for lazy-object-proxy (from asteroid<3,>=2.2.0->pylint->-r requirements.txt (line 1))
The version of lazy object proxy downloaded is 1.4.1
Im fairly new to this so maybe there is something in this error that highlights why it doesn't see or like the version that is downloaded and in the directory, any help would be much appreciated.
OS is windows 7, running python version 3.6.0
NOTE: even trying to just install lazy object proxy on its own fails saying it doesn't exist, its like its not there although it is in the folder.
I've written something in TensorFlow that makes use of some nice group theory functions that work very easily in Sage (and seem prohibitively difficult to code from scratch). The Sage part works on its own, and the TensorFlow part works on its own, but I can't figure out how to get them working together.
Specifically: I can make a file test.py using Sage functions and run it from the command line using:
sage --python test.py
with no problem. But calling a function defined in test.py from a .py file using TensorFlow fails ("Import error, no module named Sage"), presumably because Sage (6.x) uses Python 2.6.x, while TensorFlow uses Python 2.7 or 3.3+.
Is there a way around this?
Thanks!
EDIT: I'm not sure if this is relevant, but if I fire up normal Python (the kind TensorFlow uses), I get this:
from sage.env import SAGE_LOCAL
SAGE_LOCAL
which outputs '$SAGE_ROOT/local'.
However if I fire up Sage first I get this:
sage
SAGE_LOCAL
which outputs ''/usr/lib/sagemath/local'.
I just upgraded to Sage 7.0 if that matters (this didn't work in 6.10 either, though).
Here's something NOT to do (yet); don't just take whatever Sage install you happen to have and do:
$ sage -pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Even though this "works", it also had several worrying messages about upgrading numpy and six, which completely broke the numpy part of my Sage installation. This was with Sage-6.9.
Which means you have to make sure you have a Sage that has the right versions of Numpy and six. With the latest development version, we do, apparently:
$ cd /path/to/my/bleeding/edge/sage/directory
$ ./sage -pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Collecting tensorflow==0.7.1 from https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Using cached https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six>=1.10.0 in ./local/lib/python2.7/site-packages/six-1.10.0-py2.7.egg (from tensorflow==0.7.1)
Collecting protobuf==3.0.0b2 (from tensorflow==0.7.1)
Using cached protobuf-3.0.0b2-py2.py3-none-any.whl
Collecting wheel (from tensorflow==0.7.1)
Using cached wheel-0.29.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.10.1 in ./local/lib/python2.7/site-packages (from tensorflow==0.7.1)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./local/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg (from protobuf==3.0.0b2->tensorflow==0.7.1)
Installing collected packages: protobuf, wheel, tensorflow
Successfully installed protobuf-3.0.0b2 tensorflow-0.7.1 wheel-0.29.0
You are using pip version 8.0.2, however version 8.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
And then I don't get any failures.
So one has to be careful, but apparently it's possible. However, you definitely have to use it from "within" Sage; Sage-as-distribution wouldn't allow you to use your other tensorflow with it.
I'm trying to run the Flask-Bootstrap sample application from within Spyder, but I'm not seeing anything when I hit run. It sits for a long time, but I'm pretty sure it's supposed to bring up a webpage, correct? I don't get any error message. It just never does anything as far as I can tell.
When I run via python run_sample_app.py, I get an error message No module named flask_appconfig despite having installed Flask using conda install flask.
I tried installing via pip install flask-appconfig, but I have apparently already installed:
Requirement already satisfied (use --upgrade to upgrade): flask-appconfig in /Applications/Anaconda/anaconda/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): flask in /Applications/Anaconda/anaconda/lib/python2.7/site-packages (from flask-appconfig)
Requirement already satisfied (use --upgrade to upgrade): six in /Applications/Anaconda/anaconda/lib/python2.7/site-packages (from flask-appconfig)
Cleaning up...
Any ideas?
Thanks for the help, guys! I was just being dumb. I thought surely the application should pull up a browser for me, but I was wrong. It just posts the HTML/CSS to my localhost (127.0.0.1), and I didn't realize that I had to navigate there on my own. Everything is working now. Thanks again!