Is it possible to use scrapy without Anaconda? - python

So I am very new to scraping and trying to learn scrapy. Most of the tutorial on web I have seen use Anaconda for scrapy project. I just wish to know is it possible to use scrapy without Anaconda.

it's is normally possible,
You can download it with pip
check there : http://doc.scrapy.org/en/latest/intro/install.html
you should probably use a dedicated virtualenv to avoid conflicting with your system packages.
https://github.com/pypa/pipenv Pipenv is a good one

Yes, it is possible. Following steps would lead you to that(if you're using Ubuntu):
Install python-pip if (if not installed)
sudo apt-get install python 3-pip
Next install scrapy
sudo pip install Scrapy
How to install pip on Windows
Step #2 still remains the same for Windows.
Cheers!

Related

Correct location to install Python-docx globally

I am updating this question because the basis of the question was really in error to begin with.
If you're having issues with installing your modules, first check to see if you (like me) followed a tutorial saying to use sudo -i to install modules. This is HORRIBLE advice. HORRRR IIII BBBBLE advice. Your modules will be installed but not really accessible to the users, but then when you try to do things right, and use sudo pip install BeautifulSoup4 you'll get a message that the module is already installed. So if you know you've installed modules on your server / VPS, Linux, etc. but you can't access them, uninstall them, go to sudo -i and uninstall them, make sure that EVERYTHING. IS. CLEAN. of your modules, then go to your admin account and use sudo pip install <modules name>.
This will be MUCH better and you won't waste your time trying to figure out what you've done wrong. Most likely, what you did wrong was follow some stupid tutorial saying to install things as root# and it's messed you up from the start.
You'll need to uninstall it from your version of python 3.4.8, then reinstall it using sudo rights to do it globally.
python3 -m pip uninstall python-docx
sudo python3 -m pip install python-docx
I'm a little confused though, since in your screenshots, it shows MySQLdb as installed for python2.6, not python3.4. Are you aware of this, and is it part of your problem?

Ubuntu Python, install packages with user or sudo

I am new at Python and Ubuntu. I really struggle to decide the best way of installing Python packages on Ubuntu. I am using VIM so I cannot use Anaconda since they do no "talk".
I have two options:
Install with: pip3 install --user foo. This works find but according to python4astronomers you should never use --user. It will work very bad.
Use apt-get but then I got old packages
Use sudo pip, but that can get bad code into my system.
I do not want to use virtual environments. I want to be able to use all packages I have installed when I use Python. I normally use Jupyter.
I find it strange that Python installation is so difficult on Linux compared with Windows.
Which of these is the correct way?

Installing Python 2.7.11 without sudo and make

I'm doing a little project and I need to install a different version of python on a box on which I don't have superuser rights. Already tried installing python to a directory for which I do have permissions, but I would still need to build it using make, which the box doesn't have installed. Any ideas?
Sounds like you may want to use pyenv to install python
Since you can run pip, you can install a virtual environment to play in. For that, I recommend virtualenvwrapper which is installable via pip install virtualenvwrapper.
The docs on the linked site have more details on what you need to do to get it up and running.

Installing Scrapy on Python VirtualEnv

Here's my problem,
I have a shared hosting (GoDaddy Linux Hosting package) account and I'd like to create .py file to do some scraping for me. To do this I need the scrapy module (scrapy.org). Because of the shared account I can't install new modules so I installed VirtualEnv and created a new virtual env. that has pip, wheel, etc. preinstalled.
Running pip install scrapydoes NOT complete successfully because scrapy has lot of dependencies like libxml2 and it also needs python-dev tools. If I had access to 'sudo apt-get ...' this would be easy but I dont'. I can only use pip and easy_install.
So How do I install the python dev tool? And how do I install the dependencies? Is this even possible?
Cheers
You can install all the dependencies by activating the python virtual environment first.
Step 1 :
On Linux :
env/bin/activate
On Windows :
env\Scripts\activate
Step 2:
pip install lxml
I just tried and it worked for me. Please find screen shot attached.
It's not possible to do what I wanted to do on the GoDaddy plan I had.
I had some of the same issues. I found this and modified to pip3.7 install lxml==3.4.2. I was able to install successfully.

How to install pip in a new python installation

I recently installed python 2.7.2 on my Mac running OSX 10.6.8. Previously, I had version 2.6. I set my path in .bash_profile as follows:
export PATH=/usr/local/bin:$PATH
export PATH=/usr/local/share/python:$PATH
so that when I run python it will refer to my new installation. It does.
I would also like to use pip with my new installation, but the problem is that I already have the current version of pip installed at
/usr/local/bin/pip.
I tried to re-install pip with:
easy_install pip
But, of course this does not put pip in the desired new directory
/usr/local/share/python/pip
but simply refers to the existing version in /usr/local/bin/pip.
Can someone tell me how to fix this?
I would like to then use pip to install NumPy and SciPy in the correct directory (I was having trouble getting the SciPy installation to work with my old version of python, hence the new install).
If you'd like, you can visit the website where I found instructions for installing python 2.7, creating/updating my .bash_profile, installing pip, and NumPy and SciPy. Might provide some insight, or I'm happy to give more details if needed. Thanks!
http://www.thisisthegreenroom.com/2011/installing-python-numpy-scipy-matplotlib-and-ipython-on-lion/#python
Install distribute as per the instructions at http://pypi.python.org/pypi/distribute .
Make sure you specify the full path to the python executable (/usr/local/share/python/python or smth in your case).
$ curl -O https://svn.apache.org/repos/asf/oodt/tools/oodtsite.publisher/trunk/distribute_setup.py
$ /usr/local/share/python/python distribute_setup.py
Then you should have /usr/local/share/python/easy_install.
After that, run:
$ /usr/local/share/python/easy_install pip
Then you should have /usr/local/share/python/pip.
Depending on the ordering of things in your PATH, either your old, or the newly installed pip is executed when you execute the pip command, so you either might have to adapt your PATH, or specify the full path to /usr/local/share/python/pip when installing eggs.
(shameless plug:
In any case, you might consider using virtualenv for installing packages into a "project" specific isolated environment, as opposed to installing them globally.)
I needed to uninstall brew's python.
Then, I was left with python v2.7.6
Next to install, pip I ran
sudo easy_install pip
installed fine and working
I had a similar issue, try this:
$ python -m pip install --upgrade --force-reinstall pip
This will force reinstall pip with whatever version of python you use including installing the binary.
A few days ago I had a friend who was starting Python Programming and needed help with the same issue: installing pip. There are debates over which one to choose between easy_install and pip and it seems everybody is heading the pip direction. Either way, installing either of them can be frustrating.
You can use this simple tutorial : installing pip package manager the easy way
Here are what you should keep in mind as you follow the above guide:
If you already have an older version installed, uninstall it or totally remove the python installation
Once that is cleared, download an install Python.
After that, download ez_setup.py file and save it to your desktop - easily accessible from the command line
Now run it from the command line and it will install easy_install for you after which,
You can use it to install pip.
Once again, you can do this or use the above link to find a simple step-by-step guide on how to get it installed on your computer.
Good luck.
Just so that people knew, ATM we can install PIP by downloading get-pip.py from the page with docs and run it like this:
c:\python27\python.exe get-pip.py
BTW, Python 3.4 comes with PIP pre-installed.
One of the command line options lets you choose where to install to.
--install-dir (-d) install package to DIR
So something like - # easy_install pip -d /usr/local/share/python
(Please correct me if I'm wrong.)
Just wanted to say that I found a way to get around my problem. I don't know that I can explain it perfectly, since I am not very good at understanding what I am doing with this stuff just yet! But, the problem seems to have been with my PATH. I removed the PATH that I posted in my original question, and then used easy_install pip. It went straight to python 2.7.2 (my new version) with no problem. I then successfully used pip to install NumPy and SciPy in the correct location, and they both work. Thanks to ErikAllik and FakeRainBrigand for taking the time to look into it!

Categories