I've written something in TensorFlow that makes use of some nice group theory functions that work very easily in Sage (and seem prohibitively difficult to code from scratch). The Sage part works on its own, and the TensorFlow part works on its own, but I can't figure out how to get them working together.
Specifically: I can make a file test.py using Sage functions and run it from the command line using:
sage --python test.py
with no problem. But calling a function defined in test.py from a .py file using TensorFlow fails ("Import error, no module named Sage"), presumably because Sage (6.x) uses Python 2.6.x, while TensorFlow uses Python 2.7 or 3.3+.
Is there a way around this?
Thanks!
EDIT: I'm not sure if this is relevant, but if I fire up normal Python (the kind TensorFlow uses), I get this:
from sage.env import SAGE_LOCAL
SAGE_LOCAL
which outputs '$SAGE_ROOT/local'.
However if I fire up Sage first I get this:
sage
SAGE_LOCAL
which outputs ''/usr/lib/sagemath/local'.
I just upgraded to Sage 7.0 if that matters (this didn't work in 6.10 either, though).
Here's something NOT to do (yet); don't just take whatever Sage install you happen to have and do:
$ sage -pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Even though this "works", it also had several worrying messages about upgrading numpy and six, which completely broke the numpy part of my Sage installation. This was with Sage-6.9.
Which means you have to make sure you have a Sage that has the right versions of Numpy and six. With the latest development version, we do, apparently:
$ cd /path/to/my/bleeding/edge/sage/directory
$ ./sage -pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Collecting tensorflow==0.7.1 from https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Using cached https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six>=1.10.0 in ./local/lib/python2.7/site-packages/six-1.10.0-py2.7.egg (from tensorflow==0.7.1)
Collecting protobuf==3.0.0b2 (from tensorflow==0.7.1)
Using cached protobuf-3.0.0b2-py2.py3-none-any.whl
Collecting wheel (from tensorflow==0.7.1)
Using cached wheel-0.29.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.10.1 in ./local/lib/python2.7/site-packages (from tensorflow==0.7.1)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./local/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg (from protobuf==3.0.0b2->tensorflow==0.7.1)
Installing collected packages: protobuf, wheel, tensorflow
Successfully installed protobuf-3.0.0b2 tensorflow-0.7.1 wheel-0.29.0
You are using pip version 8.0.2, however version 8.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
And then I don't get any failures.
So one has to be careful, but apparently it's possible. However, you definitely have to use it from "within" Sage; Sage-as-distribution wouldn't allow you to use your other tensorflow with it.
Related
I want to install a specific version of PyTorch (inside a Dockerfile) to handle modern GPUs: 1.8.1+cu111. Please note the local version identifier. I then install (both via pip) a library that requests torch>=1.8.1 as a dependency. For some reason, this uninstalls my 1.8.1+cu111 and replaces it with vanilla 1.10.0, despite the custom version with "local version specifier" should be taken as valid as PEP 0440 says (...local version labels MUST be ignored entirely when checking...) and it's what is happening normally, e.g. here or even for me when I try to re-install PyTorch by hand:
pip install "torch>=1.8.1"
Requirement already satisfied: torch>=1.8.1 in /usr/local/lib/python3.7/dist-packages (1.8.1+cu111)
But no, when I install my library, I get this:
...
Collecting torch>=1.8.1
Using cached torch-1.10.0-cp37-cp37m-manylinux1_x86_64.whl (881.9 MB)
...
Installing collected packages: torch,...
Attempting uninstall: torch
Found existing installation: torch 1.8.1+cu111
Uninstalling torch-1.8.1+cu111:
Successfully uninstalled torch-1.8.1+cu111
How can I make it keep the pre-installed 1.8.1+cu111?
In the PEP040, you have above:
When multiple candidate versions match a version specifier, the preferred version SHOULD be the latest version as determined by the consistent ordering defined by the standard Version scheme. Whether or not pre-releases are considered as candidate versions SHOULD be handled as described in Handling of pre-releases.
I think this because you are using an inclusive specifier where local version identifiers are NOT permitted
Can you specify the exact version with pip install "torch==1.8.1+cu111" ? Maybe only 1.8.1 works.
I have been through the various Stack Overflow questions and non of the answers resolve the issue for me.
It is likely a path 'thing' that I am not seeing.
Have installed pandas, it says its there, but numpy will import and pandas won't.
Any ideas? What am I missing?
If I enter:
!pip3 install pandas
I get:
Requirement already satisfied: pandas in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (1.0.3)
Requirement already satisfied: pytz>=2017.2 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from pandas) (2020.1)
Requirement already satisfied: numpy>=1.13.3 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from pandas) (1.18.4)
Requirement already satisfied: python-dateutil>=2.6.1 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from pandas) (2.8.1)
Requirement already satisfied: six>=1.5 in c:\users\ron mcmillan\appdata\local\programs\python\python38\lib\site-packages (from python-dateutil>=2.6.1->pandas) (1.14.0)
Okay, I'm going to give a completely new, opinionated, answer.
It looks like you've got at least two installations of Python, one in ..python\python38, and one in ..\python\python38-32. pandas is installed on the first, but jupyter is running on the second. You've got a version of pip for each, plus a third one in Microsoft\WindowsApps.
This happens.
On Windows, you are probably safe to just uninstall all the versions of Python on your computer. Then download Miniconda for Windows. Miniconda is, like pip, a package manager for Python; but unlike pip, is able to also install non-Python files, which makes all sorts of things much easier. It's called 'Miniconda' to distinguish it from 'Anaconda', which is conda plus almost all the packages a data scientist might need -- Miniconda just gives you the package manager conda, and you can then choose to install what you actually want.
Once conda is installed, create a new conda environment (conda create -n MYNEWENV -- if you have twenty minutes then read the quickstart explanation) and then try to download everything with conda install...
You can specify which 'channel' you download packages from, but you probably don't need to. Just try conda install jupyter numpy pandas; or search the Conda cloud for the specifics of whatever you want.
Make sure that the python installation which is running jupyter is the one that has pandas. Run where jupyter (which on linux/mac), and make sure it matches the paths above. Also try where pip to make sure the installations of pip, python, and jupyter match.
To be sure you're using a specific version of python+pip to install a package, you can call pip the module rather than the script, e.g. with c:\users\ron mcmillan\appdata\local\programs\python\python38\python -m pip install [pkg]
I'd consider using an environment manager such as virtualenv or conda to simplify environments & installation.
Try this:
# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install pandas
There is a lengthy explanation here.
Going through the lengthy explanation above - will need to go through it again.
import sys
!{sys.executable} -m pip install numpy
returns:
'c:\users\ron' is not recognized as an internal or external command,
operable program or batch file.
import sys
sys.path
returns:
['C:\\Users\\Ron McMillan',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-
32\\python38.zip',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\DLLs',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32',
'',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\win32',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\win32\\lib',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\Pythonwin',
'c:\\users\\ron mcmillan\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\IPython\\extensions',
'C:\\Users\\Ron McMillan\\.ipython']
!type python
returns
The system cannot find the file specified.
And in the Command Prompt:
where jupyter
C:\users\ron mcmillan\appdata\local\programs\python\python38-32\Scripts\jupyter.exe
and
where pip
C:\users\ron mcmillan\appdata\local\programs\python\python38\Scripts\pip.exe
C:\users\ron mcmillan\appdata\local\programs\python\python38-32\Scripts\pip.exe
C:\users\ron mcmillan\appdata\local\Microsoft\WindowsApps\pip.exe
So, Jupyter note finding Python?
Ok, thank you.
Yes, I knew it was messy - to the point I din't get it. The picture makes sense.
So I am trying to install pylint using pip, as my work machine is offline I have downloaded pylint using pip and transferred this using a CD. As part of pylint download it also brought down asteroid, colorama, isort, lazy_object_proxy, McCabe, six, typed ast and wrapt.
However when running the install for pylint using the following command inside the directory with all the above files in:
python -m pip install --no-index --find-links . -r requirements.txt
This starts to work with it collecting pylint, isort and a couple of others, however after collecting asteroid it goes to collect lazy object proxy (which is in the directory) and gives the following error:
Could not find a version that satisfies the requirement lazy-object-proxy (from asteroid<3,>=2.2.0->pylint->-r requirements.txt (line 1)) (from versions: ) No matching distribution found for lazy-object-proxy (from asteroid<3,>=2.2.0->pylint->-r requirements.txt (line 1))
The version of lazy object proxy downloaded is 1.4.1
Im fairly new to this so maybe there is something in this error that highlights why it doesn't see or like the version that is downloaded and in the directory, any help would be much appreciated.
OS is windows 7, running python version 3.6.0
NOTE: even trying to just install lazy object proxy on its own fails saying it doesn't exist, its like its not there although it is in the folder.
I am relatively new to Python and I was installing a couple of modules - BeautifulSoup and Requests (learning how to scrape). So I installed Requests
$ pip install requests
Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2.19.1)
Requirement already satisfied: certifi>=2017.4.17 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (2018.4.16)
Requirement already satisfied: idna<2.8,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (2.7)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (1.23)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (3.0.4)
But then I when I run my file, I keep getting this error message
$ python test5.py
Traceback (most recent call last):
File "test5.py", line 1, in <module>
import requests
Could you help me with why I am having this issue?
On a separate note, all of a sudden BBEdit started asking me when I save a .py file "BBEDIT is trying to install a helper tool" and asking for a PW. Then it asks again "BBEDIT is trying to make modifications, allow it"? WHy is it doing that? Is it safe?
Thanks everyone and sorry for basic questions
A Stack Overflow answer can only cover one specific answer, not give a complete tutorial. But you probably actually want a complete tutorial. Also, if you're reading this from the distant future, whatever is written below may be as out-of-date as laptops that only do video and sound without smellovision. So, you should read the Python Packaging Authority's tutorial on installing packages.
You have (at least) two Python installations:
A Python 3.6 that you probably installed with a python.org installer, which lives in /Library/Frameworks/Python.framework/Versions/3.6/.
A Python 2.7 that Apple included as part of the OS, which lives in /System/Library/Frameworks/Python.framework/Versions/2.7/.
If you just run python, you get the Apple 2.7, not your 3.6. But you installed your packages for your Python 3.6, not your 2.7.1 (Plus, you don't want to run 2.7 anyway.)
You're supposed to run python3, as explained in PEP 394—or, better, use a virtual environment—when you have both Python 2.x and Python 3.x on the same system.2
Either will solve your problem.
If you're curious why pip installed for 3.6 instead of 2.7… well, it shouldn't work that way. But Apple's Python 2.7 is a bit out of date (I mean, even out of date for 2.7), so it doesn't even include pip. Which means the only pip on your system is the 3.6 one. So, even though python runs python2.7 instead of python3.6, pip runs pip3.6 instead of pip2.7. Which is very confusing, and pretty much unavoidable on current Macs.
This is a part of the reason current recommendations suggest never running pip directly; instead, run it like this:
python3 -m pip install requests
That guarantees that the Python installation you're installing requests into is the same on that you get when you run your script with python3 test5.py.
As things currently stand, as long as you don't ever touch Apple's 2.7, you can get away with just using pip3, or even pip. But that could easily change in, say, macOS 10.14, so better to get into the right habit.
1. You can tell because of, e.g., the paths in those "Requirement already satisfied" lines.
2. Well, the same non-Windows system, but you don't want to learn the different ways in which Windows is messed up, just stick to macOS here…
I'm trying to install a module called Scrapy. I installed it using
pip install Scrapy
I see the 'scrapy' folder in my /usr/local/lib/python2.7/site-packages, but when I try to import it in a Python program, is says there is no module by that name. Any ideas as to why this might be happening?
EDIT: Here is the output of the pip command:
Downloading/unpacking Scrapy
Downloading Scrapy-0.20.0.tar.gz (745kB): 745kB downloaded
Running setup.py egg_info for package Scrapy
no previously-included directories found matching 'docs/build'
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/local/lib/python2.7/site-packages (from w3lib>=1.2->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: Scrapy
Running setup.py install for Scrapy
changing mode of build/scripts-2.7/scrapy from 644 to 755
no previously-included directories found matching 'docs/build'
changing mode of /usr/local/bin/scrapy to 755
Successfully installed Scrapy
Cleaning up...
When I run /usr/local/bin/scrapy I get the usage for the command and the available commands. I noticed that I have a python2.7 and python2.7-32 in my /usr/local/bin, and I remember installing the 32 bit version because of a problem with Mavericks.
Here is the output of python /usr/local/bin/scrapy:
Traceback (most recent call last): File "/usr/local/bin/scrapy", line 3, in <module> from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline
And head /usr/local/bin/scrapy:
#!/usr/local/opt/python/bin/python2.7 from scrapy.cmdline import execute execute()
Are you using Homebrew or MacPorts or something? As #J.F.Sebastian said, it sounds like you are having issues mixing the default python that comes with OS X, and one that is installed via a package manager... Try /usr/local/opt/python/bin/python2.7 -m scrapy and see if that throws an ImportError.
If that works, then you may want to consider making that python executable your default. Something like alias python2.7=/usr/local/opt/python/bin/python2.7 and then always use python2.7 instead of the default python. You can likewise just point python to the /urs/local... bit, but then you won't have easy access to the system (OS X-supplied) python if you ever needed it for some reason.
EDIT: You can force pip to install to an alternate location. The details are here: Install a Python package into a different directory using pip?. If you do indeed have extra Python folders on your system, maybe you can try directing scrapy to those, even if just for a temporary solution.
Can you post the output of the pip command? Perhaps it is failing somewhere?
Also, is it possible you have two versions of Python on your machine? Pip only installs to one location, but perhaps the version of Python on your path is different.
Finally, sometimes package names given to pip are not exactly the same as the name used to import. Check the documentation of the package. I took a quick look and the import should be lowercase:
import scrapy
When all else fails you can always set the environment variable PYTHONPATH (see Permanently add a directory to PYTHONPATH for help) to the path where you installed Scrapy. (pending you're not using virtualenv -- and if you are please specify so we can help, it's generally a good idea to provide OS too)
if you run on Ubuntu:
use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.
Optionally, even if it solves your problem, it is always better to install python libraries on a virtual environment, using virtualenvwrapper to keep the libraries separated, try to examine the apt-get installation log to find out what tools where added, then remove scrapy python library and reinstall it in the virtual env. using pip
It appears that the scrapy module that is installed on the Python path is an executable file that will bootstrap a Scrapy project directory for you.
The Python code in the scrapy executable looks like this:
#!/usr/bin/env python
from scrapy.cmdline import execute
execute()
That's intended to be run from the command line rather than imported into your own Python project module.
According to the documentation for the project, running the scrapy executable with this syntax:
scrapy startproject <your-project-name>
will bootstrap a Scrapy project that has the following directory structure:
your-project-name/
scrapy.cfg
tutorial/
__init__.py
items.py
pipelines.py
settings.py
spiders/
__init__.py
...
There are a number of examples in the documentation that demonstrate how you create and run your own spiders, link extractors, etc., and how to manipulate the data that you retrieve with the application. They each demonstrate the appropriate Python imports from subdirectories in the scrapy package to get you up and running.
Hope that this helps.