I'm on Ubuntu Server 14.04.
I'm trying to use pypy 3.2 to run a django application.
But whenever I try pypy3 manage.py runserver 0:8000, it says:
(fields.E210) Cannot use ImageField because Pillow is not installed.
HINT: Get Pillow at https://pypi.python.org/pypi/Pillow or run command "pip install Pillow".
But if I try to install Pillow:
pypy3 -m pip install Pillow
It says:
Requirement already up-to-date: Pillow in /usr/local/lib/python3.4/dist-packages/Pillow-2.9.0-py3.4-linux-x86_64.egg
Then I downloaded the source and tried:
pypy3 setup.py install
And it says:
Attribute error install_layout
I don't know what to do!!
_<
You should create a proper virtualenv with pypy3 as your interpreter t run your django install - inside the virtualenv, pip should just work for both django and Pillow - without strange maneuvers like pypy3 -m pip which are subject to breakrage like teh one you've got.
To create a virtualenv inside which the default Python interpreter is pypy3 just do virtualenv -p pypy3 myprojectdir - and activate it with source myprojectdir/bin/activate - for the lazy types there is virtualenv wrapper which creates even more shortcuts. From there on, pip install anything will install thins in this directory, without messing (or checking) your system packages from any other Python.
That is the way to have a sane environment- however it seens you are just trying to be beyond the edge for what is available now - there are mentions on the internet of Pillow being compatible with Pypy (and, of course, with python3) - but I could not find a single mention about running Pillow with pypy3.
I can get it to build without errors, by installing the pypy3-devel on my Linux system, and manually setting the CFLAGS variable - (I am using fedora here - on Ubunut you should have a pypy3-dev package). But even though it does build, it does fail to run on pypy, raising ImportError: unable to load extension module 'PIL/_imaging.pypy3-24.so': PIL/_imaging.pypy3-24.so: undefined symbol: PyExc_SystemError; (On fedora 22I set CFLAGS=-I/usr/lib64/pypy3-2.4.0/include/ check your pypy3-dev package to see where do these include files are on Ubuntu).
TL;DR Either use pypy2.x or regular cPython3.4 (of course this is preferred) for your project - one of these days the dependencies will catch up. Either way Django should not benefit that much from Pypy's JIT - a heavily loaded webserver will have to deal with database connection scalability (not Python) and good caching (which can be properly configured on layers above Python as well - take a look at "varnish"). If you have that much processing on requests that your site is taking a hit for not using Pypy, consider run the bottleneck algorithms in another process using celery/RPCs - and write that part of the program to run with Pypy/Cython/pure C.
Related
I have a Windows 7 machine running Python 3.8.5 with a very large number of physics/electronics/data analysis/simulation packages. As it turned out, I must have - for some inexplicable reason - installed the 32-bit version of Python instead of the 64-bit one despite having a 64-bit system. And I didn't notice until very recently when I was trying to install some packages that require 64-bit Python. Hence I've now downloaded and installed the latest Python version that is supported by Windows 7, which seems to be 3.8.10.
Question: What is the easiest and also fail-safe way to reinstall all the user packages - that I currently have under 3.8.5 - to 3.8.10?
For some reason, I couldn't find any "canonical" solution for this online. As it seems, Python does not come with any built-in support for updating or system migration and I'm honestly wondering why...
Anyway, my first idea was to get a list of all user (= "local"?) packages currently installed under 3.8.5, but I don't know how. Reason: Doing help('modules') inside the interpreter will list all packages and I don't see a way to "selectively apply" pip to a specific Python version, e.g. something like python-3.8.5 -m pip list --local is not supported.
After getting a list of the user packages, I was thinking to pack it into a batch command pip install package_1 package_2 <...> package_N, thus reinstalling everything to Python 3.8.10. And afterwards uninstalling Python 3.8.5 and removing all environment variables from system PATH.
Is this the proper way to do this?
Anyway, my first idea was to get a list of all user (= "local"?) packages currently installed under 3.8.5, but I don't know how.
Create a list of installed packages with pip freeze > pkglist.txt or pip list --format=freeze. If you already have one, that's great.
Then uninstall 32-bit Python 3.8.5 and clean your path for all Python related variables. Now, install 64-bit Python 3.8.10.
After reinstalling, you can install back all the packages with pip install -r pkglist.txt and it will restore the exact versions of the packages.
If you insist on having both 32-bit and 64-bit versions installed and also have the Python Launcher installed, you could invoke 32 and 64 bit versions separately with py -3.8-64 -m pip and py -3.8-32 -m pip.
I don't see a way to "selectively apply" pip to a specific Python version.
This is possible with the Python Launcher on Windows. But only between major/minor versions and not the patch versions according to its help message.
I would also recommend creating a virtual environment this time before installing the packages and leaving the root environment alone. You can create one named venv with just python -m venv venv, activate it with ./venv/Scripts/activate and proceed with the installation of packages.
Nope, doesn't work. After installing the packages with the newer Python version in PATH, e.g. Jupyter won't start.
If the Jupyter error persists, you could try pinning packages to their most recent patch/minor versions to update them and yet not break your code.
As a last resort, you could try installing Python 3.10 alongside your current Python installation (without uninstall or editing the PATH) and then installing the absolute latest versions of the packages in a 3.10 virtual environment to see if it works for you. You would invoke the two versions with Py Launcher, e.g. py -3.10 and py -3.8.
If I understood correctly, you have multiple packages like NumPy, pandas etc. installed on your machine, and you want to reinstall them "automatically" on a fresh installation of python.
The method (I use) to perform such an operation is by creating a file named setup.py which includes a list of all the packages.
Bellow, I am attaching an example of such a file I use in one of my projects:
from setuptools import setup, find_packages
setup(
name='surface_quality_tools',
version='0.1',
install_requires=["matplotlib", "psutil", "numpy", "scipy", "pandas", "trimesh", "pyglet", "networkx", "protobuf",
"numpy-stl", "sklearn", "opencv-python", "seaborn", "scikit-image", "flask", "tqdm", "pytest"],
package_data={'': ['*.json']},
packages=find_packages(include=[])
)
to run the installation you should open a command prompt from inside the project directory and run:
pip install -e .
You can find a nice example in this blog page
One common way of handling packages in Python is via virtual environments. You can use Anaconda (conda), venv or any of several other solutions. For example, see this post:
https://towardsdatascience.com/virtual-environments-104c62d48c54#:~:text=A%20virtual%20environment%20is%20a,a%20system%2Dwide%20Python).
The way this works in by keeping the Python interpreter separate from the virtual environment that contains all the necessary packages.
Probably the main reason Python doesn't feature migration tools (at least as part of standard library) is because pip - the main package tool - doesn't handle conflict resolution all too well. When you update a version of Python it might so happen (especially with niche packages) that some of them won't work any more and pip often won't be able to solve the dependencies. This is why it's a good idea to keep a separate venv for different Python versions and different projects.
The other tool you could use for easy migration is Docker which is a semi-virtual machine working on top of your host OS and containing usually some linux distribution, Python along with the necessary packages necessary for running and development.
It takes a bit of time to set up a container image initially but afterwards setting everythin on a new machine or in the cloud becomes a breeze.
Listing currently installed packages is done via pip freeze command, the output of which you can then pipe into a file to keep a record of project requirements, for example pip freeze > requirements.txt.
My Django site is hosted on Azure.
It allows for users to upload photos. I need a way for the system to resize, and possibly rotate photos.
Seems simple, and I tried to use the Pillow library but while it works locally it will not deploy to Azure for a number of reasons. I can be specific if needed but this is well documented like here.
I even tried buiding a wheel of Pillow and deploying that but Azure refuses to load it saying it is the wrong platform (even though I matched the Python 2.7 version - and 32 bit). I tried to upload 64 bit versions as well, and nothing works. So at this point I just want to leave Pillow behind me and ask for another way to achieve this in Python without Pillow. Is there any other way to do this?
Notes of things I tried:
1) Installing Pillow the normal way gives this familiar error message:
ValueError: zlib is required unless explicitly disabled using --disable-zlib, aborting
2) I then created a wheel by doing: pip wheel Pillow --wheel-dir=requirements
This however yields the following error in the pip.log:
Pillow-3.4.2-cp27-cp27m-win32.whl is not a supported wheel on this platform.
Pillow-4.1.1-cp27-cp27m-win32.whl is not a supported wheel on this platform.
I am certain that I'm runing Python 2.7 on a 32bit platform so not sure why its complaining.
After days wasted, I've discovered the reason why Pillow isn't installing. It's not because the wheel is incompatible to the platform, but rather that pip is too old.
Azure is using pip version 1.5.6 at the moment - shame on them. This version doesn't recognise wheels.
Here is how I fixed this:
Goto the Kudu DebugConsole:
https://[site_name].scm.azurewebsites.net/DebugConsole
Activate your VirtualEnv:
env\Scripts\activate
Note that if you run pip --version how old that version number is.
Now upgrade this by running:
python -m pip install -U pip
Note that you cannot upgrade the default pip in D:\Python27 as you don't have access to it but you can upgrade your local pip inside fo the virtual environment.
Now run pip --version to ensure you are running the latest version (i.e. >=9.0.1).
Now inside of requirements.txt you can tell pip to look for wheels in specific folders by adding a line at the top such as:
--find-links requirements (which means it will search the requirements folder).
Here is how you create the Pillow wheel. You can run this locally or on the Kudu Console. If you run it locally ensure your python version matches what you use on Azure (2.7 or 3.X) and by default make sure you use a 32bit version.
pip install wheel (Only if you don't have wheel installed)
pip wheel Pillow --wheel-dir=requirements
This will copy two files into your requirements folder: Pillow-X.whl and olefile-X.whl. Ensure these are added to your source control if you are deploying via git push. Push these to the server.
Now in the Kudu DebugConsole you can test the .whl files are there (after deploying) and test the installing by running:
pip install --no-index -r requirements.txt
This should now work and install Pillow!
When deploying pay close attention to if it says Found compatible virtual environment. or Creating python 2-7 virtual environment.. The former is what you want. But if you see the latter it means that the deploy has blasted your env folder and reset you back to pip 1.5.6. I don't know why it does this sometimes, but try to make as few changes to the env folder as possible after deploying (i.e. just upgrade pip and thats it) to avoid this.
I can't help you much with installing Pillow on Azure platform.
But my days of using manually resizing and other stuff is long gone.
I have been using thumbor https://thumbor.org/ for quite some time.
Just setup a secured instance of the same and use it resize, crop and manage your images dynamically.
Hope it helps
There is the other SO thread Microsoft Azure Django Python setup error Pillow, which has the similar issue about installing Pillow on Azure. I think my answer for that is helpful for resolving your issue. Any concern for my solution, please feel free to let me know.
I'm not overly familiar with Linux and am trying to run a Python script that is dependent upon Python 3.4 as well as pymssql. Both Python 2.7 and 3.4 are installed (usr/local/lib/[PYTHON_VERSION_HERE]). pymssql is also installed, except it's installed in the Python 2.7 directory, not the 3.4 directory. When I run my Python script (python3 myscript.py), I get the following error:
File "myscript.py", line 2, in
import pymssql
ImportError: No module named 'pymssql'
My belief is that I need to install pymssql to the Python 3.4 folder, but that's my uneducated opinion. So my question is this:
How can I get my script to run using Python 3.4 as well as use the pymssql package (sorry, probably wrong term there)?
I've tried many different approaches, broken my Ubuntu install (and subsequently reimaged), and at this point don't know what to do. I am a relative novice, so some of the replies I've seen on the web say to use ENV and separate the versions are really far beyond the scope of my understanding. If I have to go that route, then I will, but if there is another (i.e. easier) way to go here, I'd really appreciate it, as this was supposed to just be a tiny thing I need to take care of but it's tied up 12 hours of my life thus far! Thank you in advance.
It is better if when you run python3.4 you can have modules for that version.
Another way to get the desire modules running is install pip for python 3.4
sudo apt-get install python3-pip
Then install the module you want
python3.4 -m pip install pymssql
The easiest way is to use virtual environments instead of system paths or environment scripts. See official Python package installation guide.
All you need to do is to
# Create fresh Python environemnt
virtualenv -p python3.4 my-venv
# Activate it in current shell
source my-venv/bin/activate
# Install packages
pip install mysqlclent
Note that mysqlclient is Python 3.x compatible version.
I've seen quite a few posts on this topic, but there seems to be little agreement or definitive answer. I understand that OSX (in my case 10.10.5) comes with Python2.7 installed, as detailed in the official python docs.
After following those instructions, I now have python 2.7 and python 3.4 installed (accessible by $ python and $ python3, respectively) and perhaps more versions. I find python versions in a few places:
/Applications/Python 3.4
/usr/local/bin (contains python, python2, python2.7, python3, python3.4
/usr//bin (contains python2.6, python2.7)
/System/Library/Frameworks/Python.framework/Versions (contains 2.3, 2.5, 2.6, 2.7, Current)
Moreover, I'm still having trouble when I install python packages...and I think it's because I don't completely understand where certain packages are installed, their PATH variables and how they are aliased.
Problem:
For example, after successfully installing nose with pip install nose, I cannot use $ nosetests because the command is not found.
Suggested solutions:
Inspect the nose directories and look for nosetests to alias it (but despite there being many other files within /nose, regardless of the version, there isn't anything about nosetests).
Another suggestion is to run python setup.py install within /nose but that simply gives this error:
/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file 'setup.py': [Errno 2] No such file or directory
Some posts suggested simply using sudo as a fix to the problem...but I have been warned against using sudo without understanding why I need it, and the nose docs don't mention needing sudo, so I don't want to simply put a band-aid over a deeper issue that may arise later.
My original question was basically two questions: one about conflicting versions of python and the other about installing the nose package specifically.
For the first question, I found jonrsharpe's initial comment to my question very helpful. He links to a tutorial that explains virtualenv and how to set it up. However, I was still uncertain how to install packages only within that virtualenv, because even when I'm in the virtualenv directory and I pip install, the package is still listed under pip freeze when outside of that virtualenv directory.
Solution:
This tutorial was helpful in further explaining virtualenv, what it is and how to use it. In short, virtualenv creates a copy of python in the designated virtualenv directory that has its OWN pip. That local pip must be used to install packages locally for that virtualenv, referencing it as [virtualenv_dir_you_created]/bin/pip install nose
And, by using virtualenv, problems with permission for writing to packages globally disappear.
I have a problem that comes from me following tutorials without really understanding what I'm doing. The root of the problem I think is the fact that I don't understand how the OS X filesystem works.
The problem is bigger than Python but it was when I started learning about Python that I realized how little I really understand. So in the beginning I started following tutorials which led me to use the easy_install command a lot and when a lot of tutorials recommended PIP I never got it running. So I have run a lot of commands and installed a lot of different packages.
As I have understood Lion comes with a python install. I have been using this a lot and from this I have installed various packages with easy_install. Is there any way to go back to default installation and begin from the very beginning? Is this something I want to do? If so why?
Is there any advantage of using a Python version I have installed with Homebrew? How can I see from where Python is run when I run the Python command?
When I do install something with either easy_install, homebrew, macports etc where do things actually end up?
Homebrew installs its software inside the /usr/local subdirectory on your Mac. OS X doesn't install anything there on its own; in fact, /usr/local is reserved for user-installed stuff. Since Homebrew never installs files outside /usr/local (and doesn't even have the ability to, unless you run brew using sudo - which is not recommended_) and OS X never installs files inside there, never the two shall mix.
easy_install and pip install files into system directories by default. That's why you have to run those commands with sudo to install packages with them.
I can't recommend virtualenv enough, regardless of which OS you're using. It installs a copy of Python, along with any packages or modules you want, inside a directory of your choosing. For example:
$ cd /tmp
$ virtualenv foo
New python executable in foo/bin/python
Installing setuptools............done.
Installing pip...............done.
$ cd foo
$ bin/pip install sqlalchemy
Downloading/unpacking sqlalchemy
Downloading SQLAlchemy-0.7.7.tar.gz (2.6Mb): 2.6Mb downloaded
Running setup.py egg_info for package sqlalchemy
[...]
Successfully installed sqlalchemy
Cleaning up...
[work, work, work]
[decide this was a bad idea]
$ cd /tmp; rm -rf foo
...and all traces of the project are now completely gone.
Use easy_install to install virtualenv into OS X itself - like you've done for those other packages - but then do all new development inside isolated directories that you can wipe clean at a moment's notice. This is pretty much the standard way of developing and deploying Python applications these days.
The advantage of using a Python installed via a package manager like Homebrew or MacPorts would be that this provides a simple way of removing the Python installation and reinstalling it. Also, you can install a more recent version than the one Mac OS X provides.