This is the first time I've tried developing a package rather than just scripts and a lot of it is very confusing to me. I am using Windows 10 with Python 3.7 installed and managed via Conda 4.7.11.
TLDR: Conda packages aren't properly isolated from each other when using pip install -e.
I have a basic package structured like this:
----my_package
| LICENSE
| README.md
| setup.py
|
|---my_package
my_module.py
__init__.py
In my_module.py is:
def cool_thing():
print("This is a cool thing!")
In __init__.py is:
from .my_module import cool_thing
So when I run Python from the outer my_package directory, I can do this:
$ python
Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Ana
conda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_package
>>> my_package.cool_thing()
This is a cool thing!
So far, so good. Now I want to install the package locally to a virtual Conda environment. My setup.py file has this inside:
from setuptools import setup
setup(name = "my_package",
version = "0.1",
description = "Do something cool!",
author = "Me",
license = "MIT",
packages = ["my_package"],
zip_safe = False)
In the terminal I do this:
conda create -n env-my-package
conda activate env-my-package
conda install pip
I read in a lot of places that this is how you're supposed to install local development packages:
pip install -e .
It tells me:
Obtaining file:///C:/long/path/to/my_package
Installing collected packages: my-package
Running setup.py develop for my-package
Successfully installed my-package
pip list tells me it's installed:
$ pip list
Package Version Location
------------ --------- --------------------------
certifi 2019.6.16
my-package 0.1 c:\long\path\to\my_package
pip 19.2.2
setuptools 41.0.1
wheel 0.33.4
wincertstore 0.2
And if I navigate to a different directory I can still import it in Python:
cd ..
python
$ python
Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_package
>>> my_package.cool_thing()
This is a cool thing!
So this seems fine. But then look at what happens when I switch to a different Conda environment:
conda deactivate
conda activate env-something-else
python
Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_package
The package somehow gets imported anyway, without an error message, despite the fact that we are in a totally separate Conda environment without my_package installed. On the other hand, actually trying to use the package results in an error:
>>> my_package.cool_thing()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'my_package' has no attribute 'cool_thing'
Investigating further, dir(my_package) shows it has no non-dunder attributes:
>>> dir(my_package)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
my_package.__file__ outputs None, while my_package.__path__ gives the path to the directory where the original files are: C:\long\path\to\my_package.
So it seems like the Conda environments aren't properly isolated. It can still see the package when it's not supposed to, even if it can't use functions from it. Before you ask, yes, I did make sure to run pip install -e with the original env-my-package environment activated.
This has made me completely distrust that what I'm doing in my Conda environment stays in my Conda environment. If it "leaks" in this way, what else might leak? I don't want to accidentally pollute any of my other projects while working on this package. Thankfully I'm able to uninstall it using pip and return to a clean state, but still, I can't go any further until I figure this out. My ultimate question:
What is the proper way to develop a package when using Conda?
I hope you'll forgive me for writing this all out to the point of tediousness, but I wanted to make sure I covered all the details just in case any are important.
Related
I just started having some problems with python, and would like to fix this the proper way, as a last resort, reinstalling if I have to.
Before Python 3, I had Python 2.7 installed. I then installed Python 3.7 along side it. I don't remember, if the path environment was automatically set, or if i manually set it.
My system variables path contains :
C:\Python27\;C:\Python27\Scripts;
So I believe this was set automatically.
My user variables path however contains :
C:\Python37\;C:\Python37\Scripts;
Would the python installer use the user environment? I'm not sure, so I don't know if I set that myself.
However, the user variables path also contains :
%PYTHON_DIR%\Python37\Scripts\;%PYTHON_DIR%\Python37\;%PYTHON_DIR%\Python36\Scripts\;%PYTHON_DIR%\Python36\
Again, I don't know if I set this myself, and I have no recollection of setting this in my user variables :
PYTHON_DIR = C:\Users\pcuser\AppData\Local\Programs\Python
...but it's there, and I have two folders in that path.
I also have this in my user variables path :
PYTHON = os.path.expanduser(os.getenv('PYTHON', 'C:\\Python37\\python.exe'))
I know I created a python environment. I just have to look for it, as I don't remember where to find it right now.
Here is the problem I am having.
I also am using Python in MSYS, and get this error from Python in C:\msys64\usr\bin :
Could not find platform independent libraries <prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000800018040 (most recent call first):
0 [main] python 1489 cygwin_exception::open_stackdumpfile: Dumping stack t
race to python.exe.stackdump
I set PYTHONHOME to C:\msys64\usr\bin. That doesn't solve the problem.
I also removed PYTHON_DIR to see if there was a conflict there, but that didn't solve the problem.
Rather, I got another error :
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000800018040 (most recent call first):
0 [main] python 143 cygwin_exception::open_stackdumpfile: Dumping stack tr
ace to python.exe.stackdump
So it seems to me my Python environment is messed up, and I want to learn how to set it up correctly, hopefully without having to start over.
How can I get rid of these errors, and get a proper python environment?
Personally, I use miniconda.
Miniconda is the small version of anaconda which is an excellent environment manager. Miniconda does not include a gui and I feel it is nicer to work with.
To create an environment with
conda create -n my-test-env python=3.9
You can activate it using
conda activate my-test-env
You can install other packages using
conda install numpy.
However, not all python packages are available in conda. You can any pip packages by installing pip
conda install pip
pip install numpy
Setting PYTHONHOME to a value like C:\msys64\usr\bin is definitely wrong; it should be something like C:\Python37\Lib if that's where Python installed its libraries. But try simply unsetting it.
After a while of troubleshooting, I was able to discover where the problem was.
Some files in C:\msys64\usr\lib\python3.8 somehow got deleted, or python 3.8 was installed, but not properly. So I did a reinstall, and python stopped complaining about missing encodings and modules.
Running a test on all python installations
MINGW64 ~
# python
Python 3.8.2 (default, Feb 27 2020, 05:27:33) [GCC 9.2.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
# python setup_build.py install
running install
running build
running build_ext
skipping 'geotools/geotools.c' Cython extension (up-to-date)
running install_lib
copying build/lib.mingw-3.8/geotools-cpython-38.dll -> C:/msys64/mingw64/lib/python3.8/site-packages
running install_egg_info
Writing C:/msys64/mingw64/lib/python3.8/site-packages/geotools-0.1.0-py3.8.egg-info
MINGW32 ~
# python
Python 3.8.2 (default, Feb 27 2020, 06:39:26) [GCC 9.2.0 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
# python setup_build.py install
running install
running build
running build_ext
dllwrap: WARNING: dllwrap is deprecated, use gcc -shared or ld -shared instead
skipping 'geotools/geotools.c' Cython extension (up-to-date)
running install_lib
running install_egg_info
Removing C:/msys64/mingw32/lib/python3.8/site-packages/geotools-0.1.0-py3.8.egg-info
Writing C:/msys64/mingw32/lib/python3.8/site-packages/geotools-0.1.0-py3.8.egg-info
C:\WINDOWS\system32>py -3.7 -m pip list
Package Version
----------------------------- ---------
...
C:\WINDOWS\system32>py -2.7 -m pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please
upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop s
upport for Python 2.7 in January 2021. More details about Python 2 support in pi
p can be found at https://pip.pypa.io/en/latest/development/release-process/#pyt
hon-2-support pip 21.0 will remove support for this functionality.
Package Version
----------------------------- ----------
...
C:\WINDOWS\system32>python
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\WINDOWS\\SYSTEM32\\python27.zip', 'C:\\Python27\\DLLs', 'C:\\Python27\
\lib', 'C:\\Python27\\lib\\plat-win', 'C:\\Python27\\lib\\lib-tk', 'C:\\Python27
', 'C:\\Users\\pcUser\\AppData\\Roaming\\Python\\Python27\\site-packages', 'C:\\Py
thon27\\lib\\site-packages', 'C:\\Python27\\lib\\site-packages\\pybind11-2.6.2-p
y2.7.egg']
>>>
C:\WINDOWS\system32>py -3
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\Users\\pcUser\\AppData\\Local\\Programs\\Python\\Python37\\python37.zip'
, 'C:\\Users\\pcUser\\AppData\\Local\\Programs\\Python\\Python37\\DLLs', 'C:\\User
s\\pcUser\\AppData\\Local\\Programs\\Python\\Python37\\lib', 'C:\\Users\\pcUser\\App
Data\\Local\\Programs\\Python\\Python37', 'C:\\Users\\pcUser\\AppData\\Roaming\\Py
thon\\Python37\\site-packages', 'C:\\Users\\pcUser\\AppData\\Local\\Programs\\Pyth
on\\Python37\\lib\\site-packages']
>>>
C:\WINDOWS\system32>py -3.6 -m pip list
Package Version
--------------- --------
...
My python environment is good. All working nicely. :)
What I learned...
Using PYTHONHOME and PYTHONPATH is not neccesary, as was suggested by many... although equally suggessted by many.
From my experience - speaking fron a novice perspective - I would not recommend using them.
Maybe they work for others, but I got errors regardless of how I set it up - whether
C:\Python37 or
C:\Python37\lib or
C:\Users\pcUser\AppData\Local\Programs\Python\Python37 or
C:\msys64\usr\lib\python3.8;C:\msys64\usr\lib\python3.8\site-packages;C:\msys64\usr\lib\python3.8\lib-dynload;C:\msys64\usr\lib\python3.8\distutils
They all return a heap of errors on one installment or other.
So it does not work for me. However, probably works for others.
I am happy though that after probably more than 60 hours, my python environment is back in working order. :)
While trying to install atom-lint package in Atom editor I somehow corrupted my conda installation.
I did the following things that might have caused the issue:
Installed a python dependency Flake8 using conda install.
Messed around with Atom Init Script (I can provide more info if needed)
After I did these things I encountered the following problem:
If I run conda I get
$ conda
Traceback (most recent call last):
File "/Users/me/miniconda3/bin/conda", line 12, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Strangely enough if I run python in my command line anaconda still seems to be installed an working.
$ python
Python 3.7.1 (default, Oct 23 2018, 14:07:42)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
I have also noticed that the Python3 bin has disappeared from the conda environment folder ~/miniconda3/envs/my_env/bin/
This is very strange and I thought might be related.
Any help would be much appreciated.
I ended up making a backup copy of the miniconda3/envs folder, reinstalling miniconda and copying the environment back in. It works now, not sure what caused the issue.
I'm trying to use virtualenv in my new mainly Python project. The code files are located at ~/Documents/Project, and I installed a virtual environment in there, located at ~/Documents/Project/env. I have all my packages and libraries I wanted in the env/bin folder.
The question is, how do I actually run my Python scripts, using this virtual environment? I activate it in Terminal, then open idle as a test, and try
"import django"
but it doesn't work. Basically, how can I use the libraries install in the virtual environment with my project when I run it, instead of it using the standard directories for installed Python libraries?
Check out the example below, and also the virtualenv documentation. It's actually fairly straightforward if you follow the steps:
virtualenv Project # creates a new Project dir
cd Project/bin # could just call Project/bin
. activate # should now have (Project) in the prompt name
pip install django # without this, won't be able to import django
deactivate # switch of virtual mode
I tried the above out in my Mac and worked fine. Here's a transcript for reference.
Transcript of operations
[MacMini]<Documents> :virtualenv Project
[MacMini]<Project> :cd bin/
[MacMini]<bin> :python2.7
Python 2.7.9 (v2.7.9:648dcafa7e5f, Dec 10 2014, 10:10:46)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named django
>>> quit()
[MacMini]<bin> :. activate
(Project)[MacMini]<bin> :pip install django
You are using pip version 6.0.6, however version 8.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting django
Downloading Django-1.9.4-py2.py3-none-any.whl (6.6MB)
100% |################################| 6.6MB 1.2MB/s
Installing collected packages: django
Successfully installed django-1.9.4
(Project)[MacMini]<bin> :python
Python 2.7.10 (default, Oct 23 2015, 18:05:06)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> quit()
(Project)[MacMini]<bin> :deactivate
It's also good practice to make a requires.txt file for all your dependencies. If for example your project requires Flask and pymongo, create a file with:
Flask==<version number you want here>
pymongo==<version number you want here>
Then you can install all the necessary libraries by doing:
pip install -r requires.txt
Great if you want to share your project or don't want to remember every library you need in your virtualenv.
I thought I'd move from using Tkinter to wxPython, but I'm having some troubles. All I get is this:
>>> import wx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named wx
I have installed wxPython. When I run the uninstaller I clearly see that wxPython IS installed:
1. wxPython3.0-osx-cocoa-py2.7 3.0.0.0
Enter the number of the install to examine or 'Q' to quit:
When I start Python I see that my version should match the version of wxPython:
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
I'm on OS X 10.7.5 32-bit.
Any help would be awesome! :)
Macs can have multiple versions of Python installed. Are you sure that you installed wxPython for the same python you invoke with the interpreter?
Try, which python, and make sure that this version of python has a wxredirect.pth file in site-packages pointing to the wxPython installation. (If it doesn't search for wxredirect.pth.)
Here's one version on my system...
> which python2.6
/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
> more /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/wxredirect.pth
import site; site.addsitedir('/usr/local/lib/wxPython-3.0.0.0/lib/python2.6')
#import site; site.addsitedir('/usr/local/lib/wxPython-2.9.1.1/lib/python2.6')
#import site; site.addsitedir('/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.6')
Here, btw, I can comment out lines in the wxredirect.pth to choose the version of wxPython I want to use.
I just find the same problem.
I used brew install wxpython to install it on mac.
I try the method above but no use.
I happen to find the solution when I install another package, it showen below:
brew install tbb
Downloading https://homebrew.bintray.com/bottles/tbb-2017_U7.sierra.bottle.t
############################################################ 100.0%
==> Pouring tbb-2017_U7.sierra.bottle.tar.gz
==> Caveats
Python modules have been installed and Homebrew's site-packages is not
in your Python sys.path, so you will not be able to import the modules
this formula installed. If you plan to develop with these modules,
please run:
mkdir -p /Users/ningrongye/.local/lib/python2.7/site-packages
echo 'import site; site.addsitedir("/usr/local/lib/python2.7/site-
packages")' >> /Users/ningrongye/.local/lib/python2.7/site-
packages/homebrew.pth`
this is what homebrew said and I just try those, and it works.
ningrong
I have Python 2.7 and 3.7 .
In /usr/local/bin/ there are symbolic links for 2.7 and 3.7 Python versions and also symbolic links for pip.
I've installed wxPython with pip3
pip3 install -U wxPython
Then i checked the installation for Python3
myname$ python3
Python 3.7.2 (v3.7.2:9a3ffc0492, Dec 24 2018, 02:44:43)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
>>> wx.version()
'4.0.4 osx-cocoa (phoenix) wxWidgets 3.0.5'
Antonio
In my case it worked by resetting the brew environment to 2.7:
brew link --overwrite python#2
I have build my virtual env with this command:
virtualenv env --distribute --no-site-packages
And then I have installed several modules (django etc) into env with pip, the problem is that when I wanted to run the code on the second machine it would not work, here is what I have done:
visgean#rewitaqia:~/scripty/project_name$ source ./env/bin/activate
(env)visgean#rewitaqia:~/scripty/project_name$ python
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:12:17)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.__file__
'/home/visgean/scripty/project_name/env/lib/python2.7/site-packages/django/__init__.pyc'
but when I want to run it on the second machine:
(env)user#debian:~/project_name$ python
Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named django
>>>
I wild error appears! The first machine is ubuntu, the second one is ubuntu. There seems to be some broken links in /home/user/project_name/env/lib/python2.7 , is that the problem? and if so, how should I prevent it/repair it?
If you are just copying the virtualenv to the second machine you may encounter some issues. From the virtualenv site:
Normally environments are tied to a specific path. That means that you
cannot move an environment around or copy it to another computer. You
can fix up an environment to make it relocatable with the command:
$ virtualenv --relocatable ENV
This will make some of the files
created by setuptools or distribute use relative paths, and will
change all the scripts to use activate_this.py instead of using the
location of the Python interpreter to select the environment.
Note: you must run this after you've installed any packages into the
environment. If you make an environment relocatable, then install a
new package, you must run virtualenv --relocatable again.
I have just noticed that I did have a wrong version of python on the second machine - debian does not have python2.7, installing 2.7 and pip for is the solution