Related
I have some simple cefpython code opening a url and am trying to create a stand alone executable with pyinstaller:
I copied files from https://github.com/cztomczak/cefpython/tree/master/examples/pyinstaller to a a directry named pyinstaller
I made following minor changes to pyinstaller.spec
+SECRET_CIPHER = ""
...
- ["../wxpython.py"],
+ ["../hello.py"],
...
- icon="../resources/wxpython.ico")
+ )
I can successfully compile my application on windows with python
On the same machine with python 3.5.4 64 bit and following virtualenv:
cefpython3==66.0
future==0.18.2
PyInstaller==3.2.1
pypiwin32==223
pywin32==228
I can also compile windows with python 3.6.4 64 and following virtualenv:
altgraph==0.17
cefpython3==66.0
future==0.18.2
macholib==1.14
pefile==2019.4.18
PyInstaller==3.3.1
pyinstaller-hooks-contrib==2020.9
pypiwin32==223
pywin32==228
pywin32-ctypes==0.2.0
On Linux compilation works as well, but the executable is not operational.
I get following output and error:
CEF Python 66.0
Chromium 66.0.3359.181
CEF 3.3359.1774.gd49d25f
Python 3.5.2 64bit
[1013/180954.001980:ERROR:icu_util.cc(133)] Invalid file descriptor to ICU data received.
Trace/breakpoint trap (core dumped)
version is python 3.5.2 64bit and the virtualenv is:
cefpython3==66.0
pkg-resources==0.0.0
PyInstaller==3.2.1
What could be the cause?
The code, that I try to compile is below:
import platform
import sys
from cefpython3 import cefpython as cef
def check_versions():
ver = cef.GetVersion()
print("CEF Python {ver}".format(ver=ver["version"]))
print("Chromium {ver}".format(ver=ver["chrome_version"]))
print("CEF {ver}".format(ver=ver["cef_version"]))
print("Python {ver} {arch}".format(
ver=platform.python_version(),
arch=platform.architecture()[0]))
assert cef.__version__ >= "57.0", "CEF Python v57.0+ required to run this"
def main(url="https://www.stackoverflow.com"):
sys.excepthook = cef.ExceptHook
check_versions()
settings = {}
switches = {}
browser_settings = {}
cef.Initialize(settings=settings, switches=switches)
cef.CreateBrowserSync(
url=url,
window_title="CEF_HELLO: ",
settings=browser_settings,
)
cef.MessageLoop()
cef.Shutdown()
if __name__ == "__main__":
main()
Addendum: 2020-10-14:
same error on linux with other versions:
so far I tried python 3.5 and 3.7
Is there anybody who successfully created an executable?
I could be, that this just an issue with the example project and its configuration?
As alternative, a solution could be found in PyInstaller bug 5400
Here the steps:
1- download the PyInstaller helper in CEFpython named hook-cefpython3.py from:
https://github.com/cztomczak/cefpython/tree/master/examples/pyinstaller and put in the root directory of your project
2- In that file, replace the line:
from PyInstaller.compat import is_win, is_darwin, is_linux, is_py2
with:
from PyInstaller.compat import is_win, is_darwin, is_linux
is_py2 = False
3- in your PyInstaller .spec file, add the '.' to the hookspath, e.g. hookspath=['.']. I think it is also possible to add it as PyInstaller command line option.
These steps should solve the problem, until CEFPython deliver a correct version of the hook file.
This is not really the answer I would like to accept, but it is at least one solution and contains information, that might lead to a better fix, a better answer.
After debugging with strace I found out, that the executable searches many files like for example icudtl.dat, v8_context_snapshot.bin, locales/* were searched in
'dist/cefapp/cefpython3but were copied todist/cefapp/`
An ugly work around is to do following after compilation
cd dist/cefapp/cefpython3
ln -s ../* .
and the executable works.
I'm sure there is also a nicer non-brute-force solution, but for the time being I wanted to answer in case others are stuck as well
Probably this can be fixed in the spec file but would we need one spec file for linux and one for windows then?
Perhaps there's also an option to tell the excutable to search for these files one level higer?
To solve this, you need to set this in your spec file:
hookspath=[r'YOUR_ENV_SITE_PACKAGES\cefpython3\examples\pyinstaller\']
And then rebuild, you will have things in the right place.
The following steps solved the issue for me on Windows 10, Python 3.9.5 32-bit, PyInstaller 4.3, and CEFPython 66.1:
Download the hook-cefpython3.py file from here and put it into your project root directory.
Run the pyinstaller command as usual but add the --additional-hooks-dir . command line option, so the command will look like this:
pyinstaller --additional-hooks-dir . <main-file.py>
As opposed to other answers here, this anser neither requires changes of hookspath directive in pyinstaller's spec file and, as of now, nor any changes to the downloaded hook-cefpython3.py file.
I have installed pyspark recently. It was installed correctly. When I am using following simple program in python, I am getting an error.
>>from pyspark import SparkContext
>>sc = SparkContext()
>>data = range(1,1000)
>>rdd = sc.parallelize(data)
>>rdd.collect()
while running the last line I am getting error whose key line seems to be
[Stage 0:> (0 + 0) / 4]18/01/15 14:36:32 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 123, in main
("%d.%d" % sys.version_info[:2], version))
Exception: Python in worker has different version 2.7 than that in driver 3.5, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
I have the following variables in .bashrc
export SPARK_HOME=/opt/spark
export PYTHONPATH=$SPARK_HOME/python3
I am using Python 3.
By the way, if you use PyCharm, you could add PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to run/debug configurations per image below
You should set the following environment variables in $SPARK_HOME/conf/spark-env.sh:
export PYSPARK_PYTHON=/usr/bin/python
export PYSPARK_DRIVER_PYTHON=/usr/bin/python
If spark-env.sh doesn't exist, you can rename spark-env.sh.template
This may happen also if you're working within an environment. In this case, it may be harder to retrieve the correct path to the python executable (and anyway I think it's not a good idea to hardcode the path if you want to share it with others).
If you run the following lines at the beginning of your script/notebook (at least before you create the SparkSession/SparkContext) the problem is solved:
import os
import sys
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
Package os allows you to set global variables; package sys gives the string with the absolute path of the executable binary for the Python interpreter.
I got the same issue, and I set both variable in .bash_profile
export PYSPARK_PYTHON=/usr/local/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/python3
But My problem is still there.
Then I found out the problem is that my default python version is python 2.7 by typing python --version
So I solved the problem by following below page:
How to set Python's default version to 3.x on OS X?
Just run the code below in the very beginning of your code. I am using Python3.7. You might need to run locate python3.7 to get your Python path.
import os
os.environ["PYSPARK_PYTHON"] = "/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7"
os.environ["PYSPARK_DRIVER_PYTHON"] = "/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7"
I'm using Jupyter Notebook to study PySpark, and that's what worked for me.
Find where python3 is installed doing in a terminal:
which python3
Here is pointing to /usr/bin/python3.
Now in the the beginning of the notebook (or .py script), do:
import os
# Set spark environments
os.environ['PYSPARK_PYTHON'] = '/usr/bin/python3'
os.environ['PYSPARK_DRIVER_PYTHON'] = '/usr/bin/python3'
Restart your notebook session and it should works!
Apache-Spark 2.4.3 on Archlinux
I've just installed Apache-Spark-2.3.4 from Apache-Spark website, I'm using Archlinux distribution, it's simple and lightweight distribution. So, I've installed and put the apache-spark directory on /opt/apache-spark/, now it's time to export our environment variables, remember, I'm using Archlinux, so take in mind to using your $JAVA_HOME for example.
Importing environment variables
echo 'export JAVA_HOME=/usr/lib/jvm/java-7-openjdk/jre' >> /home/user/.bashrc
echo 'export SPARK_HOME=/opt/apache-spark' >> /home/user/.bashrc
echo 'export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH' >> /home/user/.bashrc
echo 'export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH' >> /home/user/.bashrc
source ../.bashrc
Testing
emanuel#hinton ~ $ echo 'export JAVA_HOME=/usr/lib/jvm/java-7-openjdk/jre' >> /home/emanuel/.bashrc
emanuel#hinton ~ $ echo 'export SPARK_HOME=/opt/apache-spark' >> /home/emanuel/.bashrc
emanuel#hinton ~ $ echo 'export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH' >> /home/emanuel/.bashrc
emanuel#hinton ~ $ echo 'export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH' >> /home/emanuel/.bashrc
emanuel#hinton ~ $ source .bashrc
emanuel#hinton ~ $ python
Python 3.7.3 (default, Jun 24 2019, 04:54:02)
[GCC 9.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyspark
>>>
Everything it's working fine since you correctly imported the environment variables for SparkContext.
Using Apache-Spark on Archlinux via DockerImage
For my use purposes I've created a Docker image with python, jupyter-notebook and apache-spark-2.3.4
running the image
docker run -ti -p 8888:8888 emanuelfontelles/spark-jupyter
just go to your browser and type
http://localhost:8888/tree
and will prompted a authentication page, come back to terminal and copy the token number and voila, will have Archlinux container running a Apache-Spark distribution.
If you are using Pycharm , Got to Run - > Edit Configurations and click on Environment variables to add as below(basically the PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON should point to the same version of Python) . This solution worked for me .Thanks to the above posts.
To make it easier to see for people, that instead of having to set a specific path /usr/bin/python3 that you can do this:
I put this line in my ~/.zshrc
export PYSPARK_PYTHON=python3.8
export PYSPARK_DRIVER_PYTHON=python3.8
When I type in python3.8 in my terminal I get Python3.8 going. I think it's because I installed pipenv.
Another good website to reference to get your SPARK_HOME is https://towardsdatascience.com/how-to-use-pyspark-on-your-computer-9c7180075617
(for permission denied issues use sudo mv)
1. Download and Install Java (Jre)
2. It has two options, you can choose one of the following solution:-
## -------- Temporary Solution -------- ##
Just put the path in your jupyter notebook in the following code and RUN IT EVERYTIME:-
import os
os.environ["PYSPARK_PYTHON"] = r"C:\Users\LAPTOP0534\miniconda3\envs\pyspark_v3.3.0"
os.environ["PYSPARK_DRIVER_PYTHON"] = r"C:\Users\LAPTOP0534\miniconda3\envs\pyspark_v3.3.0"
os.environ["JAVA_HOME"] = r"C:\Program Files\Java\jre1.8.0_333"
----OR----
## -------- Permanent Solution -------- ##
Set above 3 variables in your Environment Variable.
I have gone through many answers but nothing works for me.
But both of these resolution worked for me. This has resolved my error.
Thanks
import os
os.environ["JAVA_HOME"] = "C:\Program Files\Java\jdk-19"
os.environ["SPARK_HOME"] = "C:\Program Files\Spark\spark-3.3.1-bin-hadoop2"
import sys
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
This worked for me in a jupyter notebook as the os library makes our life easy in setting up the environment variables. Make sure to run this cell befor running the sparksession.
I tried two methods for the question. the method in the picture can works.
add environment variables
PYSPARK_PYTHON=/usr/local/bin/python3.7;PYSPARK_DRIVER_PYTHON=/usr/local/bin/python3.7;PYTHONUNBUFFERED=1
I tried this minimal example:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
app.run(debug = True)
When I try python hello.py, everything goes well. However, when I try to run it from Textmate (Shift + Cmd + R) an error is thrown:
Traceback (most recent call last):
File "/Users/user/EventFeed/hello.py", line 1, in <module>
from flask import Flask
ImportError: No module named flask
Textmate calls pythonw instead of python. When I try pythonw myself the same error is thrown.
The man pythonw states that As of Python 2.5, python and pythonw are interchangeable though they appear not to be in this case.
Would you have an idea of what happens?
(Question Code that works with python and not with pythonw does not answer the question despite its similar title.)
The problem is that your pythonw and your python are not pointing at the same Python installations.
Why?
Most likely because you've installed a second Python 2.7 that doesn't include the obsolete pythonw, but Apple's pre-installed Python 2.7 definitely does include it.
The quickest way to check this is the which command. For example, on one of my machines:
$ which python
/usr/local/bin/python
$ which pythonw
/usr/bin/pythonw
That first one is a symlink to a Homebrew install of Python 2.7, while the second is Apple's Python 2.7. Your exact details may differ; the first one may be a symlink to /Library/Frameworks/Python.framework/Versions/2.7/bin/python, or a wrapper executable that actually lives in /usr/local/bin, or it may be in /opt/local, etc. The point is that they're not in the same directories.
At any rate, your two separate installations of Python don't share the same site-packages (and they shouldn't), so the fact that you've installed Flask for the second one doesn't help the Apple one. You can verify this by running them and printing out sys.path:
$ python
>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.7/site-packages', …]
>>> ^D
$ pythonw
>>> import sys
>>> sys.path
['', '/Library/Python/2.7/site-packages', …]
>>> ^D
Anyway, the simplest solution is to configure your editor to run python instead of pythonw—or, better, give it an absolute path to a Python interpreter like /usr/local/bin/python2.7 to make absolutely sure you know what you're running.
(I don't know TextMate very well, but from this source it looks like it has a setting named TM_PYTHON that should control this…)
When trying to launch Mercurial(hg) after a restart in my Ubuntu 9.10 Linux Box I got following message:
abort: couldn't find mercurial libraries in [/usr/bin /usr/local/lib/python2.6/dist-packages/vipy-0.4-py2.6.egg /usr/local/lib/python2.6/dist-packages/nose-0.11.1-py2.6.egg /usr/local/lib/python2.6/dist-packages/rope-0.9.2-py2.6.egg /usr/local/lib/python2.6/dist-packages/Sphinx-0.6.3-py2.6.egg /usr/local/lib/python2.6/dist-packages/django_html-0.0.1-py2.6.egg /usr/local/lib/python2.6/dist-packages/html5lib-0.11.1-py2.6.egg /home/kenny /home/kenny/Projects/soclone-read-only /home/kenny/python/Django /home/kenny/python/pysmell /home/kenny/python/Django/ropemode /home/kenny/python/Django/rope /home/kenny/python/lib /usr/lib/python2.6 /usr/lib/python2.6/plat-linux2 /usr/lib/python2.6/lib-tk /usr/lib/python2.6/lib-old /usr/lib/python2.6/lib-dynload /usr/local/lib/python2.6/dist-packages]
(check your install and PYTHONPATH)
Mysteriously other Python programs don't find their modules, including django-admin, bzr, BUT surprisingly the Python interpreter itself is launching.
Here you can find my current sys.path:
['', '/usr/local/lib/python2.6/dist-packages/vipy-0.4-py2.6.egg', '/usr/local/lib/python2.6/dist-packages/nose-0.11.1-py2.6.egg', '/usr/local/lib/python2.6/dist-packages/rope-0.9.2-py2.6.egg', '/usr/local/lib/python2.6/dist-packages/Sphinx-0.6.3-py2.6.egg', '/usr/local/lib/python2.6/dist-packages/django_html-0.0.1-py2.6.egg', '/usr/local/lib/python2.6/dist-packages/html5lib-0.11.1-py2.6.egg', '/home/kenny', '/home/kenny/Projects/soclone-read-only', '/home/kenny/python/Django', '/home/kenny/python/pysmell', '/home/kenny/python/Django/ropemode', '/home/kenny/python/Django/rope', '/home/kenny/python/lib', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/local/lib/python2.6/dist-packages']
Does anyone know how to resolve this issue?
I know this is no programming question in specific, but it disallows me to program, so I beg your comprehension!
Thanks in advance.
Try this:
update-python-modules -p
(might need to sudo that...)
Source:
http://hg.opensource.lshift.net/mercurial-server/rev/32dba1a70a54
All of the sites I've googled for this say that your PYTHONPATHis set wrong. The code that you are hitting in mercurial appears to be this:
try:
from mercurial import demandimport; demandimport.enable()
except ImportError:
import sys
sys.stderr.write("abort: couldn't find mercurial libraries in [%s]\n" %
' '.join(sys.path))
sys.stderr.write("(check your install and PYTHONPATH)\n")
sys.exit(-1)
So where is demandimport on your machine? On my windows box, it is here:
>>> from mercurial import demandimport
>>> demandimport.__file__
'C:\\Python26\\lib\\site-packages\\mercurial-1.4.1-py2.6-win32.egg\\mercurial\\demandimport.pyc'
And that works because I have mercurial in my PYTHONPATH:
>>> import sys
>>> for s in sys.path:
... print s
...
# Other crud deleted...
C:\Python26\lib\site-packages\mercurial-1.4.1-py2.6-win32.egg
C:\Windows\system32\python26.zip
C:\Python26\DLLs
C:\Python26\lib
C:\Python26\lib\plat-win
C:\Python26\lib\lib-tk
C:\Python26
C:\Python26\lib\site-packages
Your PYTHONPATH makes no mention of mercurial. At a guess, I would add this to your PYTHONPATH:
/usr/local/lib/python2.6/site-packages
and I would re-install mercurial from make. That advice worked well here.
Edit: And on my Ubuntu 9.10 box, I got these results:
>>> import mercurial
>>> mercurial.__file__
'/usr/lib/pymodules/python2.6/mercurial/__init__.pyc'
>>> import sys
>>> for s in sys.path:
... print s
...
/usr/local/lib/python2.6/dist-packages/pip-0.6.1-py2.6.egg
/usr/local/lib/python2.6/dist-packages/virtualenv-1.4.3-py2.6.egg
/usr/lib/python2.6
/usr/lib/python2.6/plat-linux2
/usr/lib/python2.6/lib-tk
/usr/lib/python2.6/lib-old
/usr/lib/python2.6/lib-dynload
/usr/lib/python2.6/dist-packages
/usr/lib/python2.6/dist-packages/PIL
/usr/lib/python2.6/dist-packages/gst-0.10
/usr/lib/pymodules/python2.6
/usr/lib/python2.6/dist-packages/gtk-2.0
/usr/lib/pymodules/python2.6/gtk-2.0
/usr/local/lib/python2.6/dist-packages
/usr/local/lib/python2.6/dist-packages/PIL
And this makes me think the problem is that this is missing for you: /usr/lib/pymodules/python2.6.
Is mercurial located in one of the library installation paths (dist-packages or site-packages)? You can use the find tool to look for it?
Did you have luck installing small libraries and access them from Python on this machine?
Thanks for the effort to all of you.
I've solved the problem thanks to hughdbrown. hughdbrown, you've made me realize that I commited a typo when defining doing $PYTHONPATH; instead of adding the path /usr/lib/pymodules/python2.6, I wrote /usr/lib/pymodules/ so python couldn't import the libraries... However, I corrected it and I'm glad to see Mercurial and Co. work again.
The only strange thing is, WHY it changed... Well, I'll be knowing from now on what to do.
You could try to reinstall affected Python programs with aptitude:
sudo aptitude reinstall mercurial
How do I find the location of my site-packages directory?
There are two types of site-packages directories, global and per user.
Global site-packages ("dist-packages") directories are listed in sys.path when you run:
python -m site
For a more concise list run getsitepackages from the site module in Python code:
python -c 'import site; print(site.getsitepackages())'
Caution: In virtual environments getsitepackages is not available with older versions of virtualenv, sys.path from above will list the virtualenv's site-packages directory correctly, though. In Python 3, you may use the sysconfig module instead:
python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
The per user site-packages directory (PEP 370) is where Python installs your local packages:
python -m site --user-site
If this points to a non-existing directory check the exit status of Python and see python -m site --help for explanations.
Hint: Running pip list --user or pip freeze --user gives you a list of all installed per user site-packages.
Practical Tips
<package>.__path__ lets you identify the location(s) of a specific package: (details)
$ python -c "import setuptools as _; print(_.__path__)"
['/usr/lib/python2.7/dist-packages/setuptools']
<module>.__file__ lets you identify the location of a specific module: (difference)
$ python3 -c "import os as _; print(_.__file__)"
/usr/lib/python3.6/os.py
Run pip show <package> to show Debian-style package information:
$ pip show pytest
Name: pytest
Version: 3.8.2
Summary: pytest: simple powerful testing with Python
Home-page: https://docs.pytest.org/en/latest/
Author: Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris Bruynooghe, Brianna Laugher, Florian Bruhin and others
Author-email: None
License: MIT license
Location: /home/peter/.local/lib/python3.4/site-packages
Requires: more-itertools, atomicwrites, setuptools, attrs, pathlib2, six, py, pluggy
>>> import site; site.getsitepackages()
['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']
(or just first item with site.getsitepackages()[0])
A solution that:
outside of virtualenv - provides the path of global site-packages,
insidue a virtualenv - provides the virtualenv's site-packages
...is this one-liner:
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
Formatted for readability (rather than use as a one-liner), that looks like the following:
from distutils.sysconfig import get_python_lib
print(get_python_lib())
Source: an very old version of "How to Install Django" documentation (though this is useful to more than just Django installation)
For Ubuntu,
python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"
...is not correct.
It will point you to /usr/lib/pythonX.X/dist-packages
This folder only contains packages your operating system has automatically installed for programs to run.
On ubuntu, the site-packages folder that contains packages installed via setup_tools\easy_install\pip will be in /usr/local/lib/pythonX.X/dist-packages
The second folder is probably the more useful one if the use case is related to installation or reading source code.
If you do not use Ubuntu, you are probably safe copy-pasting the first code box into the terminal.
This is what worked for me:
python -m site --user-site
A modern stdlib way is using sysconfig module, available in version 2.7 and 3.2+. Unlike the current accepted answer, this method still works regardless of whether or not you have a virtual environment active.
Note: sysconfig (source) is not to be confused with the distutils.sysconfig submodule (source) mentioned in several other answers here. The latter is an entirely different module and it's lacking the get_paths function discussed below. Additionally, distutils is deprecated in 3.10 and will be unavailable soon.
Python currently uses eight paths (docs):
stdlib: directory containing the standard Python library files that are not platform-specific.
platstdlib: directory containing the standard Python library files that are platform-specific.
platlib: directory for site-specific, platform-specific files.
purelib: directory for site-specific, non-platform-specific files.
include: directory for non-platform-specific header files.
platinclude: directory for platform-specific header files.
scripts: directory for script files.
data: directory for data files.
In most cases, users finding this question would be interested in the 'purelib' path (in some cases, you might be interested in 'platlib' too). The purelib path is where ordinary Python packages will be installed by tools like pip.
At system level, you'll see something like this:
# Linux
$ python3 -c "import sysconfig; print(sysconfig.get_path('purelib'))"
/usr/local/lib/python3.8/site-packages
# macOS (brew installed python3.8)
$ python3 -c "import sysconfig; print(sysconfig.get_path('purelib'))"
/usr/local/Cellar/python#3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
# Windows
C:\> py -c "import sysconfig; print(sysconfig.get_path('purelib'))"
C:\Users\wim\AppData\Local\Programs\Python\Python38\Lib\site-packages
With a venv, you'll get something like this
# Linux
/tmp/.venv/lib/python3.8/site-packages
# macOS
/private/tmp/.venv/lib/python3.8/site-packages
# Windows
C:\Users\wim\AppData\Local\Temp\.venv\Lib\site-packages
The function sysconfig.get_paths() returns a dict of all of the relevant installation paths, example on Linux:
>>> import sysconfig
>>> sysconfig.get_paths()
{'stdlib': '/usr/local/lib/python3.8',
'platstdlib': '/usr/local/lib/python3.8',
'purelib': '/usr/local/lib/python3.8/site-packages',
'platlib': '/usr/local/lib/python3.8/site-packages',
'include': '/usr/local/include/python3.8',
'platinclude': '/usr/local/include/python3.8',
'scripts': '/usr/local/bin',
'data': '/usr/local'}
A shell script is also available to display these details, which you can invoke by executing sysconfig as a module:
python -m sysconfig
Addendum: What about Debian / Ubuntu?
As some commenters point out, the sysconfig results for Debian systems (and Ubuntu, as a derivative) are not accurate. When a user pip installs a package it will go into dist-packages not site-packages, as per Debian policies on Python packaging.
The root cause of the discrepancy is because Debian patch the distutils install layout, to correctly reflect their changes to the site, but they fail to patch the sysconfig module.
For example, on Ubuntu 20.04.4 LTS (Focal Fossa):
root#cb5e85f17c7f:/# python3 -m sysconfig | grep packages
platlib = "/usr/lib/python3.8/site-packages"
purelib = "/usr/lib/python3.8/site-packages"
root#cb5e85f17c7f:/# python3 -m site | grep packages
'/usr/local/lib/python3.8/dist-packages',
'/usr/lib/python3/dist-packages',
USER_SITE: '/root/.local/lib/python3.8/site-packages' (doesn't exist)
It looks like the patched Python installation that Debian/Ubuntu are distributing is a bit hacked up, and they will need to figure out a new plan for 3.12+ when distutils is completely unavailable. Probably, they will have to start patching sysconfig as well, since this is what pip will be using for install locations.
Let's say you have installed the package 'django'. import it and type in dir(django). It will show you, all the functions and attributes with that module. Type in the python interpreter -
>>> import django
>>> dir(django)
['VERSION', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'get_version']
>>> print django.__path__
['/Library/Python/2.6/site-packages/django']
You can do the same thing if you have installed mercurial.
This is for Snow Leopard. But I think it should work in general as well.
As others have noted, distutils.sysconfig has the relevant settings:
import distutils.sysconfig
print distutils.sysconfig.get_python_lib()
...though the default site.py does something a bit more crude, paraphrased below:
import sys, os
print os.sep.join([sys.prefix, 'lib', 'python' + sys.version[:3], 'site-packages'])
(it also adds ${sys.prefix}/lib/site-python and adds both paths for sys.exec_prefix as well, should that constant be different).
That said, what's the context? You shouldn't be messing with your site-packages directly; setuptools/distutils will work for installation, and your program may be running in a virtualenv where your pythonpath is completely user-local, so it shouldn't assume use of the system site-packages directly either.
The native system packages installed with python installation in Debian based systems can be found at :
/usr/lib/python2.7/dist-packages/
In OSX - /Library/Python/2.7/site-packages
by using this small code :
from distutils.sysconfig import get_python_lib
print get_python_lib()
However, the list of packages installed via pip can be found at :
/usr/local/bin/
Or one can simply write the following command to list all paths where python packages are.
>>> import site; site.getsitepackages()
['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']
Note: the location might vary based on your OS, like in OSX
>>> import site; site.getsitepackages()
['/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/site-python', '/Library/Python/2.7/site-packages']
pip show will give all the details about a package:
https://pip.pypa.io/en/stable/reference/pip_show/ [pip show][1]
To get the location:
pip show <package_name>| grep Location
In Linux, you can go to site-packages folder by:
cd $(python -c "import site; print(site.getsitepackages()[0])")
All the answers (or: the same answer repeated over and over) are inadequate. What you want to do is this:
from setuptools.command.easy_install import easy_install
class easy_install_default(easy_install):
""" class easy_install had problems with the fist parameter not being
an instance of Distribution, even though it was. This is due to
some import-related mess.
"""
def __init__(self):
from distutils.dist import Distribution
dist = Distribution()
self.distribution = dist
self.initialize_options()
self._dry_run = None
self.verbose = dist.verbose
self.force = None
self.help = 0
self.finalized = 0
e = easy_install_default()
import distutils.errors
try:
e.finalize_options()
except distutils.errors.DistutilsError:
pass
print e.install_dir
The final line shows you the installation dir. Works on Ubuntu, whereas the above ones don't. Don't ask me about windows or other dists, but since it's the exact same dir that easy_install uses by default, it's probably correct everywhere where easy_install works (so, everywhere, even macs). Have fun. Note: original code has many swearwords in it.
An additional note to the get_python_lib function mentioned already: on some platforms different directories are used for platform specific modules (eg: modules that require compilation). If you pass plat_specific=True to the function you get the site packages for platform specific packages.
This works for me.
It will get you both dist-packages and site-packages folders.
If the folder is not on Python's path, it won't be
doing you much good anyway.
import sys;
print [f for f in sys.path if f.endswith('packages')]
Output (Ubuntu installation):
['/home/username/.local/lib/python2.7/site-packages',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages']
This should work on all distributions in and out of virtual environment due to it's "low-tech" nature. The os module always resides in the parent directory of 'site-packages'
import os; print(os.path.dirname(os.__file__) + '/site-packages')
To change dir to the site-packages dir I use the following alias (on *nix systems):
alias cdsp='cd $(python -c "import os; print(os.path.dirname(os.__file__))"); cd site-packages'
A side-note: The proposed solution (distutils.sysconfig.get_python_lib()) does not work when there is more than one site-packages directory (as recommended by this article). It will only return the main site-packages directory.
Alas, I have no better solution either. Python doesn't seem to keep track of site-packages directories, just the packages within them.
from distutils.sysconfig import get_python_lib
print get_python_lib()
You should try this command to determine pip's install location
Python 2
pip show six | grep "Location:" | cut -d " " -f2
Python 3
pip3 show six | grep "Location:" | cut -d " " -f2
Answer to old question. But use ipython for this.
pip install ipython
ipython
import imaplib
imaplib?
This will give the following output about imaplib package -
Type: module
String form: <module 'imaplib' from '/usr/lib/python2.7/imaplib.py'>
File: /usr/lib/python2.7/imaplib.py
Docstring:
IMAP4 client.
Based on RFC 2060.
Public class: IMAP4
Public variable: Debug
Public functions: Internaldate2tuple
Int2AP
ParseFlags
Time2Internaldate
For those who are using poetry, you can find your virtual environment path with poetry debug:
$ poetry debug
Poetry
Version: 1.1.4
Python: 3.8.2
Virtualenv
Python: 3.8.2
Implementation: CPython
Path: /Users/cglacet/.pyenv/versions/3.8.2/envs/my-virtualenv
Valid: True
System
Platform: darwin
OS: posix
Python: /Users/cglacet/.pyenv/versions/3.8.2
Using this information you can list site packages:
ls /Users/cglacet/.pyenv/versions/3.8.2/envs/my-virtualenv/lib/python3.8/site-packages/
I made a really simple function that gets the job done
import site
def get_site_packages_dir():
return [p for p in site.getsitepackages()
if p.endswith(("site-packages", "dist-packages"))][0]
get_site_packages_dir()
# '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages'
If you want to retrieve the results using the terminal:
python3 -c "import site;print([p for p in site.getsitepackages() if p.endswith(('site-packages', 'dist-packages')) ][0])"
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
I had to do something slightly different for a project I was working on: find the relative site-packages directory relative to the base install prefix. If the site-packages folder was in /usr/lib/python2.7/site-packages, I wanted the /lib/python2.7/site-packages part. I have, in fact, encountered systems where site-packages was in /usr/lib64, and the accepted answer did NOT work on those systems.
Similar to cheater's answer, my solution peeks deep into the guts of Distutils, to find the path that actually gets passed around inside setup.py. It was such a pain to figure out that I don't want anyone to ever have to figure this out again.
import sys
import os
from distutils.command.install import INSTALL_SCHEMES
if os.name == 'nt':
scheme_key = 'nt'
else:
scheme_key = 'unix_prefix'
print(INSTALL_SCHEMES[scheme_key]['purelib'].replace('$py_version_short', (str.split(sys.version))[0][0:3]).replace('$base', ''))
That should print something like /Lib/site-packages or /lib/python3.6/site-packages.
Something that has not been mentioned which I believe is useful, if you have two versions of Python installed e.g. both 3.8 and 3.5 there might be two folders called site-packages on your machine. In that case you can specify the python version by using the following:
py -3.5 -c "import site; print(site.getsitepackages()[1])