This question already has answers here:
Python: How do I find which pip package a library belongs to?
(2 answers)
Closed 24 days ago.
Ok, so you clone a repo, there's an import
import yaml
ok, so you do pip install yaml and you get:
ERROR: No matching distribution found for yaml
Ok, so you look for a package with yaml in it, and there's like a gazillion of them... usually adding py in front does the job, but...
How on earth should I know which one was used?!
And it's not just yaml, oh no... there's:
import cv2 # python-opencv
import PIL # Pillow
and the list goes on and on...
How can I know which import uses which package? Shouldn't there be a PEP for this? Or a naming convention, e.g. import is always the same as the package name?
There's a similar topic here, if you're not frustrated enough :)
[When I clone a repo,] How can I know which import uses which package?
In short: it is the cloned code's responsibility to explain this, and it is an expected courtesy that the cloned code includes an installer that will take care of it.
If this is just some random person's bundle of .py files on GitHub with no installation instructions, look for notes in the associated documentation; failing that, make an issue on the tracker. (Or just give up. Maybe look for a better-engineered project that does the same thing.)
However, most "serious", contemporary Python projects are meant to be installed by using some form of packaging system. These have evolved over the years, and best practices have changed many times; but generally speaking, a properly "packaged" and "distributed" project will have either a setup.py or (newer; better in many ways, but not universally adopted yet) pyproject.toml file at the top level.
A pyproject.toml file is a config file in TOML format that simply describes a bunch of project metadata. This requires a build backend conforming to PEP 517. For a while, this required third-party tools, such as Poetry; but the standard setuptools can handle this since version 40.8.0. (As of this writing, the current release is 65.7.0.)
A setup.py script is executable code that pip will invoke after downloading a package from PyPI (or another package index). Generally, this script will use either setuptools or distutils (the predecessor to setuptools; it has finally been officially deprecated in 3.10, and will be removed in 3.12) to install the project, by calling a function named setup and passing it a big dict with some project metadata.
Security warning: this file is still executable code. It is arbitrary code, and it doesn't have to be following the standard conventions. Also, the package that is actually downloaded from PyPI doesn't necessarily match the project's source shown on GitHub (or another Git provisioning website), if such is even available. (This problem also affects package managers in other languages and ecosystems, notably npm for Javascript.)
With the setup.py based approach, package dependencies are specified using a keyword argument to the setup function. The specification has changed many times; currently, projects still using a setup.py should use the install_requires keyword argument.
With the pyproject.toml based approach, using setuptools' backend, dependencies will be an array (using JSON terminology, as TOML is a superset) stored under project.dependencies. This will vary for other backends; for example, Poetry expects this information under tool.poetry.dependencies.
In any event, pip freeze will output a list of what's installed in the current environment. It's a somewhat common practice for developers to test the code in a virtual environment where the dependencies are installed, dump this output to a requirements.txt file, and include that as documentation.
[When I want to use a third-party library in my own code,] How can I know which import uses which package?
It's worth considering the question the other way around, too: given that we have installed OpenCV for Python using pip install opencv-python, and want to use it in our own code, how do we know to import cv2 specifically?
The answer: there is no convention, and certainly no requirement for the installed package name to match the PyPI name, nor the GitHub etc. repository name. Read the documentation. Everyone who intends for their code to be used as a library, will be more than willing to show how, on at least a basic level.
Watch for requirements.txt . Big projects usually have it. You can import packages from this file. Else just google.
Keep in mind that it might not be a pip package.
Probably what is happening is that the main script is trying to import a secondary script (yaml.py, in this case) with functions or utils for the main script to use.
Check if the repo contains a file named yaml.py. If it's the case make sure to run the main script while the yaml.py is in the same directory.
Also, check for a requirements.txt file.
You can install all the requirements inside the file running in shell this line:
pip install -r *path to your requirements.txt*
Hope that this helps.
Any package on PyPI or cloned from some online repository is free to set itself up with a base directory name it chooses. That base directory xyz determines the import xyz line. Additionally a package name on PyPI doesn't have to match the repository name where its source code revisions are kept (assuming there is any).
This has the disadvantage that there is no one-to-one relation between package name, repo and/or import-line. But the advantage is that you e.g. can install Pillow, which is backwards compatible with PIL and still use import PIL instead of changing all your sources to use import Pillow as PIL.
If the repo you clone has a requirements.txt look there, you can also look in the setup.py for extra_require. But there is no guarantee that these are available, or contain the names of the packages to install (e.g. I use a generic setup.py that reads its info from a datastructure in the __init__.py file when creating/installing a package).
yaml seems to be a reserved name on PyPI (at least when I tried to upload a package with that name a few years ago). So that might be the reason the package is named PyYAML, although the Py is not very informative as the python code will not function in another programming language. PyPI' search is not very helpful as it relevance ordering is not relevant (at least not for yaml).
PyPI has no entry in the metadata for the import line, but you could extract that from .whl package file as the import line is the top level directory that doesn't match .dist-info. This is normally not possible from a .tar.gz` package file. I don't know of any site that does this kind of automatic scraping.
You can click through the packages on PyPI, after searching the import term, and hope you find something that matches the import in the documentation, but that is no guarantee you get the right one.
You might be best of searching for import yaml here on stackoverflow, and hope that the question or the answer mentions the package name.
thank you very much for your help and ideas. Big thanks to Karl Knechter for his exhaustive answer.
tl;dr: I think using some sort of "package" / "distribution" as a standard, would make everyone's lives easier.
However, my question was half-theoretical, to point out something I'd call, an incoherence in Python. You are of course right, there should be setuptools or requirements.txt or at least some documentation. But, if there isn't any, we're prone to error or additional browsing.
GospelBG pointed out something important. There could be a script yaml.py in the main folder and we need to check and/or guess.
Most importantly, naming imports differently than packages is just plainly misleading. There should be a naming convention or a PEP for this. Again, you can of course eventually get the proper package etc., but it's not explicit and obvious, and it should be! Because in programming, we like it that way, don't we?
I'm no seasoned dev in Python and I'm learning C++, but e.g. in C++, you import a header file with a particular name and static or dynamic libraries by their filename. Now I know this is very "step-by-step, on foot method", but at least you use the exact filenames.
On the upper level you have CMake, which would be an equivalent of setuptools where using find_package or find_library you can import package / library. To be honest, I'm not sure if all packages have the exact equivalent name, but at least the ones I used, did match.
Thanks again for your help and answers! I'm open for discussion and comments :)
Sorry if it is a stupid question.
Normally when I need a package, I import filename.py.
How is about this https://github.com/simplegeo/python-oauth2 ? How can I import this package since I cannot find oauth2.py file.
Do I copy this oauth2 folder to project root and then do import oauth2?
I am just a beginner so can you give me detail instruction?
Thanks for your time
Read this first.
Now if you check out oauth2/__init__.py, you will see oauth already do import httplib2 so you don't have to import it yourself. (Unless you also are using httplib2, of course.)
In essence, you are correct. If you want to use a python package in a GAE app you have to have a copy of it it in the applications root directory where it can be accessed by your application as a normal import. It will then be uploaded along with your applications code and be usable when deployed too.
There are exceptions where you don't have to do this as the library is provided for you, you can read about those here: Supported 3rd Party Libraries
For those you need to edit your app.yaml.
For oauth2 you should look at google specific implementations. I don't know what you've linked to there, all I know is the endless stream of oauth/gae questions usually end up here: https://developers.google.com/appengine/docs/python/oauth/overview
I was looking for something similar to perl's Dumper functionality in python. So after googling I found one which serves me well # https://gist.github.com/1071857#file_dumper.pyamazon
So I downloaded and installed it and it works fine.
But then I came accross PyPI: http://pypi.python.org/pypi which looks like CPAN equivalent for python.
So I searched for the Dumper module there and I could not find it there. I was hoping that this seems like a very basic module and should have been listed in PyPI.
So my question is, if I have to install a python module, then should I search in PyPI first and if i do not find then look other places on google?
OR is there any other Python Module repository apart from PyPI?
I am learning python and hence this question.
thanks.
If you are using pip, pip search package_name would help you do the same as searching on the web interface provided by PyPi.
Once located, installing a python package is of course as easy as
pip install package_name
Some python libraries may be in development stage and may not directly be available on PyPi OR you may want a specific commit has (git) of that library and if you can find that library's source on github.com or on bitbucket.com for example, you can do
pip install -e git+git://github.com/the/repo/url.git#egg=package_name
And regarding your question about perl Dumper, perl's Dumper has two main uses iirc -
data persistence
debugging and inspecting objects.
As far as I know, there's no exact equivalent of perl's Dumper in python.
However, I use pickle for data persistence.
And pprint is useful for visually inspecting objects/debug.
Both of which are standard, built-in modules in Python. There's no necessity for 3rd party libraries for these functionalities.
If you want to use what is here - https://gist.github.com/1071857#file_dumper.pyamazon.
What you need to do is to copy the code and place it in a local file in your project directory. You can name the file something like pydumper.py. Or any name you prefer really, but end it with suffix .py.
In your project, you can import the functions and classes defined in pydumper.py by doing
from pydumper import *
or if you want to be specific (which is preferred. it's better to be explicit about what you are importing.)
from pydumper import Dumper
and you can start using the Dumper class in your own code.
Are you looking for something like easy_install from setuptools? I might have misunderstood your question as I don't use perl.
From the Scripts directory in the python installation directory ("c:/python27/Scripts" on my machine), you can install modules from the command line like so:
easy_install modulename
Makes life alot easier if you set the Scripts directory to your PATH variable.
Is there a documentation for Python 'gnomekeyring' lib somewhere in the Web?
There is the reference for the C library which the Python package uses. Most function names are identical (except for the "gnome_keyring_" prefix). The Bending Gnome Keyring with Python blog series should give you a good start as well.
And as the keyring package was already mentioned: If I remember correctly, that package supports gnome-keyring as a backend, so you can look at its source code to find out how to use gnome-keyring.
Apparently not. But this one is documented, maybe that's an option?
http://pypi.python.org/pypi/keyring
If not, maybe you can figure out how to use gnomekeyring from reading the source of keyring. :)
For my project I would be using the argparse library. My question is, how do I distribute it with my project. I am asking this because of the technicalities and legalities involved.
Do I just:
Put the argparse.py file along with
my project. That is, in the tar file for my project.
Create a package for it for my
distro?
Tell the user to install it himself?
What's your target Python version? It appears that argparse is included from version 2.7.
If you're building a small library with minimal dependencies, I would consider removing the dependency on an external module and only use facilities offered by the standard Python library. You can access command line parameters with sys.argv and parse them yourself, it's usually not that hard to do. Your users will definitely appreciate not having to install yet another third party module just to use your code.
It would be best for the user to install it so that only one copy is present on the system and so that it can be updated if there are any issues, but including it with your project is a viable option if you abide by all requirements specified in the license.
Try to import it from the public location, and if that fails then resort to using the included module.
You could go with Ignacio's suggestion.
But... For what it is worth, there's another library for argument parsing built into Python, which is quite powerful. Have you tried optparse? It belongs to the base Python distribution and has been there for a while...
Good luck!