I'm developing a Python application and in the process of branching off a release. I've got a PyPI server set up on a company server and I've copied a source distribution of my package onto it.
I checked that the package was being hosted on the server and then tried installing it on my local development machine. I ended up with this output:
$ pip3 install --trusted-host 172.16.1.92 -i http://172.16.1.92:5001/simple/ <my-package>
Collecting <my-package>
Downloading http://172.16.1.92:5001/packages/<my-package>-0.2.0.zip
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\setup.py", line 9, in <module>
import appdirs
ModuleNotFoundError: No module named 'appdirs'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\<me>\AppData\Local\Temp\pip-build-ubb3jkpr\<my-package>\
The reason is that I'm trying to import a third-party library appdirs in my setup.py, which is necessary for me to compute the data_files argument to setup():
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
import os
from collections import defaultdict
import appdirs
from <my-package>.version import __version__ as <my-package>_version
APP_NAME = '<my-app>'
APP_AUTHOR = '<company>'
SYSTEM_COMPONENT_PLUGIN_DIR = os.path.join(appdirs.user_data_dir(APP_NAME, APP_AUTHOR), 'components')
# ...
setup(
# ...
data_files=component_files,
)
However, I don't have appdirs installed on my local dev machine and I don't expect the end users to have it either.
Is it acceptable to rely on third-party libraries like this in setup.py, and if so what is the recommended approach to using them? Is there a way I can ensure appdirs gets installed before it's imported in setup.py, or should I just document that appdirs is a required package to install my package?
I'm ignoring licensing issues in this answer. You definetly need to take these into account before you really do a release.
Is it acceptable to rely on third-party libraries like this in setup.py
Yes, it is acceptable but generally these should be minimized, especially if these are modules which have no obvious use for the end-user. Noone likes to have packages they don't need or use.
what is the recommended approach to using them?
There are basically 3 options:
Bootstrap them (for example use pip to programmatically install packages). For example setuptools provides an ez_setup.py file that can be used to bootstrap setuptools. Maybe that can be customized to download and install appdirs.
Include them (especially if it's a small package) in your project. For example appdirs is basically just a single file module. Pretty easy to copy and maintain in your project. Be very careful with licensing issues when you do that!
Fail gracefully when it's not possible to import them and let the user install them. For example:
try:
import appdirs
except ImportError:
raise ImportError('this package requires "appdirs" to be installed. '
'Install it first: "pip install appdirs".')
You could use pip to install the package programmatically if the import fails:
try:
import appdirs
except ImportError:
import pip
pip.main(['install', 'appdirs'])
import appdirs
In some circumstances you may need to use importlib or __import__ to import the package after pip.main or referesh the PATH variable. It could also be worthwhile to include a verification if the user really wants to install that package before installing it.
I used a lot of the examples from "Installing python module within code" and I haven't personally tried used this in setup.py files but it looks like it could be a solution for your question.
You can mention install_requires with the dependencies list. Please check the python packaging guide here. Also you can provide a requirements.txt file so that it can be run at once using "pip install -r"
Related
I uploaded a package on PyPi using twine, and it went fine.
Now I'm trying to install that package and importing it into a script.
According to pip the module is already installed correctly:
PS C:\Users\alber> pip install ethbotutils
Requirement already satisfied: ethbotutils in c:\users\alber\appdata\local\programs\python\python39\lib\site-packages (1.1)
But when I try to import it in a script or in a IDE or in Python IDLE i get:
>>> import ethbotutils
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
import ethbotutils
ModuleNotFoundError: No module named 'ethbotutils'
This is the pyproject.toml file (stored in the project root):
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
And this the setup.py file (stored withing the package directory):
from setuptools import setup
setup(
name='ethbotutils',
version=1.0,
packages=["."],
install_requires=["requests~=2.25.1", "PyYAML~=5.4.1"],
python_requires=">=3.6"
)
EDIT:
What #a_guest suggested seems to be working: if I import a script present in the package, like "bot_utils" everything works, but it still doens't when I try to import the whole package by its name. How can I fix that?
The name of a distribution (the name parameter of setup) determines how a distribution (or project) is identified within the Python ecosystem; this includes the Python Package Index, where the distribution will be located at the URL https://pypi.org/project/<name>/.
This is distinct from the actual package(s) that a distribution contains (the packages parameter of setup). It is these packages that are made available when installing a distribution (e.g. via pip).
So as an example, if the setup.py file contains the following specification
setup(
name='foo',
packages=['bar'],
...
)
then this will create a distribution named foo which installs a package named bar; i.e. after doing pip install foo, the content of that distribution (the package) can be accessed via import bar. Typically the name of a distribution and the top-level package should coincide in order to avoid conflicts with other distributions which might get installed into the same virtual environment (see this answer for more information).
For the specific example of the OP, this means that the setup.py file should contain the following specification:
setup(
name='ethbotutils',
packages=['ethbotutils'],
...
)
In order for the setup to work, all relevant Python modules need to be placed inside a local folder ethbotutils which exists next to the setup.py file.
I am following along with the O'Riley Head First Python (2nd Edition) Course.
At one point you will create a webapp and deploy it to pythonanywhere (chapter5).
The webapp uses two functions, imported from a module, created earlier.
The module is called vsearch.py. I also created a readme.txt and a setup.py and used setuptools to create a source distribution file using :
python3 setup.py sdist
The code of the setup.py read as follows:
from setuptools import setup
setup(
name = "vsearch",
version = "1.0",
description = "The Head First Python Seach Tools",
author = "HF Python 2e",
author_email = "hfpy2e#gmail.com",
url = "headfirstlabs.com",
py_modules = ["vsearch"],
)
The source distribution file gets created without errors and creates a file called vsearch-1.0.tar.gz
The file then gets uploaded to pythonanywhere and installed via console using:
python3 -m pip install vsearch-1.0.tar.gz --user
Console outputs:
15:36 ~/mysite $ python3 -m pip install vsearch-1.0.tar.gz --user
Looking in links: /usr/share/pip-wheels
Processing ./vsearch-1.0.tar.gz
Building wheels for collected packages: vsearch
Running setup.py bdist_wheel for vsearch ... done
Stored in directory: /home/Mohr/.cache/pip/wheels/85/fd/4e/5302d6f3b92e4057d341443ed5ef0402eb04994663282c12f7
Successfully built vsearch
Installing collected packages: vsearch
Found existing installation: vsearch 1.0
Uninstalling vsearch-1.0:
Successfully uninstalled vsearch-1.0
Successfully installed vsearch-1.0
Now when I try to run my webapp I get the following error:
2020-03-24 16:18:14,592: Error running WSGI application
2020-03-24 16:18:14,592: ModuleNotFoundError: No module named 'vsearch'
2020-03-24 16:18:14,593: File "/var/www/mohr_eu_pythonanywhere_com_wsgi.py", line 16, in <module>
2020-03-24 16:18:14,593: from vsearch4web import app as application # noqa
2020-03-24 16:18:14,593:
2020-03-24 16:18:14,593: File "/home/Mohr/mysite/vsearch4web.py", line 3, in <module>
2020-03-24 16:18:14,593: from vsearch import search4letters
Judging from this error I assume that "vsearch" can not be found because it was installed as "vsearch-1.0". However when I try to change this line to:
from vsearch-1.0 import search4letters
I rightfully get a synthax error since I can not adress modules this way. So what can I do about this? When creating the module in the beginning I added a version number to the setup.py file because according to the lecture it is good practice. Setuptools then automatically creates the source distribution file with the "-1.0" at the end. Also when importing it using the command shown above i automatically gets importet as "vsearch-1.0" which in turn I am unable to reference in my python code because of bad synthax.
Am I doing something wrong? Is there a way to import this under another namespace? Is there a way to reference "vsearch-1.0" in my python code without getting a synthax error?
There are different python3 versions installed on PythonAnywhere. When you install something using python3 -m pip or pip3 you use default python3 that is probably not matching python version setting of your web app. Use python3.7 and pip3.7 or python3.6 and pip3.6 etc. for --user installations to be sure.
pip install --user (with emphasized --user) installed the package into your user directory: /home/Mohr/.local/lib/pythonX.Y/site-packages/.
To run your WSGI application you probably use a virtual environment in which the user-installed modules are not available. To use modules in the venv you have to install everything in the venv. So activate the venv in a terminal and install the module with the venv's pip:
pip install vsearch-1.0.tar.gz
I am using numpy.distutils to setup a package (mypackage) that has a frotran module. The problem is that if I do pip install mypackage on an environment that does not have numpy, I get the following error:
ModuleNotFoundError: No module named 'numpy'
The easy solution is to ask users (if I manage to have any) to pip install numpy before they install my package, but I do not think this is a very elegant solution.
I came up with the idea of calling setuptools.setup with only setup_requires=['numpy'] before I import numpy and it seems to work well. This is my setup.py:
import setuptools
setuptools.setup(
setup_requires=[
'numpy'
],)
from numpy.distutils.core import setup, Extension
mod = Extension(name='mypackage.amodule', sources=['source/a.f90'])
setup(name='mypackage',
packages=['mypackage'],
ext_modules=[mod],)
I honestly don't fully understand what it implies to call an empty setup() (no name, no package). Is this a good solution? Is this somehow a bad practice?
It is a common issue. How to install a build-time dependency? You might want to use a pyproject.toml file and take advantage of the build-system feature. See PEP517. And an example here:
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "numpy"]
Use the pep517 tool to build the distributions (sdist and wheel).
Installing MySQLDb (Python module) without Internet Connection
Hi all,
I need to install MySQLDb on a SLES 11 Development Server. This Dev Server does not have access to the public internet due to corporate firewall policies. I was assuming this would merely be a nuisance which would force me to do source installations etc on things more easily installed otherwise. Instead, I have hit a wall, during my attempt to ultimately install and run Django with MySQL support.
I have downloaded and unzipped MySQL-python-1.2.4, and I am attempting to run its setup.py. However, any attempt to run setup.py, even just sudo python setup.py --help, reults in the following
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.gz
And then of course the download fails.
I downloaded and installed the current version of the distribute module (0.7.3), via sudo python setup.py build install. I assumed this would keep mysql-python's setup.py from trying to download distribute. But that does not appear to matter. I did have a quick look at mysql-client's setup.py to see where the download was being forced. It appears that its setup.py does this:
from distribute_setup import use_setuptools
use_setuptools()
Which calls this:
def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
to_dir=os.curdir, download_delay=15, no_fake=True):
# making sure we use the absolute path
to_dir = os.path.abspath(to_dir)
was_imported = 'pkg_resources' in sys.modules or \
'setuptools' in sys.modules
try:
try:
import pkg_resources
if not hasattr(pkg_resources, '_distribute'):
if not no_fake:
_fake_setuptools()
raise ImportError
except ImportError:
return _do_download(version, download_base, to_dir, download_delay)
And this, the import of pkg_resources fails (I have reproduced this from the command line); and the exception handler tries a download, which of course fails.
My understanding is that distribute is deprecated anyway, and that setuptools should be used instead. I do have setuptools installed; but is the mysqldb module hardcoded to use distribute, and possibly a specific version of distribute, and that's my issue? To be honest at this point I'm a bit confused about modules, dependencies, etc in Python (I'm quite mediocre at Python).
Thanks all,
Bean
git clone https://github.com/PyMySQL/PyMySQL or download tarball. for mysqldb: https://github.com/farcepest/MySQLdb1
untar it
run sudo python setup.py install
That's all.
Guess it's too late, but for the sake of the future googlers...
I had the same problem. To solve it I had to comment these lines:
if not hasattr(pkg_resources, '_distribute'):
if not no_fake:
_fake_setuptools()
raise ImportError
After this I was able to install MySQL-python via python setup.py install.
The answer to this question appears to be version dependent. MySQLDb version 1.2.5 (newest version as of this writing) and later do not require distribute so python setup.py install will work. Version 1.2.5 was released 01/2014 so this question is just that old.
Does anybody encounter this warning when executing python setup.py install of a PyPI package?
install_requires defines what the package requires. A lot of PyPI packages have this option. How can it be an "unknown distribution option"?
python setup.py uses distutils which doesn't support install_requires. setuptools does, also distribute (its successor), and pip (which uses either) do. But you actually have to use them. I.e. call setuptools through the easy_install command or pip install.
Another way is to import setup from setuptools in your setup.py, but this not standard and makes everybody wanting to use your package have to have setuptools installed.
This was the first result on my google search, but had no answer.
I found that upgrading setuptools resolved the issue for me (and pip for good measure)
pip install --upgrade pip
pip install --upgrade setuptools
Hope this helps the next person to find this link!
I'm on a Mac with python 2.7.11. I have been toying with creating extremely simple and straightforward projects, where my only requirement is that I can run python setup.py install, and have setup.py use the setup command, ideally from distutils. There are literally no other imports or code aside from the kwargs to setup() other than what I note here.
I get the error when the imports for my setup.py file are:
from distutils.core import setup
When I use this, I get warnings such as
/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
If I change the imports (and nothing else) to the following:
from distutils.core import setup
import setuptools # noqa
The warnings go away.
Note that I am not using setuptools, just importing it changes the behavior such that it no longer emits the warnings. For me, this is the cause of a truly baffling difference where some projects I'm using give those warnings, and some others do not.
Clearly, some form of monkey-patching is going on, and it is affected by whether or not that import is done. This probably isn't the situation for everyone researching this problem, but for the narrow environment in which I'm working, this is the answer I was looking for.
This is consistent with the other (community) comment, which says that distutils should monkeypatch setuptools, and that they had the problem when installing Ansible. Ansible appears to have tried to allow installs without having setuptools in the past, and then went back on that.
https://github.com/ansible/ansible/blob/devel/setup.py
A lot of stuff is up in the air... but if you're looking for a simple answer for a simple project, you should probably just import setuptools.
ATTENTION! ATTENTION! Imperfect answer ahead. To get the "latest memo" on the state of packaging in the Python universe, read this fairly detailed essay.
I have just ran into this problem when trying to build/install ansible. The problem seems to be that distutils really doesn't support install_requires. Setuptools should monkey-patch distutils on-the-fly, but it doesn't, probably because the last release of setuptools is 0.6c11 from 2009, whereas distutils is a core Python project.
So even after manually installing the setuptools-0.6c11-py2.7.egg running setup.py only picks up distutils dist.py, and not the one from site-packages/setuptools/.
Also the setuptools documentation hints to using ez_setup and not distutils.
However, setuptools is itself provided by distribute nowadays, and that flavor of setup() supports install_requires.
This is a warning from distutils, and is a sign that you do not have setuptools installed.
Installing it from http://pypi.python.org/pypi/setuptools will remove the warning.
In conclusion:
distutils doesn't support install_requires or entry_points, setuptools does.
change from distutils.core import setup in setup.py to from setuptools import setup or refactor your setup.py to use only distutils features.
I came here because I hadn't realized entry_points was only a setuptools feature.
If you are here wanting to convert setuptools to distutils like me:
remove install_requires from setup.py and just use requirements.txt with pip
change entry_points to scripts (doc) and refactor any modules relying on entry_points to be full scripts with shebangs and an entry point.
sudo apt-get install python-dev # for python2.x installs
sudo apt-get install python3-dev # for python3.x installs
It will install any missing headers. It solved my issue
As far as I can tell, this is a bug in setuptools where it isn't removing the setuptools specific options before calling up to the base class in the standard library: https://bitbucket.org/pypa/setuptools/issue/29/avoid-userwarnings-emitted-when-calling
If you have an unconditional import setuptools in your setup.py (as you should if using the setuptools specific options), then the fact the script isn't failing with ImportError indicates that setuptools is properly installed.
You can silence the warning as follows:
python -W ignore::UserWarning:distutils.dist setup.py <any-other-args>
Only do this if you use the unconditional import that will fail completely if setuptools isn't installed :)
(I'm seeing this same behaviour in a checkout from the post-merger setuptools repo, which is why I'm confident it's a setuptools bug rather than a system config problem. I expect pre-merge distribute would have the same problem)
I've now seen this in legacy tools using Python2.7, where a build (like a Dockerfile) installs an unpinned dependancy, for example pytest. PyTest has dropped Python 2.7 support, so you may need to specify version < the new package release.
Or bite the bullet and convert that app to Python 3 if that is viable.
It works fine if you follow the official documentation:
import setuptools
setuptools.setup(...)
Source: https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py