In our project, we are using Click to create command-line interfaces. We have a folder called foo in the project, and __main__.py file there contains Click.group definition.
For now, we can access the cli using -
python foo [OPTIONS]
but, we wish to simplify it to just -
foo [OPTIONS]
I had a look at Setuptools integration using Click, and I was successful in creating command after following the instructions. But, what steps are to be done next is unclear to me.
After I introduce setup.py to the project, do I need to ask everyone using the project to run pip install --editable .?
What happens if someone adds new commands/options in the project? Will all users again need to run pip commands?
The project will always run in UNIX environment and we are wondering if there exists a shebang-way of running foo/__main__.py with just foo. If we had foo (not foo.py) as file containing Click.group definition, adding a shebang at the top of file would remove the need to use python while invoking commands.
pip install --editable . means that if you manipulate the files inside the package, it will have immediately an effect when you use the package in Python, because import yourpackagename will refer to the folder with the original package.
pip install . will copy the package folder into some other path and use that when you use import yourpackagename, so any modifications to the original package files will not have an effect until you reinstall it.
Related
I want to build an application, say it's called helloworld, with a command line interface in python. My application as multiple nested modules and a top level module main.py.
Say the layout is :
helloworld/
helloworld/
__init__.py
module1/
module2/
module3/
setup.py
main.py
At this point, once my project is installed I can run it using
#> python path/to/helloworld/main.py arg1 arg2 arg3
I want to be able to interact with the program using commands such as :
#> helloworld arg1 arg2 arg3, in a similar manner as for example, the Flask framework comes with a command line application I can use with #> flask run.
I looked around and found questions suggesting using a shebang. As far as I can tell, this is not what I want.
I want a self contained application I can install using pip and then launch directly without any reference to the python interpreter.
How should I go about this ?
EDIT
It is an important requirement to me that the package can be installed using pip and the command line interface is immediately available. Shebangs/fiddling manually with the path/creating an additional shell script do not solve my problem.
You should add main.py to your PATH. What happens when you are running flask run is that your teminal looks up the command flask in PATH and runs the program that it is pointing to. You could see it as a kind of shortcut to the program Flask.
By adding your program to your PATH, you can tell the computer that if you type helloworld in your terminal, the terminal should run /my/path/to/helloworld.py.
I don't know what OS you are on, so here are links for most common OS on how to add a PATH variable.
Windows
Linux
Mac OSX
Edit: After you gave more explanation, setuptools is what you are looking for. Please look at this reference to see how to use it.
There's an explanation here of how to tell setuptools to add a script to the Python Scripts-folder during the pip installation. You can then make that script be for example a bat-file that calls your Python-script. In that way you achieve both that you do not need to write .py for your script to run, and that it happens automatically.
I know this has already been answered, but just in case someone comes across this in the future..
Here's what worked for a new package I'm developing after reading up on this from many different sources.
tldr; use setuptools and add your command as a console_scripts option under [options.entry_points] in setup.cfg.
You will also need some other fields in setup.cfg or setup.py for package discovery to work. Check out the setuptools docs for that.
Example entry point:
[options.entry_points]
console_scripts =
mycmd = mypackage.mymodule:myfunc
For local development, you can install the package in editable (development) mode with pip install -e . which will allow you to test the command as if you pip intalled it.
Alternatively, you could create a __main__.py that initializes your module when you run python -m mypackage from the root.
from mypackage import app
if __name__ == '__main__':
app.main()
I personally opted for both. Usually your main.py (or app.py in my example) belongs in your package not project level dir. Then you would create __main__.py there as well.
I made a python (3) package and I have been trying to upload it on Github. I also know how to push and install a git using pip. To test if it works as anticipated, I made a virtual environment on my local computer (linux) and pip installed my already pushed private package in there without a problem.
The issue is that I don't know how to access it!!! (I know how to activate and use virtualenvs; I don't know how to call my package) My package has a main interface that one would need to call it in terminal as follows:
python3 myui.py some_args *.data
and it's supposed to create some files where it's called. In other words, it's not exactly a module like numpy to be imported. I have watched/read many tutorials and documentations on the web and tbh I'm lost here.
You are looking for the -m flag. If you installed everything correctly, then the following command should allow you to run your script (based on your example). Note that you shouldn't add the file extension '.py'.
python3 -m myui some args *.data
If you have an actual package (directory with __init__.py file and more) instead of a module (a single .py file), then you can add a __main__.py file to that package. Python will execute this script when you use the -m flag with the package's name, in the same way as shown above.
python3 -m mypackage some args *.data
If you want to run a different script that is nested somewhere inside of that package, you can still run it by specifying its module name:
python3 -m mypackage.subpackage.myscript some args *.data
Another common way to make your script available uses the setup script (setup.py) or setup configuration file (setup.cfg) that is used to install the module or package. In that case, you can add an entry point to map a command to a specific module/function/etc. (as described in this Python packaging tutorial) so that you can run that command instead of having to use the -m flag with Python.
$ mycommand some args *.data
I have a python application (Django based), and I have a couple of standalone maintenance scripts that go along with the application, that I have to call every now and then. They have to import parts of my application (sub-packages). Currently, I just put them in my toplevel directory:
application/
djangoproject/
djangoapp/
otherpackage/
tool1.py
tool2.py
Where tool1.py would do
from djangoproject import wsgi
from djangoapp.models import Poll
I've accumulated quite some of these tools, and would like to move them to a scripts subdirectory. Then, I would like to be able to call them via python scripts/tool1.py or maybe cd scripts; python tool1.py.
I understand (and sometimes lament) how Python's imports work, and I know that I can add some lines to each script to add the parent directory to PYTHONPATH. I am wondering if there is a widespread pattern to handle such a collection of assorted scripts. Maybe one could put the path manipulation into another file, and have every script start with import mainproject?
I am using a virtualenv, and installing dependencies with pip. But the application itself currently doesn't use a setup.py, and I think it wouldn't help to move the scripts to a separate package installed via pip, since I change them a lot during development, and there are lots of one-offs.
The ways for organizing the source code vary from project to project. From the years of my experience, the best and the most pythonic way is to always have setup.py.
In that case, you can make pip install -e . and the editable version from . dir will be pseudo-installed to the virtualenv. Actually, not really installed (i.e. copied), but "linked": the source code dir will be added to sys.path with .pth files, so you can edit & try without any special copying/installing steps afterward.
More on that, you can extend setup.py with extra dependencies for e.g. the development purposes, and install them by pip install -e .[dev]. More like a fancy consequence.
The rest depends on the nature of the scripts.
If the scripts are part of the application, they should be installed via the entry-points in setup.py.
# setup.py:
setup(
entry_points={
'console_scripts': [
'tool1 = mytools.tool1:main',
'tool2 = mytools.tool2:main',
],
},
)
In that case, after pip install -e ., they will be in the bin folder of the virtualenv, or in /usr/local/bin or alike if the system python is used. You can execute them like this:
source .venv/bin/activate
tool1 ...
# OR:
~/path/to/venv/bin/tool2
The scripts installed this way are fully aware of the virtualenv, to which they were installed, so no activation and no explicit python binary are needed.
If the scripts are for the code maintenance, and not semantically part of the application, then they are usually put into ./scripts/ directory (or any other, e.g. ./ci/), with shebang at the top (#!/usr/bin/env python). E.g., tool1.py:
#!/usr/bin/env python
def main():
pass
if __name__ == '__main__':
main()
And executed in the current virtualenv due to this shebang as follows:
source .venv/bin/activate
./scripts/tool1.py ...
# OR:
~/path/to/venv/bin/python ./scripts/tool1.py
Unlike the scripts installed via the entry points, these scripts do not know about their own virtualenv in any way, so the virtualenv should be activate or proper python used explicitly.
This way is also used when the scripts are non-python, e.g. for the bash scripts.
In both cases, the requirements.txt file is sometimes used to pin the application's & dependencies' versions (with pip freeze), so that the deployments would be persistent & predictable. But this is another story — about the deployment of the application, not about the packaging & maintenance.
The requirements.txt file is regenerated from time to time to satisfy the new unpinned (i.e. flexible) requirements in setup.py and the new package versions available. But usually it is the generated content (despite being committed in the repo), not the content maintained by hand.
If you strictly do not want to have setup.py for any reason, then either execute those scripts with the modified env var:
PYTHONPATH=. python scripts/tool1.py
Or hack the sys.path from inside:
# tools1.py
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
This is exactly what pip install -e . does, just done manually on every call, not once with the .pth file in the virtualenv. And also this looks hacky.
However, as we know, neither hacky solutions nor the duplicating solutions, especially those duplicating the standard toolkit, are considered "pythonic".
I want to add a specific library path only to python2. After adding export PYTHONPATH="/path/to/lib/" to my .bashrc, however, executing python3 gets the error: Your PYTHONPATH points to a site-packages dir for Python 2.x but you are running Python 3.x!
I think it is due to that python2 and python3 share the common PYTHONPATH variable.
So, can I set different PYTHONPATH variables respectively for python2 and python3. If not, how can I add a library path exclusively to a particular version of python?
PYTHONPATH is somewhat of a hack as far as package management is concerned. A "pretty" solution would be to package your library and install it.
This could sound more tricky than it is, so let me show you how it works.
Let us assume your "package" has a single file named wow.py and you keep it in /home/user/mylib/wow.py.
Create the file /home/user/mylib/setup.py with the following content:
from setuptools import setup
setup(name="WowPackage",
packages=["."],
)
That's it, now you can "properly install" your package into the Python distribution of your choice without the need to bother about PYTHONPATH. As far as "proper installation" is concerned, you have at least three options:
"Really proper". Will copy your code to your python site-packages directory:
$ python setup.py install
"Development". Will only add a link from the python site-packages to /home/user/mylib. This means that changes to code in your directory will have effect.
$ python setup.py develop
"User". If you do not want to write to the system directories, you can install the package (either "properly" or "in development mode") to /home/user/.local directory, where Python will also find them on its own. For that, just add --user to the command.
$ python setup.py install --user
$ python setup.py develop --user
To remove a package installed in development mode, do
$ python setup.py develop -u
or
$ python setup.py develop -u --user
To remove a package installed "properly", do
$ pip uninstall WowPackage
If your package is more interesting than a single file (e.g. you have subdirectories and such), just list those in the packages parameter of the setup function (you will need to list everything recursively, hence you'll use a helper function for larger libraries). Once you get a hang of it, make sure to read a more detailed manual as well.
In the end, go and contribute your package to PyPI -- it is as simple as calling python setup.py sdist register upload (you'll need a PyPI username, though).
You can create a configuration file mymodule.pth under lib/site-packages (on Windows) or lib/pythonX.Y/site-packages (on Unix and Macintosh), then add one line containing the directory to add to python path.
From docs.python2 and docs.python3:
A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to sys.path. Non-existing items are never added to sys.path, and no check is made that the item refers to a directory rather than a file. No item is added to sys.path more than once. Blank lines and lines beginning with # are skipped. Lines starting with import (followed by space or tab) are executed.
I found that there is no way to modify PYTHONPATH that is only for python2 or only for python3. I had to use a .pth file.
What I had to do was:
make sure directory is created in my home: $HOME/.local/lib/python${MAJOR_VERSION}.${MINOR_VERSION}/site-packages
create a .pth file in that directory
test that your .pth file is work
done
For more info on `.pth. file syntax and how they work please see: python2 docs and python3 docs.
(.pth files in a nutshell: when your python interpreter starts it will look in certain directories and see the .pth file, open those files, parse the files, and add those directories to your sys.path (i.e. the same behavior as PYTHONPATH) and make any python modules located on those directories available for normal importing.)
If you don't want to bother with moving/adding documents in lib/site-packages, try adding two lines of code in the python2.7 script you would like to run (below.)
import sys
sys.path = [p for p in sys.path if p.startswith(r'C:\Python27')]
This way, PYTHONPATH will be updated (ignore all python3.x packages) every time you run your code.
I'd like to start developing an existing Python module. It has a source folder and the setup.py script to build and install it. The build script just copies the source files since they're all python scripts.
Currently, I have put the source folder under version control and whenever I make a change I re-build and re-install. This seems a little slow, and it doesn't settle well with me to "commit" my changes to my python install each time I make a modification. How can I cause my import statement to redirect to my development directory?
Use a virtualenv and use python setup.py develop to link your module to the virtual Python environment. This will make your project's Python packages/modules show up on the sys.path without having to run install.
Example:
% virtualenv ~/virtenv
% . ~/virtenv/bin/activate
(virtenv)% cd ~/myproject
(virtenv)% python setup.py develop
Virtualenv was already mentioned.
And as your files are already under version control you could go one step further and use Pip to install your repo (or a specific branch or tag) into your working environment.
See the docs for Pip's editable option:
-e VCS+REPOS_URL[#REV]#egg=PACKAGE, --editable=VCS+REPOS_URL[#REV]#egg=PACKAGE
Install a package directly from a checkout. Source
will be checked out into src/PACKAGE (lower-case) and
installed in-place (using setup.py develop).
Now you can work on the files that pip automatically checked out for you and when you feel like it, you commit your stuff and push it back to the originating repository.
To get a good, general overview concerning Pip and Virtualenv see this post: http://www.saltycrane.com/blog/2009/05/notes-using-pip-and-virtualenv-django
Install the distrubute package then use the developer mode. Just use python setup.py develop --user and that will place path pointers in your user dir location to your workspace.
Change the PYTHONPATH to your source directory. A good idea is to work with an IDE like ECLIPSE that overrides the default PYTHONPATH.