I have a small python application that I would like to make into a downloadable / installable executable for UNIX-like systems. I am under the impression that setuptools would be the best way to make this happen but somehow this doesn't seem to be a common task.
My directory structure looks like this:
myappname/
|-- setup.py
|-- myappname/
| |-- __init__.py
| |-- myappname.py
| |-- src/
| |-- __init__.py
| |-- mainclassfile.py
| |-- morepython/
| |-- __init__.py
| |-- extrapython1.py
| |-- extrapython2.py
The file which contains if __name__ == "__main__": is myappname.py. This file has a line at the top, import src.mainclassfile.
When this is downloaded, I would like for a user to be able to do something like:
$ python setup.py build
$ python setup.py install
And then it will be an installed executable which they can invoke from anywhere on the command line with:
$ myappname arg1 arg2
The important parts of my setup.py are like:
from setuptools import setup, find_packages
setup(
name='code2flow',
scripts=['myappname/myappname.py'],
package_dir={'myappname': 'myappname'},
packages=find_packages(),
)
Current state
By running:
$ sudo python setup.py install
And then in a new shell:
$ myapp.py
I am getting a No module named error
The problem here is that your package layout is broken.
It happens to work in-place, at least in 2.x. Why? You're not accessing the package as myappname—but the same directory that is that package's directory is also the top-level script directory, so you end up getting any of its siblings via old-style relative import.
Once you install things, of course, you'll end up with the myappname package installed in your site-packages, and then a copy of myappname.py installed somewhere on your PATH, so relative import can't possibly work.
The right way to do this is to put your top-level scripts outside the package (or, ideally, into a bin directory).
Also, your module and your script shouldn't have the same name. (There are ways you can make that work, but… just don't try it.)
So, for example:
myappname/
|-- setup.py
|-- myscriptname.py
|-- myappname/
| |-- __init__.py
| |-- src/
| |-- __init__.py
| |-- mainclassfile.py
Of course so far, all this makes it do is break in in-place mode the exact same way it breaks when installed. But at least that makes things easier to debug, right?
Anyway, your myscriptname.py then has to use an absolute import:
import myappname.src.mainclassfile
And your setup.py has to find the script in the right place:
scripts=['myscriptname.py'],
Finally, if you need some code from myscriptname.py to be accessible inside the module, as well as in the script, the right thing to do is to refactor it into two files—but if that's too difficult for some reason, you can always write a wrapper script.
See Arranging your file and directory structure and related sections in the Hitchhiker's Guide to Packaging for more details.
Also see PEP 328 for details on absolute vs. relative imports (but keep in mind that when it refers to "up to Python 2.5" it really means "up to 2.7", and "starting in 2.6" means "starting in 3.0".
For a few examples of packages that include scripts that get installed this way via setup.py (and, usually, easy_install and pip), see ipython, bpython, modulegraph, py2app, and of course easy_install and pip themselves.
Related
I use pants to manage a Python project that uses protocol buffers. Pants places the generated _pb2.py and _pb2.pyi files under a separate dist/codegen tree. Is it possible to get VS Code autocomplete to work when using the _pb2 modules?
The file tree looks like this:
.
|-- dist/
| `-- codegen/
| `-- src/
| `-- project/
| |-- data_pb2.py
| `-- data_pb2.pyi
`-- src/
`-- project/
|-- __init__.py
|-- code.py
`-- data.proto
And in code.py I have import statements like this:
from project import data_pb2
I've tried setting python.analysis.extraPaths to ["dist/codegen/src"] in settings.json. This makes pylance stop complaining that data_pb2 is missing. But autocomplete still does not work, and pylance has no type information for members of data_pb2.
Replace your python.analysis.extraPaths with the following extent:
"python.analysis.extraPaths": [
"./dist/codegen/src"
],
And adding the following code to your code.py:
import sys
sys.path.append(".\dist\codegen\src")
You can use Python implicit namespace packages (PEP 420) to make this work. Namespace packages are allowed to have modules within the same package reside in different directories. Which allows pylance and other tools to work correctly when code is split between src and dist/codegen/src.
To use implicit namespace packages, you just need to remove src/package/__init__.py, and leave "python.analysis.extraPaths" set to ["dist/codegen/src"].
See also the GitHub issue microsoft/pylance-release#2855, which describes using implicit namespace packages to make pylance work correctly in a similar situation.
I'm using PyCharm CE to develop a project with a structure similar to this:
test_python/
|-- app
| |-- __init__.py
| |-- mymodule.py
| |-- mymodule.pyc
| `-- test_mymodule.py
|-- config.py
`-- tests
|-- __init__.py
|-- test_config.py
`-- test_models.py
When I try to run my test scripts such as test_config.py, I get:
$ python tests/test_config.py
Traceback (most recent call last):
File "tests/test_config.py", line 1, in <module>
from config import app_config
ImportError: No module named config
I have read a lot of other SO posts that talk about needing a init.py file in all directories that are packages (which I have done already). Many also suggest messing around with sys.path. My problem with this latter approach is that I never had to meddle with the paths previously. I'm not sure if it's something that changed with my dev environment setup, but here it is:
Python 2.7 | macOS Sierra | PyCharm CE
I have tried to install a virtual environment with virtualenv but didn't see a difference. Here is the sample project on github if you'd like to run it yourself
It seems that there is a problem with folder depth. Replace from ../config import app_config.
I think it will work if you change the working directory on your run configuration to test_python. All the packages in your import statements are relative to some entry in your Python path, and the working directory is usually in the Python path.
I have a medium sized python command line program that runns well from my source code, and I've created a source distribution file and installed it into the virtual environment using "python setup.py install"
Since this is a pure Python program, and provided that the end users have installed Python, and the required packages, my idea is that i can distribute it through PyPi for all available platforms as a source distribution.
Upon install, I get an 'appname' directory within the virtualenv site-packages directory, and it also runs correctly when I write "python 'pathtovirtualenv'/Lib/sitepackages/'myappname'
But is this the way the end user is supposed to run distutils-distributed programs from the command line.
I fnd a lot of information on how to distribute a program using distutils, but not on how the end user is supposed to launch it after installing it.
Since you already created a setup.py, I would recommend looking at the entry_points:
entry_points={
'console_scripts': [
'scriptname=yourpackage.module:function',
],
},
Here, you have a package named yourpackage, and a module named module in it, and you refer to the function function. This function will wrapped by the script called scriptname, which will be installed in the users bin folder, which is normally in the $PATH, so the user can simply type scriptname after he installed your package via pip install.
To sum up: a user will install the package via pip install yourpackage and finally be able to call the function in module via script name.
Here are some docs on this topic:
https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation
http://www.scotttorborg.com/python-packaging/command-line-scripts.html
Well, I eventually figured it out.
Initially, I wanted to just use distutils, I like it when the end user can install it with minimum of extra dependencies. But I have now discovered that setuptools is the better option in my case.
My directory structure looks like this (Subversion):
trunk
|-- appname
| |-- __init__.py # an empty file
| |-- __main__.py # calls appname.main()
| |-- appname.py # contains a main() and imports moduleN
| |-- module1.py
| |-- module2.py
| |-- ...
|-- docs
| |-- README
| |-- LICENSE
| |-- ...
|-- setup.py
And my setyp.py basically looks like this:
# This setup file is to be used with setuptools source distribution
# Run "python setup sdist to deploy
from setuptools import setup, find_packages
setup( name = "appname",
...
include_package_data = True,
packages = find_packages(),
zip_safe = True,
entry_points = {
'console_scripts' : 'appname=appname.appname:main'
}
)
The next step now is to figure out how to install the contents of the docs directory on the users computer.
But right now, I'm thinking about adding --readme, --license, --changes, --sample (and so forth) options to the main script, to display them at run time.
The source for the package is here
I'm installing the package from the index via:
easy_install hackertray
pip install hackertray
easy_install installs images/hacker-tray.png to the following folder:
/usr/local/lib/python2.7/dist-packages/hackertray-1.8-py2.7.egg/images/
While, pip installs it to:
/usr/local/images/
My setup.py is as follows:
from setuptools import setup
setup(name='hackertray',
version='1.8',
description='Hacker News app that sits in your System Tray',
packages=['hackertray'],
data_files=[('images', ['images/hacker-tray.png'])])
My MANIFEST file is:
include images/hacker-tray.png
Don't use data_files with relative paths. Actually, don't use data_files at all, unless you make sure the target paths are absolute ones properly generated in a cross-platform way insted of hard coded values.
Use package_data instead:
setup(
# (...)
package_data={
"hackertray.data": [
"hacker-tray.png",
],
},
)
where hackertray.data is a proper python package (i.e. is a directory that contains a file named __init__.py) and hacker-tray.png is right next to __init__.py.
Here's how it should look:
.
|-- hackertray
| |-- __init__.py
| `-- data
| |-- __init__.py
| `-- hacker-tray.png
`-- setup.py
You can get the full path to the image file using:
from pkg_resources import resource_filename
print os.path.abspath(resource_filename('hackertray.data', 'hacker-tray.png'))
I hope that helps.
PS: Python<2.7 seems to have a bug regarding packaging of the files listed in package_data. Always make sure to have a manifest file if you're using something older than Python 2.7 for packaging. See here for more info: https://groups.google.com/d/msg/python-virtualenv/v5KJ78LP9Mo/OiBqMcYVFYAJ
I am developing python in eclipse. As a result, python src files and test files are in different directories.
Question is: How do we run on command line specific test files in the test folder? These obviously depend on files in the src folder.
Cheers
Edit: if I run
python test/myTestFile.py
I get dependency errors, eg. ImportError: No module named SrcFile1
You need to make sure your PYTHONPATH is set correctly so the command-line interpreter can find your packages, or run your test cases from within Eclipse Pydev. Update: Another option: running your tests using nose might make things a bit easier, since it can auto-discover packages and test cases.
If your project is laid out like so:
/home/user/dev/
src/pkg1/
mod1.py
test/
mod1_test.py
Use: PYTHONPATH=$HOME/dev/src python test/mod1_test.py. I'd also recommend using distribute and virtualenv to set up your project for development.
Updated in response to question in comments:
This shows how the PYTHONPATH environment variable extends Python's package sear
ch path:
% PYTHONPATH=foo:bar python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
# exporting the variable makes it sticky for your current session. you can
# add this to your shell's resource file (e.g. ~/.profile) or source
# it from a textfile to save typing:
% export PYTHONPATH=bar:baz
% python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
% python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
The above should get you going in the short term. Using distribute and
virtualenv have a higher one-time setup cost but you get longer-term benefits
from using them. When you get a chance, read some of the many tutorials on SO for setting these up to see if they're a good fit for your project.
There are 2 principal solutions to this. Either, you need to use e.g. PYTHONPATH environment variable to tell the tests where the source is, or you need to make tests and production code part of the same module tree by inserting the relevant __init__.py files. In the latter approach, the tree may look something like this:
|-- qbit
| |-- __init__.py
| |-- master.py
| |-- policy.py
| |-- pool.py
| |-- synchronize.py
| `-- worker.py
`-- test
|-- __init__.py
|-- support.py
|-- test_policy.py
|-- test_synchronize.py
`-- test_worker.py
__init__.py can be an empty file.