I'm not sure I'm organizing my package structure correctly or am using the right options in setup.py because I'm getting errors when I try to run unit tests.
I have a structure like this:
/project
/bin
/src
/pkgname
__init__.py
module1.py
module2.py
/tests
__init__.py
test1.py
test2.py
My setup.py looks like this:
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(version='0.1',
description='Trend following library',
author='Nate Reed',
author_email='nate#natereed.com',
packages=find_packages(),
install_requires=['numpy'],
test_suite="tests",
)
When I run 'python setup.py test' I get:
nate#nate-desktop:~/PycharmProjects/trendfollowing$ sudo python setup.py test
running test
running egg_info
writing requirements to UNKNOWN.egg-info/requires.txt
writing UNKNOWN.egg-info/PKG-INFO
writing top-level names to UNKNOWN.egg-info/top_level.txt
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 11, in <module>
test_suite="tests",
File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
cmd_obj.run()
File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 137, in run
self.with_project_on_sys_path(self.run_tests)
File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 117, in with_project_on_sys_path
func()
File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 146, in run_tests
testLoader = loader_class()
File "/usr/lib/python2.6/unittest.py", line 816, in __init__
self.parseArgs(argv)
File "/usr/lib/python2.6/unittest.py", line 843, in parseArgs
self.createTests()
File "/usr/lib/python2.6/unittest.py", line 849, in createTests
self.module)
File "/usr/lib/python2.6/unittest.py", line 613, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python2.6/unittest.py", line 587, in loadTestsFromName
return self.loadTestsFromModule(obj)
File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 34, in loadTestsFromModule
tests.append(self.loadTestsFromName(submodule))
File "/usr/lib/python2.6/unittest.py", line 584, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test1'
Do the test names need to match module names? Are there other conventions I need to follow in my package structure?
Through some trial and error, I found the cause of this problem. Test names should match module names. If there is a "foo_test.py" test, there needs to be a corresponding module foo.py.
I found some guidelines on organizing package structure, which helped me reorganize my package into a structure I was confident in.
Related
So I have a script called papercutter.py and a setup.py that is set up as follows according to the py2exe tutorial:
from distutils.core import setup
import py2exe
setup(console=['papercutter.py'])
So following the tutorial, i run python setup.py py2exe and it looks successful:
c:\Codes>python setup.py install
running install
running build
running install_egg_info
Removing C:\Python\Lib\site-packages\UNKNOWN-0.0.0-py3.6.egg-info
Writing C:\Python\Lib\site-packages\UNKNOWN-0.0.0-py3.6.egg-info
but when I run the next step, python setup.py py2exe, I get error:
c:\Codes>python setup.py py2exe
running py2exe
Traceback (most recent call last):
File "setup.py", line 4, in <module>
setup(console=['papercutter.py'])
File "C:\Python\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "C:\Python\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\Python\lib\site-packages\py2exe\distutils_buildexe.py", line 188, in run
self._run()
File "C:\Python\lib\site-packages\py2exe\distutils_buildexe.py", line 267, in _run
builder.analyze()
File "C:\Python\lib\site-packages\py2exe\runtime.py", line 157, in analyze
self.mf.import_package(modname[:-2])
File "C:\Python\lib\site-packages\py2exe\mf3.py", line 91, in import_package
self.import_hook(name)
File "C:\Python\lib\site-packages\py2exe\mf3.py", line 117, in import_hook
module = self._gcd_import(name)
File "C:\Python\lib\site-packages\py2exe\mf3.py", line 267, in _gcd_import
return self._find_and_load(name)
File "C:\Python\lib\site-packages\py2exe\mf3.py", line 320, in _ find_and_load
self._scan_code(module.__code__, module)
File "C:\Python\lib\site-packages\py2exe\mf3.py", line 352, in _scan_code
for what, args in self._scan_opcodes(code):
File "C:\Python\lib\site-packages\py2exe\mf3.py", line 381, in _scan_opcodes
yield "store", (names[oparg],)
IndexError: tuple index out of range
What am I doing wrong here?
Note: this is the tutorial I am following: http://www.py2exe.org/index.cgi/Tutorial
Also note: I am using py2exe 0.9.2.0 and python 3.6.0. I think these should be fine together.
The answer is at the bottom of the Traceback. Seems like you have an indexing problem with a tuple. Indexes are 0 based in Python.
Currently py2exe supports Python up to 3.4
I'm curious about the following situation. Let's say I have two projects named project_alpha and project_bravo, both defining a top-level namespace package mymeta. The layout:
project_alpha/
-> mymeta/
-> __init__.py
-> project_alpha/
-> __init__.py
-> version.py
-> setup.py
project_bravo/
-> mymeta/
-> __init__.py
-> project_bravo/
-> __init__.py
-> version.py
-> setup.py
Both mymeta/__init__.pys contain only the line __import__('pkg_resources').declare_namespace(__name__) (according to namespace section in setuptools docs). Contents of both version.pys:
__version_info__ = (0, 9, 9, 'dev0')
__version__ = '.'.join((str(entry) for entry in __version_info__))
The setup.py script for project_alpha is pretty simple. The namespace package mymeta is declared, and the version is taken from the version module:
# project_alpha
from setuptools import find_packages, setup
from mymeta.project_alpha.version import __version__
setup(
name='mymeta.project-alpha',
version=__version__,
namespace_packages=['mymeta'],
packages=find_packages(),
)
The setup.py script for project_bravo has the same structure, but with a twist: project_bravo depends on project_alpha when building:
from setuptools import find_packages, setup
from mymeta.project_bravo.version import __version__
setup(
name='mymeta.project-bravo',
version=__version__,
namespace_packages=['mymeta'],
setup_requires=['mymeta.project-alpha'],
packages=find_packages(),
)
When building project_bravo, I get the following error:
~/project_bravo $ python setup.py sdist
Traceback (most recent call last):
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 156, in save_modules
yield saved
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 197, in setup_context
yield
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 246, in run_setup
DirectorySandbox(setup_dir).run(runner)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 276, in run
return func()
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 245, in runner
_execfile(setup_script, ns)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 47, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-ahmxos98/mymeta.project-alpha-0.9.9.dev0/setup.py", line 6, in <module>
try:
ImportError: No module named 'mymeta.project_alpha'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "setup.py", line 22, in <module>
packages=find_packages(),
File "/usr/lib64/python3.5/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/dist.py", line 315, in __init__
self.fetch_build_eggs(attrs['setup_requires'])
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/dist.py", line 361, in fetch_build_eggs
replace_conflicting=True,
File "/tmp/tstenv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 853, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/tmp/tstenv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1125, in best_match
return self.obtain(req, installer)
File "/tmp/tstenv/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1137, in obtain
return installer(requirement)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/dist.py", line 429, in fetch_build_egg
return cmd.easy_install(req)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 665, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 695, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 876, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 1115, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 1101, in run_setup
run_setup(setup_script, args)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 249, in run_setup
raise
File "/usr/lib64/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 197, in setup_context
yield
File "/usr/lib64/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 168, in save_modules
saved_exc.resume()
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 143, in resume
six.reraise(type, exc, self._tb)
File "/tmp/tstenv/lib/python3.5/site-packages/pkg_resources/_vendor/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 156, in save_modules
yield saved
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 197, in setup_context
yield
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 246, in run_setup
DirectorySandbox(setup_dir).run(runner)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 276, in run
return func()
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 245, in runner
_execfile(setup_script, ns)
File "/tmp/tstenv/lib/python3.5/site-packages/setuptools/sandbox.py", line 47, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-ahmxos98/mymeta.project-alpha-0.9.9.dev0/setup.py", line 6, in <module>
try:
ImportError: No module named 'mymeta.project_alpha'
Unfortunately, I don't understand the error here. It has something to do with the imports order, right? If I comment out the import of mymeta.project_bravo.version in project_bravo/setup.py and replace the version with some hard-coded string, suddenly the build succeeds...
Edit: I introduced a workaround for this issue. Instead of trying to import the version directly, I exec the version module to avoid the import problems. Of course, this is not a proper solution, thus not posting this as answer, but still here it is:
__version__ = None # if the exec fails, leave the version unset, the resulting build version will be 0.0.0
version_script_path = os.path.relpath(os.path.join(os.path.dirname(__file__), 'mymeta', 'project_alpha', 'version.py'))
with open(version_script_path) as version_script:
exec(version_script.read())
After the version script is read and executed, the version is initialized, thus no need to import anything from mymeta package.
Almost one year later, I once again faced this issue and the solution for python>=3.3 is to use implicit namespace packages as specified in PEP 420. The project structure is barely changed, just both the __init__.pys for pkgutil-style namespace package mymeta are gone:
project_alpha/
-> mymeta/
-> project_alpha/
-> __init__.py
-> version.py
-> setup.py
project_bravo/
-> mymeta/
-> project_bravo/
-> __init__.py
-> version.py
-> setup.py
To make setuptools happy, both setup scripts need to be adjusted as well:
...
setup(
...
packages=['mymeta.' + pkg for pkg in find_packages('mymeta')],
)
Now the imports will be resolved correctly when building project-bravo.
As mentioned in the comments: The command
$ python setup.py sdist
for mymeta.project_bravo downloads an egg of mymeta.project_alpha from a private pypi repo.
Namespace packages are very delicate concerning their imports.
e.g. I found out that when I install ns.a and ns.b regularly in my environment and supply the import path of ns.d in a .pth file in the site-packages directory, it will not work, no matter what. However when I symlink to ns.d in site-packeges/ns directory it will work.
If I install none of the components and supply all paths in a .pth in the site-packages directory, everything works fine.
If I install all of the components regularly also everything works fine.
Just when I mix concepts it will not work.
Therefore I suspect that this might also be the issue here: Different strategies for the import paths.
You might want to try to modify your __init__.py file to:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
__import__('pkg_resources').declare_namespace(__name__)
I regret my decision to do namespace packages. I would never do it again.
I have a package with a rather extensive test suite, which I maintain with very low frequency of changes. From a time to time I forget to install a component needed for testing or that my changes break the test code. And very often when this happens, setuptools causes a hiding of the real cause of the problem.
Here is an example:
$ python setup.py test
running test
running egg_info
writing requirements to pwman3.egg-info/requires.txt
writing pwman3.egg-info/PKG-INFO
writing top-level names to pwman3.egg-info/top_level.txt
writing dependency_links to pwman3.egg-info/dependency_links.txt
writing entry points to pwman3.egg-info/entry_points.txt
reading manifest file 'pwman3.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pwman3.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 361, in <module>
'console_scripts': ['pwman3 = pwman.ui.cli:main']
File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib64/python2.7/site-packages/setuptools/command/test.py", line 146, in run
self.with_project_on_sys_path(self.run_tests)
File "/usr/lib64/python2.7/site-packages/setuptools/command/test.py", line 127, in with_project_on_sys_path
func()
File "/usr/lib64/python2.7/site-packages/setuptools/command/test.py", line 167, in run_tests
testRunner=self._resolve_as_ep(self.test_runner),
File "/usr/lib64/python2.7/unittest/main.py", line 94, in __init__
self.parseArgs(argv)
File "/usr/lib64/python2.7/unittest/main.py", line 149, in parseArgs
self.createTests()
File "/usr/lib64/python2.7/unittest/main.py", line 158, in createTests
self.module)
File "/usr/lib64/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib64/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_pwman'
Sometimes, I spend a few frustrating minutes trying to understand why I get this error, despite have a python module called test_pwman inside the test directory.
The relevant code inside setup.py is:
from setuptools.command.install import install
setup(name=pwman.appname,
...
test_suite='tests.test_pwman.suite',
...
)
After spending some time staring at the screen, I would remember I can run the test suite like this:
$ python -m tests.test_pwman
Which reveals, for example a real cause for tests not running:
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/ozn/Software/pwman3/tests/test_pwman.py", line 27, in <module>
from .test_postgresql import TestPostGresql
File "tests/test_postgresql.py", line 26, in <module>
import psycopg2 as pg
File "/home/ozn/.virtualenvs/pwman3/lib/python2.7/site-packages/psycopg2/__init__.py", line 50, in <module>
from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: libpq.so.5: cannot open shared object file: No such file or directory
After installing Postgresql with Python support, I could run my tests. So my question is:
How do you cause setuptools to propagate the real exception?
Is my code calling the test suite from setup.py wrong?
See http://bugs.python.org/issue7559, it's a bug in unittest module.
As pointed out in iElectric's answer, it is a know bug that has been fixed in python 3.5. However, I see you are using python 2.7.
The bug fix has been back ported to unittest2 which will solve the problem for you in python 2.7.
To use unittest2, install it (pip install unittest2) simply replace import unittest with import unittest2.
The unittest2 is maintained by Robert Collins who is listed a an official expert of the stdlib unitest module, so you can be assured that you are in good hands.
I'm trying to make an exe file from my (2) py files. In one file is bs4 imported - import bs4
When I try to execute this script:
setup(
console = ['gui.py'],
options = {
'py2exe': {
'packages': ["bs4"]
}
}
)
the console returns:
C:\Users\uživatel\PycharmProjects\mail_checker>setup.py py2exe
running py2exe
*** searching for required modules ***
Traceback (most recent call last):
File "C:\Users\u×ivatel\PycharmProjects\mail_checker\setup.py", line 12, in <m
odule>
'packages': ["bs4"]
File "C:\Python27\lib\distutils\core.py", line 151, in setup
dist.run_commands()
File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Python27\lib\site-packages\py2exe\build_exe.py", line 243, in run
self._run()
File "C:\Python27\lib\site-packages\py2exe\build_exe.py", line 296, in _run
self.find_needed_modules(mf, required_files, required_modules)
File "C:\Python27\lib\site-packages\py2exe\build_exe.py", line 1306, in find_n
eeded_modules
mf.import_hook(f)
File "C:\Python27\lib\site-packages\py2exe\mf.py", line 719, in import_hook
return Base.import_hook(self,name,caller,fromlist,level)
File "C:\Python27\lib\site-packages\py2exe\mf.py", line 136, in import_hook
q, tail = self.find_head_package(parent, name)
File "C:\Python27\lib\site-packages\py2exe\mf.py", line 204, in find_head_pack
age
raise ImportError, "No module named " + qname
ImportError: No module named bs4
So I suppose that I haven't setup.py written correct. Could you give me an advice? Thanks
SOLUTION:
The problem was probably that I have installed bs4 (and xlsxwriter) by easy_install which creates *.egg files in site-packages folder. Py2exe couldn't find bs4 in site-packages for some reason. So I tried to open BeautifulSoup egg file and copy bs4 folder into site-packages folder, I did the same with xlsxwriter.
It helped. Program works properly.
I have been trying to get cx_freeze working on ubuntu, but when i try to run the "python setup.py build" i get the following error:
cx_Freeze.freezer.ConfigError: no initscript named Console
I have searched google and i see a lot of people have ahd this issue, but i can't seem to find a solution.
my setup.py code is as follows:
from cx_Freeze import setup, Executable
setup( name = "hello world" ,
version = "0.1" ,
description = "Hello" ,
executables = [Executable("hello.py")] ,
)
I have placed the setup.py file and hello.py in the same folder.
Any idea as to what could solve this problem?
running build
running build_exe
Traceback (most recent call last):
File "setup.py", line 7, in <module>
executables = [Executable("hello.py")] ,
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze-4.3.1-py2.7-linux-i686.egg
cx_Freeze/dist.py", line 365, in setup
distutils.core.setup(**attrs)
File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze-4.3.1-py2.7-linux-i686.egg
/cx_Freeze/dist.py", line 234, in run
metadata = metadata)
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze-4.3.1-py2.7-linux-i686.egg
/cx_Freeze/freezer.py", line 104, in __init__
self._VerifyConfiguration()
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze-4.3.1-py2.7-linux-i686.egg
/cx_Freeze/freezer.py", line 466, in _VerifyConfiguration
self._GetInitScriptFileName()
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze-4.3.1-py2.7-linux-i686.egg
/cx_Freeze/freezer.py", line 311, in _GetInitScriptFileName
raise ConfigError("no initscript named %s", name)
cx_Freeze.freezer.ConfigError: no initscript named Console
For my installation I took same problem. Looks like symbolic link for initscripts not created, so I make it manually and everything works (change cx_Freeze version to your own):
cd /usr/local/lib/python2.7/dist-packages/
cd cx_Freeze-4.3.1-py2.7-linux-i686.egg
sudo ln -s -t cx_Freeze/ ../../cx_Freeze/initscripts/
PS: Ubuntu 12.10/quantal, Python 2.7.3