I'm trying to run Hadoopy, but am getting a compiling error on OS X:
ImportError: Building module failed: ["CompileError: command 'llvm-gcc-4.2' failed with exit status 1\n"
I have /Developer/usr/bin in my $PATH, and am running latest version of XCode on OS X Lion 10.7. Cython was installed via easy_install.
Full output:
>>> import pyximport; pyximport.install()
>>> import hadoopy
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:236:22: error: getdelim.h: No such file or directory
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:236:22: error: getdelim.h: No such file or directory
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c: In function ‘__pyx_f_7hadoopy_5_main_11HadoopyTask_read_offset_value_text’:
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:4399: warning: implicit conversion shortens 64-bit value into a 32-bit value
lipo: can't open input file: /var/folders/8b/n0j5pn_13qn_x8p2v4f848zh0000gn/T//ccC8x2Ex.out (No such file or directory)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/hadoopy/__init__.py", line 22, in <module>
from _main import run, print_doc_quit
File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 335, in load_module
self.pyxbuild_dir)
File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 183, in load_module
so_path = build_module(name, pyxfilename, pyxbuild_dir)
File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 167, in build_module
reload_support=pyxargs.reload_support)
File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyxbuild.py", line 85, in pyx_to_dll
dist.run_commands()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/Cython/Distutils/build_ext.py", line 135, in run
_build_ext.build_ext.run(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/Cython/Distutils/build_ext.py", line 143, in build_extensions
self.build_extension(ext)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
depends=ext.depends)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 624, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/unixccompiler.py", line 180, in _compile
raise CompileError, msg
ImportError: Building module failed: ["CompileError: command 'llvm-gcc-4.2' failed with exit status 1\n"
For me it was an installation issue and i had fixed it sometime back using below mentioned steps:
pip install argparse
git clone https://github.com/bwhite/hadoopy.git
cd hadoopy
python setup.py install
Instead of using pyximport, build the extension modules in place with python setup.py build_ext --inplace (or install for in-place development with python setup.py develop, or just a regular install via python setup.py install). For packages you almost always want to run the setup, which will properly configure the compilation environment, build, and installation process.
pyximport is good for your personal scripts if you're using Cython to speed up your code (e.g. for scientific computing). Even then you might need to bring in other libraries, and build from multiple sources. In that case, you can use a pyxbld file to set up the sources and include_dirs. For example, say you have foo.pyx, then you can place build instructions in a foo.pyxbld in the same directory. For example:
#foo.pyxbld
def make_ext(modname, pyxfilename):
from distutils.extension import Extension
return Extension(name = modname,
sources=[pyxfilename, 'bar.c'],
include_dirs=['/myinclude'] )
def make_setup_args():
return dict(script_args=["--compiler=mingw32"])
Now using foo is as simple as the following:
import pyximport
pyximport.install()
import foo
Presuming there's no compilation errors, it's virtually transparent but for the delay the first time you import the module. Subsequent imports will look for the built extension in the .pyxbld subdirectory of your home/profile directory.
Related
I have a package with a rather extensive test suite, which I maintain with very low frequency of changes. From a time to time I forget to install a component needed for testing or that my changes break the test code. And very often when this happens, setuptools causes a hiding of the real cause of the problem.
Here is an example:
$ python setup.py test
running test
running egg_info
writing requirements to pwman3.egg-info/requires.txt
writing pwman3.egg-info/PKG-INFO
writing top-level names to pwman3.egg-info/top_level.txt
writing dependency_links to pwman3.egg-info/dependency_links.txt
writing entry points to pwman3.egg-info/entry_points.txt
reading manifest file 'pwman3.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pwman3.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 361, in <module>
'console_scripts': ['pwman3 = pwman.ui.cli:main']
File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib64/python2.7/site-packages/setuptools/command/test.py", line 146, in run
self.with_project_on_sys_path(self.run_tests)
File "/usr/lib64/python2.7/site-packages/setuptools/command/test.py", line 127, in with_project_on_sys_path
func()
File "/usr/lib64/python2.7/site-packages/setuptools/command/test.py", line 167, in run_tests
testRunner=self._resolve_as_ep(self.test_runner),
File "/usr/lib64/python2.7/unittest/main.py", line 94, in __init__
self.parseArgs(argv)
File "/usr/lib64/python2.7/unittest/main.py", line 149, in parseArgs
self.createTests()
File "/usr/lib64/python2.7/unittest/main.py", line 158, in createTests
self.module)
File "/usr/lib64/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib64/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_pwman'
Sometimes, I spend a few frustrating minutes trying to understand why I get this error, despite have a python module called test_pwman inside the test directory.
The relevant code inside setup.py is:
from setuptools.command.install import install
setup(name=pwman.appname,
...
test_suite='tests.test_pwman.suite',
...
)
After spending some time staring at the screen, I would remember I can run the test suite like this:
$ python -m tests.test_pwman
Which reveals, for example a real cause for tests not running:
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/ozn/Software/pwman3/tests/test_pwman.py", line 27, in <module>
from .test_postgresql import TestPostGresql
File "tests/test_postgresql.py", line 26, in <module>
import psycopg2 as pg
File "/home/ozn/.virtualenvs/pwman3/lib/python2.7/site-packages/psycopg2/__init__.py", line 50, in <module>
from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: libpq.so.5: cannot open shared object file: No such file or directory
After installing Postgresql with Python support, I could run my tests. So my question is:
How do you cause setuptools to propagate the real exception?
Is my code calling the test suite from setup.py wrong?
See http://bugs.python.org/issue7559, it's a bug in unittest module.
As pointed out in iElectric's answer, it is a know bug that has been fixed in python 3.5. However, I see you are using python 2.7.
The bug fix has been back ported to unittest2 which will solve the problem for you in python 2.7.
To use unittest2, install it (pip install unittest2) simply replace import unittest with import unittest2.
The unittest2 is maintained by Robert Collins who is listed a an official expert of the stdlib unitest module, so you can be assured that you are in good hands.
I'm trying to package an auto updating python 3 application using esky, but cannot get it to work on windows.
I'm trying to get the simplest example from the tutorial to work.
https://github.com/cloudmatrix/esky/tree/master/tutorial/stage1
My environment:
Z:\share_space\esky-master\tutorial\stage1>python --version
Python 3.4.3
Z:\share_space\esky-master\tutorial\stage1>pip freeze
...
cx-Freeze==4.3.4
esky==0.9.8
py2exe==0.9.2.2
...
When run using py2exe as the freezer, it fails to find the py2exe module for some reason (though freezing other apps with "python setup.py py2exe" works just fine).
Z:\share_space\esky-master\tutorial\stage1>python setup.py bdist_esky
running bdist_esky
Traceback (most recent call last):
File "setup.py", line 17, in <module>
"freezer_module":"py2exe",
File "C:\Python34\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python34\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "C:\Python34\lib\distutils\dist.py", line 973, in run_command
cmd_obj.ensure_finalized()
File "C:\Python34\lib\distutils\cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "C:\Python34\lib\site-packages\esky\bdist_esky\__init__.py", line 291, in
finalize_options
raise RuntimeError(err)
RuntimeError: freezer module not found: 'py2exe'
With cx-Freeze building the package works well but running the executable fails.
Z:\share_space\esky-master\tutorial\stage1>dist\example-app-0.1.win32\example.ex
e
ValueError: bad marshal data (unknown type code)
Fatal Python error: unable to locate initialization module
Current thread 0x00001254 (most recent call first):
In order got get a working windows build you need the following:
The latest esky version from github (not pypi).
To circumvent an esky bug ensure the character case of dependencies listed with pip freeze are correct and reinstall if not.
I can confirm using cx-Freeze works (py2exe not tested yet).
See this github issue as to how the problem was resolved.
I am attempting to use cx_Freeze to turn a .py file that I wrote in Python 3.3.4. In running the build command, I get the following error.
C:\Python33>c:\python33\python.exe setup.py build
running build
running build_exe
Traceback (most recent call last):
File "setup.py", line 22, in <module>
executables=executables
File "c:\python33\lib\site-packages\cx_Freeze\dist.py", line 365, in setup
distutils.core.setup(**attrs)
File "c:\python33\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "c:\python33\lib\distutils\dist.py", line 930, in run_commands
self.run_command(cmd)
File "c:\python33\lib\distutils\dist.py", line 949, in run_command
cmd_obj.run()
File "c:\python33\lib\distutils\command\build.py", line 126, in run
self.run_command(cmd_name)
File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "c:\python33\lib\distutils\dist.py", line 949, in run_command
cmd_obj.run()
File "c:\python33\lib\site-packages\cx_Freeze\dist.py", line 234, in run
metadata = metadata)
File "c:\python33\lib\site-packages\cx_Freeze\freezer.py", line 101, in __init
__
for n in self._GetDefaultBinPathExcludes() + binPathExcludes]
File "c:\python33\lib\site-packages\cx_Freeze\freezer.py", line 235, in _GetDe
faultBinPathExcludes
import cx_Freeze.util
ImportError: DLL load failed: The specified module could not be found.
Have I not installed cx_freeze correctly? Or am I running it all wrong?
Help is greatly appreciated.
Thanks
Well, I found an answer that works for me.
Turns out there's a bug in cx_Freeze 4.3.4 and downgrading to 4.3.3 solved this problem for me.
Try opening up the util.pyd file with a tool like Dependency Walker in order to check if the error is a result of missing dependencies. I found that it was trying to load msvcr100.dll from the Microsoft Visual C++ 2010 runtime which I did not have installed on my machine. If this is also the source of your error, you can try running the x86 / x64 installer to see if this addresses it (you want to pick the one that matches the architecture of your Python installation not of your OS).
Failed to install sqlite wrapper. Any help with installation will be really helpful.
Downloaded from
Installed sqlite3.version
'2.6.0'
Please find the error log below-
sudo python setup.py install test
running install
running build
running build_ext
SQLite: Using system sqlite include/libraries
running install_lib
running install_egg_info
Removing /Library/Python/2.7/site-packages/apsw-3.7.16.2_r1-py2.7.egg-info
Writing /Library/Python/2.7/site-packages/apsw-3.7.16.2_r1-py2.7.egg-info
running test
Traceback (most recent call last):
File "setup.py", line 857, in <module>
'win64hackvars': win64hackvars}
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "setup.py", line 102, in run
import tests
File "/Users/tj/Downloads/apsw-3.7.16.2-r1/tests.py", line 9, in <module>
import apsw
ImportError: dlopen(/Users/tj/Downloads/apsw-3.7.16.2-r1/apsw.so, 2): Symbol not found: _sqlite3_db_filename
Referenced from: /Users/tj/Downloads/apsw-3.7.16.2-r1/apsw.so
Expected in: flat namespace
in /Users/tj/Downloads/apsw-3.7.16.2-r1/apsw.so
It is picking up the system SQLite shared library which is an older version. MacOS tends to ignore the SQLite library you want it to use and force using a known system one. (Many OS components like CoreData depend on SQLite which is why they like to force using a known good version.)
The easiest fix is to use the command line as specified in the documentation. It ensures a private copy of SQLite is embedded in the extension and hence will always work.
http://apidoc.apsw.googlecode.com/hg/build.html#recommended
python setup.py fetch --all build --enable-all-extensions install test
(I'm the APSW author). BTW there is a python-sqlite mailing list which you may find useful.
I have installed psyco in one machine with no problem, but i'm getting an strange error while installing in one other machine.
I'm not able to use easy_install, since it gives me an error:
C:\Python26\Downloads\psyco-1.6>easy_install psyco
Searching for psyco
Reading http://pypi.python.org/simple/psyco/
Reading http://psyco.sourceforge.net/
Best match: psyco snapshot
Downloading http://wyvern.cs.uni-duesseldorf.de/psyco/psyco-snapshot.tar.gz
error: Can't download http://wyvern.cs.uni-duesseldorf.de/psyco/psyco-snapshot.t
ar.gz: 404 Not Found
So, I download the last version 1.6 and did "python setup.py install". I have already used it several times with no problem. I get the follwing messages:
C:\Python26\Downloads\psyco-1.6>python setup.py install
PROCESSOR = 'i386'
running install
running build
running build_py
running build_ext
building 'psyco._psyco' extension
Traceback (most recent call last):
File "setup.py", line 180, in <module>
**kwds )
File "C:\python26\lib\distutils\core.py", line 152, in setup
dist.run_commands()
File "C:\python26\lib\distutils\dist.py", line 975, in run_commands
self.run_command(cmd)
File "C:\python26\lib\distutils\dist.py", line 995, in run_command
cmd_obj.run()
File "C:\python26\lib\distutils\command\install.py", line 577, in run
self.run_command('build')
File "C:\python26\lib\distutils\cmd.py", line 333, in run_command
self.distribution.run_command(command)
File "C:\python26\lib\distutils\dist.py", line 995, in run_command
cmd_obj.run()
File "C:\python26\lib\distutils\command\build.py", line 134, in run
self.run_command(cmd_name)
File "C:\python26\lib\distutils\cmd.py", line 333, in run_command
self.distribution.run_command(command)
File "C:\python26\lib\distutils\dist.py", line 995, in run_command
cmd_obj.run()
File "C:\python26\lib\distutils\command\build_ext.py", line 340, in run
self.build_extensions()
File "C:\python26\lib\distutils\command\build_ext.py", line 449, in build_exte
nsions
self.build_extension(ext)
File "C:\python26\lib\distutils\command\build_ext.py", line 499, in build_exte
nsion
depends=ext.depends)
File "C:\python26\lib\distutils\msvc9compiler.py", line 449, in compile
self.initialize()
File "C:\python26\lib\distutils\msvc9compiler.py", line 359, in initialize
vc_env = query_vcvarsall(VERSION, plat_spec)
File "C:\python26\lib\distutils\msvc9compiler.py", line 275, in query_vcvarsal
l
raise ValueError(str(list(result.keys())))
ValueError: [u'path']
Any ideas on why I'm getting this error?
Thanks
It appears that psyco has extension modules that need to be built. The error message you're getting is not what I would expect, but it seems to indicate that it's not finding the Microsoft Visual C++ compiler, required (at least recommended over other compilers) for building extension modules for Python 2.6.
If you really want to build from sources, I suggest you download the Microsoft Visual C++ 2008 Express Edition or Microsoft Visual Studio 2008 trial.
You would probably be better off reading this thread, and downloading a pre-compiled binary from someone in the community who has already compiled it.
Edit: I just noticed that on the Psyco home page there is a link to Python 2.6 pre-compiled binaries by Michael Foord. These would be preferable.
I am trying to get around this problem too (while trying to install another module). The problem is the script msvc9compiler.py is trying to find and vcvarsall.bat which is under a folder like C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC which in turn is trying to find and execute a script under C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\Tools in my case vcvars32.bat. This script is setting-updating certain variables: PATH, LIB and 2 others that are never set/updated in your case.
My workaround was to find the *.bat manually and run it in the prompt from which I would then run the installation command. The variables are set correctly this way but only during the session of the command prompt.
Try running echo %LIB%, before and after running the script, to see that the variable is indeed set.