We Keep Coding

How to correctly import latest libarchive for use on Mac OS

The release notes for libarchive state that because of an older version of libarchive being included within MacOS they recommend changing LD_LIBRARY_PATH to point towards the location of the recent copy of libarchive.
I've used this code to try and achieve that but I get an error message when I run the script.
import os
print os.environ.get('LD_LIBRARY_PATH') #Check what the current path is
os.environ['LD_LIBRARY_PATH'] = '/Library/Python/2.7/site-packages/'
print os.environ.get('LD_LIBRARY_PATH') #Check the variable has been set
import libarchive.public
Error:
None
/Library/Python/2.7/site-packages/
Traceback (most recent call last):
File "scratch.py", line 8, in <module>
import libarchive.public
File "/Library/Python/2.7/site-packages/libarchive/public.py", line 1, in <module>
from libarchive.adapters.archive_read import \
File "/Library/Python/2.7/site-packages/libarchive/adapters/archive_read.py", line 7, in <module>
import libarchive.calls.archive_read
File "/Library/Python/2.7/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 378, in __getattr__
func = self.__getitem__(name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 383, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fb08b741000, archive_read_support_filter_all): symbol not found
I cant find a great answer to this anywhere out there.

It is not clear from the tool documentation, but based on a thread: https://github.com/dsoprea/PyEasyArchive/issues/16 I set another environment variable to the place where the underlying c library could be found. In my case, it was put there by homebrew on my mac.
os.environ['LA_LIBRARY_FILEPATH']='/usr/local/opt/libarchive/lib/libarchive.dylib'
import libarchive.public
worked for me.

How to load a python module with arguments in python?

I would like to create a python module that would be called with python -m mymodule somefile.py some_arg some_arg.
The idea is that I would be able to set up an alias alias="python -m mymodule" and call files normally with python somefile.py some_arg some_arg.
In the file mymodule/__main__.py, what is the best way to load somefile.py and pass it the argument list?
I am looking for a generic solution, that would be python2 and 3 compatible.
It would be great to be as little intrusive as possible. If somefile.py would raise an exception, mymodule should barely be seen in the traceback.
What the module does is not interesting here in detail, but it sets up some python things (traceback hooks etc.), so somefile.py should be ran pythonicly in the same process. os.system or subprocess.Popen do not fit.

Ok I found something good for python 3.5, and satisfying enough for python 2.7.
mymodule/main.py
import sys
# The following block of code removes the part of
# the traceback related to this very module, and runpy
# Negative limit support came with python 3.5, so it will not work
# with previous versions.
# https://docs.python.org/3.5/library/traceback.html#traceback.print_tb
def myexcepthook(type, value, tb):
nb_noise_lines = 3
traceback_size = len(traceback.extract_tb(tb))
traceback.print_tb(tb, nb_noise_lines - traceback_size)
if sys.version_info >= (3, 5):
sys.excepthook = myexcepthook
if len(sys.argv) > 1:
file = sys.argv[1]
sys.argv = sys.argv[1:]
with open(file) as f:
code = compile(f.read(), file, 'exec')
exec(code)
somefile.py
import sys
print sys.argv
raise Exception()
in the terminal
$ python3 -m mymodule somefile.py some_arg some_arg
['somefile.py', 'some_arg', 'some_arg']
Traceback (most recent call last):
File "somefile.py", line 3, in <module>
raise Exception()
$ python2 -m mymodule somefile.py some_arg some_arg
['somefile.py', 'some_arg', 'some_arg']
Traceback (most recent call last):
File "/usr/lib64/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/azmeuk/dev/testpy/mymodule/__main__.py", line 16, in <module>
exec(code)
File "somefile.py", line 3, in <module>
raise Exception()
$ python somefile.py some_arg some_arg
['somefile.py', 'some_arg', 'some_arg']
Traceback (most recent call last):
File "somefile.py", line 3, in <module>
raise Exception()
Exception
Still, if someone has a better proposition, it would be great!

I think the negative value of limit does not work in traceback module before python 3.5. Here is an ugly hack that works with python 2.7
import sys
import traceback
class ExcFile(object):
def __init__(self, file):
self.topline = True
self.file = file
def write(self, s):
if self.topline:
u, s = s.split('\n', 1)
self.file.write(u +'\n')
self.topline = False
if '#---\n' in s:
u, s = s.split('#---\n', 1)
self.file.write(s)
self.write = self.file.write
ExcFile._instance = ExcFile(sys.stdout)
# The following block of code removes the part of
# the traceback related to this very module, and runpy
def myexcepthook(type, value, tb):
traceback.print_exception(type, value, tb, file=ExcFile._instance)
sys.excepthook = myexcepthook
if len(sys.argv) > 1:
file = sys.argv[1]
sys.argv = sys.argv[1:]
with open(file) as f:
code = compile(f.read(), file, 'exec')
exec(code) #---
All this should be written in a separate file, to avoid clutter __main__.py.

comtypes internal method fails on LP_BSTR

I'm trying to access the .olb files that ship with ArcGIS10.1 using the comtypes module. Some of the .olb files work (esriGeometry.olb) and some of them don't (esriSystem.olb), and some of them work some of the time (esriSearch.olb).
The following code
from comtypes.client import GetModule
olb_path = 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\com\\esriSystem.olb'
m = GetModule(path)
raises this traceback and exception
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
test3()
File "D:\Data\MatthewPlourde\JAMES\gis_tools\tool.py", line 139, in test3
m = GetModule(path)
File "C:\Python27\ArcGIS10.1\lib\site-packages\comtypes\client\_generate.py", line 112, in GetModule
mod = _CreateWrapper(tlib, pathname)
File "C:\Python27\ArcGIS10.1\lib\site-packages\comtypes\client\_generate.py", line 188, in _CreateWrapper
mod = _my_import(fullname)
File "C:\Python27\ArcGIS10.1\lib\site-packages\comtypes\client\_generate.py", line 26, in _my_import
return __import__(fullname, globals(), locals(), ['DUMMY'])
File "C:\Python27\ArcGIS10.1\lib\site-packages\comtypes\gen\_5E1F7BC3_67C5_4AEE_8EC6_C4B73AAC42ED_0_10_1.py", line 5705, in <module>
( ['in'], POINTER(_midlSAFEARRAY(POINTER(BSTR))), 'pParameters' ),
File "C:\Python27\ArcGIS10.1\lib\site-packages\comtypes\safearray.py", line 18, in _midlSAFEARRAY
sa_type = _make_safearray_type(itemtype)
File "C:\Python27\ArcGIS10.1\lib\site-packages\comtypes\safearray.py", line 53, in _make_safearray_type
raise TypeError(itemtype)
TypeError: <class 'comtypes.errorinfo.LP_BSTR'>
Apparently comtypes.safearray._make_safearray_type doesn't know what to do with <class 'comtypes.errorinfo.LP_BSTR'>. If there's anyone out there using ArcGIS10.1, I'd be grateful to know whether you can reproduce this error, and especially grateful if you know the cause.

I found a solution posted on the ArcGIS forums. It simply involves modifying automation.py in the comtypes source. Add the entry POINTER(BSTR): VT_BYREF|VT_BSTR to the _ctype_to_vartype dictionary.
After this, all the .olb's load.

I too, am having this exact error and I can't get past it. If you find out what happens please update this. The only thing I can find is something saying that there may be mixing 32 bit with 64 bit libs.
(also, I don't see where I can send a reply to your question... only an answer. I don't use stackexchange much)
Let me add something that may help you, this was taken from: http://forums.arcgis.com/threads/15848-ArcMap-10-ArcObjects-and-Python-very-cool-but-help-with-a-couple-of-problems
import logging
# grab rootlogger
_loggy = logging.getLogger()
_loggy.setLevel(logging.DEBUG)
_loggy.addHandler(logging.FileHandler("derpdebug.log"))
import os
import comtypes.client
# change com_dir to whatever it is for you
com_dir = r'C:\Program Files (x86)\ArcGIS\Desktop10.0\com'
coms = [os.path.join(com_dir, x) for x in os.listdir(com_dir) if os.path.splitext(x)[1].upper() == '.OLB']
map(comtypes.client.GetModule, coms)
# check add whatever you want here.
import comtypes.gen.esriArcMapUI
import comtypes.gen.esriGeodatabase
print dir(comtypes.gen.esriArcMapUI)
Then I just did:
grep -v ^Release derpdebug.log >readable.log

Does anyone know how to configure the hunpos wrapper class on nltk?

i've tried the following code and installed
from http://code.google.com/p/hunpos/downloads/list
english-wsj-1.0
hunpos-1.0-linux.tgz
i've extracted the file onto '~/' directory
and when i tried the following python code:
import nltk
from nltk.tag import hunpos
from nltk.tag.hunpos import HunposTagger
import os, sys, re, glob
cwd = os.getcwd()
for infile in glob.glob(os.path.join(cwd, '*.txt')):
(PATH, FILENAME) = os.path.split(infile)
read = open(infile)
ht = HunposTagger('english.model')
ht.tag(read.readline())
i get the following error
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/usr/local/lib/python2.6/dist-packages/nltk-2.0b9-py2.6.egg/nltk/tag/hunpos.py", line 46, in __init__
verbose=verbose)
File "/usr/local/lib/python2.6/dist-packages/nltk-2.0b9-py2.6.egg/nltk/internals.py", line 503, in find_binary
raise LookupError('\n\n%s\n%s\n%s' % (div, msg, div))
LookupError:
===========================================================================
NLTK was unable to find the hunpos-tag executable! Use
config_hunpos-tag() or set the HUNPOS environment variable.
>>> config_hunpos-tag('/path/to/hunpos-tag')
Searched in:
- .
- /usr/bin
- /usr/local/bin
- /opt/local/bin
- /Applications/bin
- /home/ubi/bin
- /home/ubi/Applications/bin
For more information, on hunpos-tag, see:
<http://code.google.com/p/hunpos/>
===========================================================================
>>> config_hunpos-tag('~/')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'config_hunpos' is not defined
how do i configure hunpos in python? which python command do i need to enter?

You're very close to the solution. Move the hunpos-tag executable to /home/ubi/bin and it should be able to find it then. This caused me some trouble too when I was first trying to use hunpos.

Jython and the xml.sax parser - strange error

I am just getting started with Python/Jython and the SAX parser (xml.sax). I wrote a simple content handler as a test.
from __future__ import with_statement
from xml.sax import make_parser, handler
from xml.sax.handler import ContentHandler
class CountingHandler(ContentHandler):
def __init__(self):
self.counter = 0
def startElement(self, name, attrs):
self.counter += 1
def main(argv=sys.argv):
parser = make_parser()
h = CountingHandler()
parser.setContentHandler(h)
with open(argv[1], "r") as input:
parser.parse(input)
When I run this on some documents (not all), I get an error:
Traceback (most recent call last):
File "src/sciencenetworks/xmltools.py", line 93, in <module>
sys.exit(main())
File "src/sciencenetworks/xmltools.py", line 88, in main
parser.parse(input)
File "/amd.home/home/staudt/workspace/jython/Lib/xml/sax/drivers2/drv_javasax.py", line 141, in parse
self._parser.parse(JyInputSourceWrapper(source))
File "/amd.home/home/staudt/workspace/jython/Lib/xml/sax/drivers2/drv_javasax.py", line 90, in resolveEntity
return JyInputSourceWrapper(self._resolver.resolveEntity(pubId, sysId))
File "/amd.home/home/staudt/workspace/jython/Lib/xml/sax/drivers2/drv_javasax.py", line 75, in __init__
if source.getByteStream():
AttributeError: 'unicode' object has no attribute 'getByteStream'
When I look into the source code of drv_javasax.py, it seems like input is not recognized as a file like object, which it is.
Any ideas on how to fix this?

I think it's this bug: http://bugs.jython.com/issue1488. Fixed in Jython 2.5.2-b1: http://www.jython.org/latest.html

When you insert print type(input) after your with statement, what do you see?
When you revert to old-style "try/finally" code instead of "with", does it work for all files?
What is different between files that work and files that don't work?
What happens if you change the name input to something that doesn't shadow a built-in function?