We Keep Coding

How to debug vage error lxml.etree.SerialisationError: unknown error -2029930774 in python

I am using some legacy code from python2 that has to work with python3. So far so good, most of the things work as they should. However I get the most vage error from a library called lxml.
In my understanding this is a library that binds to a binary program written in c.
The problem comes from this piece of code:
with etree.xmlfile(self.temp_file, encoding='utf-8') as xf:
with xf.element('{http://www.opengis.net/citygml/2.0}CityModel', nsmap=nsmap):
with open(input_gml, mode='rb') as f:
context = etree.iterparse(f)
for action, elem in context:
if action == 'end' and elem.tag == '{http://www.opengis.net/citygml/2.0}cityObjectMember':
# Duplicate feature and subfeatures
self.duplicateFeature(xf, elem)
# Clean up the original element and the node of its previous sibling
# (https://www.ibm.com/developerworks/xml/library/x-hiperfparse/)
elem.clear()
while elem.getprevious() is not None:
del elem.getparent()[0]
del context
xf.flush()
It processes this xml file. And gets the following error:
Traceback (most recent call last):
File "/usr/local/bin/stetl", line 4, in <module>
__import__('pkg_resources').run_script('Stetl==2.0', 'stetl')
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1446, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/EGG-INFO/scripts/stetl", line 43, in <module>
main()
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/EGG-INFO/scripts/stetl", line 36, in main
etl.run()
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/stetl/etl.py", line 157, in run
chain.run()
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/stetl/chain.py", line 172, in run
packet = self.first_comp.process(packet)
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/stetl/component.py", line 213, in process
packet = self.next.process(packet)
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/stetl/component.py", line 213, in process
packet = self.next.process(packet)
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/stetl/component.py", line 213, in process
packet = self.next.process(packet)
File "/usr/local/lib/python3.6/site-packages/Stetl-2.0-py3.6.egg/stetl/component.py", line 199, in process
packet = self.invoke(packet)
File "/app/bgt/etl/stetlbgt/subfeaturehandler.py", line 144, in invoke
del context
File "src/lxml/serializer.pxi", line 925, in lxml.etree.xmlfile.__exit__
File "src/lxml/serializer.pxi", line 1263, in lxml.etree._IncrementalFileWriter._close
File "src/lxml/serializer.pxi", line 1269, in lxml.etree._IncrementalFileWriter._handle_error
File "src/lxml/serializer.pxi", line 199, in lxml.etree._raiseSerialisationError
lxml.etree.SerialisationError: unknown error -2029930774
I'm not sure what's going wrong here. It seems that something is wrong with some weird encoded character.
How to debug this?

python fabric execute error

I execute fabric with both fab.exe and run python script directly.
But it shows error.
My PC is Windows 7, Python 2.7.13
Traceback (most recent call last):
File "C:\fabric_test.py", line 3, in <module>
from fabric.api import run,env,execute,task
File "C:\Python27\lib\site-packages\fabric\api.py", line 10, in <module>
from fabric.context_managers import (cd, hide, settings, show, path, prefix,
File "C:\Python27\lib\site-packages\fabric\context_managers.py", line 41, in <module>
from fabric.state import output, win32, connections, env
File "C:\Python27\lib\site-packages\fabric\state.py", line 9, in <module>
from fabric.network import HostConnectionCache, ssh
File "C:\Python27\lib\site-packages\fabric\network.py", line 24, in <module>
import paramiko as ssh
File "C:\Python27\lib\site-packages\paramiko\__init__.py", line 31, in <module>
from paramiko.transport import SecurityOptions, Transport
File "C:\Python27\lib\site-packages\paramiko\transport.py", line 56, in <module>
from paramiko.dsskey import DSSKey
File "C:\Python27\lib\site-packages\paramiko\dsskey.py", line 27, in <module>
from cryptography.hazmat.primitives.asymmetric.utils import (
File "C:\Python27\lib\site-packages\cryptography\hazmat\primitives\asymmetric\utils.py", line 9, in <module>
from asn1crypto.algos import DSASignature
File "C:\Python27\lib\site-packages\asn1crypto\algos.py", line 23, in <module>
from ._int import fill_width
File "C:\Python27\lib\site-packages\asn1crypto\_int.py", line 56, in <module>
from ._perf._big_num_ctypes import libcrypto
File "C:\Python27\lib\site-packages\asn1crypto\_perf\_big_num_ctypes.py", line 31, in <module>
libcrypto_path = find_library('crypto')
File "C:\Python27\lib\ctypes\util.py", line 53, in find_library
fname = os.path.join(directory, name)
File "C:\Python27\lib\ntpath.py", line 85, in join
result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xbc in position 21: ordinal not in range(128)
what's the problem?

I hack the C:\Python27\lib\ntpath.py file.
add this source code to header of that file. (17 line)
reload(sys)
sys.setdefaultencoding('euc-kr')
then, error disappear. fabric works fine.
but i don't know reason.
what's the problem in my pc?
I want solve this problem without hack.

UnicodeDecodeError When I use cuda to train dataset

I used chainer to train some images but there is an error.
I don't know whether its UnicodeDecodeError or the error of installation of cupy.
P:\dcgans\chainer-DCGAN\chainer-DCGAN>python DCGAN.py
Traceback (most recent call last):
File "DCGAN.py", line 279, in <module>
train_dcgan_labeled(gen, dis)
File "DCGAN.py", line 171, in train_dcgan_labeled
zvis = (xp.random.uniform(-1, 1, (100, nz), dtype=np.float32))
File "P:\Python35\lib\site-packages\cupy\random\distributions.py", line 132, in uniform
return rs.uniform(low, high, size=size, dtype=dtype)
File "P:\Python35\lib\site-packages\cupy\random\generator.py", line 235, in uniform
rand = self.random_sample(size=size, dtype=dtype)
File "P:\Python35\lib\site-packages\cupy\random\generator.py", line 153, in random_sample
RandomState._1m_kernel(out)
File "cupy/core/elementwise.pxi", line 552, in cupy.core.core.ElementwiseKernel.__call__ (cupy\core\core.cpp:43810)
File "cupy/util.pyx", line 39, in cupy.util.memoize.decorator.ret (cupy\util.cpp:1480)
File "cupy/core/elementwise.pxi", line 409, in cupy.core.core._get_elementwise_kernel (cupy\core\core.cpp:42156)
File "cupy/core/elementwise.pxi", line 12, in cupy.core.core._get_simple_elementwise_kernel (cupy\core\core.cpp:34787)
File "cupy/core/elementwise.pxi", line 32, in cupy.core.core._get_simple_elementwise_kernel (cupy\core\core.cpp:34609)
File "cupy/core/carray.pxi", line 87, in cupy.core.core.compile_with_cache (cupy\core\core.cpp:34264)
File "P:\Python35\lib\site-packages\cupy\cuda\compiler.py", line 133, in compile_with_cache
base = _empty_file_preprocess_cache[env] = preprocess('', options)
File "P:\Python35\lib\site-packages\cupy\cuda\compiler.py", line 99, in preprocess
pp_src = pp_src.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 27-28: invalid continuation byte

It seems nvcc generated non-UTF8 output and CuPy failed to decode it.
This is a bug of CuPy (I posted an issue: #378).
A possible solution for the time being is to replace 'utf-8' in cupy/cuda/compiler.py at the line pp_src = pp_src.decode('utf-8') with something that match your environment. For example, in Japanese environment, 'cp932' should work, and 'cp936' should perhaps work for simplified Chinese.
You could also try locale.getdefaultlocale()[1] as a universal solution (be sure to import locale).
Update: The fix has been merged. It should be fixed in upcoming CuPy v1.0.3.

spyder - python IDE uft8 encoding loading issue

I am unable to get spyder to load. I am running windows 7, 64 bit.
I have installed Anaconda-2.3.0 64 bit and have attempted to run the bundled spyder.
I have also tried the latest separate spyder build and encountered the exact same issue.
When running spyder via cmd I recieve the following error message:
"
Traceback (most recent call last):
File "C:\Anaconda\scripts\spyder", line 2, in <module>
from spyderlib import start_app
File "C:\Anaconda\lib\site-packages\spyderlib\start_app.py", line 13, in <modu
le>
from spyderlib.config import CONF
File "C:\Anaconda\lib\site-packages\spyderlib\config.py", line 718, in <module
>
subfolder=SUBFOLDER, backup=True, raw_mode=True)
File "C:\Anaconda\lib\site-packages\spyderlib\userconfig.py", line 215, in __i
nit__
self.load_from_ini()
File "C:\Anaconda\lib\site-packages\spyderlib\userconfig.py", line 260, in loa
d_from_ini
self.readfp(configfile)
File "C:\Anaconda\lib\ConfigParser.py", line 324, in readfp
self._read(fp, filename)
File "C:\Anaconda\lib\ConfigParser.py", line 479, in _read
line = fp.readline()
File "C:\Anaconda\lib\codecs.py", line 678, in readline
return self.reader.readline(size)
File "C:\Anaconda\lib\codecs.py", line 533, in readline
data = self.read(readsize, firstline=True)
File "C:\Anaconda\lib\codecs.py", line 480, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe8 in position 59: invalid
continuation byte
"
Is there some way to fix this?
Thanks

Never mind. Figured it out.
uninstall/reinstall anaconda.
delete Libraries\Documents.spyder2 folder
Run spyder. Wait a long time. spyder loads.
yay.

UnicodeDecodeError in textblob tutorial

I'm trying to run through the TextBlob tutorial in Windows (using Git Bash shell) with Python 3.3.
I've installed textblob and nltk as well as any dependencies.
The Python code is:
from text.blob import TextBlob
wiki = TextBlob("Python is a high-level, general-purpose programming language.")
tags = wiki.tags
I'm getting the following error
Traceback (most recent call last):
File "textblob.py", line 4, in <module>
tags = wiki.tags
File "c:\Python33\lib\site-packages\text\decorators.py", line 18, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "c:\Python33\lib\site-packages\text\blob.py", line 357, in pos_tags
for word, t in self.pos_tagger.tag(self.raw)
File "c:\Python33\lib\site-packages\text\taggers.py", line 40, in tag
return pattern_tag(sentence, tokenize)
File "c:\Python33\lib\site-packages\text\en.py", line 115, in tag
for sentence in parse(s, tokenize, True, False, False, False, encoding).split():
File "c:\Python33\lib\site-packages\text\en.py", line 99, in parse
return parser.parse(unicode(s), *args, **kwargs)
File "c:\Python33\lib\site-packages\text\text.py", line 1213, in parse
s[i] = self.find_tags(s[i], **kwargs)
File "c:\Python33\lib\site-packages\text\en.py", line 49, in find_tags
return _Parser.find_tags(self, tokens, **kwargs)
File "c:\Python33\lib\site-packages\text\text.py", line 1161, in find_tags
map = kwargs.get( "map", None))
File "c:\Python33\lib\site-packages\text\text.py", line 967, in find_tags
tagged.append([token, lexicon.get(token, i==0 and lexicon.get(token.lower()) or None)])
File "c:\Python33\lib\site-packages\text\text.py", line 98, in get
return self._lazy("get", *args)
File "c:\Python33\lib\site-packages\text\text.py", line 79, in _lazy
self.load()
File "c:\Python33\lib\site-packages\text\text.py", line 367, in load
dict.update(self, (x.split(" ")[:2] for x in _read(self._path) if x.strip()))
File "c:\Python33\lib\site-packages\text\text.py", line 367, in <genexpr>
dict.update(self, (x.split(" ")[:2] for x in _read(self._path) if x.strip()))
File "c:\Python33\lib\site-packages\text\text.py", line 346, in _read
for line in f:
File "c:\Python33\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 16: character maps to <undefined>
Any idea what is wrong here? Adding a 'u' before the string didn't help.

Release 0.7.1 fixes this issue, which means it's time for a
$ pip install -U textblob
The problem was that the en-lexicon.txt file used for part-of-speech tagging opened the file using Windows' default platform encoding, cp1252. The file apparently had characters that Python could not decode from this encoding. This was fixed by explicitly opening the file in utf-8 mode.