UnicodeDecodeError When I use cuda to train dataset - python

I used chainer to train some images but there is an error.
I don't know whether its UnicodeDecodeError or the error of installation of cupy.
P:\dcgans\chainer-DCGAN\chainer-DCGAN>python DCGAN.py
Traceback (most recent call last):
File "DCGAN.py", line 279, in <module>
train_dcgan_labeled(gen, dis)
File "DCGAN.py", line 171, in train_dcgan_labeled
zvis = (xp.random.uniform(-1, 1, (100, nz), dtype=np.float32))
File "P:\Python35\lib\site-packages\cupy\random\distributions.py", line 132, in uniform
return rs.uniform(low, high, size=size, dtype=dtype)
File "P:\Python35\lib\site-packages\cupy\random\generator.py", line 235, in uniform
rand = self.random_sample(size=size, dtype=dtype)
File "P:\Python35\lib\site-packages\cupy\random\generator.py", line 153, in random_sample
RandomState._1m_kernel(out)
File "cupy/core/elementwise.pxi", line 552, in cupy.core.core.ElementwiseKernel.__call__ (cupy\core\core.cpp:43810)
File "cupy/util.pyx", line 39, in cupy.util.memoize.decorator.ret (cupy\util.cpp:1480)
File "cupy/core/elementwise.pxi", line 409, in cupy.core.core._get_elementwise_kernel (cupy\core\core.cpp:42156)
File "cupy/core/elementwise.pxi", line 12, in cupy.core.core._get_simple_elementwise_kernel (cupy\core\core.cpp:34787)
File "cupy/core/elementwise.pxi", line 32, in cupy.core.core._get_simple_elementwise_kernel (cupy\core\core.cpp:34609)
File "cupy/core/carray.pxi", line 87, in cupy.core.core.compile_with_cache (cupy\core\core.cpp:34264)
File "P:\Python35\lib\site-packages\cupy\cuda\compiler.py", line 133, in compile_with_cache
base = _empty_file_preprocess_cache[env] = preprocess('', options)
File "P:\Python35\lib\site-packages\cupy\cuda\compiler.py", line 99, in preprocess
pp_src = pp_src.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 27-28: invalid continuation byte

It seems nvcc generated non-UTF8 output and CuPy failed to decode it.
This is a bug of CuPy (I posted an issue: #378).
A possible solution for the time being is to replace 'utf-8' in cupy/cuda/compiler.py at the line pp_src = pp_src.decode('utf-8') with something that match your environment. For example, in Japanese environment, 'cp932' should work, and 'cp936' should perhaps work for simplified Chinese.
You could also try locale.getdefaultlocale()[1] as a universal solution (be sure to import locale).
Update: The fix has been merged. It should be fixed in upcoming CuPy v1.0.3.

Related

Error while training tensorflow object detection about checkpoint error

I have a problem about tensorflow training part.
speci:
tensorflow-gpu= 2.2.0
python= 3.7.9
cuda= 10.1
cdnn= 7.6.- (ı dont remember but it is ok with cuda).
models= ssd_resnet101_v1_fpn_1024x1024_coco17_tpu-8 and efficientdet_d7_coco17_tpu-32
reference: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html
when I start to train it gives that error:
Traceback (most recent call last):
File "model_main_tf2.py", line 113, in <module>
tf.compat.v1.app.run()
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\absl\app.py", line 300, in run
_run_main(main, args)
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "model_main_tf2.py", line 110, in main
record_summaries=FLAGS.record_summaries)
File "C:\TensorFlow\models\research\object_detection\model_lib_v2.py", line 578, in train_loop
ckpt, manager_dir, max_to_keep=checkpoint_max_to_keep)
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\checkpoint_management.py", line 635, in __init__
recovered_state = get_checkpoint_state(directory)
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\checkpoint_management.py", line 279, in get_checkpoint_state
coord_checkpoint_filename)
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 320, in read_file_to_string
return f.read()
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 116, in read
self._preread_check()
File "C:\Users\Nurullah\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 79, in _preread_check
self.__name, 1024 * 512)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 122: invalid start byte
my start-training command: And ı check this paths are correct
python model_main_tf2.py --logtostderr --model_dir=pre-trained-models/ssd_resnet101_v1_fpn_1024x1024_coco17_tpu-8 --pipeline_config_path=pre-trained-models/ssd_resnet101_v1_fpn_1024x1024_coco17_tpu-8/pipeline.config
Checkpoint path, pipeline path all of them are correct. I tried training with two different models. And cant solve. How can ı solve this problem?
I researched utf-8 errors but cant find solution. Thank you for helping. :))
I met the same issue, which made me struggled overnight.
I have the same env like yours, and I tired mask_rcnn model.
In a nutshell, it can be figured out by changing the output_dir.
Origin:
----mask_rcnn_model (output_dir)
--------checkpoint (folder)
--------saved_model (folder)
--------my_pipeline.config
After:
----output_dir
----mask_rcnn_model
--------checkpoint (folder)
--------saved_model (folder)
--------my_pipeline.config
hint from: https://github.com/tensorflow/models/issues/8892
it worked for my situation.

Rasa App breaks in Pycharm but works fine in terminal

Whenever I try to run my Rasa app using the run button in PyCharm, or try to use the debugger, I get the following error:
Traceback (most recent call last):
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/pykwalify/core.py", line 76, in __init__
self.source = yaml.load(stream)
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/ruamel/yaml/main.py", line 933, in load
loader = Loader(stream, version, preserve_quotes=preserve_quotes)
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/ruamel/yaml/loader.py", line 50, in __init__
Reader.__init__(self, stream, loader=self)
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/ruamel/yaml/reader.py", line 85, in __init__
self.stream = stream # type: Any # as .read is called
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/ruamel/yaml/reader.py", line 130, in stream
self.determine_encoding()
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/ruamel/yaml/reader.py", line 190, in determine_encoding
self.update_raw()
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/ruamel/yaml/reader.py", line 297, in update_raw
data = self.stream.read(size)
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 473: ordinal not in range(128)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/matthewspeck/project/trainer_app/app.py", line 25, in <module>
parser=False, core=True)
File "/Users/matthewspeck/project/trainer_app/rasa_model.py", line 165, in make_rasa_model
rasa_config=rasa_config
File "/Users/matthewspeck/project/trainer_app/rasa_model.py", line 66, in __init__
self._parser = create_agent(use_rasa_nlu=True, load_models=True)
File "/Users/matthewspeck/project/trainer_app/rasa.py", line 32, in create_agent
domain = create_domain()
File "/Users/matthewspeck/project/trainer_app/rasa.py", line 83, in create_domain
domain = ClarifyDomain.load(domain_path)
File "/Users/project/clarification/domain.py", line 39, in load
domain = TemplateDomain.load(filename)
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/rasa_core/domain.py", line 404, in load
cls.validate_domain_yaml(filename)
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/rasa_core/domain.py", line 438, in validate_domain_yaml
schema_files=[schema_file])
File "/Users/matthewspeck/anaconda3/envs/proj_env/lib/python3.6/site-packages/pykwalify/core.py", line 78, in __init__
raise CoreError(u"Unable to load any data from source yaml file")
pykwalify.errors.CoreError: <CoreError: error code 3: Unable to load any data from source yaml file: Path: '/'>
Process finished with exit code 1
However, when I run the app from my terminal, or from my text editor (I use VSCode), It runs with no problems whatsoever. I've looked online and every answer I see has something to do with Rasa, but nothing mentions problems with PyCharm.
I've also checked that the yaml for the domain is properly formatted, and it is. Anyone have any idea why I would be getting this error in PyCharm, but not in any other environment, and how I could fix it?
I believe your problem was fixed with Rasa version 0.12 ([changelog][1]): https://github.com/RasaHQ/rasa_core/blob/master/CHANGELOG.rst#0120---2018-11-11 .
I recommend upgrading to a newer version of Rasa Core which parses the training data correctly.

Spyder crashes at start: UnicodeDecodeError

During a Spyder session my Linux froze. After startup, I could not start Spyder; I got the following error instead:
(trusty)dreamer#localhost:~$ spyder
Traceback (most recent call last):
File "/home/dreamer/anaconda2/bin/spyder", line 2, in <module>
from spyderlib import start_app
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/start_app.py", line 13, in <module>
from spyderlib.config import CONF
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/config.py", line 736, in <module>
subfolder=SUBFOLDER, backup=True, raw_mode=True)
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/userconfig.py", line 215, in __init__
self.load_from_ini()
File "/home/dreamer/anaconda2/lib/python2.7/site-packages/spyderlib/userconfig.py", line 260, in load_from_ini
self.readfp(configfile)
File "/home/dreamer/anaconda2/lib/python2.7/ConfigParser.py", line 324, in readfp
self._read(fp, filename)
File "/home/dreamer/anaconda2/lib/python2.7/ConfigParser.py", line 479, in _read
line = fp.readline()
File "/home/dreamer/anaconda2/lib/python2.7/codecs.py", line 690, in readline
return self.reader.readline(size)
File "/home/dreamer/anaconda2/lib/python2.7/codecs.py", line 545, in readline
data = self.read(readsize, firstline=True)
File "/home/dreamer/anaconda2/lib/python2.7/codecs.py", line 492, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 2: invalid start byte
(trusty)dreamer#localhost:~$
I have found this solution, which sounds very much like my problem, but am curious if there are others, and whether anyone knows why this occurred.
My guess is that your spyder configuration file somehow got corrupted. This is the file spyder.ini, which resides in a directory like ~/.spyder2 (the exact name of the directory depends on the version you have installed). Maybe the encoding of the configuration file changed or a Unicode byte order mark was somehow introduced.
Possible solutions: use an editor to convert the file back to UTF-8; delete the configuration file; delete the whole directory containing the configuration file. The last two obviously delete any changes you made to the configuration.

spyder - python IDE uft8 encoding loading issue

I am unable to get spyder to load. I am running windows 7, 64 bit.
I have installed Anaconda-2.3.0 64 bit and have attempted to run the bundled spyder.
I have also tried the latest separate spyder build and encountered the exact same issue.
When running spyder via cmd I recieve the following error message:
"
Traceback (most recent call last):
File "C:\Anaconda\scripts\spyder", line 2, in <module>
from spyderlib import start_app
File "C:\Anaconda\lib\site-packages\spyderlib\start_app.py", line 13, in <modu
le>
from spyderlib.config import CONF
File "C:\Anaconda\lib\site-packages\spyderlib\config.py", line 718, in <module
>
subfolder=SUBFOLDER, backup=True, raw_mode=True)
File "C:\Anaconda\lib\site-packages\spyderlib\userconfig.py", line 215, in __i
nit__
self.load_from_ini()
File "C:\Anaconda\lib\site-packages\spyderlib\userconfig.py", line 260, in loa
d_from_ini
self.readfp(configfile)
File "C:\Anaconda\lib\ConfigParser.py", line 324, in readfp
self._read(fp, filename)
File "C:\Anaconda\lib\ConfigParser.py", line 479, in _read
line = fp.readline()
File "C:\Anaconda\lib\codecs.py", line 678, in readline
return self.reader.readline(size)
File "C:\Anaconda\lib\codecs.py", line 533, in readline
data = self.read(readsize, firstline=True)
File "C:\Anaconda\lib\codecs.py", line 480, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe8 in position 59: invalid
continuation byte
"
Is there some way to fix this?
Thanks
Never mind. Figured it out.
uninstall/reinstall anaconda.
delete Libraries\Documents.spyder2 folder
Run spyder. Wait a long time. spyder loads.
yay.

UnicodeDecodeError in textblob tutorial

I'm trying to run through the TextBlob tutorial in Windows (using Git Bash shell) with Python 3.3.
I've installed textblob and nltk as well as any dependencies.
The Python code is:
from text.blob import TextBlob
wiki = TextBlob("Python is a high-level, general-purpose programming language.")
tags = wiki.tags
I'm getting the following error
Traceback (most recent call last):
File "textblob.py", line 4, in <module>
tags = wiki.tags
File "c:\Python33\lib\site-packages\text\decorators.py", line 18, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "c:\Python33\lib\site-packages\text\blob.py", line 357, in pos_tags
for word, t in self.pos_tagger.tag(self.raw)
File "c:\Python33\lib\site-packages\text\taggers.py", line 40, in tag
return pattern_tag(sentence, tokenize)
File "c:\Python33\lib\site-packages\text\en.py", line 115, in tag
for sentence in parse(s, tokenize, True, False, False, False, encoding).split():
File "c:\Python33\lib\site-packages\text\en.py", line 99, in parse
return parser.parse(unicode(s), *args, **kwargs)
File "c:\Python33\lib\site-packages\text\text.py", line 1213, in parse
s[i] = self.find_tags(s[i], **kwargs)
File "c:\Python33\lib\site-packages\text\en.py", line 49, in find_tags
return _Parser.find_tags(self, tokens, **kwargs)
File "c:\Python33\lib\site-packages\text\text.py", line 1161, in find_tags
map = kwargs.get( "map", None))
File "c:\Python33\lib\site-packages\text\text.py", line 967, in find_tags
tagged.append([token, lexicon.get(token, i==0 and lexicon.get(token.lower()) or None)])
File "c:\Python33\lib\site-packages\text\text.py", line 98, in get
return self._lazy("get", *args)
File "c:\Python33\lib\site-packages\text\text.py", line 79, in _lazy
self.load()
File "c:\Python33\lib\site-packages\text\text.py", line 367, in load
dict.update(self, (x.split(" ")[:2] for x in _read(self._path) if x.strip()))
File "c:\Python33\lib\site-packages\text\text.py", line 367, in <genexpr>
dict.update(self, (x.split(" ")[:2] for x in _read(self._path) if x.strip()))
File "c:\Python33\lib\site-packages\text\text.py", line 346, in _read
for line in f:
File "c:\Python33\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 16: character maps to <undefined>
Any idea what is wrong here? Adding a 'u' before the string didn't help.
Release 0.7.1 fixes this issue, which means it's time for a
$ pip install -U textblob
The problem was that the en-lexicon.txt file used for part-of-speech tagging opened the file using Windows' default platform encoding, cp1252. The file apparently had characters that Python could not decode from this encoding. This was fixed by explicitly opening the file in utf-8 mode.

Categories