We Keep Coding

Google Colab RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Yesterday and today running the same Python notebooks that I am running the past few months, I am getting the error
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
97 Variable._execution_engine.run_backward(
98 tensors, grad_tensors, retain_graph, create_graph,
---> 99 allow_unreachable=True) # allow_unreachable flag
100
101
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
The point in the code where this error seems to be random since it changes from try to try. From what I have searched, it looks to be a compatibility issue.
Also, if I rerun the cell, I might get another error which is,
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
346 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
347 if self._pin_memory:
--> 348 data = _utils.pin_memory.pin_memory(data)
349 return data
350
/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/pin_memory.py in pin_memory(data)
53 return type(data)(*(pin_memory(sample) for sample in data))
54 elif isinstance(data, container_abcs.Sequence):
---> 55 return [pin_memory(sample) for sample in data]
56 elif hasattr(data, "pin_memory"):
57 return data.pin_memory()
/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/pin_memory.py in <listcomp>(.0)
53 return type(data)(*(pin_memory(sample) for sample in data))
54 elif isinstance(data, container_abcs.Sequence):
---> 55 return [pin_memory(sample) for sample in data]
56 elif hasattr(data, "pin_memory"):
57 return data.pin_memory()
/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils /pin_memory.py in pin_memory(data)
45 def pin_memory(data):
46 if isinstance(data, torch.Tensor):
---> 47 return data.pin_memory()
48 elif isinstance(data, string_classes):
49 return data
RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278
Does anyone else have the same problem? Did anyone solve it, how?

Finally, I solved the problem.
Somewhere in my code I use a CrossEntropyLoss function with ignore_index parameter as ignore_index = my_ignore_index. By mistake, I had my_ignore_index = -1 which as value, it is not a valid value for my data; -1 never appears in my data values. Updating correctly solved the problem. This solved the "... an illegal memory access was encou..." error.
The other thing that I did and helped to solve the problem was to use a newer version of anaconda3. This solved the CUDNN_STATUS_NOT_INITIALIZED error.
I hope that helps.

Is possible to use numba for the evaluation of a distance between 2d arrays?

I'm trying to build my own distance function but I need it quite efficient so I hoped I could use numba. The problem is that it returns an error when I try to insert more than 1 input value. This code doesn't work and I cannot understand why:
import numba
import numpy as np
#numba.njit(parallel=True)
def distance(a,b):
dist = np.zeros((len(a),len(b))
for i in numba.prange(len(a)):
for j in numba.prange(len(b)):
dist[i,j] = np.linalg.norm(a[i]-b[j])
return dist
The error I obtain is:
---------------------------------------------------------------------------
TypingError Traceback (most recent call last)
<ipython-input-18-9be257aa275d> in <module>
----> 1 dist = distance(a,b)
~/miniconda3/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
374 e.patch_message(msg)
375
--> 376 error_rewrite(e, 'typing')
377 except errors.UnsupportedError as e:
378 # Something unsupported is present in the user code, add help info
~/miniconda3/lib/python3.7/site-packages/numba/dispatcher.py in error_rewrite(e, issue_type)
341 raise e
342 else:
--> 343 reraise(type(e), e, None)
344
345 argtypes = []
~/miniconda3/lib/python3.7/site-packages/numba/six.py in reraise(tp, value, tb)
656 value = tp()
657 if value.__traceback__ is not tb:
--> 658 raise value.with_traceback(tb)
659 raise value
660
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<function norm at 0x7f21b053add0>) with argument(s) of type(s): (array(int64, 1d, C))
* parameterized
In definition 0:
TypingError: np.linalg.norm() only supported on float and complex arrays.
raised from /home/plongo/miniconda3/lib/python3.7/site-packages/numba/targets/linalg.py:847
In definition 1:
TypingError: np.linalg.norm() only supported on float and complex arrays.
raised from /home/plongo/miniconda3/lib/python3.7/site-packages/numba/targets/linalg.py:847
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: resolving callee type: Function(<function norm at 0x7f21b053add0>)
[2] During: typing of call at <ipython-input-16-e3299481baaf> (6)
File "<ipython-input-16-e3299481baaf>", line 6:
def distance(a,b):
<source elided>
for j in numba.prange(len(b)):
dist[i,j] = np.linalg.norm(a[i]-b[j])
^
This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.
To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html
For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile
If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new
I totally cannot understand what is wrong... even better would be to use CUDA but I get the same problem.. I think I missed some concept.
Thank you

what reliable method to save huge numpy arrays

I saved some arrays using numpy.savez_compressed(). One of the arrays is gigantic, it has the shape (120000,7680), type float32.
Trying to load the array gave me the error below (message caught using Ipython).
Is seems like this is a Numpy limitation:
Numpy: apparent memory error
What are other ways to save such a huge array? (I had problems with cPickle as well)
In [5]: t=numpy.load('humongous.npz')
In [6]: humg = (t['arr_0.npy'])
/usr/lib/python2.7/dist-packages/numpy/lib/npyio.pyc in __getitem__(self, key)
229 if bytes.startswith(format.MAGIC_PREFIX):
230 value = BytesIO(bytes)
--> 231 return format.read_array(value)
232 else:
233 return bytes
/usr/lib/python2.7/dist-packages/numpy/lib/format.pyc in read_array(fp)
456 # way.
457 # XXX: we can probably chunk this to avoid the memory hit.
--> 458 data = fp.read(int(count * dtype.itemsize))
459 array = numpy.fromstring(data, dtype=dtype, count=count)
460
SystemError: error return without exception set
System: Ubuntu 12.04 64 bit, Python 2.7, numpy 1.6.1

Why the source code of conjugate in Numpy cannot be found by using the inspect module?

I want to see the implementation of the conjugate function used in Numpy. Then I tried the following:
import numpy as np
import inspect
inspect.getsource(np.conjugate)
However, I received the following error message stating that <ufunc 'conjugate'> is not a module, class, method, function, traceback, frame, or code object. May someone answer why?
In [8]: inspect.getsource(np.conjugate)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-8-821ecfb71e08> in <module>()
----> 1 inspect.getsource(np.conj)
/Users/duanlx/anaconda/python.app/Contents/lib/python2.7/inspect.pyc in getsource(object)
699 or code object. The source code is returned as a single string. An
700 IOError is raised if the source code cannot be retrieved."""
--> 701 lines, lnum = getsourcelines(object)
702 return string.join(lines, '')
703
/Users/duanlx/anaconda/python.app/Contents/lib/python2.7/inspect.pyc in getsourcelines(object)
688 original source file the first line of code was found. An IOError is
689 raised if the source code cannot be retrieved."""
--> 690 lines, lnum = findsource(object)
691
692 if ismodule(object): return lines, 0
/Users/duanlx/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc in findsource(object)
149 FIXED version with which we monkeypatch the stdlib to work around a bug."""
150
--> 151 file = getsourcefile(object) or getfile(object)
152 # If the object is a frame, then trying to get the globals dict from its
153 # module won't work. Instead, the frame object itself has the globals
/Users/duanlx/anaconda/python.app/Contents/lib/python2.7/inspect.pyc in getsourcefile(object)
442 Return None if no way can be identified to get the source.
443 """
--> 444 filename = getfile(object)
445 if string.lower(filename[-4:]) in ('.pyc', '.pyo'):
446 filename = filename[:-4] + '.py'
/Users/duanlx/anaconda/python.app/Contents/lib/python2.7/inspect.pyc in getfile(object)
418 return object.co_filename
419 raise TypeError('{!r} is not a module, class, method, '
--> 420 'function, traceback, frame, or code object'.format(object))
421
422 ModuleInfo = namedtuple('ModuleInfo', 'name suffix mode module_type')
TypeError: <ufunc 'conjugate'> is not a module, class, method, function, traceback, frame, or code object
Thanks!

Numpy is written in C, for speed. You can only see the source of Python functions.

PyFITS: File already exists

I'm really close to completing a large code, but the final segment of it seems to be failing and I don't know why. What I'm trying to do here is take an image-array, compare it to a different image array, and wherever the initial image array equals 1, I want to mask that portion out in the second image array. However, I'm getting a strange error:
Code:
maskimg='omask'+str(inimgs)[5:16]+'.fits'
newmaskimg=pf.getdata(maskimg)
oimg=pf.getdata(inimgs)
for i in range (newmaskimg.shape[0]):
for j in range (newmaskimg.shape[1]):
if newmaskimg[i,j]==1:
oimg[i,j]=0
pf.writeto('newestmask'+str(inimgs)[5:16]+'.fits',newmaskimg)
Error:
/home/vidur/se_files/fetch_swarp10.py in objmask(inimgs, inwhts, thresh1, thresh2, tfdel, xceng, yceng, outdir, tmpdir)
122 if newmaskimg[i,j]==1:
123 oimg[i,j]=0
--> 124 pf.writeto('newestmask'+str(inimgs)[5:16]+'.fits',newmaskimg)
125
126
/usr/local/lib/python2.7/dist-packages/pyfits/convenience.pyc in writeto(filename, data, header, output_verify, clobber, checksum)
396 hdu = PrimaryHDU(data, header=header)
397 hdu.writeto(filename, clobber=clobber, output_verify=output_verify,
--> 398 checksum=checksum)
399
400
/usr/local/lib/python2.7/dist-packages/pyfits/hdu/base.pyc in writeto(self, name, output_verify, clobber, checksum)
348 hdulist = HDUList([self])
349 hdulist.writeto(name, output_verify, clobber=clobber,
--> 350 checksum=checksum)
351
352 def _get_raw_data(self, shape, code, offset):
/usr/local/lib/python2.7/dist-packages/pyfits/hdu/hdulist.pyc in writeto(self, fileobj, output_verify, clobber, checksum)
651 os.remove(filename)
652 else:
--> 653 raise IOError("File '%s' already exists." % filename)
654 elif (hasattr(fileobj, 'len') and fileobj.len > 0):
655 if clobber:
IOError: File 'newestmaskPHOTOf105w0.fits' already exists.

If you don't care about overwriting the existing file, pyfits.writeto accepts a clobber argument to automatically overwrite existing files (it will still output a warning):
pyfits.writeto(..., clobber=True)
As an aside, let me be very emphatic that the code you posted above is very much not the right way to use Numpy. The loop in your code can be written in one line and will be orders of magnitude faster. For example, one of many possibilities is to write it like this:
oimg[newmaskimg == 1] = 0

Yes, add clobber = True. I've used this in my codes before and it works just fine. Or, simply go and sudo rm path/to/file and get rid of them so you can run it again.

I had the same issue and as it turns out using the argument clobber still works but won't be supported in future versions of AstroPy.
The argument overwrite does the same thing and doesn't put out an error message.