We Keep Coding

File I/O error using nglview.show_biopython(structure)

So I have been trying to get into visualizing proteins in python, so I went on youtube and found some tutorials I ended up on a tutorial that was teaching you how to visualize a protein from the COVID-19 virus, so I went and setup anaconda, got jupyter notebook working vscode, and downloaded the necessary files from the PDB database, and made sure they were in the same directory as my notebook but when I run the the nglview.show_biopython(structure) function I get an ValueError: I/O opertation on a closed file. I'm stummed this is my first time using jupyter notebook so maybe there is something I'm missing, I don't know.
This what the code looks like
from Bio.PDB import *
import nglview as nv
parser = PDBParser()
structure = parser.get_structure("6YYT", "6YYT.pdb")
view = nv.show_biopython(structure)
This is the error
Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_1728\2743687014.py in <module>
----> 1 view = nv.show_biopython(structure)
c:\Users\jerem\anaconda3\lib\site-packages\nglview\show.py in show_biopython(entity, **kwargs)
450 '''
451 entity = BiopythonStructure(entity)
--> 452 return NGLWidget(entity, **kwargs)
453
454
c:\Users\jerem\anaconda3\lib\site-packages\nglview\widget.py in __init__(self, structure, representations, parameters, **kwargs)
243 else:
244 if structure is not None:
--> 245 self.add_structure(structure, **kwargs)
246
247 if representations:
c:\Users\jerem\anaconda3\lib\site-packages\nglview\widget.py in add_structure(self, structure, **kwargs)
1111 if not isinstance(structure, Structure):
1112 raise ValueError(f'{structure} is not an instance of Structure')
-> 1113 self._load_data(structure, **kwargs)
1114 self._ngl_component_ids.append(structure.id)
1115 if self.n_components > 1:
...
--> 200 return io_str.getvalue()
201
202
ValueError: I/O operation on closed file
I only get this error when using nglview.show_biopython, when I run the get_structure() function it can read the file just fine. I can visualize other molucles just fine, or maybe that's because I was using the ASE library instead of a file. I don't know, that's why I'm here.
Update: Recently I found out that I can visualize the protein using nglview.show_file() instead of using nglview.show_biopython(). Even though I can visualize proteins now and techincally my problem has been solved I would still like to know why the show_biopython() function isn't working properly.

I also figured out another way to fix this problem. After going back to the tutorial I was talking about I saw that it was made back in 2021. After seeing this I wonder if we were using the same verions of each package, turns out we were not. I'm not sure what version of nglview they were using, but they were using biopython 1.79 which was the latest verion back in 2021 and I was using biopython 1.80. While using biopython 1.80 I was getting the error seen above. But now that I'm using biopython 1.79 I get this:
file = "6YYT.pdb"
parser = PDBParser()
structure = parser.get_structure("6YYT", file)
structure
view = nv.show_biopython(structure)
view
Output:
c:\Users\jerem\anaconda3\lib\site-packages\Bio\PDB\StructureBuilder.py:89:
PDBConstructionWarning: WARNING: Chain A is discontinuous at line 12059.
warnings.warn(
So I guess there is something going on with biopython 1.80, so I'm going to stick with 1.79

I had a similar problem with:
from Bio.PDB import *
import nglview as nv
parser = PDBParser(QUIET = True)
structure = parser.get_structure("2ms2", "2ms2.pdb")
save_pdb = PDBIO()
save_pdb.set_structure(structure)
save_pdb.save('pdb_out.pdb')
view = nv.show_biopython(structure)
view
error was like in question:
.................site-packages/nglview/adaptor.py:201, in BiopythonStructure.get_structure_string(self)
199 io_str = StringIO()
200 io_pdb.save(io_str)
--> 201 return io_str.getvalue()
ValueError: I/O operation on closed file
I modified site-packages/nglview/adaptor.py:201, in BiopythonStructure.get_structure_string(self):
def get_structure_string(self):
from Bio.PDB import PDBIO
from io import StringIO
io_pdb = PDBIO()
io_pdb.set_structure(self._entity)
io_str = StringIO()
io_pdb.save(io_str)
return io_str.getvalue()
with :
def get_structure_string(self):
from Bio.PDB import PDBIO
import mmap
io_pdb = PDBIO()
io_pdb.set_structure(self._entity)
mo = mmap_str()
io_pdb.save(mo)
return mo.read()
and added this new class mmap_str() , in same file:
import mmap
import copy
class mmap_str():
import mmap #added import at top
instance = None
def __init__(self):
self.mm = mmap.mmap(-1, 2)
self.a = ''
b = '\n'
self.mm.write(b.encode(encoding = 'utf-8'))
self.mm.seek(0)
#print('self.mm.read().decode() ',self.mm.read().decode(encoding = 'utf-8'))
self.mm.seek(0)
def __new__(cls, *args, **kwargs):
if not isinstance(cls.instance, cls):
cls.instance = object.__new__(cls)
return cls.instance
def write(self, string):
self.a = str(copy.deepcopy(self.mm.read().decode(encoding = 'utf-8'))).lstrip('\n')
self.mm.seek(0)
#print('a -> ', self.a)
len_a = len(self.a)
self.mm = mmap.mmap(-1, len(self.a)+len(string))
#print('a :', self.a)
#print('len self.mm ', len(self.mm))
#print('lenght string : ', len(string))
#print(bytes((self.a+string).encode()))
self.mm.write(bytes((self.a+string).encode()))
self.mm.seek(0)
#print('written once ')
#self.mm.seek(0)
def read(self):
self.mm.seek(0)
a = self.mm.read().decode().lstrip('\n')
self.mm.seek(0)
return a
def __enter__(self):
return self
def __exit__(self, *args):
pass
if I uncomment the print statements I'll get the :
IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
error , but commenting them out I get:
while using thenglview.show_file(filename) I get:
tha's because, as could be seen looking at the pdb_out.pdb file
outputted by my code, Biopytho.PDB.PDBParser.get_structure(name , filename) doesnt retrieve the pdb header responsible for generate full CRYSTALLOGRAPHIC SYMMETRY/or biopython can't handle it (not sure about this, help if you know better), but just the coordinates.
Still don't understand what is going on with the :
--> 201 return io_str.getvalue()
ValueError: I/O operation on closed file
it could be something related to jupiter ipykernal ? hope somebody could shed more light into this, dont know how the framework runs, but is definitely different from a normal python interpreter. As an example:
Same code in one of my Python virtualenv, will run forever, so it could be ipykernel dont like StringIO()s or do something strange to them ?
OK thanks to the hint in the answer below, I went inspecting PDBIO.py in github repo for version Biopython 1.80 and compared the save method of PDBIO : def save(self, file, select=_select, write_end=True, preserve_atom_numbering=False): with the one in Biopython 1.79,
see first bit:
and last bit:
so apparently the big difference is the with fhandle: block in version 1.80.
So I realized that changing adaptor.py with adding a subclass of StringIO that looks like:
from io import StringIO
class StringIO(StringIO):
def __exit__(self, *args, **kwargs):
print('exiting from subclassed StringIO !!!!!')
pass
and modifying def get_structure_string(self): like this:
def get_structure_string(self):
from Bio.PDB import PDBIO
#from io import StringIO
io_pdb = PDBIO()
io_pdb.set_structure(self._entity)
io_str = StringIO()
io_pdb.save(io_str)
return io_str.getvalue()
was enough to get my Biopython 1.80 work in jupiter with nglview.
That told I am not sure what are the pitfalls of not closing the StringIO object we use for the visualization, but apparently its what Biopython 1.79 was doing like my first answer using a mmap object was doing too (not closing the mmap_str)

Another way to solve the probelm:
tried to understand git, I ended up with this, seems more coherent with the previous habits in the biopython project, but cant push it.
It makes use of as_handle from BIO.file :https://github.com/biopython/biopython/blob/e1902d1cdd3aa9325b4622b25d82fbf54633e251/Bio/File.py#L28
#contextlib.contextmanager
def as_handle(handleish, mode="r", **kwargs):
r"""Context manager to ensure we are using a handle.
Context manager for arguments that can be passed to SeqIO and AlignIO read, write,
and parse methods: either file objects or path-like objects (strings, pathlib.Path
instances, or more generally, anything that can be handled by the builtin 'open'
function).
When given a path-like object, returns an open file handle to that path, with provided
mode, which will be closed when the manager exits.
All other inputs are returned, and are *not* closed.
Arguments:
- handleish - Either a file handle or path-like object (anything which can be
passed to the builtin 'open' function, such as str, bytes,
pathlib.Path, and os.DirEntry objects)
- mode - Mode to open handleish (used only if handleish is a string)
- kwargs - Further arguments to pass to open(...)
Examples
--------
>>> from Bio import File
>>> import os
>>> with File.as_handle('seqs.fasta', 'w') as fp:
... fp.write('>test\nACGT')
...
10
>>> fp.closed
True
>>> handle = open('seqs.fasta', 'w')
>>> with File.as_handle(handle) as fp:
... fp.write('>test\nACGT')
...
10
>>> fp.closed
False
>>> fp.close()
>>> os.remove("seqs.fasta") # tidy up
"""
try:
with open(handleish, mode, **kwargs) as fp:
yield fp
except TypeError:
yield handleish
Anyone could pass it along ? [of course needs to be checked out, my tests are OK, but I am a novice].

How to serialize a scandir.DirEntry in Python for sending through a network socket?

I have server and client programs that communicate with each other through a network socket.
What I want is to send a directory entry (scandir.DirEntry) obtained from scandir.scandir() through the socket.
For now I am using pickle and cPickle modules and have come up with the following (excerpt only):
import scandir, pickle
s = scandir.scandir("D:\\PYTHON")
entry = s.next()
data = pickle.dumps(entry)
However, I am getting the following error stack:
File "untitled.py", line 5, in <module>
data = pickle.dumps(item)
File "C:\Python27\Lib\pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "C:\Python27\Lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\Lib\pickle.py", line 306, in save
rv = reduce(self.proto)
File "C:\Python27\Lib\copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle DirEntry objects
How can I get rid of this error?
I have heard of using marshall or JSON.
UPDATE: JSON is not dumping all the data within the object.
Is there any completely different way to do so to send the object through the socket?
Thanks in advance for any help.

Yes, os.DirEntry objects are intended to be short-lived, not really kept around or serialized. If you need the data in them to be serialized, looks like you've figured that out in your own answer -- serialize (pickle) a dict version of the attributes you need.
To deserialize into an object that walks and quacks like an os.DirEntry instance, create a PseudoDirEntry class that mimics the things you need.
Note that you can directly serialize the stat object already, which saves you picking the fields out of that.
Combined, that would look like this:
class PseudoDirEntry:
def __init__(self, name, path, is_dir, stat):
self.name = name
self.path = path
self._is_dir = is_dir
self._stat = stat
def is_dir(self):
return self._is_dir
def stat(self):
return self._stat
And then:
>>> import os, pickle
>>> entry = list(os.scandir())[0]
>>> pickled = pickle.dumps({'name': entry.name, 'path': entry.path, 'is_dir': entry.is_dir(), 'stat': entry.stat()})
>>> loaded = pickle.loads(pickled)
>>> pseudo = PseudoDirEntry(loaded['name'], loaded['path'], loaded['is_dir'], loaded['stat'])
>>> pseudo.name
'.DS_Store'
>>> pseudo.is_dir()
False
>>> pseudo.stat()
os.stat_result(st_mode=33188, st_ino=8370294, st_dev=16777220, st_nlink=1, st_uid=502, st_gid=20, st_size=8196, st_atime=1478356967, st_mtime=1477601172, st_ctime=1477601172)

Well I myself have figured out that for instances of non-standard classes like this scandir.DirEntry, the best way is to convert the class member data into a (possibly nested) combination of standard objects like (list, dict, etc.).
For example, in the particular case of scandir.DirEntry, it can be done as follows.
import scandir, pickle
s = scandir.scandir("D:\\PYTHON")
entry = s.next()
# first convert the stat object to st_
st = entry.stat()
st_ = {'st_mode':st.st_mode, 'st_size':st.st_size,\
'st_atime':st.st_atime, 'st_mtime':st.st_mtime,\
'st_ctime':st.st_ctime}
# now convert the entry object to entry_
entry_ = {'name':entry.name, 'is_dir':entry.is_dir(), \
'path':entry.path, 'stat':st_}
# one may need some other class member data also as necessary
# now pickle the converted entry_
data = pickle.dumps(entry_)
Although for my purpose, I only require the data, after the unpickling in the other end, one may need to reconstruct the unpickled entry_ to unpickled scandir.DirEntry object 'entry'. However, I am yet to figure out how to reconstruct the class instance and set the data for the behaviour of methods like is_dir(), stat().

How do I mock a class's function's return value?

I have a method in Python that looks like this (in comicfile.py):
from zipfile import ZipFile
...
class ComicFile():
...
def page_count(self):
"""Return the number of pages in the file."""
if self.file == None:
raise ComicFile.FileNoneError()
if not os.path.isfile(self.file):
raise ComicFile.FileNotFoundError()
with ZipFile(self.file) as zip:
members = zip.namelist()
pruned = self.prune_dirs(members)
length = len(pruned)
return length
I'm trying to write a unit test for this (I've already tested prune_dirs), and so for this is what I have (test_comicfile.py):
import unittest
import unittest.mock
import comicfile
...
class TestPageCount(unittest.TestCase):
def setUp(self):
self.comic_file = comicfile.ComicFile()
#unittest.mock.patch('comicfile.ZipFile')
def test_page_count(self, mock_zip_file):
# Store as tuples to use as dictionary keys.
members_dict = {('dir/', 'dir/file1', 'dir/file2'):2,
('file1.jpg', 'file2.jpg', 'file3.jpg'):3
}
# Make the file point to something to prevent FileNoneError.
self.comic_file.file = __file__
for file_tuple, count in members_dict.items():
mock_zip_file.return_value.namelist = list(file_tuple)
self.assertEqual(count, self.comic_file.page_count())
When I run this test, I get the following:
F..ss....
======================================================================
FAIL: test_page_count (test_comicfile.TestPageCount)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/mock.py", line 1157, in patched
return func(*args, **keywargs)
File "/Users/chuck/Dropbox/Projects/chiv/chiv.cbstar/test_comicfile.py", line 86, in test_page_count
self.assertEqual(count, self.comic_file.page_count())
AssertionError: 2 != 0
----------------------------------------------------------------------
Ran 9 tests in 0.010s
FAILED (failures=1, skipped=2)
OK, so self.comic_file.page_count() is returning 0. I tried placing the following line after members = zip.namelist() in page_count.
print('\nmembers -> ' + str(members))
During the test, I get this:
members -> <MagicMock name='ZipFile().__enter__().namelist()' id='4483358280'>
I'm quite new to unit testing and am quite nebulous on using unittest.mock, but my understanding is that mock_zip-file.return_value.namelist = list(file_tuple) should have made it so that the namelist method of the ZipFile class would return each of the file_tuple contents in turn. What it is doing I have no idea.
I think what I'm trying to do here is clear, but I can't seem to figure out how to override the namelist method so that my unit test is only testing this one function instead of having to deal with ZipFile as well.

ZipFile is instantiated as a context manager. to mock it you have to refer to its __enter__ method.
mock_zip_file.return_value.__enter__.return_value.namelist.return_value = list(file_tuple)
What you're trying to do is very clear, but the context manager adds complexity to the mocking.
One trick is that when a mock registers all calls made to it, in this example it is saying it has a call at:
members -> <MagicMock name='ZipFile().__enter__().namelist()' id='4483358280'>
This can guide you in registering your mocked object, replace all () with return_value

twisted xmlrpc and numpy float 64 exception

I'm using numpy to to some staff and then serve the results via a twisted/XMLRPC server. If the result is a numpy float 64, I get an exception cause probably twisted can't handle this type. Infact is I downgrade the result to float32 with x=float(x), everything is ok.
This is not so good cause if I forget this workaroud somewhere, it's a pain.
Have you any better solution?
server:
from twisted.web import xmlrpc, server
import numpy as np
class MioServer(xmlrpc.XMLRPC):
"""
An example object to be published.
"""
def xmlrpc_test_np(self):
return np.sqrt(2)
if __name__ == '__main__':
from twisted.internet import reactor
r = MioServer()
reactor.listenTCP(7080, server.Site(r))
reactor.run()
client:
import xmlrpclib
if __name__=='__main__':
x=xmlrpclib.ServerProxy('http://localhost:7080/')
print x.test_np()
Exception:
Traceback (most recent call last):
File "C:\Users\Stone\.eclipse\org.eclipse.platform_4.3.0_1709980481_win32_win32_x86\plugins\org.python.pydev_2.8.2.2013090511\pysrc\pydevd.py", line 1446, in <module>
debugger.run(setup['file'], None, None)
File "C:\Users\Stone\.eclipse\org.eclipse.platform_4.3.0_1709980481_win32_win32_x86\plugins\org.python.pydev_2.8.2.2013090511\pysrc\pydevd.py", line 1092, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "C:\Users\Stone\Documents\FastDose\src\Beagle\Prove e test\xmlrpc_client.py", line 28, in <module>
print x.test_np()
File "C:\Python27\lib\xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "C:\Python27\lib\xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "C:\Python27\lib\xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "C:\Python27\lib\xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "C:\Python27\lib\xmlrpclib.py", line 1473, in parse_response
return u.close()
File "C:\Python27\lib\xmlrpclib.py", line 793, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 8002: "Can't serialize output: cannot marshal <type 'numpy.float64'> objects">

This has nothing to do with twisted. If you read the error message you posted you see near the end that the error arises in xmlrpclib.py.
The xml-rpc implementation uses marshal to serialize objects. However, the marshalling done by xml-rpc does not support handling third party objects like numpy.ndarray. The reason that it works when you convert to float is that the built-in float type is supported.
Before offering my solution, I should point out that this exact same thing has been asked elsewhere in several places easily found via google (1 2), and I am stealing my answers from there.
To do what you want, you can convert your numpy array to something that can be serialized. The simplest way to do this is to write flatten/unflatten functions. You would then call then call the flattener when sending, and the unflattener when receiving. Here's an example (taken from this post):
from cStringIO import StringIO
from numpy.lib import format
def unflatten(s):
f = StringIO(s)
arr = format.read_array(f)
return arr
def flatten(arr):
f = StringIO()
format.write_array(f, arr)
s = f.getvalue()
return s
An even simpler thing to do would be to call
<the array you want to send>.tolist()
on the sending side to convert to a python list, and then call
np.array(<the list you received>)
on the receiving side.
The only drawback of doing this is that you have to explicitly call the flattener and unflattener when you send and receive data. Although this is a bit more typing, it's not much, and if you forget the program will fail loudly (with the same error you already experienced) rather than silently doing something wrong.
I gather from your question that you don't like this, and would rather find a way to make numpy arrays work directly with xml-rpc without any explicit flattening/unflattening. I think this may not be possible, because the xml-rpc documentation specifically says that the only third party objects that can be serialized are new style classes with a __dict__ attribute, and in this case the keys must be strings and the values must be other conformable types.
So you see, if you want to support numpy arrays directly, it seems that you have to modify the way xml-rpc's marshalling works. It would be nice if you could just add some kind of method to a subclass of ndarray to support being marshalled, but it looks like that's not how it works.
I hope this helps you understand what's going on, and why the existing solutions use explicit flattening/unflattening.
-Daniel
P.S. This has made me curious about how to extend xml-rpc in python to support other types of objects, so I've posted my own question about that.

I have found this way: before sending the result, I pass it to funcion to_float()
def iterable(x):
try:
iter(x)
except: # not iterable
return False
else: # iterable
return True
def to_float(x):
from numpy import float64,ndarray
if type(x) == dict:
res = dict()
for name in iter(x):
res[name] = to_float(x[name])
return res
elif type(x) == ndarray:
return map(float, x.tolist())
elif type(x) == float64:
return float(x)
elif iterable(x) and not isinstance(x,str):
res=[]
for item in x:
if type(item) == float64:
res.append(float(item))
elif type(x) == ndarray:
res.append(map(float, x.tolist()))
else:
res.append(item)
return res
else:
return x

Tried for quite some time to transfer a 640x480x3 image with xmlrpc. Neither the proposed "tolist()" nor the "flatten" solution worked for me. Finally found this solution:
Sender:
def getCamPic():
cam = cv2.VideoCapture(0)
img = cam.read()
transferImg = img.tostring()
return transferImg
Receiver:
transferImg = proxy.getCamPic()
rawImg = np.fromstring(transferImg.data,dtype=np.uint8)
img = np.reshape(rawImg,(640,480,3))
As I know my cam resolution the hard coding is not an issue for me. An improved version might have this information included in the transferImg?

using generators and cStringIO in python to stream strings

I'm trying to read a very large string stream using cStringIO in a python dictionary:
def stream_read(self, path):
try:
# create a string stream from the contents at 'path'
# note: the string at self._my_dict[path] is 7MB in size
stream = StringIO.StringIO(self._my_dict[path])
while True:
# buffer size is 128kB, or 128 * 1024
buf = stream.read(self.buffer_size)
if buf != '':
yield buf
else:
raise StopIteration
except KeyError:
raise IOError("Could not get content")
And in my test suite, I'm testing this function by first testing stream_write, asserting that the data exists at that path, and then calling stream_read:
def test_stream(self):
filename = self.gen_random_string()
# test 7MB
content = self.gen_random_string(7 * 1024 * 1024)
# test stream write
io = StringIO.StringIO(content)
self._storage.stream_write(filename, io)
io.close()
self.assertTrue(self._storage.exists(filename))
# test read / write
data = ''
for buf in self._storage.stream_read(filename):
data += buf
self.assertEqual(content, data)
Yet in my test suite, I'm catching an AssertionError:
======================================================================
FAIL: test_stream (test_swift_storage.TestSwiftStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/bacongobbler/.../test/test_local_storage.py", line 44, in test_stream
self.assertEqual(content, data)
AssertionError: '[squelched]' != '<cStringIO.StringI object at 0x3148e70>'
----------------------------------------------------------------------
Ran 28 tests in 20.495s
FAILED (failures=1)
It looks related to an issue I posted last week, but I'm still not quite sure I understand why stream is getting set to the Generator as a string in this case.
If anyone wants to take a closer look at the source code, it's all up at https://github.com/bacongobbler/docker-registry/blob/106-swift-storage/test/utils/mock_swift_storage.py

You store just the StringIO object when calling self._storage.stream_write(filename, io):
def put_content(self, path, content, chunk=None):
path = self._init_path(path)
try:
self._swift_container[path] = content
except Exception:
raise IOError("Could not put content")
where content is the io object you passed in.
Later on, you pass that file object to StringIO again:
stream = StringIO.StringIO(self.get_content(path))
This calls str() on self.get_content(path), storing the string representation of a cStringIO.StringI() instance:
>>> from cStringIO import StringIO
>>> str(StringIO('test data'))
'<cStringIO.StringI object at 0x1074ea470>'
Your reading code works fine, it is your writing mock that needs to actually take the data out of the StringIO object.
A .read() call will do here:
def put_content(self, path, content, chunk=None):
path = self._init_path(path)
try:
self._swift_container[path] = content.read()
except Exception:
raise IOError("Could not put content")