We Keep Coding

Function to read fasta files not working after update python

I got a good code to read fasta files:
from itertools import groupby
def is_header(line):
return line[0] == '>'
def parse_fasta(filename):
if filename.endswith('.gz'):
opener = lambda filename: gzip.open(filename, 'rb')
else:
opener = lambda filename: open(filename, 'r')
with opener(filename) as f:
fasta_iter = (it[1] for it in groupby(f, is_header))
for name in fasta_iter:
name = name.__next__()[1:].strip()
sequences = ''.join(seq.strip() for seq in fasta_iter.__next__())
yield name, sequences.upper()
And it worked fine until I update to Python 3.10.4, then when I try to use it I got this error:
Traceback (most recent call last):
File "/media/paulosschlogl/Paulo/pauloscchlogl/Genome_kmers/fasta_parser.py", line 21, in parse_fasta
sequences = ''.join(seq.strip() for seq in fasta_iter.__next__())
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/media/paulosschlogl/Paulo/pauloscchlogl/Genome_kmers/fasta_split_chr_pls.py", line 112, in <module>
sys.exit(main())
File "/media/paulosschlogl/Paulo/pauloscchlogl/Genome_kmers/fasta_split_chr_pls.py", line 81, in main
plasmid, chromosome = split_sequences_from_fasta_file(filename)
File "/media/paulosschlogl/Paulo/pauloscchlogl/Genome_kmers/fasta_parser.py", line 111, in split_sequences_from_fasta_file
for header, seq in parse_fasta(filename)
RuntimeError: generator raised StopIteration
I run my code in conda (conda 4.13.0) environment, and I try to debug the function but I got stucked.
I don't want to loose this function to try use Biopython.
If you guys have any idea to fixe it I really appreciate.
Thanks
Paulo
Example of fasta file:
> seq_name
AGGTAGGGA
The funny stuff is that, when I run the function in python interpret at the command line all worked fine, but when I call the functions from the script using the function imported thats when I got the errors.
>>> import gzip
>>> from itertools import groupby
>>> def is_header(line):
... return line[0] == '>'
...
>>> for name, seq in parse_fasta("/media/paulosschlogl/Paulo/pauloscchlogl/Genome_kmers/Genomes/Archaea/Asgard/Chr/GCA_008000775.1_ASM800077v1_genomic.fna.gz"):
... print(name, seq[:50])
...
CP042905.1 Candidatus Prometheoarchaeum syntrophicum strain MK-D1 chromosome, complete genome TAAATATTATAGCCCGTAATAGCAGAGTCACCAACACTTAAAGGTGCATC
>>> quit()

The exception you're getting is because you're manually calling the __next__ method on various iterators in your code. Eventually you do that on an iterator that doesn't have any values left, and you'll get a StopIteration exception raised.
In much older versions of Python, that was OK to leave uncaught in a generator function. The StopIteration exception would continue to bubble up just like any other exception. For a generator function, raising StopIteration is an expected part of its behavior (it happens automatically when the function ends, either with a return statement, or by reaching the end of its code). In Python 3.5, this behavior changed, with PEP 479 making it an error for a StopIteration to go uncaught in a generator.
Now, given the logic of your code, I'm not exactly sure why you're getting empty iterators. If the file is in the format you describe, the __next__ calls should always have a value to get, and the StopIteration that comes when there are no values left will be received by the for loop, which will suppress it (and just end the loop). Perhaps some of your files are not correctly formatted, with a header line by itself with no subsequent sequences?
In any case, you can better diagnose the issue if you catch the StopIteration and print out some diagnostic information. I'd try:
with opener(filename) as f:
fasta_iter = (it[1] for it in groupby(f, is_header))
for name in fasta_iter:
name = name.__next__()[1:].strip()
try:
sequences = ''.join(seq.strip() for seq in fasta_iter.__next__())
except StopIteration as si:
print(f'no sequence for {name}')
raise ValueError() from si
yield name, sequences.upper()
If you find that the missing sequence is a normal thing and happens at the end of every one of your files, then you could suppress the error by just putting a return statement in the except block, or maybe by using zip in your for loop (for name_iter, sequences_iter in zip(fasta_iter, fasta_iter):). I'd hesitate to jump to that as the first solution though, since it will throw away a header if there is an extra one, and silently losing data is generally a bad idea.

What does a "with" statement do without the "as"?

with dpg.window(label="Window"):
dpg.add_text("Hello, world")
dpg.add_button(label="Save")
dpg.add_input_text(label="string", default_value="Quick brown fox")
dpg.add_slider_float(label="float", default_value=0.273, max_value=1)
dpg.add_color_picker(label="Pick", default_value=(0, 0, 0))
This code runs without error (given the correct imports and setup)
dpg.window(label="Window")
dpg.add_text("Hello, world")
dpg.add_button(label="Save")
dpg.add_input_text(label="string", default_value="Quick brown fox")
dpg.add_slider_float(label="float", default_value=0.273, max_value=1)
dpg.add_color_picker(label="Pick", default_value=(0, 0, 0))
This code does not. A runtime error occurs on line 2. I do not understand how a with statement with no as affects the contents of the with. I've seen another similar post, but I can't understand how the explanation answers my question.
Here is the stacktrace for the error:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\18328\PycharmProjects\sandbox\main.py", line 10, in <module>
dpg.add_text("Hello, world")
File "C:\Users\18328\AppData\Local\Programs\Python\Python39\lib\site-packages\dearpygui\dearpygui.py", line 7019, in add_text
return internal_dpg.add_text(default_value, label=label, user_data=user_data, use_internal_label=use_internal_label, tag=tag, indent=indent, parent=parent, before=before, source=source, payload_type=payload_type, drag_callback=drag_callback, drop_callback=drop_callback, show=show, pos=pos, filter_key=filter_key, tracked=tracked, track_offset=track_offset, wrap=wrap, bullet=bullet, color=color, show_label=show_label, **kwargs)
SystemError: <built-in function add_text> returned a result with an error set

On the Python side: Context managers run completely arbitrary code on enter and exit. Python doesn't force that code to do things "behind your back", but neither does it present them from doing so; so the implementation of the specific context manager at hand needs to be inspected to know why it's modifying behavior.
From a DearPyGui perspective: The C++ library that dpg backends into maintains a "container stack". Entering a container as a context manager ensures that it's on the top of your stack. When you create a new DearPyGui object that needs to be attached to a container, it refers to that stack, and attaches itself to the thing that is currently on top.

Using the translation provided by PEP-343, you can see that
with dpg.window(label="Window"):
...
is equivalent to
mgr = dpg.window(label="Window")
exit = type(mgr.__exit__)
value = type(mgr).__enter__(mgr)
exc = True
try:
try:
...
except:
exc = False
if not exit(mgr, *sys.exc_info()):
raise
finally:
if exc:
exit(mgr, None, None, None)
You can compare the above to the fuller translation in PEP-343 to see the single line I omitted that results from the use of as.
In your examples, dpg.window(label="Window").__enter__ is called in the first example, but not in the second.

Safest way to open a file in python 3.4

I was expecting the following would work but PyDev is returning an error:
try fh = open(myFile):
logging.info("success")
except Exception as e:
logging.critical("failed because:")
logging.critical(e)
gives
Encountered "fh" at line 237, column 5. Was expecting: ":"...
I've looked around and cannot find a safe way to open a filehandle for reading in Python 3.4 and report errors properly. Can someone point me in the correct direction please?

You misplaced the :; it comes directly after try; it is better to put that on its own, separate line:
try:
fh = open(myFile)
logging.info("success")
except Exception as e:
logging.critical("failed because:")
logging.critical(e)
You placed the : after the open() call instead.
Instead of passing in e as a separate argument, you can tell logging to pick up the exception automatically:
try:
fh = open(myFile)
logging.info("success")
except Exception:
logging.critical("failed because:", exc_info=True)
and a full traceback will be included in the log. This is what the logging.exception() function does; it'll call logging.error() with exc_info set to true, producing a message at log level ERROR plus a traceback.

Searching for equivalent of FileNotFoundError in Python 2

I created a class named Options. It works fine but not not with Python 2.
And I want it to work on both Python 2 and 3.
The problem is identified: FileNotFoundError doesn t exist in Python 2.
But if I use IOError it doesn t work in Python 3
Changed in version 3.3: EnvironmentError, IOError, WindowsError, VMSError, socket.error, select.error and mmap.error have been merged into OSError.
What should I do ???(Please do not discuss my choice of portability, I have reasons.)
Here s the code:
#!/usr/bin/python
#-*-coding:utf-8*
#option_controller.py
#Walle Cyril
#25/01/2014
import json
import os
class Options():
"""Options is a class designed to read, add and change informations in a JSON file with a dictionnary in it.
The entire object works even if the file is missing since it re-creates it.
If present it must respect the JSON format: e.g. keys must be strings and so on.
If something corrupted the file, just destroy the file or call read_file method to remake it."""
def __init__(self,directory_name="Cache",file_name="options.json",imported_default_values=None):
#json file
self.option_file_path=os.path.join(directory_name,file_name)
self.directory_name=directory_name
self.file_name=file_name
#self.parameters_json_file={'sort_keys':True, 'indent':4, 'separators':(',',':')}
#the default data
if imported_default_values is None:
DEFAULT_INDENT = 2
self.default_values={\
"translate_html_level": 1,\
"indent_size":DEFAULT_INDENT,\
"document_title":"Titre"}
else:
self.default_values=imported_default_values
def read_file(self,read_this_key_only=False):
"""returns the value for the given key or a dictionary if the key is not given.
returns None if it s impossible"""
try:
text_in_file=open(self.option_file_path,'r').read()
except FileNotFoundError:#not 2.X compatible
text_in_file=""#if the file is not there we re-make one with default values
if text_in_file=="":#same if the file is empty
self.__insert_all_default_values()
text_in_file=open(self.option_file_path,'r').read()
try:
option_dict=json.loads(text_in_file)
except ValueError:
#if the json file is broken we re-make one with default values
self.__insert_all_default_values()
text_in_file=open(self.option_file_path,'r').read()
option_dict=json.loads(text_in_file)
if read_this_key_only:
if read_this_key_only in option_dict:
return option_dict[read_this_key_only]#
else:
#if the value is not there it should be written for the next time
if read_this_key_only in self.default_values:
self.add_option_to_file(read_this_key_only,self.default_values[read_this_key_only])
return self.default_values[read_this_key_only]
else:
#impossible because there is not default value so the value isn t meant to be here
return None
else:
return option_dict
def add_option_to_file(self,key,value):#or update
"""Adds or updates an option(key and value) to the json file if the option exists in the default_values of the object."""
option_dict=self.read_file()
if key in self.default_values:
option_dict[key]=value
open(self.option_file_path,'w').write(\
json.dumps(option_dict,sort_keys=True, indent=4, separators=(',',':')))
def __insert_all_default_values(self):
"""Recreate json file with default values.
called if the document is empty or non-existing or corrupted."""
try:
open(self.option_file_path,'w').write(\
json.dumps(self.default_values,sort_keys=True, indent=4, separators=(',',':')))
except FileNotFoundError:
os.mkdir(self.directory_name)#Create the directory
if os.path.isdir(self.directory_name):#succes
self.__insert_all_default_values()
else:
print("Impossible to write in %s and file %s not found" % (os.getcwd(),self.option_file_path))
#demo
if __name__ == '__main__':
option_file_object=Options()
print(option_file_object.__doc__)
print(option_file_object.read_file())
option_file_object.add_option_to_file("","test")#this should have no effect
option_file_object.add_option_to_file("translate_html_level","0")#this should have an effect
print("value of translate_html_level:",option_file_object.read_file("translate_html_level"))
print(option_file_object.read_file())

If FileNotFoundError isn't there, define it:
try:
FileNotFoundError
except NameError:
FileNotFoundError = IOError
Now you can catch FileNotFoundError in Python 2 since it's really IOError.
Be careful though, IOError has other meanings. In particular, any message should probably say "file could not be read" rather than "file not found."

You can use the base class exception EnvironmentError and use the 'errno' attribute to figure out which exception was raised:
from __future__ import print_function
import os
import errno
try:
open('no file of this name') # generate 'file not found error'
except EnvironmentError as e: # OSError or IOError...
print(os.strerror(e.errno))
Or just use IOError in the same way:
try:
open('/Users/test/Documents/test') # will be a permission error
except IOError as e:
print(os.strerror(e.errno))
That works on Python 2 or Python 3.
Be careful not to compare against number values directly, because they can be different on different platforms. Instead, use the named constants in Python's standard library errno module which will use the correct values for the run-time platform.

The Python 2 / 3 compatible way to except a FileNotFoundError is this:
import errno
try:
with open('some_file_that_does_not_exist', 'r'):
pass
except EnvironmentError as e:
if e.errno != errno.ENOENT:
raise
Other answers are close, but don't re-raise if the error number doesn't match.
Using IOError is fine for most cases, but for some reason os.listdir() and friends raise OSError instead on Python 2. Since IOError inherits from OSError it's fine to just always catch OSError and check the error number.
Edit: The previous sentence is only true on Python 3. To be cross compatible, instead catch EnvironmentError and check the error number.

For what it's worth, although the IOError is hardly mentioned in Python 3's official document and does not even showed up in its official Exception hierarchy, it is still there, and it is the parent class of FileNotFoundError in Python 3. See python3 -c "print(isinstance(FileNotFoundError(), IOError))" giving you a True. Therefore, you can technically write your code in this way, which works for both Python 2 and Python 3.
try:
content = open("somefile.txt").read()
except IOError: # Works in both Python 2 & 3
print("Oops, we can not read this file")
It might be "good enough" in many cases. Although in general, it is not recommended to rely on an undocumented behavior. So, I'm not really suggesting this approach. I personally use Kindall's answer.

AssertRaises fails even though exception is raised

I am running into the following rather strange problem:
I am developing a django app and in my models class I am defining an exception that should be raised when a validation fails:
class MissingValueException(Exception):
"""Raise when a required attribute is missing."""
def __init__(self, message):
super(MissingValueException, self).__init__()
self.message = message
def __str__(self):
return repr(self.message)
This code is called from a publication class in a validation method:
def validate_required_fields(self):
# Here is the validation code.
if all_fields_present:
return True
else:
raise MissingValueException(errors)
In my unit test I create a case where the exception should be raised:
def test_raise_exception_incomplete_publication(self):
publication = Publication(publication_type="book")
self.assertRaises(MissingValueException, publication.validate_required_fields)
This produces the following output:
======================================================================
ERROR: test_raise_exception_incomplete_publication (core_knowledge_platform.core_web_service.tests.logic_tests.BusinessLogicTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/media/data/Dokumente/Code/master_project/core_knowledge_platform/../core_knowledge_platform/core_web_service/tests/logic_tests.py", line 45, in test_raise_exception_incomplete_publication
self.assertRaises(MissingValueException, method, )
File "/usr/lib/python2.7/unittest/case.py", line 465, in assertRaises
callableObj(*args, **kwargs)
File "/media/data/Dokumente/Code/master_project/core_knowledge_platform/../core_knowledge_platform/core_web_service/models.py", line 150, in validate_required_fields
raise MissingValueException(errors)
MissingValueException: 'Publication of type book is missing field publisherPublication of type book is missing field titlePublication of type book is missing field year'
So it looks like the exception is raised (which is the case - I even checked it in an interactive IPython session), but it seems that assertRaises is not catching it.
Anyone has any idea why this might happen?
Thanks

This could happen if your tests and your product code are importing your exception class through two different paths, so asserRaises doesn't realize that the exception you got was the one you were looking for.
Look at your imports, make sure that they are the same in both places. Having the same directories available in two different ways in your PYTHONPATH can make this happen. Symbolic links in those entries can also confuse things.