I am trying to write a plugin for nose that produces and displays additional debug information for certain kinds of exceptions in a GUI. The reason I want to do this in a plugin is because I want the GUI to be launched only when the --enable-gui option has been given, and plugins are the only way to add command-line options to the nose runner.
According to the documentation, I need to override addFailure(step, err) and addError(step, err), and they say that err is the sys.exc_info() tuple:
http://nose.readthedocs.org/en/latest/plugins/interface.html
Unfortunately, I'm getting something else entirely: The exception is replaced with the string value representing it. Here's my code:
def addError(self, test, err):
info = ', '.join((type(x).__name__) for x in err)
open('/tmp/xxxxx', 'a').write(info + '\n')
def addFailure(self, test, err):
info = ', '.join((type(x).__name__) for x in err)
open('/tmp/xxxxx', 'a').write(info + '\n')
Here's the output:
type, str, traceback
type, str, traceback
So, instead of exc_type, exc_value, exc_tb, I'm getting exc_type, str(exc_value), exc_tb.
Here's the stack of the call to my overriden methods:
Traceback (most recent call last):
File "runtests.py", line 6, in <module>
nose.main(module=tests, addplugins=tests.plugins.get_all_plugins())
File "/usr/lib/python2.7/dist-packages/nose/core.py", line 118, in __init__
**extra_args)
File "/usr/lib/python2.7/unittest/main.py", line 95, in __init__
self.runTests()
File "/usr/lib/python2.7/dist-packages/nose/core.py", line 197, in runTests
result = self.testRunner.run(self.test)
File "/usr/lib/python2.7/dist-packages/nose/core.py", line 61, in run
test(result)
File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 176, in __call__
return self.run(*arg, **kw)
File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 223, in run
test(orig)
File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 176, in __call__
return self.run(*arg, **kw)
File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 223, in run
test(orig)
File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 176, in __call__
return self.run(*arg, **kw)
File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 223, in run
test(orig)
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 45, in __call__
return self.run(*arg, **kwarg)
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 138, in run
result.addError(self, err)
File "/usr/lib/python2.7/dist-packages/nose/proxy.py", line 134, in addError
plugins.addError(self.test, err)
File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 94, in __call__
return self.call(*arg, **kw)
File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 162, in simple
result = meth(*arg, **kw)
File "<snip>/plugins.py", line 31, in addError
open('/tmp/xxxxx', 'a').write(info + '\n')
I can't extract the exception from sys.exc_info(), because it has already been replaced with another one (in particular, the UnicodeEncodeError caught during __str__ of the raised exception).
Is there any way to extract exc_value from somewhere, say, the traceback?
A potential workaround: I know I can make my plugin a global variable, and instead of handling exceptions in it, I can directly send the information to the plugin itself. Unfortunately, that's not a very clean solution, so I'd like to avoid it.
Why I need a GUI: The error that I'm getting is coloured HTML traceback created by twisted, which is unreadable in the console whether I'm printing the HTML or using the html2text representation.
The err tuple is actually: (exception type, actual exception, traceback). You should be able to access the information you need from either the exception or the traceback.
Note that in your code you write everything to a string:
info = ', '.join((type(x).__name__) for x in err)
This means it will indeed be cast to a string, which, if I understand correctly, is what you are complaining about...
Related
I really don't know how to help myself, being unfamiliar with this kind of error, and not finding anything on the Google landscape really. My last hope is one of you guys since I don't know where else to go with this. I tried reinstalling all libraries and setting up a new venv. For more action I don't trust myself enough in these kinds of things.
The code triggering the error:
from wetterdienst import DWDObservationData
observations_daily = DWDObservationData(
station_ids=station_ids_d,
parameter=params_daily,
time_resolution=TimeResolution.DAILY,
start_date="2015-01-01",
end_date="2020-10-10",
tidy_data=True,
humanize_column_names=True,
)
for df in observations_hourly.collect_data():
name = str(df.STATION_ID.iloc[0]).strip(".0")
df.to_csv('./data/hourly/{}.csv'.format(name))
print('{} done'.format(name))
API is found here: https://github.com/earthobservations/wetterdienst
Error:
Traceback (most recent call last):
File "/Users/sashakaun/PycharmProjects/wetter2.0/main.py", line 83, in <module>
for df in observations_hourly.collect_data():
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/api.py", line 178, in collect_data
df_parameter = self._collect_parameter_from_station(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/api.py", line 243, in _collect_parameter_from_station
df_period = collect_climate_observations_data(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 82, in collect_climate_observations_data
filenames_and_files = download_climate_observations_data_parallel(remote_files)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 106, in download_climate_observations_data_parallel
return list(zip(remote_files, files_in_bytes))
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
yield fs.pop().result()
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 124, in _download_climate_observations_data
return BytesIO(__download_climate_observations_data(remote_file=remote_file))
File "<decorator-gen-2>", line 2, in __download_climate_observations_data
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 1356, in get_or_create_for_user_func
return self.get_or_create(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 954, in get_or_create
with Lock(
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/lock.py", line 185, in __enter__
return self._enter()
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/lock.py", line 94, in _enter
generated = self._enter_create(value, createdtime)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/lock.py", line 178, in _enter_create
return self.creator()
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 920, in gen_value
self.backend.set(key, value)
File "/Users/sashakaun/PycharmProjects/wetter2.0/venv/lib/python3.8/site-packages/dogpile/cache/backends/file.py", line 239, in set
dbm[key] = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
_gdbm.error: Database needs recovery
Thanks a lot!!
A GDBM file has been corrupted. You need to use gdbmtool to recover the database. Install gdbmtool then run
gdbmtool FILENAME
Where FILENAME is the name of the GDBM database. A prompt will appear, then you can enter
gdbmtool> recover summary
If the database can be recovered it will display a summary of the recovery results, eg:
Recovery succeeded.
Keys recovered: 6870650, failed: 5, duplicate: 0
Buckets recovered: 64830, failed: 2
I am trying to check if a certain dataset exists in bigquery using the Google Api Client in Python. It always worked untill the last update where I got this strange error I don't know how to fix:
Traceback (most recent call last):
File "/root/miniconda/lib/python2.7/site-packages/dsUtils/bq_utils.py", line 106, in _get
resp = bq_service.datasets().get(projectId=self.project_id, datasetId=self.id).execute(num_retries=2)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 598, in new_request
self._refresh(request_orig)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 864, in _refresh
self._do_refresh_request(http_request)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 891, in _do_refresh_request
body = self._generate_refresh_request_body()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 1597, in _generate_refresh_req
uest_body
assertion = self._generate_assertion()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/service_account.py", line 263, in _generate_ass
ertion
key_id=self._private_key_id)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/crypt.py", line 97, in make_signed_jwt
signature = signer.sign(signing_input)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/_pycrypto_crypt.py", line 101, in sign
return PKCS1_v1_5.new(self._key).sign(SHA256.new(message))
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Signature/PKCS1_v1_5.py", line 112, in sign
m = self._key.decrypt(em)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 174, in decrypt
return pubkey.pubkey.decrypt(self, ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
plaintext=self._decrypt(ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 235, in _decrypt
r = getRandomRange(1, self.key.n-1, randfunc=self._randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 123, in getRandomRange
value = getRandomInteger(bits, randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 104, in getRandomInteger
S = randfunc(N>>3)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 202, in read
return self._singleton.read(bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 178, in read
return _UserFriendlyRNG.read(self, bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 137, in read
self._check_pid()
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 153, in _check_pid
raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()
Is someone understanding what is hapening?
Note that I also get this error with other bricks like GCStorage.
Note also that I use the following command to load my Google credentials:
from oauth2client.client import GoogleCredentials
def get_credentials(credentials_path): #my json credentials path
logger.info('Getting credentials...')
try:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
credentials = GoogleCredentials.get_application_default()
return credentials
except Exception as e:
raise e
So if anyone know a better way to load my google credentials using my json service account file, and which would avoid the error, please tell me.
It looks like the error is in the PyCrypto module, which appears to be used under the hood by Google's OAuth2 implementation. If your code is calling os.fork() at some point, you may need to call Crypto.Random.atfork() afterward in both the parent and child process in order to update the module's internal state.
See here for PyCrypto docs; search for "atfork" for more info:
https://github.com/dlitz/pycrypto
This question and answer might also be relevant:
PyCrypto : AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
I'm trying to set up a regular expression for grabbing variables from a custom server log. The variables are grabbed by naming each portion of a regular expression. Here's an example:
/^(?<time>[^ ]+) (?<host>[^ ]+) (?<process>[^:]+): (?<message>((?<key>[^ :]+)[ :])? ?((to|from)=<(?<address>[^>]+)>)?.*)$/
I want to do the same for a custom format. The format is as follows:
[11/Jul/2014 19:35:38] ERROR [django.request.tastypie:273] Internal Server Error: /v1/notes/
Traceback (most recent call last):
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 195, in wrapper
response = callback(request, *args, **kwargs)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 426, in dispatch_list
return self.dispatch('list', request, **kwargs)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 458, in dispatch
response = method(request, **kwargs)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 1320, in post_list
updated_bundle = self.obj_create(bundle, **self.remove_api_resource_names(kwargs))
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 2084, in obj_create
return self.save(bundle)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 2230, in save
bundle.obj.save()
File "./notification/models.py", line 193, in save
handler.handle_notification()
File "./notification/handler.py", line 31, in handle
getattr(self, '_set_{}_payload'.format(preference))()
File "./notification/notification_handler.py", line 89, in _set_payload
raise Exception(error)
All I care about is grabbing the various components on the first line, putting them into variables, and then putting the entire traceback in a variable.
I realize this may be too specific of a question, but I feel a basic run down of the regex that would cover this would be beneficial to many people trying to parse server logs.
You mean something along these lines?
\[(?P<timestamp>.*?)\] (?P<level>\w+) \[(?P<location>.*?)\] (?P<message>.*?)\n(?P<details>.*?)(?=\n\[|$)
See http://regex101.com/r/zS1fN5/1
Works for the given example, but depends on what quirks and exceptions there can be to this log format.
Can anyone help me with this strange error I'm getting when adding a new document to a Whoosh index?
Here's the code:
def add_to_index(self, doc):
ix = index.open_dir(self.index_dir)
writer = AsyncWriter(ix) # use async writer to prevent write lock errors
writer.add_document(**self.get_doc_args(doc))
writer.commit()
def get_doc_args(self, doc):
return {
'id': u""+str(doc['id']),
'org': doc['org__id'],
'created': doc['created_date'],
'date': doc['received_date'],
'from_addr': doc['from_addr'],
'subject': doc['subject'],
'body': doc['messagebody__cleaned_message']
}
I get the following error:
TypeError('ord() expected a character, but string of length 0 found',)
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/celery/execute/trace.py", line 36, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/usr/local/lib/python2.6/dist-packages/celery/app/task/__init__.py", line 232, in __call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/celery/app/__init__.py", line 172, in run
return fun(*args, **kwargs)
File "/mnt/deploy/prod/chorus/src/chorus/../chorus/search/__init__.py", line 131, in index_message
MessageSearcher().add_to_index(message)
File "/mnt/deploy/prod/chorus/src/chorus/../chorus/search/__init__.py", line 29, in add_to_index
writer.commit()
File "/usr/local/lib/python2.6/dist-packages/whoosh/writing.py", line 423, in commit
self.writer.commit(*args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filewriting.py", line 501, in commit
new_segments = mergetype(self, self.segments)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filewriting.py", line 78, in MERGE_SMALL
reader = SegmentReader(writer.storage, writer.schema, seg)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filereading.py", line 63, in __init__
self.termsindex = TermIndexReader(tf)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 590, in __init__
super(TermIndexReader, self).__init__(dbfile)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 502, in __init__
OrderedHashReader.__init__(self, dbfile)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 379, in __init__
HashReader.__init__(self, dbfile)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 187, in __init__
self.hashtype = dbfile.read_byte()
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/structfile.py", line 219, in read_byte
return ord(self.file.read(1))
Strangely, the exact same code using a standard writer (i.e. not AsyncWriter) works just fine. What am I missing here? Note that in production I have to use AsyncWriter in order to avoid LockErrors.
This error is caused by some kind of index corruption. In my case the machine crashed by another reason while index was being rebuild.
You can easily solve it by deleting whoosh_index folder contents completely and rebuilidng index.
Ended up finding a solution; it's called Solr :-)
I'm trying out Dowser, it's really cool but I'm stuck with a little problem, and I couldn't find anything helpful on Google, so here I am.. ^^;
I'm running a CherryPy+SQLAlchemy application.. It works normally, except that when I enable Dowser (that is, after I call dowser.Root()), now and then I get exceptions like:
SystemError:
../Objects/tupleobject.c:809: bad
argument to internal function
on innocent-looking istructions, such as accessing a SQLA-mapped field.. Relevant part of a traceback:
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/attributes.py", line 158, in __get__
return self.impl.get(instance_state(instance), instance_dict(instance))
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/attributes.py", line 377, in get
value = callable_()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/strategies.py", line 586, in __call__
result = q.all()
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/query.py", line 1267, in all
return list(self)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/query.py", line 1422, in instances
rows = [process[0](context, row) for row in fetch]
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/query.py", line 2032, in main
return _instance(row, None)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/mapper.py", line 1653, in _instance
identitykey = identity_key(row)
File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/mapper.py", line 1594, in identity_key
return (identity_class, tuple(row[column] for column in pk_cols))
Could this be related to the Dowser thread, accessing the garbage collector? Any hint on what I could check out?
I'm running Python 2.6.2 on Xubuntu Jaunty.
Thanks for your attention!