EDIT: I had another random error pop up which I successfully caught in command prompt, this time pointing to line 69- segmentation fault calling whether length of a tuple in a different dictionary is equal to a number....
I have a long running (up to a week) script that I designed to test SQLlite3 insert times for different structures. Unfortunately the script intermittently crashes python without outputting error messages to the python GUI, below is the error message that windows gives in the 'python has stopped working' window;
Full error message:
Problem Event Name: APPCRASH
Application Name: pythonw.exe
Application Version: 3.5.150.1013
Application Timestamp: 55f4dccb
Fault Module Name: python35.dll
Fault Module Version: 3.5.150.1013
Fault Module Timestamp: 55f4dcbb
Exception Code: c0000005
Exception Offset: 000e800e
OS Version: 6.1.7601.2.1.0.768.3
Locale ID: 2057
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
Script I was running (warning, 1.5k lines...)
From observing what had and had not been printed I know that it was caused, or that it at least happened coincidentally at this time, with the following piece of code (starting from line 1450 on the link):
with open(r"C:\BMRA\LOG\less tbl log.csv",'a') as log:
log.write(my_date.strftime("%Y-%m-%d"))
log.write(", ")
seconds=sql_time.seconds
log.write(str(seconds))
log.write("\n")
item_collector=[]
The Log csv file appears to have written fine, so my assumption is that the error must lie with the last line.
Item_collector is a (large, ~700mb) dictionary of lists of tuples that had just been written to an sqllite3 database (the tuples containing only str, int, or floats).
As I understand it, the error refers to an application writing to memory it shouldn't and windows consequently shutting everything down to stop it messing things up. However I don't see how changing a normal vanilla python object full of other vanilla python objects should create such an error.
Does anyone have any ideas about what could underlay this, or alternatively ways to figure that out given python doesn't give an error message pointing to the specific issue? I did after a previous issue implement a logging module wrapper below around my script, but it did not catch anything
Some initial research suggested that I get a mini dump from the task manager before closing the process- I have it, but debugging hasn't been succesful- apparently I need something called python35.pdb, which as far as I can make out isn't around (for 3.5)
The script recently had a similar problem before which gave a similar error message
The advice I received was to implement the logging module around my script as so:
import logging
logging.basicConfig(filename=r'C:\BMRA\ERROR_LOG.log', level=logging.DEBUG)
try:
main()
except BaseException:
logging.getLogger(__name__).exception("Program terminated")
raise
and;
def logging_decorator(func):
def wrapper_function(self, *args, **kwargs):
logging.getLogger(__name__).debug(
"Calling %s: %r %r", func.__name__, args, kwargs)
ret = func(self, *args, **kwargs)
logging.getLogger(__name__).debug(
"%s returned %r", func.__name__, ret)
return ret
return wrapper_function
class MyConnect(sqlite3.Connection):
def cursor(self):
return super(MyConnect, self).cursor(MyCursor)
commit = logging_decorator(sqlite3.Connection.commit)
class MyCursor(sqlite3.Cursor):
execute = logging_decorator(sqlite3.Cursor.execute)
However this does not appear to have caught the error, with the script still crashing without sending any info to the designated file.
Apologies if I've not included something necessary.
After the 50th time the script crashed in a random place inexplicably, I ran the windows memory diagnostic tool.
Unfortunately it appears that my system has ram/hardware errors, which I understand would cause issues like this.
Related
So, I'm in process of learning python with libvirt module. Here is a little script that I made which checks if connection with libvirtd is successfully established and checks for one domain. I'm not developer and I'm taking some shortcuts so I don't understand how python or libvirt module works. But my real problem at this moment why is my script closing if connection is not established or domain is not found.
#!/usr/bin/env python3
from __future__ import print_function
import sys
import libvirt
domName = 'server1'
conn = libvirt.open('qemu:///system')
if conn == None:
print('Failed to open connection to qemu:///system', file=sys.stderr)
exit(1)
else:
print('Connection opened sucessfully')
dom = conn.lookupByName(domName)
if dom == None:
print('Failed to find the domain '+domName, file=sys.stderr)
exit(1)
else:
print('Domain '+domName+' was found')
conn.close()
exit(0)
For example libvirtd service is stopped and connection is not established and instead going further down the lines into if statement it just prints some errors and stops, so there is an if statement which should check for this, but like this it does not has any functionality. It looks like this
[root#localhost Documents]# ./virt.py
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Traceback (most recent call last):
File "./virt.py", line 11, in <module>
conn = libvirt.open('qemu:///system')
File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 277, in open
if ret is None:raise libvirtError('virConnectOpen() failed')
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
[root#localhost Documents]#
I managed to suppress errors but then it just the same thing but without errors. Also I found this script here.
I found this script here (...)
Well, then you've learned one first lesson: you shouldn't rely on copy-pasting code found "here" (nor in most other places actually). Actually, you can consider that about 80% of the code you'll find on the net is crap (and I'm being generous).
I'm in process of learning python
Did you do the full official Python tutorial ? If no, then that's really what you want to start with (assuming you already get the basic concepts like types, variables, conditionals, iteration, functions, etc - else you want this instead)
For example libvirtd service is stopped and connection is not established and instead going further down the lines into if statement it just prints some errors and stops
Like most modern languages, Python uses a mechanism named "exceptions" to signal errors. This is much more powerful, usable and reliable than returning error codes or special values from functions...
This is all explained in the tutorial, so I won't bother posting corrected code, just following the tutorial should be enough to let you fix this code by yourself.
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
This error message suggests that the libvirtd daemon is not actually running on this host.
Your script still needs changes though if you want to catch errors. libvirt APIs will raise exceptions when things go wrong, so instead of checking the return value against "None", you need a try/except block to catch & handle it
I am overwriting connection.notices of psycopg2.
My goal is to send a notice from a PostgreSQL trigger to my application.
In python I want to write the current stacktrace to a log file to see which python code is triggering the trigger in the database.
It works, but unfortunately I can't extract the whole stacktrace.
I only get the lines below psycopg, not the above (lines of the callers).
Here is my code:
# Overwriting connetion.notices via Django
class MyAppConfig(AppConfig):
def ready(self):
connection_created.connect(connection_created_check_for_notice_in_connection)
class ConnectionNoticeList(object):
def append(self, message):
if not 'some_magic_of_db_trigger' in message:
return
logger.warn('%s %s' % (message, ''.join(traceback.format_stack())))
def connection_created_check_for_notice_in_connection(sender, connection, **kwargs):
connection.connection.notices=ConnectionNoticeList()
I see this in the logs:
'NOTICE: some_magic_of_db_trigger: 17909
File "/snap/pycharm-community/128/helpers/pycharm/_jb_pytest_runner....ork/foo/apps.py", line 47, in append
logger.warn(\'%s %s\' % (message, \'\'.join(traceback.format_stack())))
'
traceback.format_stack() inside ConnectionNoticeList.append() extracts not the callers.
Is there a way to get the lines of the upper methods?
(This is related to an older question: Log Stacktrace of current Python Interpreter via PostgreSQL trigger)
Above code works. I see the whole traceback.
I don't know why the traceback was cut in PyCharm. In production I could see the whole traceback and I could find the broken code which modified the data in way which should not happen.
Many thanks to Daniele Varrazzo who provided the hint to overwrite connection.notices:
https://www.postgresql.org/message-id/CA%2Bmi_8a5rij_5xchPkYPFS3moAtw9i6TrwKndynWfbJ%3DkFA4fg%40mail.gmail.com
Background
I am using Python 2.7.6 to parse chunks of very large files (20+ GB) in parallel with the multiprocessing module. I have the worker processes extract information from the input file and put the results in a shelved dictionary for later processing. To prevent simultaneous writes to the pseudo-database, I am using a managed lock. I have also implemented a context manager for the database access to ensure it is always closed, because the shelve module doesn't natively support context manager functionality until Python 3.4.
The Problem
I would like to measure overall run time with the Linux time command. However, when I run the script with the time command, I get a SyntaxError exception that I don't get if I run it normally. Example code:
import multiprocessing
import shelve
from contextlib import contextmanager
DB_NAME = 'temp_db'
# manually implemented context manager - not natively implemented until Python 3.4
# I could use contextlib.closing, but this method makes the "with" statements cleaner
#contextmanager
def open_db(db_name, flag='c'):
db = shelve.open(db_name, flag=flag)
try:
yield db
finally:
db.close()
db_lock = multiprocessing.Manager().Lock()
with db_lock, open_db(DB_NAME) as db:
db['1'] = 'test_value1'
db['2'] = 1.5
with db_lock, open_db(DB_NAME) as db:
for key, val in db.iteritems():
print("{0} : {1}\n".format(key, val))
Running python test_script.py produces the expected output:
2 : 1.5
1 : test_value1
On the other hand, running time python test_script.py causes an exception:
File "test_script.py", line 21
with db_lock, open_db(DB_NAME) as db:
^
SyntaxError: invalid syntax
0.005u 0.002s 0:00.01 0.0% 0+0k 0+0io 0pf+0w
The Question
Why would the time command affect what the interpreter considers valid syntax?
Other Notes
I assume the time command is being invoked correctly because it does produce the timing information, and the presence of the exception shows that the interpreter is finding the correct script.
If I eliminate either the acquisition of the lock or the database opening, the exception disappears, so the problem appears to be caused by the comma in the with statement.
Something is causing the python executable (and version) to change. Try these commands:
which python
python -V
time which python
time python -V
For the overall project, consider having each worker just return data to the parent, which then stores info in a file or database. This simplifies the code because you don't need locking -- only the parent has access.
I am trying to have a Gtk.Application which stays unique and handles opening files. I am using python 2.7.3 on Ubuntu 12.04 with Gtk3 (fairly new to both python and Gtk)
The application runs fine without parameters, but it fails to get the file list when I run it trying to open a file. Here is the code, as minimalistic as I could make it:
#!/usr/bin/env python
import sys
from gi.repository import Gtk, Gio
def do_open(app, files, *hint):
print(app)
print(files)
print(hint)
def do_activate(app):
print "activate"
test = Gtk.Application(application_id="a.b", flags=Gio.ApplicationFlags.HANDLES_OPEN)
test.set_inactivity_timeout(10000)
test.connect("open", do_open)
test.connect("activate", do_activate)
test.run(sys.argv)
When I run the program without arguments it just prints "activate", which is fine. When I run it with a parameter (like ./test.py test.py) I get the following:
/usr/lib/python2.7/dist-packages/gi/types.py:43: Warning: g_value_get_boxed: assertion `G_VALUE_HOLDS_BOXED (value)' failed
return info.invoke(*args, **kwargs)
<Application object at 0x1c75230 (GtkApplication at 0x1cba0b0)>
[]
(1, '')
Does anyone understand why that assertion is failing and why I am getting an empty list of files?
As common as this task appears to be, I couldn't find any working example online either.
There is a bug in PyGObject. It is already reported in GNOME Bugzilla, check the bug report titled "Does not handle GFile in signal arguments".
Update: The bug was fixed in 2013. No more assertion and it returns the list of files (GFiles). In other words, the code works as expected (at least using 3.14):
$ python test.py test.py
<Application object at 0x7fcaf18f5d20 (GtkApplication at 0x11892b0)>
[<__main__.GLocalFile object at 0x7fcaf18a6050 (GLocalFile at 0x11aea00)>]
(1, '')
I create a file called foo_module.py containing the following code:
import shelve, whichdb, os
from foo_package.g import g
g.shelf = shelve.open("foo_path")
g.shelf.close()
print whichdb.whichdb("foo_path") # => dbhash
os.remove("foo_path")
Next to that file I create a directory called foo_package than contains an empty __init__.py file and a file called g.py that just contains:
class g:
pass
Now when I run foo_module.py I get a weird error message:
Exception TypeError: "'NoneType' object is not callable" in ignored
But then, if I rename the directory from foo_package to foo, and change the import line in foo_module.py, I don't get any error. Wtf is going on here?
Running Python 2.6.4 on WinXP.
I think you've hit a minor bug in 2.6.4's code related to the cleanup at end of program. If you run python -v you can see exactly at what point of the cleanup the error comes:
# cleanup[1] foo_package.g
Exception TypeError: "'NoneType' object is not callable" in ignored
Python sets references to None during the cleanup at the end of program, and it looks like it's getting confused about the status of g.shelf. As a workaround you could set g.shelf = None after the close. I would also recommend opening a bug in Python's bug tracker!
After days of hair loss, I finally had success using an atexit function:
import atexit
...
cache = shelve.open(path)
atexit.register(cache.close)
It's most appropriate to register right after opening. This works with multiple concurrent shelves.
(python 2.6.5 on lucid)
This is indeed a Python bug, and I've posted a patch to the tracker issue you opened (thanks for doing that).
The problem is that shelve's del method calls its close method, but if the shelve module has already been through cleanup, the close method fails with the message you see.
You can avoid the message in your code by adding 'del g.shelf' after g.shelf.close. As long as g.shelf is the only reference to the shelf, this will result in CPython calling the shelve's del method right away, before the interpreter cleanup phase, and thus avoid the error message.
It seems to be an exception in a shutdown function registered by the shelve module. The "ignored" part is from the shutdown system, and might get its wording improved sometime, per Issue 6294. I'm still hoping for an answer on how to eliminate the exception itself, though...
for me a simple shelve.close() on an unclosed one did the job.
shelve.open('somefile') returns a "persistent dictionary for reading and writing" object which i used throughout the app's runtime.
when I terminated the app I received the "TypeError" Exception as mentioned.
I plased a 'close()' call in my termination sequence and that seemed to fix the problem.
e.g.
shelveObj = shelve.open('fileName')
...
shelveObj.close()
OverByThere commented on Jul 17, 2018
This seems to be fixable.
In short open /usr/lib/python3.5/weakref.py and change line 109 to:
def remove(wr, selfref=ref(self), _atomic_removal=_remove_dead_weakref):
And line 117 to:
_atomic_removal(d, wr.key)
Note you need to do this with spaces, not tabs as this will cause other errors.