So, I'm in process of learning python with libvirt module. Here is a little script that I made which checks if connection with libvirtd is successfully established and checks for one domain. I'm not developer and I'm taking some shortcuts so I don't understand how python or libvirt module works. But my real problem at this moment why is my script closing if connection is not established or domain is not found.
#!/usr/bin/env python3
from __future__ import print_function
import sys
import libvirt
domName = 'server1'
conn = libvirt.open('qemu:///system')
if conn == None:
print('Failed to open connection to qemu:///system', file=sys.stderr)
exit(1)
else:
print('Connection opened sucessfully')
dom = conn.lookupByName(domName)
if dom == None:
print('Failed to find the domain '+domName, file=sys.stderr)
exit(1)
else:
print('Domain '+domName+' was found')
conn.close()
exit(0)
For example libvirtd service is stopped and connection is not established and instead going further down the lines into if statement it just prints some errors and stops, so there is an if statement which should check for this, but like this it does not has any functionality. It looks like this
[root#localhost Documents]# ./virt.py
libvirt: XML-RPC error : Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Traceback (most recent call last):
File "./virt.py", line 11, in <module>
conn = libvirt.open('qemu:///system')
File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 277, in open
if ret is None:raise libvirtError('virConnectOpen() failed')
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
[root#localhost Documents]#
I managed to suppress errors but then it just the same thing but without errors. Also I found this script here.
I found this script here (...)
Well, then you've learned one first lesson: you shouldn't rely on copy-pasting code found "here" (nor in most other places actually). Actually, you can consider that about 80% of the code you'll find on the net is crap (and I'm being generous).
I'm in process of learning python
Did you do the full official Python tutorial ? If no, then that's really what you want to start with (assuming you already get the basic concepts like types, variables, conditionals, iteration, functions, etc - else you want this instead)
For example libvirtd service is stopped and connection is not established and instead going further down the lines into if statement it just prints some errors and stops
Like most modern languages, Python uses a mechanism named "exceptions" to signal errors. This is much more powerful, usable and reliable than returning error codes or special values from functions...
This is all explained in the tutorial, so I won't bother posting corrected code, just following the tutorial should be enough to let you fix this code by yourself.
libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
This error message suggests that the libvirtd daemon is not actually running on this host.
Your script still needs changes though if you want to catch errors. libvirt APIs will raise exceptions when things go wrong, so instead of checking the return value against "None", you need a try/except block to catch & handle it
Related
I have an application that is time sensitive to the time on a remote server - it needs to be run at a particular instant. Looping and retrieving the server time continuously does not get me as close as i'd like. I am trying to retrieve the server time and update my system's time (Python 3.7, windows 10) in advance so the module can restart as close as possible to the server's time. I have found the SetSysemTime function in win32api.
Here is the two line module i created to test various approaches:
import win32api
win32api.SetSystemTime(2020,9,1,21,9,10,10,0)
When i run this i get the following error:
Traceback (most recent call last):
File "C:\Users\pinev\AppData\Local\Programs\Python\Python37\Projects\Threading Tests\settime.py", line 4, in
win32api.SetSystemTime(2020,9,1,21,9,10,10,0)
pywintypes.error: (1314, 'SetSystemTime', 'A required privilege is not held by the client.')
So it seems i have the right solution, but somehow the module's privilege needs to be changed in order for Win10 to process the request. I have researched ways to set the privilege in a Python module and can't seem to find anything that works. If anyone could could provide a solution or a reference it would be greatly appreciated.
Thanks in advance.
If you just need to elevate privileges to run Python programs, you can launch the program as an administrator.
This is also the simplest and most direct method I think.
Test code:
import win32api
import ctypes, sys
def is_admin():
try:
return ctypes.windll.shell32.IsUserAnAdmin()
except:
return False
if is_admin():
win32api.SetSystemTime(2020,9,1,21,9,10,10,0)
else:
# Re-run the program with admin rights
ctypes.windll.shell32.ShellExecuteW(None, "runas", sys.executable, __file__, None, 1)
EDIT: I had another random error pop up which I successfully caught in command prompt, this time pointing to line 69- segmentation fault calling whether length of a tuple in a different dictionary is equal to a number....
I have a long running (up to a week) script that I designed to test SQLlite3 insert times for different structures. Unfortunately the script intermittently crashes python without outputting error messages to the python GUI, below is the error message that windows gives in the 'python has stopped working' window;
Full error message:
Problem Event Name: APPCRASH
Application Name: pythonw.exe
Application Version: 3.5.150.1013
Application Timestamp: 55f4dccb
Fault Module Name: python35.dll
Fault Module Version: 3.5.150.1013
Fault Module Timestamp: 55f4dcbb
Exception Code: c0000005
Exception Offset: 000e800e
OS Version: 6.1.7601.2.1.0.768.3
Locale ID: 2057
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
Script I was running (warning, 1.5k lines...)
From observing what had and had not been printed I know that it was caused, or that it at least happened coincidentally at this time, with the following piece of code (starting from line 1450 on the link):
with open(r"C:\BMRA\LOG\less tbl log.csv",'a') as log:
log.write(my_date.strftime("%Y-%m-%d"))
log.write(", ")
seconds=sql_time.seconds
log.write(str(seconds))
log.write("\n")
item_collector=[]
The Log csv file appears to have written fine, so my assumption is that the error must lie with the last line.
Item_collector is a (large, ~700mb) dictionary of lists of tuples that had just been written to an sqllite3 database (the tuples containing only str, int, or floats).
As I understand it, the error refers to an application writing to memory it shouldn't and windows consequently shutting everything down to stop it messing things up. However I don't see how changing a normal vanilla python object full of other vanilla python objects should create such an error.
Does anyone have any ideas about what could underlay this, or alternatively ways to figure that out given python doesn't give an error message pointing to the specific issue? I did after a previous issue implement a logging module wrapper below around my script, but it did not catch anything
Some initial research suggested that I get a mini dump from the task manager before closing the process- I have it, but debugging hasn't been succesful- apparently I need something called python35.pdb, which as far as I can make out isn't around (for 3.5)
The script recently had a similar problem before which gave a similar error message
The advice I received was to implement the logging module around my script as so:
import logging
logging.basicConfig(filename=r'C:\BMRA\ERROR_LOG.log', level=logging.DEBUG)
try:
main()
except BaseException:
logging.getLogger(__name__).exception("Program terminated")
raise
and;
def logging_decorator(func):
def wrapper_function(self, *args, **kwargs):
logging.getLogger(__name__).debug(
"Calling %s: %r %r", func.__name__, args, kwargs)
ret = func(self, *args, **kwargs)
logging.getLogger(__name__).debug(
"%s returned %r", func.__name__, ret)
return ret
return wrapper_function
class MyConnect(sqlite3.Connection):
def cursor(self):
return super(MyConnect, self).cursor(MyCursor)
commit = logging_decorator(sqlite3.Connection.commit)
class MyCursor(sqlite3.Cursor):
execute = logging_decorator(sqlite3.Cursor.execute)
However this does not appear to have caught the error, with the script still crashing without sending any info to the designated file.
Apologies if I've not included something necessary.
After the 50th time the script crashed in a random place inexplicably, I ran the windows memory diagnostic tool.
Unfortunately it appears that my system has ram/hardware errors, which I understand would cause issues like this.
The following code causes Java to segfault:
import os.path
import neo4j
from paste import httpserver, fileapp
import tempfile
from webob.dec import wsgify
from webob import Response, Request
HOST = '127.0.0.1'
PORT = 8080
class DebugApp(object):
#wsgify
def __call__(self, req):
# db = neo4j.GraphDatabase(tempfile.mkdtemp())
db = neo4j.GraphDatabase(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'data'))
return Response(body='it worked')
def main():
app = DebugApp()
httpserver.serve(app, host=HOST, port=PORT)
if __name__ == '__main__':
main()
To reproduce, first save that code into a file (say, app.py), and then run python app.py. Then try http://localhost:8080 in your browser; you should see the Java crash handler.
The top of the Java stack trace looks like this:
Stack: [0xb42e7000,0xb4ae8000], sp=0xb4ae44f0, free space=8181k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [_jpype.so+0x26497] JPJavaEnv::NewObjectA(_jclass*, _jmethodID*, jvalue*)+0x37
C [_jpype.so+0x3c0e8] JPMethodOverload::invokeConstructor(_jclass*, std::vector<HostRef*, std::allocator<HostRef*> >&)+0x178
C [_jpype.so+0x3a417] JPMethod::invokeConstructor(std::vector<HostRef*, std::allocator<HostRef*> >&)+0x47
C [_jpype.so+0x1beba] JPClass::newInstance(std::vector<HostRef*, std::allocator<HostRef*> >&)+0x2a
C [_jpype.so+0x67b9c] PyJPClass::newClassInstance(_object*, _object*)+0xfc
C [python+0x96822] PyEval_EvalFrameEx+0x4332
C [python+0x991e7] PyEval_EvalCodeEx+0x127
I believe that's neo4j.GraphDatabase in Python triggering JPype to go looking for EmbeddedGraphDatabase in neo4j, under Java.
Running this code in an interactive Python session doesn't segfault:
>>> import webob
>>> import app
>>> debug_app = app.DebugApp()
>>> response = debug_app(webob.Request.blank('/'))
>>> response.body
'it worked'
Presumably that's because I'm avoiding Paste altogether in that example. Perhaps this has something to do with Paste's use of threads getting in the way of neo4j? I noted a somewhat similar problem in the neo4j forums: http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-CPython-Pylons-and-threading-td942435.html
...but that only occurs on shutdown.
The issue is not with Paste per se, but with the neo4j Python bindings, which use JPype. Paste creates threads to handle incoming requests; neo4j is supposed to be thread-safe, but JPype comes with this caveat from the documentation (1):
"For the most part, python threads based on OS level threads (i.e posix threads), will work without problem. The only thing to remember is to call jpype.attachThreadToJVM() in the thread body to make the JVM usable from that thread. For threads that you do not start yourself, you can call isThreadAttachedToJVM() to check."
I couldn't find the code that does this, but I think that some of the Java code in the neo4j bindings may call attachThreadToJVM at import time. If so, when a request is handed to a worker thread by paste, and that thread then goes to fetch data from neo4j, it is crossing thread boundaries, and the JVM attachment rule may not be satisfied.
You can avoid the crash by only running import neo4j from within a single thread. In the case above, this is the callable targeted by threading.Thread.
Unfortunately, this means that even though neo4j is thread-safe, it must be constrained to a single thread when used from Python. But that's not too disappointing, considering.
Update: the maintainers responded(2) and investigated the problem, and checked in a fix. I don't know which release of neo4j this was available in, and I can no longer find the commit to their github repo(3), so this stands for re-testing.
I create a file called foo_module.py containing the following code:
import shelve, whichdb, os
from foo_package.g import g
g.shelf = shelve.open("foo_path")
g.shelf.close()
print whichdb.whichdb("foo_path") # => dbhash
os.remove("foo_path")
Next to that file I create a directory called foo_package than contains an empty __init__.py file and a file called g.py that just contains:
class g:
pass
Now when I run foo_module.py I get a weird error message:
Exception TypeError: "'NoneType' object is not callable" in ignored
But then, if I rename the directory from foo_package to foo, and change the import line in foo_module.py, I don't get any error. Wtf is going on here?
Running Python 2.6.4 on WinXP.
I think you've hit a minor bug in 2.6.4's code related to the cleanup at end of program. If you run python -v you can see exactly at what point of the cleanup the error comes:
# cleanup[1] foo_package.g
Exception TypeError: "'NoneType' object is not callable" in ignored
Python sets references to None during the cleanup at the end of program, and it looks like it's getting confused about the status of g.shelf. As a workaround you could set g.shelf = None after the close. I would also recommend opening a bug in Python's bug tracker!
After days of hair loss, I finally had success using an atexit function:
import atexit
...
cache = shelve.open(path)
atexit.register(cache.close)
It's most appropriate to register right after opening. This works with multiple concurrent shelves.
(python 2.6.5 on lucid)
This is indeed a Python bug, and I've posted a patch to the tracker issue you opened (thanks for doing that).
The problem is that shelve's del method calls its close method, but if the shelve module has already been through cleanup, the close method fails with the message you see.
You can avoid the message in your code by adding 'del g.shelf' after g.shelf.close. As long as g.shelf is the only reference to the shelf, this will result in CPython calling the shelve's del method right away, before the interpreter cleanup phase, and thus avoid the error message.
It seems to be an exception in a shutdown function registered by the shelve module. The "ignored" part is from the shutdown system, and might get its wording improved sometime, per Issue 6294. I'm still hoping for an answer on how to eliminate the exception itself, though...
for me a simple shelve.close() on an unclosed one did the job.
shelve.open('somefile') returns a "persistent dictionary for reading and writing" object which i used throughout the app's runtime.
when I terminated the app I received the "TypeError" Exception as mentioned.
I plased a 'close()' call in my termination sequence and that seemed to fix the problem.
e.g.
shelveObj = shelve.open('fileName')
...
shelveObj.close()
OverByThere commented on Jul 17, 2018
This seems to be fixable.
In short open /usr/lib/python3.5/weakref.py and change line 109 to:
def remove(wr, selfref=ref(self), _atomic_removal=_remove_dead_weakref):
And line 117 to:
_atomic_removal(d, wr.key)
Note you need to do this with spaces, not tabs as this will cause other errors.
I'm having an interesting problem with threads and the tempfile module in Python. Something doesn't appear to be getting cleaned up until the threads exit, and I'm running against an open file limit. (This is on OS X 10.5.8, Python 2.5.1.)
Yet if I sort of replicate what the tempfile module is doing (not all the security checks, but just generating a file descriptor and then using os.fdopen to produce a file object) I have no problems.
Before filing this as a bug with Python, I figured I'd check here, as it's much more likely that I'm doing something subtly wrong. But if I am, a day of trying to figure it out hasn't gotten me anywhere.
#!/usr/bin/python
import threading
import thread
import tempfile
import os
import time
import sys
NUM_THREADS = 10000
def worker_tempfile():
tempfd, tempfn = tempfile.mkstemp()
tempobj = os.fdopen(tempfd, 'wb')
tempobj.write('hello, world')
tempobj.close()
os.remove(tempfn)
time.sleep(10)
def worker_notempfile(index):
tempfn = str(index) + '.txt'
# The values I'm passing os.open may be different than tempfile.mkstemp
# uses, but it works this way as does using the open() function to create
# a file object directly.
tempfd = os.open(tempfn,
os.O_EXCL | os.O_CREAT | os.O_TRUNC | os.O_RDWR)
tempobj = os.fdopen(tempfd, 'wb')
tempobj.write('hello, world')
tempobj.close()
os.remove(tempfn)
time.sleep(10)
def main():
for count in range(NUM_THREADS):
if count % 100 == 0:
print('Opening thread %s' % count)
wthread = threading.Thread(target=worker_tempfile)
#wthread = threading.Thread(target=worker_notempfile, args=(count,))
started = False
while not started:
try:
wthread.start()
started = True
except thread.error:
print('failed starting thread %s; sleeping' % count)
time.sleep(3)
if __name__ == '__main__':
main()
If I run it with the worker_notempfile line active and the worker_tempfile line commented-out, it runs to completion.
The other way around (using worker_tempfile) I get the following error:
$ python threadtempfiletest.py
Opening thread 0
Opening thread 100
Opening thread 200
Opening thread 300
Exception in thread Thread-301:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 460, in __bootstrap
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 440, in run
File "threadtempfiletest.py", line 17, in worker_tempfile
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tempfile.py", line 302, in mkstemp
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tempfile.py", line 236, in _mkstemp_inner
OSError: [Errno 24] Too many open files: '/var/folders/4L/4LtD6bCvEoipksvnAcJ2Ok+++Tk/-Tmp-/tmpJ6wjV0'
Any ideas what I'm doing wrong? Is this a bug in Python, or am I being bone-headed?
UPDATE 2009-12-14:
I think I've found the answer, but I don't like it. Since nobody was able to replicate the problem, I went hunting around our office for machines. It passed on everything except my machine. I tested on a Mac with the same software versions I was using. I even went hunting for a Desktop G5 with the EXACT same hardware and software config I had -- same result. Both tests (with tempfile and without tempfile) succeeded on everything.
For kicks, I downloaded Python 2.6.4, and tried it on my desktop, and same pattern on my system as Python 2.5.1: tempfile failed, and notempfile succeeded.
This is leading me to the conclusion that something's hosed on my Mac, but I sure can't figure out what. Any suggestions are welcome.
I am unable to reproduce the problem with (Apple's own build of) Python 2.5.1 on Mac OS X 10.5.9 -- runs to completion just fine!
I've tried both on a Macbook Pro, i.e., an Intel processor, and an old PowerMac, i.e., a PPC processor.
So I can only imagine there must have been a bug in 10.5.8 which I never noticed (don't have any 10.5.8 around to test, as I always upgrade promptly whenever software update offers it). All I can suggest is that you try upgrading to 10.5.9 and see if the bug disappears -- if it doesn't, I have no idea how this behavior difference between my machines and yours is possible.
I think your answer can be found here. You have to explicitly os.close() the file descriptor given as the first part of the tuple that mkstemp gives you.
Edit: no, the OP is already doing what is supposed to be done. I'm leaving the answer up for the nice link.
I just tested your code on my Ubuntu Linux computer here, and it worked perfectly for me.
I have one suggestion for you to try. I don't know that it will help but it can't hurt. Rewrite your code to use with:
from __future__ import with_statement
def worker_tempfile():
tempfd, tempfn = tempfile.mkstemp()
with os.fdopen(tempfd, 'wb') as tempobj:
tempobj.write('hello, world')
os.remove(tempfn)
time.sleep(10)
The with statement is supposed to make sure that the file object gets closed no matter what. Perhaps it might help?
Good luck. Great job on the question, by the way.
Why do you think the error is not genuine? You are launching 10000 threads, each opening a file, while the maximum number of open files is typically 1024 under Unix systems.
First try to keep manually track of the number of files currently open and check whether it bumps past the OS limit.
Since nobody was able to replicate the problem, I went hunting around our office for machines. It passed on everything except my machine. I tested on a Mac with the same software versions I was using. I even went hunting for a Desktop G5 with the EXACT same hardware and software config I had -- same result. Both tests (with tempfile and without tempfile) succeeded on everything.
For kicks, I downloaded Python 2.6.4, and tried it on my desktop, and same pattern on my system as Python 2.5.1: tempfile failed, and notempfile succeeded.
This is leading me to the conclusion that something's hosed on my Mac, so this isn't likely to be a problem that anyone else runs into ever.
Thanks VERY much to everyone (especially Alex Martelli) who helped on this!