We Keep Coding

Can you use mock_open to simulate serial connections?

Morning folks,
I'm trying to get a few unit tests going in Python to confirm my code is working, but I'm having a real hard time getting a Mock anything to fit into my test cases. I'm new to Python unit testing, so this has been a trying week thus far.
The summary of the program is I'm attempting to do serial control of a commercial monitor I got my hands on and I thought I'd use it as a chance to finally use Python for something rather than just falling back on one of the other languages I know. I've got pyserial going, but before I start shoving a ton of commands out to the TV I'd like to learn the unittest part so I can write for my expected outputs and inputs.
I've tried using a library called dummyserial, but it didn't seem to be recognising the output I was sending. I thought I'd give mock_open a try as I've seen it works like a standard IO as well, but it just isn't picking up on the calls either. Samples of the code involved:
def testSendCmd(self):
powerCheck = '{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK']).encode('utf-8')
read_text = 'Stuff\r'
mo = mock_open(read_data=read_text)
mo.in_waiting = len(read_text)
with patch('__main__.open', mo):
with open('./serial', 'a+b') as com:
tv = SharpTV(com=com, TVID=999, tvInput = 'DVI')
tv.sendCmd(SharpCodes['POWER'], SharpCodes['CHECK'])
com.write(b'some junk')
print(mo.mock_calls)
mo().write.assert_called_with('{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK']).encode('utf-8'))
And in the SharpTV class, the function in question:
def sendCmd(self, type, msg):
sent = self.com.write('{0}{1:>4}\r'.format(type,msg).encode('utf-8'))
print('{0}{1:>4}\r'.format(type,msg).encode('utf-8'))
Obviously, I'm attempting to control a Sharp TV. I know the commands are correct, that isn't the issue. The issue is just the testing. According to documentation on the mock_open page, calling mo.mock_calls should return some data that a call was made, but I'm getting just an empty set of []'s even in spite of the blatantly wrong com.write(b'some junk'), and mo().write.assert_called_with(...) is returning with an assert error because it isn't detecting the write from within sendCmd. What's really bothering me is I can do the examples from the mock_open section in interactive mode and it works as expected.
I'm missing something, I just don't know what. I'd like help getting either dummyserial working, or mock_open.

To answer one part of my question, I figured out the functionality of dummyserial. The following works now:
def testSendCmd(self):
powerCheck = '{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK'])
com = dummyserial.Serial(
port='COM1',
baudrate=9600,
ds_responses={powerCheck : powerCheck}
)
tv = SharpTV(com=com, TVID=999, tvInput = 'DVI')
tv.sendCmd(SharpCodes['POWER'], SharpCodes['CHECK'])
self.assertEqual(tv.recv(), powerCheck)
Previously I was encoding the dictionary values as utf-8. The dummyserial library decodes whatever you write(...) to it so it's a straight string vs. string comparison. It also encodes whatever you're read()ing as latin1 on the way back out.

Dll causes Python to crash when using memset

I've been working on a project where I'm trying to use an old CodeBase library, written in C++, in Python. What I want is to use CodeBase to reindex a .dbf-file that has a .cdx-index. But currently, Python is crashing during runtime. A more detailed explanation will follow further down.
As for Python, I'm using ctypes to load the dll and then execute a function I added myself which should cause no problems, since it doesn't use a single line of code that CodeBase itself isn't using.
Python Code:
import ctypes
cb_interface = ctypes.CDLL("C4DLL.DLL")
cb_interface.reindex_file("C:\\temp\\list.dbf")
Here's the CodeBase function I added, but it requires some amount of knowledge that I can't provide right now, without blowing this question up quite a bit. If neccessary, I will provide further insight, as much as I can:
S4EXPORT int reindex_file(const char* file){
CODE4 S4PTR *code;
DATA4 S4PTR *data;
code4initLow(&code, 0, S4VERSION, sizeof(CODE4));
data = d4open(code, file);
d4reindex(data);
return 1;
}
According to my own debugging, my problem happens in code4initLow. Python crashes with a window saying "python.exe has stopped working", when the dll reaches the following line of code:
memset( (void *)c4, 0, sizeof( CODE4 ) ) ;
c4 here is the same object as code in the previous code-block.
Is there some problem with a dll trying to alter memory during runtime? Could it be a python problem that would go away if I were to create a .exe-file from my python script?
If someone could answer me these questions and/or provide a solution for my python-crashing-problem, I would greatly appreciate it.
And last but not least, this is my first question here. If I have accidently managed to violate a written or unwritten rule here, I apologize and promise to fix that as soon as possible.

First of all the pointer code doesn't point anywhere as it is uninitialized. Secondly you don't actually try to fill the structure, since you pass memset a pointer to the pointer.
What you should probably do is declare code as a normal structure instance (and not a pointer), and then use &code when passing it to d4open.

Like Joachim Pileborg saying the problem is, to pass a Nullpointer to code4initLow. ("Alternative") Solution is to allocate Memory for the struct CODE4 S4PTR *code = (CODE4*)malloc(sizeof(CODE4));, then pass the Pointer like code4initLow(code, 0, S4VERSION, sizeof(CODE4));.

How to debug Python memory fault?

Edit: Really appreciate help in finding bug - but since it might prove hard to find/reproduce, any general debug help would be greatly appreciated too! Help me help myself! =)
Edit 2: Narrowing it down, commenting out code.
Edit 3: Seems lxml might not be the culprit, thanks! The full script is here. I need to go over it looking for references. What do they look like?
Edit 4: Actually, the scripts stops (goes 100%) in this, the parse_og part of it. So edit 3 is false - it must be lxml somehow.
Edit 5 MAJOR EDIT: As suggested by David Robinson and TankorSmash below, I've found a type of data content that will send lxml.etree.HTML( data ) in a wild loop. (I carelessly disregarded it, but find my sins redeemed as I've paid a price to the tune of an extra two days of debug! ;) A working crashing script is here. (Also opened a new question.)
Edit 6: Turns out this is a bug with lxml version 2.7.8 and below (at
least). Updated to lxml 2.9.0, and bug is gone. Thanks also to the fine folks over at this follow-up question.
I don't know how to debug this weird problem I'm having.
The below code runs fine for about five minutes, when the RAM is suddenly completely filled up (from 200MB to 1700MB during the 100% period - then when memory is full, it goes into blue wait state).
It's due to the code below, specifically the first two lines. That's for sure. But what is going on? What could possibly explain this behaviour?
def parse_og(self, data):
""" lxml parsing to the bone! """
try:
tree = etree.HTML( data ) # << break occurs on this line >>
m = tree.xpath("//meta[#property]")
#for i in m:
# y = i.attrib['property']
# x = i.attrib['content']
# # self.rj[y] = x # commented out in this example because code fails anyway
tree = ''
m = ''
x = ''
y = ''
i = ''
del tree
del m
del x
del y
del i
except Exception:
print 'lxml error: ', sys.exc_info()[1:3]
print len(data)
pass

You can try Low-level Python debugging with GDB. Probably there is a bug in Python interpreter or in lxml library and it is hard to find it without extra tools.
You can interrupt your script running under gdb when CPU usage goes to 100% and look at stack trace. It will probably help to understand what's going on inside script.

it must be due to some references which keep the documents alive. one must always be careful with string results from xpath evaluation. I see you have assigned None to tree and m but not to y,x and i .
Can you also assign None to y,x and i .

Tools are also helpful when trying to track down memory problems. I've found guppy to be a very useful Python memory profiling and exploration tool.
It is not the easiest to get started with due to a lack of good tutorials / documentation, but once you get to grips with it you will find it very useful. Features I make use of:
Remote memory profiling (via sockets)
Basic GUI for charting usage, optionally showing live data
Powerful, and consistent, interfaces for exploring data usage in a Python shell

Loading a document on OpenOffice using an external Python program

I'm trying to create a python program (using pyUNO ) to make some changes on a OpenOffice calc sheet.
I've launched previously OpenOffice on "accept" mode to be able to connect from an external program. Apparently, should be as easy as:
import uno
# get the uno component context from the PyUNO runtime
localContext = uno.getComponentContext()
# create the UnoUrlResolver
resolver = localContext.ServiceManager.createInstanceWithContext(
"com.sun.star.bridge.UnoUrlResolver", localContext)
# connect to the running office
ctx = resolver.resolve("uno:socket,host=localhost,port=2002;"
"urp;StarOffice.ComponentContext")
smgr = ctx.ServiceManager
# get the central desktop object
DESKTOP =smgr.createInstanceWithContext("com.sun.star.frame.Desktop", ctx)
#The calling it's not exactly this way, just to simplify the code
DESKTOP.loadComponentFromURL('file.ods')
But I get an AttributeError when I try to access loadComponentFromURL. If I make a dir(DESKTOP), I've see only the following attributes/methods:
['ActiveFrame', 'DispatchRecorderSupplier', 'ImplementationId', 'ImplementationName',
'IsPlugged', 'PropertySetInfo', 'SupportedServiceNames', 'SuspendQuickstartVeto',
'Title', 'Types', 'addEventListener', 'addPropertyChangeListener',
'addVetoableChangeListener', 'dispose', 'disposing', 'getImplementationId',
'getImplementationName', 'getPropertySetInfo', 'getPropertyValue',
'getSupportedServiceNames', 'getTypes', 'handle', 'queryInterface',
'removeEventListener', 'removePropertyChangeListener', 'removeVetoableChangeListener',
'setPropertyValue', 'supportsService']
I've read that there are where a bug doing the same, but on OpenOffice 3.0 (I'm using OpenOffice 3.1 over Red Hat5.3). I've tried to use the workaround stated here, but they don't seems to be working.
Any ideas?

It has been a long time since I did anything with PyUNO, but looking at the code that worked last time I ran it back in '06, I did my load document like this:
def urlify(path):
return uno.systemPathToFileUrl(os.path.realpath(path))
desktop.loadComponentFromURL(
urlify(tempfilename), "_blank", 0, ())
Your example is a simplified version, and I'm not sure if you've removed the extra arguments intentionally or not intentionally.
If loadComponentFromURL isn't there, then the API has changed or there's something else wrong, I've read through your code and it looks like you're doing all the same things I have.
I don't believe that the dir() of the methods on the desktop object will be useful, as I think there's a __getattr__ method being used to proxy through the requests, and all the methods you've printed out are utility methods used for the stand-in object for the com.sun.star.frame.Desktop.
I think perhaps the failure could be that there's no method named loadComponentFromURL that has exactly 1 argument. Perhaps giving the 4 argument version will result in the method being found and used. This could simply be an impedance mismatch between Python and Java, where Java has call-signature method overloading.

This looks like issue 90701: http://www.openoffice.org/issues/show_bug.cgi?id=90701
See also http://piiis.blogspot.com/2008/10/pyuno-broken-in-ooo-30-with-system.html and http://udk.openoffice.org/python/python-bridge.html

In Python, How Do I Set an Expiration Date on a User Object in Active Directory?

Setup: I have the user object in hand, via win32com.client.Dispatch('ADsNameSpaces'), in a standard Windows environment, using ActiveState Python of the 2.6 flavor. Apparently, Get() and Put()/SetInfo() methods are the appropriate ways to read from and write to properties of the object. My approach has been to simply adapt examples from the Active Directory Cookbook to Python. I can set other properties (password, sn, givenName, etc) of the user object and otherwise activate the account, add it to groups, but setting the expiration date seems to be problematic.
In VBScript (I know, I know, that's what's in the book) you can do a:
objUser.AccountExpirationDate = '12/31/2010'
objUser.SetInfo
I also know that there's a separate accountExpires property. I can perform a times = objUser.Get('accountExpires') and receive an object with a .HighPart and a .LowPart, as I know the information is internally stored, so I feel I am on the right track.
Additionally, I have written functions which convert human readable dates to Unix Epoch time, and from there to the 64-bit Microsoft time format (100 nanosecond intervals since January 1, 1601; stored as a 32-bit HighPart and a 32-bit LowPart), should these be required.
Failed attempts:
1)
objUser.AcccountExpirationDate = '03/20/2010'
>>AttributeError: Property '<unknown>.AcccountExpirationDate' can not be set.
2)
objUser.Put('AccountExpirationDate', '03/20/2010')
>> File "<COMObject <unknown>>", line 2, in SetInfo
pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, u'Active Directory', u'The specified directory service attribute or value does not exist.\r\n', None, 0, -2147016694), None)
3)
times = objUser.Get('accountExpires')
print 'highpart: ' + str(times.HighPart)
# "highpart" variable computed elsewhere
times.Put('HighPart', highpart)
>>AttributeError: <unknown>.Put
Failed inspection attempts for the derived "times" object and the parts within it (HighPart, LowPart):
1) dir() has not been helpful with my attempts to inspect the object, putting forth nothing that looks unusual.
2) The "Guide to Python introspection" article (http://www.ibm.com/developerworks/library/l-pyint.html), which interesting, has not provided any more insight.
3) I have a great deal of bewildering, but highly generic-looking, output from the inspect module; nothing says "Hi, put a 32-bit integer in me!"
Where do I go from here?

Well I never used ADsNameSpaces COM objects before. But looking at the examples Here, I think you may want to try something like:
objUser.AccountExpirationDate(date)