I have read about unit testing in python, but all examples I found are based on trivial examples of mocking objects. I have no idea how to really implement tests for the project in a way it can be tested without accessing COM port. project is mainly used for control of another application via COM API plus evaluating data produced by app and making reports, so let's say it is higher abstraction level api made as python library. COM interface is non-trivial, with main object that is exposing some managers, the managers exposing containers of objects, that are having references to another objects, so a web of conencted objects responsible for different things controlled application. Architecture of python library somehow follows the COM structure. During library import main COM object is dispatched and stored in central module, manager-level modules on import are getting references to COM manager objects from central module and then manager-level methods are using this COM manager objects in their methods. Examples for better understanding
#__init__
from managera import ManagerA
from centralobject import CentralObject
central_object = CentralObject()
manager_a = ManagerA() #here all initialisation to make everything work,
manager_b = ManagerB() #so full com object needed to import the package
...
#centalobject
class CentalObject():
def __init__():
self.com = Dispatch("application")
...
#managera
from project import central_object
class ManagerA():
def __init__():
self.com = central_object.com.manager_a
def manager_a_method1(x):
foo = self.com.somecontainer[x].somemethod()
foo.configure(3)
return foo
...
In current state it is hard to test. It is even not possible import without connection to the COM app. Dispatch could be moved into some init function that is executed after the import, but I don't see how it would make the testing possible. One solution would be to make test double that have structutre similar to the original application, but for my unexperienced in testing mind it seems a bit overkill to do this in each method test. Maybe it is not and that's how it should be done, I am asking somebody's more experience advice.
Second solutuon that came to my mind is to dispatch COM object every time any COM call is made, and then mock it but it seems a lot of dispatches and I don't see how it makes the code better.
Tried also with managers not being defined as classes, but modules, however it seemed even more hard to test as then COM managers was refrerenced during module import, not object instantiation.
What should be done to make testing posible and nice without accessing COM? Interested in everything: solution, advice or just topic that I should read more about.
Related
I am relatively new to Python, so please bear with my lack of technical knowledge.
I have a lock-in amplifier setup with a GPIB to USB cable that is plugged into my computer. I am able to read the instrument fine with Pyvisa. I am additionally using other driver classes (labdrivers) to write and read values to and from the instruments.
Here is the __init__ method for the Python class I am referencing. There is of course more to it in terms of other methods that specifically run the commands using the instrument read from the resource manager, but I believe the error lies in this method when specifically reading the instrument. I have added three stars (***) to the beginning and end of the resourcemanager lines that are likely causing the issue for easier reference.
import pyvisa
# create a logger object for this module
logger = logging.getLogger(__name__)
# added so that log messages show up in Jupyter notebooks
logger.addHandler(logging.StreamHandler())
class Sr830:
"""Interface to a Stanford Research Systems 830 lock in amplifier."""
def __init__(self, gpib_addr):
"""Create an instance of the Sr830 object.
:param gpib_addr: GPIB address of the SR830
"""
try:
# the pyvisa manager we'll use to connect to the GPIB resources
***self.resource_manager = pyvisa.ResourceManager()***
except OSError:
logger.exception("\n\tCould not find the VISA library. Is the VISA driver installed?\n\n")
self._gpib_addr = gpib_addr
self._instrument = None
***self._instrument = self.resource_manager.open_resource("GPIB::%d" % self._gpib_addr)***
The code I am running on my test file is this.
from labdrivers.srs import sr830
sr_gpib_address = 8
lock_in = sr830.Sr830(sr_gpib_address)
The reason why I create the object using sr830.Sr830 is because this is the file path.
\anaconda3\lib\site-packages\labdrivers\srs\sr830.py.
The error output from the terminal is this.
Traceback (most recent call last):
File ~\test.py:14 in <module>
lock_in = sr830.Sr830(sr_gpib_address)
File ~\anaconda3\lib\site-packages\labdrivers\srs\sr830.py:28 in __init__
self._instrument = self.resource_manager.open_resource("GPIB::%d" % self._gpib_addr)
AttributeError: 'Sr830' object has no attribute 'resource_manager'
My files are in different directories. Could this be a possible issue? I am not sure if I am just creating the object wrong (I may be referencing the files incorrectly) or if there is some issue with pyvisa, although the pyvisa commands seem correct.
I am running Python 3.9 on Spyder.
Please let me know any more information that I should provide.
Thank you for all and any help.
I was able to resolve my issue.
I believe the main problem was using a driver class written from someone else, which may have included other information or incompatible reference programs. By simply calling creating the Pyvisa resource manager object by myself in a separate file and passing information to the machine with the methods specified in the manual, the error no longer appeared. It is most likely necessary to make the methods simpler as well, as the driver methods included a great deal of other information which may have needed specific installations or versions of the supposed installations. It was not an issue of the files being in different directories. Additionally, I would advise making the class and file names as different as possible, as that led to some conflicts when I was creating objects. The file name was Sr830 which was similar to the class name of Sr830. This other error persisted despite my attempts to work around it, such as:
from labdrivers import Sr830
from Sr830 import Sr830
Can't provide much context given the complexity, but I'm hoping for some insight/thought-provoking questions as to why this is happening.
I'm testing a process which loads files into a database, so I'm patching the credentials for a database connection using unittest.mock.patch to use test and not production credentials. We have a series of mocks that are applied as a contextmanager, simplified version here:
from contextlib import ExitStack
def context_stack(contexts):
stack = ExitStack()
for context in contexts:
stack.enter_context(context)
return stack
def patch_mocks():
mocks = [
patch('db_config.ReadWrite', db_mocks.ReadWrite),
patch('db_config.ReadWrite', db_mocks.ReadWrite)
]
return context_stack(mocks)
It gets used as such (simplified):
with patch_mocks():
LoadFiles(file_list)
LoadFiles will iterate over each file in file_list and attempt to insert the contents into the database. The underlying methods connect to the database using db_config.ReadWrite but of course they are patched by db_mocks.ReadWrite. This works pretty consistently except, seemingly very randomly, it will fail as it tries to instead use db_config.ReadWrite when trying to create the connection.
So for example, there could be a hundred files, and it will patch the most of them successfully, but it will randomly stop using the patch halfway through and fail the test. What conditions/variables could be causing this patch to not be applied? Is there a limit to the number of patches that can be applied? Should it be applied in another way?
My first line of investigation would involve this warning from the docs on .patch():
target should be a string in the form 'package.module.ClassName'. The target is imported and the specified object replaced with the new object, so the target must be importable from the environment you are calling patch() from. The target is imported when the decorated function is executed, not at decoration time.
and this further explanation on Where to patch
The basic principle is that you patch where an object is looked up, which is not necessarily the same place as where it is defined.
I would try to find a broken case and check the status of the import environment there to make sure the same import you're using everywhere else is reachable from there.
Do not patch/mock, instead use the repository pattern to access the database.
You would then have two implementations of the Repository interface:
in memory: keeps all data in-memory
using a DB-driver/connector: actually writes to the DB
So we have a rather large program written in Python using PySide/QT to get most of its GUI work done. We use Chaco to do some plotting.
This program also has a full CLI interface.
Every now and then a developer accidentally creates an import chain that causes our CLI runs to try and import something from PySide or Chaco. This causes our CLI runs to die with "cannot connect to x server" as either PySide or Chaco is trying to initialize X via QT.
Any tips on how to prevent this? Can we stub out and override some function that is doing this? Some flag we can pass along? Currently our prevention mechanism is track down the bad import and refactor.
So one semi-solution is to do the following. Essentially stub out the QApplication class and have it print a stack trace on init. This will break things but you will get a stack trace to do the first spot QtApplication tries to initialize.
_oldQtApplication = QtGui.QApplication
class BogusQApplication(QtGui.QApplication):
def __init__(self, *args):
import traceback
print traceback.print_stack()
_oldQtApplication.__init__(self, args)
QtGui.QApplication = BogusQApplication
A better solution would be to essentially stub the whole QtApplication class in such a way that users of it still work but essentially have a NullQtApplication. Unfortunately this appears to be a ton of work based on the usage of QtApplication in libraries like PySide of Chaco.
Rather than avoiding initialising Qt altogether, perhaps you could use QCoreApplication in your CLI applications. See the detailed description of QApplication in http://doc.qt.io/qt-5/qapplication.html for some example code you could adapt.
The good practice for such kind of pattern is to have a proper organisation of the code where the code using the user interface can't be imported when running on the CLI.
You can do that by designing your application in a pluggable way and only load the plugins needed by the CLI when running in CLI mode and the full set of plugins when running with the UI. We extensively use the Envisage framework to build pluggable application and solve those kind of problems. It requires a bit more upfront effort in designing your app.
For reference:
https://pypi.python.org/pypi/envisage
http://docs.enthought.com/envisage/
https://github.com/enthought/envisage
Most python windows service examples based on the win32serviceutil.ServiceFramework use the win32event for synchronization.
For example:
http://tools.cherrypy.org/wiki/WindowsService (the example for cherrypy 3.0)
(sorry I dont have the reputation to post more links, but many similar examples can be googled)
Can somebody clearly explain why the win32events are necessary (self.stop_event in the above example)?
I guess its necessary to use the win32event due to different threads calling svcStop and svcRun? But I'm getting confused, there are so many other things happening: the split between python.exe and pythonservice.exe, system vs local threads (?), python GIL..
For the top of PythonService.cpp
PURPOSE: An executable that hosts Python services.
This source file is used to compile 2 discrete targets:
* servicemanager.pyd - A Python extension that contains
all the functionality.
* PythonService.exe - This simply loads servicemanager.pyd, and
calls a public function. Note that PythonService.exe may one
day die - it is now possible for python.exe to directly host
services.
What exactly do you mean by system threads vs local threads? You mean threads created directly from C outside the GIL?
The PythonService.cpp just related the names to callable python objects and a bunch of properties, like the accepted methods.
For example a the accepted controls from the ServiceFramework:
def GetAcceptedControls(self):
# Setup the service controls we accept based on our attributes. Note
# that if you need to handle controls via SvcOther[Ex](), you must
# override this.
accepted = 0
if hasattr(self, "SvcStop"): accepted = accepted | win32service.SERVICE_ACCEPT_STOP
if hasattr(self, "SvcPause") and hasattr(self, "SvcContinue"):
accepted = accepted | win32service.SERVICE_ACCEPT_PAUSE_CONTINUE
if hasattr(self, "SvcShutdown"): accepted = accepted | win32service.SERVICE_ACCEPT_SHUTDOWN
return accepted
I suppose the events are recommended because that way you could interrupt the interpreter from outside the GIL, even if python is in a blocking call from the main thread, e.g.: time.sleep(10) you could interrupt from those points outside the GIL and avoid having an unresponsive service.
Most of the win32 services calls are in between the python c macros:
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS
It may be that, being examples, they don't have anything otherwise interesting to do in SvcDoRun. SvcStop will be called from another thread, so using an event is just an easy way to do the cross-thread communication to have SvcDoRun exit at the appropriate time.
If there were some service-like functionality that blocks in SvcDoRun, they wouldn't necessarily need the events. Consider the second example in the CherryPy page that you linked to. It starts the web server in blocking mode, so there's no need to wait on an event.
I'm using Pywin32 to communicate with Bloomberg through its COM-library. This works rather good! However, I have stumbeled upona a problem which I consider pretty complex. If I set the property QueueEvents of the Com object to True I the program fails. In the documentation they have a section regarding this,
If your QueueEvents property is set to
True and you are performing low-level
instantiation of the data control
using C++, then in your data event
handler (invoke) you will be required
to initialize pvarResult by calling
the VariantInit() function. This will
prevent your application from
receiving duplicate ticks.
session = win32com.client.DispatchWithEvents(comobj, EventHandler)
session.QueueEvents = True <-- this trigger some strange "bugs" in execution
if "pvarResult" is not initialized
I think I understand the theoretical aspects here, you need to initialize a datastructure before the comobject can write to it. However, how do you do this from Pywin32? That I have no clue about, and would appreciate any ideas or pointers(!) to how this can be done.
None of the tips below helped. My program doesn't throw an exception, it just returns the same message from the COM object again and again and again...
From the documentation:
If your QueueEvents property is set to
True and you are performing low-level
instantiation of the data control
using C++, then in your data event
handler (invoke) you will be required
to initialize pvarResult by calling
the VariantInit() function. This will
prevent your application from
receiving duplicate ticks. If this
variable is not set then the data
control assumes that you have not
received data yet, and it will then
attempt to resend it. In major
containers, such as MFC and Visual
Basic, this flag will automatically be
initialized by the container. Keep in
mind that this only pertains to
applications, which set the
QueueEvents property to True.
I'm not sure if this will help for your issue, but to have working COM events in Python you shouldn't forget about:
setting COM apartment to free
threaded at the beginning of script
file. This could be done using
following lines
import sys
sys.coinit_flags = 0
generating wrapper for com library before calling first DispatchWithEvents
from win32com.client.makepy import GenerateFromTypeLibSpec
GenerateFromTypeLibSpec("ComLibName 1.0 Type Library")
If you could post how the program fails (COM object fails or maybe python trows some exceptions) maybe I could advice more.