We Keep Coding

Squish test functions always pass when used with squishtest module

We are using Squish for Qt 6.6.2 on Windows 10 with Python 3.8.7 and running our tests using squishtest module with Robot Framework 4.0.1.
We are having an issue with the test functions provided by the Squish API where any verifications done with such a call (for example squishtest.test.imagePresent) will Pass. The issue itself was quite simple to pinpoint to the fact that although the verification failed, the function call itself was passing without raising exceptions. This can also be verified from the report provided by the squishrunner where we have <scriptedVerificationResult type="FAIL" time="--"> on the passed execution.
The question is, can we in any way get the actual verification result passed to the Robot so we can fail the test accordingly? Preferrably in real time rather than parsing the report afterwards.
In Squish this works perfectly fine
def main():
startApplication("AUT")
snooze(2)
test.imagePresent("image.png", {"tolerant": True, "threshold": 85},
waitForObjectExists(names.sceneContainer_GraphWidget))
but with Robot this is always passing
# In testSuite.robot
*** Settings ***
Library MySquishLib
*** Test Cases ***
Test Image
Start AUT
Verify Image image.png {"tolerant": True, "threshold": 85} names.sceneContainer_GraphWidget
# In MySquishLib.py
import squishtest
import names
def start_aut():
squishtest.startApplication("AUT")
def verify_image(imageFile, imageParams, imageArea):
squishtest.test.imagePresent(imageFile, imageParams, imageArea)

Have a look at the documented: bool testSettings.throwOnFailure flag.
In your robot you can set this and you would not have to patch / rewrite every test.vp, test.compare, ... method.
When running from within the Squish IDE, this flag can be unset. Probably robotframework provides some environment variables to detect wether the test case is running from within itself.
https://doc.froglogic.com/squish/latest/rgs-squish.html#rgs-testsettings

The test functions are not supposed to raise exceptions during execution in order to allow test to continue even if a single VP was failed. The function does however return a boolean value just as expected. By using
def verify_image(imageFile, imageParams, imageArea):
if not squishtest.test.imagePresent(imageFile, imageParams, imageArea):
raise Exception("Image was not found")
I'm able to fail the Robot test without any issues.

Connecting to local dockerized Perforce server in Python unittests

I maintain a Python tool that runs automation against a Perforce server. For obvious reasons, parts of my test suite (which are unittest.TestCase classes run with Pytest) require a live server. Until now I've been using a remote testing server, but I'd like to move that into my local environment, and make server initialization part of my pre-test setup.
I'm experimenting with dockerization as a solution, but I get strange connection errors when trying to run Perforce commands against the server in my test code. Here's my test server code (using a custom docker image, Singleton metaclass based on https://stackoverflow.com/a/6798042, and with the P4Python library installed):
class P4TestServer(metaclass=Singleton):
def __init__(self, conf_file='conf/p4testserver.conf'):
self.docker_client = docker.from_env()
self.config = P4TestServerConfig.load_config(conf_file)
self.server_container = None
try:
self.server_container = self.docker_client.containers.get('perforce')
except docker.errors.NotFound:
self.server_container = self.docker_client.containers.run(
'perforce-server',
detach=True,
environment={
'P4USER': self.config.p4superuser,
'P4PORT': self.config.p4port,
'P4PASSWD': self.config.p4superpasswd,
'NAME': self.config.p4name
},
name='perforce',
ports={
'1667/tcp': 1667
},
remove=True
)
self.p4 = P4()
self.p4.port = self.config.p4port
self.p4.user = self.config.p4superuser
self.p4.password = self.config.p4superpasswd
And here's my test code:
class TestSystemP4TestServer(unittest.TestCase):
def test_server_connection(self):
testserver = P4TestServer()
with testserver.p4.connect():
info = testserver.p4.run_info()
self.assertIsNotNone(info)
So this is the part that's getting to me: the first time I run that test (i.e. when it has to start the container), it fails with the following error:
E P4.P4Exception: [P4#run] Errors during command execution( "p4 info" )
E
E [Error]: 'TCP receive failed.\nread: socket: Connection reset by peer'
But on subsequent runs, when the container is already running, it passes. What's frustrating is that I can't otherwise reproduce this error. If I run that test code in any other context, including:
In a Python interpreter
In a debugger stopped just before the testserver.p4.run_info() invokation
The code completes as expected regardless of whether the container was already running.
All I can think at this point is that there's something unique about the pytest environment that's tripping me up, but I'm at a loss for even how to begin diagnosing. Any thoughts?

I had a similar issue recently where I would start postgres container and then immediately run a python script to setup database as per my app requirement.
I had to introduce a sleep command in between the two steps and that resolved the issue.
Ideally you should check if the start sequence of the docker container is done before trying to use it. But for my local development use case, sleep 5 seconds was good enough workaround.

Robot Framework: How to merge old/new statuses using rebot --merge and keep the only one?

I'm using rebot tool to merge outputs after re-execution of failed tests.
robot --output original.xml /path/to/dir/.
robot --rerunfailed original.xml --output rerun.xml /path/to/dir/.
rebot -o machine1.xml -l machine1.html --merge original.xml rerun.xml
The same action is provided on few test machines.
Test suite is identical and is executed against each VM. All VMs are considered as identical, however, they are not stable and I receive different results on each machine.
I want to merge all results from all machines and retrieve the maximum number of Passed tests, to understand whether the test is REALLY failing or it's just unstable environment and test is OK by itself.
Other words, if test passed at least on 1 machine, but failed on other 3 machines, it needs to be considered as Passed in the final report.
However I receive False for such case.
Is it possible to change the behavior somehow?
Example from the final report:
Status: FAIL (critical)
Message: Re-executed test has been merged.
New status: FAIL
New message: Re-executed test has been merged.
New status: FAIL
New message: IndexError: Given index 0 is out of the range 0--1.
Old status: FAIL
Old message: IndexError: Given index 0 is out of the range 0--1.
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message:
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message:
Old status: PASS
Old message:

Resolved by using simple script for prerebotmodifier written by myself. Maybe it will be helpful for somebody else.
So, I created TestStatusChecker.py which helps to merge reports with the following logic: test status will be failed ONLY if it was failing in ALL reports.
from robot.api import SuiteVisitor
class TestStatusChecker(SuiteVisitor):
def __init__(self, *args):
pass
def visit_test(self, test):
if 'PASS' in test.message and 'Re-executed test has been merged' in test.message:
test.status = 'PASS'
test.message = 'Test passed because it passed at least once.'
CLI command for merging the results:
rebot -l final_log.html --prerebotmodifier TestStatusChecker.py --merge 1.xml 2.xml 3.xml 4.xml
report

SonarQube Python Plugin - scanning python code: Fail to decorate

I am trying to run a scan in the sample project (I am actually trying to scan a much larger project, but the problem is the same. I am working on the sample project because is much simpler) and it is giving me the following error:
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
Total time: 29.790s
Final Memory: 14M/379M
INFO: ------------------------------------------------------------------------
ERROR: Error during Sonar runner execution
ERROR: Unable to execute Sonar
ERROR: Caused by: Fail to decorate 'org.sonar.api.resources.File#56ffc188[key=src/__init__.py,deprecatedKey=__init__.py,path=src/__init__.py,dir=[root],filename=__init__.py,language=Python]'
ERROR: Caused by: 0
ERROR:
ERROR: To see the full stack trace of the errors, re-run SonarQube Runner with the -e switch.
ERROR: Re-run SonarQube Runner using the -X switch to enable full debug logging.
Anyone has any idea of what this means? This is the sample project right out of the box, and my sonnar runner config is firly simple:
#----- Default SonarQube server
sonar.host.url=http://localhost:9000
#----- MySQL
sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8
#----- Global database settings
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
#----- Default source code encoding
sonar.sourceEncoding=UTF-8
#----- Security (when 'sonar.forceAuthentication' is set to 'true')
sonar.login=admin
sonar.password=admin
I have enabled all pylint rules, within a new profile just with those rules. With the default profile as default (Sonar Way) the error does not happen. Curious thing is that I enabled a random rule from pylint and it worked. So maybe one (or more) rule is messing with the analysis.
Pylint integration is essential for me.
The stacktrace, although not very helpful is the following:
ERROR: Error during Sonar runner execution
org.sonar.runner.impl.RunnerException: Unable to execute Sonar
at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:91)
at org.sonar.runner.impl.BatchLauncher$1.run(BatchLauncher.java:75)
at java.security.AccessController.doPrivileged(Native Method)
at org.sonar.runner.impl.BatchLauncher.doExecute(BatchLauncher.java:69)
at org.sonar.runner.impl.BatchLauncher.execute(BatchLauncher.java:50)
at org.sonar.runner.api.EmbeddedRunner.doExecute(EmbeddedRunner.java:102)
at org.sonar.runner.api.Runner.execute(Runner.java:100)
at org.sonar.runner.Main.executeTask(Main.java:70)
at org.sonar.runner.Main.execute(Main.java:59)
at org.sonar.runner.Main.main(Main.java:53)
Caused by: org.sonar.api.utils.SonarException: Fail to decorate 'org.sonar.api.resources.File#3f3674b2[key=src/__init__.py,deprecatedKey=__init__.py,path=src/__init__.py,dir=[root],filename=__init__.py,language=Python]'
at org.sonar.batch.phases.DecoratorsExecutor.executeDecorator(DecoratorsExecutor.java:103)
at org.sonar.batch.phases.DecoratorsExecutor.decorateResource(DecoratorsExecutor.java:86)
at org.sonar.batch.phases.DecoratorsExecutor.decorateResource(DecoratorsExecutor.java:78)
at org.sonar.batch.phases.DecoratorsExecutor.decorateResource(DecoratorsExecutor.java:78)
at org.sonar.batch.phases.DecoratorsExecutor.execute(DecoratorsExecutor.java:70)
at org.sonar.batch.phases.PhaseExecutor.execute(PhaseExecutor.java:126)
at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:222)
at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:93)
at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:78)
at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:235)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:230)
at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:223)
at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:93)
at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:78)
at org.sonar.batch.scan.ScanTask.scan(ScanTask.java:65)
at org.sonar.batch.scan.ScanTask.execute(ScanTask.java:52)
at org.sonar.batch.bootstrap.TaskContainer.doAfterStart(TaskContainer.java:128)
at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:93)
at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:78)
at org.sonar.batch.bootstrap.BootstrapContainer.executeTask(BootstrapContainer.java:171)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:95)
at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:67)
at org.sonar.runner.batch.IsolatedLauncher.execute(IsolatedLauncher.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:87)
... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at org.sonar.plugins.core.issue.tracking.FileHashes.getHash(FileHashes.java:75)
at org.sonar.plugins.core.issue.IssueTracking.setChecksumOnNewIssues(IssueTracking.java:69)
at org.sonar.plugins.core.issue.IssueTracking.track(IssueTracking.java:54)
at org.sonar.plugins.core.issue.IssueTrackingDecorator.doDecorate(IssueTrackingDecorator.java:138)
at org.sonar.plugins.core.issue.IssueTrackingDecorator.decorate(IssueTrackingDecorator.java:112)
at org.sonar.batch.phases.DecoratorsExecutor.executeDecorator(DecoratorsExecutor.java:95)
... 36 more
Thanks for any help!

I encountered the same problem. I had to deactivate two rules to stop this error message:
Missing docstring
docstring should be defined
I also deactivate the deprecated rules but I don't know if it had any effect.

How to skip the rest of tests in the class if one has failed?

I'm creating the test cases for web-tests using Jenkins, Python, Selenium2(webdriver) and Py.test frameworks.
So far I'm organizing my tests in the following structure:
each Class is the Test Case and each test_ method is a Test Step.
This setup works GREAT when everything is working fine, however when one step crashes the rest of the "Test Steps" go crazy. I'm able to contain the failure inside the Class (Test Case) with the help of teardown_class(), however I'm looking into how to improve this.
What I need is somehow skip(or xfail) the rest of the test_ methods within one class if one of them has failed, so that the rest of the test cases are not run and marked as FAILED (since that would be false positive)
Thanks!
UPDATE: I'm not looking or the answer "it's bad practice" since calling it that way is very arguable. (each Test Class is independent - and that should be enough).
UPDATE 2: Putting "if" condition in each test method is not an option - is a LOT of repeated work. What I'm looking for is (maybe) somebody knows how to use the hooks to the class methods.

I like the general "test-step" idea. I'd term it as "incremental" testing and it makes most sense in functional testing scenarios IMHO.
Here is a an implementation that doesn't depend on internal details of pytest (except for the official hook extensions). Copy this into your conftest.py:
import pytest
def pytest_runtest_makereport(item, call):
if "incremental" in item.keywords:
if call.excinfo is not None:
parent = item.parent
parent._previousfailed = item
def pytest_runtest_setup(item):
previousfailed = getattr(item.parent, "_previousfailed", None)
if previousfailed is not None:
pytest.xfail("previous test failed (%s)" % previousfailed.name)
If you now have a "test_step.py" like this:
import pytest
#pytest.mark.incremental
class TestUserHandling:
def test_login(self):
pass
def test_modification(self):
assert 0
def test_deletion(self):
pass
then running it looks like this (using -rx to report on xfail reasons):
(1)hpk#t2:~/p/pytest/doc/en/example/teststep$ py.test -rx
============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev17
plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov, timeout
collected 3 items
test_step.py .Fx
=================================== FAILURES ===================================
______________________ TestUserHandling.test_modification ______________________
self = <test_step.TestUserHandling instance at 0x1e0d9e0>
def test_modification(self):
> assert 0
E assert 0
test_step.py:8: AssertionError
=========================== short test summary info ============================
XFAIL test_step.py::TestUserHandling::()::test_deletion
reason: previous test failed (test_modification)
================ 1 failed, 1 passed, 1 xfailed in 0.02 seconds =================
I am using "xfail" here because skips are rather for wrong environments or missing dependencies, wrong interpreter versions.
Edit: Note that neither your example nor my example would directly work with distributed testing. For this, the pytest-xdist plugin needs to grow a way to define groups/classes to be sent whole-sale to one testing slave instead of the current mode which usually sends test functions of a class to different slaves.

If you'd like to stop the test execution after N failures anywhere (not in a particular test class) the command line option pytest --maxfail=N is the way to go:
https://docs.pytest.org/en/latest/usage.html#stopping-after-the-first-or-n-failures
if you instead want to stop a test that is comprised of multiple steps if any of them fails, (and continue executing the other tests) you should put all your steps in a class, and use the #pytest.mark.incremental decorator on that class and edit your conftest.py to include the code shown here
https://docs.pytest.org/en/latest/example/simple.html#incremental-testing-test-steps.

The pytest -x option will stop test after first failure:
pytest -vs -x test_sample.py

It's generally bad practice to do what are you doing. Each test should be as independent as possible from the others, while you completely depend on the results of the other tests.
Anyway, reading the docs it seems like a feature like the one you want is not implemented.(Probably because it wasn't considered useful).
A work-around could be to "fail" your tests calling a custom method which sets some condition on the class, and mark each test with the "skipIf" decorator:
class MyTestCase(unittest.TestCase):
skip_all = False
#pytest.mark.skipIf("MyTestCase.skip_all")
def test_A(self):
...
if failed:
MyTestCase.skip_all = True
#pytest.mark.skipIf("MyTestCase.skip_all")
def test_B(self):
...
if failed:
MyTestCase.skip_all = True
Or you can do this control before running each test and eventually call pytest.skip().
edit:
Marking as xfail can be done in the same way, but using the corresponding function calls.
Probably, instead of rewriting the boiler-plate code for each test, you could write a decorator(this would probably require that your methods return a "flag" stating if they failed or not).
Anyway, I'd like to point out that,as you state, if one of these tests fails then other failing tests in the same test case should be considered false positive...
but you can do this "by hand". Just check the output and spot the false positives.
Even though this might be boring./error prone.

You might want to have a look at pytest-dependency. It is a plugin that allows you to skip some tests if some other test had failed.
In your very case, it seems that the incremental tests that gbonetti discussed is more relevant.

Based on hpk42's answer, here's my slightly modified incremental mark that makes test cases xfail if the previous test failed (but not if it xfailed or it was skipped). This code has to be added to conftest.py:
import pytest
try:
pytest.skip()
except BaseException as e:
Skipped = type(e)
try:
pytest.xfail()
except BaseException as e:
XFailed = type(e)
def pytest_runtest_makereport(item, call):
if "incremental" in item.keywords:
if call.excinfo is not None:
if call.excinfo.type in {Skipped, XFailed}:
return
parent = item.parent
parent._previousfailed = item
def pytest_runtest_setup(item):
previousfailed = getattr(item.parent, "_previousfailed", None)
if previousfailed is not None:
pytest.xfail("previous test failed (%s)" % previousfailed.name)
And then a collection of test cases has to be marked with #pytest.mark.incremental:
import pytest
#pytest.mark.incremental
class TestWhatever:
def test_a(self): # this will pass
pass
def test_b(self): # this will be skipped
pytest.skip()
def test_c(self): # this will fail
assert False
def test_d(self): # this will xfail because test_c failed
pass
def test_e(self): # this will xfail because test_c failed
pass

UPDATE: Please take a look at #hpk42 answer. His answer is less intrusive.
This is what I was actually looking for:
from _pytest.runner import runtestprotocol
import pytest
from _pytest.mark import MarkInfo
def check_call_report(item, nextitem):
"""
if test method fails then mark the rest of the test methods as 'skip'
also if any of the methods is marked as 'pytest.mark.blocker' then
interrupt further testing
"""
reports = runtestprotocol(item, nextitem=nextitem)
for report in reports:
if report.when == "call":
if report.outcome == "failed":
for test_method in item.parent._collected[item.parent._collected.index(item):]:
test_method._request.applymarker(pytest.mark.skipif("True"))
if test_method.keywords.has_key('blocker') and isinstance(test_method.keywords.get('blocker'), MarkInfo):
item.session.shouldstop = "blocker issue has failed or was marked for skipping"
break
def pytest_runtest_protocol(item, nextitem):
# add to the hook
item.ihook.pytest_runtest_logstart(
nodeid=item.nodeid, location=item.location,
)
check_call_report(item, nextitem)
return True
Now adding this to conftest.py or as a plugin solves my problem.
Also it's improved to STOP testing if the blocker test has failed. (meaning that the entire further tests are useless)

Or quite simply instead of calling py.test from cmd (or tox or wherever), just call:
py.test --maxfail=1
see here for more switches:
https://pytest.org/latest/usage.html

To complement hpk42's answer, you can also use pytest-steps to perform incremental testing, this can help you in particular if you wish to share some kind of incremental state/intermediate results between the steps.
With this package you do not need to put all the steps in a class (you can, but it is not required), simply decorate your "test suite" function with #test_steps:
from pytest_steps import test_steps
def step_a():
# perform this step ...
print("step a")
assert not False # replace with your logic
def step_b():
# perform this step
print("step b")
assert not False # replace with your logic
#test_steps(step_a, step_b)
def test_suite_no_shared_results(test_step):
# Execute the step
test_step()
You can add a steps_data parameter to your test function if you wish to share a StepsDataHolder object between your steps.
import pytest
from pytest_steps import test_steps, StepsDataHolder
def step_a(steps_data):
# perform this step ...
print("step a")
assert not False # replace with your logic
# intermediate results can be stored in steps_data
steps_data.intermediate_a = 'some intermediate result created in step a'
def step_b(steps_data):
# perform this step, leveraging the previous step's results
print("step b")
# you can leverage the results from previous steps...
# ... or pytest.skip if not relevant
if len(steps_data.intermediate_a) < 5:
pytest.skip("Step b should only be executed if the text is long enough")
new_text = steps_data.intermediate_a + " ... augmented"
print(new_text)
assert len(new_text) == 56
#test_steps(step_a, step_b)
def test_suite_with_shared_results(test_step, steps_data: StepsDataHolder):
# Execute the step with access to the steps_data holder
test_step(steps_data)
Finally, you can automatically skip or fail a step if another has failed using #depends_on, check in the documentation for details.
(I'm the author of this package by the way ;) )