I'm trying to run Appium tests written in python 3 on AWS Device Farm.
As stated in the documentation,
Python 2.7 is supported in both the standard environment and using Custom Mode. It is the default in both when specifying Python.
Python 3 is only supported in Custom Mode. To choose Python 3 as your python version, change the test spec to set the PYTHON_VERSION to 3, as shown here:
phases:
install:
commands:
# ...
- export PYTHON_VERSION=3
- export APPIUM_VERSION=1.14.2
# Activate the Virtual Environment that Device Farm sets up for Python 3, then use Pip to install required packages.
- cd $DEVICEFARM_TEST_PACKAGE_PATH
- . bin/activate
- pip install -r requirements.txt
# ...
I did run tests successfully on Device Farm in the past, using a python 3 custom environment, with this spec file (I include the install phase only):
phases:
install:
commands:
# Device Farm support two major versions of Python, each with one minor version: 2 (minor version 2.7), and 3 (minor version 3.7.4).
# The default Python version is 2, but you can switch to 3 by setting the following variable to 3:
- export PYTHON_VERSION=3
# This command will install your dependencies and verify that they're using the proper versions that you specified in your requirements.txt file. Because Device Farm preconfigures your environment within a
# Python Virtual Environment in its setup phase, you can use pip to install any Python packages just as you would locally.
- cd $DEVICEFARM_TEST_PACKAGE_PATH
- . bin/activate
- pip install -r requirements.txt
# ...
Now, when I run the tests, I get this log an then the test crashes due to incompatible code.
[DEVICEFARM] ########### Entering phase test ###########
[DeviceFarm] echo "Navigate to test package directory"
Navigate to test package directory
[DeviceFarm] cd $DEVICEFARM_TEST_PACKAGE_PATH
[DeviceFarm] echo "Start Appium Python test"
Start Appium Python test
[DeviceFarm] py.test tests/ --junit-xml $DEVICEFARM_LOG_DIR/junitreport.xml
============================= test session starts ==============================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /tmp/scratchrPuGRa.scratch/test-packageM5E89M/tests, inifile:
collected 0 items / 3 errors
==================================== ERRORS ====================================
____________________ ERROR collecting test_home_buttons.py _____________________
/usr/local/lib/python2.7/dist-packages/_pytest/python.py:610: in _importtestmodule
mod = self.fspath.pyimport(ensuresyspath=importmode)
Is python 3.x not supported anymore, or have there been undocumented changes?
Is there a new way to run tests in a python 3 environment?
The documented way is still correct.
I found there was an error during the installation of the pytest package in the virtual environment. This made the py.test command refer to the default environment.
In my case the issue was resolved by installing an older version of pytest and bundling it in requirements.txt with other packages.
pip install pytest==6.2.4
This works on Windows without problems, but crashes on Linux.
python setup.py test --addopts '--config myconf.ini'
I have defined a custom argument for testing to provide a test user credentials via a config file:
tests/conftest.py:
def pytest_addoption(parser):
parser.addoption('--config', default='conf.ini', help='Path to the testing config file.')
#pytest.fixture(scope='session')
def configfile(request):
return request.config.getoption("--config")
This is how I use it in testing files:
def test_foo(configfile):
print(configfile)
On Windows, this works fine. On Linux, however, I get this (setup.cfg is configured to invoke pytest, so this is pytest's err output)
usage: setup.py [options] [file_or_dir] [file_or_dir] [...]
setup.py: error: unrecognized arguments: --config
inifile: None
rootdir: /uupmaker
What is interesting though, if I pass a filename to --config fn that does not exist, no error is thrown and the tests run normally (albeit as if there was no config file defined).
Have I overlooked something? Why won't this work?
Edit: pip freeze --all output:
certifi==2017.7.27.1
chardet==3.0.4
colorama==0.3.9
coverage==4.4.1
defusedxml==0.5.0
idna==2.6
jenkinsapi==0.3.4
jira==1.0.10
lxml==3.8.0
oauthlib==2.0.3
pbr==3.1.1
pip==9.0.1
py==1.4.34
pytest==3.2.2
pytest-runner==2.12.1
python-dateutil==2.6.1
pytz==2017.2
reaver==0.1.5
requests==2.18.4
requests-oauthlib==0.8.0
requests-toolbelt==0.8.0
setuptools==28.8.0
six==1.10.0
teamcity-messages==1.21
urllib3==1.22
Using Python 3.6.1
setup.cfg:
[aliases]
test=pytest
This is the minimal working example for me: https://github.com/melkamar/pytest-setuppy-err
Simply clone that folder and run python3 setup.py test --addopts "-s --config foo.bar" and it should fail on Linux.
So OP no longer needs it, but I had this exact issue and was sad to see it was not resolved.
Instead of using:
--config configfile
try with:
--config=configfile
That fixed it for me.
I used to run the tests for my packages from the Makefile as a way to perform three tasks in one: setup a virtual environment, install requirements and call the testing suite with the corresponding arguments:
test: venv
env/bin/pip install -r test_requirements.txt
env/bin/nosetests --rednose --with-coverage --cover-pagacke my_module
Then I read that requirements.txt files are discouraged in favor of setup.py, so I modified the setup.py file aiming to get the same result:
setup(
...
tests_require=['nose', 'rednose', 'coverage'],
test_suite=['nose.collector'])
Now I could change the Makefile with
test: venv
coverage run --source=my_module/ setup.py test
But that requires installing the testing dependencies before running the setup.py file. I'm also not sure how to include other arguments such as rednose. What is the best way to do this?
Tox is good and all, but here's how to do it without having to install any other package beforehand.
List the testing dependencies as setup_requires instead of tests_require in the setup.py file
setup(
setup_requires=['nose', 'rednose', 'coverage'],
install_requires=[] # fill in other arguments as usual
)
Optionally add test parameters to the setup.cfg file.
[nosetests]
rednose=1
detailed-errors=1
with-coverage=1
cover-package=server
cover-xml=1
verbosity=2
Run the tests with the following command
python setup.py nosetests
Source: http://nose.readthedocs.io/en/latest/api/commands.html
Package Settings
I have built a Python package which uses nose for testing. Therefore, setup.py contains:
..
test_suite='nose.collector',
tests_require=['nose'],
..
And python setup.py test works as expected:
running test
...
----------------------------------------------------------------------
Ran 3 tests in 0.065s
OK
Running with XUnit output
Since I'm using Jenkins CI, I would like to output the nose results to JUnit XML format:
nosetests <package-name> --with-xunit --verbose
However, python setup.py test is far more elegant, and it installs the test requirements without having to build a virtual environment.
Is there a way to pass the --with-xunit (or any other parameter) to nose, when calling nose via python setup.py test?
You can set nosetests option using setup.cfg
For example in you setup.cfg
[nosetests]
with-xunit=1
Further information can be found at http://nose.readthedocs.io/en/latest/api/commands.html
Nose provides its own setuptools command (nosetests) which accepts command line arguments:
python setup.py nosetests --with-xunit
More information can be found here:
http://nose.readthedocs.io/en/latest/setuptools_integration.html
subtitle: Not only sdist
I am trying to get the setup.py file of a package I'm working on to play nicely with sdist. The relevant parts of the setup.py file are:
from setuptools.command.test import test
[...]
class Tox(test):
"as described in
http://tox.readthedocs.org/en/latest/example/basic.html?highlight=setuptools#integration-with-setuptools-distribute-test-commands"
[...]
def run_tests(self):
if self.distribution.install_requires:
self.distribution.fetch_build_eggs(
self.distribution.install_requires)
if self.distribution.tox_requires:
self.distribution.fetch_build_eggs(self.distribution.tox_requires)
# import here, cause outside the eggs aren't loaded
import tox
import shlex
args = self.tox_args
if args:
args = shlex.split(self.tox_args)
else:
args = ""
errno = tox.cmdline(args=args)
sys.exit(errno)
entry_points ={}
distutils_ext = {'distutils.setup_keywords': [
"tox_requires = setuptools.dist:check_requirements", ]
}
entry_points.update(distutils_ext)
setup(
install_requires=['six', 'numpy', 'matplotlib', 'scipy', 'astropy>=1',
'Pillow', ],
cmdclass={
'test': PyTest, # this is to run python setup.py test
'tox': Tox,
},
# list of packages and data
packages=find_packages(),
# tests
tests_require=['pytest', 'pytest-cov'],
tox_requires=['tox'],
# other keywords, mostly metadata
)
If I run python setup.py sdist, I get a warning at the beginning:
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tox_requires'
warnings.warn(msg)
but then the sdist works fine and it creates a tar.gz file that I can use to install my package.
But if I run it a second time, it starts with (it's the beginning of Pillow building):
warning: no previously-included files found matching '.editorconfig'
Building using 4 processes
_imaging.c: In function ‘getink’:
and begins building all the required packages into .eggs directory.
If I remove the *egg-info directory I can rerun the command.
If I comment out the tox_requires=[...] line, I can build the sdist as many times as I want.
Now according to the setuptools documentation the command above should be the correct way to run add new arguments to the setup function.
As per the subtitle, the problem is not only with sdist but it's probably due to my non-understanding on how setuptools and requirements work.
If I run python setup.py tox in a place without tox already installed I get, after the installation of some testing package it should not install (namely pytest and pytest-cov):
Traceback (most recent call last):
[...]
File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 127, in with_project_on_sys_path
func()
File "setup.py", line 65, in run_tests
if self.distribution.tox_requires:
AttributeError: Distribution instance has no attribute 'tox_requires'
[Update] The tox_requires also confuse very badly pip during installation. If it is commented out I can install the package without any issue; otherwise it begins to compile the source of the packages and it systematically fails because it doesn't find numpy while building stuff like scipy
How can I get setuptools to recognize and properly use tox_requires?
Once this issue is fixed, I think that I can get rid of the spurious installations here doing a better job at implementing the Tox class, maybe overriding more things from test or deriving it directly from Command
Complete (working) solution described below consist of 8 files (incl. short
README.rst) and has in total 43 lines of code. This is less then code in your
original question.
Despite of being so short, it supports many development and testing scenarios
in very convenient way.
Anyway, it does not answer exactly your question, but I am sure, it fulfils the
requirements which were behind it.
Three lines long setup.py
Technically it may be possible to put test command including tox automation into your setup.py, however, the result may be very messy and difficult to understand.
The same result can be achieved in much simpler way:
for developer assume:
using git
having tox installed into system
for package user:
there are no special requirements to install the resulting package
(optional) if you want your users to test the package by single command and keep test reports collected in central server:
install devpi-server and give your users access to it
ask your users to install $ pip install devpi
The solution builds on following tools and packages:
pbr: simplify package creation incl. versioning via git tags and
creation of AUTHORS and ChangeLog from git commit messages.
pytest: excelent testing framework, but any other framework can
be used instead of it.
tox: excellent build and test automation tool.
coverage: tools to measure test coverage (working simpler then
pytest-cov)
Optionally you may also use:
devpi-server: private PyPi server with password protected
access. Allows simple testing and provides test reports collection.
devpi: tool similar to pip. Apart from installation also supports
running tox defined tests (install, run tests, publish reports in on step).
Authoring the package
Create new project directory and initialize git:
$ mkdir francesco
$ cd francesco
$ git init
Create a package or module
Here we create single module francesco, but the same works for more
modules or packages.
francesco.py
def main():
print("Hi, it is me, Francesco, keeping things simple.")
requirements.txt
Create list of packages for actual installation of the package:
six
test_requirements.txt
Define packages required for testing:
pytest
coverage
tests/test_it.py
Initiate the test suite:
from francesco import main
def test_this():
main()
print("All seems fine to me")
assert True
setup.py
Did you dream of stupid simple setup.py? Here it goes:
from setuptools import setup
setup(setup_requires=["pbr"], pbr=True)
setup.cfg
Metadata belong to configuration file:
[metadata]
name = francesco
author = Francesco Montesano
author-email = fm#acme.com
summary = Nice and simply installed python module supporting testing in different pythons
description-file = README.rst
[files]
modules=francesco
[entry_points]
console_scripts =
francesco = francesco:main
tox.ini
To configure tox automated builds and tests:
[tox]
envlist = py27, py34
[testenv]
commands =
coverage run --source francesco -m pytest -sv tests
coverage report
coverage html
deps =
-rtest_requirements.txt
README.rst
Never forget README.rst:
===========================================
Complex package with 3 line long `setup.py`
===========================================
Can we keep`setup.py` simple and still support automated testing?
...
tox: build sdist and run tests in all supported python versions
Being in the project directory root, just run single command tox:
$ tox
GLOB sdist-make: /home/javl/sandbox/setuppy/setup.py
py27 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py27 runtests: PYTHONHASHSEED='2409409075'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py27 runtests: commands[2] | coverage html
py34 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py34 runtests: PYTHONHASHSEED='2409409075'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py34 runtests: commands[2] | coverage html
___________________________________ summary ____________________________________
py27: commands succeeded
py34: commands succeeded
congratulations :)
Getting the sdist
ls .tox/dist
francesco-0.0.0.zip
Developing in Python 2.7 virtualenv
Activate Python 2.7 virtualenv
$ source .tox/py27/bin/activate
Run tests
(py27) $ py.test -sv tests
==============================================================================================
test session starts
===============================================================================================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
-- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7 cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile: collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things
simple. All seems fine to me PASSED
============================================================================================
1 passed in 0.01 seconds
============================================================================================
Measure test coverage
(py27)$ coverage run --source francesco -m pytest -sv tests
.....
(py27)$ coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
View coverage report in web browser
(py27)$ coverage html
(py27)$ firefox htmlcov/index.html
Release new package version
(optional) Install local devpi-server
The installation of devpi-server is not covered here, but is very
simple, especially, if you install only to your local machine for your
personal testing.
Commit source code, assign version tag
Make sure, all your source code is commited.
Assing version tag:
$ git tag -a 0.1
Rerun the tests by tox and build sdist
Make sure, you have deactivated virtualenv (otherwise it conflicts with
tox):
(py27)$ deactivate
Run the tox:
$ tox
.....
...it builds as usual, may fail, if you have forgotten to commit some changes or files...
Find the sdist for new version of your package:
$ ls .tox/dist/francesco-0.1.0.
.tox/dist/francesco-0.1.0.zip
You are done. You may distribute your new tested versions of your package
to users as usually.
(optional) Upload the sdist to devpi-server and test it locally
Following steps assume, you have devpi-server installed and running.
$ devpi login javl
...enter your password...
$ devpi upload .tox/dist/francesco-0.1.0.zip
Test the package in clean environment
(deactivate virtualenv if active) :
$ cd /tmp
$ mkdir testing
$ cd testing
$ devpi test francesco
received http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
unpacking /tmp/devpi-test6/downloads/francesco-0.1.0.zip to /tmp/devpi-test6/zip
/tmp/devpi-test6/zip/francesco-0.1.0$ tox --installpkg /tmp/devpi-test6/downloads/francesco-0.1.0.zip -i ALL=http://localhost:3141/javl/dev/+simple/ --recreate --result-json /tmp/devpi-test6/zip/toxreport.json
-c /tmp/devpi-test6/zip/francesco-0.1.0/tox.ini
py27 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27
py27 installdeps: -rtest_requirements.txt
py27 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py27 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='3916044270'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py27 runtests: commands[2] | coverage html
py34 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34
py34 installdeps: -rtest_requirements.txt
py34 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py34 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py34 runtests: PYTHONHASHSEED='3916044270'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py34 runtests: commands[2] | coverage html
____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
py27: commands succeeded
py34: commands succeeded
congratulations :)
wrote json report at: /tmp/devpi-test6/zip/toxreport.json
posting tox result data to http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
successfully posted tox result data
You may check the test results in web browser:
$ firefox http://localhost:3141
then search for "francesco" package, click the package name, find in table
column named "tox results", click there to show environment set up and test
results.
Let your users test the package
Let's assume, your devpi-server is running and your user has access to it.
The user shall have devpi command installed:
$ pip install devpi
(note, this tool is not installing anything from the devpi-server)
Help your user to gain access to devpi-server (not covering here).
Then the user just runs the test:
$ devpi test francesco
After the test is run (it is automatically using tox, but user does not have
to care about that), you will find test results on the same place on devpi web
interface as you found yours before.