I'm trying to write some unit tests for a flask application i'm building, this is the file that contains the flask application
app.py
from flask import Flask
app = Flask(__name__)
#app.route('/')
def home():
return 'hello world'
and then the file that contains the tests
tests.py
from unittest import TestCase, main
from app import app
from multiprocessing import Process
import requests
class Test(TestCase):
#classmethod
def setUpClass(cls):
cls.hostname = 'http://localhost:8000'
with app.app_context():
p = Process(target=app.run, kwargs={'port': 8000})
p.start()
cls.p = p
def test_options(self):
# print(self.p.is_alive()) # returns True but requests doesn't connect
result = requests.post(self.hostname).text
self.assertEqual(result, 'hello world')
#classmethod
def tearDownClass(cls):
cls.p.terminate() # works fine if i comment it out
pass
if __name__ == '__main__':
main()
I came up with an idea to use the requests module to test the application instead of the test_client that comes with flask.
From my understanding of the unittest module,
The setUpClass method is called once, that creates a new process and starts the application.
After that i'm free to run various tests with request.post or request.get.
Then at the end of the day, when all the tests have been run tearDownClass is supposed to terminate the process.
When i try to run the tests this is what i get (The stacktrace is actually longer than this, but it all boils down to this)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7faeeba183c8>: Failed to establish a new connection: [Errno 111] Connection refused',))
Because the flask process doesn't start and requests is unable to connect to localhost via port 8000.
but the strange part is if i comment out cls.p.terminate everything works properly and the test passes, the only problem is that i won't be able to kill the flask process again.
I know i can make the process daemonic or use the test_client like this:
class Test(TestCase):
#classmethod
def setUpClass(cls):
cls.hostname = 'http://localhost:8000'
with app.app_context():
app.config['TESTING'] = True
cls.app = app.test_client()
def test_options(self):
# print(self.p.is_live())
result = self.app.get().data
self.assertEqual(result, b'hello world')
and just move on with my life, but i really want to understand why my code doesn't work when i try to kill the process when i'm tearing down the test suite.
I've tried to use pdb to debug, but for some reason i don't know my terminal gets messed up when i try to type anything in the pdb prompt (Some letters don't show up in the terminal as i type, but when i hit enter i discover that the letters were actually there but were not visible, E.g to type print i might end up having prrrinnnttt because r and t refused to show up when i was pressing them)
I don't know what's happening in your case, but in one of my projects I use a bash script like this to test:
#!/bin/bash
set -eux
python app.py &
sleep 3
set +e
python tests.py
result=$?
kill $(ps aux | grep app.py | grep -v grep | awk '{print $2}')
exit $result
This handles bringing up the server, running the tests, and then killing the server, and there is no mention of handling the server in the python tests.
Related
First of all I am aware of flask-testing library with LiveServerTestCase class but it hasn't updated since 2017 and GitHub full of issues of it not working neither on Windows or MacOs and I haven't found any other solutions.
I am trying to write some tests for flask app using selenium to validate FlaskForms inside this app.
Simple test like this:
def test_start(app):
driver.get("http://127.0.0.1:5000/endpoint")
authenticate(driver)
falls on selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_CONNECTION_REFUSED error. (As far as I understood in my case app creates in #pytest.fixtures and immediately shuts down and I need to find a way to keep it running for the whole test duration)
My question is: Is it possible to to create some live server in each test that will remain working so I could call API endpoints via selenium?
Simple fixtures if it helps:
#pytest.fixture
def app():
app = create_app()
...
with app.context():
# creating db
...
yield app
also:
#pytest.fixture
def client(app):
"""Test client"""
return app.test_client()
Finally got it all working. My conftest.py
import multiprocessing
import pytest
from app import create_app
#pytest.fixture(scope="session")
def app():
app = create_app()
multiprocessing.set_start_method("fork")
return app
#pytest.fixture
def client(app):
return app.test_client()
Important note that using python <3.8 line multiprocessing.set_start_method("fork") is not necessary (as far as I understood in v.3.8 they refactored multiprocessing module so further upon without this line you would get pickle Error on windows and Mac).
And one simple test looks like
def test_add_endpoint_to_live_server(live_server):
#live_server.app.route('/tests-endpoint')
def test_endpoint():
return 'got it', 200
live_server.start()
res = urlopen(url_for('.te', _external=True))# ".te is a method path I am calling"
assert url_for('.te', _external=True) == "some url"
assert res.code == 200
assert b'got it' in res.read()
Also I am using url_for. The point is every time live server starts on a random port and url_for function generates url with correct port internally. So now live server is running and it is possible to implement selenium tests.
I run my proyect on PyCharm last version.
When I run my main.py it runs good and behaivor is desired. But If I restart (or re-run) the main.py, I got error
OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
No other isntances running. Python is the only thing on computer now. No other usages of ports. The only way to skip this error is or restarting computer or deleting every python from task manager.
Also, after stopping service, Swagger is still accesible with browser, I think that this swagger is using the port for main (because they run together) and that is why I can't run my main.py.
One way to reproduce this error is run this MWE, then stop and close PyCharm or just close Pycharm, then try to restart it and the message error comes again, and swagger is still up!. This happen while I'm coding and try to run the new things I wrote.
MWE:
from tornado.ioloop import IOLoop
from tornado_swagger.setup import setup_swagger
from tornado.options import parse_command_line,define,options
from tornado.web import Application,url,RequestHandler
class TestRequests(RequestHandler):
def get(self,var1):
"""
---
tags:
- Testing
summary: minimal working example
description: some dummy text
produces:
- requestsBody:
- application/json
parameters:
- name: var1
in: path
description: dummy tet
required: true
type: string
responses:
200:
description: dummy response for dummy requests here and now
"""
print(f"Reveived GET tests for {var1}")
res = {"result": "hello",
"product": str(var1)+"-foo"}
self.write(res)
class Applications(Application):
urls = [
url(r"/api/(?P<var1>[a-zA-Z0-9]+)/test",TestRequests)
]
def __init__(self):
settings = {'debug': True}
setup_swagger(self.urls,
swagger_url = '/swagger',
api_base_url="/Base",
description = "test swagger",
title = "test title")
super(Applications,self).__init__(self.urls,**settings)
define('port', default='8088', help='Port to listen on')
if __name__ == "__main__":
print("running")
parse_command_line()
app = Applications()
app.listen(options.port)
IOLoop.current().start()
I am trying to test a Flask web app within a docker container, which is new for me. My stack is the following:
firefox
selenium
pytest-selenium
pytest-flask
Here is my Flask app file:
from flask import Flask
def create_app():
app = Flask(__name__)
return app
app = create_app()
#app.route('/')
def index():
return render_template('index.html')
Now, my test file which verifies the title of my index page:
import pytest
from app import create_app
# from https://github.com/pytest-dev/pytest-selenium/issues/135
#pytest.fixture
def firefox_options(request, firefox_options):
firefox_options.add_argument('--headless')
return firefox_options
# from https://pytest-flask.readthedocs.io/en/latest/tutorial.html#step-2-configure
#pytest.fixture
def app():
app = create_app()
return app
# from https://pytest-flask.readthedocs.io/en/latest/features.html#start-live-server-start-live-server-automatically-default
#pytest.mark.usefixtures('live_server')
class TestLiveServer:
def test_homepage(self, selenium):
selenium.get('http://0.0.0.0:5000')
h1 = selenium.find_element_by_tag_name('h1')
assert h1 == 'title'
When I run my tests with:
pytest --driver Firefox --driver-path /usr/local/bin/firefox test_app.py
I get the following error (which seems due to firefox not in headless mode).
selenium.common.exceptions.WebDriverException: Message: Service /usr/local/bin/firefox unexpectedly exited. Status code was: 1
Error: no DISPLAY environment variable specified
I am able to run firefox --headless but it seems my pytest fixture didn't manage to do the setup. Is there a better way to do this?
Now, if I replace selenium.get() by urlopen just to try the correct initialization of the app and its connection:
def test_homepage(self):
res = urlopen('http://0.0.0.0:5000')
assert b'OK' in res.read()
assert res.code == 200
I get the error:
urllib.error.URLError:
Do I need to boot the live server differently? Or should I change my host + port config somewhere?
Regarding the problem with a direct call with urllib:
Pytest's live server uses random port by default. You can add this parameter to pytest invocation:
--live-server-port 5000
Or without this parameter you can make direct calls to live server like:
import pytest
import requests
from flask import url_for
#pytest.mark.usefixtures('live_server')
def test_something():
r = requests.get(url_for('index', _external=True))
assert r.status_code == 200
I suppose you have view function called index. It would add a correct port number automatically.
But this doesn't have to do anything with docker, how do you run it?
Regarding the problem with Selenium itself - I can imagine docker networks related problem. How do you use it? Do you have eg. docker-compose configuration? Can you share it?
The referenced pytest-selenium issue has:
#pytest.fixture
def firefox_options(firefox_options, pytestconfig):
if pytestconfig.getoption('headless'):
firefox_options.add_argument('-headless')
return firefox_options
Note the - (single dash) preceding headless in add_argument()
(Source)
For late comers, it might be worthwhile taking a look at Xvfb and even more helpful can be this tutorial
Then (in Linux shell) you can enter:
Xvfb :99 &
export DISPLAY=:99
pytest --driver Firefox --driver-path /usr/local/bin/firefox test_app.py
This provides a virtual frame buffer (fake screen) for the application and it outputs all the graphical content there.
Note that I did not encountered this problem, just providing a solution that helped me overcome the mentioned error with an another app.
I am running a Twisted application with the following command:
twistd -y service.py
The code looks like this:
from twisted.web.server import Site
from twisted.application import service, internet
from twisted.web.resource import Resource
class WebRoot(Resource):
isLeaf = True
def render_GET(self, request):
return "web resource".encode("utf-8")
application = service.Application("testapp")
root = internet.TCPServer(8084, Site(WebRoot()))
root.setServiceParent(application)
The command starts the application as a daemon, and I am then able to enter other commands into the terminal session with no issue. However, if I close the terminal session, the program stops running. I don't understand why this is happening to a daemonic service such as this, so any help will be appreciated.
I want to write Selenium tests with server as fixture:
import pytest
#pytest.fixture()
def driver(request):
from selenium import webdriver
d = webdriver.Firefox()
request.addfinalizer(d.close)
return d
#pytest.fixture()
def server():
from server import run
run(host="localhost", port=8080)
def test_can_see_echo(driver,server):
page = TestPage(driver)
page.fill_text_in_input("test")
page.click_send()
print page.get_returnet_value()
Function run in server fixture is bottle run function. The problem is that, when I call run() programs go into infinite loop and body of test is not executed. Should I call run in same thread? Is my design fine? In future I want use server fixture to integrate to server state. For example make test "add comment" using Selenium and in the end use server fixture to ask server if this action really happened.
The tests hang because your run(host="localhost", port=8080) starts a server which waits forever. You should start that server in a different thread/process.
Look into something like pytest-xprocess for running external server processes for your tests.