I am in charge of testing of a web application using Selenium Webdriver with Python. Over the past year I created a large script (20K+ lines) where each test is a separate function. Now my boss wants me to document my tests explaining in plan English what each test does. What tool would you recommend to document the steps your tests make?
I think this is a great question. Many people and companies don't bother managing their existing tests properly which leads to redundant and repeated code without having a clear idea what is covered by automated tests.
There is no single answer to this question but in general you can consider the following options:
Testing framework built in reporting. In Java, for example, you have the unit testing libraries like jUnit and TestNG. When they run, they generate certain output that can later be formatted and reviewed as the need arises. I am sure there an implementation of unit testing framework like this in Python too.
You can also consider using a BDD tool like Cucumber. This is a bit different and might not be suitable in certain cases when the tests are low level system checks. It can however help you organize your test scenarios and keep them an a readable form. It is also very good for reporting to a non-technical person.
Related
I'm trying to process a bunch of files, which aren't known until the testing begins, and I want to add testing mechanisms to the processing so I know if there are any errors and get a report out at the end.
This processing requires a little bit of setup so a setup test needs runs once before the processing actually begins. Does anyone have any good examples of this kind of process being done in Python?
I've been researching this for the past few days and I haven't found any good solutions. A few options that I've seen are:
unittest: Using dynamically generated tests with a setUpClass method to do the setup.
Nose doesn't seem like an option at all because of the lack of continued support. This testing needs to last for a long time.
pytest (?): Not quite sure on this, the documentation isn't very good and I haven't found any concrete examples.
Basically I need a testing framework that has the ability to dynamically create parameterized tests with dependencies.
I have the need to scale up some testing efforts for web application. I'm most familiar with using selenium (with python bindings) for functional testing amongst other things. Now that I need to also do concurrent load/stress testing I think I need to take different approach. I like the look of locust, but I'm not sure how to integrate the functional test requirements as well. The basic test outline for an individual user is this:
login to site with credentials
"click" relevant angular elements to navigate the site
"click" and initiate download of various reports
Ideally, I could scale this with 10-50-100 concurrent users and get a log file with results (times, failures, etc.)
Any best practices tips from the frequently unsung test heros would be sincerely appreciated!
EDIT:
I realize this is a bit non-standard. Just the nature of what I am trying replicate with new relic running the background for analytics. Currently, I'm trying to figure out if Selenium can be combined with Locust in an appropriate way.
You are right that your first choice was Locust. The main strength of Locust that it is the Python code based tool and you can do there almost everything else what you can do in pure Python.
if you are looking for some functional testing aspect, you can even do it in integration with your load tests with default Python assertions library.
Check this article, it should give you some thoughts on how to make functional checks within your Locust performance tests using Python:
https://www.blazemeter.com/blog/locust-assertions-a-complete-user-manual
I'm teaching myself backend and frontend web development (I'm using Flaks if it matters) and I need few pointers for when it comes to unit test my app.
I am mostly concerned with these different cases:
The internal consistency of the data: that's the easy one - I'm aiming for 100% coverage when it comes to issues like the login procedure and, most generally, checking that everything that happens between the python code and the database after every request remain consistent.
The JSON responses: What I'm doing atm is performing a test-request for every get/post call on my app and then asserting that the json response must be this-and-that, but honestly I don't quite appreciate the value in doing this - maybe because my app is still at an early stage?
Should I keep testing every json response for every request?
If yes, what are the long-term benefits?
External APIs: I read conflicting opinions here. Say I'm using an external API to translate some text:
Should I test only the very high level API, i.e. see if I get the access token and that's it?
Should I test that the returned json is what I expect?
Should I test nothing to speed up my test suite and don't make it dependent from a third-party API?
The outputted HTML: I'm lost on this one as well. Say I'm testing the function add_post():
Should I test that on the page that follows the request the desired post is actually there?
I started checking for the presence of strings/html tags in the row response.data, but then I kind of gave up because 1) it takes a lot of time and 2) I would have to constantly rewrite the tests since I'm changing the app so often.
What is the recommended approach in this case?
Thank you and sorry for the verbosity. I hope I made myself clear!
Most of this is personal opinion and will vary from developer to developer.
There are a ton of python libraries for unit testing - that's a decision best left to you as the developer of the project to find one that fits best with your tool set / build process.
This isn't exactly 'unit testing' per se, I'd consider it more like integration testing. That's not to say this isn't valuable, it's just a different task and will often use different tools. For something like this, testing will pay off in the long run because you'll have piece of mind that your bug fixes and feature additions aren't impacting your end to end code. If you're already doing it, I would continue. These sorts of tests are highly valuable when refactoring down the road to ensure consistent functionality.
I would not waste time testing 3rd party APIs. It's their job to make sure their product behaves reliably. You'll be there all day if you start testing 3rd party features. A big reason to use 3rd party APIs is so you don't have to test them. If you ever discover that your app is breaking because of a 3rd party API it's probably time to pick a different API. If your project scales to a size where you're losing thousands of dollars every time that API fails you have a whole new ball of issues to deal with (and hopefully the resources to address them) at that time.
In general, I don't test static content or html. There are tools out there (web scraping tools) that will let you troll your own website for consistent functionality. I would personally leave this as a last priority for the final stages of refinement if you have time. The look and feel of most websites change so often that writing tests isn't worth it. Look and feel is also really easy to test manually because it's so visual.
I recently stumbled over this (aged) article:
http://imranontech.com/2007/01/04/unit-testing-the-final-frontier-legacy-code/
where the author allegedly wrote a perl script to automatically generate test cases.
His strategy went like this (cited):
Read in the header files I gave it.
Extracted the function prototypes.
Gave me the list of functions it found and let me pick
which ones I wanted to create unit tests for.
It then created a dbx
(Solaris debugger) script which would break-point every time the
selected function was called, save the variables that were passed to
it and then continue until the function returned at which point it
would save the return value.
Run the executable under the dbx
script, and which point I proceeded to use the application as
normal, and just ran through lots of use cases which I thought would
go through the code in question and especially cases where I thought
it would hit edge cases in the functions I want to create unit tests
for.
The perl script then took all of the example runs, stripped out
duplicates, and then autogenerated a C file containing unit tests
for each of the examples (i.e pass in the input data and verify the
return value is the same as in the example run) Compiled/Linked/Ran
the unit tests and threw away ones which failed (i.e. get rid of
inputs which cause the function to behave non-deterministically)
I have a lot of legacy code of all kinds in the languages Python and Fortran. The article is from 2007. Is there anything like this implemented in current Unit testing frameworks?
How would i go about writing such a script?
Very C-like. Also, OS dependent, I think (Solaris debugger)? I'd say you should look at "record/capture and playback" tools, though somehow I think the "generate" part never really took off.
Python's testing tools taxonomy would be a great place to start. I'd say you either record your way through application using Selenium or Dogtail. The link takes you right to that section, Web testing tools, but check others as well: fuzzy testing is a technique similar to Golden Master, which sometimes may help with legacy apps, and is a "record / playback" technique. Feathers calls such tests "characterization" test, for they characterize legacy system's behaviours.
Very good point in article you cite:
Have a look at your own source code repository and see which
functions/classes have had the most bugfix checkins applied, 80% of
bugfixes tend to be made to about 20% of the code. There’s sound logic
behind this – often that 20% of the code is poorly written with dozens
or hundreds of “special case” hacks.
This is where I'd actually start. Have you got these parts identified? Simple Git/SVB log usage scripts and coverage tools section from the taxonomy would come in handy with this.
Unfortunately more than that I can't help you - my Python experience is limited and Fortran - non-existing.
I have the 'luck' of develop and enhance a legacy python web application for almost 2 years. The major contribution I consider I made is the introduction of the use of unit test, nosestest, pychecker and CI server. Yes, that's right, there are still project out there that has no single unit test (To be fair, it has a few doctest, but are broken).
Nonetheless, progress is slow, because literally the coverage is limited by how many unit tests you can afford to write.
From time to time embarrassing mistakes still occur, and it does not look good on management reports. (e.g. even pychecker cannot catch certain "missing attribute" situation, and the program just blows up in run time)
I just want to know if anyone has any suggestion about what additional thing I can do to improve the QA. The application uses WebWare 0.8.1, but I have expermentially ported it to cherrypy, so I can potentially take advantage of WSGI to conduct integration tests.
Mixed language development and/or hiring an additional tester are also options I am thinking.
Nothing is too wild, as long as it works.
Feather's great book is the first resource I always recommend to anybody in your situation (wish I had it in hand before I faced it my first four times or so!-) -- not Python specific but a lot of VERY useful general-purpose pieces of advice.
Another technique I've been happy with is fuzz testing -- low-effort, great returns in terms of catching sundry bugs and vulnerabilitues; check it out!
Last but not least, if you do have the headcount & budget to hire one more engineer, please do, but make sure he or she is a "software engineer in testing", NOT a warm body banging at the keyboard or mouse for manual "testing" -- somebody who's rarin' to write and integrate all sorts of automated testing approaches as opposed to spending their days endlessly repeating (if they're lucky) the same manual testing sequences!!!
I'm not sure what you think mixed language dev't will buy you in terms of QA. WSGI OTOH will give you nice bottlenecks/hooks to exploit in your forthcoming integration-test infrastructure -- it's good for that (AND for sundry other things too;-).
Automated testing seems to be as a very interesting approach. If you are developping a web app, you may be interested in WebDriver http://code.google.com/p/webdriver/
Since it is a web app, I'm wondering whether browser-based testing would make sense for you. If so, check out Selenium, an open-source suite of test tools. Here are some items that might be interesting to you:
automatically starts and stops browser instances on major platforms (linux, win32, macos)
tests by emulating user actions on web pages (clicking, typing), Javascript based
uses assertions for behavioral results (new web page loaded, containing text, ...)
can record interactive tests in firefox
can be driven by Python test scripts, using a simple communication API and running against a coordination server (Selenium RC).
can run multiple browsers on the same machine or multiple machines
It has a learning curve, but particularly the Selenium RC server architecture is very helpful in conducting automated browser tests.
Have a look at Twill, it's a headless web browser written in Python, specifically for automated testing. It can record and replay actions, and it can also hook directly into a WSGI stack.
Few things help as much as testing.
These two quotes are really important.
"how many unit tests you can afford to write."
"From time to time embarrassing mistakes still occur,"
If mistakes occur, you haven't written enough tests. If you're still having mistakes, then you can afford to write more unit tests. It's that simple.
Each embarrassing mistake is a direct result of not writing enough unit tests.
Each management report that describes an embarrassing mistake should also describe what testing is required to prevent that mistake from ever happening again.
A unit test is a permanent prevention of further problems.