I'm teaching myself backend and frontend web development (I'm using Flaks if it matters) and I need few pointers for when it comes to unit test my app.
I am mostly concerned with these different cases:
The internal consistency of the data: that's the easy one - I'm aiming for 100% coverage when it comes to issues like the login procedure and, most generally, checking that everything that happens between the python code and the database after every request remain consistent.
The JSON responses: What I'm doing atm is performing a test-request for every get/post call on my app and then asserting that the json response must be this-and-that, but honestly I don't quite appreciate the value in doing this - maybe because my app is still at an early stage?
Should I keep testing every json response for every request?
If yes, what are the long-term benefits?
External APIs: I read conflicting opinions here. Say I'm using an external API to translate some text:
Should I test only the very high level API, i.e. see if I get the access token and that's it?
Should I test that the returned json is what I expect?
Should I test nothing to speed up my test suite and don't make it dependent from a third-party API?
The outputted HTML: I'm lost on this one as well. Say I'm testing the function add_post():
Should I test that on the page that follows the request the desired post is actually there?
I started checking for the presence of strings/html tags in the row response.data, but then I kind of gave up because 1) it takes a lot of time and 2) I would have to constantly rewrite the tests since I'm changing the app so often.
What is the recommended approach in this case?
Thank you and sorry for the verbosity. I hope I made myself clear!
Most of this is personal opinion and will vary from developer to developer.
There are a ton of python libraries for unit testing - that's a decision best left to you as the developer of the project to find one that fits best with your tool set / build process.
This isn't exactly 'unit testing' per se, I'd consider it more like integration testing. That's not to say this isn't valuable, it's just a different task and will often use different tools. For something like this, testing will pay off in the long run because you'll have piece of mind that your bug fixes and feature additions aren't impacting your end to end code. If you're already doing it, I would continue. These sorts of tests are highly valuable when refactoring down the road to ensure consistent functionality.
I would not waste time testing 3rd party APIs. It's their job to make sure their product behaves reliably. You'll be there all day if you start testing 3rd party features. A big reason to use 3rd party APIs is so you don't have to test them. If you ever discover that your app is breaking because of a 3rd party API it's probably time to pick a different API. If your project scales to a size where you're losing thousands of dollars every time that API fails you have a whole new ball of issues to deal with (and hopefully the resources to address them) at that time.
In general, I don't test static content or html. There are tools out there (web scraping tools) that will let you troll your own website for consistent functionality. I would personally leave this as a last priority for the final stages of refinement if you have time. The look and feel of most websites change so often that writing tests isn't worth it. Look and feel is also really easy to test manually because it's so visual.
Related
I am in charge of testing of a web application using Selenium Webdriver with Python. Over the past year I created a large script (20K+ lines) where each test is a separate function. Now my boss wants me to document my tests explaining in plan English what each test does. What tool would you recommend to document the steps your tests make?
I think this is a great question. Many people and companies don't bother managing their existing tests properly which leads to redundant and repeated code without having a clear idea what is covered by automated tests.
There is no single answer to this question but in general you can consider the following options:
Testing framework built in reporting. In Java, for example, you have the unit testing libraries like jUnit and TestNG. When they run, they generate certain output that can later be formatted and reviewed as the need arises. I am sure there an implementation of unit testing framework like this in Python too.
You can also consider using a BDD tool like Cucumber. This is a bit different and might not be suitable in certain cases when the tests are low level system checks. It can however help you organize your test scenarios and keep them an a readable form. It is also very good for reporting to a non-technical person.
I need help coming up with a good order to build a web application in. I am building it in django. Its a web app thats going to have alot of things like voting, liking, creating lists, commenting etc. So to keep my options open in the future for simplicity of building a multi-platform webapp, I was thinking building the website "API up".
Not sure if I am using the right terms, but essentially I want to build all the function parts using REST e.g. to like a comment would be url.com/api/like/commentID/user/Pass where user pass is optional since it can use a web session if it has one. Once I build the functions (like, comment, etc) I plan on building views and templates on top of the api (e.g the like button is a simple href to the api like function).
I think this is kinda different from my typical way in the sense that I'd usually have to have multiple methods one for an api type of thing, and one for view/templates forms and such.
Essentially I am asking, would building an website on top of your API be a good idea?
Keep it simple and stupid.
Start by writing a simple list of user stories - don't over-design your project. Group them by app. Try to divide your project in many apps but not too many.
Controlling complexity is the essence of computer programming. — Brian Kernigan
Start your project with Pinax or something like that. So that you get all the user management/settings/registration/etc/etc ... out of the box. You can now start focusing on what makes your website different.
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. — Brian W. Kernighan and P. J. Plauger in The Elements of Programming Style.
Code the first version of the first app, that should include a few models, urls, views, templates and staticfiles.
There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. — C.A.R. Hoare, The 1980 ACM Turing Award Lecture
Postpone features which doesn't fit into the standard Django way of coding (e.g. requires hacks, code bloat, etc ..). Rely on external apps as much as possible, improve them rather than re-inventing the wheel.
Life is too short to run proprietary software. — Bdale Garbee
Code the other apps, again try to keep them simple if something is too complex then postpone it.
The cheapest, fastest, and most reliable components are those that aren't there — Gordon Bell
You should then have a first working version of your project. Time has gone by while you did that. Depending on the amount of time left, decide which of the features you postponed are most important for the first release.
The computing scientist's main challenge is not to get confused by the complexities of his own making." — E. W. Dijkstra
Do the first release, at least private, your project should be in production and you should have succeeded in deadlines by now. The code should be elegant and the user stories implemented somehow with simplicity in the code. Set up a rolling release system - that is make a script that can push from test to production.
And folks, let's be honest. Sturgeon was an optimist. Way more than 90% of code is crap." — Al viro
You're now ready to work on the more complex stuff. But don't forget: your website's usage (users and business model) should now be able to dictate the priorities. You should have a sane project that evolves by itself now.
It's a curious thing about our industry: not only do we not learn from our mistakes, we also don't learn from our successes." — Keith Braithwaite
I am working on developing two systems:
A system that will constantly retrieve economic data from a 3rd party data feed and push it into a MySQL DB (using sqlalchemy)
A server that will allow anyone to query the data in the db over a JSON AJAX API (similar to Yelp or Yahoo API for example)
I have two main questions:
Which Python framework should I use in 2)? Pyramid is my first choice, but if you strongly suggest against it or in favor of something else like Django or Pylons I am definitely wiling to consider it.
Should I develop the two system separately? Or should 1) be a part of 2), running within the framework (using crontab or celery for example)?
Depends on what stage you are at, I would suggest to develop 2 systems because the load to pull data from 3rd party and the load to handle the API would be different. You can scale them into a different types of nodes if you want.
Django-Tastypie (https://github.com/toastdriven/django-tastypie) is not bad, it supports all JSON, XML and YAML. Also you can add OAuth easily. Though, Django itself maybe a bit heavy for your needs at this time.
You might want to check out web2py's new functionality for easily generating RESTful API's, particularly its parse_as_rest and smart_query functions. You might also consider using web2py's database abstraction layer to handle #1.
If you need any help, ask on the mailing list.
I agree with Anthony, you should look at Web2Py. It is very easy to get started, very low learning cure and easy to deploy on many systems including Linux, Windows and Amazon.
So far I have found nothing that Web2Py can not do. But more importantly it does things how you would think they should be done, so if you are not sure, very often a guess is good enough and it just works. If you do get stuck, it has by far the best and most up to date documentation for any Python Web Framework.
Even with all it's great features, easy use and up to date documentation, you will also find that the web2py user group on Google, is like having a paid for help desk staffed 24 hours a day. Most questions are answered with a couple minutes and Massimo (The original creator of Web2Py) goes out of his way not only to help, but to implement new ideas, suggestions and bug fixes within days of them being raised in the group.
There is a way to login into adwords without use api, in order to parsing it to get informations about: campaings, keywords clicks and so on...
something like:
url = 'https://www.google.com/accounts/ClientLogin'
data = [('Email', 'foo#gmail.com'), ('Passwd', 'secret'),
('accountType', 'GOOGLE'), ('service', 'adwords')]
urllib.urlopen(url, urllib.urlencode(data))
startpage = urllib.urlopen('https://adwords.google.com.br/select/snapshot').readlines()
I am a GUI test automation programmer, and som my experience is highly relevant to this question. I create tests designed to interact with a system, as a user would, at the request of my employer, who develop the software I am trying to test. This is an extremely complicated process even if you have to hand the entire source code and implementation documentation - in fact it's a full time job, and it's how I make my living.
So bearing tis in mind, you can do this, but please save yourself the hassle and don't. You'd just be creating another API, except with layers of leaky abstraction introduced by the Adwords' UI that will make it a nightmare to maintain. Someone else has done the hard work for you already, why do you want to reinvent the wheel when you're not even using as good tools as the first guy?
You say that:
Each AdWords API operation performed consumes a certain number of API
units, that's why I don't want to use adwords API
Paying $0.25 per 1000 units will be cheaper to maintain that your custom API will be to develop AND maintain, and you don't even have the resources to hand which I would working in my day job; Not to mention the fact that I imagine there will be some sort of bot-detection on the adwords website.
Do you feel like writing a Captcha decoder?
Just optimise your code to use as few units as possible, have one less coffee every week, and put your feet up instead of battling against google like this.
Or if you must, make it open source so others can learn from it...
I have the 'luck' of develop and enhance a legacy python web application for almost 2 years. The major contribution I consider I made is the introduction of the use of unit test, nosestest, pychecker and CI server. Yes, that's right, there are still project out there that has no single unit test (To be fair, it has a few doctest, but are broken).
Nonetheless, progress is slow, because literally the coverage is limited by how many unit tests you can afford to write.
From time to time embarrassing mistakes still occur, and it does not look good on management reports. (e.g. even pychecker cannot catch certain "missing attribute" situation, and the program just blows up in run time)
I just want to know if anyone has any suggestion about what additional thing I can do to improve the QA. The application uses WebWare 0.8.1, but I have expermentially ported it to cherrypy, so I can potentially take advantage of WSGI to conduct integration tests.
Mixed language development and/or hiring an additional tester are also options I am thinking.
Nothing is too wild, as long as it works.
Feather's great book is the first resource I always recommend to anybody in your situation (wish I had it in hand before I faced it my first four times or so!-) -- not Python specific but a lot of VERY useful general-purpose pieces of advice.
Another technique I've been happy with is fuzz testing -- low-effort, great returns in terms of catching sundry bugs and vulnerabilitues; check it out!
Last but not least, if you do have the headcount & budget to hire one more engineer, please do, but make sure he or she is a "software engineer in testing", NOT a warm body banging at the keyboard or mouse for manual "testing" -- somebody who's rarin' to write and integrate all sorts of automated testing approaches as opposed to spending their days endlessly repeating (if they're lucky) the same manual testing sequences!!!
I'm not sure what you think mixed language dev't will buy you in terms of QA. WSGI OTOH will give you nice bottlenecks/hooks to exploit in your forthcoming integration-test infrastructure -- it's good for that (AND for sundry other things too;-).
Automated testing seems to be as a very interesting approach. If you are developping a web app, you may be interested in WebDriver http://code.google.com/p/webdriver/
Since it is a web app, I'm wondering whether browser-based testing would make sense for you. If so, check out Selenium, an open-source suite of test tools. Here are some items that might be interesting to you:
automatically starts and stops browser instances on major platforms (linux, win32, macos)
tests by emulating user actions on web pages (clicking, typing), Javascript based
uses assertions for behavioral results (new web page loaded, containing text, ...)
can record interactive tests in firefox
can be driven by Python test scripts, using a simple communication API and running against a coordination server (Selenium RC).
can run multiple browsers on the same machine or multiple machines
It has a learning curve, but particularly the Selenium RC server architecture is very helpful in conducting automated browser tests.
Have a look at Twill, it's a headless web browser written in Python, specifically for automated testing. It can record and replay actions, and it can also hook directly into a WSGI stack.
Few things help as much as testing.
These two quotes are really important.
"how many unit tests you can afford to write."
"From time to time embarrassing mistakes still occur,"
If mistakes occur, you haven't written enough tests. If you're still having mistakes, then you can afford to write more unit tests. It's that simple.
Each embarrassing mistake is a direct result of not writing enough unit tests.
Each management report that describes an embarrassing mistake should also describe what testing is required to prevent that mistake from ever happening again.
A unit test is a permanent prevention of further problems.