Pankaj Nakhat's blog : All About Agile Testing

1)      As a QA I am involved early on user stories with product owners and business analysts.

2)      I give my feedback on user stories while they are being analysed.

3)      I work with business owners to define Acceptance tests for user stories before story is developed.

4)      I get involved in the story kick off with developer and business analysts to give QA input.

5)      I work with developers while he is developing story to give early feedback and automating acceptance tests.

6)      As a QA I encourage automation testing at all levels – Unit, Integration and UI. And I give feedback on all kinds of testing.

7)      I look at the software to give feedback, even before developer check in the code in source control.

8)      I make sure I understand the implication of the newly added feature and I look to improve automation tests by adding and refactoring existing tests.

9)      I use my QA skills to do effective exploratory testing to test the software beyond what an automated test would do.

10)   I constantly look to engage with different stakeholders on the project to improve my knowledge on the application which then I use to find gaps in the requirement.

I have been practicing Acceptance Test Driven development for years and since last few years I have been using BDD . BDD is a collaborative process to engage with business at the same time offering other benefits in Automation testing. There is no doubt BDD has changed the way we collaborate in an Agile team but also enhanced communication among stakeholders.

Some of the challenges I have seen while working on various projects, which I learnt from on my journey to do Behaviour Driven Development.

Process challenges

1. How to get business engage with this Idea of doing Testing  ?- I am sure this is easier said then done. How do I get my business stakeholder to engage in this process ? This scenario is like a sales scenario where you want to sell a great new product, but the problem is features are not immediately obvious.  Yes its about selling,  before its benefits becomes obvious.   Good thing about is BDD is natural language of expression, so getting involved is easier. BDD Scenarios makes so much sense in terms of a requirement. You can drive a product by an example in your own language, which is great. Its a great comfort when the Gap between requirement to development is negligible. Technical and Non technical people are speaking same language and you can validate your product in the very same language .
2. How to be consistent with the language  used in BDD ?  The whole idea of this process is about using a language which all stakeholders can understand and make sense from ? But one of the smells of BDD is using multiple style to define your behaviors. And result is simple, people are not consistent with their language of communication, so further consequence is mis communication and chaos.  Reason is simple “a” thing in my behaviors can mean few different thing. Other aspect of the same problem is different level of abstracted language people use, should I define my behaviors at 10000 ft or should it be more granular ? Well this depend  on the context, I don’t mind any approach where the language speaks the language of the problem we are trying to solve and its consistent.
3. Who should write the BDD ? I think it doesn’t matter who writes the BDD Scenarios. As long as different stakeholder participate in the process, and it is used as paradigm of communication it does not matter who writes it eventually. I have heard people getting in to discussion whether QA/BA/SME should write the acceptance tests. I think idea of doing BDD is to share the responsibility and getting everyone engaged in this process.
4. How to maintain my Acceptance tests going forward ?  I think the most important challenge in doing Acceptance Testing using BDD is, how does change in feature/code/data be managed. How the change should be managed in the acceptance tests? How not to fall in big trap of technical debt when big re factoring comes along and acceptance tests are well forgotten ? And I think most teams falls apart on these challenges more often then not. I have seen developers changing the acceptance tests without liaising with BAs and QAs when pressure of delivering eventually mounts. Similarly I have seen QA/BAs not managing to change the acceptance tests to reflect the change in feature. So basically challenge is to follow the same process when things are tight. Every change in feature should be managed as if its a new feature and should have a respective acceptance tests, agreed together with different SMEs.  Individual people on team should refrain making changes to acceptance tests, event though it may look trivial.Sometime it is difficult to trace acceptance tests against requirements, as in there is no obvious tooling in place to support trace the requirements against the tests. And if you are working on a large project it can be a nightmare to do so.  In this case having some means of tracing mechanism in place definitely helps. It can be as simple as a spreadsheet or usage of story maps if you are using tools like jbehave.

Software testing has changed dramatically since the inception of Agile and Scrum. These methodologies are now widely accepted and without any doubt offers better way of delivering software. However Agile and Scrum also highlighted some of the challenges in the testing processes which possessed greater risk of failure , if not thought about in the over all process.

Some of these challenges becomes inevitable , if some of the critical processes in scrum like story writing, estimation and TDD falls apart.

Some of the high level Testing challenges in Agile and Scrum

And I think these are some of the core points if taken care can make huge impact on the overall testing process and for that matter in delivery.

Having the right Requirement

I see this as a biggest pain points and root of other problems. Having a good story can change how the requirement is delivered and tested. I refer this from the book “User stories Applied”, that a story should inhibit following features INVEST

I – INVEST
N – NEGOTIABLE
V – VALUABLE
E – ESITMATABLE
S – SMALL
T – TESTABLE

Each of these characteristics are self explainable and however trivial they may sound, they have huge impact on the quality of the requirement delivered in the end.

For instance a large story which can pretty much eat good few sprints is more likely to introduce bugs, as it will not be ready (DONE) by end of the sprint for review. And also large stories create confusion for developer as they get out of context very soon, similarly its nightmare for testing as well since focusing on a big chunk of feature creates opportunity for bug and overlooking.

Also, a badly written story can lead a wrong estimation which can cause capacity issues for testing as chances are some stuff will have to rolled over to next sprint or rushed through. Not only that a badly estimated story can lead to greater risks in further sprints, as it will be eat up some of the valuable time needed to prepare for next sprint’s requirement.

I strongly believe that testers in Agile project can add lot of value by looking at requirements nice early and reviewing it from a tester’s point of view. Some of the questions a tester need to asks are

- Does this acceptance criteria makes sense?
- How can I test it? (What, where and When )
- Is it not a duplication of work we did earlier ?
- Is this story too small or big ?

Automate Automate Automate

Automation will help maintaining a tight grip on the changes. Automation should be done at all levels and with involvement of QAs). Why do I say that? As Agile accepts changes quiet frequently and focuses on delivering features every sprint. it is obviously you would not get much time to plan, write and manage those tests, hence it is imperative that test automation is thought about while writing the code. And QAs should play important role in discussing, planning and thinking about those tests upfront, and discuss them with BAs and Developers. Build as much input as possible and then take right decisions as in which tests to be automate as unit, integration or UI tests.

Automation are assets and needs to be maintained, re factored and looked after like production code. I have seen automation tests becoming big bottleneck to maintain and trace and these problems can be solved by doing some simple things regularly .

- Make sure your automated tests are traceable against requirements (There are tools available to do that , if not use spreadsheet ). This will make sure you can track the tests against a requirement and if needed re factor, extend or retire it.
- Automated at all levels – UI tests are expensive and not conducive to tests each and everything. Good unit and integration test helps in building better coverage and helps in giving faster feedback. End to End tests are important and should be exclusive and focus on workflow and driving application end to end.
- Get testers involved in writing unit and integration tests, they will provide valuable feedback.

Defining the DONE Criteria

I think this is the first and foremost process to be in place from Day1.

What is DONE Criteria ?

DONE criteria are high level check list items which needs to be satisfied in order to claim a story is DONE. DONE Criteria should have technical, process, functional, non-functional, configuration aspects of the software. DONE criteria forces teams to take holistic view on the stories rather then code completing the feature.

Some of the common aspects of done criteria

- All acceptance criteria met. (Functional)
- Performance not degraded. (Non Functional)
- Code coverage achieved and all acceptance tests automation. (Technical )
- All build passed and application deployed on UAT/Staging environment. (Environment/Configuration)
- Exploratory testing performed and QA signed off.
- Demoed to stake holders on proper environment. (Process)
- All known bugs closed and tested (Process)

By doing this teams will make sure that software produced is of great quality and all aspects of the applications are taken care. Doing this repeatedly will make sure testers are not doing catchup but they are part of the story and sign off process. Software produced in the end can be released and should be production ready.

Exploratory Testing

This is the most misunderstood and ignored aspect of Agile Testing. Exploratory Testing advocated testes/BAs to test a feature by exploring it beyond the test scripts. At the same time it is not about clicking around the application without proper context.

Exploratory testing should be done every sprint on set of features on a proper environment. Testers should use all their skills to think out of the box and try to fearlessly test the system within the context of the features, however not limited to that feature.

Success of Agile Testing depends on working collaboratively between different stakeholders of the project and learning and rectifying quickly.

If you are using Jbehave and wonder how to execute your scenarios with different set of data without repeating the steps, so that your scenarios don’t blow up repeating same steps with different set of data.

You can also refer to this scenario as data driven testing, as each of the row in your data table becomes one test case.

In the example below lets say I am testing a calculator sum function, and for that I need to have several test cases to test with different data. So basically steps looks like this, however we need to give different input and assert output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

 
 
Scenario: As as user I want to test calculator sum function 
Meta:
@author pankaj
 
Given calculator takes <input1> and <input2>
Then the sum is <sum>
 
Examples:
|input1|input2|sum|
|2|3|5|
|4|5|9|
|10|12|22|

And steps implementation for the above written steps will need to use Named parameter. So basically Named parameter (Which is usually in <>), will be replaced by the data in the table below the steps in the story. In the above example there are three named parameter in my story.

In the steps implementation, you step needs to explicitly use the named parameter in the method. Using named parameter in the steps tells jbehave that it needs to replace this with the data in the table specified below/or in another file.

Steps class would look like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package org.qainfolabs.jbehave.steps;
 
import java.math.BigDecimal;
 
import junit.framework.Assert;
 
import org.jbehave.core.annotations.Given;
import org.jbehave.core.annotations.Named;
import org.jbehave.core.annotations.Then;
import org.qainfolabs.app.Calculator;
 
public class CalulatorSteps {
 
	private BigDecimal input1;
	private BigDecimal input2;
 
	@Given("calculator takes <input1> and <input2>")
	public void readFruits(@Named("input1") BigDecimal input1,
			@Named("input2") BigDecimal input2) {
		this.input1 = input1;
		this.input2 = input2;
	}
 
	@Then("the sum is <sum>")
	public void doSum(@Named("sum") BigDecimal sum) {
		Calculator calc = new Calculator();
		Assert.assertEquals(sum, calc.sum(input1, input2));
	}
}

After the scenario is executed Jbehave will produce a detailed report of the scenario, showing the steps repeating with replaced parameter with each data set.

Testers on agile projects have to often deal with multiple tasks in an iteraton.

Behaviour driven development is now widely accepted process in Agile community. Now the question to be asked what is that make this simple discipline powerful and effective? And how to use the power of communication?

Agile development practices have been preaching for years on the aspect of writing acceptance tests for the requirements upfront. These acceptance tests used to be written in various and tools. Fitnesse definitely levraged among all other tools, simply because it enabled developement teams to use power of communication. No longer there was divide between tests being too technical for the people to read who actually cared about them, ofcourse I am talking about business , BAs, Testers.

Behaviour Driven Development is refined form for writing Acceptance tests, where requirements are expressed in natural language of communication (English, French etc) and implemented in the same form. This enabled no loss of communication from Requirements to Tests. And also it forces business to be involved with requirement throughout, as they have visibility on what they asked WORKS !

Tools like Cucumber, Concordian, Jbehave, Easyb enabled a uniform language of communication for Requirements and Tests , which looks like a simple discipline but it is very powerful.

Not only business gets lot of value from it at the same time development teams are equally benefited, as they have clear understanding of expected software behaviour in the form of executable requirement. Also, the power of BDD lies in thinking about business problem and expressing the intent without thinking about the implementation details of the problem.

A BDD can express the intent in form of Simple Sentences, Tables, Examples etc. The most common form of Acceptance testing format is Given – When – Then, which is widely being used in BDDs.

Given – Is a prerequisite or precondition for an action.
When – A action or Event happens.
Then – Results of the action.

Example – Saving bank account withdraw scenario

Given there are 1000$ in saving bank account number XXXXXXX
When a cash with drawl of $900 is done
Then available balance is $100

The above example is a simple explanation of How a Given When Then can express an intent in consistent and effective way.

DSLs have been buzzing around for a while now, and I thought of putting some thoughts together.

DSL – Domain Specific Languages, what does it mean for Testing? What are the benefits..

Over the years automation testing has changed from record and play back mechanism to more structured, framework oriented approach. Benefits were evident Test were readable, Scalable and Maintainable.

But there was more needed to be done, as most automation tests would turn in to over head rather then assets.  Framework based approach was good..but came up with its own sets of problem. Some of them were Code like tests, which meant they were not human readable and needed either hours of

documentation to make them explicit to humans.  So if the application changed underneath both test and documentation needed to change. And after a while these will not match each other by any means.

Frameworks like keyword driven testing enabled naive to write Tests, however created the problem of highly non abstracted tests.

Since the introduction of methodologies like Agile, it was inevitable that those problem needed to be sorted fast. Tests needed to be changed quickly with often changing requirements.

DSLs allows you to write tests in your own domain..be it banking, Online Gaming or Travel Industry.  Which means tests are human readable and highly contextual to domain.

For example – I need to write a Test for a baking application for testing a Scenario of Transfer of a fund from Account A to Account B.  Note, I am talking in terms of any specific tool here, and that I will keep up for a separate post.

<Test>

1. Login in my online banking application with user “A”
2. Go to transfer funds page
3. Check initial balance it should be “£ 500″
4. Transfter amount “£100″ to account “B”
5. Check final balance , it should be “£ 400″
6. logout from the application.

</Test>

In above example , Test exactly looks like manual test scripts but there nature is Executable in whatever technology and tool we choose.

What problems does it solve?

• Maintenance is easier now as test are human readable.
• Adding and Amending a step in Test just requires finding the right step, rather then changing the implementation  (In Most of the cases).
• For example – In above Test I need to add a new requirement of a welcome page after login. We simply introduce another DSL step after step1 “Verify user is shown welcome page with him name on it”, and rest of the test is still unchanged.
• Test can be extended easily to do more comprehensive end to end test.
• If the application changes underneath in terms of UI Objects , tests will remain unchanged, only the underlying implementation will change.
• If a business process changes completely, changed can be accommodated easily by changing the DSL definition. For example in above Test the transfer process changes and now required another level of authentication before transferring the funds. We would change the definition of Step4 to ‘Transfter amount “£100″ to account “B” and enter authentication password for user “A”‘

DSLs can also come with its own challenges, which I would like to discuss in my next post.

[quicksubscribe]

A Tester’s perspective on agile projects

Agile is no longer a buzzword or an unknown territory in the industry. Agile has come forward leaps and bounds since last few years and has seen a great transition from a newbie to a widely accepted methodology.

Testing on agile project was a paradigm shift for the traditional testing roles.  It needed a change in tester’s attitude from a laid back relay race oriented approach to an upfront involved role.  No longer had QAs needed to get something over the line to know the quality, as approach was rather to do the things right first time. But it’s easier said than done, how does it happen in reality? Does it actually happen?

Agile is a philosophy which believes in delivering constant value to customers incrementally and frequently, which is based on simple values of communication and feedback.  These two ingredients are the ingredients of a successful agile recipe.

A tester has an important role to play in agile projects throughout different phase of iterations. Iteration is generally a two week long activity where small requirements are analysed, coded and tested.

What QAs needs to do? I am outlining few of the basic QA activities in agile projects. Let’s simply break the QA activities in three phases in agile projects. However, this is no golden rule and it can be flexible according to the project situation. Agile QAs role is not limited to set of pre defined processes, as the methodology they are expected to play an agile role as per the situation.

Pre iteration : This is the time where requirements are analysed in detailed by the BAs and acceptance criteria are detailed out for that story. As QAs are immediate consumers of those requirements, so it is really important to verify these requirements early and often.

Story verification (Requirement Verification) : Agile testing is all about giving feedback early, not necessarily only by testing the requirements, but doing requirements testing early on. What should one be focusing on while doing requirements testing?

Looking at requirements for Clarity and Testability  - QAs really need to look at the requirement / stories upfront to figure out that requirement are unambiguous and testable.

Unambiguous- I think unambiguous in agile has little different context compared to typical waterfall project.

- Requirement should be small enough to make sense in the context.

- Acceptance criteria (Stories are generally broken in to acceptance criteria) should not be duplicate or overlapping from different stories.

However, doing that can be really difficult and can only be achieved with really good communication between Dev/BA/QA.

Testable: Testability aspect of the story requires QA to scan through the story to see what needs to be done to test the story. These factors can generally be.

- Finding hidden requirements

- Environment

- Test data

- Dependency on other requirement.

Getting these details early helps the story to be prioritised accordingly in the backlog, and allows smooth execution of the story in the iteration.

QAs also do participate in the iteration planning meeting to help estimating and giving testing perspective to the team to come up with developer estimate. QAs have big role to play in the iteration planning, as some of the implicit requirements gets escalated by

QAs.

a) QA Activities In  The Iteration

Once QAs are happy with the acceptance criteria of the story they can work towards defining the acceptance tests for the story. Acceptance tests are requirements in terms of a test, so that it can be executed before hand to know what is expected out of requirements. These acceptance tests are generally automated and used by developers to drive development,  Acceptance tests should not be covering too many corner case scenarios as this would create unnecessary delay in getting the story done by the developers and may end up in producing too many similar automated tests.

People often fail to understand that acceptance testing in agile projects is different from traditional projects, as in agile projects it happens before the software is delivered unlike traditional project where it happens at end of the software lifecycle. In agile projects acceptance tests are generally automated so that it can be run as regression tests.

Automation testing is really important for any agile project as frequent builds requires a short feedback cycle hence regression testing needs to be quick and accurate. Automation testing is different from traditional automation testing. In agile projects automation testing is practised by all levels be it Developers, QAs or BAs. It does not necessarily mean that everyone needs to be writing the test code. Involvement from everyone on automation testing increases the relevance of the tests and often helps in finding and implementing the right tests.

It’s always been debatable as who owns the automation tests in agile projects? And for me it’s more of a responsibility then a role.  In my experience most effective automation tests are achieved when Dev/QAs work together on it.

Use Automation to the advantage not for sake of doing it

Automation in agile is quiet controversial, as people try to automated everything and end up in a trap of having a long feedback cycle. Automation is meant for giving early feedback on the latest code, and automation should be limited to what is worth Automating and what is not. Every automation test written has a cost against it. The cost can be seen as

Cost of continuously running it (Increased feedback cycle). This cost of automation should be compared against cost of not running it. So, a question needs to be asked what if a test is not automated. What we going to lose, what would be the cost of fixing the stuff around the code for which we are losing the coverage? Is it cheaper to test manually?

And it might not be straight forward finding the worth of a test. It’s a contextual decision and also depends on size of the project and number of people involved in it. Simply putting this in other words mean Longer feedback cycle = More people losing time in getting instant feedback.

The typical QA activities in the iteration are to continuously measure the Quality of the software. QAs participate in the story handover to the developers to make them understand the testing requirements of the story, so that developers are enabled for Test Driven Development (TDD) for their code, which is a backbone for agile development practices. Also handing over the acceptance tests and making developers understand about the testability aspect of the story often catches the most common defects even before they are delivered to QAs.  These activities require a high level of communication with the developer and BAs to clarify things on the fly and making sure we are building the product right first time. QAs may also have to pair with developers work together on the story or tests for the story to get the better understanding of the requirements.

QAs try to resolve as much issue before hand by actively participating in the overall process.  It also becomes imperative that once the story is delivered, it is tested properly with all the aspects and ideally on a proper environment.  Once the QAs are happy with the stories/requirements, they sign off that piece of work in a typical manufacturing way and hand it over for further process in the line which generally a showcase to the business.

As a QA it is also important to think beyond the written requirements and to have a desire to break attitude, and that’s where QAs do exploratory testing to execute ‘out of the box’ scenarios and also do negative testing to make sure software is really robust. Exploratory testing is a not at all about executing a pre defined testing scenarios, it is more of exploring of software beyond test cases but at the same time keeping the focus around some specific requirements.

I have tried covering most of the aspects of the basic QA activities of a QA in an agile project, however it is not limited to these activities and QAs should play more collaborative role as and when needed to be able to provide more visible and transparent feedback.

Download the article

I want to make few points quickly in favor of that.

1) No support for programming languages like Java, .Net so that it allows it to be used as an acceptance testing tool.
2) VBSCript doesn’t help in every-one’s participation in the automation process, which is a key in agile projects.
3) No IDE support : QTP’s IDE is no way near Ideal hence makes maintainable a tough battle.
4) NO support for standard IDE like eclipse .
5) Multi browser support is limited.
6) Cant execute scripts on Linux and Mac platforms.
7) No continuous integration support, makes it rally tough to be used as an acceptance testing tool.

8)Test execution is slower compared to other tools, hence makes feedback cycle slower as well.

9) And last but not the least , Bloody expensive

P.S : I am not anti QTP and love QTP as a tool, but QTP is not fit for agile process ..until HP realises that and comes up with solution to rectify the problems mentioned above…..oops that means HP may end up building (Acquiring) a new tool altogether.

Disclaimer : Opinions in this post are my personal opinions and does not intend to pursue any commercial interest. If you think otherwise please ignore the post.

I am happy to receive  any feedback and comment on this post.

Twist is going GA on 27th April.