I have been working on Agile projects for years know and I have seen it evolving over the years. I have made mistakes in the past and always learnt the Hard way. I thought it will be worth sharing my experience and learning in Agile testing.

Mistake 1: Not getting involved early enough in the Sprint. So often I have seen teams falling in this trap. And consequences are expensive – QAs not getting involved early fail to provide important feedback on the requirement.  Its that missing perspective which can add lot of value early on. Viz looking at the testability of the requirements, and collaborating with different stakeholders to gain valuable context in to the story. I can go on here and talk about so many obvious and non  obvious cost of not doing this right.

However, there is one aspect worth talking about – Why do teams fall in this trap , where QAs are not getting involved early enough? I am highlighting someone of them from my experience

• Catching up on testing from previous iteration – Its a common symptom where teams leave QAs to deal with the stories delivered at the fag end of the iteration.  And I personally think that leaving testing problems to just QAs in the team is a sign of broken process. Team should look to rectify this and come together to try to get all stories to done (Properly ) before the end of the sprint. That may require people from cross discipline need to roll their sleeves up.  Doing this will enables team to have a greater disciplines and control over their iterations and most importantly avoid that ‘catch-up’ mode problem.
• Testing as seen outside the scope of sprint – And again I have seen it often in projects, where progress is shown/claimed based on development complete stories. However in true sense it does not matter if story is dev complete, Half QA done or not started at all. In Scrum/Agile model stories are considered Done only when all required phase of work is done or in other terms definition of done is met (DOD I will referring it as DOD in rest of my post). Coming out of this mindset helps team to focus on real value of work rather then claiming false sense of progress.
• Loose definition of Done – I spoke about definition of done in previous point. Now how important is it to have definition of done and what consequences can it bring if not well defined? I am not going to talk about what DOD is and how is it defined.  DOD is a yard stick against which a requirement is said to be measured and upon successful meeting the criteria, it is said to be DONE in the process.  DOD usually have attributes like – Acceptance Tests Passed, Exploratory Testing done, No Red builds, NFR met etc etc.  So in first place not having a DOD does not help team in measuring themselves against some standard quality tools.  So, it does help teams to be consistent with their approach in terms of delivering different piece if functionality. Result is obvious – Inconsistency in Quality and Communication breakdown which then leads to poor  product.
• Other scenario being not following DOD religiously for various reasons like delivery pressure, assumption of quality or may be anything else. which results in similar consequences as mentioned above.

I will continue with other reasons on my subsequent post.

2)      I give my feedback on user stories while they are being analysed.

3)      I work with business owners to define Acceptance tests for user stories before story is developed.

4)      I get involved in the story kick off with developer and business analysts to give QA input.

5)      I work with developers while he is developing story to give early feedback and automating acceptance tests.

6)      As a QA I encourage automation testing at all levels – Unit, Integration and UI. And I give feedback on all kinds of testing.

7)      I look at the software to give feedback, even before developer check in the code in source control.

8)      I make sure I understand the implication of the newly added feature and I look to improve automation tests by adding and refactoring existing tests.

9)      I use my QA skills to do effective exploratory testing to test the software beyond what an automated test would do.

10)   I constantly look to engage with different stakeholders on the project to improve my knowledge on the application which then I use to find gaps in the requirement.

I have created this project  to give some starting point for people who want to use Jbehave , Selenium with Page objects. Please clone this github repository.

Some of the challenges I have seen while working on various projects, which I learnt from on my journey to do Behaviour Driven Development.

Process challenges

1. How to get business engage with this Idea of doing Testing  ?- I am sure this is easier said then done. How do I get my business stakeholder to engage in this process ? This scenario is like a sales scenario where you want to sell a great new product, but the problem is features are not immediately obvious.  Yes its about selling,  before its benefits becomes obvious.   Good thing about is BDD is natural language of expression, so getting involved is easier. BDD Scenarios makes so much sense in terms of a requirement. You can drive a product by an example in your own language, which is great. Its a great comfort when the Gap between requirement to development is negligible. Technical and Non technical people are speaking same language and you can validate your product in the very same language .
2. How to be consistent with the language  used in BDD ?  The whole idea of this process is about using a language which all stakeholders can understand and make sense from ? But one of the smells of BDD is using multiple style to define your behaviors. And result is simple, people are not consistent with their language of communication, so further consequence is mis communication and chaos.  Reason is simple “a” thing in my behaviors can mean few different thing. Other aspect of the same problem is different level of abstracted language people use, should I define my behaviors at 10000 ft or should it be more granular ? Well this depend  on the context, I don’t mind any approach where the language speaks the language of the problem we are trying to solve and its consistent.
3. Who should write the BDD ? I think it doesn’t matter who writes the BDD Scenarios. As long as different stakeholder participate in the process, and it is used as paradigm of communication it does not matter who writes it eventually. I have heard people getting in to discussion whether QA/BA/SME should write the acceptance tests. I think idea of doing BDD is to share the responsibility and getting everyone engaged in this process.
4. How to maintain my Acceptance tests going forward ?  I think the most important challenge in doing Acceptance Testing using BDD is, how does change in feature/code/data be managed. How the change should be managed in the acceptance tests? How not to fall in big trap of technical debt when big re factoring comes along and acceptance tests are well forgotten ? And I think most teams falls apart on these challenges more often then not. I have seen developers changing the acceptance tests without liaising with BAs and QAs when pressure of delivering eventually mounts. Similarly I have seen QA/BAs not managing to change the acceptance tests to reflect the change in feature. So basically challenge is to follow the same process when things are tight. Every change in feature should be managed as if its a new feature and should have a respective acceptance tests, agreed together with different SMEs.  Individual people on team should refrain making changes to acceptance tests, event though it may look trivial.Sometime it is difficult to trace acceptance tests against requirements, as in there is no obvious tooling in place to support trace the requirements against the tests. And if you are working on a large project it can be a nightmare to do so.  In this case having some means of tracing mechanism in place definitely helps. It can be as simple as a spreadsheet or usage of story maps if you are using tools like jbehave.

Jbehave reports can be ebabled to show something which is called Storynavigator. Its a client side javascript library which enabled browser to read meta data generated by cross referencing in Jbehave. Cross referencing can be enabled very easily from the configuration and it then generated the meta data which can be used by this story navigator.

How to enable cross referencing in Jbehave configuration?

 @Override
    public Configuration configuration() {
    	CrossReference xref = new CrossReference();
 
        Class embeddableClass = this.getClass();
        // Start from default ParameterConverters instance
        ParameterConverters parameterConverters = new ParameterConverters();
        // factory to allow parameter conversion and loading from external resources (used by StoryParser too)
        ExamplesTableFactory examplesTableFactory = new ExamplesTableFactory(new LocalizedKeywords(), new LoadFromClasspath(embeddableClass), parameterConverters);
        // add custom coverters
        parameterConverters.addConverters(new DateConverter(new SimpleDateFormat("yyyy-MM-dd")),
                new ExamplesTableConverter(examplesTableFactory));
        parameterConverters.addConverters(new MyConverter());
        parameterConverters.addConverters(new StudentConverter(new ExamplesTableFactory()));
 
		return new MostUsefulConfiguration()
            .useStoryLoader(new LoadFromClasspath(embeddableClass))
            .useStoryParser(new RegexStoryParser(examplesTableFactory)) 
            .useStoryReporterBuilder(new StoryReporterBuilder()
                .withCodeLocation(CodeLocations.codeLocationFromClass(embeddableClass))
                .withDefaultFormats()
                .withFormats(CONSOLE, TXT, HTML, XML)
                .withCrossReference(xref))
            .useParameterConverters(parameterConverters)    
            .useStepMonitor(xref.getStepMonitor());
    }

Now the next step is to copy the JS libraries in your project and make sure they are copied in the view directory. Download the all the files and folder from here https://github.com/jbehave/jbehave-core/tree/master/jbehave-navigator/src/main/resources. Copy them under src/main/storynavigator folder in your source.

Add this confif in your POM files, so that these files are copied in the Jbehave view directory

        ${basedir}/src/main/storynavigator
        ${project.build.directory}/jbehave/view
        false

Now every run will generate a file called navigator.html inside /target/jbehave/view.

First tab is Stories, it will allow searching on story description, narrative and meta tag. Second tab is the Steps Tab, this will allow searching all the steps in your step classes.

Bit of learning – You will not see data in Stories tab for you story if no meta tags are used in the report.

Some of these challenges becomes inevitable , if some of the critical processes in scrum like story writing, estimation and TDD falls apart.

Some of the high level Testing challenges in Agile and Scrum

And I think these are some of the core points if taken care can make huge impact on the overall testing process and for that matter in delivery.

Having the right Requirement

I see this as a biggest pain points and root of other problems. Having a good story can change how the requirement is delivered and tested. I refer this from the book “User stories Applied”, that a story should inhibit following features INVEST

I – INVEST
N – NEGOTIABLE
V – VALUABLE
E – ESITMATABLE
S – SMALL
T – TESTABLE

Each of these characteristics are self explainable and however trivial they may sound, they have huge impact on the quality of the requirement delivered in the end.

For instance a large story which can pretty much eat good few sprints is more likely to introduce bugs, as it will not be ready (DONE) by end of the sprint for review. And also large stories create confusion for developer as they get out of context very soon, similarly its nightmare for testing as well since focusing on a big chunk of feature creates opportunity for bug and overlooking.

Also, a badly written story can lead a wrong estimation which can cause capacity issues for testing as chances are some stuff will have to rolled over to next sprint or rushed through. Not only that a badly estimated story can lead to greater risks in further sprints, as it will be eat up some of the valuable time needed to prepare for next sprint’s requirement.

I strongly believe that testers in Agile project can add lot of value by looking at requirements nice early and reviewing it from a tester’s point of view. Some of the questions a tester need to asks are

- Does this acceptance criteria makes sense?
- How can I test it? (What, where and When )
- Is it not a duplication of work we did earlier ?
- Is this story too small or big ?

Automate Automate Automate

Automation will help maintaining a tight grip on the changes. Automation should be done at all levels and with involvement of QAs). Why do I say that? As Agile accepts changes quiet frequently and focuses on delivering features every sprint. it is obviously you would not get much time to plan, write and manage those tests, hence it is imperative that test automation is thought about while writing the code. And QAs should play important role in discussing, planning and thinking about those tests upfront, and discuss them with BAs and Developers. Build as much input as possible and then take right decisions as in which tests to be automate as unit, integration or UI tests.

Automation are assets and needs to be maintained, re factored and looked after like production code. I have seen automation tests becoming big bottleneck to maintain and trace and these problems can be solved by doing some simple things regularly .

- Make sure your automated tests are traceable against requirements (There are tools available to do that , if not use spreadsheet ). This will make sure you can track the tests against a requirement and if needed re factor, extend or retire it.
- Automated at all levels – UI tests are expensive and not conducive to tests each and everything. Good unit and integration test helps in building better coverage and helps in giving faster feedback. End to End tests are important and should be exclusive and focus on workflow and driving application end to end.
- Get testers involved in writing unit and integration tests, they will provide valuable feedback.

Defining the DONE Criteria

I think this is the first and foremost process to be in place from Day1.

What is DONE Criteria ?

DONE criteria are high level check list items which needs to be satisfied in order to claim a story is DONE. DONE Criteria should have technical, process, functional, non-functional, configuration aspects of the software. DONE criteria forces teams to take holistic view on the stories rather then code completing the feature.

Some of the common aspects of done criteria

- All acceptance criteria met. (Functional)
- Performance not degraded. (Non Functional)
- Code coverage achieved and all acceptance tests automation. (Technical )
- All build passed and application deployed on UAT/Staging environment. (Environment/Configuration)
- Exploratory testing performed and QA signed off.
- Demoed to stake holders on proper environment. (Process)
- All known bugs closed and tested (Process)

By doing this teams will make sure that software produced is of great quality and all aspects of the applications are taken care. Doing this repeatedly will make sure testers are not doing catchup but they are part of the story and sign off process. Software produced in the end can be released and should be production ready.

Exploratory Testing

This is the most misunderstood and ignored aspect of Agile Testing. Exploratory Testing advocated testes/BAs to test a feature by exploring it beyond the test scripts. At the same time it is not about clicking around the application without proper context.

Exploratory testing should be done every sprint on set of features on a proper environment. Testers should use all their skills to think out of the box and try to fearlessly test the system within the context of the features, however not limited to that feature.

Success of Agile Testing depends on working collaboratively between different stakeholders of the project and learning and rectifying quickly.

You can also refer to this scenario as data driven testing, as each of the row in your data table becomes one test case.

In the example below lets say I am testing a calculator sum function, and for that I need to have several test cases to test with different data. So basically steps looks like this, however we need to give different input and assert output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

 
 
Scenario: As as user I want to test calculator sum function 
Meta:
@author pankaj
 
Given calculator takes <input1> and <input2>
Then the sum is <sum>
 
Examples:
|input1|input2|sum|
|2|3|5|
|4|5|9|
|10|12|22|

And steps implementation for the above written steps will need to use Named parameter. So basically Named parameter (Which is usually in <>), will be replaced by the data in the table below the steps in the story. In the above example there are three named parameter in my story.

In the steps implementation, you step needs to explicitly use the named parameter in the method. Using named parameter in the steps tells jbehave that it needs to replace this with the data in the table specified below/or in another file.

Steps class would look like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package org.qainfolabs.jbehave.steps;
 
import java.math.BigDecimal;
 
import junit.framework.Assert;
 
import org.jbehave.core.annotations.Given;
import org.jbehave.core.annotations.Named;
import org.jbehave.core.annotations.Then;
import org.qainfolabs.app.Calculator;
 
public class CalulatorSteps {
 
	private BigDecimal input1;
	private BigDecimal input2;
 
	@Given("calculator takes <input1> and <input2>")
	public void readFruits(@Named("input1") BigDecimal input1,
			@Named("input2") BigDecimal input2) {
		this.input1 = input1;
		this.input2 = input2;
	}
 
	@Then("the sum is <sum>")
	public void doSum(@Named("sum") BigDecimal sum) {
		Calculator calc = new Calculator();
		Assert.assertEquals(sum, calc.sum(input1, input2));
	}
}

After the scenario is executed Jbehave will produce a detailed report of the scenario, showing the steps repeating with replaced parameter with each data set.

-          Working closely with BA on upcoming stories

-          Where ever possible produce acceptance criteria / test of the stories to be done in coming sprint

-          Understanding review of the stories

-          Challenging the ambiguous requirements

-          Challenging big stories

In the sprint

-          Work with development team on implantation of the acceptance tests of the stories.

-          Pair with developers if it helps the team to get stories moving quickly and reducing the feedback time

-          Discuss different levels of testing to be done for the stories . Don’t think you don’t have any responsibilities towards unit and integration tests.

-          Have an understating of implementation of the tests being written and challenge a implementation wherever you think the code will not be exercised in the right way.

-          Make sure the feedback is fast and delivered stories are tested on proper environments and meets the Done criteria.

-          Make sure you builds are green and tests are passing all the time

-          Work with business / BA on the delivered stories to sign off it.

-          Document and communicate the bugs according to your team process.

-          Implement more comprehensive automated tests if needed.

-          Make sure Non functional requirements are met

Post sprint

-          Work with business/ BAs on the demo of the story, as I think QAs develop deep understanding of the requirements.

-          More exploratory testing.

-          Look at performance/load/stress criteria of your application every now and then.

-          Close the fixed bugs and make sure proper regression tests are added.

This list is not a comprehensive tasks lists and it is meant only for guideline purposes.