Jbehave reports can be ebabled to show something which is called Storynavigator. Its a client side javascript library which enabled browser to read meta data generated by cross referencing in Jbehave. Cross referencing can be enabled very easily from the configuration and it then generated the meta data which can be used by this story navigator.

How to enable cross referencing in Jbehave configuration?

    @Override
    public Configuration configuration() {
    	CrossReference xref = new CrossReference();
 
        Class embeddableClass = this.getClass();
        // Start from default ParameterConverters instance
        ParameterConverters parameterConverters = new ParameterConverters();
        // factory to allow parameter conversion and loading from external resources (used by StoryParser too)
        ExamplesTableFactory examplesTableFactory = new ExamplesTableFactory(new LocalizedKeywords(), new LoadFromClasspath(embeddableClass), parameterConverters);
        // add custom coverters
        parameterConverters.addConverters(new DateConverter(new SimpleDateFormat("yyyy-MM-dd")),
                new ExamplesTableConverter(examplesTableFactory));
        parameterConverters.addConverters(new MyConverter());
        parameterConverters.addConverters(new StudentConverter(new ExamplesTableFactory()));
 
		return new MostUsefulConfiguration()
            .useStoryLoader(new LoadFromClasspath(embeddableClass))
            .useStoryParser(new RegexStoryParser(examplesTableFactory)) 
            .useStoryReporterBuilder(new StoryReporterBuilder()
                .withCodeLocation(CodeLocations.codeLocationFromClass(embeddableClass))
                .withDefaultFormats()
                .withFormats(CONSOLE, TXT, HTML, XML)
                .withCrossReference(xref))
            .useParameterConverters(parameterConverters)    
            .useStepMonitor(xref.getStepMonitor());
    }

Now the next step is to copy the JS libraries in your project and make sure they are copied in the view directory. Download the all the files and folder from here https://github.com/jbehave/jbehave-core/tree/master/jbehave-navigator/src/main/resources. Copy them under src/main/storynavigator folder in your source.

Add this confif in your POM files, so that these files are copied in the Jbehave view directory

        ${basedir}/src/main/storynavigator
        ${project.build.directory}/jbehave/view
        false

Now every run will generate a file called navigator.html inside /target/jbehave/view.

First tab is Stories, it will allow searching on story description, narrative and meta tag. Second tab is the Steps Tab, this will allow searching all the steps in your step classes.

Bit of learning – You will not see data in Stories tab for you story if no meta tags are used in the report.

Some of these challenges becomes inevitable , if some of the critical processes in scrum like story writing, estimation and TDD falls apart.

Some of the high level Testing challenges in Agile and Scrum

And I think these are some of the core points if taken care can make huge impact on the overall testing process and for that matter in delivery.

Having the right Requirement

I see this as a biggest pain points and root of other problems. Having a good story can change how the requirement is delivered and tested. I refer this from the book “User stories Applied”, that a story should inhibit following features INVEST

I – INVEST
N – NEGOTIABLE
V – VALUABLE
E – ESITMATABLE
S – SMALL
T – TESTABLE

Each of these characteristics are self explainable and however trivial they may sound, they have huge impact on the quality of the requirement delivered in the end.

For instance a large story which can pretty much eat good few sprints is more likely to introduce bugs, as it will not be ready (DONE) by end of the sprint for review. And also large stories create confusion for developer as they get out of context very soon, similarly its nightmare for testing as well since focusing on a big chunk of feature creates opportunity for bug and overlooking.

Also, a badly written story can lead a wrong estimation which can cause capacity issues for testing as chances are some stuff will have to rolled over to next sprint or rushed through. Not only that a badly estimated story can lead to greater risks in further sprints, as it will be eat up some of the valuable time needed to prepare for next sprint’s requirement.

I strongly believe that testers in Agile project can add lot of value by looking at requirements nice early and reviewing it from a tester’s point of view. Some of the questions a tester need to asks are

- Does this acceptance criteria makes sense?
- How can I test it? (What, where and When )
- Is it not a duplication of work we did earlier ?
- Is this story too small or big ?

Automate Automate Automate

Automation will help maintaining a tight grip on the changes. Automation should be done at all levels and with involvement of QAs). Why do I say that? As Agile accepts changes quiet frequently and focuses on delivering features every sprint. it is obviously you would not get much time to plan, write and manage those tests, hence it is imperative that test automation is thought about while writing the code. And QAs should play important role in discussing, planning and thinking about those tests upfront, and discuss them with BAs and Developers. Build as much input as possible and then take right decisions as in which tests to be automate as unit, integration or UI tests.

Automation are assets and needs to be maintained, re factored and looked after like production code. I have seen automation tests becoming big bottleneck to maintain and trace and these problems can be solved by doing some simple things regularly .

- Make sure your automated tests are traceable against requirements (There are tools available to do that , if not use spreadsheet ). This will make sure you can track the tests against a requirement and if needed re factor, extend or retire it.
- Automated at all levels – UI tests are expensive and not conducive to tests each and everything. Good unit and integration test helps in building better coverage and helps in giving faster feedback. End to End tests are important and should be exclusive and focus on workflow and driving application end to end.
- Get testers involved in writing unit and integration tests, they will provide valuable feedback.

Defining the DONE Criteria

I think this is the first and foremost process to be in place from Day1.

What is DONE Criteria ?

DONE criteria are high level check list items which needs to be satisfied in order to claim a story is DONE. DONE Criteria should have technical, process, functional, non-functional, configuration aspects of the software. DONE criteria forces teams to take holistic view on the stories rather then code completing the feature.

Some of the common aspects of done criteria

- All acceptance criteria met. (Functional)
- Performance not degraded. (Non Functional)
- Code coverage achieved and all acceptance tests automation. (Technical )
- All build passed and application deployed on UAT/Staging environment. (Environment/Configuration)
- Exploratory testing performed and QA signed off.
- Demoed to stake holders on proper environment. (Process)
- All known bugs closed and tested (Process)

By doing this teams will make sure that software produced is of great quality and all aspects of the applications are taken care. Doing this repeatedly will make sure testers are not doing catchup but they are part of the story and sign off process. Software produced in the end can be released and should be production ready.

Exploratory Testing

This is the most misunderstood and ignored aspect of Agile Testing. Exploratory Testing advocated testes/BAs to test a feature by exploring it beyond the test scripts. At the same time it is not about clicking around the application without proper context.

Exploratory testing should be done every sprint on set of features on a proper environment. Testers should use all their skills to think out of the box and try to fearlessly test the system within the context of the features, however not limited to that feature.

Success of Agile Testing depends on working collaboratively between different stakeholders of the project and learning and rectifying quickly.

You can also refer to this scenario as data driven testing, as each of the row in your data table becomes one test case.

In the example below lets say I am testing a calculator sum function, and for that I need to have several test cases to test with different data. So basically steps looks like this, however we need to give different input and assert output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

 
 
Scenario: As as user I want to test calculator sum function 
Meta:
@author pankaj
 
Given calculator takes <input1> and <input2>
Then the sum is <sum>
 
Examples:
|input1|input2|sum|
|2|3|5|
|4|5|9|
|10|12|22|

And steps implementation for the above written steps will need to use Named parameter. So basically Named parameter (Which is usually in <>), will be replaced by the data in the table below the steps in the story. In the above example there are three named parameter in my story.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package org.qainfolabs.jbehave.steps;
 
import java.math.BigDecimal;
 
import junit.framework.Assert;
 
import org.jbehave.core.annotations.Given;
import org.jbehave.core.annotations.Named;
import org.jbehave.core.annotations.Then;
import org.qainfolabs.app.Calculator;
 
public class CalulatorSteps {
 
	private BigDecimal input1;
	private BigDecimal input2;
 
	@Given("calculator takes <input1> and <input2>")
	public void readFruits(@Named("input1") BigDecimal input1,
			@Named("input2") BigDecimal input2) {
		this.input1 = input1;
		this.input2 = input2;
	}
 
	@Then("the sum is <sum>")
	public void doSum(@Named("sum") BigDecimal sum) {
		Calculator calc = new Calculator();
		Assert.assertEquals(sum, calc.sum(input1, input2));
	}
}

-          Working closely with BA on upcoming stories

-          Where ever possible produce acceptance criteria / test of the stories to be done in coming sprint

-          Understanding review of the stories

-          Challenging the ambiguous requirements

-          Challenging big stories

In the sprint

-          Work with development team on implantation of the acceptance tests of the stories.

-          Pair with developers if it helps the team to get stories moving quickly and reducing the feedback time

-          Discuss different levels of testing to be done for the stories . Don’t think you don’t have any responsibilities towards unit and integration tests.

-          Have an understating of implementation of the tests being written and challenge a implementation wherever you think the code will not be exercised in the right way.

-          Make sure the feedback is fast and delivered stories are tested on proper environments and meets the Done criteria.

-          Make sure you builds are green and tests are passing all the time

-          Work with business / BA on the delivered stories to sign off it.

-          Document and communicate the bugs according to your team process.

-          Implement more comprehensive automated tests if needed.

-          Make sure Non functional requirements are met

Post sprint

-          Work with business/ BAs on the demo of the story, as I think QAs develop deep understanding of the requirements.

-          More exploratory testing.

-          Look at performance/load/stress criteria of your application every now and then.

-          Close the fixed bugs and make sure proper regression tests are added.

This list is not a comprehensive tasks lists and it is meant only for guideline purposes.

Agile development practices have been preaching for years on the aspect of writing acceptance tests for the requirements upfront. These acceptance tests used to be written in various and tools. Fitnesse definitely levraged among all other tools, simply because it enabled developement teams to use power of communication. No longer there was divide between tests being too technical for the people to read who actually cared about them, ofcourse I am talking about business , BAs, Testers.

Behaviour Driven Development is refined form for writing Acceptance tests, where requirements are expressed in natural language of communication (English, French etc) and implemented in the same form. This enabled no loss of communication from Requirements to Tests. And also it forces business to be involved with requirement throughout, as they have visibility on what they asked WORKS !

Tools like Cucumber, Concordian, Jbehave, Easyb enabled a uniform language of communication for Requirements and Tests , which looks like a simple discipline but it is very powerful.

Not only business gets lot of value from it at the same time development teams are equally benefited, as they have clear understanding of expected software behaviour in the form of executable requirement. Also, the power of BDD lies in thinking about business problem and expressing the intent without thinking about the implementation details of the problem.

A BDD can express the intent in form of Simple Sentences, Tables, Examples etc. The most common form of Acceptance testing format is Given – When – Then, which is widely being used in BDDs.

Given – Is a prerequisite or precondition for an action.
When – A action or Event happens.
Then – Results of the action.

Example – Saving bank account withdraw scenario

Given there are 1000$ in saving bank account number XXXXXXX
When a cash with drawl of $900 is done
Then available balance is $100

The above example is a simple explanation of How a Given When Then can express an intent in consistent and effective way.

I am assuming you have got a basic Java project setup where you can write tests in Java, and also you can choose any unit testing framework of your choice. WebDriver is a generic interface which then gets implemented by different available drivers (browsers) to achieve the consistency in the API. However, there are still some difference between the drivers, but we will ignore that for this posts, as basic API is consistent across the driver. Having an interface independent of driver implementation helps to write driver independent  tests, so we can write a test in WebDriver and run that against any available driver (Firefox, Chrome, IE and Iphone driver).

I must admit that chrome driver is little immature at the moment, but I am sure it should be good soon. For my example I will be using Eclipse as the IDE and TestNG as the unit testing framework.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

package org.wd.tests;
 
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.Assert;
import org.testng.annotations.Test;
import org.wd.annotation.DataDriven;
 
public class TestGoogle{
 
	/* This creates a field of type WebDriver Interface*/
	protected WebDriver driver;
 
	@Test
	public void testGoogle(){
		/*This will crate a new Instance of Firefox driver*/
		this.driver = new FirefoxDriver();	
		/*This will launch www.google.co.uk on the browser*/
		driver.get("http://www.google.co.uk");	
 
		/*Identify the elements to interact with */		
		WebElement searchTextBox = driver.findElement(By.name("q"));
		WebElement searchButton = driver.findElement(By.name("btnG"));
 
		/*Take actions now - Set the text field*/
		searchTextBox.sendKeys("Pankaj Nakhat");
		/*Click on the searchButton*/
		searchButton.click();		
 
		/*Verify HTML response contains some text*/
		Assert.assertEquals(driver.getPageSource(), "Pankaj Nakhat");
		driver.quit();
	}
}

Agile testing is not a rocket science and you still need people with good testing skills, that is a prerequisite.  But most of the organizations get confused between hiring a QA in terms of desired technical skills and QA Skills.

If you see most of the job specs floating around in the market, they have standard specs  viz Selenium, Fitnesse etc etc.  So, basically people seek technical tester to do test automation and hence they get glued to some particular job specs while hiring Agile Testers.  Where do you draw a line that a tester is technical enough to do the job, at the same time posses necessary Testing Skills?

Do people miss a trick here, in forming a good checklist for an Agile Tester?

A good Agile tester should be technically sound, there is no second thought about it, but then why can’t people hire good developers to do testing job in Agile projects.  That’s where I think QA specific skills should be looked for while hiring Agile Testers,  at the same time assessing their technical skills. I will get to the point of “QA Skills” in a while, but before that I want to touch upon on Technical skills of an Agile Tester.

Technical skills of an Agile Tester should not be only limited to knowing some specific tools, I think an Agile Tester should have a good programming aptitude. So this will enable them to pick up any new technology quiet quickly.  They should then apply their automation/tools experience with any given technology.

Also, an Agile Tester should be aware of the application architecture, hardware, tools and interaction between different systems. That enables them to think about the testing from much more wider perspective.

What QA skills one must look for whilst hiring Agile Testers?

1)      Good analytical skills to think about the requirement from users/testers perspective to verify a requirement and also testing it on adverse scenarios.

2)      Ability to communicate with business to challenge and asses a requirement.

3)      Proactive: An agile environment needs testers to be proactive to foresee any issues that may block the team in anyways.

4)      Ability to communicate well with developers:  I think this is equally important like interaction with business.   Testers play a role of a bridge between Business/BAs and Developers. And thats where they should not be technically deprived that one end of bridge become weaker.  I am not saying that you just need technical skill to be able to communicate with the developers.

5)      Good time/task management:  A Tester needs to play different roles in any  iterations/sprints and he should be able to priorities his work to be able to carry out the most important activities in the context of a sprint.

6)      Quick Learners : This is an attribute which is great to have in anyone, but I would asses a candidate on his learning ability.

DSL – Domain Specific Languages, what does it mean for Testing? What are the benefits..

Over the years automation testing has changed from record and play back mechanism to more structured, framework oriented approach. Benefits were evident Test were readable, Scalable and Maintainable.

But there was more needed to be done, as most automation tests would turn in to over head rather then assets.  Framework based approach was good..but came up with its own sets of problem. Some of them were Code like tests, which meant they were not human readable and needed either hours of

documentation to make them explicit to humans.  So if the application changed underneath both test and documentation needed to change. And after a while these will not match each other by any means.

Frameworks like keyword driven testing enabled naive to write Tests, however created the problem of highly non abstracted tests.

Since the introduction of methodologies like Agile, it was inevitable that those problem needed to be sorted fast. Tests needed to be changed quickly with often changing requirements.

DSLs allows you to write tests in your own domain..be it banking, Online Gaming or Travel Industry.  Which means tests are human readable and highly contextual to domain.

For example – I need to write a Test for a baking application for testing a Scenario of Transfer of a fund from Account A to Account B.  Note, I am talking in terms of any specific tool here, and that I will keep up for a separate post.

<Test>

1. Login in my online banking application with user “A”
2. Go to transfer funds page
3. Check initial balance it should be “£ 500″
4. Transfter amount “£100″ to account “B”
5. Check final balance , it should be “£ 400″
6. logout from the application.

</Test>

In above example , Test exactly looks like manual test scripts but there nature is Executable in whatever technology and tool we choose.

What problems does it solve?

• Maintenance is easier now as test are human readable.
• Adding and Amending a step in Test just requires finding the right step, rather then changing the implementation  (In Most of the cases).
• For example – In above Test I need to add a new requirement of a welcome page after login. We simply introduce another DSL step after step1 “Verify user is shown welcome page with him name on it”, and rest of the test is still unchanged.
• Test can be extended easily to do more comprehensive end to end test.
• If the application changes underneath in terms of UI Objects , tests will remain unchanged, only the underlying implementation will change.
• If a business process changes completely, changed can be accommodated easily by changing the DSL definition. For example in above Test the transfer process changes and now required another level of authentication before transferring the funds. We would change the definition of Step4 to ‘Transfter amount “£100″ to account “B” and enter authentication password for user “A”‘

DSLs can also come with its own challenges, which I would like to discuss in my next post.