Pankaj Nakhat's blog : All About Agile Testing

I have been working on Agile projects for years know and I have seen it evolving over the years. I have made mistakes in the past and always learnt the Hard way. I thought it will be worth sharing my experience and learning in Agile testing.

Mistake 1: Not getting involved early enough in the Sprint. So often I have seen teams falling in this trap. And consequences are expensive – QAs not getting involved early fail to provide important feedback on the requirement.  Its that missing perspective which can add lot of value early on. Viz looking at the testability of the requirements, and collaborating with different stakeholders to gain valuable context in to the story. I can go on here and talk about so many obvious and non  obvious cost of not doing this right.

However, there is one aspect worth talking about – Why do teams fall in this trap , where QAs are not getting involved early enough? I am highlighting someone of them from my experience

• Catching up on testing from previous iteration – Its a common symptom where teams leave QAs to deal with the stories delivered at the fag end of the iteration.  And I personally think that leaving testing problems to just QAs in the team is a sign of broken process. Team should look to rectify this and come together to try to get all stories to done (Properly ) before the end of the sprint. That may require people from cross discipline need to roll their sleeves up.  Doing this will enables team to have a greater disciplines and control over their iterations and most importantly avoid that ‘catch-up’ mode problem.
• Testing as seen outside the scope of sprint – And again I have seen it often in projects, where progress is shown/claimed based on development complete stories. However in true sense it does not matter if story is dev complete, Half QA done or not started at all. In Scrum/Agile model stories are considered Done only when all required phase of work is done or in other terms definition of done is met (DOD I will referring it as DOD in rest of my post). Coming out of this mindset helps team to focus on real value of work rather then claiming false sense of progress.
• Loose definition of Done – I spoke about definition of done in previous point. Now how important is it to have definition of done and what consequences can it bring if not well defined? I am not going to talk about what DOD is and how is it defined.  DOD is a yard stick against which a requirement is said to be measured and upon successful meeting the criteria, it is said to be DONE in the process.  DOD usually have attributes like – Acceptance Tests Passed, Exploratory Testing done, No Red builds, NFR met etc etc.  So in first place not having a DOD does not help team in measuring themselves against some standard quality tools.  So, it does help teams to be consistent with their approach in terms of delivering different piece if functionality. Result is obvious – Inconsistency in Quality and Communication breakdown which then leads to poor  product.
• Other scenario being not following DOD religiously for various reasons like delivery pressure, assumption of quality or may be anything else. which results in similar consequences as mentioned above.

I will continue with other reasons on my subsequent post.

Software testing has changed dramatically since the inception of Agile and Scrum. These methodologies are now widely accepted and without any doubt offers better way of delivering software. However Agile and Scrum also highlighted some of the challenges in the testing processes which possessed greater risk of failure , if not thought about in the over all process.

Some of these challenges becomes inevitable , if some of the critical processes in scrum like story writing, estimation and TDD falls apart.

Some of the high level Testing challenges in Agile and Scrum

And I think these are some of the core points if taken care can make huge impact on the overall testing process and for that matter in delivery.

Having the right Requirement

I see this as a biggest pain points and root of other problems. Having a good story can change how the requirement is delivered and tested. I refer this from the book “User stories Applied”, that a story should inhibit following features INVEST

I – INVEST
N – NEGOTIABLE
V – VALUABLE
E – ESITMATABLE
S – SMALL
T – TESTABLE

Each of these characteristics are self explainable and however trivial they may sound, they have huge impact on the quality of the requirement delivered in the end.

For instance a large story which can pretty much eat good few sprints is more likely to introduce bugs, as it will not be ready (DONE) by end of the sprint for review. And also large stories create confusion for developer as they get out of context very soon, similarly its nightmare for testing as well since focusing on a big chunk of feature creates opportunity for bug and overlooking.

Also, a badly written story can lead a wrong estimation which can cause capacity issues for testing as chances are some stuff will have to rolled over to next sprint or rushed through. Not only that a badly estimated story can lead to greater risks in further sprints, as it will be eat up some of the valuable time needed to prepare for next sprint’s requirement.

I strongly believe that testers in Agile project can add lot of value by looking at requirements nice early and reviewing it from a tester’s point of view. Some of the questions a tester need to asks are

- Does this acceptance criteria makes sense?
- How can I test it? (What, where and When )
- Is it not a duplication of work we did earlier ?
- Is this story too small or big ?

Automate Automate Automate

Automation will help maintaining a tight grip on the changes. Automation should be done at all levels and with involvement of QAs). Why do I say that? As Agile accepts changes quiet frequently and focuses on delivering features every sprint. it is obviously you would not get much time to plan, write and manage those tests, hence it is imperative that test automation is thought about while writing the code. And QAs should play important role in discussing, planning and thinking about those tests upfront, and discuss them with BAs and Developers. Build as much input as possible and then take right decisions as in which tests to be automate as unit, integration or UI tests.

Automation are assets and needs to be maintained, re factored and looked after like production code. I have seen automation tests becoming big bottleneck to maintain and trace and these problems can be solved by doing some simple things regularly .

- Make sure your automated tests are traceable against requirements (There are tools available to do that , if not use spreadsheet ). This will make sure you can track the tests against a requirement and if needed re factor, extend or retire it.
- Automated at all levels – UI tests are expensive and not conducive to tests each and everything. Good unit and integration test helps in building better coverage and helps in giving faster feedback. End to End tests are important and should be exclusive and focus on workflow and driving application end to end.
- Get testers involved in writing unit and integration tests, they will provide valuable feedback.

Defining the DONE Criteria

I think this is the first and foremost process to be in place from Day1.

What is DONE Criteria ?

DONE criteria are high level check list items which needs to be satisfied in order to claim a story is DONE. DONE Criteria should have technical, process, functional, non-functional, configuration aspects of the software. DONE criteria forces teams to take holistic view on the stories rather then code completing the feature.

Some of the common aspects of done criteria

- All acceptance criteria met. (Functional)
- Performance not degraded. (Non Functional)
- Code coverage achieved and all acceptance tests automation. (Technical )
- All build passed and application deployed on UAT/Staging environment. (Environment/Configuration)
- Exploratory testing performed and QA signed off.
- Demoed to stake holders on proper environment. (Process)
- All known bugs closed and tested (Process)

By doing this teams will make sure that software produced is of great quality and all aspects of the applications are taken care. Doing this repeatedly will make sure testers are not doing catchup but they are part of the story and sign off process. Software produced in the end can be released and should be production ready.

Exploratory Testing

This is the most misunderstood and ignored aspect of Agile Testing. Exploratory Testing advocated testes/BAs to test a feature by exploring it beyond the test scripts. At the same time it is not about clicking around the application without proper context.

Exploratory testing should be done every sprint on set of features on a proper environment. Testers should use all their skills to think out of the box and try to fearlessly test the system within the context of the features, however not limited to that feature.

Success of Agile Testing depends on working collaboratively between different stakeholders of the project and learning and rectifying quickly.

Testers on agile projects have to often deal with multiple tasks in an iteraton.