Reaping the Benefits of Agile Testing in Medical Device Development

Traditionally, many medical device manufacturers have chosen the Waterfall process as the best way of managing their development with the assumption that regulatory bodies preferred this method.

While the regulations do not prescribe a method for designing and developing a product, some FDA QSR related guidance, such as the FDA’s “Design Control Guidance for Medical Device Manufacturers”, use a language that points in this particular direction.

The medical device manufacturers’ focus, due to the uncompromising effects of not being compliant, has always been on creating documents and reviewing those documents - with organization, compliance, and quality being more important than end-user-focused and efficient development. Testing processes have also stayed true to this path. As most of the development in the medical device industry is subjected to these forces, we often see the following “testing truths”

• Testing starts on completion of components at the end of the development cycle
• Since formal testing is documentation heavy, most testing is one-off.
• Involving the end-user tends to begin at design validation after the product is either completed or nearly completed
• The rush to decrease the time-to-market causes hasty testing without nearly enough meaningful test coverage
• Manual testing is done really manually – meaning printed tests, checkoff sheets, and then feedback and results are input manually back into the Word test plan.
• Regression testing is also done manually which causes burned-out testers who are more prone to make errors over time.

With time, these drawbacks have become more and more obvious to the industry as they result in increased cost, lower quality, and lower end-user acceptance.

Although Agile development appears to be a promising alternative, its (unwarranted) reputation of being unstructured and informal, focusing on speed rather than quality, has made the traditional medical device manufacturer shun it.

After all, does it not seem wiser to use a tried and tested, compliant development process, known and accepted by notified bodies, than to gamble with an unknown alternative, even though the cost might be a bit higher?

On the contrary. The truth is that many of the industry’s major players, including GE Healthcare, Medtronic, Roche, St. Jude Medical, Cochlear, Boston Scientific and Toshiba have adopted the agile, iterative development approach. However, going agile is not always an all-or-nothing proposition.

It is up to the manufacturer to pick and choose the parts from the agile methodology that makes the most sense for their business.

Using an iterative approach will allow you to enjoy earlier and more frequent testing. One of the clear benefits of the Agile process is that testing is introduced earlier in the development process. With shorter development cycles, testers have the advantage of finding issues earlier, which should provide an improved overall quality further down-stream. This is also the case for formative usability testing, in line with IEC 62366-1:2015. The earlier we receive feedback from our target customer, the better we can steer development towards producing a viable and accepted product upon release.

Test early, fix early

Introducing test activities early in the development process allows early fixing of detected quality issues. It is a long proven fact that it is much easier and less expensive to fix issues and solve problems earlier in the development cycle when there are fewer dependencies and design is fresh in the mind.

Once your development has progressed, it is natural to forget why something was done and the difficulty to avoid serious impact when introducing a change increases dramatically.

By having frequent release cycles, you have an opportunity to adjust as you go so that your path fits your testing – both for errors as well as for acceptance. Requirements never stop changing and evolving - your testing should mirror this and your development should be in line with the results.

Involve the end-user on a regular basis

The idea of Agile, or the iterative development approach, is to release working prototypes often, review the work done, improve on that work in the next iteration or sprint.

While prototypes may not always fit the development needs of medical devices, the iterative approach focuses on feature-driven iterations which can be tested. Either way, the focus is customer-based – accepting and implementing requirement changes as they come in to better fit the customers’ expectations.

Shorter release schedules allow for more frequent reviews, which helps the development team stay on track while improving both quality and compliance adherence over time.

Testing – especially usability testing, a prescribed activity by MDR and FDA QSR 820 – will be frequent and timely, enabling teams to build quality into the iterations at a much faster speed. This will in turn shorten time-to-market and deliver a higher quality product as you will see an increase in the test coverage capabilities with a more frequent and early testing process.

Working features will be produced in every iteration and verification will be achieved on these frequent builds through unit tests, manual verification & regression testing. By finding the issues early, there is a significant lowering of development risks for the project and ultimately the product.

Addressing high risks with early testing

Using a risk-based approach, an established best practice in medical device development and prescribed by standards like ISO 13485, implies prioritizing design areas of high risk, address these early, and direct efforts and resources to tasks targeted to minimize said risks.

This includes the implementation of the corresponding risk-mitigating features but also on the verification of the effectiveness of these features, as stated in ISO 14971.

Only by verifying that an implemented feature effectively reduces the identified risk, can it be proven that the initial risk has been reduced. If it cannot be proven that a feature actually reduces risk, the risk is considered to be unmitigated which can jeopardize the entire project.

An early proof of risk reduction effectiveness through verification, will in turn lower the business risk of the project.

Automated testing in medical device development

To increase efficiency and continue to lower time-to-market, automating testing wherever possible is a great way to increase test coverage, decrease cost, and lower overall project risk. Regression testing in particular is very labor-intensive and can lead to mistakes due to the stress that it inherently causes.

By automating these and other tests you will see the reliability and predictability of your test plan increase. Testing visibility and transparency will enable you to better budget for future projects in terms of labor, finance, and effort.  

The key to avoiding delays and lowering development risk is to shorten development iterations, which enables you to test early, test frequently, and adjust development to fit the user needs sooner rather than later.

Early identification of risks, issues, and product validation problems can help overcome them before they become project or product killers. Usability testing, when possible, should be done throughout development to maintain the validity of the project and keep development on track.

Automate regression and any other labor-intensive testing wherever possible.

Happy testers tend to be more accurate testers.

A well-tested, compliant product with early usability acceptance should be everybody’s goal – especially when it arrives on schedule.