How to maximize Formative Evaluations in Usability Engineering
Formative evaluations are a medical device development team's secret weapon, holding the power to catch and fix safety and usability issues early—before they become costly problems.
By observing how representative users interact with prototypes or by having experts evaluate early designs, formative evaluations allow us to refine, rethink, and even reimagine our approach. This process drives the design in the right direction of becoming a safe and effective medical device.
When to Conduct Formative Evaluations
To maximize their impact, formative evaluations should start early in the development process. This timing ensures that designers can identify usability issues and safety risks while changes are still relatively easy and cost-effective. Delaying formative evaluations to later stages risks valuable insights being lost or minimized, as changes become more challenging and costly to incorporate as the design nears completion.
Planning Formative Evaluations
Conducting a formative evaluation requires a representation of the medical device, such as a simple mock-up, sketch, or functional prototype, depending on the development stage. The prototype should be evaluated against planned use interactions, described in scenarios, use cases, or tasks.
Selecting scenarios for evaluation should be risk-based, prioritizing scenarios that represent higher threats to user safety and to effective use. Therefore, it is a prerequisite to analyze these scenarios for potential risks before beginning formative evaluations. This ensures evaluations are focused for the final validation step and aligned with critical safety aspects, as risk assessments serve as inputs to formative evaluation activities.
How to Perform Formative Evaluations
Methods for formative evaluations fall into two main categories: expert-based methods (e.g., cognitive walkthroughs, heuristic evaluations) and user-based methods, where users are observed and later interviewed about their interactions with the device. Expert-based methods are ideal for early-stage evaluations, while user-based methods yield greater insights when the design is more concrete.
In both cases, the objective is to identify potential use difficulties, close calls and use errors. These observations highlight risks that should be addressed in the device's risk assessment, ideally through design changes (e.g., User Interface Specifications). Risk assessments thus serve a dual purpose, functioning both as inputs and outputs of formative evaluation activities.
Beware of Bias
Bias can influence the outcomes of formative evaluations, both from observers and test participants. Emotional investment in the current design might impact an observer's impartiality or openness to feedback from participants. Having a design team member serve as an observer may lead to biased interpretations of participant comments.
Repeated use of the same individuals as test participants in multiple usability tests can also introduce bias, as they may implicitly compare the current design with previous versions rather than provide an unbiased assessment. Planning, assessing, and mitigating bias is, therefore, fundamental to a successful formative evaluation.
Why Formative Evaluations Are Essential
Formative evaluations do not yield binary "pass/fail" results; they generate qualitative observations. The goal is not to achieve a "passed" or "failed" outcome, but to gather insights that inform design. Ultimately, the formative evaluation's purpose is to drive the design forward, aligning it with user needs and safety—not to verify it.
Conclusion
Formative evaluations are valuable opportunities to shape a safer and easy-to-use device design, with each insight guiding us closer to a truly safe and effective medical device. By prioritizing early evaluations, we not only reduce costly design changes but also ensure a development process deeply rooted in user-centered principles.
About the Expert
Andrea Schuetz Frikart is a Senior Project Manager with extensive experience in the user-centered approach for designing safe and effective medical devices in the EU and U.S. regulatory framework. Her expertise lies in human factors engineering, particularly in compliance with IEC 62366 standards, the FDA guidance and all related human research studies. She is a passionate lecturer for professional training, workshops and academic-level courses.
As a specialist in human factors methodologies, she holds a Master of Advanced Studies (MAS) in Applied Psychology. Find here Linkedin profile here.
