Skip to main content

5 points when evaluating an ALM for Medical Device Development

Written by Karl Larsson on . Posted in Tips & Tricks.

So you have realized that an Application Lifecycle Management software probably would make your medical device development more efficient?

Maybe you have experienced that your team:

  • spends a lot of time documenting Design Controls?
  • has difficulty maintaining the traceability for your product?
  • lacks confidence in your current technical file?

There are many good ALM tools out there so it should not be too hard to pick one, right?

However, acquiring an ALM tool is a significant investment and you want to make sure that you pick one that addresses the particular challenges facing your team.

So where do you start?

Writing a Request for Proposal

A common starting point is to establish your organization's needs in a “Request for Proposal” document a.k.a. RFP. The RFP will be sent to a number of vendors and will help you 

This approach has several benefits. First of all, to explicitly spell out what the tool shall do for you will force you to concretize your needs. You shall also include other stakeholders to make sure you are informed about and address their concerns as well. This exercise will also reveal conflicting and ambiguous requirements which are always better to address sooner rather than later.

We recommend this RFP practice since it provides structure to the evaluation process. However, the RFP phase can sometimes lead to an “Analysis Paralysis” situation where an unproportionate effort is spent on establishing the “complete” RFP. Sometimes this is caused by incorporating too many stakeholders in the process, conflicting visions or needs or simply as a fear of failure when making a significant investment. Worst case, the RFP process can drag on for months, swell uncontrollably in size and, in the end, cost much more than the tool itself.     

We recommend you prioritizing the core needs that initially triggered the evaluation. Keep the RPF short and concise!

For a medical device ALM, this RFP should at least include requirements for:

  • Requirement Management
  • Establishing Traceability
  • Risk Analysis capabilities
  • Management of Verification & Validation activities
  • Proper versioning of Documentation and Artifacts
  • Establish reports that satisfies standards like ISO 13485 and ISO 14971
  • An automatic Audit Trail of all changes
  • Enable you to fulfill your Quality Management System

(We consider these the minimum features required for compliant medical device development).

Since we operate within the medical device scope, 21 CFR Part 11 Electronic Records (most of the artifacts in an ALM!) and Electronic Signatures (depending on how you implement the sign-off in your organisation) shall be addressed by the ALM. This will generate a number of requirements here summarized as:

  • 21 CFR Part 11 Ready (note that a tool in itself can never be 21 CFR Part 11 compliant. Final compliance always depends on processes within the organization)

As always, remember to make sure the requirements are testable.

Testability equals CSV

In our line of business, a Computer Software Validation, CSV, (more on that here) will eventually be used to validate the ALM. The RFP can provide valuable input to this activity and you should be able to derive your CSV User Requirements from your RFP.

Even though low CSV efforts may not be a formal requirement in itself, the CSV activity will definitely be a cost driver that should have an impact on your purchasing decision. In some cases, the CSV cost can surpass the cost of the tool itself.

The CSV effort is typically driven by:

  • The complexity of your QMS (e.g., number of Design Item types, number of different relationships between design items as well as number of SOPs to test in the validation).
  • The complexity of your CSV process (the rigorousness and granularity required)
  • Approach to tailor the tool itself a.k.a. customizable or configurable.

Finding out the vendors tailoring approach must definitely be part of the RFP. Do not forget to consider required integrations between the ALM and external tools. Integrations are notorious for requiring additional programming efforts.

If the tool has to be scripted/programmed to tailor it to your organization, the system falls into the CSV category “customizable“ for which the validation efforts for a system is expected to be higher (including deeper testing, managing risks, code reviews, etc).   

A configurable ALM is therefore preferable since:

  • The initial tailoring costs are lower
  • The initial and subsequent CSV costs are lower
  • Customization may impact the compatibility with future ALM releases and forward compatibility
  • Administrating the tailoring over time will require specialized personnel

Require the ALM to fit with your Quality Management System

Your QMS, development processes, and templates have been carefully developed to specifically address your company’s unique situation. Compromising your QMS may have significant effects on compliance and efficiency.

Some ALM  tools are pre-configured and you are required to make substantial changes to your QMS in order to fully utilize its potential. Unless you are considering rewriting your QMS for other reasons, we would discourage you from doing too big changes to an existing QMS since it may affect existing certifications and jeopardize compliance.

Another option to bridge the difference between the QMS and the ALM to set up processes and documentation that translates/maps between the tool and your QMS. However, operations that require manual mapping as described are notoriously error-prone and subjects your technical files to an increased risk of errors and inconsistencies.

Instead, the ALM should ideally adapt to your QMS, reflect your processes, use your templates and present your terms and notions in the GUI. This will lead to a higher level of acceptance, lower the learning curve, speed up the tool adoption process and reduce the risk of errors. For these reasons, we at Aligned always strive to make the tool follow the QMS.

The impact on your QMS should ideally not be more than an additional work instruction/SOP, stating that an ALM is used to fulfill certain activities defined by QMS. This work instruction/SOP will be a valuable input to your CSV activities.

To get you started, we have put together an example RFP spreadsheet. Download it here: Medical Device ALM Request for Proposal Template.xls

One Tool to Rule Them All or a Landscape of Tools?

What is the required scope of your ALM? This is a very common question often discussed in terms of cost. Intuitively, one IT tool must be cheaper to maintain than multiple tools, right?

A large tool spanning many areas of activity may have the benefit of providing integration, but from experience, dedicated tools usually are really good at what they do and definitely have their eligibility as well. Within a large organisation there will be tool boundaries. This fact may even simplify the introduction of an ALM tool and lower any business risk if you aim at building a working tool landscape rather than replacing it all at once.

The consideration of integrating with other tools can also be part of the RFP. We recommend however to keep the priorities right between core functionality and ‘nice to have’s’. The need to integrate different tools may be fulfilled over time.

Go through your RFP again and check if tools already existing in your organisation may already fulfill your requirements and adjust any priorities accordingly.

Next Step, gather Hands-on Experience!

Your RFP is not going to get you answers to all of your questions. Now it is time to experience the tool first hand. This is a crucial step in your evaluation. Most vendors will affirm every one of your requirements but you will quickly realize that the “how” is more important than the “what”.

Ask if you can access and evaluate the tool yourself. The vendor’s response will give you some important indications. Do they trust the ALM to be self-explanatory or will they require you to take training?

At this point, there are a few points on what to expect from the vendor:

  • You shouldn’t need to buy any infrastructure to install and run the evaluation version.
  • The evaluation should not require a significant time investment on your part
  • Any training and/or customization cost for the evaluation shall be minimal

Try to account for the subjective impression from the hands-on experience in your evaluation. They will be an important factor in embracing the tool within your organisation.   

Estimating the total cost

To prepare for the last step, it is time to request quotes from your vendors. Make sure these quotes include all cost-driving factors through-out the ALM’s lifecycle to accurately facilitate comparison between vendors.

An accurate lifetime cost estimation shall include:

  • Cost of Licenses
  • Cost of Service and Support (find out what is included)
  • Cost of configuration/customizing (how long until the system can be used?)
  • Cost of training
  • Cost of CSV activities
  • Cost of import/transfer of legacy data into the ALM
  • Cost of IT Infrastructure (include cloud costs, software licenses for OS, databases etc.)

Lifetime costs will inevitably arise from maintaining the tool:

  • Is the tool so complex to administrate that we need to build an internal team (or optionally buy this service from the vendor or 3rd party service provider) covering this new expertise?
  • Or you can rely on the vendor (ideally this is already part of the support agreement) for maintenance

Wrapping up the evaluation

Having the input from the RFP and the hands-on experience, you can now hopefully make a well-informed decision based on the capabilities of the ALM. You have gathered knowledge about how to make the tool will fit in your organisation and IT landscape.

At this stage, as a note of caution: a given tool will not solve all your problems. Tools do speeds up processes, provide structure and checks but in essence, they rely on improving something that already works.

To summarize the points:

  1. Formulate an RFP, focused on medical device development
  2. Plan ahead for CSV
  3. Check for fit with your QMS
  4. Gain hands-on experience
  5. Establish total lifetime cost

Best of luck with your evaluation!