What is the Case for Quality (CfQ)?
FDA launched the Case for Quality in 2011 to elevate the focus of all medical device stakeholders from baseline regulatory compliance to sustained, predictive practices that advance medical device quality and safety to achieve better patient outcomes.
FDA established the Case for Quality following an in-depth review of device quality data and feedback from regulatory and industry stakeholders. The FDA’s analysis flagged manufacturing quality risks. It also showed that firms that manage those risks by driving quality organization-wide are more productive, with fewer complaints and investigations per batch, and often with smaller quality units with lower quality-related costs than their competitors.
In other words, investing in quality pays.
Firms with an established quality culture are able to avoid quality failures and shift the resulting freed resources to accelerate device design, innovation, and market introduction.
Dr. Jeffrey Shuren, director of the FDA’s Center for Devices and Radiological Health, affirmed the Case for Quality’s role in advancing solutions to ensure patient access to high quality medical devices and the FDA’s commitment to their shared goal to improve patient outcomes.
A cultural shift of this magnitude demands the engagement of regulators, industry, providers, payers and patients. So in 2014, FDA enlisted the Medical Device Innovation Consortium (MDIC) to foster a new level of collaboration between the Agency and industry in a far-reaching Case for Quality movement.
The Case for Quality offers a unique forum for medical device stakeholders to work together to enhance device quality and patient safety. Participants in the Case for Quality include FDA and a growing number of medical device companies, healthcare providers and payer organizations, as well as trade associations and educational institutions.
In 2017, MDIC is focused on expanding on the Maturity Model pilot, completed in 2016. The Maturity Model team is working on addressing the barriers to assessment, enrollment and measurement (metrics) for the implementation of the Maturity Model as part of the CDRH third party voluntary program to be announced in December 2017. MDIC is also continuing work on the development of an analytics dashboard to use accessible quality data to inform a user-friendly dashboard make it possible to compare medical devices based on quality. Such a dashboard also makes it possible for medical device companies to compete based on the quality of their products.
What is the CDRH Voluntary Medical Device Manufacturing and Product Quality Pilot?
The CDRH pilot program leverages a proven maturity model, the CMMI framework, by which medical device organizations may measure their capability to produce high quality devices and increase patient safety. This measurement can then be used by the organization to drive targeted continuous improvement activities throughout their facility.
In recognition of manufacturers who complete this independent assessment of quality maturity, FDA will adjust that manufacturer’s engagement activities and change submission requirements. For additional information, see this 2-page program overview.
Why did CDRH choose CMMI for this activity?
Through collaboration with MDIC over the last 2 years, CMMI was selected for this voluntary pilot program, after a review of several maturity models was performed. CMMI is a framework supporting process level improvement and training, and includes an appraisal method for objectivity and consistency. This program is administered by the CMMI Institute and helps organizations focus the value that they can deliver by building capability in their people and processes. This model has been successfully used in various industries, including information technology, healthcare, automotive, defense, and aerospace, to consistently deliver high quality products and reduce waste and defects.
What is CMMI?
The Capability Maturity Model Integration (CMMI®) is a capability improvement model that can be adapted to solve any performance issue at any level of the organization in any industry. The Model provides guidelines and recommendations for helping your organization diagnose problems and improve performance. Used by over 10,000 organizations from more than 100 countries all over the world, CMMI helps you identify and achieve measurable business goals.
Who is the CMMI Institute?
CMMI® Institute is the global leader in the advancement of best practices in people, process, and technology. The Institute provides the tools and support for organizations to benchmark their capabilities and build maturity by comparing their operations to best practices and identifying performance gaps. For over 25 years, thousands of high-performing organizations in a variety of industries, including aerospace, finance, healthcare, software, defense, transportation, and telecommunications, have earned a CMMI maturity level rating and proved they are capable business partners and suppliers.
Are there any examples of medical device manufacturers who have publicized their participation in the CMMI program?
The CMMI Institute has a published list of organizations from a broad set of markets available online. In addition to these data, multiple medical device manufacturers piloted CMMI appraisals in coordination with the CMMI Institute, the FDA, and MDIC to determine how appraisal activity provided continuous and quality improvement feedback to the manufacturer. All organizations participating in the pilot were provided feedback that served as improvements to the quality of their QSR systems, product development and organization capability.
Why should my company participate? What are the tangible benefits from CDRH? When do I recognize them?
There are multiple reasons your company should participate in this program as part of the FDA’s Case for Quality. Industry participation is critical to long term implementation of the maturity model as an alternative to the traditional path of a routine FDA inspection. The FDA is seeking to leverage the maturity model program as a method to drive improvements in areas that are also beneficial to manufacturers, such as
- Increased manufacturing and product confidence
- Faster time to market
- Better information to drive regulatory decisions
- Improved resource allocation
- A continuous improvement focus on what is most important to patients
The FDA has confidence in companies that are accepted into this voluntary program, and will remove them from the routine FDA inspection plans at the point of approval into the program and will use independent means to monitor sustained performance.
Additionally, the Agency’s confidence and assurance in your quality system and performance (from program participation) means modifying and reducing the PMA submission requirements for the following manufacturing reviews:
- 30-Day Notices
- Site Changes
- PMA Original manufacturing module
The FDA will also benefit from this program by potentially reducing the internal resources required for evaluation of inspections and manufacturing review submissions. That capacity can be used for manufacturers who have not demonstrated high capability and to engage earlier in the development life cycle of new products.
The benefits provided can also accelerate the implementation of quality improvements at a faster rate and reduce time to market of product improvements and enhancements for patients.
How do I share this program and its value with senior management in their language?
Participation in the CDRH Voluntary Quality program will require a small investment of resource, both personnel and money, however there are several benefits. In general, you should expect to improve your processes and reduce variability that could lead to increased costs of quality and decreased rework.
In addition, the FDA has communicated that there will be increased confidence in companies that are enrolled in this voluntary program, so they may be removed from FDA routine inspection plans. There will also be some modifications and reductions to certain PMA submissions (30-Day notices, Site Changes, and Manufacturing modules).
Each company will be able to estimate a direct cost benefit by utilizing their own internal data and determining appraisal costs as part of appraisal scheduling with the CMMI Institute. You can estimate potential of saved costs for quality, submissions and audits. For instance, as shared at the March 30 MDIC Case for Quality Forum, Boston Scientific has estimated that if they were participating in this program, they could have redeployed $500,000 USD of resources who were assigned to write and submit their sixty nine 30 day notices for the year 2016.
What is expected from companies participating and NOT participating?
Companies may request to enroll into the program through an online application process.
Minimum enrollment requirements:
- Must be a U.S. based company/site or global company that distributes medical devices in the U.S. (Class I, II, III).
- Facilities must be those responsible for the manufacture and processing of medical devices.
- Facilities must not be under Official Action Indicated (OAI) status or subject to a judicial action.
- Companies with OAI can become eligible for benefits when:
- they have provided FDA with confirmation that appropriate corrective action has been implemented, and
- those actions have been verified by FDA during a follow-up inspection.
- Companies must have  prior history or compliance profile,  site registration, and  listing with FDA.
- Examples of ineligibility include: new manufacturers, start-up companies, no FDA inspections, no marketed products in the U.S., etc.
Expectations of Particpants:
- Annual subscription fee of $5,000 to be paid once you are accepted into the pilot program.
- Participation in a Medical Device Discovery Appraisal within the 90-day target and payment of the associated costs.
- Submission of metrics at defined frequencies to confirm progress and improvement from initial appraisal results.
- Sustained favorable compliance profile/history as defined in the enrollment requirements.
*Companies not participating will not see any changes in their engagement.
What are the risks of participating?
- Legal/Legislative Risks
- What happens when something goes wrong (questions from congress)
- Exposure of quality data (not releasable for FDA)
- Participant Risks
- Exposing too much information
- Greater public scrutiny; certifications need to be public
- Raising barrier to entry
- No alignment with districts
- Watch dog group will watch this
- CMMI barrier to entry
- Economic risk and driving consumer market
- Data Integrity Risks
- Data management/security/hacking
- Access to data
What is the difference between small and large companies’ costs and benefits?
There are no differences in benefits provided between large and small organizations. There will be two tiers for costs, please see “How much does it cost to enroll?” and “What does an appraisal cost?” Costs of an appraisal are determined as part of the scoping of that activity and consider a variety of factors including (but not limited to) the number of practice areas as the criteria, the number of products reviewed, and quality system complexities.
What benefits do I get with enrollment?
Once you are approved into the organization, the FDA will send you a letter (via email) listing the benefits you are now receiving: removal from the inspection list.
What are the requirements to enroll?
Must medical device manufacturing facility that distributes medical devices in the US and has a good compliance history with the FDA.
Do I enroll my organization or a facility?
Each application is intended to be for a specific facility. You can, however, enroll more than one facility per application if they share the same address. Additionally, organizations can enroll multiple facilities located at different addresses with multiple applications.
Should I enroll multiple sites? How?
This depends on whether or not you’re looking to make submission changes across multiple sites, whether or not you have multiple facilities at the same address, and whether or not it makes sense from a business perspective. To enroll multiple sites at the same address, select that option on the application. To enroll multiple sites with different addresses, please submit one application for each site.
How much does it cost to enroll?
We’re looking at a 2-tiered structure similar to the FDA’s and MDIC’s, with the division at $30M. At the first tier, for organizations making $30M in revenue or more, the program fee will be $5k for each facility enrolled, and at the second tier, for organizations making less than $30M in revenue, the program fee will be $3k for each facility enrolled.
What information do I need to enroll?
Be prepared to submit your Operation Number and Registration Number, both accessible via search here: https://www.accessdata.fda.gov/SCRIPTS/cdrh/cfdocs/cfRL/rl.cfm
Who should I list on the enrollment form?
Under contact information, please enter the name of the person who will be the main point of contact for your organization. This may be an executive, this may be someone in quality, this may be the site coordinator.
How long do I need to wait for approval into the program by the FDA?
Once you’ve enrolled, the CMMI Institute PMO will review your application within 1-business day and reach out to you if there are any concerns. If not, the PMO will send the application to the FDA, who will approve or deny the application within 5-business days.
How will I know if I have been approved into the program?
If you are approved into the program, the FDA will send you a letter of approval (via email) within 5 business days of receiving your application.
What if I am not approved into the program?
If you are denied from the program, the FDA will send you a letter of denial (via email) within 5 business days, containing additional information about the issues, and the steps for resolving those issues.
What happens after I am enrolled?
After you are approved into the program via a letter from the FDA, the CMMI Institute PMO will reach out to your Point of Contact to get a signed program agreement, signed SOW for the program fee, and request for your completed metrics collection form (which will be attached to the email). The PMO will also be requesting your availability to schedule the initial scoping conversation between you and a lead appraiser.
What is the purpose of an initial scoping call?
The first call will allow the appraiser to better understand the your organization’s business, products, and operations, as well as help your organization to better understand the relationship with the appraiser. The second (and possibly third) call will allow both parties to delve into the organization’s various processes for families of devices, and will inform the length of time needed onsite, the number of appraisal team members, the best timeframe for both parties, and the pricing.
How long is the initial scoping call?
We request 1-hour for the initial scoping call and 1-hour for each subsequent scoping call.
Who schedules the initial scoping call?
The CMMI Institute PMO will reach out to your organization for availability, and connect the key people with one of the MDDAP lead appraisers.
Who needs to be on that initial scoping call?
Both the executive sponsor and site coordinator should be on the call from your organization; however, you may extend the invitation to other parties you deem relevant to the conversation.
What should I do to prepare for the initial scoping call?
There’s nothing you need to do to prepare for the initial scoping call. It’s preferable if you come to the meeting with an open mind and let the lead appraiser guide you through the questions around the topics you already know: your operations, your products, your employees, your process, etc
What sort of questions are asked in the scoping calls?
This will vary slightly from organization to organization, but the basic set of questions, such as “How many of X product do you manufacture each year” and “How many employees do you have in Y department”, are intended to help the appraiser understand your organization’s business, products, processes, and operations.
What if we don’t get a chance to cover everything in the first scoping call?
The CMMI Institute PMO will continue to schedule follow-up calls until both the lead appraiser and organization have reached an agreement on the scope of work (based on an understanding of the facility’s operations), timing of the appraisal, and cost of the appraisal.
When do we need to have the initial scoping call?
It’s important to schedule the initial call within 30-days of enrollment and complete these scoping calls within 60-days of enrollment to ensure that the appraisal can be conducted within 90-days of enrollment.
What do I need to do to complete this baseline performance metrics collection form?
The CMMI Institute PMO will send you instructions when requesting the completed form. You can also view those instructions here.
What is the baseline performance metrics for used for?
There are two main purposes to these metrics. First, they will help us to understand how your organization measures its risk and quality, so that we can start looking for trends (increases or decreases) related to our activity. While we hope to see decrease in risk measurements and increase in quality measurements, the most important thing is understanding why we’re seeing certain changes. Second, the FDA wants to be able to have some insight into these trends as well so that they can understand how removing the current set of regulatory burdens support organizations in their efforts to mitigate risk and improve quality. There is additional room for feedback as to what other removed regulatory burdens your organization believes would assist that effort further. As we move forward, the FDA will consider that feedback in the next iteration of the program.
What is the longest period of time that I can wait from enrollment to the appraisal occurring?
The current target is for an organization to complete the appraisal with 90-days from the date they were approved into the program. If the site is not appraised within 90-days of enrollment, then it is expected that the organization has a conversation with both the PMO and the FDA to determine why. If there is no engagement after continued efforts, then the facility will be removed from the program and the inspection benefit will be halted. In order to be reconsidered for the program, the organization must have a discussion with the FDA and PMO.
Does a site have to be appraised to get the regulatory modifications?
Yes. In order to receive the submission benefits, a site must be appraised.
How is this different than an FDA inspection?
The Medical Device Discovery Appraisal (MDDA) focuses on actual capabilities and activities that are of value to the organization and looks beyond the CFR and the strict compliance of an inspection or audit. The MDDA leverages interviews and data collection from the people who actually perform the work, not just those in the "front room" or those who usually manage audits and inspections. Also, the MDDA drives a conversation for how to actually improve in a way that makes sense to the business rather than a corrective action list to "be in compliance".
What if I want more information about the CMMI model?
You can find more information in this high-level model overview. We highly encourage organizations not to read too much into the model beforehand because (1) it can seem daunting or overwhelming to someone who hasn’t been trained in the model, or (2) organizations may feel comfortable enough with the model to attempt “scripted” answers based on their interpretation of what they believe the answer should be. Please let your lead appraiser, who has been trained in the model and executed numerous appraisals, guide you through the first discovery appraisal.
Can I extend the scope of my appraisal to include other CMMI practice areas?
Not in your discovery appraisal, but your Lead Appraiser will make recommendations for which Practice Areas to consider in your organization's improvement efforts for the next appraisal.
What does an appraisal cost?
We’re looking at a 2-tiered structure similar to the FDA’s and MDIC’s, with the division at $30M. At the first tier, for organizations making $30M in revenue or more, the appraisal daily rate will be CMMI Institute’s standard rate of $2,250k/person/day, and at the second tier, for organizations making less than $30M in revenue, the CMMI Institute will cut into its standard rate for an appraisal daily rate of $2,000k/person/day. This does not include reimbursable travel expenses. The actual appraisal cost varies depending on the scope of the work, which is determined by looking at the various products manufactured at your site, and understanding the key similarities and differences in the processes for each. Each appraisal will require a team of two, the Lead Appraiser (7 days) and an Appraisal Team Member (5 days). Our experience to date is that most appraisals will only need one team of two (2-total), but some will need two teams of two (4-tota), and a small few might need three teams of two (6-total). To understand the specifics more accurately, you will need to have the initial scoping conversation with your lead appraiser after you’ve enrolled and been assigned one.
What scores do I need to get to “pass” the appraisal?
Any “score” will pass the appraisal; in other words, there is no minimum “score” required to pass the appraisal. This initial appraisal is intended to be a discovery appraisal; meaning that the appraisers want to understand how your organization currently functions, and how it is doing against a set of proven best practices. We expect that most, if not all, organizations will score medium to low in their discovery appraisal- this is not a reflection of the facility or organization’s competency. The goal after that would be to determine the areas of focus that make the most sense of the organization’s business needs, and working together to improve in those areas. There are currently no requirements for how much improvement must be made in any given timeframe.
What does the site need to do to prepare for the appraisal? What people from the site are expected to participate in the appraisal?
Your lead appraiser will begin to have these discussions with you shortly after determining an agreed upon scope, timing, and pricing for the appraisal. The lead appraiser will work with your site coordinator to determine when interviews will be conducted with which group of people. Generally speaking, the appraisal team will need at least one conference room for the duration of the 5-days onsite, plus 1 additional conference room for any additional 2-person team. For further details, please reach out to your lead appraiser.
Who are the people performing the appraisal? Can I pick who they are?
Currently we have a pool of appraisers that have been vetted by the CMMI Institute PMO based on the quality of their appraisal reviews, their knowledge base, and their alignment in the efforts of this pilot to focus on continuous improvement. For 2018, appraisers are subject to availability and will be assigned based on availability. In the future, there may be an option for organizations to choose their lead appraiser.
How is CMMI institute providing oversight to appraisal methodology and appraisers?
The CMMI Institute has a historically proven model, specific qualifications and certifications for lead appraisers, and a clearly documented methodology for executing appraisals. In addition, for this Medical Device Discovery Appraisal Program, specific expectations and training have been defined collaboratively by the CMMI Institute, the FDA, MDIC, and industry partners. Appraiser and appraisal team performance are regularly monitored by the CMMI Institute Project Management Office (PMO).
How many appraisers participate in the appraisal?
This varies depending on the scope of the work, but at least one team of two appraisers will be onsite. If the scope is larger there may be multiple teams of two appraisers (ex: 3 teams of 2 appraisers is 6 total appraisal team members; but only one will be your lead appraiser).
Can I provide my own appraisers?
In order to maintain the integrity of an independent third party, we are not accepting appraisers or appraisal team members from the organization at this time. However, if you are interested in training opportunities for members of your organization, you may reach out to your lead appraiser or contact us at email@example.com.
Approximately how long would an appraisal be?
To be respectful of everyone’s time, we have been trying to maintain a target appraisal length of 5-days onsite.
Do you have an approximate schedule or agenda for an appraisal?
We will work with your organization to determine the details of the schedule, but generally speaking, during the first three to four days, we will be conducting 1-2.5 hour interviews with the individuals who actually do the work in your facility. During the fourth and fifth day, we will be reviewing our interview notes with those individuals to verify and validate what we’ve learned. At the end of the fifth day, we will present these findings with the organization (this includes weaknesses, strength, gaps, and suggestions for improvement).
Can I get a copy of the questions that will be asked in the appraisal?
In the spirit of shifting from a compliance mindset to a quality mindset, we’ve found that it’s best not to share questions beforehand in order to avoid the “scripted answer”. However, your lead appraiser can provide some examples of the questions you’ll be hearing during the appraisal.
What is the heat map tool?
The appraisal data tool with heat map is a standardized document for collecting interview data and documented evidence as needed. Specifically, the heat map is a roll-up of the appraisal results based on the interviews against the varying devices manufactured against the 11 practice areas. It is intended to be an easy-to-read visual representation of your organization’s current standing against the CMMI model. The FDA will only be able to view the cumulative percentages of this heat map, and not the specific interview results, documented evidence, or specific results in each practice.
Where do my appraisal results and performance metrics go? Who has access to them and to what level of detail?
Both of these data will initially go to the CMMI Institute PMO. Your lead appraiser will remove any attributes from the appraisal results before sending (specific names or project details, for example). The PMO will review the appraisal results for quality assurance (approximately 10 - 30 days), then anonymize the data and submit a subsection of the heatmap (only the cumulative percentage of each PA) to the FDA along with the performance metrics collection form. The PMO will anonymize and aggregate the data for trends and baseline purposes, those results will only be available to organizations participating in the pilot. Some general comments about those trends might be made available for public consumption, but not the actual data.
What will be shared and with whom?
How appraisal data is shared depends on who is the recipient of the information. The organization being appraised will receive a set of results reflecting strengths, weaknesses, and notes to consider against the CMMI model’s practices. These results are typically shared in a presentation with all the members of the organization that participated in appraisal activities. The organization’s appraisal sponsor (this is a named member that internal to the appraised organization) will be delivered a documented set of these results.
These are also aggregated at the process area level and then are considered in an overall average. The appraisal team will also deliver these results and the heat map to the CMMI Institute PMO. The PMO will hold the specific data in the confidence but will share the appraised organization’s process area aggregate scores and the overall cumulative score with the FDA. The PMO will also perform analysis of these data over multiple appraisals for trend analysis. No organization’s specific appraisal data will be shared by the PMO or the FDA with any other organization. Trends provided by the PMO to the FDA and the broader MDIC participants will be only provided in a manner that assures participating organizations cannot be identified.
Will there be a public dashboard? Can I see comparative data?
Aggregate data will be shared among the participants during the pilot program on a regular basis once a representative sample size is achieved. This shared information will be deidentified but will allow comparative review across the participants.
Do I get to see other participants’ appraisal results?
Yes, once you’re enrolled in the pilot program and have participated in an appraisal you will be able to see other participants’ deidentified heat maps; however, this data will be anonymized so that there are no names associated with each organization. You will also have access to the trends at that time.
Am I obligated to correct or improve my organization’s appraisal results?
In short, not necessarily. This pilot encourages organizations to focus their improvement efforts on the areas that make the most sense to their facility and operational needs / objectives. We do expect to see effort to improve in those areas, but not a specific amount over a specific timeframe.
Can I work with my lead appraiser further to determine if we are addressing our findings appropriately?
Yes, you do not necessarily need to wait until your next check point; however, the lead appraiser will not be able to update your results outside of those check points.
How frequently do I need to do a regular appraisal?
At present, annually from the date of your initial discovery appraisal.
What if I want to pursue a maturity rating?
You can choose to engage your lead appraiser (or another lead appraiser) in a full appraisal activity resulting in a “level” in addition to your activity within the pilot program. Choosing to do so, or choosing not to do so, will not impact your participation in the pilot program one way or the other.
What happens if I disagree with a finding?
First, please understand that this is a discovery appraisal, that the results do not reflect poorly on your facility via the FDA, and that the results are based on verified and validated interviews with your employees who are doing the work. Second, we expect that most organizations, if not all, will have a medium to low “score” in their initial appraisal because these best practices are not embedded in the industry. Third, if there is some other issue with the way the appraisal was conducted, the CMMI Institute PMO’s Quality Analysts will be able to determine that issue and work with the lead appraiser to resolve it. If there is still a concern, please contact the CMMI Institute PMO at firstname.lastname@example.org
What happens if I want to express concerns about my appraiser or the appraisal team?
If there is concern before the appraisal, please reach out to the CMMI Institute PMO at email@example.com and the Institute will try to address those concerns then (whether directly with the appraisal team, or by assigning your facility a new team). If there are concerns after the appraisal, please also reach out via that email and we will address those concerns via the Quality Analysts.
What regulatory modifications do I receive after the appraisal? When does this happen, how do I know? How long are these good for?
Once the lead appraiser has completed the appraisal, those results will be submitted to the CMMI Institute PMO for a quality review (typically 15-30 days). After successful review, the results will be anonymized and submitted to the FDA. Once the FDA has received and reviewed these results, they will send your organization a letter (via email) to notify you of your submission benefits [more detail from FDA slides]. You should continue to be eligible for these benefits for as long as you’re enrolled in the program, and meet the expectation of continued engagement during your enrollment in the program.
What is the check point process?
The goals of the check point process are to (1) encourage continuous process improvement and (2) encourage engagement in the program. These check points allow for trending of demonstrated progress and regular identification of areas of concern. Check points with the CMMI Lead Appraiser occur at least every six months but can occur quarterly if the organization chooses to do so. The appraisal data tool and “heat map” is updated at these check points as appropriate. Check point occurs on the phone/digitally over the course of 1-2 hours.
How is CMMI and 820 compatible?
There are no incompatibilities between quality system regulations and the CMMI framework. The CMMI framework provides an approach that allows organizations to further improve the quality in their products and services.
Will I need to engage with the FDA?
Most of your engagement in this program will be with and through the CMMI Institute PMO, however, if you have specific questions for the FDA, you can submit those at firstname.lastname@example.org