The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium 9781604300932

Citation preview

The 21st Century Take on Observational Studies Using Real-World Evidence in the New Millennium

/

Copyright© 2018 by FDAnews. Digital version ISBN: 978-1-60430-093-2. Price: $397. All rights reserved. Photocopying or reproducing this report in any form, including electronic or facsimile transmission, scanning or electronic storage, is a violation of federal copyright law and is strictly prohibited without the publisher’s express written permission. This report may not be resold. FDAnews only sells its publications directly or through authorized resellers. Information concerning authorized resellers may be obtained from FDAnews, 300 N. Washington St., Suite 200, Falls Church, VA 22046-3431. Main telephone: (703) 538-7600. Toll free: (888) 838-5578. While every effort has been made by FDAnews to ensure the accuracy of information in this report, this organization accepts no responsibility for errors or omissions. The report is sold as is, without warranty of any kind, either express or implied, respecting its contents, including but not limited to implied warranties for the report’s quality, performance, merchantability, or fitness for any particular purpose. Neither FDAnews nor its dealers or distributors shall be liable to the purchaser or any other person or entity with respect to any liability, loss, or damage caused or alleged to be caused directly or indirectly by this report.

The 21 st Century Take on Observational Studies Using Real-World Evidence in the New Millennium

Table of Contents Introduction 21st Century Research: A New Approach....................................................................................... 3 The Need for Evidence..................................................................................................................... 5 Observational Studies: Research in the Real World..................................................................... 7 How Industry is Using Observational Studies.............................................................................. 10 Challenges of Observational Studies........................................................................................... 17 Trends in Observational Research................................................................................................ 19 Conclusion: Great Expectations................................................................................................... 21 Appendix......................................................................................................................................... 22 A. FDA Guidance for Industry: Use of Real-World Evidence to Support Regulatory DecisionMaking for Medical Devices

Introduction Passage of the 21st Century Cures Act heralds a new era in drug and device development. No longer are sponsors bound to traditional clinical trials to prove safety and efficacy. More flexible, faster and cheaper research methods, including observational studies, are now acceptable in the FDA’s approval process, either as supplements to or substitutes for clinical trials. Observational studies allow drug- and devicemakers to collect real-world usage information from patients and physicians as well as harvest data from existing studies – from university research to patient data registries – to use as evidence of a product’s efficacy and safety. Data from these studies can save manufacturers time and money compared to lengthy clinical research and help them get to market faster. The drawback is that successful observational research requires an entirely different set of procedures and careful planning to ensure the realworld evidence collected is valid and reliable. And it requires an understanding of the kind of evidence the FDA and, perhaps more importantly, other marketplace stakeholders will accept. This management report addresses the opportunities and challenges that observational studies can present. It covers the most effective uses of observational research in both the pre-approval and post-approval phases; how to identify stakeholders and determine what kind of data they need; and how the FDA’s view of observational research is evolving. This report is largely drawn from comments made during an FDAnews webinar by Jeff Trotter, president of Continuum Clinical, which in 2017 published the 10th edition of its Survey on Observational Research. A 30-year veteran of the healthcare industry, Trotter is a thought leader in the design and implementation of research studies generating real-world evidence.

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

21 st Century Research: A New Approach The 21st Century Cures Act, signed into law in December 2016, has mapped a new route to approval of drug and device products that bypasses the traditional clinical trial avenue in favor of studies relying on evidence gathered from real-world practices and experiences. The question on many minds is, will this new approach provide results that are just as reliable as traditional clinical research? Congress and the FDA made their position clear with the passage of the law, and drug and devicemakers lauded its promise for greater innovation, efficiency and expediency in drug approval. Consumer groups, on the other hand, expressed concern that real world-based evidence couldn’t meet the same high bar as the decades-old practice of randomized clinical trials and would lead to public health risks. It’s now up to the FDA to develop real-world evidence standards and a system for evaluating new studies to make sure they have the same level of reliability and control to bridge that gap. The agency issued a final guidance on the use of real-world evidence in developing medical devices in August 2017 (see Appendix A) and began holding public meetings and workshops on the drug side in September 2017. Why the Shift? It has long been acknowledged that controlled clinical trials do not reflect the reality of clinical care. Until recently, however, they have provided the best source of evidence on a new product’s efficacy and safety. Today, many more sources of healthcare information are available that can show how a drug or device performs in normal, uncontrolled practice, a fact that industry and researchers have embraced. “Most patients and most clinicians are bound together by electronic medical records, making data a lot more available through just the normal interaction between physicians and their patients,” says Jeff Trotter, President of Continuum Clinical. “That may have the potential to employ sophisticated analytical tools and really generate the scientific evidence needed to make good decisions.” Trotter points to former FDA Commissioner Robert Califf’s emphasis on leveraging realworld evidence to inform the agency’s decisionmaking. “Rather than relying primarily on randomized clinical trials,” Trotter says, “Califf saw the idea of well-designed, more pragmatic studies as a way of generating high-quality evidence” to use either alongside clinical trials or as a substitute. The Patient’s Voice The 21st Century Cures Act also emphasizes involvement of patient opinions and experiences in drug development. The FDA created several commissions to examine how to bring the voice of the patient into the drug approval process and better understand how to use patientreported information and experience data in reviewing new products. 3

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

“Patient centricity,” Trotter says, “is another very important component of the Cures Act, more formally addressing how we can ensure we are being relevant to you and I as patients, making sure that our voice is represented in clinical trial design and in the measures being implemented.” In addition to therapeutic benefits, the Cures Act addresses economic issues. “The potential cost-effectiveness of drugs,” he says, “is an important variable that is not necessarily critical for drug approval, but it is critical in conversations [with payers] about the value of drugs.” Lower Costs, Faster Approval? Will observational studies and other real world-based studies lead to lower development costs and earlier market introduction? Trotter believes it’s possible. “Observational studies can be cheaper if they’re operationalized appropriately,” he says. They may not be much faster than clinical trials because they often look at long-term outcomes, whereas clinical trials study immediate effects. “But if the idea is that we’re capturing reasonable scientific evidence of a drug working in the real world, and that being potentially a substitute for a longer time period evaluating how a product works under more artificial conditions, then indeed we may see faster approvals.” Trotter doubts that drug prices will be affected, however. “I think there is an opportunity for the industry to achieve some economies and to benefit from greater operational expediency” he says, but don’t count on that lowering market prices. “That’s probably a political issue that we’ll see perhaps in the years to come.” Why Wait? The push for real-world studies didn’t begin with the Cures Act, but more and more it’s a regulatory issue, Trotter says. “Certainly, from a regulatory process, the use of observational methods may change the way we submit products for approval. But there’s ample evidence to suggest that real-world data is critical in documenting the value of a product,” whether from a clinical standpoint, an economic standpoint or patient feelings about the quality of life associated with a product. “All of these in actual medical practice are starting to be used and really have been over the last several years.”

4

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

The Need for Evidence The pharmaceutical and medical device worlds have long relied on controlled clinical research to generate solid evidence of a product’s efficacy and safety to present to regulators for marketing approval. But more and more, the industry is looking at evidence-gathering as a spectrum or a continuum that covers a product’s entire life cycle. Post-marketing surveillance requirements placed on approved products force firms to go beyond the clinical research that is the mainstay of the pre-approval process. And beyond the demands of regulators, once a product is on the market, manufacturers need real-world information on usage, effects and experiences to support its continued viability and acceptance by various stakeholders. That’s where observational studies can be most useful. But for companies that have relied on traditional approaches to controlled clinical research, it’s a whole new world. Many companies fall into the trap of thinking ‘a study is a study is a study’. But there are big differences between how you conduct controlled clinical trials and observational studies. The reason for conducting clinical trials is clear: You need objective, scientific data to present to regulators in a premarket application. But the reasons for performing observational studies are not that cut and dried and can be harder to pin down. You know you need proof of your product’s continuing safety and effectiveness, but what specific kind of evidence do you look for? For instance, are you looking for evidence that your drug can be used safely in elderly patients by surveying hospitals and skilled nursing facilities? Or do you need to interview users of your medical device to determine if a recent rise in complaints is due to unclear user instructions rather than device malfunction? Known v. Unknown Former U.S. Secretary of Defense Donald Rumsfeld, who was once the CEO of a pharmaceutical company, offers this analogy. He said there are known knowns. These are the things we know that we know. They are facts, such as the earth is round. These are the truths in the world. There are known unknowns. These are things that we know we don’t know. To find answers, we need to explore further. So we conduct randomized clinical trials to find out. In this case, you want to understand specific things. You have a hypothesis and want to be able to prove or disprove it. To be able to do that with some level of definitiveness, you need a controlled experiment. But there are also unknown unknowns. These are the things that we don’t know we don’t know. The real world is kind of the world of unknown unknowns. We never know what might be out there that we didn’t have the opportunity to research because we really just didn’t even ask the right questions. We may find that there are some very interesting trends that we can really only uncover through real-world observational research.

5

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Evidence for Different Stages, Different Stakeholders The need for evidence changes as a product moves through its life cycle. In the pre-approval timeframe, you need evidence that a product is safe and efficacious. In the postmarket timeframe, the issues can be dramatically different. You’ve already established the basic efficacy and safety of a product to get it to market and you will continue to gather that kind of evidence, but the focus now shifts to the question of “value.” There are three specific dimensions of value.

ºº Clinical – Does the product achieve its intended clinical outcome? ºº Economic -- Is the product worth it? Is it cost effective? ºº Humanistic -- How do patients really feel about it? What level of quality of life or workplace productivity is associated with a particular product?

Your intended audience also changes. In the pre-approval stage, without question, you are gathering evidence for regulatory authorities. You need to establish efficacy and safety to get a product approved in the first place. Once the product is marketed, however, there may be different stakeholders involved. In the postmarket stage, you are gathering evidence for various “market players,” a much wider group that includes physicians, patients and payers, as well as regulators. (See Figure 1.) Figure 1. The Spectrum of Need for Evidence

Efficacy

Safety

Pre-approval Regulators

Value*

Postmarket Market Players**

*Clinical, economic, humanistic ** Physicians, patients, payers, regulators Source: Continuum Clinical

Your audience’s needs and expectations will determine what information you collect and how you go about collecting it. And while the rules of controlled clinical research are well established, few companies have experience with the more open-ended nature of observational studies.

6

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Observational Studies: Research in the Real World The conditions under which products are tested and evaluated for regulatory approval are often quite different from the conditions under which products are actually used. In the real world, providers and patients have the opportunity to use an approved product in ways a clinical trial may not have addressed. Real-world conditions produce unexpected results. A product may end up being used by a more heterogeneous population than that targeted in controlled trials. Patients often are using other products concurrently or not following usage instructions. And physicians frequently prescribe products for off-label indications. It’s impossible to control the conditions under which a product is used in the real world. That’s why observational studies are a particularly effective tool to gauge actual use in the postmarket phase. Because they don’t rely on controlled conditions or mandated procedures, they have the flexibility to respond to factors generally not comfortably embraced in clinical trial protocols. But that’s also what makes observational studies more challenging to implement. There is no black or white, no right or wrong answers. In observational studies, you are basically “looking over the shoulders of physicians and patients” to see what happens when they use the product. At the same time, to ensure that the findings are meaningful, you must conduct observational studies in an appropriately rigorous and scientific manner. Different Models for Studies Observational studies can be used in a variety of ways. Here are some examples.

ºº Identifying risk factors. Observing and studying groups of people over a period of

time can be an effective way to gather information on what factors contribute to developing a particular health condition. The five decade-long Framingham Heart Study is a prime example of this approach (see box).

ºº Understanding actual practice conditions. This includes factors such as how clinicians manage patients, how patients flow through the system and what kinds of outcomes are available in the real world.

ºº Documenting the pre-approval status quo. Observational studies can be used in the

postmarket phase to gather evidence that what was promised in a marketing application is actually occurring once the product is distributed.

ºº Documenting real-world value. This requires a determination of what happens from a clinical perspective, from an economic standpoint and from a humanistic standpoint.

ºº Establishing treatment guidelines. This means understanding the optimal use of products and the optimal outcomes. One of the more interesting uses of real-world research is finding out whether there’s a cohort of patients that seem to be doing really well. If that is the case, what are the predictors of those positive outcomes? Are they clinical

7

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

factors, economic factors or an aspect of quality of life? Can we standardize treatments based on those guidelines so that we can optimize outcomes going forward?

ºº Engaging with patients and practitioners in their “natural habitats.” Observational

research can be used to help understand the dynamic between patients and practitioners, and the impact of such interactions on outcomes.

Identifying Risk – The Framingham Heart Study The Framingham Heart Study is a project of the National Heart, Lung and Blood Institute and Boston University. It began in 1948, when study scientists and participants undertook an ambitious project to identify risk factors for heart disease. The study continues to this day and remains a center for cutting-edge heart, brain, bone and sleep research. The Framingham project has followed the development of cardiovascular disease over a long period of time in three generations of participants. It began with an original cohort of 5,209 men and women between the ages of 30 and 62 who lived in the town of Framingham, Mass. Researchers interviewed those participants about their lifestyle and conducted physical exams that they would later analyze for common patterns related to the development of heart disease. Since 1948, the participants have returned every two years for a detailed medical history, physical exam and lab tests. The study has enrolled a second generation that includes 5,124 of the original participants’ adult children and their spouses. In 2002, the study enrolled a third generation, the grandchildren of the original group. Since the first study, the project has continued and has led to the identification of major risk factors for heart disease as well as the effects of blood pressure, blood triglyceride, cholesterol levels, age, gender and psychosocial issues. Researchers today continue to investigate risk factors for other conditions, including dementia, and study genetic patterns. Ways to Use Observational Studies Q&A Question: How can observational studies be used in formal postmarket safety assessments? Answer: In the past couple of years, use of post-approval safety surveillance initiatives has increased. In Europe in particular, almost all observational studies sponsored by pharmaceutical companies must reflect regulatory guidelines for postmarket safety surveillance. And while there is always an obligation to capture adverse event data in any observational study while you may not have intended the study to be used for postmarket surveillance, regulatory authorities increasingly are interested in capitalizing on the opportunity to capture the kind of data those studies can provide. So when you design an observational study aimed at safetyrelated issues, you have to consider the expectations of regulators, which may be quite different from other stakeholders in your audience. Q: What about global or multinational observational studies? Can you use the same operational approach in all countries? 8

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

A: You can use the same methods and processes, but you need to adjust for the reality of each different country. Your study needs to reflect differences in how treatment is delivered, how products are used. And then there are issues of privacy laws in terms of how to analyze the information and whether to use patient-identifiable information. You need to account for the laws and regulations in each different country. There may also be situations in which one particular country views the seemingly benign use of a questionnaire to a patient as interventional, whereas another country might view that as observational. You may need to change the very processes that need to be undertaken given those two conflicting perspectives. The reality is that there isn’t such a thing as a purely observational study. Even by virtue of asking a question, we may be inspiring a physician or a patient to think a little bit differently. It’s really hard to gauge the impact of that. The bottom line is that there are great opportunities to conduct these studies globally, recognizing that we can use the same machinery, but we have to accommodate nuances that may exist on a country-by-country basis.

9

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

How Industry is Using Observational Studies Two recent surveys of companies currently using observational research highlight some of the issues they face. Continuum Clinical Survey Currently in its ninth iteration, Continuum Clinical’s Survey on Observational Research was initiated in response to requests for help from research sponsors that clearly didn’t understand the nature of observational studies. “They would ask for us to conduct perhaps a 10,000-patient or a 5,000-patient observational study,” says Continuum president Jeff Trotter, “and at the same time, within the same document, there were instructions for randomization schedules and fixed visit schedules and all sorts of things that seemed to contradict the whole concept of undertaking an observational study.” The industry in general wants things to be black and white, Trotter says, but observational research lies somewhere in between, in a shade of gray that sometimes organizations have trouble with. “Our goal with the survey was to pinpoint areas of uncertainty, identify best practices, try to shed some light on some of these issues that seem to be the source for this lack of consistency and the contradictions that we saw in many of the [requests].” The survey reflects the perspectives of a broad sample of the industry, with about one-third of responses coming from pharmaceutical companies, device manufacturers and biotech organizations. Another third of the responses are from consulting organizations and contract research organizations. The remaining third are from other organizations, with a heavy representation from academic institutions. Sometimes, payers and providers also respond to the survey, as well as regulators, Trotter says. (For a copy of the survey summary, “Real-World Research: Opportunities and Challenges,” go to http://continuumclinical.com/research.) There are a lot of disciplines involved in carrying out observational studies, Trotter says. Individuals come from the fields of epidemiology, pharmacovigilance, biostatistics and consultants to the industry—all of them focusing on a variety of different issues. The studies can also involve executive management, clinical study operations, outsourcing and procurement personnel, medical affairs, and marketing and product management, as well as academics. This diversity makes reaching a consensus on study design and implementation difficult. People have different training, different skill sets, and different perspectives that they bring to the table. You need to respect and capitalize on those different perspectives to create an optimal observational study. Survey Finding – When Observational Research Takes Place Eighty-two percent of survey respondents said they conduct observational research as part of the postmarket phase. The remaining 18 percent use it primarily as part of the pre-approval process. (See Figure 2.) 10

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Figure 2. Pre-approval v. Postmarket Use of Observational Studies 18.0% 82.0%

Primarily pre-approval Primarily post-approval

Source: Continuum Clinical

Certainly, observing how a specific product performs in the real world can only be undertaken post-approval when the product is actually used by patients. But observational studies also have a role to play in the pre-approval process. Pre-approval research isn’t focused necessarily on a particular product—since the product isn’t available at that point—but rather on articulating the landscape of a particular area of need to develop a baseline against which to evaluate the world once the product is released. Or such a study could help a company understand the opportunity for improvement by documenting the outcomes and the treatments that are available today. Studies can also help medical professionals understand the best treatment for their patients. For instance, many interesting findings have resulted from the National Registry of Myocardial Infarction, a large prospective U.S. registry with data on more than two million heart attack patients. In one case, researchers looked at how those patients were triaged to determine the optimal time to begin thrombolytic therapy to achieve the best outcomes. Survey Finding – Reasons for Using Observational Studies Just why are organizations conducting observational studies? Trotter says it’s important to think about the rationale for the study. Why are you doing the study? Is there a scientific premise? Or a business purpose? What are your expectations for using this information? Is it going to be used to influence a regulatory body? Is it going to be used to influence a payer or a physician? What information are you looking to use, and have you reflected the values of those people that you’re trying to influence? “It’s extremely important to kind of back the train up to the station and understand why we are doing this study in the first place,” Trotter says. The two top reasons for using observational studies cited in the survey were: 1. To document real-world clinical outcomes (see box for an example of a clinical outcomes study); and 2. To document real-world product cost effectiveness.

11

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Assessing Real-World Effects – The Air Force Health Study The Air Force Health Study was undertaken by the U.S. Air Force in 1979 to assess the possible health effects on Air Force personnel exposed to Agent Orange and other chemical defoliants used during the Vietnam War. The study primarily examined members of Operation Ranch Hand, the unit responsible for aerial spraying of herbicides from 1962 to 1971, looking for problems caused by Agent Orange shortly after exposure, such as skin rashes. But a longer-term aspect of the study followed the subjects until 2006 to see if other problems, such as infertility and cancer, might surface further down the road after exposure. The reports that resulted from the study suggested links between Agent Orange exposure and nine diseases that included Hodgkin’s disease, multiple myeloma and respiratory cancers. Other reasons included:

ºº To establish product effectiveness; ºº To provide support for reimbursement coverage or payment; ºº To examine direct medical research utilization or cost; ºº To explore product compliance and adherence; ºº To establish the safety profile of a product; ºº To satisfy regulators’ requirements for postmarketing studies and data; ºº To document “real-world” practice; ºº To understand the sequencing of therapies; and ºº To characterize the current standard of practice. Survey Finding – Goals of Observational Studies Now let’s consider organizations’ goals for their observational studies. What is the specific output or deliverable sought? Many organizations just solicit support for studies that they expect they will be submitted to the FDA or EMA, Trotter says. But that may not be the reason for undertaking the study in the first place. It’s important to think about the outputs. The goal may not be to gather data to provide a submission to a regulatory agency, but rather exclusively to publish the results of a study. For instance, you may want to publish your study to show how your drug compares to others in treating a disease. Physicians may use that data to make decisions about what drug they prescribe to their patients. Survey respondents listed a wide variety of goals for their observational studies, including:

ºº Gathering data for publications or presentations; ºº Gathering data for value/reimbursement discussions with health authorities and/or payers; 12

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

ºº Documenting the role of a particular product in “real-world” management of a health issue; ºº Documenting (post-approval) product safety for regulatory purposes; and ºº Supporting labeling changes. Tufts University Study More recently, Tufts University’s Center for the Study of Drug Development (CSDD) gathered insights on how the industry is responding to the challenge. “We went out into the literature and tried to find some benchmarking data on the use of realworld evidence within sponsor and CRO companies,” says CSDD Associate Director of Sponsored Research Mary Jo Lamberti. “We didn’t find a lot.” CSDD’s study aimed to include a baseline assessment of the current and planned uses of real-world evidence and real-world data, operational approaches and return on investment (ROI) measurements. The center began by interviewing with 19 respondents from five stakeholder groups across 16 organizations. Respondents hold various roles and responsibilities, but the majority occupy director-level positions and above. They represent stakeholders such as regulators, vendors, payers, health systems and patient advocacy groups. These interviews found that, among the respondents, there is no common definition of either real-world evidence or real-world data. What the stakeholders do have in common, Lamberti says, is a lot of challenges, such as lack of standardization of data, data privacy and use of unstructured data. Using lessons learned from these interviews, CSDD created a questionnaire for a web survey of 75-100 global pharmaceutical and biotechnology companies and CROs. The survey examined current data uses, sources of data and how data is being integrated, operational support of real-world data use (e.g., functions, personnel, roles and responsibilities, skill sets) and the ROI/performance areas impacted. Survey Finding – Real-World Data Function and Cost Nearly all companies surveyed have a real-world data function, and those centralized functions are supported by an average of six other functions and have 11 in-house datasets available (see Figures 3 and 4). Figure 3. Area of Expertise of Real-World Data Function Staff Area of Expertise

Responses

Epidemiology

18

Data Management/Analytics

11

Health Economics and Outcomes Research

8

Statistics/Biostatistics

7

Clinical Research

4

Source: CSDD

13

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Figure 4. Departments to Which Real-World Data Function Reports Department

Responses (n=27)

Medical Affairs

11

Global Statistics

2

Marketing

2

Information Technology

2

CEO

1

Commercialization and Outcomes Research

1

Chief Medical Office

1

Clinical Platform

1

Innovative Health

1

Patient Access and Quality

1

Health Economics and Outcomes Research (within R&D)

1

Scientific Affairs

1

Commercial

1

Source: CSDD

Of the companies surveyed, 86 percent indicate that their anticipated real-world evidence budget will be increasing in the coming three years, most commonly by 11-25 percent. At present, only 20 percent of companies spend more than $5 million on analytics, while 60 percent of companies spend more than $5 million on research studies and data. Survey Finding – Top Challenges Survey respondents identified availability of data as their biggest challenge, Lamberti says. “That’s really looking at, is there robust data available to be used? And there are gaps in the data as well,” she says. The second biggest challenge is lack of external stakeholder trust in real-world evidence, followed by the cost of acquiring data and the cost and effort of data integration. “Some companies really have to assess whether they want to invest and put the money into integrating the different sources of data.” Survey Finding – Labeling Survey respondents have very little direct experience with regulatory agencies accepting real-world evidence for labeling or labeling changes, Lamberti says, but have more experience with it being used post-approval for safety assessments and in postmarketing studies. Survey Finding – Return on Investment The most valuable ROI measures for current real-world evidence activities are market access, greater insight into therapeutic area needs and greater insight into drug effectiveness. There is little change expected in the top ROI measures for the next one to three years. 14

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

“We didn’t really see a lot of companies,” Lamberti says, “using [real-world data] to assess patient experience or quality measures or patient reported outcomes.” “I think we’re going to see more of the social media data,” she says, “the wearables and the personal health devices as we move forward. I think that’s a more promising area, gathering patient data.” Figure 5. Top Challenges of Working With Real-World Data 70% 60% 50% 40% 30% 20% 10% 0%

Source: CSDD

Examples of Studies Q&A Question: Can you give some examples of studies you’ve undertaken recently and, importantly, their strategic and scientific goals? Answer: A slight trend that we’re seeing is more and more disease advocacy organizations starting to develop their own observational studies. These organizations really have the pulse of the patient and interact with them through internet-based technology. Many of the larger organizations and foundations are starting to develop their own observational studies, which tend to be direct-to-patient programs. The strategy behind them is to ultimately be consistent with their advocacy goal and to find out if there are opportunities to improve patient outcomes just by virtue of improving connectivity with various patient groups. Some of the observational studies are actually subsets of larger initiatives that seek to facilitate improved social connections between patients within particular disease areas. 15

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

But there are some subsets of some of those initiatives that really focus on gathering observational data and to solicit responses from patients on their lifestyle, on their experiences, on how they view their disease, quality of life and issues like that. There may be some challenges in doing these studies in terms of really understanding how specific drugs are utilized. And we can’t expect the average patient to have a good comfort level with clinical lexicon. Fundamentally however, the strategy underlying these studies is to really perpetuate the advocacy role of the disease organization. The classic observational studies we’ve seen, however, and still the vast majority of the studies that we get involved with tend to be sponsored by a pharmaceutical company or a device company. The company either has a product where they really want to understand how it’s working in the real world. Or, as part of the pre-approval process, they want to understand the landscape against which to evaluate the world once their product is released. That’s a theme we see a fair amount of these days. The strategy underlying the study is to develop evidence, to develop information that ultimately will demonstrate the product’s value. The science behind these is really intrinsic to every study that’s done. We need to, in many ways, work backwards from the key themes and the issues that we want to explore. So, for example, if we want to look at what’s the most cost-effective treatment regimen in a particular disease area, we need to think about the definition of cost effectiveness. What are we going to be measuring? What effectiveness measure are we going to be using? How are we going to capture cost information? And then we work backwards to ensure, ultimately, that the data we collect are going to be as valid as possible. We think about the limitations of the study in terms of what we actually can say, given the fact that there may be biases every step of the way in how sites are selected and patients enrolled. Again, these are the complexities that are intrinsic to studies in the real world. But employing scientific methods and ensuring that we’re looking very carefully at how we want to analyze the information, what level of granularity we need to be able to make statements, that’s really intrinsic in every study. Again, the strategy may vary, but the science, frankly, shouldn’t.

16

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Challenges of Observational Studies There are three common mistakes companies typically make when trying to plan observational studies. The first is lack of clarity about the purpose of the study. Too often, the question “why” is an afterthought. The starting point of an observational study should be knowing what you want to achieve, both scientifically and strategically. Companies that jump into an observational study without establishing its fundamental purpose are at a disadvantage from the get-go. While that approach may work for traditional preapproval clinical research, where the obvious purpose is to get approval for the product, there can be various strategic rationales underlying the need for observational research. For example, you may want to assess the post-approval safety of a product or the current treatments for a particular disease before you move forward with development of a new medication. Each requires a different operational approach. Determine first why you are doing a study and who you are trying to influence. Then work backwards and consider what you want to examine; what analysis you need to perform to explore those questions or issues; what data is required to support the analyses and how do you capture that data? Cross-Functional Cooperation The second common mistake is failing to ensure cross-functional cooperation in funding, designing and implementing observational studies. Organizations assign responsibility for observational studies in a variety of ways. Typically, the health economics and outcomes research departments are heavily involved in all aspects of these studies. In some organizations, medical affairs may be involved in funding and overall coordination while clinical study operations may get involved in implementation. Or funding may come from the commercial side, with design by medical affairs or health economics, and implementation by clinical operations. Dealing with individuals who have fundamental differences in perspective, training, interests, etc., can be problematic. The key to a successful effort is keeping all parties focused on the essential “why” of the study. Specific Operating Procedures Many pharmaceutical and medical device companies do not have SOPs that are appropriate for observational research. Without specific procedures, companies run the risk of falling back on the more familiar clinical research procedures, which aren’t suited to the nature of observational research at all. So it’s worth taking the time to craft new SOPs. Observational research SOPs should address issues such as site selection, data quality, monitoring, site management, data management and analytical conclusiveness. 17

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

One resource companies can turn to for guidance in planning and conducting observational studies is the STROBE Initiative at the University of Bern, Switzerland. This international collaborative initiative is made up of epidemiologists, methodologists, statisticians, researchers and journal editors involved in conducting and disseminating observational studies. You can find resources for developing SOPs on the STROBE website at www.strobestatement.org.

18

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Trends in Observational Research Observational research generally focuses on gathering information from current activities. But the increasing availability of electronic medical records and other new technology presents opportunities for capturing information retrospectively and, potentially, in the “purest” observational mode. A retrospective study looks back at events that already have taken place. Investigators collect data from past records rather than following patients during a period of time as they would in a prospective study. For instance, researchers may look at medical records of groups of individuals exposed to suspected risk factors in relation to an outcome. The big distinction in retrospective studies is that all of the cases of disease have already occurred before the investigators initiated a study. Rather than establishing a study group and following members over time to see if there is the development of a disease, researchers look back. For example, they could look to medical records to determine whether use of a particular drug was associated with the desired outcome, whether there were lots of side effects, whether concomitant agents were used, etc. Example of a Retrospective Study – Source of Infection In 2003, epidemiologists from the Massachusetts Department of Public Health began a retrospective investigation of an outbreak of giardia –a microscopic parasite that causes a diarrheal illness—in a Boston suburb to identify the source of the infection. The investigation began after the start of the outbreak, which was already winding down, and concentrated on members of a particular golf club. The club had two swimming pools, one for adults and a wading pool for small children. Investigators thought the cases were most likely related to contamination of the wading pool by a child shedding giardia into their stool. By separating club members by those who said they had used the wading pool from those who had not and determining how many people in each group were infected, investigators determined the risk level for each group and found those who used the wading pool were more than three times more likely to have had contracted the illness. One reason for choosing to conduct retrospective studies is that it can be quicker and more cost effective to gather data that already exists in electronic sources than to start from the beginning. But despite improvements in electronic data gathering, retrospective studies have their drawbacks. For instance, data systems often don’t “talk” to each other, making it difficult to aggregate information and draw valid conclusions. And rarely do we see patient-reported outcomes included routinely in these databases. So an important dimension -- the patient perspective – may be missing from the study. Fundamentally, in retrospective research, you are entirely dependent on the data that were collected, which may not be the data you’re seeking. Innovative Uses Other trends in the observational research world include more direct-to-patient observational studies in which researchers solicit the patient perspective with no physician involvement. 19

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Also, increasingly over the past couple of years, more disease advocacy groups and foundations are starting to undertake their own observational studies using patient registries. Respondents to the Continuum Clinical survey say that there are times when observational research can accelerate product approval, be used as the basis to support a critical clinical endpoint and even be used to justify a label change. More and more companies are pushing the envelope and trying to use observational data to explore the opportunity to change a product label. The intent is to gather enough evidence that a product is being used in a certain way to convince the FDA to expand its label, but the jury is still out on whether such an approach is sufficiently definitive. Quality of Data Observational studies do present some challenges in terms of the quality of the data, how valid it is, and how researchers can monitor information gathered from patients. And it is important to recognize that research needs to be done transparently. One survey respondent reported being specifically told by a sponsor not to collect a vital piece of information in an observational study because it might show evidence of a potential adverse event. That negates the whole purpose of an observational study: to find out what truly happens in the real world. You can’t just brush some important bad news under the table. New frontiers The industry is entering an exciting era in which observational information can be gleaned very naturally through the use of advanced technology. These data sources, in fact, may be ideally suited for observational research. Where clinical research relies on strict rules and protocols to protect data integrity and validity, observational studies can access information more informally and quickly by using, for example, cell phones for information gathering or wearable devices for tracking activity. We can collect data just by virtue of the fact that patients are wearing or carrying a device in their pocket.

20

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Conclusion: Great Expectations When pharmaceutical and medical device companies sponsor research, they expect to get what they pay for – information they can use to develop or improve products and enhance the bottom line. Clinical research provides conclusive, statistically significant data they can act on. But what level of conclusiveness can they expect from observational research? It’s important to understand that observational research is more about exploring issues than about finding answers. Most respondents to the Continuum Clinical survey agree that findings from observational studies should be more influential and directional than conclusive (see Figure 3). But how do you assign a value to something so subjective? Figure 3. Expected Level of Conclusiveness As definitive as findings from RCTs Definitive, but with caveats Not necessarily conclusive, but influential/directional Exploratory only: no expectation of conclusiveness Not at all conclusive

Source: Continuum Clinical

Again, we must go back to why the study was conducted in the first place. A study may not produce a definitive answer to a specific question but still be considered a success as long as the results match the rationale. If payers, for example, are looking for evidence of a product’s or treatment’s real-world value, an observational study that focuses on how patients and practitioners are using it and how satisfied they are with it can be just as informative as controlled clinical data on its effectiveness. But what about regulators’ expectations? As authorities like the FDA grant accelerated approval – without the usual lengthy research phase – to more products, it’s increasingly important to keep a close eye on how those products perform in the real world, a purpose uniquely suited to observational studies. The FDA has been cautious about accepting observational research in the past but is now showing signs of broadening its perspective. It’s a rapidly evolving area, and research sponsors need to stay on top of developments at the FDA and other regulators around the world to make sure their observational studies meet the criteria needed to gain acceptance. At the same time, study sponsors must recognize that stakeholders other than the FDA are potentially more important when it comes to findings from observational research.

21

The 21st Century Take on Observational Studies: Using Real-World Evidence in the New Millennium

Appendix A. FDA Guidance for Industry: Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices

22

Appendix A: FDA Guidance for Industry: Use of RealWorld Evidence to Support Regulatory DecisionMaking for Medical Devices

Contains Nonbinding Recommendations

Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices Guidance for Industry and Food and Drug Administration Staff Document issued on August 31, 2017. The draft of this document was issued on July 27, 2016 For questions about this document regarding CDRH-regulated devices, contact the Office of Surveillance and Biometrics (OSB) at 301-796-5997 or [email protected]. For questions about this document regarding CBER-regulated devices, contact the Office of Communication, Outreach, and Development (OCOD) at 1-800-835-4709 or 240-402-8010.

U.S. Department of Health and Human Services Food and Drug Administration Center for Devices and Radiological Health Center for Biologics Evaluation and Research

1

Contains Nonbinding Recommendations

Preface Public Comment You may submit electronic comments and suggestions at any time for Agency consideration to http://www.regulations.gov . Submit written comments to the Dockets Management Staff, Food and Drug Administration, 5630 Fishers Lane, Room 1061, (HFA-305), Rockville, MD 20852. Identify all comments with the docket number FDA-2016-D-2153. Comments may not be acted upon by the Agency until the document is next revised or updated.

Additional Copies CDRH Additional copies are available from the Internet. You may also send an e-mail request to [email protected] to receive a copy of the guidance. Please use the document number 1500012 to identify the guidance you are requesting.

CBER Additional copies are available from the Center for Biologics Evaluation and Research (CBER), Office of Communication, Outreach, and Development (OCOD), 10903 New Hampshire Ave., Bldg. 71, Room 3128, Silver Spring, MD 20993-0002, or by calling 1-800-835-4709 or 240-4028010, by email, [email protected] or from the Internet at http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guida nces/default.htm.

2

Contains Nonbinding Recommendations

Table of Contents I. Introduction and Scope ......................................................................................................... 4 II. Background .......................................................................................................................... 5 III. Real-World Evidence ........................................................................................................... 8 IV. Regulatory Context in Which RWE May be Used ............................................................... 9 A. General considerations for the use of RWE ......................................................................... 9 B. Application of Investigational Device Exemption (IDE) Requirements in 21 CFR 812 to the Collection of RWD .............................................................................................................. 11 V. Characteristics of RWD...................................................................................................... 12 A. Relevance ........................................................................................................................... 13 B. Reliability........................................................................................................................... 15 (1) Data accrual ................................................................................................................ 15 (2) Data assurance - Quality Control................................................................................ 16 VI. Examples Where RWE is Used .......................................................................................... 17 A. Expanded Indications for Use ............................................................................................ 17 B. Postmarket Surveillance Studies (Section 522) ................................................................. 18 C. Post-Approval Device Surveillance as Condition of Approval ......................................... 19 D. Control Group .................................................................................................................... 19 E. Supplementary Data ........................................................................................................... 20 F. Objective Performance Criteria and Performance Goals ................................................... 20 VII. Glossary .............................................................................................................................. 21

3

Contains Nonbinding Recommendations

Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices Guidance for Industry and Food and Drug Administration Staff This guidance represents the Food and Drug Administration’s (FDA’s) current thinking on this topic. It does not create or confer any rights for or on any person and does not operate to bind FDA or the public. You can use an alternative approach if the approach satisfies the requirements of the applicable statutes and regulations. If you want to discuss an alternative approach, contact the FDA staff responsible for implementing this guidance. If you cannot identify the appropriate FDA staff, call the appropriate number listed on the title page of this guidance.

I.

Introduction and Scope

FDA is issuing this guidance to clarify how we evaluate real-world data to determine whether they are sufficient for generating the types of real-world evidence that can be used in FDA regulatory decision-making for medical devices. This guidance is applicable to all devices, as that term is defined under section 201(h) of the Federal Food, Drug, and Cosmetic Act (the FD&C Act), including software that meets the definition of a device. Real-World Data (RWD) are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. Examples of RWD include data derived from electronic health records (EHRs), claims and billing data, data from product and disease registries, patient-generated data including in homeuse settings, and data gathered from other sources that can inform on health status, such as mobile devices. RWD sources (e.g., registries, collections of EHRs, and administrative and healthcare claims databases) can be used as data collection and analysis infrastructure to support many types of trial designs, including, but not limited to, randomized trials, such as large simple trials, pragmatic clinical trials, and observational studies (prospective and/or retrospective ). Real-World Evidence (RWE) is the clinical evidence regarding the usage, and potential benefits or risks, of a medical product derived from analysis of RWD. 4

Contains Nonbinding Recommendations

Under the right conditions, data derived from real world sources can be used to support regulatory decisions. RWD and associated RWE may constitute valid scientific evidence depending on the characteristics of the data. This guidance should not be construed to alter, or change in any way, the existing evidentiary standards applicable to FDA’s regulatory decisionmaking; rather, it describes the circumstances under which RWD may be used to support a variety of FDA decisions based on the existing evidentiary standards. While FDA encourages the use of relevant and reliable RWD, this guidance neither mandates its use nor restricts other means of providing evidence to support regulatory decision-making. This guidance highlights some of the potential uses of RWD, and describes the factors that FDA considers when evaluating whether specific RWD is of sufficient quality to inform or support a regulatory decision. It also clarifies when an Investigational Device Exemption (IDE) may be needed to prospectively collect and use RWD for purposes of determining the safety and effectiveness of a device. This document does not address the use of non-clinical data, adverse event reports, secondary use of clinical trial data (e.g., post hoc analyses), or systematic literature reviews. Nor does it address study design/conduct or analytical methodologies. While it does describe the factors that FDA considers when evaluating RWD or RWE, it does not provide a specific set of pass/fail criteria or other scoring tools for making a determination about the suitability of RWD or RWE for a particular regulatory decision. This guidance does not affect any federal, state or local laws or regulations, or foreign laws or regulations that may be applicable to the use or collection of RWD, or that provide protections for human subjects (including informed consent requirements) or patient privacy. This guidance should be used to complement, but not supersede, other device-specific and good clinical practice guidance documents. FDA's guidance documents, including this guidance, do not establish legally enforceable responsibilities. Instead, guidance documents describe the Agency’s current thinking on a topic and should be viewed only as recommendations, unless specific regulatory or statutory requirements are cited. The use of the word should in Agency guidance means that something is suggested or recommended, but not required.

II.

Background

To protect and promote the public health, FDA needs to understand and evaluate the available evidence related to regulated products.1 For medical devices, available evidence is traditionally comprised of non-clinical and, in some cases, clinical studies conducted and provided to FDA by the device manufacturer or sponsor. However, FDA recognizes that a wealth of RWD covering medical device experience exists and is routinely collected in the course of treatment and management of patients. Data collected during clinical care or in the home setting may not have the same quality controls as data collected within a clinical trial setting. Even so, under certain 1

FDA’s What We Do (http://www.fda.gov/AboutFDA/WhatWeDo/default.htm).

5

Contains Nonbinding Recommendations

circumstances RWD may be of sufficient quality to help inform or augment FDA’s understanding of the benefit-risk profile of devices at various points in their life cycle. RWD, which are typically collected for non-regulatory purposes in EHRs, registries, and administrative and claims data, may provide new insights into the performance and clinical outcomes associated with medical device use. This information can potentially be used by sponsors to demonstrate compliance with regulatory requirements and to aid FDA in our regulatory decision-making. FDA has issued guidance documents on premarket and postmarket data collection,2 benefit-risk determinations,3 patient preference information,4 and expedited access to medical devices for unmet medical needs in order to streamline the process for bringing new technologies to market while maintaining the assessment of reasonable assurance of safety and effectiveness of medical devices. FDA has also issued plans for, and has begun implementation of, the National Evaluation System for health Technology (NEST)5,6,7,8 to leverage RWD in order to more quickly identify safety problems and to better understand the benefit-risk profile of devices used in clinical care. FDA believes that, if leveraged correctly, the NEST may also help to reduce the time and cost of generating the types of evidence used to support the marketing authorization of FDA-regulated products and to meet postmarket study and reporting requirements. Devices are often used in routine clinical practice for uses that are not within their cleared or approved indications for use. However, the knowledge gained from all uses of a device in medical practice is often not realized because the data collected are not systematically characterized, aggregated, and analyzed in a way that can be relied upon to inform regulatory decision-making. By recognizing the value of RWE as an important contributing factor for understanding and regulating medical devices, we hope to encourage the medical community to learn more from routine clinical care than we do today. 2

Balancing Premarket and Postmarket Data Collection for Devices Subject to Premarket Approval (http://www.fda.gov/ucm/groups/fdagov-public/@fdagov-meddev-gen/documents/document/ucm393994.pdf). 3 Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications (http://www.fda.gov/ucm/groups/fdagov-public/@fdagov-meddevgen/documents/document/ucm517504.pdf) 4 Patient Preference Information – Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling (Patient Preference Information – Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm446680.pdf). 5 Strengthening Our National System for Medical Device Postmarket Surveillance (http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe ports/UCM301924.pdf). 6 Strengthening Our National System for Medical Device Postmarket Surveillance: Update and Next Steps - April 2013 (http://www.fda.gov/downloads/MedicalDevices/Safety/CDRHPostmarketSurveillance/UCM348845.pdf). 7 Strengthening Patient Care: Building a National Postmarket Medical Device Surveillance System (http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe ports/UCM435112.pdf). 8 Recommendations for a National Medical Device Evaluation System: Strategically Coordinated Registry Networks to Bridge the Clinical Care and Research - August 2015 (http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe ports/UCM459368.pdf).

6

Contains Nonbinding Recommendations

FDA will use the criteria described in this guidance to evaluate whether RWD are of sufficient quality to support regulatory decision-making, including potentially generating valid scientific evidence.9 FDA relies only upon valid scientific evidence to determine whether there is a reasonable assurance that a device is safe and effective. However, it is possible that RWE can meet this threshold under conditions where the underlying RWD were accurately and reliably captured at clinically relevant time intervals throughout the device lifecycle. Under the right conditions, RWE may be suitable to support the clearance or approval of a new device, or the expansion of the indications for use of devices that are already on the market. RWE may also be used to supplement the total evidence required for such clearances or approvals. Other applications of RWE in premarket decision-making may also be possible, particularly as RWD systems and analysis methodology advance. Additionally, aggregation of RWD (e.g., in medical device registries) may prove useful as a postmarket control suitable for providing ongoing device safety surveillance and additional evidence for effectiveness. FDA has long applied postmarket controls as a way to reduce premarket data collection, while still ensuring that the statutory standard of reasonable assurance of safety and effectiveness is met.10 FDA believes that applying postmarket controls to reduce premarket data collection, when appropriate, can help improve patient access to safe and effective medical devices.11 In some cases, a “traditional” clinical trial may be impractical or excessively challenging to conduct. Ethical issues regarding treatment assignment, and other similar challenges, may present themselves when developing and attempting to execute a high quality clinical trial. Analyses of RWD, using appropriate methods, may in some cases provide similar information with comparable or even superior characteristics to information collected and analyzed through a traditional clinical trial. For example, RWD collected using a randomized exposure assignment within a registry can provide a sufficient number of patients for powered subgroup analyses, which could be used to expand the device’s indications for use. However, not all RWD are collected and maintained in a way that provides sufficient reliability. As such, the use of RWE for specific regulatory purposes will be evaluated based on criteria that assess their overall

9

“Valid scientific evidence is evidence from well-controlled investigations, partially controlled studies, studies and objective trials without matched controls, well-documented case histories conducted by qualified experts, and reports of significant human experience with a marketed device, from which it can fairly and responsibly be concluded by qualified experts that there is reasonable assurance of the safety and effectiveness of a device under its conditions of use. The evidence required may vary according to the characteristics of the device, its conditions of use, the existence and adequacy of warnings and other restrictions, and the extent of experience with its use.” [21 CFR 860.7(c)(2)] (http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?FR=860.7) 10 The Least Burdensome Provisions of the FDA modernization Act of 1997: Concept and Principles, (http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm085999.p df) 11 Balancing Premarket and Postmarket Data Collection for Devices Subject to Premarket Approval, (http://www.fda.gov/ucm/groups/fdagov-public/@fdagov-meddev-gen/documents/document/ucm393994.pdf).

7

Contains Nonbinding Recommendations

relevance and reliability. If a sponsor is considering using RWE to satisfy a particular FDA regulatory requirement, the sponsor should contact FDA through the pre-submission process.12

III. Real-World Evidence RWE can exist across a wide spectrum, ranging from observational studies within an existing dataset to studies that incorporate planned interventions with or without randomization at the point of care.13 Because of the rapidly advancing methodology for generating and interpreting RWD, this guidance will not elaborate on the methodological approaches that can be used. However, when reviewing the use of RWE to support a regulatory decision, FDA will rely on scientifically robust methods and approaches to determine whether submitted RWE is of sufficient quality to support a particular regulatory decision. Clinical trials are designed to control variability through detailed eligibility criteria and carefully designed clinical protocols performed by specialized research personnel. They require intensive monitoring and data auditing to demonstrate that use of a device produces the expected results. Although useful in establishing a baseline for device performance, clinical trials may be narrow in scope due to practical challenges. In contrast, studies leveraging RWD can potentially provide information on a wider patient population, thus providing information that cannot be obtained through a traditional clinical trial alone. An existing RWD source, however, may have some inherent bias that could limit its value for drawing causal inferences between medical device exposures and outcomes. Therefore, to mitigate potential bias, careful study design is needed, and a study protocol and analysis plan should be created prior to accessing, retrieving, and analyzing RWD, regardless of whether the RWD are already collected (retrospective) or if they are to be collected in the future (prospective design). Protocols and analysis plans for RWD should address the same elements that a traditional clinical trial protocol and statistical analysis plan would cover. FDA recommends use of the pre-submission process when considering the development of a study using RWD in a regulatory submission. When considering a prospective study design, one should consider whether RWD collection instruments and analysis infrastructure are sufficient to serve as the mechanism for conducting the study, and if they are not, whether it is possible to modify them for such a purpose. Ultimately, if the sources of bias can be mitigated, RWD collected using a prospective study design may be used to generate or contribute to the totality of evidence needed to assess medical device performance. Because of its nature, the quality of RWE can vary greatly across different data types and sources. Likewise, there are many types of FDA regulatory decisions spanning the Total Product Life Cycle (TPLC) that necessitate different levels of evidence. This guidance does not change 12

Requests for Feedback on Medical Device Submissions: The Pre-Submission Program and Meetings with Food and Drug Administration Staff (http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM311176. pdf) 13 Sherman RE, Steven A, Dal Pan GJ, et.al.; Real-World Evidence – What Is It and What Can it Tell Us?; NEJM, Dec 2016; 375; 23

8

Contains Nonbinding Recommendations

FDA’s evidentiary standards for regulatory decision-making, and in each context we will evaluate whether the available RWE is of sufficient quality to address the specific regulatory decision being considered. FDA believes that the increased use of electronic data systems in the healthcare setting has the potential to generate substantial amounts of RWD. Because these systems can vary greatly in terms of quality, not all RWD can by itself generate sufficient evidence to support an FDA regulatory decision. Nevertheless, these RWD may still provide a valuable contribution to the totality of evidence considered for the decision. Furthermore, in order to use RWD, the device must be sufficiently identified with the level of detail necessary to address the regulatory question. For example, to evaluate a particular device, the Unique Device Identifier (UDI) or serial/model number may be necessary to identify the device within a RWD source that contains data on many similar devices. When RWE is intended to be used for purposes of evaluating a regulatory issue, it is important that the RWD not only follow the criteria described in section V, but are also presented to FDA according to recognized data standards (i.e. in standardized file formats and data structures, utilizing standardized variables and definitions, etc.) when applicable. This includes discussions of the methodology used to analyze RWD and assess clinically relevant differences as well as statistical significance.

IV. Regulatory Context in Which RWE May be Used A. General considerations for the use of RWE FDA will consider the use of RWE to support regulatory decision-making for medical devices when it concludes that the RWD used to generate the RWE are of sufficient quality to inform or support a particular regulatory decision. The threshold for sufficient quality will depend on the specific regulatory use of the evidence. For example, a specific registry might be leveraged for postmarket surveillance, but not be adequate to support a premarket determination of reasonable assurance of safety and effectiveness or substantial equivalence. The collection or aggregation of RWD sources outside of medical records is usually performed for specific pre-determined non-regulatory purposes. For example, medical administrative claims data are typically kept for purposes of billing/payment for medical care. Disease-specific RWD sources sponsored by patient advocacy organizations may be useful for tracking progression or outcomes of specific rare or poorly understood diseases. Treatment-specific RWD sources coordinated by one or more professional societies may have several purposes, including assessment and tracking of overall outcomes, providing data for quality assessment (QA), informing performance improvement (PI) initiatives, or providing risk prediction and benchmarking data for specific therapies. Therefore, the use of RWD for specific regulatory decisions necessitates an understanding of the strengths and limitations of the RWD, and how these qualities impact the relevance and reliability factors described below. RWD may potentially be used as some or all of the evidence necessary for understanding medical device performance at different points in the TPLC. Some purposes for which RWD may potentially be used include the following: 9

Contains Nonbinding Recommendations

·

for generating hypotheses to be tested in a prospective clinical study;

·

as a historical control, a prior in a Bayesian trial14, or as one source of data in a hierarchical model or a hybrid data synthesis;

·

as a concurrent control group or as a mechanism for collecting data related to a clinical study to support device approval or clearance in a setting where a registry or some other systematic data collection mechanism exists;

·

as evidence to identify, demonstrate, or support the clinical validity of a biomarker;

·

as evidence to support approval or granting of an Humanitarian Device Exemption, Premarket Approval Application (PMA), or De Novo request;

·

as support for a petition for reclassification of a medical device under section 513(e) or (f)(3) of the FD&C Act;

·

as evidence for expanding the labeling of a device to include additional indications for use or to update the labeling to include new information on safety and effectiveness15,16

·

for public health surveillance efforts. Through ongoing surveillance, signals are at times identified that suggest there may be a safety issue with a medical device. RWE may be used to refine these signals for purposes of informing appropriate corrective actions and communication;17,18

·

to conduct post-approval studies that are imposed as a condition of device approval or to potentially preclude the need for postmarket surveillance studies ordered under section 522 of the FD&C Act;

14

Guidance for Industry and FDA Staff: Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials, (https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm071121. pdf) 15 What We Mean When We Talk About EvGen Part II: Building Out a National System for Evidence Generation, (http://blogs.fda.gov/fdavoice/index.php/2016/04/what-we-mean-when-we-talk-about-evgen-part-i-laying-thefoundation-for-a-national-system-for-evidence-generation/) 16 Guidance for Industry: General/Specific Intended Use (http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm073945.p df). 17 Public Notification of Emerging Postmarket Medical Device Signals (“Emerging Signals”) Guidance for Industry and Food and Drug Administration Staff (http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm479248.pdf). 18 Strengthening Patient Care: Building an Effective National Medical Device Surveillance System (http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe ports/UCM435112.pdf).

10

Contains Nonbinding Recommendations

·

in certain circumstances, for use in generating summary reports of Medical Device Reports (MDRs); and

·

to provide postmarket data in lieu of some premarket data.

B. Application of Investigational Device Exemption (IDE) Requirements in 21 CFR 812 to the Collection of RWD An approved IDE permits a device to be shipped lawfully for the purpose of conducting investigations of the device without complying with certain other requirements of the FD&C Act that would apply to devices in commercial distribution. The purpose of this, per 21 CFR 812.1, “is to encourage, to the extent consistent with the protection of public health and safety and with ethical standards, the discovery and development of useful devices intended for human use, and to that end to maintain optimum freedom for scientific investigators in their pursuit of this purpose.” As explained in Part 812, the IDE regulations apply to all clinical investigations of devices to determine safety and effectiveness, with certain limited exceptions. In many cases, an approved IDE is required before initiating a clinical investigation. An investigation is defined as “a clinical investigation or research involving one or more subjects to determine the safety or effectiveness of a device.19” Whether the collection of RWD for a legally-marketed device requires an IDE depends on the particular facts of the situation. Specifically, if the device is being used in the normal course of medical practice, an IDE would likely not be required. Because FDA does not regulate the practice of medicine, this could include administration of a legally marketed device for uncleared or unapproved uses as long as the device is being administered under the authority of a healthcare practitioner within a legitimate practitioner-patient relationship. 20 However, if data are being gathered to determine the safety and effectiveness of the device, and the process for gathering the data would influence treatment decisions, such administration would likely not be within the normal course of medical practice, and an IDE may be required. For example, a registry designed to determine the safety and effectiveness of an approved device for a new intended use would likely be subject to IDE requirements if physicians are instructed to treat specific patients or otherwise administer the device in a prescribed way for purposes of data generation, or when certain follow-up activities are performed for the purpose of research. Because the gathering of RWD is different from traditional investigations, we recommend that you contact FDA if you have questions about whether an IDE is required. As stated above, FDA does not regulate the practice of medicine, and recognizes that RWD may be collected during the routine care of patients that provides information related to the actual use of an approved or cleared device. Such observations may include RWD from a use of a medical device that was not within the cleared or approved indications for use. If such RWD collection 19

See 21 CFR 812.3(h) FDA will not limit or interfere with the authority of a health care practitioner to prescribe or administer any legally marketed device to a patient for any condition or disease within a legitimate health care practitioner-patient relationship. Section 1006 of the FD&C Act, 21 USC 396. 20

11

Contains Nonbinding Recommendations

does not impact how the device is administered, and the administration is within the normal course of medical care, an IDE would likely not be required. For example, retrospective analyses of existing RWD involving the use of a device that was not within the cleared or approved indications for use would generally not be subject to IDE regulations. In such cases, treatment decisions were made in the best interest of patients according to their clinician’s medical judgment at that time. However, if the plan to conduct such analyses impacts patient care, then the study may be subject to IDE requirements. Should a sponsor or Institutional Review Board (IRB) be unclear regarding the applicability of the IDE regulations to a particular RWD collection activity or use, the sponsor or IRB should contact FDA. If an IDE is determined to be required, FDA will work with the IDE sponsor to develop the least burdensome approach to facilitate the efficient generation of high-quality RWE. Note that regardless of the applicability of 21 CFR 812, FDA regulations at 21 CFR 56 (IRB review), 21 CFR 50 (Informed Consent) and 21 CFR 54 (Financial Disclosure) may apply to RWE generation activities, as may other federal, state, and local laws regarding human subject protections.

V.

Characteristics of RWD

FDA does not endorse one type of RWD over another. Sponsors should select appropriate RWD sources based on their suitability to address specific regulatory questions. Whether the RWD resides within paper or electronic medical records, is collected by administrative databases, is abstracted, aggregated and stored in disease- or treatment-specific databases (i.e., registries), or collected and aggregated through other means, accuracy when compared to verifiable source documentation is essential. Verifiable source documentation for RWD elements includes, but is not limited to: paper or electronic inpatient and outpatient medical records and case histories, diagnostic laboratory and imaging data, patient preference information, patient-reported outcome measures, UDI and other device identifiers, and performance data that exist within the device such as self-diagnostics, error codes, and patient diagnoses/treatments delivered. Requirements and needs for individual source data verification will vary with specific regulatory questions, and will contribute to the overall understanding of data quality for that source. In order to determine the suitability of RWD for regulatory decision-making, FDA will assess the relevance and reliability of the source and its specific elements. This assessment will be used to determine whether the RWD source(s) and the proposed analysis can generate evidence that is sufficiently robust to be used for a given regulatory purpose. Whether evidence is sufficiently relevant and reliable for use will, in part, depend on the level of quality necessary to make a particular regulatory decision. FDA will evaluate the same factors to assess RWD across all data sources and regulatory decisions. In cases where RWE is derived from multiple RWD sources, each RWD source will be evaluated individually and together in the aggregate to determine the relevance and reliability of the RWD. When developing a new RWD source, consultation with FDA and other stakeholders is recommended to ensure that relevance and reliability are addressed in the initial design.

12

Contains Nonbinding Recommendations

A. Relevance The relevance of RWD, RWD sources, and resultant analysis is assessed by evaluating several factors as outlined below. These factors can help determine if the data adequately address the applicable regulatory question or requirement, in part or in whole. Questions about the applicability of RWD to a specific case should be discussed with FDA through the presubmission process.21 Relevance of RWD for regulatory decision-making can be assessed prior to a regulatory submission, such as via the pre-submission process, or during the regulatory review process. The overall assessment of relevance should determine whether the existing RWD source is adequate for evaluating the performance of a device in the identified regulatory context (as a sole source or partial source of evidence). Since RWD sources are usually developed for non-regulatory purposes (e.g., to document care in the case of EHRs or to submit insurance claims for reimbursement in administrative and claims data), FDA will assess whether the individual data elements contained within an existing RWD source are sufficient to be used for a regulatory purpose. The data should be accurate, as complete as possible, and have an appropriate scope to address the question at hand (i.e., data adequacy). The need for review or adjudication of specific outcomes of interest (e.g. stroke or major bleeding) at the patient level may also be assessed. For analysis and interpretation of RWD, it is important to have a pre-defined common set of data elements, a common definitional framework (i.e., data dictionary), and pre-specified time intervals for data element collection and outcome analyses. In assessing the relevance of RWD, FDA will also consider, if warranted, the ability to supplement the available RWD through linkage with other data sources to provide additional or confirmatory data, e.g., with EHR and/or administrative claims data. Important relevance factors that FDA will assess to determine if the RWD are suitable for regulatory use include, but are not limited to, whether: ·

the RWD contain sufficient detail to capture the use of the device, exposures, and the outcomes of interest in the appropriate population (i.e. the data apply to the question at hand);

·

the data elements available for analysis are capable of addressing the specified question when valid and appropriate analytical methods are applied (i.e. the data are amenable to sound clinical and statistical analysis); and

·

the RWD and RWE they provide are interpretable using informed clinical/scientific judgment. Important considerations for the assessment of this factor include:

21

Requests for Feedback on Medical Device Submissions: The Pre-Submission Program and Meetings with Food and Drug Administration Staff: Guidance for Industry and Food and Drug Administration Staff (http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM311176. pdf)

13

Contains Nonbinding Recommendations

o whether the use of the device in a real-world population is representative as captured within the data source, and is generalizable to the relevant population being evaluated; o whether the RWD source is used regionally, nationally and/or internationally; o the overall percentage of patient exposure to the device that are captured in the RWD source; o the validation protocols and resultant data that are used to evaluate how well the RWD source reflects the patient population’s experience; o whether the RWD study design, study protocol, and/or analysis plan is appropriate to address the regulatory question and capable of being accomplished in a sufficiently timely manner; o whether the RWD contains elements to capture specific device identification information (e.g., unique device identifier); o whether the RWD adequately captures patient medical history and preexisting conditions, as well as follow-up information needed evaluate the question being addressed (e.g., whether administrative claims data have adequate continuity of coverage); o whether sufficient data elements are collected to adjust for confounding factors that may impact the exposure or outcomes of interest; o whether any linkages performed are scientifically appropriate and account for differences in coding and reporting across sources; o the RWD source reporting schedule, including time interval between database close and release, and length of reporting periods; o the prior documented (e.g., peer reviewed publications or practice guidelines) use of the RWD source for determining outcomes-based quality assessments, validated predictive risk modeling, signal detection, performance improvement, benchmarking, and other clinically-meaningful uses; o whether the data elements collected are sufficient for assessing outcomes (including adjudication, if necessary); and o whether supplemental data sources are available and sufficient to provide any missing information or evidence required for an informed decision.

14

Contains Nonbinding Recommendations

B. Reliability The reliability of RWD, RWD sources, and resultant analyses is assessed by evaluating several factors as outlined below. The primary factors FDA considers for assessing the reliability of RWD include how the data were collected (data accrual), and whether the people and processes in place during data collection and analysis provide adequate assurance that errors are minimized and that data quality and integrity are sufficient (data assurance). Additionally, the RWD analysis protocol should be prospectively defined as described in section III above. FDA will consider existing data accrual, data assurance, and analysis approaches in its assessment of the fitness of a given RWD source and its data.

(1)

Data accrual

To ensure the reliability of RWD, the RWD source should have an operational manual or other documentation that pre-specifies the data elements to be collected, data element definitions (i.e., data dictionary to provide a common definitional framework), methods for data aggregation and documentation (e.g., common case report form, abstraction from verifiable sources), and the relevant time windows for data element collection (i.e., common temporal framework). Some RWD sources such as EHRs or claims data may not fulfill all of these characteristics, but may still demonstrate sufficient reliability to support regulatory decision-making. Factors FDA will consider in making this determination include (but are not limited to): o the preparedness of individual sites for complete and accurate collection of RWD (e.g., whether there are defined processes, site training and support, and qualified personnel); o whether a common data capture form was used; o whether a common definitional framework (i.e., data dictionary) was used; o adherence to a common temporal framework for collection of key data points; o the timing of establishing the study plan, protocol, and/or analysis plan relative to collection or retrieval of the RWD; o the sources and technical methods used for data element capture (e.g., chart abstraction, point of care entry, EHR integration, UDI capture, data records from the device, and linkage to claims data); o whether patient selection and enrollment criteria minimize bias and ensure a representative real-world population (e.g., all-comer’s design, consecutive patient enrollment); o the timeliness of data entry, transmission, and availability; and

15

Contains Nonbinding Recommendations

o whether necessary and adequate patient protections were in place (e.g., methods to protect patient privacy, and need for informed consent as determined by the reviewing IRB and in compliance with FDA regulations).

(2)

Data assurance - Quality Control

Data quality control is essential for providing confidence in the reliability of RWD and RWE sources. RWD quality can generally be improved by following published recommendations concerning registries, such as those by the Agency for Health Care Quality, Patient-Centered Outcomes Research Institute,22 the National Medical Device Registry Task Force,23 and the Regulators Forum (IMDRF) Registry Working Group. 24 However, certain sources of RWD, such as some administrative and healthcare claims databases or EHRs, may not have established data quality control processes and may not be capable of fully implementing or following the above recommendations. When considering a source of RWD for regulatory purposes, it is important to consider any methods and systems used to help ensure sufficient data quality. Potential RWD sources should be evaluated in accordance with the data QA plan and procedures developed for the data source itself. Since evaluation of RWD sources may not always permit specific line item source verification, important factors for consideration include: o the quality of data element population (e.g., whether abstracted from a verifiable source to assess transcription errors or automatically populated through a data extraction algorithm); o adherence to source verification procedures and data collection and recording procedures for completeness and consistency; o completeness (i.e., minimized missing or out of range values) of data necessary for specified analyses, including adjustment for confounding factors; o data consistency across sites and over time; 25

22

AHRQ Registries for Evaluating Patient Outcomes: A User’s Guide 3rd Edition, Volume 1 Chapter 11, Data Collection and Quality Assurance and Volume 2 Chapter 25, Assessing Quality. (http://www.effectivehealthcare.ahrq.gov/search-for-guides-reviews-andreports/?pageaction=displayproduct&mp=1&productID=1897) 23 A Report from the Medical Device Registry Task Force & the Medical Devices Epidemiology Network. Recommendations for a National Medical Device Evaluation System. Strategically Coordinated Registry Networks to Bridge Clinical Care and Research, August 2015. Available at: http://www.fda.gov/downloads/aboutfda/centersoffices/officeofmedicalproductsandtobacco/cdrh/cdrhreports/ucm45 9368.pdf. Accessed August 6, 2016. 24 IMDRF Registry Essential Principles. Available at: http://www.imdrf.org/docs/imdrf/final/consultations/imdrf-cons-essential-principles-151124.pdf. Accessed August 3, 2016 25 PCORI Conduct of Registry Studies http://www.pcori.org/sites/default/files/Standards-in-the-Conduct-of-Registry-Studies-for-Patient-CenteredOutcomes-Research1.pdf

16

Contains Nonbinding Recommendations

o evaluation of on-going training programs for data collection and use of data dictionaries at participating sites; o evaluation of site and data monitoring practices; and o the use of data quality audit programs. The use of routine medical care data for additional analyses often relies on data cleaning and cross-referencing. These techniques can confirm the data’s internal consistency and identify missing values, but cannot fully determine data accuracy and authenticity. In traditional clinical research, an audit that compares source documents to entered data is often an essential step to verify the accuracy and completeness of the data. Study monitoring through various methods, such as remote monitoring, also plays an important role.26 These types of data verification activities are equally important for RWD that is intended to be used for regulatory purposes. Regardless of the original purpose for collecting the RWD, procedures for data collection and quality assurance should be put into place during the data source design and development stages (when applicable) to optimize the reliability, quality and usefulness of the data. The data collection procedures used for data platforms such as registries should be clearly defined and described in a detailed data management standard operating procedures (SOP) manual. When implementing a retrospective study using RWD, standardizing procedures to ensure the use of uniform and systematic methods for collecting and cleaning data are vital to ensuring data quality. For example, a “quality system approach” with a risk-based quality assurance and monitoring plan is a practical strategy for data platforms such as registries that can be challenging to audit. The RWD source organization or entity considering using the RWD for regulatory purposes should retain records regarding the assessment of adherence to the RWD source’s established data quality assurance and quality control policies and procedures.

VI. Examples Where RWE is Used The following examples are generalized from actual uses of RWE in support of regulatory decision making. These examples do not represent a comprehensive list of all potential uses or sources of RWD, but do describe some situations where RWE might be used to support regulatory decision-making.

A. Expanded Indications for Use The National Cardiovascular Data Registry (NCDR) was created in 1997 by the American College of Cardiology (ACC) as “an exploration into strategies for improving cardiovascular care through the use and application of clinical data.” These registries are designed to help participants measure, benchmark, and improve cardiovascular care. In particular, the Registry for diagnostic cardiac CATHeterization and Percutaneous Coronary Intervention (Cath-PCI 26

Oversight of Clinical Investigations: A Risk-Based Approach to Monitoring (PCORI Conduct of Registry Studies (http://www.pcori.org/sites/default/files/Standards-in-the-Conduct-of-Registry-Studies-for-Patient-CenteredOutcomes-Research1.pdf)

17

Contains Nonbinding Recommendations

Registry) “assesses the characteristics, treatments and outcomes of cardiac disease patients who receive diagnostic catheterization and/or percutaneous coronary intervention (PCI) procedures, measuring adherence to ACC/AHA clinical practice guideline recommendations, procedure performance standards and appropriate use criteria for coronary revascularization.” As a registry collecting RWD on patients treated in routine clinical practice with an approved or cleared device, an IDE is not required even though a substantial volume of RWD is generated for uses outside of the device’s cleared or approved indications for use. RWD from this registry could be used to identify opportunities to expand the labeling of devices captured in the registry. Should a manufacturer wish to expand indications, this type of RWD might provide sufficient evidentiary support, depending on the specific devices, indications, and analyses. Another example is a Class III device that, due to technological advances in its design, has seen an expansion of clinically acceptable use that is outside of the device’s approved indications for use. There is little published data to support a reasonable assurance of safety and effectiveness of the new use. To address the lack of data to support new indications for use, relevant medical societies have established a national registry to collect safety and effectiveness information for all patients implanted with this specific device at participating institutions. This registry also uses a validated matching algorithm to link registry records with administrative healthcare claims as a supplemental dataset for capturing long-term outcomes. A study using the registry data collection and analysis infrastructure was initiated with an approved IDE application since the study focused on a use of this device that was not within the approved indications for use and imposed changes to the collection regiment of specific follow-up data that might not otherwise have occurred as part of routine medical care. FDA is hopeful that the RWD may be of sufficient quality to address critical safety questions and to potentially support labeling changes or other regulatory decisions for this device.

B. Postmarket Surveillance Studies (Section 522) FDA has issued a series of postmarket surveillance orders, related to investigating patient safety issues for a type of class II device, under the authority of section 522 of the FD&C Act. These 522 orders covered multiple devices from different manufacturers that are similar in intended use, design, and other characteristics, such that the surveillance questions were identical. To comply with the orders, many manufacturers decided to collaborate with a clinical professional society in this field and with FDA to develop a patient registry that would collect the needed data to address the public health questions. An IDE was not required because the 522 order was focused on device uses consistent with the labeling. The resultant registry was designed to collect RWD on all patients with the relevant condition, including those treated with the devices of interest, with other devices, and through medical management. Manufacturers are able to share the comparator group consisting of treatments that do not use the devices of interest. In addition, because the registry was designed at the outset to produce regulatory-quality RWD in addition to meeting research and quality improvement purposes, appropriate data quality checks and electronic controls were a part of the initial design and implementation. Since this registry development process took a substantial amount of time, FDA was willing to grant extensions to manufacturers to respond to the 522 orders as long as progress was being made. The registry was also designed to allow for its use (with additional protocols and other traditional study

18

Contains Nonbinding Recommendations

operational elements) in conducting premarket studies that could support future premarket submissions.

C. Post-Approval Device Surveillance as Condition of Approval Permanent implants are typically designed to serve patients for a time period that is much longer than what can reasonably be captured in a premarket clinical trial. For example, a trial that follows patients for two years after implantation would not produce data for the designed life span of 7 to10 years for that implanted device. Traditionally, FDA would require extended follow-up of the premarket patient cohort and an additional new-enrollment study designed to capture hundreds to thousands of patients with follow-up for the life of the implanted device. Some clinical professional societies have developed registries that collect RWD on patients receiving these devices. FDA has worked with manufacturers and professional societies to evaluate the registries and has found that they can be reliable for certain health outcomes of interest. Should sufficient RWD exist that are capable of addressing the questions for which a Post-Approval Study (PAS) requirement may be issued, FDA may instead issue a condition of approval that requires collection/reporting of these RWD on the device. For example, a new breakthrough Class III medical device was approved based on prospective, randomized, and controlled clinical trial data. Early in the PMA review process, the manufacturer began to consider postmarket commitments, and began discussions with FDA and other stakeholders. A registry was launched that generated RWD that could meet FDA’s data requirements, as well as others. Because the new registry was constructed early enough to collect information about all patients receiving this device upon approval, FDA could provide an earlier device approval conditioned on further robust RWD collection and reporting in the postmarket setting. This registry has since been used to a) collect surveillance data on subsequent devices with similar designs and indications, b) collect and retrospectively analyze RWD on all uses of the devices to support new expanded indications for use, and c) support embedded prospective clinical investigations under IDE for new devices and new generations of approved devices. No IDE is necessary for the general data collection activities of the registry, as it collects RWD on all uses of otherwise approved medical devices and it does not influence the treatment decisions and/or follow-up care that patients receive. The retrospective analysis of RWD for uses that are outside the approved indications for use did not require an IDE because treatment decisions were not influenced by the expectation of conducting the future analysis, but was still reviewed by an IRB for human subject protection issues. However, prospective enrollment in a clinical investigation using the registry infrastructure to study a new, unapproved significant risk device would require an IDE. Similarly, a prospective, non-observational clinical investigation of a new indication for an approved device may require an IDE, depending on the risk determination.

D. Control Group A manufacturer approached FDA during the development of a next generation medical device that had substantial technological changes from previous iterations of that specific device and other similar devices from other manufacturers. FDA determined that clinical evidence was needed to support an approval decision for this device modification. A registry exists that captures RWD on all uses of medical devices with a similar intended use. The manufacturer 19

Contains Nonbinding Recommendations

designed a clinical study that compared the use of the new device to a non-randomized concurrent control group derived from the registry. The existing registry was evaluated by FDA and the manufacturer according to the factors cited in this guidance and was found to provide sufficiently relevant and reliable RWD on the control population, such that the manufacturer did not have to collect additional data from these patients or influence the course of their clinical care in any way. The patients who received the investigational device were enrolled under an approved IDE. However, the patients who contributed to the control group were not considered part of the IDE because they were enrolled in a national registry that collected RWD on FDA approved devices, and their treatment was not influenced by the existence of the study.

E. Supplementary Data Periodically, FDA identifies an issue related to the safety of a marketed medical device that was not detected in premarket trials. In the case where RWD has been systematically collected, FDA has used these RWD, in combination with case reports, publications, adverse event reports, engineering and nonclinical test data, and other sources of information available to FDA to gain a better understanding of the severity of the issue, precipitating factors, affected populations, and alternative therapies. The addition of RWD has proven extremely valuable as a means to develop a course of action that best protects public health in these instances. For example, a class III device was under review for a new indication. The manufacturer provided data from a prospective clinical trial with limited follow-up information and inadequate data from the control group, which made interpretation of results difficult. However, a preexisting registry was already collecting and reporting RWD on the control therapies. The registry data were used to supplement and help interpret the clinical trial data, allowing FDA to come to an appropriate regulatory decision without requiring additional clinical trial data. Without the RWE, additional study subjects could have been exposed to a device with a questionable risk-benefit balance. Coming to a final decision more quickly in this case protected subjects’ health while also spurring development of new designs for the medical device.

F. Objective Performance Criteria and Performance Goals An Objective Performance Criterion (OPC) refers to a numerical target value derived from historical data from clinical studies and/or registries and may be used in a dichotomous (pass/fail) manner for the review and comparison of safety or effectiveness endpoints.27 An OPC is usually developed when device technology has sufficiently matured and can be based on publicly available information or on information pooled from all available studies on a particular kind of device. Similar to OPC, a performance goal (PG) refers to a numerical value that is considered sufficient for use in the evaluation of an investigational device regarding a safety and/or effectiveness endpoint. But, generally, the device technology is not as well-developed or 27

Design Considerations for Pivotal Clinical Investigations for Medical Devices - Guidance for Industry, Clinical Investigators, Institutional Review Boards and Food and Drug Administration Staff for more information on OPCs and PGs. http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM373766. pdf).

20

Contains Nonbinding Recommendations

mature for use of a PG as for an OPC, and the data used to generate a PG are not considered as robust as those used to develop an OPC. A PG might be considered for patient populations that are difficult to study or if there is no clinical equipoise for any control. From a sufficiently relevant and reliable RWD source, a PG can be constructed using appropriate statistical methods, such as a subject-level meta-analysis. As technology evolves over time, an OPC or PG could be updated using RWD.

VII. Glossary The following definitions are supplied to provide the reader with an understanding of the specific terms used in this guidance. These definitions should not be construed to be new interpretations or clarification of the use of similar words or phrases in the FD&C Act, related code or regulation, other federal, state, or local laws, or other guidance documents. ·

Bias—“Bias is any systematic error in the design, conduct, analysis, interpretation, publication, or review of a study and its data that results in a mistaken estimate of a treatment’s effect on disease. This systematic error results from flaws in the method of selecting study participants, in the procedures for gathering data, and in the decision of how and whether to publish the results. These flaws can lead to observed study results that tend to be different from the “true” results. Bias can be minimized by ensuring that the study design is appropriate for addressing the study hypotheses and establishing and carefully monitoring procedures of data collection that are valid and reliable.”28

·

Confounding—“A situation in which a non-causal association between a given exposure or treatment and an outcome is observed as a result of the influence of a third variable designated as a confounder. The confounding variable needs to be related to both the treatment and the outcome under study. Confounding is distinct from bias because this association, while not causal, is real.”29

·

Electronic Health Record (EHR)—“An electronic record of health-related information on an individual that conforms to nationally recognized/utilized interoperability standards and that can be created, managed, and consulted by authorized clinicians and staff across more than one health care organization.”30

·

Electronic Medical Record (EMR)—“An electronic record of health-related information on an individual that can be created, gathered, managed, and consulted by authorized clinicians and staff within one health care organization.”31

28

JM Last. A dictionary of Epidemiology (3rd edition). New York: Oxford University Press, 1995) (M Szklo & FJ Nieto. Epidemiology: Beyond the basics. Gaithersburg, MD: Aspen Publishers, Inc., 2000 29 L Gordis. Epidemiology. Philadelphia: WB Saunders, Co., 1996 30 The National Alliance for Health Information Technology Report to the Office of the National Coordinator for Health Information Technology on Defining Key Health Information Technology Terms April 28, 2008. (http://www.himss.org/national-alliance-health-information-technology-report-office-national-coordinator-health). 31 The National Alliance for Health Information Technology Report to the Office of the National Coordinator for Health Information Technology on Defining Key Health Information Technology Terms April 28, 2008. (http://www.himss.org/national-alliance-health-information-technology-report-office-national-coordinator-health)

21

Contains Nonbinding Recommendations

·

Interventional Study—“A clinical study in which participants are assigned to receive one or more interventions (or no intervention) so that researchers can evaluate the effects of the interventions on biomedical or health-related outcomes. The assignments are determined by the study protocol. Participants may receive diagnostic, therapeutic, or other types of interventions.”32

·

Large Simple Trial – “A large simple trial (LST) is a type of randomized clinical trial (RCT) ideally suited to answer many important clinical questions and because it typically answers only one or two questions in a broader patient population, is generally more efficient and less expensive than other large RCTs. LSTs have a large sample size and statistical power to detect clinically relevant treatment effects, providing unambiguous results and minimizing the effects of random errors.” 33

·

Medical Administrative Claims Data—“Claims data arise from a person’s use of the health care system [and reimbursement of health care providers for that care].”34

·

Medical Device Registry – “Organized system that continuously and consistently collects relevant data in conjunction with routine clinical care, evaluates meaningful outcomes and comprehensively covers the population defined by exposure to particular device(s) at a reasonably generalizable scale (e.g. international, national, regional, and health system) with a primary aim to improve the quality of patient care.” 35

·

Medically Recognized Standards of Care—“Medically recognized standards of care are treatments or procedures that have been accepted by medical experts as appropriate treatments or procedures for a given type of disease or condition and are commonly used by health care professionals. The medical recognition of standards of care is typically represented by publication in a peer-reviewed journal or some form of recognition by a professional medical society. The evidentiary bases for these recognized standards of care vary.”36

·

Observational Study— “A study that does not involve any intervention (experimental or otherwise) on the part of the investigator; e.g., a population study in which changes in health status are studied in relation to changes in other characteristics. Most analytical epidemiological designs (notably, case-control and cohort studies) are properly called observational because investigators observe without intervening other than to record, classify, count, and analyze results.” 37

32

https://www.clinicaltrials.gov/ct2/about-studies/glossary Project: Large Simple Trials; Clinical Trials Transformation Initiative (https://www.ctticlinicaltrials.org/projects/large-simple-trials) 34 Strom, Brian. Pharmacoepidemiology. Chichester, England: John Wiley and Sons, 2005. 35 IMDRF Registry Essential Principles. Available at: http://www.imdrf.org/docs/imdrf/final/consultations/imdrf-cons-essential-principles-151124.pdf. Accessed August 3, 2016 36 Ethical Review and Oversight Issues in Research Involving Standard of Care Interventions: Workshop in Brief 2015, Institute of Medicine (https://www.nap.edu/read/21668/chapter/1) 37 A Dictionary of Epidemiology (6 ed.) Oxford Reference http://irea.ir/files/site1/pages/dictionary.pdf 33

22

Contains Nonbinding Recommendations

·

Postmarket surveillance— “Postmarket surveillance is the active, systematic, scientifically valid collection, analysis and interpretation of data or other information about a marketed device.”38

·

Pragmatic clinical trial (PCT) - A clinical trial “designed for the primary purpose of informing decision-makers regarding the comparative balance of benefits, burdens and risks of a biomedical or behavioral health intervention at the individual or population level.”39

·

Prospective Study—“A prospective study design (also called a concurrent cohort study) defines the original population of interest at the start of the study and collects exposure/treatment and outcome data from that time point forward. The start of the study is defined as the time the research protocol for the specific study question was initiated.”40

·

Real-World Data (RWD) are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.

·

Real-World Evidence (RWE) is the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.

·

Registry—“An organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves one or more predetermined scientific, clinical or policy purposes.”41

·

Retrospective Study—“A retrospective study design (also called a retrospective cohort study, a historical cohort, or non-concurrent prospective study) defines the population and determines the exposure/treatment from historical data (i.e., data generated prior to the initiation of the study). The variables and outcomes of interest are determined at the time the study is initiated. Some studies are a combination of concurrent and retrospective cohort designs where the exposure/treatment is ascertained from existing objective records (e.g., medical records, claims data), and follow up and measurement of the outcome continues into the future.”42

38

21 CFR 822.3 Adapted from Califf R and Sugarman J; Exploring the Ethical and Regulatory Issues in Pragmatic Clinical Trials; Clin Trials, 2015 Oct;12(5):436-41. doi: 10.1177/1740774515598334. Epub 2015 Sep 15 40 JM Last. A dictionary of Epidemiology (3rd edition). New York: Oxford University Press, 1995) (M Szklo & FJ Nieto. Epidemiology: Beyond the basics. Gaithersburg, MD: Aspen Publishers, Inc., 2000 39

41

Registries for Evaluating Patient Outcomes: A User's Guide (https://www.effectivehealthcare.ahrq.gov/ehc/products/420/1897/registries-guide-3rd-edition-vol-1-140430.pdf) 42 JM Last. A dictionary of Epidemiology (3rd edition). New York: Oxford University Press, 1995) (M Szklo & FJ Nieto. Epidemiology: Beyond the basics. Gaithersburg, MD: Aspen Publishers, Inc., 2000

23

Contains Nonbinding Recommendations

·

Surveillance—“Surveillance is a continuous and systematic process of collection, analysis, interpretation, and dissemination of descriptive information for monitoring health problems.”43

·

Traditional clinical trial—Traditional clinical trials are typically conducted in specialized research settings with specific populations. They often utilize measures designed to control variability and ensure data quality, such as detailed eligibility criteria, detailed case report forms that exist apart from ordinary medical records, and intensive monitoring and auditing designed to ensure precise adherence to study procedures and rigorous precision in data collection. They typically also include substantial efforts to ensure compliance with treatments and to avoid concomitant treatments that might influence the randomized treatment effect.

43

JW Buehler. Surveillance (Ch. 22) pages 435-458 in KJ Rothman & S Greenland (editors) Modern Epidemiology 2nd edition. Philadelphia: Lippincott-Raven Publishers, 1998

24