This white paper offers advice from two industry experts, Jill Northup from Medical College of Wisconsin, and Philip Cotter from ResearchDx, in the area of setting up a clinical NGS laboratory
Please provide the following for access to content in our resource library
The face of medicine is changing, driven in large part by two factors. Greater understanding of the genetic basis of disease garnered in the decade since the completion of the human genome project has provided the potential for targeted, personalized therapies. Simultaneously, the costs of sequencing a whole human genome have dropped faster than Moore’s Law. As a result, clinicians now have the ability to, for example, sequence a patient’s tumor and prescribe treatments based on the specific mutations found.
The drive toward delivering personalized medicine has led many labs to consider conducting clinical tests employing next-generation sequencing (NGS). Labs capable of offering clinical NGS services generally fall into one of two categories: research labs already performing NGS that want to begin offering clinical services, and clinical labs providing other clinical diagnostics that want to add NGS tests to their arsenal. In either case, the decision to offer clinical NGS services requires labs to consider how they will conduct this work under the auspices of federal (and in some cases, state) clinical laboratory improvement amendments (CLIA) or College of American Pathologists (CAP) requirements.
“Recently CAP has added some checklist questions relating to NGS, but in general, the requirements for offering a NGS test are essentially no different from a blood chemistry test,” explained Phillip D. Cotter, principal at ResearchDx, a contract diagnostics organization that provides consulting services to labs seeking CLIA certification alongside standard clinical lab assay development. Yet only minimal guidance, such as the checklist questions, currently exists to help labs decipher the requirements and set up processes appropriately to conduct clinical NGS. Additionally, NGS presents unique challenges for labs —from managing the unprecedented volume of data produced by the techniques, to determining how to appropriately validate processes on the NGS scale.
“The fact is that everything that is done—from the time you receive patient samples to reporting and storing results—has to be tracked and controlled,” said Jill Northup, administrator at the Human and Molecular Genetics Center at the Medical College of Wisconsin.
In this paper, Cotter and Northup explain the general principles of CLIA certification and their specific application to clinical NGS, from the perspective of both types of labs that typically seek CLIA certification for NGS tests. Labs can set themselves up for success by:
Beginning in the 1970s, several ad hoc standards were implemented to raise the quality of tests conducted on humans. These standards were codified in 1988 into CLIA, which are managed by the Centers for Medicare & Medicaid Services (CMS) of the United States. Although all U.S. clinical laboratories must be properly certified to receive Medicare or Medicaid payments, CLIA has no direct Medicare or Medicaid program responsibilities. Currently, CLIA covers approximately 225,000 laboratory entities.1
The purpose of CLIA is to require labs to implement a quality management system, which means defining a host of elements associated with laboratory testing such as:
Labs can secure and maintain CLIA certification by two routes. In ‘CLIA by compliance’, a lab submits its initial applications and, if it is accepted, can legally start testing. Two or three months after beginning to offer tests, the lab is inspected by a local CLIA office and, provided it passes inspection, is granted CLIA by compliance that will be validated through inspection every two years thereafter. The other mechanism is ‘CLIA by accreditation’, in which one of seven organizations, identified by the CMS to inspect on CLIA’s behalf, evaluates a lab processes according to a predefined checklist. The most well-known of these organizations is CAP, which is why many organizations refer to CAP-CLIA when referring to regulatory requirements. A lab that passes this inspection receives a certificate of accreditation, which is usually good for two years.
In addition to the federal license, some states require separate licenses that may, in some cases, be more stringent than federal guidelines. Labs in these states must receive certification both from the federal CLIA agency and the state. Additionally, some states also specify that to accept samples from their state, labs in states must secure the relevant state licensure from the receiving state. So CLIA can, in some cases, require three different levels of accreditation: federal, in state, and requirements mandated by states in which a lab does business. While this may sound daunting, Cotter noted that “whatever permutation you go through, the basic quality management requirements are the same—they are the quality procedures you must have in place to operate a safe and effective clinical lab.”
The basic elements for running a test under CLIA are the same, regardless of the type of test performed. NGS does not change the requirements—but it does make those requirements harder to achieve. That’s because the scale of NGS changes how labs go about validating tests.
The purpose of validation, according to Cotter, is not to comment on the utility of an assay. It’s to verify that the assay accurately and routinely delivers the same results. In traditional assay validation, labs take a minimum of 20-25 samples and perform an accuracy study, in which they run the samples through the SOPs and check at the end, whether they get the correct answers each time. Labs typically define these SOPs by independently validating each component; however, such an approach fails to scale for an NGS test.
“I cannot validate every known mutation for every gene in a panel,” said Cotter. “Even if I could find them all, it would be too expensive to do.” For NGS tests, explained Cotter, labs must move away from validating specific datapoints to validating a process. Labs will specify that for a known set of samples, they will perform a given set of tasks with the aim of achieving a particular concordance rate. “You never aim for 100% concordance, because there are errors in public databases, discordance between platforms, changes in bioinformatics, and so much else to account for,” said Cotter. “You’re validating your process and essentially running a screening test, and it’s assumed you’ll go back and do other procedures to confirm your results.”
Northup, who recently helped the Human and Molecular Genetics Center at the Medical College of Wisconsin implement CLIA certified NGS tests in its existing clinical and research labs, explained that the process validation required for NGS tests forces a mindset shift among clinical lab personnel, who aren’t always prepared for the costs involved. “Clinical labs must face the reality that NGS is expensive technology,” said Northup. “Spending as much as $50,000 just to get a test online is unheard of for your average clinical diagnostic test, but it’s standard for NGS. And then there’s the cost of failures if, for example, a machine goes down and you lose reagents, which can also be significant.”
In addition to acclimating to the costs associated with validating NGS tests, clinical labs must also prepare themselves for the data deluge associated with NGS. “Molecular labs are extremely skilled at all types of wet-lab methodologies, so many assume that they just need to put some sequencers in place and they’ll be able to start offering NGS services,” said Northup. “But they aren’t ready for the data windfall—terabytes upon terabytes of data. Most lab staff honestly wouldn’t even know or care about the size of the dataset associated with a standard clinical test. NGS brings that to the forefront because there’s so much more to manage.”
“Our clinical lab has a storage capacity double that of our entire clinical affiliate hospital,” Northup pointed out.
Research labs familiar with running NGS tests must make similar ‘attitude’ adjustments to run these tests in a clinical setting. “Research cares about quality, but tracking that quality is not on the same level as a clinical lab,” said Northup. Regular proficiency tests, validating reagents, and adhering to a formal QA plan can seem onerous to researchers, as can the idea that clinical tests require an endpoint. “In research, you explore a problem until you discover something, and protocols are always being tweaked to improve them,” Northup explained. “In a clinical setting, though, you need to first define what you are doing and then preserve a trail of changes to document and validate that it’s still what you are doing. And if you don’t find something, that doesn’t mean you failed… it just means it’s outside the limits of the test.”
Northup said that she frequently injects a dose of reality to counter labs who think that doing NGS just means putting some sequencers in place. “I’ll say, ‘What will you do when you have data coming off the machines? Do you know what the data looks like, how to understand it, and what to do with an incidental finding?’ It really reaches every aspect of healthcare delivery and ultimately at the other end of that test is an ill patient.”
Northup pointed out that most labs are good with the technical execution of the wet-lab procedures; where they get hung up is on the front end, with receiving samples— and on the back end, with reporting and archiving results and data. This is especially true for research labs translating NGS into a clinical diagnostic. Northup advises research labs to start at the backend—from the final report—and work backwards to determine what SOPs need to be in place to generate that report. “Some of the things you need to think about won’t hit you right away: like preanalytical variables, whether you have adequate samples and what do you do if you don’t, ways to track and receive test requests, and how you’re going to accession and track samples,” she said. Similarly, labs need to consider how they plan to distribute reports, store reports and data, and archive patient files. “The more you can do upfront to set this up, the more it helps when you become operational,” explained Northup.
Planning, of course, often means anticipating what could go wrong. And NGS is so new that many labs, frankly, can’t envision the possibilities. Both Northup and Cotter emphasized that labs should seek help from someone experienced both with NGS procedures and with running these tests in a clinical setting.
“Until fairly recently, labs couldn’t pick up paperwork that defined what to put in place to do NGS in a clinical setting,” said Cotter. “And while lab directors may have experience doing clinical work, that doesn’t mean they know how to set up a clinical lab from the get-go.”
Northup admitted that many labs don’t consider asking for help. “I don’t know if it’s just genetics, but there’s this pride factor that says, ‘We can do it ourselves.’ Don’t fall into that trap. Look for resources, be it a consultant or simply someone else in your organization who has experience implementing and running tests—specifically NGS—in a clinical setting.” Northup is a case in point: She was chosen for her role at the Medical College of Wisconsin because she was able to understand and bridge the demands of clinical testing with those of research NGS.
Cotter and Northup acknowledged that there’s no regulatory requirement under CLIA to implement informatics, but several elements required under CLIA make informatics a logical choice. First, a clinical workflow inherently demands information management. “NGS consists of complicated processes, and anywhere that a human being interacts with a sample and that human being has to write a value down, there’s an opportunity for a mistake, with the worst being a sample mix up,” said Cotter. Informatics designed to support standardized, prescribed tasks and move data through the workflow minimizes and potentially eliminates humans as a source of error.
If sample management and tracking is a necessity for clinical NGS, the next question is which type of system to deploy. Northup mentioned that Excel spreadsheets can manage this type of data, but cautioned that this may not be the best choice for clinical work. “How easy is it to mess up a spreadsheet?” she said. “Sure, you can protect it, but where are the checks and balances necessary for doing this in a regulated, clinical setting? Basic spreadsheet software won’t cut it clinically.”
As an alternative, many organizations consider building their own informatics, but Northup noted that such endeavors are often shortsighted. “You may have a strong IT development group, but if you start adding up the time associated with performing system maintenance, updates, and upgrades, it may not make sense,” she said.
While a laboratory information management system (LIMS) may not be a requirement for a lab seeking to do clinical NGS, LIMS certainly sets a lab up for success. “Security and accuracy are so important in a clinical setting that if there are tools that can help you better achieve these goals, why wouldn’t you be better off having them?” said Northup. “Automating even tiny things can add up to a ton of time and cost savings. And you gain the ability to, at any point, check the current state of a sample, or to identify a sample and connect it to results without having to match up bits and pieces. You’re more CLIA compliant than you are aware that you are, if you have a rock solid LIMS to manage everything.”
Cotter concurred, pointing out that a LIMS not only supports work as it proceeds, it provides data that feeds into metrics and assists with audit tracking and quality improvement. “From a corrective action perspective, having a LIMS is brilliant,” Cotter said. “And honestly, if you’re already spending $600,000 on a HiSeq instrument, the additional cost for a LIMS is nothing.”
Labs running NGS in a clinical setting can position themselves for success by focusing on the purpose of CLIA: ensuring quality in clinical testing. Northup pointed to the unique position a lab now holds in the treatment cycle. “Ultimately the lab is an intermediary between the doctor and the patient,” Northup said. “There’s a lot of information to decipher, and it’s up to the clinical lab to make that information usable for the clinician.”
Cotter noted that as the most complicated clinical tests become routine, labs can view testing as a commodity, and samples as units processed in the facility. “We need to remember that someone—a real person—is attached to that sample, and that person is waiting for the result of your test to determine what to do next about their illness or condition,” Cotter advised. “That’s the whole reason we’re so focused on quality.”