Pioneering Precision: The Evolution and Future of Accuracy-Based Proficiency Testing

Learn more about the CAP's proficiency testing program.

Listen to other episodes in the 75 Years of Proficiency Testing Series:

• Enhancing Public Health Readiness: Laboratory Preparedness Exercise
• Infectious Disease Program: Continuously Evolving to Bolster Public Health

Lisa Tomcko:

Welcome to the latest edition of the College of American Pathologists' CAPcast. I'm Lisa Tomcko, content strategist with the CAP. This year marks the 75th anniversary of the CAP's proficiency testing, a.k.a. PT, programs. Accuracy-based programs are a key component of the CAP's PT offerings. That's because they use matrix matched materials and reference method comparisons to help laboratories ensure the accuracy of their results, allowing those results to be used reliably within national and international guidelines.

In this episode, CAP members Dr. David Sacks and Dr. Andy Hoofnagle discuss the history of the accuracy-based PT programs and look ahead to the future discussing potential directions and opportunities for growth and development. So welcome both of you and thank you for being here. Would you like to introduce yourselves? Dr. Hoofnagle, You can start us off if you'd like.

Dr. Andy Hoofnagle:

Sure. Thanks, Lisa. I'm Andy Hoofnagle. I'm a professor of laboratory medicine up at the University of Washington in Seattle, and I'm the current chair of the Accuracy-Based Programs Committee at the College of American Pathologists. I have a research lab on the side, which kind of specializes in using mass spectrometry and the development of novel blood tests and urine tests, et cetera. So I enjoy being a part of the scientific progress in improving accuracy of testing as well as the clinical and implementation of accurate blood tests.

Lisa Tomcko:

Great. Happy to have you here. And Dr. Sacks?

Dr. David Sacks:

I'm David Sacks. I'm the chief of clinical chemistry and a senior investigator at the NIH, and I'm a member of the CAP Chemistry Resource Committee.

Lisa Tomcko:

Excellent, welcome. And with that, I'll let you both take us through accuracy-based PT.

Dr. Andy Hoofnagle:

I have some questions for you, Dave. I know you have some questions for me. So just to get things going, there's a long history of proficiency testing here at the College of American Pathologists as Lisa mentioned, and there's new things on the block every once in a while. But what you thought about what you did back for the accuracy-based proficiency testing for hemoglobin A1C was really revolutionary. So can you kind of think about that out loud? How did you decide that accuracy-based proficiency testing was important? How did you come up with this idea when it first got started?

Dr. David Sacks:

Thank you for that very kind introduction, Andy and those kind comments. So the main reason was to deal with the matrix effects, and just briefly, for those who may not be familiar, matrix effects are change in quantification of an analyte that results when there are other substances in the sample. And that's usually when the lab measures something using an artificial material rather than blood. So this results in different methods from different manufacturers giving different results on the same sample due to matrix effects. Even though on a patient sample they might give the same results.

Dr. Andy Hoofnagle:

So you really focused on matrix matching there rather than accuracy-based. And certainly the witch's brew of stuff that we mix together and send out for proficiency testing is very different from matrix matching. You were also interested in accuracy in actually getting the right answer. Why was that important back in the day?

Dr. David Sacks:

The reason we started a new proficiency testing program for hemoglobin A1C was based entirely on clinical trials that were conducted. And I'll just briefly summarize the background. In 1993, the DCCT or Diabetes Control and Complications trial was published and it was a prospective study of 1,441 individuals, all of whom had type one diabetes, and they were randomly assigned to either intensive insulin therapy or standard glycemic control. And this was the very first study that showed that people who got intensive glycemic control had delayed onset and reduced risk of microvascular complications. And the importance of this study for hemoglobin A1C was that it was measured in all of these patients. And the study showed that a relatively small change in hemoglobin A1C, for example, from 8% to 7% resulted in a reduction of microvascular complications of 35%. So that is an enormous clinical impact. And this was followed four years later when the UK PDS study, which evaluated patients with type two diabetes, came up with essentially identical results due to the same method being used. The problem at the time was that there was a decisive lack of accuracy in hemoglobin A1C measurements in clinical laboratories. For example, in 1993, the College of American Pathologists hemoglobin A1C survey of 742 labs showed a range the same sample of 2.5% to 7.5%. So this indicated that an accurate hemoglobin A1C method was essential.

Dr. Andy Hoofnagle:

But do numbers really matter? I mean, if lower is better, why can't I just have my sample measured in the same lab over and over again? And why? Who cares if each of the platforms is a little bit different?

Dr. David Sacks:

Because all the clinical outcomes data are based on one method. So if your lab reports a hemoglobin A1C that has a value of 7%, reports it as 10%, then clearly the patient will be treated aggressively and will develop hypoglycemia. And conversely, if your lab has a low bias, the patients will be mistreated. It's really important to have a reference lab, a method. Can you just describe why reference methods are important for the field?

Dr. Andy Hoofnagle:

I think you hit the nail on the head that there has to be a number that we care about, right? I was being flippant when I said, well, lower is better, who cares? But that's actually true for assays where a number isn't important, then relative numbers are probably okay. But once we know that there is some sort of medical decision point above which we will treat patients differently than below, and I really liked your example of being treated aggressively with somebody who has a hemoglobin A1C of 7%, suddenly developing hypoglycemia, which as we all know is one of the more important horrible outcomes potentially as if you're driving or riding a bike or doing something else and actually become hypoglycemic. Bad things can happen. So I think there are real outcomes there and things to keep in mind with hemoglobin A1C, but it's true for others too.

Vitamin D is an example. Whether or not vitamin D and treatment really changes outcomes is obviously a huge matter of debate, but there was a number there from the Institute of Medicine, and that was one of the reasons that vitamin D and its accuracy-based materials were developed. But now there's others, right? Testosterone is a great example. We are now using a standardized harmonized reference range from the CDC as the line below which we should begin treating, begin treating patients with exogenous testosterone, so appropriate versus inappropriate therapy. And so for me, having the right number, in other words, getting it to be 2.35 instead of what I say flippantly, just one, I'm going to call it one, who cares if it's 2.35? We need to know the real number when there's a medical decision, a different therapeutic approach when the number is different. So having those reference methods that can tell us what the right answer is, having matrix matched materials so that there aren't those weird matrix effects that change from platform to platform can really help the field and help patients when they travel from hospital to hospital, insurance company to insurance company to get the right answer no matter where they are. When our therapeutic strategy centers around specific numeric cutoffs that have been developed by those amazing clinical studies like the ones that you described.

Dr. David Sacks:

The contribution of the clinical labs to obtaining accurate results for hemoglobin A1C had a very big impact on the field of diabetes and patient care. The way it was done was due to the combined efforts of the College of American Pathologists and the NGSP. So just briefly, the NGSP was formed almost immediately after the DCCT was published in 1993, and the goal was to standardize hemoglobin A1C methods so that the results in an individual laboratory that measures patient samples would be equivalent to those in the two clinical trials. Clinicians who treating their patients who are going for target of 7%, which is the usual target for treatment, would know that the 7% from the lab they sent their patient sample to would give the same result as if it was sent to the DCCT. Prior to 2007, the College of American Pathologists used peer group grading, and in 2007, this was changed to accuracy-based grading for hemoglobin A1C.

And the target values were assigned by the NGSP network, which has several labs, and each lab analyzed each PT sample in triplicate on two separate days. And the target value was the mean. And initially when in 2007 the passing value for hemoglobin A1C was plus minus 15% was deemed acceptable, and this was progressively tightened going down 6% in 2013. This resulted in a marked improvement in accuracy among laboratories, and the CVs went from more than 20% to currently less than 2.5%. In addition to clinicians having more confidence in the accuracy of their values for management, it resulted in 2010 in for the very first time hemoglobin A1C being actually recommended for diagnosis of diabetes worldwide. So this was a huge change in both the screening diagnosis and management of patients.

Dr. Andy Hoofnagle:

So you said that we're at two and a half percent, but that doesn't line up with what CMS recently recommended for their CLIA.

Dr. David Sacks:

The current college acceptability criteria plus minus 6%. The CMS decided initially wanted to change it to plus minus 10%, but subsequently, after many people wrote in, reduced it to plus minus 8%. And that's what the law now states.

Dr. Andy Hoofnagle:

When we see this huge discrepancy, your work, NGSP, others have helped make this incredible achievement. And now CMS is saying, ah, it doesn't matter. What can the college do to help guarantee and to continue the path forward in improving hemoglobin A1C's accuracy?

Dr. David Sacks:

The College of American Pathologists is planning to have a grading for participants. They're going to give them two results, both plus minus 8%, which is required for passing, but also plus minus 6% in labs can see how well they perform. And also they plan to start having labs that are accredited by CAP evaluated to make sure that they reach the plus minus 6% criteria. And in fact, currently more than 95% of labs actually participants in the CAP program meet these criteria.

Dr. Andy Hoofnagle:

So I mean, just at a different way when I think about it, being CAP accredited will actually be a step up even for hemoglobin A1C, it'll mean something a little bit more than just being CLIA. To go back historically, if you don't mind for just a second, this was again, a big leap, not just for the field philosophically, but practically speaking, you guys decided to use actual whole blood as a PT material, and that's expensive, right?

We have to collect it, we have to process it, we got to get it sent out. Was there another kind of sort of matrix match material that you thought about or did you go straight to whole blood? And how did you convince people that this was going to be worth the expense and all that?

Dr. David Sacks:

It's more expensive, but in the long run, it's essential to avoid matrix effects. And the only way to do this was to use whole blood. And despite the expense the College of American Pathologists prioritized patient care over cost and succeeded in changing the field as I said.

Dr. Andy Hoofnagle:

The CAP has to be commended for all of its efforts in getting matrix matched materials in trying to partner with folks to make reference methods. While there's programs outside the United States, there are no others that I'm aware of that really come close to the quality of what CAP is trying to do for accuracy-based materials.

Dr. David Sacks:

Yes, I agree with that. Do you see any reason to have non-accuracy-based proficiency testing programs?

Dr. Andy Hoofnagle:

Not accuracy-based, not matrix matched? Well, let me tease that apart a little bit. First, accuracy-based, I think we have to decide between harmonization and standardization. And I think the benefits of the accuracy part, having the reference method and having everything come back to the right answer argues for standardization. And there's not that many assays in chemistry that have literal numbers that are the same across all guidelines across all hospitals. And kind of alluding back to what we were talking about before, if your cholesterol, yes, if your vitamin D, if your others, there's a line, we need to get that line right. If not, harmonization is probably okay. Not having access to a reference method may not be the end of the world, provided we can get labs again, because changing from hospital to hospital, lab to lab, at least everybody getting the same answer is awesome.

So harmonization should always be our goal, but getting the absolute right accurate answer is probably less important. So I do, I think that there's any role for non accuracy-based proficiency testing material? I really hope that we begin thinking about matrix matching so that we can harmonize at least and reference methods when it's necessary to standardize. And so the answer the other half, I think I already kind of alluded to it. I really do think the more and more that we move to a matrix similar matrix matching, the better we're going to do for our patients because we're going to be able to harmonize this whole peer group grading approach. While really important for maintaining proficiency within a laboratory, doesn't do anything to help those patients who do have to move from system to system.

Dr. David Sacks:

Developing reference methods is very labor intensive. The methods are expensive often, and sometimes it's very difficult to establish a reference method. And so what do you recommend in a situation like that where the reference method is difficult to obtain?

Dr. Andy Hoofnagle:

There aren't many examples of proteins as an example that have reference methods. And if there are reference methods, there are immunoassays instead of mass spec, which of course we hold up as the gold standard. I think the field is moving in the direction of more and more assays by mass spectrometry being the reference method or at least chromatographic separation as in the case of hemoglobin A1C. When it is hard and there isn't a number that we're treating to, I think harmonization is okay, I will set it aside, but when it's difficult, but we really need the method, we really do need a reference method. We have to collaborate with our friends and colleagues at the large metrological institutes in the world, whether it's Japan, France, here in the United States at NIST, the CDC has done a lot of work in trying to develop reference methods in addition to NIST.

And we have other labs throughout the country that are just trying to chisel away at the problem. They see that there's a need for accuracy. There's the group out of what was formerly AACC, ADLM, the harmonization group. They've identified the biomarkers that need reference methods. They're trying to prioritize them. They're trying to find ways for federal governments, other institutions to help pay for the development of reference methods. And I hope that we focus on mass spec as the cornerstone. IFCC is currently working on reference methods for apolipoproteins, both A1 and B as well as LPa. And so there are efforts to make reference methods. They're going to get rolled out. Hopefully they'll get approved by JCTLM, which is the organization that approves reference methods. I think it's one step at a time.

Lisa Tomcko:

We are going to take a quick break from today's episode to talk about an exciting new program from the CAP, now open for registration. When it comes to successfully starting a new job. Preparation is key. Introducing the Job Prep Bootcamp from the CAP, a fast-paced interactive virtual review designed just for pathologists like you refresh your skills in signing out less familiar cases, access the library of resources to ensure thorough and accurate case workups, learn to recognize and avoid common pitfalls, get guidance from experts and tap into a supportive network exclusive to the Job Prep Bootcamp alumni community. You can find the link in the episode description to get more details and register. And now back to the episode.

Dr. David Sacks:

So how does one actually determine which assays to develop a reference method to?

Dr. Andy Hoofnagle:

Having groups of experts think about what the priority for each of the tests should be I think is super helpful. The International Consortium for Harmonization of Clinical Laboratory results is made up of content experts who are thinking about how lab tests are being used, whether different labs getting different results could be misleading for patients or patient care in general.

Dr. David Sacks:

I think that's a very important consideration before one invests a lot of time, effort, and resources in developing a reference method.

Dr. Andy Hoofnagle:

You were able to change the field not just because of NGSP and CAP, but you got the manufacturers to change their ways. And that's hard. What did you do?

Dr. David Sacks:

That's actually a very important consideration. And in fact, I can unequivocally state that accuracy of hemoglobin A1C measurements would not have, certainly not reach the stage that they have today without very active participation of the manufacturers. So what we did was we started a program of certification, the NGSP certifies methods of hemoglobin A1C methods, and the manufacturers participate voluntarily. There's a sample comparison, whole blood, everything's done with whole blood between the manufacturer and the NGSP reference lab. And if the manufacturer shows stringent accuracy, then they obtain a certificate. And this certificate is only good for one year.

And initially the NGSP started off with quite wide criteria, relatively wide criteria for accuracy. And these have been progressively tightened over the years. And the manufacturers participate voluntary, and in fact, they actually pay to do this because it's a very expensive for the NGSP. This was driven initially, I think by manufacturers. Once one manufacturer had a certificate and they could go to labs and say, look, my hemoglobin A1C method is NGSP-certified, you've got to use this. That stimulated the field. And in fact, what's happened subsequently is that almost all methods now in the US and much of the world, in fact, not just the US are NGSP-certified, and the guidelines which are followed by widely, the American Diabetes Association guidelines and the AACC guidelines now specify that labs should use only hemoglobin A1C assay methods that are certified by the NGSP. And moreover, they also say that for hemoglobin A1C labs should participate in accuracy-based PT program that uses fresh whole blood with targets set by the NGSP.

Initially, we had the certificate program, but then the way that we got the manufacturers to improve their methods was to meet with them regularly. We meet with them every year at the AACC. And during the time when the CAP was tightening the acceptability criteria for proficiency testing, we notified the manufacturers, we discussed this with the manufacturers, we advised them what the ultimate goal was and what the timeframe would be to reduce it so that they could actually modify their methods or even discontinue certain methods and switch to new ones because some of the manufacturers said, and they realized that their method could never achieve the accuracy that was necessary. They spent millions and millions of dollars. I have no idea how many, but they really participated in this. And obviously at all these meetings, we continue to emphasize the clinical value and clinical necessity of having accurate results.

Dr. Andy Hoofnagle:

 Should we be tying accuracy to payment or is that taking it a step too far?

Dr. David Sacks:

That's a very difficult question. I mean, I think that it's very important, in my opinion, to have accurate results for the analytes that we've discussed earlier. And if you have negative impacts on patient care, then the clinicians will make wrong decisions if they get the wrong result. And it could be catastrophic for patients. There has to be a way to convince labs to use accurate methods, and I think that's particularly different. Difficult now with cost considerations. I don't know what your thoughts are regarding that.

Dr. Andy Hoofnagle:

Interesting. So you're saying we know that things are getting more expensive already, so now adding an additional layer would be more expensive. What's the concern around that? Is that what you're asking?

Dr. David Sacks:

Well, I'm asking you if you think that might result by forcing labs to use methods that are certified.

Dr. Andy Hoofnagle:

I think it will get more expensive. I think it would have to, because you have to administer the program for accuracy-based certification, which as you mentioned, well, you were talking about two different costs. One is to kind of what's the cost of being certified? But then the other is, if I make a change to an assay, how much does it cost for me to go to FDA and say, look, I made a change in my assay. Is that okay? And that's where we've gotten a lot of pushback on C-peptides. The manufacturers say things like, well, that's too much. If I change my calibration system, I'd have to go back to the FDA and get it approved. So your question is, is it going to get more expensive? I think the answer is it has to, doesn't it?

Dr. David Sacks:

Presumably manufacturers will be forced to pass the costs on to the labs.

Dr. Andy Hoofnagle:

Which then passes it onto the patient.

Dr. David Sacks:

Right.

Dr. Andy Hoofnagle:

Yeah, you're right, it is. So we've talked about now proficiency testing materials being matrix matched, there's a cost to that. We've talked about them being accurate, but in other words, having a reference method, there's a cost to that. There's the concept of getting certified and building a program around that to help manufacturers have a blue ribbon to say, I am the best. And then finally having, after being certified, then the continuous program around continuing to be certified, making changes at FDA, et cetera if needed. All of that is additional cost. However, all of that additional cost would improve quality and certainly would improve accuracy for the patient, which they're relying on to make the right medical decision, which as you said before, it can have catastrophic consequences if they make the wrong one. So yes.

Dr. David Sacks:

I agree. Yeah, I think that it also, if you look at the big picture as to healthcare costs in this country or other countries preventing complications in patients and getting the diagnosis correct, the costs for that are relatively small compared to the costs of treating these patients with irreversible complications often.

Dr. Andy Hoofnagle:

That's also a great call, right? Picking the wrong med for the wrong disease or the wrong diagnosis, can, yeah, you're right. Watching what you did with hemoglobin A1C and watching what we haven't been able to do with C-peptide, knowing what we haven't been able to do with testosterone and estradiol, knowing that we really do need the manufacturers to grab on for the ride and they really do have to help us steer. Do you have any other tricks besides blue ribbons besides tying payment? Any other ways that you can think of getting manufacturers recognize how important holding onto the reins of this process is?

Dr. David Sacks:

I don't think there's a simple answer, and I don't think there's one answer that would fit all assays. I mean, clearly the emphasis needs to be on the clinical implications of an accurate result versus an inaccurate result. And I think if manufacturers can be convinced that having an accurate assay is of great benefit to patient care, I think they will be more inclined to participate in this. It ties into the clinical value of the assay and whether there's evidence and guidelines from clinical organizations that emphasize how important that assay is in patient care. So if it's an assay that's rarely used, then I think it would be much more difficult. But if it's something that is really fundamental to patient care the way hemoglobin A1C is to diabetes, then I think it's easier.

Dr. Andy Hoofnagle:

There's a calculus in your head, there's a number of people with the disease, the number of people with the disease that will be tested, and then what could be done with that result. And that's all coming together to make it more or less likely that manufacturers be excited to participate and really steer the process.

Dr. David Sacks:

Yes, and I also think it's a bit of the chicken and egg because clinicians don't want to use assays if they're not accurate. And how do you show that an analyte is a value in a common disease if you don't have an accurate assay? So it becomes very difficult to convince manufacturers if the clinicians don't, are not convinced that the analyte is of clinical value.

Dr. Andy Hoofnagle:

Yeah, absolutely. We do need large clinical studies that have outcomes tied to them to convince people to make changes, and to have the assay to do that, you have to spend time and resources to make a good assay that will last through a large study. And then as you've kind of alluded to, be adaptable to the future after the study is done. So it's a lot.

Dr. David Sacks:

You also need people like you and the members of your committee who are going to champion these and are willing to put in a lot of time and effort to affect these changes.

Dr. Andy Hoofnagle:

So where do you want to see us go next? Where do you want our committee to be working? What do you think is most helpful?

Dr. David Sacks:

Well, I'm not a member of your committee, so you're the chair, you're the one who has to answer that question.

Dr. Andy Hoofnagle:

Well, I know where we're headed, right? We're going to be trying to expand certain programs. Proficiency testing programs that used to be concocted now would actually rely on matrix matched materials, trying to identify all the assays that really do need reference methods, and then working with providers of those reference methods to give us timely results. You kind of alluded to cost before. We have to be able to make it so that it's cost effective for the laboratories that want to participate in these kinds of programs. But I really think that we can get more accuracy-based around eptide and get more and more, I think this is just me throwing this out there. I think one day C-peptide could be important in trying to identify which patients get specific diabetes treatments based on what their beta cell reserve. I think that's possible. But to get there, we have to have an assay that's reliable, but also albumin and urine.

Hugely important cutoff in patients who are at risk of developing chronic kidney disease or even categorizing current chronic kidney disease. We don't have the best reference method for measuring albumin in urine. And so our committee is working with groups around the world to try to identify what is the reference method? Is there a group of reference methods that can answer this question so that we can get manufacturers to really hone in on the right answer? So I think the committee is well poised to expand what assays are being tested proficiency testing wise with matrix match materials. We'll try to get as many reference methods as we can for those analytes where the number really matters and try to do it in a cost-effective way so that the patient doesn't bear the brunt of a huge cost of laboratory testing, but knows that the accuracy as they move from hospital to hospital is constantly getting better.

Dr. David Sacks:

Sounds like the committee is doing good work, and it sounds like you're going to be very busier for the next two years, so you'll be very active and you've got lots to do, but it sounds like you're going about it the right way, in my opinion.

Lisa Tomcko:

Well, thank you both for all the insights into the CAP's accuracy-based PT program. We talked about how it came into existence as well as the impact it's made and where it's going. Truly, this has been a fascinating discussion to sit in on.

Dr. Andy Hoofnagle:

Thank you, David, for doing this. I really appreciate you taking the time. Thanks, Lisa for organizing, and thanks to all the people who are actually participating in the accuracy-based programs. We would love to get more and more participants to help statistically know what we're doing right and what we're doing wrong.

Dr. David Sacks:

I think we've covered a large area, and I want to thank Andy for coming up with the questions and leading the discussion.

Lisa Tomcko:

And thank you all for listening. We also invite our listeners to check out the other episodes in our 75 years of Proficiency Testing and External Quality Assessment series. So far, we've covered the LPX and Infectious Disease programs, and there are more to come. You can also learn more about the CAP's proficiency testing program via the link in the show notes. And for more information about the CAP visit cap.org.

Download Transcript