Replication a Challenge in Quest for Alzheimer’s Blood Test

As the Alzheimer’s field matures, it has begun the task of moving fluid biomarkers from the discovery stage toward the clinic. For cerebrospinal fluid markers of AD, researchers have made great strides toward standardization, but the field’s more nascent blood-based biomarkers remain plagued by problems with replication and methodology. Case in point: A 2014 Nature Medicine report from Howard Federoff, then at Georgetown University Medical Center, Washington, D.C., and Mark Mapstone, then at the University of Rochester School of Medicine, New York, described a panel of 10 plasma lipids that reportedly presaged cognitive impairment within the next few years. Both Federoff and Mapstone are now at the University of California, Irvine. The paper generated excitement and headlines around the world. It also kicked off efforts by multiple groups to repeat the findings. The first of these has now been published. In the January 21 Alzheimer’s & Dementia, researchers led by Madhav Thambisetty at the National Institute on Aging in Baltimore reported they were unable to reproduce the findings in two larger, independent cohorts.

Does this mean the 10-lipid panel has no validity across populations? Not necessarily. Federoff and Mapstone note that methodological differences between the studies may explain the discrepancies. Meanwhile, replication attempts by them and others continue. More broadly, biomarker researchers point to consistent evidence that phospholipids in blood do associate with AD, and maintain that the hunt for a metabolite-based blood test will eventually bear fruit. The National Institutes of Health has put its weight behind this effort because of the advantages in cost and convenience that a blood test would have over imaging or spinal taps for routine clinical use.

Researchers lauded the current focus on replication and standardization. “This is an excellent study and a solid contribution to the field,” said Sid O’Bryant at the University of North Texas Health Science Center, Fort Worth, of Thambisetty’s work. “It does exactly what we have to do: take findings from preliminary studies into large cohorts,” O’Bryant added. Henrik Zetterberg at the University of Gothenburg, Mölndal, Sweden, noted, “These types of independent replication studies are essential. In other fields, it is extremely hard to publish these types of negative findings in high-impact journals.”

Despite much research in this area, a blood-based test for Alzheimer’s has remained elusive (see Jun 2013 webinar; AlzBiomarker database). Although numerous plasma markers differ between people with AD and healthy controls, these typically lack the sensitivity and specificity necessary for a diagnostic test (see Kivipelto et al., 2006; Doecke et al., 2012; Burnham et al., 2013). Ideally, researchers want a test that would predict who among a cognitively healthy population will go on to develop AD. A small 2007 study reported finding a panel of 18 blood proteins that might do the trick, but no one has been able to replicate that data (see Oct 2007 news). 

Thus, great enthusiasm greeted the 2014 report from Federoff and Mapstone of a predictive lipid panel. The 10 plasma metabolites distinguished 28 people who went on to develop cognitive impairment within three years from 53 controls who remained healthy, and did so with 90 percent sensitivity and specificity, the authors reported (see Mar 2014 news). The authors have since refined their test to a 24-metabolite panel that they claim has somewhat better predictive power, using data from the same cohort (see Fiandaca et al., 2015). 

To repeat the findings, Thambisetty and colleagues used serum samples from two different longitudinal cohorts, the Baltimore Longitudinal Study of Aging (BLSA) and the Age, Gene/Environment Susceptibility-Reykjavik Study (AGES-RS). The latter is a collaborative study between the NIA and the Icelandic Heart Association to look at risk factors for Alzheimer’s and other age-related diseases in a long-running longitudinal cohort. First author Ramon Casanova analyzed baseline and five-year serum samples using the same commercial metabolomics platform as the Federoff study did, namely the AbsoluteIDQ p180 assay from Biocrates Life Science AG, Innsbruck, Austria. In the BLSA study, the authors compared 93 people who progressed to AD with 99 others who did not. From the AGES-RS cohort, Casanova et al. analyzed 100 progressors vs. 100 who stayed stable. All participants were cognitively normal at baseline. Diagnosis was determined by clinical consensus among neuropsychologists and clinicians in the respective studies.

The 10-lipid panel performed poorly in both cohorts, with a sensitivity of around 50 percent for predicting people who would progress to AD, the authors reported. To find out if any metabolites in the AbsoluteIDQ platform correlated with disease progression, the authors then analyzed all 187 markers. They found a set of markers that had moderate predictive power in the BLSA cohort, as well as a slightly different set that distinguished people with current AD from healthy controls in the BLSA, but neither replicated in the AGES-RS cohort. While there may be a metabolite profile associated with AD, the specific panels explored so far are not clinically useful, Thambisetty told Alzforum.

Mapstone disagrees. He contends that differences in methodology, such as the use of serum instead of plasma, may account for the failure to find a signal. “We do not believe valid conclusions can be drawn from this attempt at replication,” Mapstone wrote to Alzforum (see full comment below). Speaking with Alzforum, he added, “There are certain conditions we controlled for in our study that were not taken into account in this one. If you’re doing a replication study, you need to do it prospectively and not use samples on hand.” He said he and his colleagues would detail the issues in a letter to the journal.

O’Bryant agreed that differences in the blood fraction used could play a role. “When you look across serum and plasma, the results are often different,” he told Alzforum, adding, “The jury is still out on the Federoff panel.” O’Bryant was not involved in either study.

He and others stressed that the field must standardize methodology for collection, processing, and storage of samples. In 2015, the international Blood-Based Biomarker Interest Group (BBBIG) and STandards for Alzheimer's Research in Blood biomarkers (STAR-B) published the first guidelines for processing Alzheimer’s blood protein biomarkers (see O’Bryant et al., 2015). This group comprises largely AD and assay development researchers in academia, industry, and diagnostics companies. They incorporate lessons learned from the CSF field. The group is currently preparing additional guidelines for protein biomarker validation studies, O’Bryant said.

Meanwhile, other groups are turning their attention to small molecules. The National Institute on Aging supports the Alzheimer’s Disease Metabolomics Consortium, a large initiative of different groups. It brings together eight metabolomics and informatics centers, led by Rima Kaddurah-Daouk at the Duke Institute for Brain Studies, Durham, North Carolina (see Feb 2011 news series). The consortium is applying a standardized approach to define metabolic changes in Alzheimer patients, Kaddurah-Daouk said. Numerous factors affect blood metabolites, including sex, age, diet, and medications, and they must be accounted for in analyses, she added. The consortium partners with the Alzheimer’s Disease Neuroimaging Initiative to identify new biomarkers and drug targets using ADNI samples.

As part of this effort, Kaddurah-Daouk has examined the Federoff panel in 800 participants in ADNI1, who ranged from cognitively healthy to having dementia. This particular 10-lipid panel did not predict impairment in this cohort, she told Alzforum. However, she did see a consistent correlation between Alzheimer’s disease and perturbed phospholipid metabolism, supporting other findings in the field such as those from Federoff and Thambisetty. “Many papers have come to the conclusion that there is a problem in phospholipid metabolism in AD, but it will take rigorous standardized studies to zoom in on [specific] biomarkers,” Kaddurah-Daouk said. She emphasized the need to deal with confounding factors such as medications that could have affected the findings in the reported studies. Profiling ADNI2/GO samples with use of matched plasma and serum will provide additional insight about the limitations of the data available thus far, she added.

Other biomarker experts concur that the field is slowly homing in on reliable blood markers, even if it has not found quite the right combination yet. Proteins such as clusterin and C-reactive protein, for example, consistently produce a signal in multiple studies (see Jongbloed et al., 2015; Liang et al., 2015; Yarchoan et al., 2013). “It is encouraging that different groups agree on some of the pathways/metabolites that seem to play a role in disease development and progression,” Eugenia Trushina at the Mayo Clinic in Rochester, Minnesota, wrote to Alzforum (see full comment below). Thambisetty agreed, “A large body of evidence suggests there is a blood signal associated with core pathological features of AD.”

The challenge for the field now is to refine the procedures enough to get repeatable results across different populations. This will be necessary to take these markers to the clinic. “The gap between discovery and clinical implementation is all about methods,” O’Bryant said.—Madolyn Bowman Rogers

Comments

  1. Two studies reported contradicting findings on the attempt to identify blood biomarkers of preclinical AD. While Mapstone et al. reported the identification of a panel of 10 markers specific for diagnosis of AD in blood, Casanova et al. failed to confirm this panel in larger cohorts of patients using similar techniques. The former study used 28 samples from converters, the later utilized almost 200. The discrepancy in the findings is troubling since such controversy may undermine the application of an otherwise promising metabolomics approach for early diagnosis of AD.

    Both studies included well-characterized patient populations. Disease diagnosis in both studies was carefully conducted using established and consistent tests.  However, one obvious discrepancy in the study groups is an inconsistent ratio between men and women. Emerging data suggest the presence of sex-specific differences in the development of AD where metabolic pathways, especially those involved in energy production, are affected differentially. Therefore, it is surprising that neither of the studies discussed the potential effect of group composition on the results. Similarly, the studies failed to conduct a separate analysis based on sex. Another pitfall was the lack of detailed list of medications in study groups. Since metabolomics can also detect small molecules produced during drug metabolism, it is imperative to carefully control for that.

    Another important point, which was raised recently at meetings of the Metabolomics Society, is the transparency and availability of the raw metabolomics data for direct comparison. As was pointed out by Casanova et al., it is unclear how the 10-biomarker panel was determined by Mapstone et al. in the first place. In order for comparative analyses to be possible, publications presenting findings using systems biology approaches should include uploaded raw data along with the data analysis and interpretation. Without this level of transparency, it will be impossible to understand what particular parameters account for the lack of reproducibility.

    Additionally, it is very important to standardize sample collection, processing and storage. It has been estimated that up to 46 percent of laboratory errors come from pre-analytic processing (Becan-McBride, 1999). Factors related to blood collection devices (needle gauge, tube lubricants, tube walls) can impact blood marker levels. While Casanova et al. discussed some of the parameters, details on sample collection/storage are missing in the paper of Mapstone et al.

    It is clear that validation of metabolomics data in larger patient cohorts is essential for blood biomarker discovery. What is encouraging is that different groups agree on some of the pathways/metabolites that seem to play a role in disease development and progression.  With the availability of blood samples from AD patients from such initiatives as ADNI, it is now possible to design studies to validate results using different or similar platforms in the same subset of samples. This could help to develop strategies for better accuracy and reproducibility of metabolomics data.

    References:

    Becan-McBride K. Laboratory sampling. Does the process affect the outcome?. J Intraven Nurs. 1999 May-Jun;22(3):137-42. PubMed.

    View all comments by Eugenia (Jania) Trushina

  2. While we applaud the efforts of Casanova et al., we do not believe valid conclusions can be drawn from this attempt at replication. As scientists, we all understand that independent reproducibility of an experiment requires the use of the same experimental method. The use of pre-existing data and methods that are not commensurate with the original experiment do not represent a rigorous attempt at replication.

    While these results may be informative on the broader issue of lipid metabolism in preclinical AD, given multiple differences in the experimental design and approach and sample types, it does not represent a true replication of our work. We have prepared a response, which we are submitting to the Journal.

    View all comments by Mark Mapstone

  3. The search for blood biomarkers is obviously of high interest for Alzheimer’s disease diagnosis, and it is always disappointing to see promising results not being replicated. However, it is not a big surprise, and I encourage people working in this field to look back over the genetic research several years ago.

    Over a period of sixteen years, we were unable to validate a new genetic risk factor in AD. We had many problems related to the selection of the genes to be studied, and the quality and size of the populations. The proliferation of association studies (with small populations of fewer than 100 cases and controls) has resulted in finding numerous—too many—associations (false positive results). Inversely, low statistical power associated with these small association studies also leads us to reject potentially relevant genes due to false negative results. Consequently, discoveries have stagnated, potentially interesting results are barely noticeable, and the genetic approach, especially association studies, has lost a bit of credibility.

    Unfortunately, the classical epidemiological issues are exacerbated by the technical complexity associated with proteomics/metabolomics methodologies. That is why it is really important to circumvent at best the limitations that can be controlled without too many difficulties by well-designed studies, e.g., well-diagnosed cases, sex- and age- matched controls, and, above all, large numbers of samples. We also need adapted statistical approaches, e.g., to avoid overly complex statistical tools and/or over-stratification according to the number of samples.

    This is the basis to overcome the winner’s curse issue we encountered for many years in the genetic field. I hope that the biomarker field will learn from our errors and will not waste as much time as we did.

    View all comments by Jean-Charles Lambert

  4. I would like to thank Dr. Trushina for her comment, while also pointing out that our methods for sample collection, processing, and storage are outlined in the supplemental material of our manuscript. I direct the Alzforum audience's attention to the section labeled “Blood Collection, Shipment, and Specimen Processing Protocols.” This also details clinical and cognitive methods, tandem MS data, and statistical information supporting our methods.

    In regard to Dr. Trushina's point about medication effects, we agree completely. We attempted to minimize medication effects in our study by requiring medication withholding overnight before the blood draw. This is stated in the methods section of our manuscript. It is not clear if other studies utilize this step before drawing blood.

    We fully endorse transparency in biomarker studies. As stated in the methods section of our paper, we made the raw targeted metabolomic data from our study available upon publication. I am pasting the text from our manuscript below for the convenience of interested colleagues: "Accession codes. Lipodomics data were deposited in the European Bioinformatics Institute MetaboLights database with accession code MTBLS72.” We welcome replication attempts, but emphasize that these should be approached thoughtfully and rigorously.

    View all comments by Mark Mapstone

Make a Comment

To make a comment you must login or register.

References

Webinar Citations

  1. Webinar: O Blood-Based Biomarker, Where Art Thou?

News Citations

  1. A Blood Test for AD?
  2. Do Lipids Hold the Key to Blood-Based Alzheimer’s Test?

Series Citations

  1. Metabolomics Comes of Age

Paper Citations

  1. Kivipelto M, Ngandu T, Laatikainen T, Winblad B, Soininen H, Tuomilehto J. Risk score for the prediction of dementia risk in 20 years among middle aged people: a longitudinal, population-based study. Lancet Neurol. 2006 Sep;5(9):735-41. PubMed.
  2. Doecke JD, Laws SM, Faux NG, Wilson W, Burnham SC, Lam CP, Mondal A, Bedo J, Bush AI, Brown B, De Ruyck K, Ellis KA, Fowler C, Gupta VB, Head R, Macaulay SL, Pertile K, Rowe CC, Rembach A, Rodrigues M, Rumble R, Szoeke C, Taddei K, Taddei T, Trounson B, Ames D, Masters CL, Martins RN, . Blood-Based Protein Biomarkers for Diagnosis of Alzheimer Disease. Arch Neurol. 2012 Jul 16;:1-8. PubMed.
  3. Burnham SC, Faux NG, Wilson W, Laws SM, Ames D, Bedo J, Bush AI, Doecke JD, Ellis KA, Head R, Jones G, Kiiveri H, Martins RN, Rembach A, Rowe CC, Salvado O, Macaulay SL, Masters CL, Villemagne VL, . A blood-based predictor for neocortical Aβ burden in Alzheimer's disease: results from the AIBL study. Mol Psychiatry. 2013 Apr 30; PubMed.
  4. Fiandaca MS, Zhong X, Cheema AK, Orquiza MH, Chidambaram S, Tan MT, Gresenz CR, FitzGerald KT, Nalls MA, Singleton AB, Mapstone M, Federoff HJ. Plasma 24-metabolite Panel Predicts Preclinical Transition to Clinical Stages of Alzheimer's Disease. Front Neurol. 2015;6:237. Epub 2015 Nov 12 PubMed.
  5. O'Bryant SE, Gupta V, Henriksen K, Edwards M, Jeromin A, Lista S, Bazenet C, Soares H, Lovestone S, Hampel H, Montine T, Blennow K, Foroud T, Carrillo M, Graff-Radford N, Laske C, Breteler M, Shaw L, Trojanowski JQ, Schupf N, Rissman RA, Fagan AM, Oberoi P, Umek R, Weiner MW, Grammas P, Posner H, Martins R, STAR-B and BBBIG working groups. Guidelines for the standardization of preanalytic variables for blood-based biomarker studies in Alzheimer's disease research. Alzheimers Dement. 2015 May;11(5):549-60. Epub 2014 Oct 1 PubMed.
  6. Jongbloed W, van Dijk KD, Mulder SD, van de Berg WD, Blankenstein MA, van der Flier W, Veerhuis R. Clusterin Levels in Plasma Predict Cognitive Decline and Progression to Alzheimer's Disease. J Alzheimers Dis. 2015;46(4):1103-10. PubMed.
  7. Liang HC, Russell C, Mitra V, Chung R, Hye A, Bazenet C, Lovestone S, Pike I, Ward M. Glycosylation of Human Plasma Clusterin Yields a Novel Candidate Biomarker of Alzheimer's Disease. J Proteome Res. 2015 Dec 4;14(12):5063-76. Epub 2015 Nov 4 PubMed.
  8. Yarchoan M, Louneva N, Xie SX, Swenson FJ, Hu W, Soares H, Trojanowski JQ, Lee VM, Kling MA, Shaw LM, Chen-Plotkin A, Wolk DA, Arnold SE. Association of plasma C-reactive protein levels with the diagnosis of Alzheimer's disease. J Neurol Sci. 2013 Oct 15;333(1-2):9-12. Epub 2013 Aug 23 PubMed.

Other Citations

  1. AlzBiomarker

External Citations

  1. Age, Gene/Environment Susceptibility-Reykjavik Study 
  2. Alzheimer’s Disease Metabolomics Consortium

Further Reading

Primary Papers

  1. Casanova R, Varma S, Simpson B, Kim M, An Y, Saldana S, Riveros C, Moscato P, Griswold M, Sonntag D, Wahrheit J, Klavins K, Jonsson PV, Eiriksdottir G, Aspelund T, Launer LJ, Gudnason V, Legido Quigley C, Thambisetty M. Blood metabolite markers of preclinical Alzheimer's disease in two longitudinally followed cohorts of older individuals. Alzheimers Dement. 2016 Jul;12(7):815-22. Epub 2016 Jan 21 PubMed.

Annotate

To make an annotation you must Login or Register.