TADPOLE Challenge: Winners Forecast AD Symptoms

Results are in for the TADPOLE Challenge—a prize contest that invited researchers to come up with their own ways of predicting dementia symptoms in participants enrolled in the Alzheimer’s Disease Neuroimaging Initiative (ADNI). In a live webinar on June 14, organizers announced who came out ahead. Contestants had submitted computer code to forecast the clinical status, cognitive scores, and/or imaging characteristics of ADNI3 participants. Once submissions were in, the data were collected and results tabulated. The best methods garnered prize money in the amount of £30,000, donated by the Alzheimer’s Society in London, Alzheimer’s Research U.K., and the Alzheimer’s Association.

  • Winners of the TADPOLE challenge took home £5,000 to £10,000 each.
  • Their codes predicted clinical status and ventricular volume in ADNI3.
  • “Consensus scores” outperformed any single entry.

“The challenge results give a snapshot of current capabilities in predicting progression and conversion,” wrote co-organizer Daniel Alexander, University College London, to Alzforum. “Current technology can add useful predictions of some variables, such as clinical status and ventricular volume. In other tasks, such as estimating future cognitive scores, current methods struggle.”

Announced in June of 2017, the TADPOLE Challenge, short for The Alzheimer's Disease Prediction of Longitudinal Evolution, is a collaboration between the European Progression of Neurological Disease (EuroPOND) consortium and ADNI (Jul 2017 news). Contestants used the longitudinal data sets of ADNI1, ADNIGO, and ADNI2 to develop ways to predict one or all of the following three features in ADNI3 data:

  • A participant’s clinical status (normal, mild cognitive impairment, or probable AD),
  • Score on the Alzheimer’s Disease Assessment Scale-Cognitive (ADAS-Cog),
  • Ventricular volume measured by magnetic resonance imaging (MRI) scans.

By November 2017, 33 teams had submitted a range of methods. Contestants came from Australia, Canada, Europe, Israel, Mexico, Singapore, the U.K. and the U.S.. Most were early career researchers in teams at universities, biotech companies, even high school.

“We received a lot of enthusiasm from challenge participants,” wrote Neil Oxtoby, also at UCL. “Professors told us of excitement from their Ph.D. students and postdocs, who loved the high-impact motivation of improving clinical trials, the unique design of forecasting, and the fun, competitive nature complete with prize pool.”

Baseline data collection from ADNI3 wrapped up in March 2019. TADPOLE organizers downloaded that data, which came from 219 participants who rolled over to ADNI3 from a previous ADNI phase. Then they evaluated which contest submissions most closely predicted the different outcomes. The winners from each category took home £5,000.

For clinical status, the winning team was led by Keli Liu, Stanford University, who was working with a team at Genentech at the time. Combining longitudinal cognitive test scores, clinical diagnosis, regional brain imaging markers, APOE status, and cerebrospinal fluid biomarkers into a machine-learning approach, these scientists achieved an impressive area under the curve of 0.931. This team also took the prize for best overall TADPOLE submission, using their method to score among the top 10 teams across all three categories. This clinical status category saw additional prizes, for a high school team led by Gang Chen, Medical College Wisconsin, and a university team led by Manon Ansart from the Brain and Spinal Cord Institute, Paris.

For ventricular volume, the winning group was Vikram Venkatraghavan’s. He is now a Ph.D. student at Erasmus MC, Rotterdam, the Netherlands, after working as an engineer for some years. Venkatraghavan and colleagues built a combined machine-learning and disease-progression model using both manually and automatically selected longitudinal features from ADNI. Their method predicted ventricle volume with the lowest mean absolute error.

Alas, the ADAS-Cog score resisted prediction. No submission did better than chance, so the organizers gave no prize in this category and reallocated the spoils to two other teams. One half went to the runner-up in the high school category, a group from the Vasile Lucaciu National College in Baia Mare, Romania. The other went to a team led by Steven Hill at the University of Cambridge that used only the most recent cross-sectional measurement from ADNI to forecast values for ADNI3. Their statistical-regression and disease-progression model forecast the ADNI3 measures with the greatest overall precision among entrants who used only cross-sectional ADNI data. Such a method would be most useful in the clinic and for trials, as it enables a likely prognosis for a given person from a single evaluation, rather than requiring longitudinal data and knowledge about their prior rates of change.

Fully understanding the challenge’s findings will require additional analysis, but some preliminary observations jump out. For one, averaging all teams’ forecasts for each measurement into a “consensus” score far outperformed any individual entry. In other words, many heads were better than one. “Such a finding is only possible through an initiative like TADPOLE, where we pull the community together to submit forecasts on the same task using diverse kinds of methods,” Alexander wrote to Alzforum. “It suggests industry might consider crowd-sourcing models with appropriate incentives to get the most robust and reliable predictions.”

For another, ApoE status, CSF biomarkers, and diffusion tensor imaging seemed most informative for future forecasts, while PET measurements of tau, amyloid, or FDG uptake were less helpful.

Organizers are encouraging the participating teams to share their project code publicly. Alexander believes that these methods can be a starting point from which the research community develops ways to make even more robust predictions.—Gwyneth Dickey Zakaib

Comments

No Available Comments

Make a Comment

To make a comment you must login or register.

References

News Citations

  1. TADPOLE Challenge Seeks Best Predictors of Alzheimer’s

External Citations

  1. webinar
  2. TADPOLE Challenge

Further Reading

Papers

  1. Marinescu RV, Oxtoby NP, Young AL, Bron EE, Toga AW, Weiner MW, Barkhof F, Fox NC, Klein S, Alexander DC, the EuroPOND Consortium, for the Alzheimer's Disease Neuroimaging Initiative. TADPOLE Challenge: Prediction of Longitudinal Evolution in Alzheimer's Disease. Cornell University arχiv:1805.03909 [q-bio.PE]. May 10, 2018

Annotate

To make an annotation you must Login or Register.