Guest post: the perils of genetic risk prediction in autism

This guest post from Daniel Howrigan, Benjamin Neale, Elise Robinson, Patrick Sullivan, Peter Visscher, Naomi Wray and Jian Yang (see biographies at end of post) describes their recent rebuttal of a paper claiming to have developed a new approach to genetic prediction of autism. This story has also been covered by Ed Yong and Emily Willingham. Genomes Unzipped authors Luke Jostins, Jeff Barrett and Daniel MacArthur were also involved in the rebuttal.

Last year, in a paper published in Molecular Psychiatry, Stan Skafidas and colleagues made a remarkable claim: a simple genetic test could be used to predict autism risk from birth. The degree of genetic predictive power suggested by the paper was unprecedented for a common disease, let alone for a disease as complex and poorly understood as autism. However, instead of representing a revolution in autism research, many scientists felt that the paper illustrated the pitfalls of pursuing genetic risk prediction. After nearly a year of study, two papers have shown how the Skafidas et al. study demonstrates the dangers of poor experimental design and biases due to important confounders.

The story in a nutshell: the Skafidas paper proposes a method for generating a genetic risk score for autism spectrum disorder (ASD) based on a small number of SNPs. The method is fairly straightforward – analyze genetic data from ASD case samples and from publicly available controls to develop, test, and validate a prediction algorithm for ASD. The stated result – Skafidas et al. claim successful prediction of ASD based on a subset of 237 SNPs. For the downstream consumer, the application is simple – have your doctor take a saliva sample from your newborn baby, send in the sample to get genotyped, and get a probability of your child developing ASD. It would be easy to test fetuses and for prospective parents to consider abortions if the algorithm suggested high risk of ASD.

The apparent simplicity is refreshing and, from the lay perspective, the result will resonate above all the technical jargon of multiple-testing correction, linkage disequilibrium (LD), or population stratification that dominates our field. This is what makes this paper all the more dangerous, because lurking beneath the appealing results is flawed methodology and design as we describe below.

We begin our critique with the abstract from Skafidas et al. (emphasis added):

Autism spectrum disorder (ASD) depends on a clinical interview with no biomarkers to aid diagnosis. The current investigation interrogated single-nucleotide polymorphisms (SNPs) of individuals with ASD from the Autism Genetic Resource Exchange (AGRE) database. SNPs were mapped to Kyoto Encyclopedia of Genes and Genomes (KEGG)-derived pathways to identify affected cellular processes and develop a diagnostic test. This test was then applied to two independent samples from the Simons Foundation Autism Research Initiative (SFARI) and Wellcome Trust 1958 normal birth cohort (WTBC) for validation. Using AGRE SNP data from a Central European (CEU) cohort, we created a genetic diagnostic classifier consisting of 237 SNPs in 146 genes that correctly predicted ASD diagnosis in 85.6% of CEU cases. This classifier also predicted 84.3% of cases in an ethnically related Tuscan cohort; however, prediction was less accurate (56.4%) in a genetically dissimilar Han Chinese cohort (HAN). Eight SNPs in three genes (KCNMB4, GNAO1, GRM5) had the largest effect in the classifier with some acting as vulnerability SNPs, whereas others were protective. Prediction accuracy diminished as the number of SNPs analyzed in the model was decreased. Our diagnostic classifier correctly predicted ASD diagnosis with an accuracy of 71.7% in CEU individuals from the SFARI (ASD) and WTBC (controls) validation data sets. In conclusion, we have developed an accurate diagnostic test for a genetically homogeneous group to aid in early detection of ASD. While SNPs differ across ethnic groups, our pathway approach identified cellular processes common to ASD across ethnicities. Our results have wide implications for detection, intervention and prevention of ASD.

We recently published a letter of response in Molecular Psychiatry, which uses data from over 5,000 individuals with ASDs in a case/pseudo-control design, as well as a 4,500 unrelated case/control design. Our analyses suggest that there is no clear association between autism and the set of genetic markers or biological pathways named by Skafidas et al.

How could our findings be so different from those in the original report, given the extraordinary claims in the Skafidas et al. manuscript? The impressive predictive ability from such a small sample far outstrips the performance of similar analyses for other complex traits. For example, similar levels of prediction have not been achieved in schizophrenia with far larger sample sizes. Furthermore, if a small set of variants were so predictive of autism, the single-locus association results for those variants should have been highly significant (which they were not).

Simply put, these results are inconsistent with current understanding of the genetics of autism. This dissonance with current scientific results was the primary motivation for two separate groups to investigate and respond to the work.

We believe it possible that methodological errors contributed to findings reported by Skafidas et al. In particular, population stratification—when cases and controls have on average different ancestral origin—was a prominent concern for both letters of response. In order to accurately estimate the relationship between a genetic variant and a disease, one must make sure that there are no other differences between cases and controls that confound the association. Population stratification is a classic example of this type of confounding.

Skafidas et al. studied ASD cases from the Autism Genetic Research Exchange (AGRE) and controls from the HapMap3 Northern and Western European (CEU) reference population. The cases and controls include individuals that, on average, differ in genetic ancestry. In a letter responding to the Skafidas et al. manuscript, Belgard et al. suggest that these ancestral differences are inflating the power of the predictor:

• The cases are of diverse European ancestry (as expected in an American sample), whereas the controls are from Northwestern Europe exclusively. This means that Eastern European ancestry is likely to be found in the cases, but not in the controls.
• Any SNPs with different allele frequencies between areas of Europe represented in the ancestry of the cases, but not in the ancestry of the controls, could therefore appear associated with autism.
• To test this, Belgard et al. compared the minor allele frequency of Skafidas et al.’s 30 published SNPs between the Northwestern CEU population and an Estonian (Eastern European) sample.
• Most of the SNPs that were genotyped in both studies (16 of 18) differed between Eastern and Northwestern Europeans. They differed in direction one would expect if the Skafidas et al. predictor were confounded by population structure.

Aside from the omission of controlling for population stratification, the possibility of technical errors strongly suggest the need for independent testing and validation of the Skafidas et al. predictor before being used as a classifier or test for ASD.

The fundamental rules for development of a risk predictor are simple:

1. Use a “training” sample to identify risk alleles and estimate their effect sizes
2. Create risk scores for individuals in a validation sample for example by summing the risk alleles carried by each individual and weighting them by the effect sizes estimated in the training sample
3. Calculate the efficacy of the predictor in discriminating cases from controls in a completely separate sample. Independence of the validation sample from the training sample ensures that biases in predictive ability are avoided.

Unfortunately it appears that Skafidas et al. broke these rules in two ways. First, the authors used a sample of 975 individuals (732 cases and 123 controls) to identify a set of 775 SNPs that discriminated cases and controls. This sample was subsequently split into training and “validation” sets. In other words, the validation set contributed to the selection of the 775 SNP set that was used in the training sample to establish the predictor.

Second, although difficult to pick up in the published methodology, we have strong reason to believe that the risk alleles and weights of the 237 SNPs identified in the training sample (from the set of 775 SNPs) as being most discriminative between ASD cases and controls were re-estimated in the validation samples to optimize their discriminative ability in the validation samples. In other words, the risk score generated for individuals in the validation sample may result from very different weights of risk alleles – including those of opposite sign – compared to the risk scores generated for individuals in the training sample. Since the purpose of risk prediction is to make inference about the population from which samples are drawn it does not make sense that SNP effects and risk alleles should be sample specific.

In conclusion, we believe that the published Skafidas et al. study is marred by a number of methodological and analytical errors and that consequently their inference about the efficacy of the constructed genetic predictor for ASD is not justified by the data presented.

It is important to note, however, that real progress is being made in other areas of autism and neuropsychiatric genetics. As discussed above, sample size has proven to be the primary driver of genetic discovery, and large, collaborative efforts in both exome sequencing and genome-wide association studies have yielded important insights over the last couple of years. ASDs have been associated with rare genetic events, such as de novo loss of function variants, as well as common genetic influences acting collectively across the genome. These findings have nominated several specific genes for follow up analysis and have greatly advanced our understanding of the complex and diverse genetic architecture of autism. Efforts such as these have enormous potential to improve our understanding of the biology of ASDs and other developmental disorders. As such, it is especially important to concentrate on replicating and validated findings, and not be distracted by spurious claims of genetic prediction.

About the authors:
Daniel Howrigan, PhD is a postdoctoral research fellow in the Analytic and Translational Research Unit at Massachusetts General Hospital and the Broad Institute of MIT and Harvard. Benjamin Neale, PhD is an Assistant Professor in the Analytic and Translational Research Unit at Massachusetts General Hospital and the Broad Institute of MIT and Harvard. Elise Robinson, ScD, MPH is an Instructor in the Analytic and Translational Research Unit at Massachusetts General Hospital and the Broad Institute of MIT and Harvard. Patrick Sullivan, MD, FRANZCP is Distinguished Professor of Genetics and Psychiatry at the University of North Carolina (Chapel Hill, NC, USA) and lead PI of the Psychiatric Genomics Consortium. Peter Visscher, PhD is Professor and Chair of Quantitative Genetics in the Queensland Brain Institute at the University of Queensland and an NHMRC Senior Principal Research Fellow. Naomi Wray, PhD is an Associate Professor of Statistical and Psychiatric Genetics in the Queensland Brain Institute at the at the University of Queensland and an NHMRC Honorary Senior Research Fellow. Jian Yang, PhD is an NHMRC Career Development Fellow in the Queensland Brain Institute at the University of Queensland.