A missed opportunity: what the GAO report could have told us about DTC genetic testing

The recent United States Government Accountability Office report on direct-to-consumer (DTC) genetic tests was the star attraction of a bruising Congressional hearing into the DTC industry, and sparked widespread headlines about “bogus” results from the genetic testing industry.

The report is prefaced by the statement:

GAO did not conduct a scientific study but instead documented observations that could be made by any consumer.

While it is the GAO’s prerogative to conduct their study as they see fit, we believe that they missed a valuable opportunity to survey the DTC genetic testing industry and systematically evaluate what is and isn’t being done well. In this post, we discuss how the discoveries that the GAO reported were already largely known, and assess the opportunities that the GAO missed to provide genuine insight. What could the data they collected have told us if they had decided to add a little more scientific rigour to their investigation?  

Overview

The GAO report is based on two key sources of information. Firstly, samples from five donors were submitted anonymously and in duplicate to four different personal genomics companies, with the two samples from each individual being labelled with different ages and ethnic backgrounds (one factual and a second fictitious); undercover telephone calls were then made to the companies by these “customers” seeking additional medical advice based on genetic data. All four of these companies are high-profile companies offering genome scans; they were selected as being “frequently cited as being credible by the media and in scientific publications”.

Secondly, the GAO approached 15 companies (including the original four) from a much broader spectrum of the DTC market to investigate their marketing approaches. These companies were approached “both by phone and in person” by GAO employees posing as potential customers, and asked a series of questions about “the reliability and usefulness of test results, privacy policies regarding consumers’ genetic information, and the sale of supplements or other products”. These conversations were recorded.

In addition to the report itself, the GAO released highly edited excerpts of taped conversations as a YouTube video.

The names of all 15 of the companies investigated were obscured in the report, being referred to only by numbers, but in the Congress hearing the first four “credible” companies were revealed as (1) 23andMe, (2)deCODEme, (3) Pathway Genomics and (4) Navigenics, while the remaining 11 remain nameless. The report takes pains to note that the companies it sampled were “nonrepresentative selections” from the industry.

Risk prediction comparisons

The GAO report ordered 40 tests for 5 individuals; two for each individual from each of four “credible” companies. This is not a cheap investigation; the retail price of the tests alone would have cost in the region of $30k (£19k). They compared the risk profiles for 15 diseases, and highlighted a number of cases where the predicted disease risk differed markedly between companies, or differed from the actual medical history of the individuals.

This result is not new: discrepancies in the predictions reported by different DTC companies for the same individual have been reported since the industry’s inception, and their basis was explored in a meeting between three major personal genomics companies in July 2008. In October 2009, a paper was published in Nature by Craig Venter and colleagues that took a similar approach to the GAO (purchasing kits from multiple DTC companies and comparing the results) but included a far more thorough analysis. As well as validating the accuracy of the raw genotyping data, the authors compared disease risk from the two companies, and found that they often disagreed, largely due to differences in which markers are included.

Thus the main message of the GAO report was already well known. The report fails to acknowledge this, despite citing the Nature paper. It also fails to explore the actual basis of these discrepancies, an investigation that would have been of far greater value than simply highlighting previously reported findings.

The larger and more complete dataset collected by the GAO could have been used to performed a number of useful assessments of the industry; they could, for instance, have used their replicate samples to get an independent assessment of genotyping error (both random and systematic) for each company. They also could generated quantitative measures of risk agreement, as the Pathway Genomics response did, showing that the companies agree around 77% of the time, and give opposite predictions around 8% of the time.

Most importantly, the investigation could have moved beyond a simple comparison of numbers to explore the actual basis for the discrepancies: to what extent are these due to differences in the markers used, the risk prediction algorithms employed, or the epidemiological literature used to define population risk estimates? The report makes much of its extensive consultation with experts in the field (led by Dr James Evans at the University of North Carolina); such an analysis would (hopefully!) not have been beyond their capabilities. If the report is to be used as the basis for increased regulation of the industry – and that was the clear subtext throughout the Congressional hearing – it would be ideal if this was based on a detailed understanding of where companies need to improve their analysis, rather than a simplistic pointing out of differences.

Advertising evaluation

The purview of the GAO investigation was all companies selling DTC genetic tests (not just those providing disease risk prediction), and for their assessment of privacy policies, test reliability, and product sales they surveyed a total of 15 companies.  Unfortunately, they did not separate their results by the type of product each company was marketing.  Thus companies providing information on ancestry and risk prediction based upon scientific research were lumped together with the much more dubious companies hawking “personalised” dietary supplements or promising to predict children’s athletic prowess, which have little or no scientific basis.

The GAO report does not explicitly state the criteria against which advertising claims were evaluated, and selectively reports negative assessments. It’s not clear from the report whether all companies were asked the same set of questions (e.g. whether they would accept a DNA sample submitted on someone else’s behalf, or whether their tests are “diagnostic”).  Nor is it clear whether (or how) the GAO validated all of the advertising claims each company makes about its product.

Again, a more thorough investigation could have provided far more useful outcomes. The GAO could have generated a census of the marketing claims being made by these companies, both justified and unjustified, and systematically recorded the accuracy of responses made by customer service representatives. This could have provided an initial framework for appraising the claims of DTC genetics companies.  Instead, we have only a series of cherry-picked statements selected by the GAO as misleading or deceptive, and no way of knowing how widespread such statements were within the industry as a whole.

Conclusions

In terms of scope of data collection, the GAO’s investigation was admirable: 4 major products, 5 individuals, with replicates, recorded follow-up inteviews, investigation of the advertising, a head-to-toe assessment of the industry. But frustratingly, the only conclusions drawn were simple facts that were already well-known:

• companies give differing risk predictions;
• predictions are not diagnoses;
• some snake-oil companies sell dodgy “genetics” products.

The one new result, the two observed instances of major, otherwise reputable companies making clearly problematic statements in phone interviews, is rendered difficult to interpret due to the lack of full transcripts and information on questions asked. It is disappointing that the GAO decided to forsake a constructive investigation for an antagonistic attack, given that, with a little extra work and greater transparency, something far more useful could have been produced.

However, the data still, presumably, exists. Third parties can investigate the risk prediction for the 15 diseases the GAO looked at (as Pathway Genomics did); however, more useful analyses would require the GAO publicly release the product descriptions for each company, full telephone transcripts, information on all the websites surveyed and the criteria used to check the validity of claims. This information could then be used as a resource by the community as a whole to make fine detailed assessments of each aspect of each company and product, rather than broad brush denouncements of the entire industry.

The GAO logo is taken from Wikimedia Common. It is for illustration purposes only, and should not be taken to imply that this content has the endorsement, support or consent of the GAO.