This is the first of a new format on Genomes Unzipped: as we acquire tests from more companies, or get data from others who have been tested, we’ll post reviews of those tests here. The aim of this series is to help potential genetic testing customers to make an informed decision about the products on the market. We’re still tweaking the format, so if you have any suggestions regarding additional analyses or areas that should be covered in more detail, let us know in the comments.
Lumigenix is a relative newcomer to the personal genomics scene: the Australian-based company launched back in March this year, offering a SNP chip-based genotyping service similar in concept to those provided by 23andMe, deCODEme and Navigenics.
The company kindly provided Genomes Unzipped with 12 free “Comprehensive” kits, which provide genotypes at over 700,000 positions in the genome, to enable us to review their product. We note that the company offers several other services, including a lower-priced “Introductory” test that covers fewer SNPs, and whole-genome sequencing for the more ambitious personal genomics enthusiast. This review should be regarded as entirely specific to the Comprehensive test.
The Lumigenix Comprehensive platform is based on Illumina’s HumanOmniExpress+, the same chip used by 23andMe. Lumigenix test for around 732,000 standard variants, along with around 2,700 specially selected ones, which appear to be mostly mitchondrial or Y chromosome markers. This is somewhat less than 23andMe (734,000 variants plus around 230,000 custom ones) and deCODEme (1.1 million variants).
Lumigenix do try to make the limitations of genetic risk projections clear in their pre-purchase information and explicitly differentiate themselves from clinical genetic testing services. They focus on the advantages of awareness and information as a step to enabling individuals to improve their own health, if they choose, and make it clear that genetic testing alone does not accomplish this. White papers describe their approach to health risk calculation (database curation and algorithm – PDFs) as well as ancestry calculations (PDF).
Information regarding the limitations of the service offered is detailed in one of the information pages, but only there is only a scant discussion of how many variants are included for different ethnic groups. Although it is possible to select from 10 different ethnicities, it is hard to find out exactly what the ramifications of this are without going to the page of each trait individually. It would be much more transparent if all DTC companies followed Pathway Genomic’s approach and provided a list of conditions with ethnicities covered. Ideally, information would also be available about the predictive capacity of the variants included for each ethnicity as well.
The testing experience
The Lumigenix testing experience will be familiar to anyone who has tested with virtually any of the major personal genomics companies: the testing kit that arrives in the mail is basically the same boxed Oragene saliva collection kit that 23andMe provides, albeit somewhat more plainly packaged. The spit kit itself is straightforward to use: spit in the tube until you reach the line, then close the hinged lid to dump preservative into the tube, and then finally replace that lid with a permanent screw-on lid for return to the company.
One small difference between the Lumigenix and 23andMe kits is that Lumigenix tucks the return envelope and biohazard bag in a separate compartment in the kit box – several of us missed this entirely and had to be guided by other (more perceptive) members.
We’ve taken a number of different approaches to estimate error rates for the various platforms that have been used by the group so far, only one of which is presented here (Luke will have more details in an upcoming post). For this analysis we took advantage of the fact that one individual in the group (Caroline) has been genotyped independently on three separate platforms (23andMe, deCODEme and now Lumigenix), and that there are around 329,000 sites that are assessed by all three companies. In this three-way comparison we looked for sites where two companies agreed on Caroline’s genotype and the third disagreed; we assume that such sites represent genotyping errors by the third company.
There is a major caveat to this analysis: sites that are shared between all three platforms are not necessarily a representative sample of the sites on any one of the chips – in fact, it’s extremely likely that these sites will be biased towards more common and better-behaved markers (i.e. this analysis likely underestimates the true error rate). Consistent with this, a less biased approach that Luke has performed on the 23andMe platform (using data from himself and his parents) suggests that the overall error rate for the 23andMe chip is substantially higher than the number above suggests, perhaps closer to 13 errors per 100,000 sites. As we don’t have family data for Lumigenix we can’t perform this analysis here.
Anyway, given that caveat, Caroline’s different genotype data-sets do tell us two interesting things about the calling approaches taken by the three companies, at least for these 329,000 shared sites. Firstly, Lumigenix are tend to be more zealous in their genotype calling, giving non-missing genotypes for 99.8% of variants (compared to 99.3% and 98.5% for 23andMe and deCODEme respectively) – in other words, if there is uncertainty at a site they are more likely to provide a “best guess” genotype rather than assign the site as “unknown”. Secondly, while error rates for all chips are extremely low, Lumigenix has a higher error rate than either of the other two: 7.4 errors per 100,000 variants, compared to 2.5 and 1.9 for 23andMe and deCODEme respectively. This is almost certainly because Lumigenix is more likely to make an educated guess at hard-to-genotype sites, so while it makes more guesses, it also gets more of thme wrong.
(We’re currently in the process of collecting the group’s raw data from the Lumigenix test, which will be posted here.)
Results are returned for 81 heath-related conditions. Lumigenix essentially use a pared-down version of the now-familiar 23andMe-style of risk communication. Traits are split into high, low, and average risk for an individual and for each condition your risk, the average risk and your relative risk, and the strength of evidence behind the association, are reported. The strength of evidence is a five point scale where 1-4 points indicates a preliminary finding and 5 points indicates an ‘established’ finding, defined as follows:
The highest evidence rating is given to established condition reports, where at least two independent studies have demonstrated an association between the same sets of SNPs. A lower evidence rating is assigned to associations built from single studies.
The user can than click through to a disease-specific page (see example below) containing information from the Mayo Clinic on disease symptoms, causes, risk factors, complications, treatments, and clinical tests for diagnosis. From this page it is also possible to retrieve the information on which SNPs were tested and the user’s genotype and odds ratio, plus references for all the SNPs included.
On the positive side, Lumigenix’s collaboration with the Mayo Clinic means that they are able to make quite a bit of medical information available about each “Heath” association. Additionally, for those who are interested in the information going into Lumigenix’s risk prediction, the details of the SNPs included are easily accessible from the “Genetics In Depth” tab for each “Health” association. However, the “Health” pages lack many of the various visual guides to interpretation that 23andMe use both to convey information and make the experience more entertaining.
Evidence supporting disease association
Most of you who have followed the personal genomics field will be aware of the fact that predicted risks can differ substantially between companies, for a number of reasons – some of which are mundane and depressingly preventable, while others are more difficult to decipher.
Comparing our Lumigenix and 23andMe results provided plenty of examples of such discrepancies: for example, Joe’s 23andMe results give him a much higher risk for prostate cancer than Lumigenix. Why the difference? Well, it’s hard to say for sure. 23andMe has more markers typed for this disease, despite Lumigenix citing the same papers. When you click on a variant in 23andMe, it tells you the precise references for that individual variant, whereas Lumigenix only gives a list of general references for the disease. That makes it very difficult to figure out exactly which references have been used to define the frequencies and effect sizes for each variant and identify the source of the discrepancies.
We believe it’s essential that personal genomics companies make the basis for their risk calculations as transparent as possible to allow customers to fully explore the evidence supporting a company’s tests. While Lumigenix does better than some companies in the industry by actually providing references at all, it could do better: we would like to see explicit links between each variant and the corresponding reference.
Completeness of tested markers
Lumigenix claim to have better coverage of common diseases, compared to 23andMe and deCODEme, based on number of SNPs genotyped. However, it’s important to note that it’s not just the number of SNPs genotyped that matters, but also which SNPs are selected. One way we can evaluate this is to use a number called the AUC (for “area under the curve”) for a set of variants. The AUC is a number between 0.5 and 1 that measures how well a test classifies individuals with and without a disease: a value of 0.5 suggests that the test has no ability to predict disease (i.e. you might as well flip a coin), and the cluser the value is to 1 the better the test is (for more information see Wray et al. 2010).
As an example, we examined the SNPs included for Crohn’s disease (CD), a well-studied common complex disease, from Lumigenix, 23andMe, and deCODEme. Luke identified all published markers with a replicated genome-wide significant association with CD (note: we did not require that the replication study be completely independent), and calculated the variance explained using the sets of variants each company uses. In each case, he used the odds ratios and frequencies from the Franke et al. (2010) meta-analysis to calculate variance explained by this variant set:
|Variant set||Number of variants||Variance explained||AUC|
|Luke’s “best set”: variant with lowest p-value in each CD region plus NOD2 variants||73||11.4%||0.76|
|Lumigenix set (includes 1 NOD2 variant)||16||5.1%||0.68|
|23andMe set (includes 3 NOD2 variants)||12||5.5%||0.69|
|deCODEme set (no NOD2 variants)||30||6.0%||0.70|
The main thing to take away from these data is that all three sets of SNPs from the DTC companies have about the same AUC, despite including different numbers of SNPs. 23andMe has a carefully hand-curated database of risk variants, and have designed their chip to optimally tag these. They have specially included the three major mutations in the important gene NOD2, which make up about 20% of the known genetic risk for Crohn’s disease. However, their set is also much smaller, so only 12/73 known variants are included. At the other extreme, deCODEme has not included any of the NOD2 mutations, but has included many more of the other variants. Lumigenix is somewhere between the two, including one of the NOD2 mutations, and a few more of the other variants. Notice that none of the companies actually do as well as they could here, as none of them include both the NOD2 mutations, and all the other variants (though 23andMe could, if they updated their database accordingly).
The user also receives results for nine “traits”: alcohol flush reaction, bitter taste perception, caffeine metabolism, earwax type, episodic memory, eye colour, freckling, fullscale IQ, and hair colour. These are simply reported as the likely outcome (e.g. “Likely brown or black hair”), and clicking through to the trait page brings up a bit of information about the trait and the specific SNPs tested. It’s not clear whether the associations reported here are subject to the same strength-of-evidence standards as the health-related results.
Lumigenix have put some effort into making their ancestry section both informative and entertaining. Their human migration map (see screenshot below) traces the historical movement of different haplogroups in an interactive way.
However, while the map is pretty, it’s also curiously disconnected from the rest of the site – rather than automatically zooming in to show your specific haplogroups, you must first browse through your ancestry information, remember your haplogroups (e.g. I2 and R1a1a), then click on the map and manually find your own haplogroup from continent-specific lists. This could certainly be made more user-friendly.
Lumigenix also provide chromosomal ancestry mapping – diagrams indicating which chunks of your genome come from which continent – which seems to work reasonably well. And then, rather surprisingly, that’s where the ancestry information ends. More detailed analyses of autosomal markers (such as a principal components analysis-based map) are not available, and as sharing between users is not currently supported, there is no facility for sharing data with relatives, or identifying new ones in the user population.
We generally believe that it is the right of personal genomics customers to have complete access to any raw genetic data generated by a company, and the responsibility of companies to provide such data in a usable form. Lumigenix does well on this front: they provide customers with the ability to freely download their complete genotype data for third-party analysis, and the company was also responsive to our suggestions regarding changes to their raw data format. The current raw data format (as of 30th November 2011) is a tab-delimited file containing fields for marker ID, chromosome number and position (relative to the build 36 reference sequence), customer genotype, and a final column indicating the two possible alleles for that marker.
Further improvements to the data format could certainly be made: for instance, a header specifying the coordinate system used, the date the file was downloaded and the field titles would be useful, and it would probably minimise confusion if the right-most column contained the customer genotype rather than the two possible alleles. However, in the absence of any industry-wide consensus on raw data format it’s hard to be too picky, and we think Lumigenix has done a comparable or better job than most of the other companies in the market in terms of providing usable data.
Unlike 23andMe, but similar to pretty much all other health-related personal genomics companies, Lumigenix provides no direct mechanism for sharing your genetic results with other users of the service. Buying a Lumigenix test will not allow you to find new relatives or socialise with people who share your genetically determined earwax consistency.
Firstly, we should be clear that there are many things that Lumigenix has done well, especially compared to companies occupying the lower end of the direct-to-consumer genetics market – this is absolutely not a cowboy operation. We particularly liked Lumigenix’s collaboration with the Mayo Clinic, which is a sensible way of obtaining reliable disease information without needing to invest heavily in in-house curation.
However, Lumigenix can’t help but suffer from comparisons to 23andMe, a company that beat it to market by over three years and now thoroughly dominates the direct-to-consumer genomics industry. The overall experience feels more-or-less like a less satisfying version of 23andMe: the common disease, trait and ancestry prediction systems are all broadly similar, but with a less sophisticated interface, and many of the more interesting components of the 23andMe service (such as carrier testing, pharmacogenomic markers, genome sharing, family inheritance patterns, Relative Finder) feel quite conspicuous in their absence. Then there’s the price: the Lumigenix test currently retails for USD$479 while 23andMe is USD$399 (without subscription).
We’re all for competition in the personal genomics market: new companies in the space should bring new ideas and help drive prices down. But when the new companies offer services that are basically less useful versions of those offered by the market leader, at a higher price, it’s hard not to be disappointed.
Still, nothing stands still in personal genomics – and Lumigenix has assembled a reasonably solid base from which to branch out into more innovative directions. We hope it chooses to take paths less travelled in its future products.