| News and Announcements » |
This tutorial explains the use of several different mapping file category comparison techniques that are available in compare_categories.py. A majority of the currently available comparison techniques are based on the ANOVA family of statistical methods, which is a statistical method wherein observed variance in a particular variable is partitioned into components attributable to different sources of variation. Most of the methods available in compare_categories.py determine whether the grouping of samples by a given category is statistically significant. Many of these methods are nonparametric, which means they use permutations of the data to determine the p-value, or statistical significance.
Note: Many of these methods have been evaluated on empirical and simulated data sets to verify their usefulness on microbial community data. It has been found that these methods seem to suffer from low specificity (i.e. they always seem to detect significance of grouping even when there doesn’t appear to be any). Even though the methods do not assume normality of the data due to their nonparametric nature, many of the methods assume that each group of samples has equal variance. Thus, it may be possible that the low specificity of the methods is due to violations of this equal variance assumption. Therefore, the methods may be detecting significant differences in the variability of sample groups instead of differences in center (i.e. mean). Another potential explanation of this problem is that as the number of samples increases, the more likely you are to see significant p-values. Thus, it is important to look at the effect size of the test as well as the statistical significance (e.g. look at ANOSIM’s R value and its p-value). These methods need to be tested more thoroughly to verify these claims, so please be cautious with your use of these methods until then.
You can obtain the files used in this tutorial here. The files are taken from a study (Fierer et al., 2010) where samples were collected from the fingertips of three individuals and the keys of their keyboards. Researchers found that an individual’s fingertip samples clustered very closely with his or her corresponding keyboard samples. These results have potential application in the field of forensic science. Throughout this tutorial, we will explore various ways of determining whether the clustering of fingertip and keyboard samples is statistically significant by analyzing the groups of samples.
Depending on which statistical method you run, the output file(s) generated by this script will vary. They will all be placed in the directory specified by the required -o option. Most of the output files will be tab-separated text files containing information about the test that was performed and its results. These can easily be viewed in a spreadsheet program such as Excel.
adonis is a nonparametric statistical method that takes a QIIME distance matrix file such as a UniFrac distance matrix, a mapping file, and a category in the mapping file to determine sample grouping from. It computes an R2 value (effect size) which shows the percentage of variation explained by the supplied mapping file category, as well as a p-value to determine the statistical significance.
adonis creates a set by first identifying the relevant centroids of the data and then calculating the squared deviations from these points. After that, significance tests are performed using F-tests based on sequential sums of squares from permutations of the raw data.
Suppose we wanted to determine whether the samples grouped by host individual (i.e. person) are statistically significant. Run the following command:
compare_categories.py --method adonis -i unweighted_unifrac_dm.txt -m map.txt -c HOST_SUBJECT_ID -o adonis_out -n 999
When broken down the command syntax means:
compare_categories.py - This is the script being run.
--method adonis - This is the method to perform, in this case it's 'adonis'.
-i unweighted_unifrac_dm.txt - This is the unweighted UniFrac distance matrix being passed in for this statistical test.
-m map.txt - This is the metadata mapping file associated with this data set.
-c HOST_SUBJECT_ID - This specifies the category in the metadata mapping file to analyze. In this case it's 'HOST_SUBJECT_ID'.
-o adonis_out - This specifies the output directory to place the results of this method in. In this case it's 'adonis_out'.
-n 999 - This specifies the number of permutations that will be used to calculate the p-value.
This command will create a new output directory named adonis_out, which will contain a single text file called adonis_results.txt. Open up adonis_results.txt to see the results of the test:
Call:
adonis(formula = as.dist(qiime.data$distmat) ~ qiime.data$map[[opts$category]], permutations = opts$num_permutations)
Terms added sequentially (first to last)
Df SumsOfSqs MeanSqs F.Model R2 Pr(>F)
qiime.data$map[[opts$category]] 10 7.1804 0.71804 5.1788 0.33243 0.001
Residuals 104 14.4196 0.13865 0.66757
Total 114 21.6000 1.00000
qiime.data$map[[opts$category]] ***
Residuals
Total
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
The p-value of 0.001 indicates that at an alpha of 0.05, the grouping of samples by individual is statistically significant. The R2 value indicates that approximately 33% of the variation in distances is explained by this grouping. These results confirm the results obtained by the original researchers of this study.
ANOSIM is a method that tests whether two or more groups of samples are significantly different (similar to adonis, above). You can specify a category in the metadata mapping file to separate samples into groups and then test whether there are significant differences between those groups. For example, you might test whether Control samples are significantly different from Fast samples. Since ANOSIM is nonparametric, statistical significance is determined through permutations.
Note: ANOSIM only works with a categorical variable that is used to do the grouping. Mantel is recommended for continuous variables.
The R statistic that is calculated by ANOSIM is determined by the following formula:
R = (rb-rw)/(N(N-1)/4)
where rb is the mean rank of all distances between groups and rw is the mean rank of all distances within groups. An R value near +1 means that there is dissimilarity between the groups, while an R value near 0 indicates no significant dissimilarity between the groups (http://folk.uio.no/ohammer/past/multivar.html).
We will attempt to answer the same question posed in the adonis section (above). Run the following command:
compare_categories.py --method anosim -i unweighted_unifrac_dm.txt -m map.txt -c HOST_SUBJECT_ID -o anosim_out -n 999
When broken down the command syntax means:
compare_categories.py - This is the script being run.
--method anosim - This is the method to perform, in this case it's 'anosim'.
-i unweighted_unifrac_dm.txt - This is the unweighted UniFrac distance matrix being passed in for this statistical test.
-m map.txt - This is the metadata mapping file associated with this data set.
-c HOST_SUBJECT_ID - This specifies the category in the metadata mapping file to analyze. In this case it's 'HOST_SUBJECT_ID'.
-o anosim_out - This specifies the output directory to place the results of this method in. In this case it's 'anosim_out'.
-n 999 - This specifies the number of permutations that will be used to calculate the p-value.
This command will create a new output directory named anosim_out, which will contain a single text file called anosim_results.txt. Open up anosim_results.txt to see the results of the test:
Method Name R-value P-value
ANOSIM 0.794026410205 0.001
The p-value of 0.001 indicates that at an alpha of 0.05, the grouping of samples by individual is statistically significant. The R value of 0.794 is fairly close to +1, indicating that the grouping of samples based on individual is strong. These results confirm the results obtained by the original researchers of this study, as well as our results when we applied adonis to this dataset (above).
Fierer, N. et al. Forensic identification using skin bacterial communities. Proc. Natl. Acad. Sci. USA. 107: 6477-6481 (2010).
