Instructions and supporting data for the QIIME/IPython/StarCluster demo at the 2012 NIH Cloud Computing the Microbiome workshop and our corresponding paper in the ISME Journal.
The analysis made use of the IPython Notebook, QIIME, StarCluster, PyCogent, and PrimerProspector. All of these tools are pre-installed in the ami-9f69c1f6 public Amazon EC2 instance, which was used in this study.
Supporting Files
The IPython notebooks supporting this study can be viewed here and are available here in PDF format:
* Note that the Timing notebook is for reference as related to the paper only - it will not be directly reproducible on re-runs of the above notebooks as it relies on the semi-manual creation of the tasks.log file. The tasks.log file used to generate the original timing data is available for download here.
The Greengenes reference OTU collection used in this study is available for download here.
The IPython notebook files (.ipynb) are available for download here.
The tree metadata mapping file used in generating the coloring categories in the 3D PCoA plot is available here.
The paper for this analysis, "Collaborative cloud-enabled tools allow rapid, reproducible biological insights", is available here.
Reproducing the analysis
Four m2.4xlarge instances were booted using StarCluster to create a 32 core cluster with approximately 280GB of RAM (70GB per 8 core instance). This was used for the full analysis (a more complete analysis then was done during the workshop, where the workshop analysis was optimized to run quickly). To support the large quantity of data that is generated during the analysis, you should create an EBS volume which will be attached to the running instance. A 20 GB volume will be sufficient. The volume used for running these notebooks is available as snap-75eb8005.
To reproduce the analyses presented in this paper you should install StarCluster locally, and configure it according to the instructions on the StarCluster website. You can then add the following to your ~/.starcluster/config file:
[plugin ipcluster]
setup_class = starcluster.plugins.ipcluster.IPCluster
enable_notebook = true
# If you leave notebook_passwd out, a random password
# will be generated instead.
notebook_passwd = YOUR-PASSWORD
[cluster qiime-ipython]
node_image_id = ami-9f69c1f6
cluster_user = ubuntu
keyname = YOUR-KEY
cluster_size = 4
node_instance_type = m2.4xlarge
plugins = ipcluster
volumes = qiime-ipython-data
[volume qiime-ipython-data]
VOLUME_ID = YOUR-VOLUME-ID
MOUNT_PATH = /home/ubuntu/data
You can then boot this cluster by running:
starcluster start -c qiime-ipython mycluster
You will be presented with the URL of your IPython notebook. You can upload our .ipynb files to re-run the analysis directly on your own hardware, or tweak it to perform your own analysis.
Citing this work
You can cite this work as follows:
Collaborative cloud-enabled tools allow rapid, reproducible biological insights.
Benjamin Ragan-Kelley, William Anton Walters, Daniel McDonald, Justin Riley, Brian E. Granger, Antonio Gonzalez, Rob Knight, Fernando Perez and J. Gregory Caporaso.
ISME Journal, in press (2012).