1 Environment

1.1 System

1.1.1 System Platform

Platform: x86-64   
Version: 12.04 LTS
OS: Ubuntu (64-bit)

1.1.2 Hardware

CPU(s): 16
Threads: 32
RAM: 128G
Hard disk: <2T


1.2 Qiime

1.2.1 Qiime version

Version: 1.9.1

1.2.2 System information

          Platform: linux2
Python/GCC version: 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
 Python executable: /home/qiime/miniconda3/bin/python

1.2.3 QIIME default reference information

                QIIME library version:      1.9.1
                 QIIME script version:      1.9.1
      qiime-default-reference version:      0.1.2
                        NumPy version:      1.9.2
                        SciPy version:      0.15.1
                       pandas version:      0.16.1
                   matplotlib version:      1.4.3
                  biom-format version:      2.1.4
                         h5py version:      2.4.0 (HDF5 version: 1.8.4)
                         qcli version:      0.1.1
                         pyqi version:      0.3.2
                   scikit-bio version:      0.2.3
                       PyNAST version:      1.2.2
                      Emperor version:      0.9.51
                      burrito version:      0.9.1
             burrito-fillings version:      0.1.1
                    sortmerna version:      SortMeRNA version 2.0, 29/11/2014
                    sumaclust version:      SUMACLUST Version 1.0.00
                        swarm version:      Swarm 1.2.19 [May 26 2015 13:50:14]
                                gdata:      Installed.
RDP Classifier version (if installed):      rdp_classifier-2.2.jar                  

1.2.4 QIIME config values (print_qiime_config.py -tf)

For definitions of these settings and to learn how to configure QIIME, see here:
http://qiime.org/install/qiime_config.html
http://qiime.org/tutorials/parallel_qiime.html

QIIME config values

For definitions of these settings and to learn how to configure QIIME, see here:
http://qiime.org/install/qiime_config.html
http://qiime.org/tutorials/parallel_qiime.html

                      blastmat_dir:      None
       pick_otus_reference_seqs_fp:      /usr/local/lib/python2.7/sitepackages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     python_exe_fp:      python
                          sc_queue:      all.q
       topiaryexplorer_project_dir:      None
      pynast_template_alignment_fp:      /usr/local/data/core_set_aligned.fasta.imputed
                   cluster_jobs_fp:      None
 pynast_template_alignment_blastdb:      None
 assign_taxonomy_reference_seqs_fp:      /usr/local/lib/python2.7/sitepackages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                      torque_queue:      friendlyq
               qiime_test_data_dir:      None
    template_alignment_lanemask_fp:      /usr/local/data/lanemask_in_1s_and_0s.txt
                     jobs_to_start:      1
                        slurm_time:      None
                 cloud_environment:      False
                 qiime_scripts_dir:      /usr/local/bin
             denoiser_min_per_core:      50
                       working_dir:      None
 assign_taxonomy_id_to_taxonomy_fp:      /usr/local/lib/python2.7/sitepackages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                          temp_dir:      /tmp/
                      slurm_memory:      None
                       slurm_queue:      None
                       blastall_fp:      blastall
                  seconds_to_sleep:      2


2. Data

2.1 original sequences

format of sequences: fastq.gz (Illumina Miseq)
Number of fastq.gz files: 30
Availability of data and materials: ENA accession no. ERP109571 (Run Accessions: ERX3185111-ERX3185140)

3. Scripts

3.1 qiime scripts

<Merging paired-end illumina Miseq sequence reads>

$ join_paired_ends.py -f Bek1_1.fastq.gz -r Bek1_2.fastq.gz -o join_output_Bek1/
$ join_paired_ends.py -f Bek2_1.fastq.gz -r Bek2_2.fastq.gz -o join_output_Bek2/
$ join_paired_ends.py -f Bek3_1.fastq.gz -r Bek3_2.fastq.gz -o join_output_Bek3/
$ join_paired_ends.py -f Bek4_1.fastq.gz -r Bek4_2.fastq.gz -o join_output_Bek4/
$ join_paired_ends.py -f Bek5_1.fastq.gz -r Bek5_2.fastq.gz -o join_output_Bek5/
$ join_paired_ends.py -f Comm1_1.fastq.gz -r Comm1_2.fastq.gz -o join_output_Comm1/
$ join_paired_ends.py -f Comm2_1.fastq.gz -r Comm2_2.fastq.gz -o join_output_Comm2/
$ join_paired_ends.py -f Comm3_1.fastq.gz -r Comm3_2.fastq.gz -o join_output_Comm3/
$ join_paired_ends.py -f Comm4_1.fastq.gz -r Comm4_2.fastq.gz -o join_output_Comm4/
$ join_paired_ends.py -f Comm5_1.fastq.gz -r Comm5_2.fastq.gz -o join_output_Comm5/
$ join_paired_ends.py -f Cutt1_1.fastq.gz -r Cutt1_2.fastq.gz -o join_output_Cutt1/
$ join_paired_ends.py -f Cutt2_1.fastq.gz -r Cutt2_2.fastq.gz -o join_output_Cutt2/
$ join_paired_ends.py -f Cutt3_1.fastq.gz -r Cutt3_2.fastq.gz -o join_output_Cutt3/
$ join_paired_ends.py -f Cutt4_1.fastq.gz -r Cutt4_2.fastq.gz -o join_output_Cutt4/
$ join_paired_ends.py -f Cutt5_1.fastq.gz -r Cutt5_2.fastq.gz -o join_output_Cutt5/
$ join_paired_ends.py -f Ins1_1.fastq.gz -r Ins1_2.fastq.gz -o join_output_Ins1/
$ join_paired_ends.py -f Ins2_1.fastq.gz -r Ins2_2.fastq.gz -o join_output_Ins2/
$ join_paired_ends.py -f Ins3_1.fastq.gz -r Ins3_2.fastq.gz -o join_output_Ins3/
$ join_paired_ends.py -f Ins4_1.fastq.gz -r Ins4_2.fastq.gz -o join_output_Ins4/
$ join_paired_ends.py -f Ins5_1.fastq.gz -r Ins5_2.fastq.gz -o join_output_Ins5/
$ join_paired_ends.py -f Jap1_1.fastq.gz -r Jap1_2.fastq.gz -o join_output_Jap1/
$ join_paired_ends.py -f Jap2_1.fastq.gz -r Jap2_2.fastq.gz -o join_output_Jap2/
$ join_paired_ends.py -f Jap3_1.fastq.gz -r Jap3_2.fastq.gz -o join_output_Jap3/
$ join_paired_ends.py -f Jap4_1.fastq.gz -r Jap4_2.fastq.gz -o join_output_Jap4/
$ join_paired_ends.py -f Jap5_1.fastq.gz -r Jap5_2.fastq.gz -o join_output_Jap5/
$ join_paired_ends.py -f Whip1_1.fastq.gz -r Whip1_2.fastq.gz -o join_output_Whip1/
$ join_paired_ends.py -f Whip2_1.fastq.gz -r Whip2_2.fastq.gz -o join_output_Whip2/
$ join_paired_ends.py -f Whip3_1.fastq.gz -r Whip3_2.fastq.gz -o join_output_Whip3/
$ join_paired_ends.py -f Whip4_1.fastq.gz -r Whip4_2.fastq.gz -o join_output_Whip4/
$ join_paired_ends.py -f Whip5_1.fastq.gz -r Whip5_2.fastq.gz -o join_output_Whip5/

<Demultiplexing (Split libraries according to barcodes specified in mapping file)>

$ split_libraries_fastq.py -i join_output_Bek1/fastqjoin.join.fastq -o OUTPUT/Bek1/ --sample_ids Bek1 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Bek2/fastqjoin.join.fastq -o OUTPUT/Bek2/ --sample_ids Bek2 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Bek3/fastqjoin.join.fastq -o OUTPUT/Bek3/ --sample_ids Bek3 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Bek4/fastqjoin.join.fastq -o OUTPUT/Bek4/ --sample_ids Bek4 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Bek5/fastqjoin.join.fastq -o OUTPUT/Bek5/ --sample_ids Bek5 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Comm1/fastqjoin.join.fastq -o OUTPUT/Comm1/ --sample_ids Comm1 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Comm2/fastqjoin.join.fastq -o OUTPUT/Comm2/ --sample_ids Comm2 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Comm3/fastqjoin.join.fastq -o OUTPUT/Comm3/ --sample_ids Comm3 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Comm4/fastqjoin.join.fastq -o OUTPUT/Comm4/ --sample_ids Comm4 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Comm5/fastqjoin.join.fastq -o OUTPUT/Comm5/ --sample_ids Comm5 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Cutt1/fastqjoin.join.fastq -o OUTPUT/Cutt1/ --sample_ids Cutt1 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Cutt2/fastqjoin.join.fastq -o OUTPUT/Cutt2/ --sample_ids Cutt2 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Cutt3/fastqjoin.join.fastq -o OUTPUT/Cutt3/ --sample_ids Cutt3 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Cutt4/fastqjoin.join.fastq -o OUTPUT/Cutt4/ --sample_ids Cutt4 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Cutt5/fastqjoin.join.fastq -o OUTPUT/Cutt5/ --sample_ids Cutt5 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Ins1/fastqjoin.join.fastq -o OUTPUT/Ins1/ --sample_ids Ins1 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Ins2/fastqjoin.join.fastq -o OUTPUT/Ins2/ --sample_ids Ins2 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Ins3/fastqjoin.join.fastq -o OUTPUT/Ins3/ --sample_ids Ins3 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Ins4/fastqjoin.join.fastq -o OUTPUT/Ins4/ --sample_ids Ins4 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Ins5/fastqjoin.join.fastq -o OUTPUT/Ins5/ --sample_ids Ins5 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Jap1/fastqjoin.join.fastq -o OUTPUT/Jap1/ --sample_ids Jap1 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Jap2/fastqjoin.join.fastq -o OUTPUT/Jap2/ --sample_ids Jap2 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Jap3/fastqjoin.join.fastq -o OUTPUT/Jap3/ --sample_ids Jap3 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Jap4/fastqjoin.join.fastq -o OUTPUT/Jap4/ --sample_ids Jap4 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Jap5/fastqjoin.join.fastq -o OUTPUT/Jap5/ --sample_ids Jap5 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Whip1/fastqjoin.join.fastq -o OUTPUT/Whip1/ --sample_ids Whip1 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Whip2/fastqjoin.join.fastq -o OUTPUT/Whip2/ --sample_ids Whip2 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Whip3/fastqjoin.join.fastq -o OUTPUT/Whip3/ --sample_ids Whip3 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Whip4/fastqjoin.join.fastq -o OUTPUT/Whip4/ --sample_ids Whip4 -q 19 -n 0 --barcoded_type 'not-barcoded'
$ split_libraries_fastq.py -i join_output_Whip5/fastqjoin.join.fastq -o OUTPUT/Whip5/ --sample_ids Whip5 -q 19 -n 0 --barcoded_type 'not-barcoded'

<Merging all the demultiplexed files>
$ cat */*.fna > merged.demultiflexed.fna

<Chimeric sequence removal (implementation of usearch61)>

$ chmod +x usearch7.0/usearch7.0.1090_i86linux32 (Downloaded from https://www.drive5.com/usearch/download.html)
$ usearch7.0/usearch7.0.1090_i86linux32 -uchime_ref merged.demultiplexed.fna -db qiime.files/97_otus.fasta -uchimeout usearch7.0/OUTPUT.uchime -nonchimeras chimera.filtered.merged.demultiplexed.fna -strand plus -threads 32

<Open-reference OTU clustering>

$ pick_otus.py -i chimera.filtered.merged.demultiplexed.fna -r qiime.files/97_otus.fasta -o pick_otus/ --denovo_otu_id_prefix denovo_ -m uclust_ref --threads 32 

<Pick representative set of sequences>

$ pick_rep_set.py -i pick_otus/chimera.filtered.merged.demultiplexed_otus.txt -f chimera.filtered.merged.demultiplexed.fna -o rep_seqs_cephalopods.fna

<Assign taxonomy using RDP classifier>

$ assign_taxonomy.py -i rep_seqs_cephalopods.fasta -m rdp -r 97_otus.fasta -t 97_otus_taxonomy.txt -o taxonomy60/ -c 0.6

<Make phylogeny>

$ align_seqs.py -i rep_seqs_cephalopods.fasta -t qiime_files/core_set_aligned.fasta.imputed -o pynast_align/ 
$ filter_alignment.py -i pynast_align/rep_seqs_cephalopods_aligned.fasta -m qiime_files/lanemask_in_1s_and_0s -o filtered_align/
$ make_phylogeny.py -i filtered_align/rep_set_cephalopods_aligned.fasta -o rep_seqs_cephalopods_tree.tre

<Make OTU table>

$ make_otu_table.py -i pick_otus/chimera_filtered_otus.txt -t assign_taxa/rep_seqs_tax_assignments.txt -o otu_table.biom -e pynast_aligned/failures.fasta

<Noise OTU filtering>

$ filter_otus_from_otu_table.py -i otu_table.biom -o otu_table.filtered.biom -n 15

<Perform rarefaction on an otu table>

$ single_rarefaction.py -i otu_table.filtered.biom -o otu_table.filtered.rarefied6000.biom -d 6000

<Calculate alpha-diversity> 

$ alpha_diversity.py -i otu_table.filtered.rarefied6000.biom -o alpha_div/ -m PD_whole_tree,observed_species,chao1,shannon,simpson_e -m cephalopods_mapping.txt -t rep_seqs_cephalopods_tree.tre

<Calculate beta-diversity>

$ beta_diversity.py -i otu_table.filtered.rarefied6000.biom -m unweighted_unifrac,weighted_unifrac,bray_curtis,binary_jaccard -o beta_div/ -t rep_seqs_cephalopods_tree.tre

<Generate PCoA plots>

$ principal_coordinates.py -i beta_div/ -o beta_coords/
$ make_2d_plots.py -i beta_coords/ -o 2d_plots/ -m cephalpods_mapping.txt

<Performing taxonomy summaries and plots>

$ summarize_taxa_through_plots.py -i otu_table.filtered.rarefied.biom -o Tax_composition_cephalopods -m cephalopods_mapping.txt

3.2 picrust scripts

<Picking closed reference OTUs with QIIME for integration to metagenome prediction pipeline, PICRUSt>

$ pick_closed_reference_otus.py -i merged_nonchimeras_cephalopods.fna -r qiime.files/97_otus.fasta -t 97_otus_taxonomy.txt -o pick_closed_otus/ --assign_taxonomy (quality-filtered and chimeric sequence removed input fasta file was brought from precedent 16S rRNA gene sequence analysis)

<Normalize OTU Table>

$ normalized_by_copy_number.py -i pick_closed_otus/otu_table.biom -o picrust/normalized_otus_table.biom