README FILE FOR SNIPER
May 2011

To use Sniper.py, make sure you have Python v2.5 or greater installed.
http://python.org/

Testing the demo:

## 1) If you already have a preexisting read map file (SAM-formatted)
##    follow this example to perform genotyping only:

Issue the following command using the preexisting map file (mydemo.sam):

> python sniper.py -r REFERENCE.fasta -i ./ --exO --index -id mydemo -s

This will organize the read map by creating unique and degenerate 
partitions (--exO), index these files (--index), then perform 
genotyping (-s) on sample mydemo (--id). Note, if your map file is 
already sorted, include --nosort to prevent resorting.

This will create new map files in the sniper_demo directory as well as a 
sniper_results directory that contains the snp calls 
(inside results-L0.67,E35,PESE,ALL/snps/).

One file is the "mother" which contains genotyping calls for all nucleotide 
sites. The other file contains significant calls only.

## 2) To perform mapping using Sniper via Bowtie:

First you must have Bowtie installed
http://bowtie-bio.sourceforge.net/index.shtml

Then, generate the reference index:

> bowtie-build REFERENCE.fasta REFERENCE

Next, call Sniper:

> python sniper.py -r REFERENCE.fasta -m demo_map.txt --qual33 -z

A new directory called sniper_results will be created within the 
sniper_demo/ directory. The --qual33 flag tells Bowtie what kind of quality 
values to expect (not needed for newer Illumina sequencing). As above, 
SNP calls are located in the results-L0.67,E35,PESE,ALL/snps/ directory.

Both approaches (1) and (2) should yield identical genotype files: 
- mydemo.mother.txt
- mydemo.aQ0.05,50.txt

## 3) Adjusting significance cutoffs

By default, Sniper uses a Bonferroni-style confidence cutoff for significant 
SNPs that accounts for multiple testing. The two parameters are the stringency 
alpha and read length. To adjust these, use the --fwer alpha,read_length 
option. For example, a 0.01 model confidence using 36 bp reads:

> python sniper.py -r REFERENCE.fasta -m demo_map.txt --qual33 -s 
   --fwer 0.01,36

## 4) Comparison to true SNP sites

Since the above example is based on simulated data, you can compare the 
predictions to the true SNP sites in the file True_SNP_sites.txt.

## 5) Compiling C library for performance improvement

To use a C implementation of the likelihood computation, you must compile the snipercore.c file as follows:

> python setup.py build

Then move the build/libXX/snipercore.so file so it is in your PYTHONPATH, or add this directory to your environment. For example in bash:

'export PYTHONPATH="$PYTHONPATH:/path/to/sniper/build/lib.macosx-10.6-universal-2.6/snipercore.so'

Sniper will report whether it found the module at runtime.

## Sniper.py output format

EX line:
REFERENCE 182 T CT 160 35 3 3 0.0 Y A:0.0,T:0.57,C:0.43,G:0.0

Column definitions:
1. Chromosome
2. Position in reference genome (0-offset)
3. Reference genotype
4. Predicted maximum a posteriori genotype
5. Confidence in genotype call (posterior probability given data)
6. Total read depth (number of reads overlapping site)
7. Number of non-unique reads overlapping site
8. Total number of alignments resulting from non-unique reads
9. Number of alignments per read (harmonic mean)
10. Significant (Y/N)
11. Variant allele frequency distribution


## Help instructions

Full help menu: sniper.py --help/-h
Description and requirements: sniper.py --version/-v
Web page: http://kim.bio.upenn.edu/software/sniper.shtml
Latest version: http://kim.bio.upenn.edu/software/sniper/sniper_pkg.zip

Usage: sniper.py -r ref.fa (--map mapfile.txt|--id samplename) [options]

Genotyping with an existing read map file:
   The read map file should be named <mapdir>/samplename.map, where samplename
   is provided in SampleID field of the map file (see below).
   
   If the read map file contains unique reads only (if already sorted, 
   include --nosort to avoid resorting):
   > sniper.py -r ref.fa -i <mapdir> --link --index --id samplename -s
   
   If the read map file contains both unique and non-unique reads, use 
   --exO to partition the file into unique and non-unique portions:
   > sniper.py -r ref.fa -i <mapdir> --exO --index --id samplename -s

Read mapping and genotyping:
   > sniper.py -r ref.fa -m mapfile.txt -z
   
   This will align raw NGS reads, organize and index reads, perform 
   genotyping, and provide diagnostics and new assemblies for all 
   samples listed in the map file. Results will be stored in a new 
   directory 'sniper_results' in your current working directory.
   
   Note, -z is equivalent to writing 
   "-a -O --index -s -d --saveass -Q 40 --fwer 0.05,50"

Use sniper.py -h for a complete description of arguments.


Standard arguments
==================
*  -m <STR>:       map file (see below for format)
*  -r <STR>:       reference genome (required, fasta format)
*  -i <STR>:       directory to find pre-existing map file(s)
*  -o <STR>:       directory in which to save results 
                   (default: current working directory ./)
* --name <STR>:    name of the output/results directory 
                   (default: sniper_results)
*  -z:             perform/redo all procedures (equivalent to 
                   -a -O --index -s -d --saveass -Q 40 --fwer 0.05,50)
*  -a/-ra:         (re)align reads (generate sam-format read maps with bowtie)
*  -O/-rO:         (re)organize read maps into 4 files: 
                     SE.unique, PE.unique, SE.degenerate, PE.degenerate
                   then build an index for each file
* --splitmap       Separate single-end and paired-end reads into different map files
*  --link          if you only have a unique map, this links to that file 
                   directly rather than organizing new files
*  -exO:           organize a pre-existing read map into the 4 files above 
                   (Default: sampleID.sam, where sampleID is in mapfile)
*  -s/-rs:         perform/redo SNP calling (and significance filtering)
* --sig:           perform significance filtering on existing genotypes
* --resig:         redo significance filtering on existing genotypes
* --diag:          generate diagnostics file summarizing SNP calls
* --saveass:       create a new fasta file of the reference genome modified 
                   by significant SNP calls (IUPAC format). NB, by default
                   -Q 40 is used, but you can return multiple assemblies: 
                   Ex: --saveassembly -Q 20,30,40,100

Key parameters
==============
EX: sniper.py -m <map_file.txt> -r <ref_genome.fasta> -z -k 2 
    -d 10 -e 30 -l 0.5 --prior resequence --theta 0.01

Arguments:
* -k <1,2,3>:      maximum number of alignment mismatches (default: 2)
* -d <INT>:        maximum number of alignments per read (default: 50)
* -e <INT>:        global sequencing error rate (specified as phred value; 
                   E.g. Q=30 == P<0.001) (default: 35)
* -l <FLOAT>:      reweight the likelihood model towards read-specific or 
                   global binomial probability (default: 0.67 towards binomial)
* --prior <STR>:   prior probability distribution (default: resequence)
                   Options: 
                   resequence:     h0:1-theta-theta^2, t1:theta/2, h2:theta/2, t2:theta^2
                   resequence-het: h0:1-theta-theta^2, t1:theta, h2:theta^2/2, t2:theta^2/2
                   maq:            h0:(1-theta-theta^2)/2, t1:theta, h2:(1-theta-theta^2)/2, t2:theta^2
                   uniform:        h0:0.25, t1:0.25, h2:0.25, t2:0.25
                   haploid:        h0:1-theta, t1:theta
* --theta <FLOAT>: expected divergence (heterozygous and homozygous) from 
                   reference genome (default: 0.001)

Read map indexing
=================
EX: sniper.py -m <map_file.txt> -r <reference_genome.fasta> --index

Arguments:
* --index:         sorts and indexes the unique SE and PE map files
* --reindex:       replaces an existing index
* --(re)indexuni:  index unique maps only
* --(re)indexdeg:  index degenerate maps only
* --nosort:        index only, assuming map files are already sorted

MEMORY / RUNTIME / SCHEDULING options
=====================================
EX: sniper.py -m <map_file.txt> -r <reference_genome.fasta> -s -x 5
For each sampleID, break the genome into 5 pieces, then launch and maintain 
up to 5 parallel processes on your local machine. The total number of jobs 
equals 5 times the number of sampleIDs in the map file. If --sge is added 
schedule jobs across CPUs using Sun Grid Engine.

EX: sniper.py -m <map_file.txt> -r <reference_genome.fasta> -s -x 15 
    --rest <restriction_file.txt>

This will launch and maintain up to 15 parallel processes, where the total 
number of jobs is equal to the number of sampleIDs in the map file times the 
rows in <restriction_file.txt>.
* --stream:     stream the degenerate index (Default: NA)
* --buffer <INT>: sets the buffer for streaming to approx. <INT> MB
*  -x <INT>:    distribute jobs over <INT> processes on 1 machine (multi-core)
* --sge:        distribute jobs using Sun Grid Engine. If -x > 1 launch 
                <INT> jobs per sampleID, otherwise launch one job for each 
                different sample to be processed
* --lip <INT>:  partition map into <INT> chunks to lower memory requirements
                (NB, requires <INT> scans of the map file, increasing runtime)
* --sgemem:     Required memory to launch process on SGE (default: 16G)
* --rest:       Specify a region or file containing a list of regions over 
                which to perform genotyping (default: none)
                EX: "--rest chr1"; genotype the entirety of chr1 only
                EX: "--rest chr1.chr5"; genotype chrom1 and chr5 only
                EX: "--rest chr1,0,1000"; genotype positions 0 to 1000 
                    of chr1, inclusive
                EX: "--rest chr1,0,1000,-"; genotype all of chr1 excluding 
                    positions 0 to 1000
                A restriction file should be a tab-delimited list of one or 
                more regions similarly described, replacing commas with tabs.
                Execute this shell command to generate one:
                echo -e "chrom1\\nchrom2\\nchrom3" > myrest.txt, or
                echo -e "chrom1\\t0\\t1000\\nchrom5\\t500\\t50000" > myrest.txt
* --ids <STR[,STR,...,STR>: restrict genotyping to a subset of SampleIDs

Other notable arguments
=======================
* --mindepth <INT>: minimum number of reads to evaluate a nucleotide locus
* --maxdepth <INT>: maximum number of reads to evaluate a nucleotide locus.
                    If > <INT> reads only the first <INT> will be used.
* --fasta:       read files do not contain quality scores
* --qual33:      quality scores follow a phred 33 scale instead of default 
                 phred 64 scale
* --peonly:      only use paired end reads for SNP identification
* --all:         use all reads for mapping (based on --policy bestm with 
                 maximum of d alignments)
* --uniq:        only use uniquely mapping reads for SNP identification. Also
                 if provided with -a for mapping
                 only return an alignment if it is unique in the reference 
                 genome given specified maximum mismatches
                 bowtie arguments: '-v mismatches -a -m 1 --best
* --best:        generate a best-guess read map, returning an alignment if it
                 is the one with the fewest mismatches
                 given specified maximum mismatches
                 bowtie arguments: '-v mismatches -k 1 --best'
* --bestno:      similar to --best but discard reads which have multiple 
                 equally best alignments
* --mapqual      use read quality values during mapping (bowtie -n mode with
                 default parameters)
* --colorspace:  input files are in ABI SOLiD colorspace format 
                 (suffix .csfasta and .qual)
                 NB, currently only works with single-end reads
* --ins <FLOAT>: estimate insert size range for paired-end reads as the 
                 <FLOAT> percentile of the empirical distribution 
                 (default: 0.99) (see below for more details)
* --globalerror: estimate expected per-nucleotide sequencing error rate from 
                 your read map
* --cstheta <FLOAT>: Haploid proportion of divergence between sample and 
                     reference genomes (default: 3/2 theta)
* -Q <INT[,INT,...,INT]>: Comma-delimited list of phred-quality scores for
                          which to compute significance (Default: 40)
* -fwer <FLOAT,INT>: Filter significant SNPs using a family-wise error rate 
                     of <FLOAT> and an average read length of <INT>. This 
                     corrects the posterior probability for each genotyped 
                     locus for multiple testing of all loci that utilize the 
                     same set of NGS alignments. The Q stringency required for
                     significance is -10 log( <FLOAT> / (2 <INT> dbar) ), 
                     where <FLOAT> is the error rate (e.g. 0.01), <INT> is the
                     read length (e.g. 50), and dbar is the estimated mean 
                     number of total alignments for all reads that overlap the
                     current locus of interest (Default: 0.05,50).
* --nominal:     Instead of waiting until the whole genome is processed before
                 returning significant SNPs, this updates a file 
                 sampleID.nominal.txt with potential significant SNPs
                 concurrent with processing.
* --suffix <STR>:    Change default map file suffix to FILENAME.<STR> [Default: sam]

Bowtie mapping options
======================
* --bowtie/-btd: Specify the base directory of your bowtie installation 
                 (e.g. --btd /usr/bin/bowtie-0.12.5) (default: '')
* --keepbowtie/--kb: Do not delete the original bowtie SAM file following 
                     read organization
* --btx <INT>:   number of processors used for mapping
* --policy:      Specify Bowtie alignment policy (default: bestk)
                 Options:
                 * 'basic': 'bowtie -n <mismatches> -m 1 (Maq-like)
                 * 'all': 'bowtie -v <mismatches> -a (end-to-k all alignments)
                 * 'bestk': -v <mismatches> -k <max num alignments> --best
                 * 'bestm': -v <mismatches> -a -m <max num alignments> --best
* --kpe: Permit a maximum number of alignment mismatches for paired-end reads
* --kse: Permit a maximum number of alignment mismatches for single-end reads
* --trimreads n5,n3: For alignment, consider read substring from 1+n5...|r|-n3
* --realign/--ra: perform new Bowtie alignment, replacing existing alignments

Map file format
===============
The map file is a standard tab-delimited text file indicating how each raw 
sequence reads file should be processed by sniper. The format is organized as 
one row per sequenced lane and requires 7 columns of information row, as 
follows:

#!Run Lane SampleID       Alias RefGenome           PairedEnd Path
  1   1    myfirstgenome  foo   /path/to/ref_genome 0         /path/to/run1/
  2   5    mysecondgenome bar   /path/to/ref_genome 1         /path/to/run2/

(If you copy/paste this, make sure to convert spaces to tabs.)

In this example, the first sample was run on lane 1 and contains single-end 
reads. The second sample was sequenced on lane 5 and contains paired-end reads.
The standard mapper used with sniper.py is bowtie, so the specified reference 
is the name of the respective index file, excluding file suffix.

The first line shows the headers but is treated as a comment; 
comments are discarded by Sniper.

Raw sequence read files should contain as a substring the label indicated 
in the run column and placed in the /path/to/reads/directory/ directory. 

For example the first sample file may be named 's_1_sequence.txt' or 
'testing1.txt'. The second sample should have 2 files since it is 
paired-end, named 's_5_1_sequence.txt' and 's_5_2_sequence.txt'. 
Colorspace files should take the format 's_1_sequence.csfasta' 
and 's_1_sequence.qual' for the fasta and quality files, respectively. 

Additional fields may be added to the map file after the required columns 
(e.g. concentration, date sequenced, author, etc.).

To specify the prior model and theta parameter value in the map file, 
include headings "Prior" and "Theta" after "Path" and include the 
particular prior and theta values for each sample. Missing values are 
tolerated, in which case those samples will be processed with the 
default values.

Estimating insert size distribution for paired-end data
=======================================================
Two-step execution for paired-end read data:
1. sniper.py -m <map_file.txt> -r <reference_genome.fasta> --ins
2. sniper.py -m <map_file.txt> -r <reference_genome.fasta> -z

By default paired-end reads are mapped using an insert size range of 
0 to 350 nt. Users may specify custom min and max bounds using

* --minins: Specify the minimium insert size for all samples
* --maxins: Specify the maximum insert size for all samples

Alternatively use '--ins <FLOAT>' to estimate the empirical insert size 
distribution for each sample specified in the map file:

* --ins <FLOAT>: Estimate insert size distributions for all samples specified 
                 in the map file using the percentile <FLOAT>[0,1] of the 
                 insert size distribution for alignment (default: 0.99)

Note that alignments must first be generated using '-a'. 
A file 'PE_insert_size.txt' will be created in the specified output directory 
that Sniper will use by default if samples are mapped again 
(using '--realign/--ra').
