Debian Med Next generation sequencing packages

The BEDTools utilities allow one to address common genomics tasks such as finding feature overlaps and computing coverage. The utilities are largely based on four widely-used file formats: BED, GFF/GTF, VCF, and SAM/BAM. Using BEDTools, one can develop sophisticated pipelines that answer complicated research questions by streaming several BEDTools together.

The groupBy utility is distribued in the filo package.

This package addresses the problem to interpret the results from the latest (2010) DNA sequencing technologies. Those will yield fairly short stretches and those cannot be interpreted directly. It is the challenge for tools like Bowtie to give a chromosomal location to the short stretches of DNA sequenced per run.

Bowtie aligns short DNA sequences (reads) to the human genome at a rate of over 25 million 35-bp reads per hour. Bowtie indexes the genome with a Burrows-Wheeler index to keep its memory footprint small: typically about 2.2 GB for the human genome (2.9 GB for paired-end).

Burrows-Wheeler Aligner (BWA) is a program that aligns relatively short nucleotide sequences against a long reference sequence such as the human genome. It implements two algorithms, bwa-short and BWA-SW. The former works for query sequences shorter than 200 bp and the latter for longer sequences up to around 100 kbp. Both algorithms do gapped alignment. They are usually more accurate and faster on queries with low error rates.

The FASTX-Toolkit is a collection of command line tools for preprocessing short nucleotide reads in FASTA and FASTQ formats, usually produced by Next-Generation sequencing machines. The main processing of such FASTA/FASTQ files is mapping (aligning) the sequences to reference genomes or other databases using specialized programs like BWA, Bowtie and many others. However, it is sometimes more productive to preprocess the FASTA/FASTQ files before mapping the sequences to the genome—manipulating the sequences to produce better mapping results. The FASTX-Toolkit tools perform some of these preprocessing tasks.

The following tools are available as part of the filo package:

groupBy – mimics the “groupBy” clause in database systems.

shuffle – randomize the order of lines in a file.

stats – computes descriptive statistic on a given column of a tab-delimited file or stream.

Because their name is too generic, ‘shuffle’ and ‘stats’ are relocated in /usr/lib/filo.

LAST is software for comparing and aligning sequences, typically DNA or protein sequences. LAST is similar to BLAST, but it copes better with very large amounts of sequence data. Here are two things LAST is good at:

• Comparing large (e.g. mammalian) genomes.
• Mapping lots of sequence tags onto a genome.

The main technical innovation is that LAST finds initial matches based on their multiplicity, instead of using a fixed size (e.g. BLAST uses 10-mers). This allows one to map tags to genomes without repeat-masking, without becoming overwhelmed by repetitive hits. To find these variable-sized matches, it uses a suffix array (inspired by Vmatch). To achieve high sensitivity, it uses a discontiguous suffix array, analogous to spaced seeds.

Maq (short for Mapping and Assembly with Quality) builds mapping assemblies from short reads generated by the next-generation sequencing machines. It was particularly designed for Illumina-Solexa 1G Genetic Analyzer, and has a preliminary functionality to handle ABI SOLiD data. Maq is previously known as mapass2.

Developmemt of Maq stopped in 2008. Its successors are BWA and SAMtools.

The mira genome fragment assembler is a specialised assembler for sequencing projects classified as 'hard' due to high number of similar repeats. For expressed sequence tags (ESTs) transcripts, miraEST is specialised on reconstructing pristine mRNA transcripts while detecting and classifying single nucleotide polymorphisms (SNP) occuring in different variations thereof.

The assembler is routinely used for such various tasks as mutation detection in different cell types, similarity analysis of transcripts between organisms, and pristine assembly of sequences from various sources for oligo design in clinical microarray experiments.

The package provides the following executables: Binaries provided:

• mira: for assembly of genome sequences
• miramem: estimating memory needed to assemble projects. Realised through link to mira.
• convert_project: for converting project file types into other types
• caf2fasta, caf2gbf, caf2text, caf2html, gbf2caf and gbf2fasta are some frequently used file converters (realised through links to convert_project)
• scftool: set of tools useful when working with SCF trace files
• fastatool: set of tools useful when working with FASTA trace files

Scripts provided:

• fasta2frag.tcl: fragmenting sequences into smaller, overlapping subsequences. Useful for simulating shotgun sequences. Can create subsequences in both directions (/default) and also paired-end sequences.
• fastaselect.tcl: given a FASTA file (and possibly a FASTA quality file) and a file with names of reads, select the sequences from the input FASTA (and quality file) and writes them to an output FASTA
• fastqselect.tcl: like fastaselect.tcl, only for FASTQ
• fixACE4consed.tcl: Consed has a bug which incapacitates it from reading consensus tags in ACE files written by the MIRA assembler (and possibly other programs). This script massages an ACE file so that consed can read the consensus tags.

Mothur seeks to develop a single piece of open-source, expandable software to fill the bioinformatics needs of the microbial ecology community. It has incorporated the functionality of dotur, sons, treeclimber, s-libshuff, unifrac, and much more. In addition to improving the flexibility of these algorithms, a number of other features including calculators and visualization tools were added.

SAM (Sequence Alignment/Map) format is a generic format for storing large nucleotide sequence alignments. Picard Tools includes these utilities to manipulate SAM and BAM files: BamToBfq IlluminaBasecallsToSam BuildBamIndex MarkDuplicates CalculateHsMetrics MeanQualityByCycle CleanSam MergeBamAlignment CollectAlignmentSummaryMetrics MergeSamFiles CollectGcBiasMetrics NormalizeFasta CollectInsertSizeMetrics QualityScoreDistribution CollectRnaSeqMetrics ReplaceSamHeader CompareSAMs RevertSam CreateSequenceDictionary SamFormatConverter ExtractIlluminaBarcodes SamToFastq EstimateLibraryComplexity SortSam FastqToSam ValidateSamFile FixMateInformation ViewSam

QIIME (canonically pronounced ‘Chime’) is a pipeline for performing microbial community analysis that integrates many third party tools which have become standard in the field. A standard QIIME analysis begins with sequence data from one or more sequencing platforms, including

• Sanger,
• Roche/454, and

• Illumina GAIIx. With all the underlying tools installed, of which not all are yet available in Debian (or any other Linux distribution), QIIME can perform

• library de-multiplexing and quality filtering;

• denoising with PyroNoise;
• OTU and representative set picking with uclust, cdhit, mothur, BLAST, or other tools;
• taxonomy assignment with BLAST or the RDP classifier;
• sequence alignment with PyNAST, muscle, infernal, or other tools;
• phylogeny reconstruction with FastTree, raxml, clearcut, or other tools;
• alpha diversity and rarefaction, including visualization of results, using over 20 metrics including Phylogenetic Diversity, chao1, and observed species;
• beta diversity and rarefaction, including visualization of results, using over 25 metrics including weighted and unweighted UniFrac, Euclidean distance, and Bray-Curtis;
• summarization and visualization of taxonomic composition of samples using pie charts and histograms and many other features.

QIIME includes parallelization capabilities for many of the computationally intensive steps. By default, these are configured to utilize a mutli-core environment, and are easily configured to run in a cluster environment. QIIME is built in Python using the open-source PyCogent toolkit. It makes extensive use of unit tests, and is highly modular to facilitate custom analyses.

Bioconductor package for differential expression analysis of whole transcriptome sequencing (RNA-seq) and digital gene expression profiles with biological replication. It uses empirical Bayes estimation and exact tests based on the negative binomial distribution. It is also useful for differential signal analysis with other types of genome-scale count data.

This tool allows one to display very long data vectors in a space-efficient manner, by organising it along a 2D Hilbert curve. The user can then visually judge the large scale structure and distribution of features simultaenously with the rough shape and intensity of individual features.

In bioinformatics, a typical use case is ChIP-Chip and ChIP-Seq, or basically all the kinds of genomic data, that are conventionally displayed as quantitative track ("wiggle data") in genome browsers such as those provided by Ensembl or UCSC.

Samtools is a set of utilities that manipulate nucleotide sequence alignments in the binary BAM format. It imports from and exports to the ascii SAM (Sequence Alignment/Map) format, does sorting, merging and indexing, and allows to retrieve reads in any regions swiftly. It is designed to work on a stream, and is able to open a BAM (not SAM) file on a remote FTP or HTTP server.

Tools for reading the SRA archive, generally by converting individual runs into some commonly used format such as fastq.

The textual dumpers "sra-dump" and "vdb-dump" are provided in this release as an aid in visual inspection. It is likely that their actual output formatting will be changed in the near future to a stricter, more formalized representation[s]. PLEASE DO NOT RELY UPON THE OUTPUT FORMAT SEEN IN THIS RELEASE.

The "help" information will be improved in near future releases, and the tool options will become standardized across the set. More documentation will also be provided documentation on the NCBI web site.

Tool options may change in the next release. Version 1 tool options will remain supported wherever possible in order to preserve operation of any existing scripts.

The Short Sequence Assembly by K-mer search and 3′ read Extension (SSAKE) is a genomics application for aggressively assembling millions of short nucleotide sequences by progressively searching for perfect 3′-most k-mers using a DNA prefix tree. SSAKE is designed to help leverage the information from short sequences reads by stringently clustering them into contigs that can be used to characterize novel sequencing targets.

Tabix indexes files where some columns indicate sequence coordinates: name (usually a chromosme), start and stop. The input data file must be position sorted and compressed by bgzip (provided in this package), which has a gzip like interface. After indexing, tabix is able to quickly retrieve data lines by chromosomal coordinates. Fast data retrieval also works over network if an URI is given as a file name.

TopHat aligns RNA-Seq reads to mammalian-sized genomes using the ultra high-throughput short read aligner Bowtie, and then analyzes the mapping results to identify splice junctions between exons. TopHat is a collaborative effort between the University of Maryland Center for Bioinformatics and Computational Biology and the University of California, Berkeley Departments of Mathematics and Molecular and Cell Biology.

ECHO is an error correction algorithm designed for short-reads from next-generation sequencing platforms such as Illumina's Genome Analyzer II. The algorithm uses a Bayesian framework to improve the quality of the reads in a given data set by employing maximum a posteriori estimation.

VCFtools is a program package designed for working with VCF files, such as those generated by the 1000 Genomes Project. The aim of VCFtools is to provide methods for working with VCF files: validating, merging, comparing and calculate some basic population genetic statistics.

Velvet is a de novo genomic assembler specially designed for short read sequencing technologies, such as Solexa or 454, developed by Daniel Zerbino and Ewan Birney at the European Bioinformatics Institute (EMBL-EBI), near Cambridge, in the United Kingdom.

Velvet currently takes in short read sequences, removes errors then produces high quality unique contigs. It then uses paired read information, if available, to retrieve the repeated areas between contigs.

Cufflinks assembles transcripts, estimates their abundances, and tests for differential expression and regulation in RNA-Seq samples. It accepts aligned RNA-Seq reads and assembles the alignments into a parsimonious set of transcripts. Cufflinks then estimates the relative abundances of these transcripts based on how many reads support each one.

MosaikBuild converts various sequence formats into Mosaik’s native read format. MosaikAligner pairwise aligns each read to a specified series of reference sequences. MosaikSort resolves paired-end reads and sorts the alignments by the reference sequence coordinates. Finally, MosaikText converts alignments to different text-based formats.

At this time, the workflow consists of supplying sequences in FASTA, FASTQ, Illumina Bustard & Gerald, or SRF file formats and producing results in the BLAT axt, the BAM/SAM, the UCSC Genome Browser bed, or the Illumina ELAND formats.

ANNOVAR is an efficient software tool to utilize update-to-date information to functionally annotate genetic variants detected from diverse genomes (including human genome hg18, hg19, as well as mouse, worm, fly, yeast and many others). Given a list of variants with chromosome, start position, end position, reference nucleotide and observed nucleotides, ANNOVAR can perform:

 1. Gene-based annotation: identify whether SNPs or CNVs cause protein coding
    changes and the amino acids that are affected. Users can flexibly use RefSeq
    genes, UCSC genes, ENSEMBL genes, GENCODE genes, or many other gene definition
    systems.
 2. Region-based annotations: identify variants in specific genomic regions,
    for example, conserved regions among 44 species, predicted transcription
    factor binding sites, segmental duplication regions, GWAS hits, database
    of genomic variants, DNAse I hypersensitivity sites, ENCODE
    H3K4Me1/H3K4Me3/H3K27Ac/CTCF sites, ChIP-Seq peaks, RNA-Seq peaks, or many
    other annotations on genomic intervals.
 3. Filter-based annotation: identify variants that are reported in dbSNP,
    or identify the subset of common SNPs (MAF>1%) in the 1000 Genome Project,
    or identify subset of non-synonymous SNPs with SIFT score>0.05, or many
    other annotations on specific mutations.
 4. Other functionalities: Retrieve the nucleotide sequence in any
    user-specific genomic positions in batch, identify a candidate gene list
    for Mendelian diseases from exome data, identify a list of SNPs from
    1000 Genomes that are in strong LD with a GWAS hit, and many other
    creative utilities.

In a modern desktop computer (3GHz Intel Xeon CPU, 8Gb memory), for 4.7 million variants, ANNOVAR requires ~4 minutes to perform gene-based functional annotation, or ~15 minutes to perform stepwise "variants reduction" procedure, making it practical to handle hundreds of human genomes in a day.

Forge Genome Assembler is a parallel, MPI based genome assembler for mixed read types.

Forge is a classic "Overlap layout consensus" genome assembler written by Darren Platt and Dirk Evers. Implemented in C++ and using the parallel MPI library, it runs on one or more machines in a network and can scale to very large numbers of reads provided there is enough collective memory on the machines used. It generates a full consensus alignment of all reads, can handle mixtures of sanger, 454 and illumina reads. There is some support for solid color space and it includes built in tools for vector trimming and contamination screening.

Forge and was originally developed at Exelixis and they have kindly agreed to place the software which underwent much subsequent development outside Exelixis, into the public domain. Forge works with most of the common MPI implementations.