BLAT Suite Program Specifications and User Guide

This documentation for the BLAT program suite explains how to use the software, including command line switches and options. Licensed users may contact us for additional tech support.

General

BLAT produces two major classes of alignments: at the DNA level between two sequences that are of 95% or greater identity, but which may include large inserts, and at the protein or translated DNA level between sequences that are of 80% or greater identity and may also include large inserts. The output of BLAT is flexible. By default it is a simple tab-delimited file which describes the alignment, but which does not include the sequence of the alignment itself. Optionally it can produce BLAST and WU-BLAST compatable output as well as a number of other formats.

The main programs in the BLAT suite are:

Building an index of the genome typically takes 10 or 15 minutes. Typically for interactive applications one uses gfServer to build a whole genome index. At that point gfClient or webBlat can align a single query within few seconds. If one is aligning a lot of sequences in a batch mode then blat can be more efficient, particularly if run on a cluster of computers. Each blat run is typically done against a single chromosome, but with a large number of query sequences.

Other programs in the BLAT suite are:

Running the Programs

The command line options of each of the programs is described below. Similar summaries of usage are printed when a command is run with no arguments. For information on installing webBlat, see Setting up WebBlat (below).

usage:

where:

options:

Common usage scenarios

Here are some BLAT settings you may find handy for specific alignment scenarios.

1) Mapping ESTs to the genome within the same species:

-ooc=11.ooc

2) Mapping full length mRNAs to the genome in the same species:

-ooc=11.ooc -fine -q=rna

3) Mapping ESTs to the genome across species:

-q=dnax -t=dnax

4) Mapping mRNA to the genome across species:

-q=rnax -t=dnax

5) Mapping proteins to the genome:

-q=prot -t=dnax

6) Mapping DNA to DNA in the same species:

-ooc=11.ooc -fastMap

7) Mapping DNA from one species to another species:

-ooc=11.ooc
When mapping DNA from one species to another the query side of the alignment should be cut up into chunks of 25kb or less for best performance.

To set up a server:

To remove a server:

To query a server with DNA sequence:

To query a server with protein sequence:

To query a server with translated dna sequence:

To process one probe fa file against a .nib format genome (not starting server):

To figure out usage level:

To get input file list:

Options:

Usage:

Where:

Options:

webBlat

webBlat generally is not run from the command line. See Setting up WebBlat (below) for information on this program.

Usage:

Options:

Usage:

Options:

Usage:

Usage:

Usage:

Options:

Usage:

Where:

Options:

Usage:

Options:

It's a really good idea if the psl file is sorted by target if it contains multiple targets. Otherwise this will be very very slow. The target and query lists can either be fasta files, nib files, or a list of fasta and/or nib files one per line. Currently this only handles nucleotide based psl files.

Usage:

Options:

INSTALLING WEBBLAT

Installing a web-based BLAT Server involves four major steps:

1. CREATING SEQUENCE DATABASES

You create databases with the program faToTwoBit. Typically you'll create a separate database for each genome you are indexing. Each database can contain up to four billion bases of sequence in an unlimited number of records. The databases for webPcr and webBlat are identical.

The input to faToTwoBit is one or more fasta format files each of which can contain multiple records. If the sequence contains repeat sequences, as is the case with vertebrates and many plants, the repeat sequences can be represented in lower case and the other sequence in upper case. The gfServer program can be configured to ignore the repeat sequences. The output of Page 10 of 13 faToTwoBit is a file which is designed for fast random access and efficient storage. The output files store four bases per byte. They use a small amount of additional space to store the case of the DNA and to keep track of runs of N's in the input. Non-N ambiguity codes such as Y and U in the input sequence will be converted to N.

Here's how a typical installation might create a mouse and a human genome database:

There's no need to put all of the databases in the same directory, but it can simplify bookkeeping.

The databases can also be in the .nib format which was used with blat and gfClient/gfServer until recently. The .nib format only packed 2 bases per byte, and could only handle one record per nib file. Recent versions of blat and related programs can use .2bit files as well.

2. CREATING IN-MEMORY INDICES WITH GFSERVER

The gfServer program creates an in-memory index of a nucleotide sequence database. The index can either be for translated or untranslated searches. Translated indexes enable protein-based blat queries and use approximately two bytes per unmasked base in the database. Untranslated indexes are used nucleotide-based blat queries as well as for In-silico PCR. An index for normal blat uses approximately 1/4 byte per base. For blat on smaller (primer-sized) queries or for In-silico PCR a more thorough index that requires 1/2 byte per base is recommended. The gfServer is memory intensive but typically doesn not require a lot of CPU power. Memory permitting multiple gfServers can be run on the same machine.

A typical installation might go:

3. EDITING THE WEBBLAT.CFG FILE

The webBlat.cfg file tells the webBlat program where to look for gfServers and for sequence. The basic format of the .cfg file is line oriented with the first word of the line being a command. Blank lines and lines starting with # are ignored. The webBlat.cfg and webPcr.cfg files are similar.

The webBlat.cfg commands are:

The background and company commands are optional. The webBlat.cfg file must have at least one valid gfServer or gfServerTrans line, and a tempDir line.

Here is a webBlat.cfg file that you might find at a typical installation:

4. PUTTING WEBBLAT WHERE THE WEB SERVER CAN EXECUTE IT

The details of this step vary highly from web server to web server. On a typical Apache installation it might be:

assuming that you've put the executable and config file in kent/webBlat.

The program will create some files in the trash directory. It is good to periodically clean out old files from this directory.

On Mac OS-X instead you might do:

Unless you are administering your own computer you will likely need to ask your local system administrators for help with this part of the webBlat installation.

File Formats

.psl files

A .psl file describes a series of alignments in a dense, easily parsed text format. It begins with a five line header which describes each field. Following this is one line for each alignment with a tab between each field. The fields are described below in a format suitable for many relational databases.

matches int unsigned , # Number of bases that match that aren't repeats
misMatches int unsigned , # Number of bases that don't match
repMatches int unsigned , # Number of bases that match but are part of repeats
nCount int unsigned , # Number of 'N' bases
qNumInsert int unsigned , # Number of inserts in query
qBaseInsert int unsigned , # Number of bases inserted in query
tNumInsert int unsigned , # Number of inserts in target
tBaseInsert int unsigned , # Number of bases inserted in target
strand char(2) , # + or - for query strand, optionally followed by + or – for target strand
qName varchar(255) , # Query sequence name
qSize int unsigned , # Query sequence size
qStart int unsigned , # Alignment start position in query
qEnd int unsigned , # Alignment end position in query
tName varchar(255) , # Target sequence name
tSize int unsigned , # Target sequence size
tStart int unsigned , # Alignment start position in target
tEnd int unsigned , # Alignment end position in target
blockCount int unsigned , # Number of blocks in alignment
blockSizes longblob , # Size of each block in a comma separated list
qStarts longblob , # Start of each block in query in a comma separated list
tStarts longblob , # Start of each block in target in a comma separated list

Currently the program does not distinguish between matches and repMatches. repMatches is always zero. There is a little gotcha in the .psl format. It has to do with how coordinates are handled on the negative strand. In the qStart/qEnd fields the coordinates are where it matches from the point of view of the forward strand (even when the match is on the reverse strand). However on the qStarts[] list, the coordinates are reversed.

Here's an example of a 30-mer that has 2 blocks that align on the minus strand and 2 blocks on the plus strand (this sort of stuff happens in real life in response to assembly errors sometimes).

0 1 2 3 tens position in query
0123456789012345678901234567890 ones position in query
++++ +++++ plus strand alignment on query
-------- ---------- minus strand alignment on query

Plus strand:

qStart 12 qEnd 31 blockSizes 4,5 qStarts 12,26

Minus strand:

qStart 4 qEnd 26 blockSizes 10,8 qStarts 5,19

Essentially the minus strand blockSizes and qStarts are what you would get if you reverse complemented the query.However the qStart and qEnd are non-reversed. To get from one to the other:

qStart = qSize - revQEnd
qEnd = qSize - revQStart

.2bit files

A .2bit file can store multiple DNA sequence (up to 4 gig total) in a compact randomly accessible format. The two bit files contain masking information as well as the DNA itself. The file begins with a 16 byte header containing the following fields:

All fields are 32 bits unless noted. If the signature value is not as given, the reader program should byte swap the signature and see if the swapped version matches. If so all multiple-byte entities in the file will need to be byteswapped. This enables these binary files to be used unchanged on different architectures.

The header is followed by a file index. There is one entry in the index for each sequence. Each index entry contains three fields:

The index is followed by the sequence records. These contain 9 fields:

.nib files

A .nib file describes a DNA sequence packing two bases into each byte. Each nib file contains only a single sequence. A nib file begins with a 32 bit signature which is 0x6BE93D3A in the architecture of the machine that created the file, and possibly a byte-swapped version of the same number on another machine. This is followed by a 32 bit number in the same format which describes the number of bases in the file. This is followed by the bases themselves packed two bases to the byte. The first base is packed in the high order 4 bits (nibble), the second base in the low order four bits. In C code:

byte = (base1<<4) + base2

The numerical values for the bases are:

0 – T, 1 – C, 2 – A, 3 – G, 4 – N (unknown)

The most significant bit in a nibble is set if the base is masked.

Limits

The gfServer program requires approximately 1 byte for every 3 bases in the genome it is indexing in DNA mode, and 1.5 bytes for each unmasked base in translated mode. The blat program requires approximately two bytes for each base in the genome in DNA mode, and three bytes for each base in translated mode. The other programs use relatively little memory.

Last revised: October 29, 2021