nucleotide substitution models

The use of maximum likelihood (ML) algorithms in developing phylogenetic hypotheses requires a model of evolution. The frequently used General Time Reversible (GTR) family of nested models encompasses 64 models with different combinations of parameters for DNA site substitution. The models are listed here from the least complex to the most parameter rich.

Jukes-Cantor (JC, nst=1): equal base frequencies, all substitutions equally likely (PAUP* rate classification: aaaaaa, PAML: aaaaaa)* (Jukes and Cantor 1969)

Felsenstein 1981 (F81, nst=1): variable base frequencies, all substitutions equally likely (PAUP*: aaaaaa, PAML: aaaaaa)** (Felsenstein 1981)

Kimura 2-parameter (K80, nst=2): equal base frequencies, variable transition and transversion frequencies (PAUP*: abaaba, PAML: abbbba) (Kimura 1980)

Hasegawa-Kishino-Yano (HKY, nst=2): variable base frequencies, variable transition and transversion frequencies (PAUP*: abaaba, PAML: abbbba) (Hasegawa et. al. 1985)

Tamura-Nei (TrN): variable base frequencies, equal transversion frequencies, variable transition frequencies (PAUP*: abaaea, PAML: abbbbf) (Tamura Nei 1993)

Kimura 3-parameter (K3P): variable base frequencies, equal transition frequencies, variable transversion frequencies (PAUP*: abccba, PAML: abccba) (Kimura 1981)

transition model (TIM): variable base frequencies, variable transitions, transversions equal (PAUP*: abccea, PAML: abccbe)

transversion model (TVM): variable base frequencies, variable transversions, transitions equal (PAUP*: abcdbe, PAML: abcdea)

symmetrical model (SYM): equal base frequencies, symmetrical substitution matrix (A to T = T to A) (PAUP*: abcdef, PAML: abcdef) (Zharkikh 1994)

general time reversible (GTR, nst=6): variable base frequencies, symmetrical substitution matrix (PAUP*: abcdef, PAML: abcdef) (e.g., Lanave et al. 1984, Tavare 1986, Rodriguez et. al. 1990)

In addition to models describing the rates of change from one nucleotide to another, there are models to describe rate variation among sites in a sequence. The following are the two most commonly used models.

gamma distribution (G): gamma distributed rate variation among sites

proportion of invariable sites (I): extent of static, unchanging sites in a dataset

Substitutions are themselves grouped hierarchically: simple, general base substitution, transitions and transversions, purine to purine and pyrimidine to pyrimidine transitions, and AC/GT and AT/CG transversions. The groupings are symbolized as rate classifications according to the PAUP* and PAML matrices below. Substitution types that are constrained to be equal in rate assume the leftmost letter symbol.

PAUP* substitution rate matrix     PAML substitution rate matrix
    A  C  G  T                         T  C  A  G
A   -  a  b  c                     T   -  a  b  c
C      -  d  e                     C      -  d  e
G         -  f=1                   A         -  f=1
T            -                     G            -

Modeltest also uses the special case of equal base frequencies for the models that have variable frequencies.