On the compatibility of quartet trees

Phylogenetic tree reconstruction is a fundamental biological problem. Quartet trees, trees over four species, are the minimal informational unit for phylogenetic classification. While every phylogenetic tree over n species defines (₄ⁿ) quartets, not every set of quartets is compatible with some phylogenetic tree. Here we focus on the compatibility of quartet sets. We provide several results addressing the question of what can be inferred about the compatibility of a set from its subsets. Most of our results use probabilistic arguments to prove the sought characteristics. In particular we show that there are quartet sets Q of size m = cn log n in which every subset of cardinality c′n/ log n is compatible, and yet no fraction of more than 1/3 + ε of Q is compatible. On the other hand, in contrast to the classical result stating when Q is the densest, i.e., m = (₄ⁿ) and the compatibility of any set of three quartets implies full compatibility, we show that even for m = Θ ( (₄ⁿ) ) there are (very) incompatible sets for which every subset of large constant cardinality is compatible. Our final result relates to the conjecture of Bandelt and Dress regarding the maximum quartet distance between trees. We provide asymptotic upper and lower bounds for this value.