We show how the fusion rules can be used to associate with every rational conformal field theory a discrete group, the center. The center is generated by primary fields having unique fusion rules with any other field. The existence of a non-trivial center implies the existence of non-diagonal modular invariants, which are related to extended integer or fractional spin algebras. Applied to Kac-Moodt algebras this method yields all known as well as many new infinite series of modular invariants. Some results on exceptional invariants are also presented, including an example of an exceptional integer spin invariant that does not correspond to a conformal embedding.