An analytical three-dimensional model is presented for free-electron lasers (FELs) operating in the small-signal linear regime. The excitation of radiation and space-charge waves is found by expanding the total electromagnetic field in terms of transverse eigenmodes in a waveguide of arbitrary cross section and solving the evolution of their amplitudes from a set of coupled excitation equations. Coupled-mode theory is employed to derive dispersion relations for the space-charge waves and for the gain. The eigenmodes of the FELs (''supermodes'') and the gain for each of them are derived after diagonalization of the coupled-mode system. It is found that for the case of degenerate coupled modes (equal axial wave numbers), the normal modes satisfy the well known FEL gain dispersion equation with a modified gain parameter. The gain of the supermode, calculated according to the presented coupled-mode theory, is higher than the gain of the individual modes if calculated on the basis of a single-mode model. We demonstrate the formalism by finding the gain of the TE01, and the coupled TE21 and TM21 modes excited simultaneously in a rectangular waveguide.