We present an analysis of a free-electron laser within a three-dimensional model, in which the finite transverse dimensions of the electron beam are taken into account together with the confinement of the electromagnetic modes by a guiding structure. The analysis is based on the expansion of the beams self-potential into an infinite set of modes that interact simultaneously with the wiggler and signal fields. Explicit gain-dispersion equations are developed for several cases including a waveguide tube completely filled by an electron beam of uniform density, and a uniform-density electron beam partially filling a waveguide (or in free space). This is carried out in both the magnetized beam limit and the general case which includes surface currents. The results bear significant effect on free-electron-laser gain operating parameters in the collective regimes and on the threshold of absolute instability oscillation.