Coded aperture correlation holography (COACH) is a recently introduced technique for recording incoherent digital holograms of general three-dimensional scenes. In COACH, a random-like coded phase mask (CPM) is used as a coded aperture. Even though the CPM is optimized to reduce background noise, there is still a substantial amount of noise, mitigating the performance of COACH. In order to reduce the noise, we first modify the hologram reconstruction method. Instead of computing the correlation between a complex hologram of the entire object and a hologram of a source point, in this study the numerical correlation is performed with a phase-only filter. In other words, the phase function of the Fourier transform of the source point hologram is used as the spatial filter in the correlation process. Furthermore, we propose and demonstrate two additional methods for reducing the background noise in COACH. The first is based on the integration of a quadratic phase function, as used in Fresnel incoherent correlation holography (FINCH), with the CPM of COACH. This hybrid COACHFINCH system enables a dynamic trade-off between the amount of background noise and the axial resolution of the system. The second method is employed by recording COACH holograms with multiple independent CPMs and averaging over the reconstructed images. The results of the above two techniques are compared with FINCH and with a regular imaging system.