Under vibrational strong coupling (VSC), the formation of molecular polaritons may significantly modify the photo-induced or thermal properties of molecules. In an effort to understand these intriguing modifications, both experimental and theoretical studies have focused on the ultrafast dynamics of vibrational polaritons. Here, following our recent work [Li et al., J. Chem. Phys. 154, 094124 (2021)], we systematically study the mechanism of polariton relaxation for liquid CO2 under a weak external pumping. Classical cavity molecular dynamics (CavMD) simulations confirm that polariton relaxation results from the combined effects of (i) cavity loss through the photonic component and (ii) dephasing of the bright-mode component to vibrational dark modes as mediated by intermolecular interactions. The latter polaritonic dephasing rate is proportional to the product of the weight of the bright mode in the polariton wave function and the spectral overlap between the polariton and dark modes. Both these factors are sensitive to parameters such as the Rabi splitting and cavity mode detuning. Compared to a Fermi's golden rule calculation based on a tight-binding harmonic model, CavMD yields a similar parameter dependence for the upper polariton relaxation lifetime but sometimes a modest disagreement for the lower polariton. We suggest that this disagreement results from polariton-enhanced molecular nonlinear absorption due to molecular anharmonicity, which is not included in our analytical model. We also summarize recent progress on probing nonreactive VSC dynamics with CavMD.