We describe a mechanism by which nonlinear phononics allows ultrafast coherent and directional control of transient structural distortions. With ErFeO3 as a model system, we use density functional theory to calculate the structural properties as input into an anharmonic phonon model that describes the response of the system to a pulsed optical excitation. We find that the trilinear coupling of two orthogonal infrared-active phonons to a Raman-active phonon causes a transient distortion of the lattice. In contrast to the quadratic-linear coupling that has been previously explored, the direction of the distortion is determined by the polarization of the exciting light, introducing a novel mechanism for nonlinear phononic switching. Since the occurrence of the coupling is determined by the symmetry of the system we propose that it is a universal feature of orthorhombic and tetragonal perovskites.