Much experimental effort is invested these days in fabricating nanoelectromechanical systems (NEMS) that are sufficiently small, cold and clean, so as to approach quantum mechanical behavior as their typical quantum energy scale ℏΩ becomes comparable with that of the ambient thermal energy kBT. Such systems will hopefully enable one to observe the quantum behavior of human-made objects, and test some of the basic principles of quantum mechanics. Here, we expand and elaborate on our recent suggestion (Katz et al 2007 Phys. Rev. Lett. 99 040404) to exploit the nonlinear nature of a nanoresonator in order to observe its transition into the quantum regime. We study this transition for an isolated resonator, as well as one that is coupled to a heat bath at either zero or finite temperature. We argue that by exploiting nonlinearities, quantum dynamics can be probed using technology that is almost within reach. Numerical solutions of the equations of motion display the first quantum corrections to classical dynamics that appear as the classical-toquantum transition occurs. This provides practical signatures to look for in future experiments with NEMS resonators.