The estimation of a person's age based on a face image is a common biometric task conducted effortlessly by human observers. We present a dual Convolutional Neural Network (CNN) and Support Vector Regression (SVR) approach for face-based age estimation. A CNN is trained for representation learning, followed by Metric Learning, after which SVR is applied to the learned features. This allows to overcome the lack of large datasets with age annotations, by initially training the CNN for face recognition. The proposed scheme was applied to the MORPH-II and FG-Net datasets and compares favorably with contemporary state-of-the-art approaches. In particular, we show that domain adaptation which is essential for analyzing small-scale datasets, such as the FG-Net, can be achieved by retraining the SVR layer, rather than the CNN.