Search for the best decision rules with the help of a probabilistic estimate

The problem of how to find the best decision rule in the course of a search, based on sample set analysis, is considered. Specifically the problem of selecting the best subset of regressors is highlighted. In the considered formulation the problem is an important case of how to learn a dependence by examples. The concept of Predictive Probabilistic Estimate (PPE) is introduced and its properties are discussed. Asymptotic properties of PPE are based on Vapnik-Chervonenkis theory of uniform convergence of a set of sample estimates. Finite-sample-size properties of PPE demonstrate how PPE takes into account the presence of a search process and the complexity of a regression formula, while estimating quality of fit. Some practical and model examples are presented.