TY - JOUR

T1 - Optimal ensemble averaging of neural networks

AU - Naftaly, Ury

AU - Intrator, Nathan

AU - Horn, David

PY - 1997/8

Y1 - 1997/8

N2 - Based on an observation about the different effect of ensemble averaging on the bias and variance portions of the prediction error, we discuss training methodologies for ensembles of networks. We demonstrate the effect of variance reduction and present a method of extrapolation to the limit of an infinite ensemble. A significant reduction of variance is obtained by averaging just over initial conditions of the neural networks, without varying architectures or training sets. The minimum of the ensemble prediction error is reached later than that of a single network. In the vicinity of the minimum, the ensemble prediction error appears to be flatter than that of the single network, thus simplifying optimal stopping decision. The results are demonstrated on sunspots data, where the predictions are among the best obtained, and on the 1993 energy prediction competition data set B.

AB - Based on an observation about the different effect of ensemble averaging on the bias and variance portions of the prediction error, we discuss training methodologies for ensembles of networks. We demonstrate the effect of variance reduction and present a method of extrapolation to the limit of an infinite ensemble. A significant reduction of variance is obtained by averaging just over initial conditions of the neural networks, without varying architectures or training sets. The minimum of the ensemble prediction error is reached later than that of a single network. In the vicinity of the minimum, the ensemble prediction error appears to be flatter than that of the single network, thus simplifying optimal stopping decision. The results are demonstrated on sunspots data, where the predictions are among the best obtained, and on the 1993 energy prediction competition data set B.

UR - http://www.scopus.com/inward/record.url?scp=0001066312&partnerID=8YFLogxK

U2 - 10.1088/0954-898x/8/3/004

DO - 10.1088/0954-898x/8/3/004

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:0001066312

VL - 8

SP - 283

EP - 296

JO - Network: Computation in Neural Systems

JF - Network: Computation in Neural Systems

SN - 0954-898X

IS - 3

ER -