TY - GEN
T1 - Improving training in the vicinity of temporary minima
AU - Roth, Ido
AU - Margaliot, Michael
N1 - Funding Information:
Research supported in part by research grants from the Israel Science Foundation (ISF) and the Israeli Ministry of Science and Technology.
PY - 2009
Y1 - 2009
N2 - An important problem in learning using gradient descent algorithms (such as backprop) is the slowdown incurred by temporary minima (TM). We consider this problem for an artificial neural network trained to solve the XOR problem. The network is transformed into the equivalent all permutations fuzzy rule-base which provides a symbolic representation of the knowledge embedded in the network. We develop a mathematical model for the evolution of the fuzzy rule-base parameters during learning in the vicinity of TM. We show that the rule-base becomes singular and tends to remain singular in the vicinity of TM. Our analysis suggests a simple remedy for overcoming the slowdown in the learning process incurred by TM. This is based on slightly perturbing the values of the training examples, so that they are no longer symmetric. Simulations demonstrate the usefulness of this approach.
AB - An important problem in learning using gradient descent algorithms (such as backprop) is the slowdown incurred by temporary minima (TM). We consider this problem for an artificial neural network trained to solve the XOR problem. The network is transformed into the equivalent all permutations fuzzy rule-base which provides a symbolic representation of the knowledge embedded in the network. We develop a mathematical model for the evolution of the fuzzy rule-base parameters during learning in the vicinity of TM. We show that the rule-base becomes singular and tends to remain singular in the vicinity of TM. Our analysis suggests a simple remedy for overcoming the slowdown in the learning process incurred by TM. This is based on slightly perturbing the values of the training examples, so that they are no longer symmetric. Simulations demonstrate the usefulness of this approach.
UR - http://www.scopus.com/inward/record.url?scp=68749122251&partnerID=8YFLogxK
U2 - 10.1007/978-3-642-02478-8_17
DO - 10.1007/978-3-642-02478-8_17
M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???
AN - SCOPUS:68749122251
SN - 3642024777
SN - 9783642024771
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 131
EP - 139
BT - Bio-Inspired Systems
T2 - 10th International Work-Conference on Artificial Neural Networks, IWANN 2009
Y2 - 10 June 2009 through 12 June 2009
ER -