Learning and control of exploration primitives

Goren Gordon*, Ehud Fonio, Ehud Ahissar

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Animals explore novel environments in a cautious manner, exhibiting alternation between curiosity-driven behavior and retreats. We present a detailed formal framework for exploration behavior, which generates behavior that maintains a constant level of novelty. Similar to other types of complex behaviors, the resulting exploratory behavior is composed of exploration motor primitives. These primitives can be learned during a developmental period, wherein the agent experiences repeated interactions with environments that share common traits, thus allowing transference of motor learning to novel environments. The emergence of exploration motor primitives is the result of reinforcement learning in which information gain serves as intrinsic reward. Furthermore, actors and critics are local and ego-centric, thus enabling transference to other environments. Novelty control, i.e. the principle which governs the maintenance of constant novelty, is implemented by a central action-selection mechanism, which switches between the emergent exploration primitives and a retreat policy, based on the currently-experienced novelty. The framework has only a few parameters, wherein time-scales, learning rates and thresholds are adaptive, and can thus be easily applied to many scenarios. We implement it by modeling the rodent’s whisking system and show that it can explain characteristic observed behaviors. A detailed discussion of the framework’s merits and flaws, as compared to other related models, concludes the paper.

Original languageEnglish
Pages (from-to)259-280
Number of pages22
JournalJournal of Computational Neuroscience
Volume37
Issue number2
DOIs
StatePublished - 1 Oct 2014
Externally publishedYes

Keywords

  • Bayesian inference
  • Hierarchical control
  • Information gain
  • Intrinsic reward
  • Motor primitives
  • Reinforcement learning

Fingerprint

Dive into the research topics of 'Learning and control of exploration primitives'. Together they form a unique fingerprint.

Cite this