Event details from: PhD Seminar

Date
7 May 2015
Time
13:30 – 14:30

Title: Exploration vs. Exploitation with Partially Observable Gaussian Autoregressive Arms

Abstract

The classical multi-armed bandit problem is a Markovian decision problem that deals with the selection of one-armed bandits (slot machines) to play on, with the objective of maximising the expected discounted or average rewards accumulated over time. In this setting it makes sense to assume that the state of the bandit's arm is frozen unless it is played. For other applications -- for example the selection of transmission channels in wireless networks -- it is more realistic that channel states evolve irrespective of whether the channels are selected for transmission. In this talk, we consider a restless bandit problem with Gaussian autoregressive arms, where the state of an arm is only observed when it is played and the state-dependent reward is collected. Since arms are only partially observable, a good decision policy needs to account for the fact that information about the state of an arm becomes more and more obsolete while the arm is not being played: The decision maker faces a tradeoff between exploiting those arms that are believed to be currently the most rewarding (i.e. those with the largest conditional mean), and exploring arms with a high conditional variance. A computationally tractable policy that gives some priority to exploration is the Whittle index policy, for which we establish structural properties. Furthermore, we examine the many-arm behaviour of the system under a parametric index policy, identifying equations describing its asymptotic dynamics.

Location: KdVI meeting room, SP 105-107, F3.20