Papers by Erin J . Talvitie
Proceedings of the Thirty-Fifth International Conference on Machine Learning (ICML), 2018
In model-based reinforcement learning it is typical to decouple the problems of learning the dyna... more In model-based reinforcement learning it is typical to decouple the problems of learning the dynamics model and learning the reward function. However, when the dynamics model is flawed, it may generate erroneous states that would never occur in the true environment. It is not clear a priori what value the reward function should assign to such states. This paper presents a novel error bound that accounts for the reward model's behavior in states sampled from the model. This bound is used to extend the existing Hallucinated DAgger-MC algorithm, which offers theoretical performance guarantees in deterministic MDPs that do not assume a perfect model can be learned. Empirically, this approach to reward learning can yield dramatic improvements in control performance when the dynamics model is flawed.
Journal of Artificial Intelligence Research (JAIR), 2018
The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of build... more The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games. It supports a variety of different problem settings and it has been receiving increasing attention from the scientific community, leading to some high-profile success stories such as the much publicized Deep Q-Networks (DQN). In this article we take a big picture look at how the ALE is being used by the research community. We show how diverse the evaluation methodologies in the ALE have become with time, and highlight some key concerns when evaluating agents in the ALE. We use this discussion to present some methodological best practices and provide new benchmark results using these best practices. To further the progress in the field, we introduce a new version of the ALE that supports multiple game modes and provides a form of stochasticity we call sticky actions. We conclude this big picture look by revisiting challenges posed when the ALE was introduced, summarizing the state-of-the-art in various problems and highlighting problems that remain open.
Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI), 2017
When an agent cannot represent a perfectly accurate model of its environment's dynamics, model-ba... more When an agent cannot represent a perfectly accurate model of its environment's dynamics, model-based reinforcement learning (MBRL) can fail catastrophically. Planning involves composing the predictions of the model; when flawed predictions are composed, even minor errors can compound and render the model useless for planning. Hallucinated Replay (Talvitie 2014) trains the model to " correct " itself when it produces errors, substantially improving MBRL with flawed models. This paper theoretically analyzes this approach, illuminates settings in which it is likely to be effective or ineffective , and presents a novel error bound, showing that a model's ability to self-correct is more tightly related to MBRL performance than one-step prediction error. These results inspire an MBRL algorithm for deterministic MDPs with performance guarantees that are robust to model class limitations.
Proceedings of the 15th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 2016
The recently introduced Deep Q-Networks (DQN) algorithm has gained attention as one of the first ... more The recently introduced Deep Q-Networks (DQN) algorithm has gained attention as one of the first successful combinations of deep neural networks and reinforcement learning. Its promise was demonstrated in the Arcade Learning Environment (ALE), a challenging framework composed of dozens of Atari 2600 games used to evaluate general competency in AI. It achieved dramatically better results than earlier approaches, showing that its ability to learn good representations is quite robust and general. This paper attempts to understand the principles that underlie DQN's impressive performance and to better contextualize its success. We systematically evaluate the importance of key representational biases encoded by DQN's network by proposing simple linear representations that make use of these concepts. Incorporating these characteristics, we obtain a computationally practical feature set that achieves competitive performance to DQN in the ALE. Besides offering insight into the strengths and weaknesses of DQN, we provide a generic representation for the ALE, significantly reducing the burden of learning a representation for each game. Moreover, we also provide a simple, reproducible benchmark for the sake of comparison to future work in the ALE.
Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI), 2015
Monte Carlo planning has been proven successful in many sequential decision-making settings, but ... more Monte Carlo planning has been proven successful in many sequential decision-making settings, but it suffers from poor exploration when the rewards are sparse. In this paper, we improve exploration in UCT by generalizing across similar states using a given distance metric. When the state space does not have a natural distance metric, we show how we can learn a local manifold from the transition graph of states in the near future. to obtain a distance metric. On domains inspired by video games, empirical evidence shows that our algorithm is more sample efficient than UCT, particularly when rewards are sparse.
Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI), 2015
Much of the focus on finding good representations in reinforcement learning has been on learning ... more Much of the focus on finding good representations in reinforcement learning has been on learning complex non-linear predictors of value. Policy gradient algorithms, which directly represent the policy, often need fewer parameters to learn good policies. However, they typically employ a fixed parametric representation that may not be sufficient for complex domains. This paper introduces the Policy Tree algorithm, which can learn an adaptive representation of policy in the form of a decision tree over different instantiations of a base policy. Policy gradient is used both to optimize the parameters and to grow the tree by choosing splits that enable the maximum local increase in the expected return of the policy. Experiments show that this algorithm can choose genuinely helpful splits and significantly improve upon the commonly used linear Gibbs softmax policy, which we choose as our base policy.
Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI), 2015
While model-based reinforcement learning is often studied under the assumption that a fully accur... more While model-based reinforcement learning is often studied under the assumption that a fully accurate model is contained within the model class, this is rarely true in practice. When the model class may be fundamentally limited, it can be difficult to obtain theoretical guarantees. Under some conditions the DAgger algorithm promises a policy nearly as good as the plan obtained from the most accurate model in the class, but only if the planning algorithm is near-optimal, which is also rarely the case in complex problems. This paper explores the interaction between DAgger and Monte Carlo planning, specifically showing that DAgger may perform poorly when coupled with a sub-optimal planner. A novel variation of DAgger specifically for use with Monte Carlo planning is derived and is shown to behave far better in some cases where DAgger fails.
Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, 2014
When an imperfect model is used to generate sample rollouts, its errors tend to compound -- a fla... more When an imperfect model is used to generate sample rollouts, its errors tend to compound -- a flawed sample is given as input to the model, which causes more errors, and so on. This presents a barrier to applying rollout-based planning algorithms to learned models. To address this issue, a training methodology called ``hallucinated replay'' is introduced, which adds samples from the model into the training data, thereby training the model to produce sensible predictions when its own samples are given as input. Capabilities and limitations of this approach are studied empirically. In several examples hallucinated replay allows effective planning with imperfect models while models trained using only real experience fail dramatically.
Proceedings of the 31st International Conference on Machine Learning, 2014
Context Tree Weighting is a powerful probabilistic sequence prediction technique that efficiently... more Context Tree Weighting is a powerful probabilistic sequence prediction technique that efficiently performs Bayesian model averaging over the class of all prediction suffix trees of bounded depth. In this paper we show how to generalize this technique to the class of K-skip prediction suffix trees. Contrary to regular prediction suffix trees, K-skip prediction suffix trees are permitted to ignore up to K contiguous portions of the context. This allows for significant improvements in predictive accuracy when irrelevant variables are present, a case which often occurs within record-aligned data and images. We provide a regret-based analysis of our approach, and empirically evaluate it on the Calgary corpus and a set of Atari 2600 screen prediction tasks.
Advances in Neural Information Processing Systems 25, 2012
This paper introduces timeline trees, which are partial models of partially observable environmen... more This paper introduces timeline trees, which are partial models of partially observable environments. Timeline trees are given some specific predictions to make and learn a decision tree over history. The main idea of timeline trees is to use temporally abstract features to identify and split on features of key events, spread arbitrarily far apart in the past (whereas previous decision-tree-based methods have been limited to a finite suffix of history). Experiments demonstrate that timeline trees can learn to make high quality predictions in complex, partially observable environments with high-dimensional observations (e.g. an arcade game).
Journal of AI Research, Volume 42, 2011
When faced with the problem of learning a model of a high-dimensional environment, a common appro... more When faced with the problem of learning a model of a high-dimensional environment, a common approach is to limit the model to make only a restricted set of predictions, thereby simplifying the learning problem. These partial models may be directly useful for making decisions or may be combined together to form a more complete, structured model. However, in partially observable (non-Markov) environments, standard model-learning methods learn generative models, i.e. models that provide a probability distribution over all possible futures (such as POMDPs). It is not straightforward to restrict such models to make only certain predictions, and doing so does not always simplify the learning problem. In this paper we present prediction profile models: non-generative partial models for partially observable systems that make only a given set of predictions, and are therefore far simpler than generative models in some cases. We formalize the problem of learning a prediction profile model as a transformation of the original model-learning problem, and show empirically that one can learn prediction profile models that make a small set of important predictions even in systems that are too complex for standard generative models.
"An agent behaving in an unknown environment may wish to learn a model that allows it to make pre... more "An agent behaving in an unknown environment may wish to learn a model that allows it to make predictions about future events and to anticipate the consequences of its actions. Such a model can greatly enhance the agent’s ability to make good decisions. However, in environments like the physical world in which we live, which is stochastic, partially observable, and high dimensional, learning a model is a challenge.
One natural approach when faced with a di�?cult model learning problem is not to model the entire system. Instead, one might focus on capturing the most important aspects of the environment and give up on modeling complicated, irrelevant phenomena. This intuition can be formalized using partial models, which are models that make only a restricted set of (abstract) predictions in only a restricted set of circumstances. Because a partial model has limited prediction responsibilities, it may be signi�?cantly simpler than a complete model.
Partial models (and similar ideas) have been studied in many contexts, mostly under the Markov assumption, where the agent is assumed to have access to the full state of the world. In this setting, predictions can typically be learned directly as functions of state and the process of learning a partial model is often as simple as estimating only the desired predictions and omitting the rest from the model. As such, much of the relevant work has focused on the interesting and challenging question of which partial models should be learned (rather than how to learn them). In the partially observable case, however, where the state of the world is (more naturally) assumed to be hidden from the agent, just the basic problem of how to learn a partial model poses signi�?cant challenges.
The goal of this thesis is to provide general results and methods for learning partial models in partially observable systems. Some of the main challenges posed by partial observability are formalized and learning methods are developed to address some of these issues. The learning methods presented are demonstrated empirically to be able to learn partial models in systems that are too complex for standard, complete model learning methods. Finally, many partial models are learned and composed to form complete models that are used for model-based planning in high dimensional arcade game examples."
Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI), 2009
A common approach to the control problem in partially observable environments is to perform a dir... more A common approach to the control problem in partially observable environments is to perform a direct search in policy space, as de�?ned over some set of features of history. In this paper we consider predictive features, whose values are conditional probabilities of future events, given history. Since predictive features provide direct information about the agent’s future, they have a number of advantages for control. However, unlike more typical features de�?ned directly over past observations, it is not clear how to maintain the values of predictive features over time. A model could be used, since a model can make any prediction about the future, but in many cases learning a model is infeasible. In this paper we demonstrate that in some cases it is possible to learn to maintain the values of a set of predictive features even when a learning a model is infeasible, and that natural predictive features can be useful for policy-search methods.
Advancements in Neural Information Processing Systems 22 (NIPS), 2009
We present a novel mathematical formalism for the idea of a “local model” of an uncontrolled dyna... more We present a novel mathematical formalism for the idea of a “local model” of an uncontrolled dynamical system, a model that makes only certain predictions in only certain situations. As a result of its restricted responsibilities, a local model may be far simpler than a complete model of the system. We then show how one might combine several local models to produce a more detailed model. We demonstrate our ability to learn a collection of local models on a large-scale example and do a preliminary empirical comparison of learning a collection of local models and some other model learning methods.
Proceedings of the Tenth International Symposium on Artificial Intelligence and Mathematics (ISAIM), 2008
We consider an agent that seeks to make abstract predictions about the world by only distinguishi... more We consider an agent that seeks to make abstract predictions about the world by only distinguishing between certain features of observations. Making accurate abstract predictions of this form may not require a fully detailed model of the world, though in general it will require that the agent make some �?ner distinctions than those that interest it. We assume the agent has a partition of the observation space of a dynamical system induced by the features of interest. The goal of this paper is to �?nd a minimal re�?nement of that partition such that a model of the re�?ned system will make accurate predictions with respect to the features of interest. We provide algorithms and worst-case bounds on the di�?culty of performing this task. Our results apply generally to all discrete dynamical systems.
Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (IJCAI), 2007
A long-lived agent continually faces new tasks in its environment. Such an agent may be able to u... more A long-lived agent continually faces new tasks in its environment. Such an agent may be able to use knowledge learned in solving earlier tasks to produce candidate policies for its current task. There may, however, be multiple reasonable policies suggested by prior experience, and the agent must choose between them potentially without any a priori knowledge about their applicability to its current situation. We present an “experts” algorithm for ef�?ciently choosing amongst candidate policies in solving an unknown Markov decision process task. We conclude with the results of experiments on two domains in which we generate candidate policies from solutions to related tasks and use our experts algorithm to choose amongst them.
Uploads
Papers by Erin J . Talvitie
One natural approach when faced with a di�?cult model learning problem is not to model the entire system. Instead, one might focus on capturing the most important aspects of the environment and give up on modeling complicated, irrelevant phenomena. This intuition can be formalized using partial models, which are models that make only a restricted set of (abstract) predictions in only a restricted set of circumstances. Because a partial model has limited prediction responsibilities, it may be signi�?cantly simpler than a complete model.
Partial models (and similar ideas) have been studied in many contexts, mostly under the Markov assumption, where the agent is assumed to have access to the full state of the world. In this setting, predictions can typically be learned directly as functions of state and the process of learning a partial model is often as simple as estimating only the desired predictions and omitting the rest from the model. As such, much of the relevant work has focused on the interesting and challenging question of which partial models should be learned (rather than how to learn them). In the partially observable case, however, where the state of the world is (more naturally) assumed to be hidden from the agent, just the basic problem of how to learn a partial model poses signi�?cant challenges.
The goal of this thesis is to provide general results and methods for learning partial models in partially observable systems. Some of the main challenges posed by partial observability are formalized and learning methods are developed to address some of these issues. The learning methods presented are demonstrated empirically to be able to learn partial models in systems that are too complex for standard, complete model learning methods. Finally, many partial models are learned and composed to form complete models that are used for model-based planning in high dimensional arcade game examples."
One natural approach when faced with a di�?cult model learning problem is not to model the entire system. Instead, one might focus on capturing the most important aspects of the environment and give up on modeling complicated, irrelevant phenomena. This intuition can be formalized using partial models, which are models that make only a restricted set of (abstract) predictions in only a restricted set of circumstances. Because a partial model has limited prediction responsibilities, it may be signi�?cantly simpler than a complete model.
Partial models (and similar ideas) have been studied in many contexts, mostly under the Markov assumption, where the agent is assumed to have access to the full state of the world. In this setting, predictions can typically be learned directly as functions of state and the process of learning a partial model is often as simple as estimating only the desired predictions and omitting the rest from the model. As such, much of the relevant work has focused on the interesting and challenging question of which partial models should be learned (rather than how to learn them). In the partially observable case, however, where the state of the world is (more naturally) assumed to be hidden from the agent, just the basic problem of how to learn a partial model poses signi�?cant challenges.
The goal of this thesis is to provide general results and methods for learning partial models in partially observable systems. Some of the main challenges posed by partial observability are formalized and learning methods are developed to address some of these issues. The learning methods presented are demonstrated empirically to be able to learn partial models in systems that are too complex for standard, complete model learning methods. Finally, many partial models are learned and composed to form complete models that are used for model-based planning in high dimensional arcade game examples."