Thesis Committee

Laura E. Ray, Ph.D. (Chair)

Christopher Amato, Ph.D.

Richard H. Granger, Ph.D.

Jerald D. Kralik, Ph.D.

Minh Q. Phan, Ph.D.

Abstract

This research focuses on learning space representation using reinforcement learning. In order to reduce the complexity of an agent’s learning space, two approaches are utilized: state abstraction and hierarchical state representation. Using state abstraction, a framework is proposed for satisficing state abstraction – one that automatically reduces state dimensionality by eliminating useless or irrelevant state dimensions via state-reward relevance testing. State abstraction approach is effective in improving the learning performance when the scale of the problem is limited. In order to solve large scale problem, we use the Q-Tree algorithm, a hierarchical state representation algorithm that concurrently learns task policy and state representations. The Q-Tree algorithm splits the state space into subspaces by evaluating state space separability, a feature based on the difference in Q-values for transitions that occur from each visited state.

Based on these learning space complexity reduction algorithms, we have developed computational frameworks for modeling insight problem solving. Insight problem solvers must overcome pre-potent responses and learn to restructure the state representation in order to reach a solution. The Q-Tree learning framework models pre-potent knowledge representation and subsequently restructures the pre-potent knowledge to reach a solution to an insight problem. The algorithm is evaluated on the nine-dot problem and the reverse reward problem and results show that the dynamics of incubation and insight can be elicited and observed. A Multi-level Problem solving model is presented as a more general framework for modeling complex insight problem solving. In this model, the state space of the problem domain is organized by a hierarchy of subsets of the original state space, and policies are learned under a subset of the state space using Q-Tree algorithm. A variant of the Wisconsin Card Sorting Test – a cognitive test of set shifting is successfully modeled by this framework.  Demonstration of insight problem solving using RL algorithms is conducted using a robot platform with constrained memory and computation capability. Additionally, the robot learning experiments start with visual percepts, as the platform perceives the environment information from a camera.

Finally, the thesis investigates application of the multi-level model to observational learning, in which a robot observes a human perform the Card Sorting task and learns the sorting rules through observation.

.