Soar (cognitive architecture)

Soar’s main processing cycle arises from the interaction between procedural memory (its knowledge about how to do things) and working memory (its representation of the current situation) to support the selection and application of operators. Information in working memory is represented as a symbolic graph structure, rooted in a state. The knowledge in procedural memory is represented as if-then rules (sets of conditions and actions), that are continually matched against the contents of working memory. When the conditions of a rule matches structures in working memory, it fires and performs its actions. This combination of rules and working memory is also called a production system. In contrast to most production systems, in Soar, all rules that match, fire in parallel.

Instead of having the selection of a single rule being the crux of decision making, Soar’s decision making occurs through the selection and applications of operators, that are proposed, evaluated, and applied by rules. An operator is proposed by rules that test the current state and create a representation of the operator in working memory as well as an acceptable preference, which indicates that the operator should be considered for selection and application. Additional rules match with the proposed operator and create additional preferences that compare and evaluate it against other proposed operators. The preferences are analyzed by a decision procedure, which selects the preferred operator and installs it as the current operator in working memory. Rules that match the current operator then fire to apply it and make changes to working memory. The changes to working memory can be simple inferences, queries for retrieval from Soar’s long-term semantic or episodic memories, commands to the motor system to perform actions in an environment, or interactions with the Spatial Visual System (SVS), which is working memory’s interface to perception. These changes to working memory lead to new operators being proposed and evaluated, followed by the selection of one and its application.

Soar supports reinforcement learning, which tunes the values of rules that create numeric preferences for evaluating operators, based on reward. To provide maximal flexibility, there is a structure in working memory where reward is created.

If the preferences for the operators are insufficient to specify the selection of a single operator, or there are insufficient rules to apply an operator, an impasse arises. In response to an impasse, a substate is created in working memory, with the goal being to resolve the impasse. Additional procedural knowledge can then propose and select operators in the substate to gain more knowledge, and either create preferences in the original state or modify that state so the impasse is resolved. Substates provide a means for on-demand complex reasoning, including hierarchical task decomposition, planning, and access to the declarative long-term memories. Once the impasse is resolved, all of the structures in the substate are removed except for any results. Soar’s chunking mechanism compiles the processing in the substate which led to results into rules. In the future, the learned rules automatically fire in similar situations so that no impasse arises, incrementally converting complex reasoning into automatic/reactive processing. Recently, the overall Universal Subgoaling procedure has been extended through a mechanism of goal-directed and automatic knowledge base augmentation that allows to solve an impasse by recombining, in an innovative and problem-oriented way, the knowledge possessed by a Soar agent.[3]

Symbolic input and output occurs through working memory structures attached to the top state called the input-link and the output-link. If structures are created on the output-link in working memory, these are translated into commands for external actions (e.g., motor control).

To support interaction with vision systems and non-symbolic reasoning, Soar has its Spatial Visual System (SVS). SVS internally represents the world as a scene graph, a collection of objects and component subobjects each with spatial properties such as shape, location, pose, relative position, and scale. A Soar agent using SVS can create filters to automatically extract features and relations from its scene graph, which are then added to working memory. In addition, a Soar agent can add structures to SVS and use it for mental imagery. For example, an agent can create a hypothetical object in SVS at a given location and query to see if it collides with any perceived objects.

Semantic Memory (SMEM) in Soar is designed to be a very large long-term memory of fact-like structures. Data in SMEM is represented as directed cyclic graphs. Structures can be stored or retrieved by rules that create commands in a reserved area of working memory. Retrieved structures are added to working memory.

SMEM structures have activation values that represent the frequency or recency of usage of each memory, implementing the base-level activation scheme originally developed for ACT-R. During retrieval, the structure in SMEM that matches the query and has the highest activation is retrieved. Soar also supports spreading activation, where activation spreads from SMEM structures that have been retrieved into working memory to other long-term memories that they are linked to.[4] These memories in turn spread activation to their neighbor memories, with some decay. Spreading activation is a mechanism for allowing the current context to influence retrievals from semantic memory.

Episodic Memory (EPMEM) automatically records snapshots of working memory in a temporal stream. Prior episodes can be retrieved into working memory through query. Once an episode has been retrieved, the next (or previous) episode can then be retrieved. An agent may employ EPMEM to sequentially play through episodes from its past (allowing it to predict the effects of actions), retrieve specific memories, or query for episodes possessing certain memory structures.

Each of Soar’s long-term memories have associated online learning mechanisms that create new structures or modify metadata based on an agent’s experience. For example, Soar learns new rules for procedural memory through a process called chunking and uses reinforcement learning to tune rules involved in the selection of operators.

Throughout its history, Soar has been used to implement a wide variety of classic AI puzzles and games, such as Tower of Hanoi, Water Jug, Tic Tac Toe, Eight Puzzle, Missionaries and Cannibals, and variations of the Blocks world. One of the initial achievements of Soar was showing that many different weak methods would naturally arise from the task knowledge that was encoded in it, a property called, the Universal Weak Method. [6]

The first large-scale application of Soar was R1-Soar, a partial reimplementation by Paul Rosenbloom of the R1 (XCON) expert system John McDermott developed for configuring DEC computers. R1-Soar demonstrated the ability of Soar to scale to moderate-size problems, use hierarchical task decomposition and planning, and convert deliberate planning and problem solving to reactive execution through chunking.[7]

NL-Soar was a natural language understanding system developed in Soar by Jill Fain Lehman, Rick Lewis, Nancy Green, Deryle Lonsdale and Greg Nelson. It included capabilities for natural language comprehension, generation, and dialogue, emphasizing real-time incremental parsing and generation. NL-Soar was used in an experimental version of TacAir-Soar and in NTD-Soar.[8]

The second large-scale application of Soar involved developing agents for use in training in large-scale distributed simulation. Two major systems for flying U.S. tactical air missions were co-developed at the University of Michigan and Information Sciences Institute (ISI) of University of Southern California. The Michigan system was called TacAir-Soar and flew (in simulation) fixed-wing U. S. military tactical missions (such as close-air support, strikes, CAPs, refueling, and SEAD missions). The ISI system was called RWA-Soar and flew rotary-wing (helicopter) missions. Some of the capabilities incorporated in TacAir-Soar and RWA-Soar were attention, situational awareness and adaptation, real-time planning and dynamic replanning, and complex communication, coordination, and cooperation among combinations of Soar agents and humans. These systems participated in DARPA’s Synthetic Theater of War (STOW-97) Advanced Concept Technology Demonstration (ACTD), which at the time was the largest fielding of synthetic agents in a joint battlespace over a 48-hour period, and involved training of active duty personnel. These systems demonstrated the viability of using AI agents for large-scale training.[9]

One of the important outgrowths of the RWA-Soar project was the development of STEAM by Milind Tambe,[10] a framework for flexible teamwork in which agents maintained models of their teammates using the joint intentions framework by Cohen & Levesque.[11]

NTD-Soar was a simulation of the NASA Test Director (NTD), the person responsible for coordinating the preparation of the NASA Space Shuttle before launch. It was an integrated cognitive model that incorporated many different complex cognitive capabilities including natural language processing, attention and visual search, and problem solving in a broad agent model.[12]

Soar has been used to simulate virtual humans supporting face-to-face dialogues and collaboration within a virtual world developed at the Institute of Creative Technology at USC. Virtual humans have integrated capabilities of perception, natural language understanding, emotions, body control, and action, among others.[13]

Game AI agents have been built using Soar for games such as StarCraft,[14] Quake II,[15] Descent 3,[16] Unreal Tournament,[17] and Minecraft, supporting capabilities such as spatial reasoning, real-time strategy, and opponent anticipation. AI agents have also been created for video games including Infinite Mario[18] which used reinforcement learning, and Frogger II, Space Invaders, and Fast Eddie, which used both reinforcement learning and mental imagery.[19]

Soar can run natively on mobile devices. A mobile application for the game Liar’s Dice has been developed for iOS which runs the Soar architecture directly from the phone as the engine for opponent AIs.[20]

Many different robotic applications have been built using Soar since the original Robo-Soar was implemented in 1991 for controlling a Puma robot arm.[21] These have ranged from mobile robot control to humanoid service REEM robots,[22] taskable robotic mules[23] and unmanned underwater vehicles.[24]

A current focus of research and development in the Soar community is Interactive Task Learning (ITL), the automatic learning of new tasks, environment features, behavioral constraints, and other specifications through natural instructor interaction.[25] Research in ITL has been applied to tabletop game playing[26] and multi-room navigation.[27]

Early on, Merle-Soar demonstrated how Soar could learn a complex scheduling task modeled after the lead human scheduler in a windshield production plant located near Pittsburgh.[28]