"CONVERSE"

INTRODUCTION

"CONVERSE" consists of a number of independent software modules which are co-ordinated by a shell structure. The total extent of the code is approximately 600 KB, plus 28.8 MB of data. The software can currently converse intelligently on 60 different topics, using its knowledge about several thousand people and places. In addition it can provide answers to questions about any of the people and places using this same knowledge base. CONVERSE employs a parser written by Prospero Software, a UK company, and a lexical database called "WordNet" created at Princeton University under the direction of Professor George Miller.

When a topic of conversation has been exhausted or when there is a clear indication that the user wants to talk about something else CONVERSE will switch topics. In order to avoid boring the user, once a topic has been exhausted it is not returned to until all other topics have also been discussed.

Currently the software has the persona of a woman called Catherine who is a journalist. She is British but lives in the States. Her personality and facts about her are stored in a "Person Database" (PDB). Different personalities can be added simply by creating new data for the Person Database.

One commercial application of the system is that CONVERSE can be made, by adding the relevant data, to provide information and help to the user on any subject. For example, a corporation that has a help line to assist customers in the use of their products could employ a version of CONVERSE containing data about each of its products. Another application is providing on-line help to computer users who are having difficulty with a software package - data could be written for each aspect of that software package and CONVERSE would then be able to deal with natural language questions related to the use of the package. A socially useful application for CONVERSE is simply as a conversational partner or "virtual friend".

CATEGORIES OF MODULE

There are five categories of software module in CONVERSE. The input modules function independently of the rest of the system. Their function is to provide a number of "clauses" which represent the user's latest utterance. The dispatch module (called "Where-To-Go") and the action modules process the parser output - they store extracted information and produce one or more outputs which make up the program's next utterance. The method of extracting information from the post-parser (PPP) output is called the "microQuery" system. This system is consulted by any module seeking information about the PPP output.

The action modules with the dispatcher together make the main engine of the system. Some modules retain information about the current state of the conversation. Other modules (such as the "SpecialCaseBox") hold no state information between utterances but are simply triggered as required by the rest of the system. Data storage modules contain information which can be accessed by any portion of the system and provide an interface for storing and retrieving the data.

SYSTEM OVERVIEW

This section contains a brief outline of the data flow through the system from the point of view of the shell.

A system cycle consists of the following:

• The shell prompts the user for an input utterance.
• The utterance is spell-corrected and elisions are expanded.
• Input cycle:
  • The utterance is divided into its sentences and the head sentence is separated.
  • The shell passes the current separated sentence to the pre-processor tagger and stores the resultant tags structure.
  • The shell calls the Prospero tagger and the Prospero parser with the current sentence.
  • The shell calls the Parser Post-Processor (PPP) with its pre-processor tags and the Prospero outputs. The output "clause" is stored by the shell.
  • This input cycle loop is repeated with the next portion of the user's utterance until no further sentences remain.
    

• The clause stack is copied (by the shell) onto the stack of previous user utterances. There is a special module to facilitate storage and retrieval of the user's previous utterances.
• The shell now calls the "Where to Go" (WTG) module with the clause stack. WTG passes each clause to each of the action modules to obtain "bids" which determine which action module processes the clause. After the bidding cycle the winning action modules are called again by WTG and told to process the clause(s) for which their bid was accepted. WTG stores the text output of the action modules and produces an output array.
• The shell then passes this output array to the text generator, which decides exactly how to express what it wants to say.
• The final output utterance is then printed by the shell and the next system cycle is started.

INPUT MODULES

This is the first module to examine the user's input utterance. Each sentence is spell-checked and corrected in order to form the sentence which is passed to the later modules. An acronym / elision expansion is employed in this module. (For more on elisions see section i2 below.)

i2 Pre-processor elision expansion

This module replaces words such as I'll, we'll, they've and it's with their expanded forms (I will, we will, they have, it is). The elisions and their expansions are stored explicitly as a table.

i3 Pre-processor tagger

This module provides additional information to the PPP, which augments output from the Prospero tagger and parser.

Only one sentence at a time from the corrected user utterance is tagged. Currently person names, place names and dates are tagged. First names are tagged with an identified sex (or as being unisex). Places are tagged with extra "place information" data.

i5 Trivial punctuation correction

This module rectifies the most trivial punctuation problems, where the last sentence in the user's utterance is not terminated with a full-stop or a question mark. The module decides whether or not this sentence is a question and adds a missing full stop or question mark as appropriate.

i6 Prospero tagger

The Prospero tagger marks words with their parts-of-speech in preparation for the parsing stage.

i7 Parser Post-processor (PPP)

This module examines the Prospero output and builds a stack of "clauses" which, between them, represent simplifications of all the sentences in the user's utterance. These clauses can therefore be used by the rest of the program to determine what the user utterance consisted of.

The PPP's tasks are two-fold: to simplify the input utterance as parsed by Prospero and to fill holes in the original Prospero parser. The PPP output is used by the microQuery system (see section d3) as the data provider for the query mechanism.

DISPATCHER MODULE

WTG presents each clause from the input to each action module in turn, keeping track of the bids for each clause from each agent. An action module's agent returns a bid, describing that agent's confidence that it has a valid response to the current clause. This bid lies between 0 (no bid -"definitely don't ask me to process this"), to 100 ("The user said something which I understood 100% and to which I can respond"). Some special cases may require a clause to have more than one winner.

The calculation of how well a user utterance is understood depends on the action modules themselves. Their schemes are described in sections a1-a4.

After the bidding is complete WTG decides which clauses are presented to which action module. Each clause can only be processed by one action module. Some clauses may not be processed at all. Those that are not processed are presented to the information scavenger module (see a3). If an action module wins a bid and is presented a clause for processing, then it will store any data that it extracts from this clause in the Person Database.

The Special Case module may sometimes have text which it wishes to add to any other output created for this clause by the system. This is signalled to the WTG module by bidding between 1001 and 1100 for the clause.

ACTION MODULES

This module stores CONVERSE's knowledge about a particular topic. The topic agent is responsible for checking that the conversation is still running in what it considers to be a sensible way.

A weighting system is employed, both to jump around topics based on the user's utterance and to guide the conversation at the end of a topic. When CONVERSE comes to the end of its knowledge about a topic the system switches to the most relevant new topic.

a2 Pisces and Agent

Pisces answers the user's questions and (where it can) replies to requests. All questions that can be categorised are currently answered. "What is a X?" or "Who is Y?" is answered using information contained in WordNet.

The Pisces module fields questions which can be answered from any of the databases (typically the Person Database). However, questions can come in a number of forms - they can be simple questions, or commands / requests eliciting information. The forms of question that can be answered with content are definitions (from WordNet), queries about the system ("What's your name?" etc.) and queries about previously presented information (e.g. "What did I tell you about my cat?")

In providing a mechanism to answer questions about itself this module also helps to create the system's personality.

a3 Information Scavenger

The function of this module is to extract any possible information from the given clause if the clause is a statement. This module is the penultimate one called by WTG. It is called only if no other action modules have extracted information from the current clause.

The implementation of this module relies on functions provided by the Tag Formation Module (TFM). By forming a tagset from the input clause and using the PDB's fuzzy matching routine to see if similar information is already present in the PDB, the module will attempt to build a piece of information that can be added to the database.

a4 Special Case and Agent

The Special Case module takes care of situations such as rudeness and rubbish typed by the user. There are also a few situations where Special Cases looks for slightly more complex problems - the user repeating himself exactly or repeating, using different words, information which the user provided earlier in the same conversation (or in an earlier conversation).

WTG calls the Special Case module last because this module requires information as to what all the other action modules have extracted.

OUTPUT MODULE

This module determines exactly what to say to the user, i.e. it converts what the system wishes to impart to the user into the exact form of words to be used in imparting it.

DATA MODULES

WordNet is a lexical database which stores not only words and definitions but also hierarchical information. WordNet is used in several ways:

1. By the PPP:

   For word "un-inflection" or morphology (in terms of the function that is required).

2. By microQueries:

   Constituent nanoQueries can search for matches in WordNet's synonym sets and genus hierarchies. For example, a query may search for the parents of a user's utterance word, such as "Persian" for "cat", or "red" for "colour"

3. By Pisces:

   Pisces requires word definitions. These are supplied by WordNet.

An additional "layer" on top of WordNet has been added, allowing higher-level access routines to be written. These routines have access to internal WordNet data and can optimise performance to more than is allowed by the sole use of the low-level function. This extra layer is provided as "WNextras". One example of this is the WN_HyperDefinition function, a routine which provides a definition for a given sense of a word. If the word does not have a definition the hierarchy is traversed upwards until a definition is found. This is the definition call used by the Pisces module.

The WordNet database has been augmented by adding proper names and their definitions (taken under licence from a Harper-Collins dictionary).

d2 Person Database

This is the module which is responsible for storing all the information for each user, as well as information about the system's own persona - Catherine.

Pisces answers most questions by fuzzy matching with the PDB. By calling the TFM (see section d5 below), a potential tagset will be formed from the user's utterance. This tagset will be passed to the fuzzy-matching portion of the PDB, which will attempt to find a "closest match" tagset within the database. Given a good enough probability that the tagset matches, Pisces will then be able to supply a response, hopefully answering the question.

d3 Query (microQuery) providers

MicroQueries are used in order to extract information from the output of the parsing stage.

d4 Dialog History

This module stores PPP output clauses so that, for example, they may be matched to check for repetition.

d5 Person Database Tag Formation Module (TFM)

This module provides a basis for Pisces and the Information Scavenger. Given an input clause, a set of functions will be provided allowing a tagset to be derived.

The module will attempt to build a tagset by identifying the subject of any information to be extracted, identifying other attributes and actions from the sentence and trying to match them to possible tags within the Person Database.

For example: "What colour are my eyes?" should yield a tagset such as

USER SELF EYE COLOUR BE WHAT DATA

* * * * * * *
For further information about CONVERSE please contact:
David Levy,
Intelligent Research Ltd.,
89, Constantine Road,
London NW3 2LP,
England.
Tel: +44 171 485 9146
Fax: +44 171 482 0672
e-mail: DavidL@intrsrch.demon.co.uk