Application Information and Rules for the 2002 Loebner Prize

The Loebner Prize Medal and a cash award is presented annually to the designer of the computer system that best succeeds in passing a variant of the Turing Test.

The Loebner Prize Competition in Artificial Intelligence was established in 1990 by Hugh Loebner and was first held at the Boston Computer Museum, Boston, Massachusetts, USA in 1991.

In accordance with the requirements of the Donor, as published in the June 1994 Communications of the ACM, the winner of the US$100,000 Gold Prize must be prepared to deal with audio visual input, and appropriate competitions will be held once Competitors have reached Turing's 50:50 likelihood level of being mistaken for a human.

An intermediate Silver Prize of US$25,000 will be offered for reaching this level in a text-only test.

There is also a Bronze Prize which is awarded annually to the designer of the "most human computer" as rated by a panel of Judges.

In 2002, this Bronze Prize will be worth $2,000.

There is no entry fee.

Applications must be accompanied by e-mail protocols recording interactions between the computer system to be entered and one or more human subjects. Protocols must not exceed 2500 words.

Applications must be submitted by e-mail to the Cambridge Center. The Cambridge Center may request an opportunity to interact with Computer Entries. e-mail to: center@behavior.org

Please review the accompanying rules carefully and provide the following information:

OFFICIAL RULES

2002 Loebner Prize Competition in Artificial Intelligence

  1. The objective of the Loebner Prize Competition in Artificial Intelligence is to identify the computer system that can best succeed in passing a modern variant of the Turing Test. Judges will attempt to distinguish computer systems ("Computer Entries") from one or more human beings ("Human Confederates") based on interactions with these Computer Entries.

  2. Contestants may be individuals, organisations, businesses, schools, corporations, institutions, or other entities. Individuals need not have institutional affiliations. Contestants may be of any nationality or age. Parental consent is required for Contestants who are aged under 18. Contestants may submit only one entry during any year.

  3. If there are two or more Computer Entries there will be a Loebner Prize contest. The Medal and cash award will go to the Contestant whose Computer Entry wins the highest score, based on the decision of the Judges.

  4. If there is only one Computer Entry, the Loebner Prize Medal and the US$2000 cash award will be awarded to that Entry.

  5. If there are no Computer Entries in the 2002 Loebner Prize Competition, the US$2000 cash prize will be added to the cash award for 2003, making the award for 2003 US$4000.

  6. If a Computer Entry passes the text-only Turing Test the Contestant will be awarded the Silver Prize of a Silver Medal and US$25,000.

  7. If a Computer Entry passes the Turing Test dealing with audio visual input the Contestant will be awarded the Gold Prize of US$100,000 and a Gold Medal.

  8. The closing date for applications for the 2002 Competition is 3 September 2002.
  9. No more than eight finalists will be chosen to compete in a simultaneous and real-time competition on 12 October 2002 at the Cambridge Center, Concord Massachusetts (date and location subject to change). Pre-selection will be carried out by the Cambridge Center.

  10. The selection process may entail interaction between Cambridge Center personnel and Computer Entries. Contestants will be notified of the selection decision by 16 September 2002.

  11. Judges will be selected by the Cambridge Center and may include children, people with disabilities, experts in Psychology, Linguistics or Artificial Intelligence, journalists, etc and are intended to be in part representative of the community at large and part of the cognitive science community.

  12. It is the task of the Computer Entries to respond to the communications of the Judges in such a manner as to imitate the responses of a human. The decision of the Cambridge Center will be final. The Competition is widely publicized, the transcripts will be published, and all Contestants must assume responsibility for their Computer Entries. As in any normal conversation, both participants have a role in determining the direction of the conversation and are free to decide what topics they do or do not wish to discuss. Computer Entries must be prepared to communicate for an indefinite period of time.

  13. Judges will have at least one opportunity to interact with each of the Computer Entries (on the computer terminals available concurrently) during the Competition. Judges will be allowed unrestricted communications. They will be informed that at least one of the terminals is controlled by Human Confederates and that at least two of the terminals are controlled by Computer Entries.

  14. Judges will not be allowed to interact with each other and will be instructed to provide individual ratings of each of the appropriateness and responsiveness of the replies.

  15. Computer Entries must communicate in the English language.

  16. Computer Entries may contain standard or customised software and hardware. The system may be of any type as long as it contains no genetic material and as long as its replies are in no manner controlled by human beings or other organic systems in real time.

  17. Entries will be required to run on hardware located at the Cambridge Center or an alternative competition site or an associated site. NO TELECOMMUNICATIONS WILL BE PERMITTED. Finalists chosen to participate may submit their Computer Entries as programs recorded on standard machine-readable media (magnetic or optical storage) together with operating documentation. Computersrunning Microsoft (R)Windows will be available on-site for the Contest.

  18. Appropriate steps will be taken to prevent the unauthorised duplication or publication of Contestants' entries. However, neither the Cambridge Center nor any of its agents can guarantee absolute security. Contestants, by entering this Competition, confirm their understanding of this and agree that the Cambridge Center and its agents will be held harmless should there be any unauthorised duplication or publication of Computer Entries. Contestants who require absolute security will be allowed to operate their Computer Entries and/or provide their own hardware on-site.

  19. Computer Entries will be required to record the conversations as text files on magnetic media. The recordings will remain the property of the Cambridge Center, which will also retain the copyright on transcripts or other representations, magnetic or otherwise, of the recordings. By entering the Contest the Contestants shall assign copyright in the conversations recorded by the Computer Entries and in the contribution to those conversations made by the Computer Entries during the Competition and during the period of temporary display at the Cambridge Center.

  20. The Contestants will retain copyright and all other intellectual property ownership of their Entries.

  21. Unless the Contestants agree otherwise, the Cambridge Center will wipe all Competition Entries from the Competition hardware after the Competition.

  22. Applications must be accompanied by e-mail transcripts recording actual interactions between the system to be entered and one or more human beings. The protocols may not exceed 2500 words.

  23. The names 'Loebner Prize' and 'Loebner Prize Competition' may be used by contestants in advertising only by advance written permission of the Cambridge Center. Advertising is subject to approval by representatives of the Cambridge Center. Improper or misleading advertising may result in revocation of the Prize and/or other actions.

  24. If, in the opinion of the Judges, a Computer Entry is not compliant with the Rules, the Computer Entry will be disqualified.

  25. The Judges' decision will be final. Neither the Cambridge Center nor its agents accept liability in connection with the Prize.

Data Entry Protocol and File Format Key Entry -

Terminal Display Each Computer Entry program (also "program") will operate in two modes: "set-up" and "contest".

Upon initial execution each program will operate in set-up mode.

Set-up mode will permit contest officials to prepare the program for the contest. To that end, while in set-up mode the program will:

  1. request a file name to append the transcripts of the interaction to
  2. assume that set-up "judge00" is at the terminal and prepare an initial comment;
  3. prompt with a sign "+" requesting a response from "judge00" (contest official);
  4. interact with "judge00" in the normal way with the sole difference that the "+" sign is used as the prompt;
  5. write a transcript to the specified file as usual.

Contest mode

Upon receipt of the string "@@T[CR]" the program will toggle to "contest" mode and behave as follows:

  1. The key sequence "@,@,T,[CR]" when entered in response to a "+" prompt terminates the conversation with the set-up Judge, and will cause the program to clear the screen and make its initial comment to the first competition Judge.
  2. The key sequence "@,@,nn,[CR](nn=01-99)" when entered in response to a ">" prompt will indicate that a new Judge, "JUDGEnn", is now entering data. Each Judge will thus identify himself/herself when moving to a new terminal.
    NOTE This is a change from previous contests. Only a single [CR] is necessary following any @@ command.
  3. The program will make an initial response to the Judges' input.
  4. The program's response may be multiline.
  5. Upon completion of the program's response, the program will prompt the Judge for input with a ">" character and then echo on the screen, character by character, the Judge's entry.
  6. Judges' questions and comments can be multi-line. Each question or comment will be entered one line at a time. Each line will be terminated by a carriage return [CR]. Judges will key in questions and comments in response to a ">" prompt from the program.
  7. Entry of two consecutive carriage returns will indicate that the Judge's question or comment is complete and that the terminal (program or human) must respond.
  8. The terminal will display character data in a monotype font (all characters of equal width).
  9. The key sequence "@,@,X,[CR]" when entered in response to a ">" prompt will cause the program to exit.

EXAMPLE: Assume program is responding to current interaction with Judge 3.

[some program comment-completed]

>@@04[CR]

(new Judge, number 4)
Welcome Judge 4 (Comment by program or human)
>Who do you think[CR] ( multiline question from Judge 4)
>was responsible for the[CR] (question continued line 2)
>Antrax attack on the postal service?[CR] (question ended (first [CR])
>[CR] (second [CR])
I think entities should not be multiplied (answer, line 1)
beyond necessity - nor should terrorists (answer, line 2)
It was whoever attacked the WTC (answer, line 3
> (Cursor waits for input. ">" prompt indicates that Judge is to respond)

Data File Format - Intent: Each program entered in the Loebner Prize Contest will produce a text file transcript of the interactions with Judges. The file should be readable by standard text-reader programs.

  1. Each Computer Entry in the 2002 Loebner Prize Contest will append to a text file on disk containing the transcript of keyboard input and program output. The program should not discard the existing contents of the file, but should open the file in append mode (the file should be created automatically if it does not exist yet).
  2. The file will be named according to the input during set-up operation.
  3. The file will be in ASCII text format suitable for input into a standard word processing program.
  4. The first three lines are headers containing the following: (c)2002 Cambridge Center, all rights reserved [Program Name] [Contestant Name] Start at: [YYYY/MM/DD HH:MM:SS]
  5. Each succeeding line will EITHER mirror one line displayed on the screen, preceded by the source ("JUDGEnn") or "PROGRAM") and time in brackets OR indicate a change of Judges: "***JUDGEnn***"

EXAMPLE: For the above interaction:

***JUDGE04***
PROGRAM[14:12:25] Welcome judge 4
JUDGE04[14:12:32] Who do you think
JUDGE04[14:12:39] was responsible for the
JUDGE04[14:12:55] Antrax attack in the postal service?
PROGRAM[14:13:15] I think entities should not be multiplied
PROGRAM[14:13:17] beyond necessity - nor should terrorists
PROGRAM[14:13:20] It was whoever attacked the WTC