(Rendered into HTML by HGL 15 March 1999)
 
 

DATED   3 February1999
 
 

THE BOARD OF TRUSTEES OF

THE SCIENCE MUSEUM

- and -

HUGH GENE LOEBNER

______________________________________________  
 

AGREEMENT

for

running the Loebner Prize Competition

______________________________________________

FARRER & CO

66 Lincoln's Inn Fields

London WC2A 3LH

CONTENTS

Clause
1	Definitions
2.	Condition Precedent
3.	Museum's Obligations
4.	Founder's Obligations
5.	Prizes
6.	Duration
7.	Contestants' Intellectual Property
8.	Rights in the Prize
9.	Acknowledgments
10.	Sponsorship by Third Parties
11.	New Prize
12.	Termination
13.	Notices
14.	Entire Agreement
15.	Waiver
16.	Governing Law

BETWEEN:

"Annual Bronze Prize"	means the prize awarded to the best Entry in any one year which is not capable of winning either the Gold Prize or the Silver Prize;
"Competition"	means the annual competition created by the Founder in 1990;
"Contestants"	means the persons, teams or organisations who participate in the Competition;
"Dartmouth College"	means Dartmouth College, New Hampshire, USA, which is to run the Competition in 2000;
"Entries"	means the software programs submitted by the Contestants;
"Gold Prize"	means the prize awarded for the entry which is able to pass an updated variant of the Turing Test dealing with audio and visual input/the state of the art technology of the day;
"Medals"	means the Gold, Silver and Bronze Medals which have been struck for the Competition;
"Prize Fund"	means the sum of US$125,000 provided by the Founder for the Competition;
"Rules"	means the draft rules for the Competition as set out in Schedule 1;
"Silver Prize"	means the prize awarded to the first entry able to pass a text-only Turing Test.
"Turing Test"	means the test invented by Turing in 1950.

 

AS WITNESS the signatures of the parties or their authorised signatories on the date first before written  
 

SCHEDULE ONE

Draft Application Information and Rules for the 2001 Loebner Prize

Presented by the Science Museum

(These Rules will be distributed with suitable amendments each year)

The Loebner Prize Medal and cash award is awarded annually to the designer of the computer system that best succeeds in passing a variant of the Turing Test.

The Loebner Prize Competition in Artificial Intelligence was established in 1990 by Hugh Loebner and was first held at the Boston Computer Museum in 1991. The Competition is to be run by Dartmouth College, USA, in 2000 and thereafter by the Science Museum, London.

In accordance with the requirements of the Founder, Dr Loebner, as published in the June 1994 Communications of the ACM, the winner of the US$100,000 Gold Prize must be prepared to deal with audio visual input, and appropriate competitions will be held once Competitors have reached Turing's 50:50 likelihood level of being mistaken for a human. An intermediate Silver Prize of US$25,000 will be offered for reaching this level in a text-only test. There is also an Annual Bronze Prize which is awarded to the designer of the "most human computer" as rated by a panel of Judges. In 2001, this prize will be worth $2000.

There is no entry fee.

Applications must be accompanied by e-mail protocols recording interactions between the computer system to be entered and one or more human subjects. Protocols must not exceed 2,500 words. Applications must be submitted by e-mail to the Science Museum. The Science Museum may request an opportunity to interact with Computer Entries. Transcripts and scores from the [1991, 1992, 1993 and 1994] [and 1995 to 1999?] Competitions may be obtained by contacting the Cambridge Center for Behavioral Studies at 617-491-9020 (e-mail center@behavior.org). Please review the accompanying rules carefully and provide the following information:-

• Name of Applicant:
• Age of Applicant:
• Proof of parental consent if aged under 18:
• Professional title of Applicant (if applicable):
• Affiliation (if applicable):
• Contact person (if applicable):
• Postal address (with postcode and country):
• Fax number (with international country and area code):
• Telephone number(s):
• E-mail address(es):
• Computer hardware to be employed:
• Computer software to be employed (optional):
• Media to be employed for submitting software:
• Enclose a protocol recording interactions between the computer system to be entered and one or more human subjects (maximum 2,500 words).
• E-mail completed applications to: s.foster@nmsi.ac.uk

Sabiha Foster, Science Museum
 

2001 Loebner Prize Competition in Artificial Intelligence

Presented by the Science Museum

1. The objective of the Loebner Prize Competition in Artificial Intelligence is to identify the computer system that can best succeed in passing a modern variant of the Turing Test. Judges will attempt to distinguish computer systems ("Computer Entries") from one or more human beings ("Human Confederates") based on interactions with these Computer Entries.
   
2. Contestants may be individuals, organisations, businesses, schools, corporations, institutions, or other entities. Individuals need not have institutional affiliations. Contestants may be of any nationality or age. Parental consent is required for Contestants who are aged under 18. Contestants may submit only one entry during any year.
   
3. If there are two or more Computer Entries there will be a Loebner Prize contest. The Medal and cash award will go to the Contestant whose Computer Entry wins the highest score, based on the decision of the Judges.
   
4. If there is only one Computer Entry, the Loebner Prize Medal and the US$2,000 cash award will be awarded to that Entry.
   
5. If there are no Computer Entries in the 2001 Loebner Prize Competition, the US$2,000 cash prize will be added to the cash award for 2002, making the award for 2002 US$4,000.
   
6. If a Computer Entry passes the text-only Turing Test the Contestant will be awarded the Silver Prize of a Silver Medal and $25,000.
   
7. If a Computer Entry passes the Turing Test [dealing with audio visual input] the Contestant will be awarded the Gold Prize of $100,000 and a Gold Medal.
   
8. The closing date for applications for the 2001 Competition is [                  ] 2001. No more than eight finalists will be chosen to compete in a [simultaneous and real-time] competition on [                  ] 2001 at the Science Museum, London. Pre-selection will be carried out by the Science Museum. The decision of the Science Museum will be final.
   
9. The selection process may entail interaction between Science Museum personnel and Computer Entries. Contestants will be notified of the selection decision by [                    ] 2001, although this may be provisional contingent on appropriate arrangements being possible to accommodate unusual hardware.
   
10. No more than eight Entries will be selected as finalists to compete in a [real-time and simultaneous] contest to be held on [                              ] 2001 at the Science Museum (date and location subject to change).
    
11. Judges will be selected by the Science Museum and may include children, people with disabilities, experts in Psychology, Linguistics or Artificial Intelligence, etc and are intended to be in part representative of the community at large and in part of the cognitive science community.
    
12. It is the task of the Computer Entries to respond to the communications of the Judges in such a manner as to imitate the responses of a human. No constraints will be placed on the Judgesí, Human Confederatesí or Contestantsí conversations. However, the Competition is widely publicised, the transcripts will be published, and all Contestants must assume responsibility for their own Computer Entries. As in any normal conversation, both participants have a role in determining the direction of the conversation and are free to decide what topics they do or do not wish to discuss. Computer Entries must be prepared to communicate for an indefinite period of time.
    
13. Judges will have one or more opportunities to interact with each of the Computer Entries [on the computer terminals available concurrently] during the Competition. Judges will be allowed unrestricted communications. They will be informed that at least one of the [terminals] is controlled by Human Confederates and that at least two of the [terminals] are controlled by Computer Entries [or re-word for sequential format with only one terminal].
    
14. One or more referees may be present to limit communications of the Human Confederates.
    
15. Judges will not be allowed to interact with each other and will be instructed to provide individual ratings of each of the appropriateness and responsiveness of the replies.
    
16. Computer Entries must communicate in the English language.
    
17. Computer Entries may contain standard or customised software and hardware. The system may be of any type as long as it contains no genetic material and as long as its replies are in no manner controlled by human beings or other organic systems in real time.
    
18. Entries will be required to run on hardware located at the Science Museum or an alternative competition site or an associated site. NO TELECOMMUNICATIONS WILL BE PERMITTED. Finalists chosen to participate may submit their Computer Entries as programs recorded on standard machine-readable media (magnetic or optical storage) together with operating documentation. Macintosh, DOS/Windows and Unix computers with standard mass storage peripherals will be available on-site for the contest. Efforts will be made to secure the use of other standard equipment as required, but this cannot be guaranteed. Where appropriate hardware cannot be provided by the organisers, Entrants will be required to provide/arrange the hardware for operation at an agreed site. Science Museum personnel will be available to operate the Computer Entries according to documentation provided by the Contestant.
    
19. Appropriate steps will be taken to prevent the unauthorised duplication or publication of Contestantsí entries. However, neither the Science Museum nor any of its agents can guarantee absolute security. Contestants, by entering this Competition, confirm their understanding of this and agree to hold the Science Museum and its agents harmless should there be any unauthorised duplication or publication of Computer Entries. Contestants who require absolute security will be allowed to operate their Computer Entries and/or provide their own hardware on-site. Computer Entries may be used by the Science Museum for demonstration or publicity purposes after the contest in a temporary exhibition lasting no more than one month.
    
20. Computer Entries will be required to record the conversations as text files on magnetic media. The recordings will remain the property of the Science Museum, which will also retain the copyright on transcripts or other representations, magnetic or otherwise, of the recordings. By entering the contest the Contestants shall assign copyright in the conversations recorded by the Computer Entries and in the contribution to those conversations made by the Computer Entries during the Competition and during the period of temporary display at the Science Museum.
    
21. The Contestants will retain copyright in their Entries and will grant to the Science Museum a licence to use the Computer Entries for the judging and promotion of the Competition and/or the temporary exhibition but for no other purpose.
    
22. Unless the Contestants agree otherwise, the Science Museum will wipe all Competition Entries from the Competition hardware after the Competition and the temporary exhibition.
    
23. Applications must be accompanied by e-mail transcripts recording actual interactions between the system to be entered and one or more human beings. The protocols may not exceed 2,500 words.
    
24. The names "Loebner Prize" and "Loebner Prize Competition" may be used by contestants in advertising only by advance written permission of the Science Museum. Advertising is subject to approval by representatives of the Science Museum. Improper or misleading advertising may result in revocation of the Prize and/or other actions.
    
25. To avoid giving Judges clues as to the nature of the Computer Entries, Computer Entries run from a shell should use the default I/O (stdin/stdout). Computer Entries run in a windowed environment should use black on white in a 25x80 window or full screen mode, using the standard font, and should seek to mimic the Confederate communications program (download or screendump possible): it displays a menu bar with Connection Font TTY View Help. Providing a View menu option to set the title (for the title bar of the window) is desirable but not obligatory. Since ideally all Computer Entries will be connected to the communications program over serial lines or (preferably) via a network, like the Human Confederates, Computer Entries are advised to provide a commandline or Communication menu option to communicate via a serial line or TCP socket, but are again not obliged to do so.
    
26. If in the opinion of the Judges a Computer Entry is not compliant with the Rules, the Computer Entry will be disqualified.
    
27. The Judgesí decision will be final. Neither the Science Museum nor its agents accept liability in connection with the Prize.
     
     

    ---

Data Entry Protocol and File Format

Key Entry - Terminal Display

Each Computer Entry program (also "program") will operate in two modes: "set-up" and "contest". Upon initial execution each program will operate in set-up mode.

Set-up mode will permit contest officials to prepare the program for the contest. To that end, while in set-up mode the program will:-

1. request a file name to append the transcripts of the interaction to;
2. assume that "judge00" is at the terminal and prepare an initial comment;
3. prompt with a sign "+" requesting a response from "judge00" (contest official);
4. interact with "judge00" in the normal way with the sole difference that the "+" sign is used as the prompt;
5. write a transcript to the specified file as usual.

Contest mode.

Upon receipt of the string "@@T[CR][CR]" the program will toggle to "contest" mode and behave as follow:-

1. The key sequence @,@,T,[CR][CR] when entered in response to a ">" or "+" prompt terminates the conversation with the current Judge, and will cause the program to clear the screen, prompt with a "?" and await the next Judge.
2. The key sequence @,@,nn,[CR][CR](nn=01-99) when entered in response to a ">" or "?" prompt will indicate that a new Judge, Judge nn, is now entering data. Each Judge will thus identify himself/herself when moving to a new terminal.
3. The program will make an initial response to the Judgeís input.
4. The programís response may be multiline.
5. Upon completion of the programís response, the program will prompt the Judge for input.
6. The computer will prompt the Judge with a ">" character and then echo on the screen, character by character, the Judgeís entry.
7. Judgesí questions and comments can be multiline. Each question or comment will be entered one line at a time. Each line will be terminated by a carriage return [CR]. Judges will key in questions and comments in response to a ">" prompt from the program.
8. Entry of two consecutive carriage returns will indicate that the Judgeís question or comment is complete and that the terminal (program or human) must respond.
9. The terminal will display character data in a monotype font (all characters of equal width).
10. The key sequence @,@,X,[CR],[CR] when entered in response to a ">" or "?" prompt will cause the program to exit.

EXAMPLE: Assume program is responding to current interaction with Judge 3.

[some program comment-completed]

>@@04[CR]	( New Judge, number 4 )
>[CR]	( two lines required )
Welcome judge 4	( Comment by program (or human)
> Do you think that the [CR]	( Multiline question from judge 4 )
> Republicans can succeed [CR]	( question continued line 2 )
> in winning the White House? [CR]	( question ended (first [CR])
> [CR]	( second [CR] )
Only if Newt succeeds in	( Answer, line 1 )
Developing a more tolerant image.	( Answer, line 2 )
>	( Cursor waits for input ">" prompt - judge to respond )

Data File Format

Intent: Each program entered in the Loebner Prize Contest will produce a text file transcript of the interactions with Judges. The file should be readable by standard text-reader programs.

1. Each Computer Entry in the 2001 Loebner Prize Contest will append to a text file on disk containing the transcript of keyboard input and program output. The program should discard the existing contents of the file, but should open the file in append mode (the file should be created automatically if it does not exist yet).
2. The file will be named according to the input during set-up operation.
3. The file will be in ASCII text format suitable for input into a standard word processing program.
4. The first three lines are headers containing the following: 

   (c) 2001 Science Museum, London, all rights reserved

   [Program Name] [Contestant Name]

   Start at: [YYYY/MM/DD HH:MM:SS]

5. Each succeed line will EITHER mirror one line displayed on the screen, preceded by the source "JUDGEnn" or "PROGRAM" and time in brackets OR indicate a change of Judges: "***JUDGEnn***".

E.g: For the above interaction:-

*** JUDGE04 ***

PROGRAM[14:12:25]Welcome judge 4

JUDGE04[14:12:32] Do you think that the

JUDGE04[14:12:39] Republicans can succeed

JUDGE04[14:12:55] In winning the White House?

PROGRAM[14:13:15] Only if Newt succeeds in

PROGRAM[14:13:17] developing a more tolerant image

SCHEDULE TWO
Letter of Agreement

Return to top
	The National
Science	Museum Museum of
Museum	Science and Industry

Dear Sirs

This letter relates to the Loebner Prize Competition ("the Competition") which was founded by Dr Hugh Gene Loebner c/o Crown Industries, Inc. 155 North Park Street, East Orange, NJ 07017, USA ("Dr Loebner") and is to be run by the Board of Trustees of the Science Museum, Exhibition Road London SW7 2DD ("the Museum") from the year 2001.

The Cambridge Center for Behavioral Studies, 336 Baker Avenue, Concord, MA 01742 USA ("the Cambridge Center") currently has in its possession a stock of one gold, one silver and 44 bronze prize medals ("the Medals") plus the remaining Competition prize fund ("Cambridge Center Fund").

The Competition will be run by a third party in the Year 2000 and the Cambridge Center will retain one Bronze medal for the winner of that competition.

In consideration of the following:

1. the Agreement between Dr Loebner and the Museum to transfer the running of the Competition to the Museum ("the Agreement");
   
2. Dr. Loebner undertaking to make up the Cambridge Center Fund to the full prize fund amount of US$125,000 ("the Prize Fund")
   
3. the Museum paying any reasonable costs in transferring the Prize Fund and Medals from the Cambridge Center to the Museum and to any winner of the Silver or Gold medals in the Year 2000,
   
4. Dr Loebner paying any reasonable cost of return of 1 Bronze medal from the Cambridge Centre to himself as a personal momento

you agree to be bound by the terms and conditions set out below.

1. The Cambridge Center will transport the Medals to the Museum to be delivered no later than 6 months after the date of the Agreement. Should the Silver or Gold medal be awarded in the Competition run in the year 2000, the Museum undertakes to transfer the medal to the winner within 3 months of the Competition result.
   
2. Dr. Loebner will transfer to the Cmbridge Center the funds to make the Cambridge Center Fund up to the Prize Fund no later than 30 June 2000.
   
3. The Cambridge Center will transfer the Cambridge Centre Fund to the Science Museum no later than 1 July 2000.
   
4. The Cambridge Center confirms that it has no further rights in the above mentioned medals or prize fund.

I should be grateful if you would counter-sign and date this letter where indicated below (retaining the enclosed copy for your records) in order to give effect to the terms set out above and to signify your agreement to them.

Yours sincerely

/s/ John Durant

/s/ Hugh Gene Loebner

/s/ Betsy J Constantine

SIGNED by JOHN DURANT )

on behalf of THE BOARD )

OF TRUSTEES OF THE SCIENCE )

MUSEUM in the presence of: )

Witness Signature:

Witness Name:

Address:
 

Occupation:
 
 

SIGNED by HUGH GENE LOEBNER )

in the presence of: )

Witness Signature:

Witness Name:

Address:
 
 
 
 

Occupation: