Loebner Prize 2004 Rules
$2000 and a Bronze Medal
$25,000 and the Silver Medal will be at risk.
The Competition "Final Four" round will take place:
10:00 - 16:00, Sunday, 19 Sept 2004, at 220 W. 98th St #2B, New York, NY.
The "Final Four" contestants have been chosen They are:
Richard Wallace et al
- If there is no compliant Entry for the 2004 Competition the $2000 prize will be added to the 2005 Competition prize making the 2005 prize $4000, and the 2005 Competition will be held under these rules. If there is only one Entry compliant with these rules the submitter(s) of that Entry will receive $2000 and a Bronze Medal. If there are two or more Entries compliant with these rules a Competition will be held and the submitter(s) of the winning Computer Entry will receive $2000 and a Bronze Medal.
- Contest management reserves the right to enter one or more publicly available open source programs, IF THERE IS ALSO AVAILABLE an associated communications program as described in Rule 8. If such an open source program wins and it is infeasible to acknowledge a deserving recipient for the award, the $2000 will be added to the 2005 prize fund and the Bronze Medal will be reserved.
- The 2004 Competition will be a RESTRICTED CONVERSATION contest. Every Entity (Human Confederate and Computer Entry) must begin its interactions with each Judge by asserting: "Hello, my name is X and I am the human" where "X" is replaced by a "reasonable" human name subject to my approval prior to the contest date. This assertion must be made on the first interaction between the Entity and every Judge, but not necessarily more than once for each Judge. If a Judge re-questions an Entity it is not expected or required that the Entity reassert its humanity (unless a line of question dictates this). Following the initial forced assertion each Entity should respond in such a manner as to attempt to convince the Judge that the Entity is a human responding truthfully.
- There are no other restrictions on conversations or topics.Confederates and the submitters of Computer Entries must understand that the transcripts of the interactions between the Entities and the Judges will be published, and may be entered as evidence in the case Loebner v. Cambridge Center, et al., (03 CV 10959 RCL Fed. Dist. Ct., Mass.) Judges, Confederates and submitters of Computer Entities are responsible for the content of their or their entries' utterances.
- Computer Entries need not act in a server/client/web mode, but are encouraged to be entered as stand-alone applications running on a single computer. Contest management will make available ONE computer, running either Windows XP or Macintosh OS X, for each submitter. Other computer/operating systems are welcome, but resource limitations require that the submitters provide their own hardware. Submitters are free to enter web based/enhanced Computer Entities, but must provide their own second computer and associated communications paraphernalia. ALL PROGRAMS WILL RUN ON SITE. Computer Entry must respond CHARACTER BY CHARACTER rather than buffering output.
- Each Computer Entry must be accompanied by a MATCHING COMMUNICATIONS PROGRAM. The Com Prog must (a) run as two programs on two computers (b) communicate CHARACTER BY CHARACTER with each other VIA EASILY IMPLEMENTED COMMUNICATIONS (c) mimic the appearance and behavior of the matching Computer Entity. "Easily implemented communications" means directly two computers with a cable having standard plugs or connecting each to an Ethernet hub.
- It is the intent of rules 5 and 6 to provide a framework by which the Competition may be held in a frugal and simple manner while being meaningful. Although submitters are constrained to create and submit communications programs matching their Entries, they are relieved of any obligation to interface with, or imitate, or otherwise mimic programs not of their own devising. They, better than anyone, should know their programs' appearances and behaviors. It is also the intent of rule 6 to facilitate judging by the method of paired comparisons.
- Judging and Selection of finalists will depend on the number of Entries according to the following Selection Rules.
- No program will be accepted that does not have an acceptable matching communications program.
- If there is only one compliant program, the submitter(s) will be declared the winner and awarded the medal and money.
- If there are 2 - 4 compliant Entries all will be entered in the Final Four Round. The Final Four Round shall consist of paired comparisons between the Computer Entries and the Confederates.
- If there are 5 - 8 compliant Entries all will be entered into The Prelim Eight Round. The four entries having the highest scores in that round will enter the Final Four Round. The Prelim Eight Round shall consist of real-time questions and answers between Judges and Computer Entries only.
- If there are 9 - 32 entries they will be prescreened down to eight. These eight will be entered in the Prelim Eight Round.
- Prescreening involves
- Downloading the Computer Entries onto local computers.
- Challenging the Computer Entries with a prepared test suit of questions.
- Scoring the responses and publishing the results.
- If there are more than 32 Entries the first 32 compliant Entries submitted will be entered in the prescreening round. HOWEVER, I reserve the right to invite into the prescreening round compliant Entries of outstanding merit.
- Logging will be similar to the rules for the 2003 Competition, with the difference that logs will be marked "This transcript is in the public domain" and references to the Cambridge Center or the University of Surrey will be eliminated.
- Scoring For the "Final Four".
- The Final Four Competition will be scored using the Method of Paired Comparisons.
- Each Judge will evaluate four pairs of terminal. One terminal of each pair will be controlled by a computer program (Entry.) The other terminal will be the matching communications program supplied along with the Entry. That program will be controlled by a human (Confederate).
- The task of each Judge will be to apportion 100 points between the two terminal based upon how "human" they seem. The judgement is relative, and ties are forbidden; one of the two terminal must be awarded at least 51 points.
We wish (a) each Entry to be compared with every Confederate; (b) each Judge to evaluate every Entry, (c) each Judge to evaluate every Confederate.
Label the four Entries E1..E4, four Confederates C1..C4, and four judges J1..J4
The following matrix has Judges as rows and Entries as columns. The intersection of each row and column shows which Confederate is assigned to the combination of Entry and Judge.
....... E1 .... E2 .... E3 .... E4
J1 .... C1 .... C2 .... C3 .... C4
J2 .... C4 .... C1 .... C2 .... C3
J3 .... C3 .... C4 .... C1 .... C2
J4 .... C2 .... C3 .... C4 .... C1
For example, reading across the row 2 we see that J2 compares E1 with C4, E2 with C1, E3 with C2, and E4 with C3. J2 will have scored every Entry and every Confederate, but in different combinations than J's 1, 3 and 4.
Reading down the third column, we see in the first row that E3 is judged by J1 against C3. Let us enter a 1 in that cell if E3 scored
51 or higher against C3, and a 0 otherwise. We may continue down the column, entering a 1 in the second row if E3 scored 51 or higher against C2, etc. The sum of the column will be the number of times E3 was judged as "more human" than a Confederate. We may do this for each Entry.
- The Entry with the highest column total will be declared the winner.
- **** The Silver Medal and $25,000 will be awarded **** if the score is "2" or higher.
( I believe that an Entry which has been independently rated by two different Judges as "more human" than two different Confederates in direct paired comparisons deserves the Silver Medal. )
- If two or more Entries tie for high column totals, the scores that each tied Entry received shall be totaled, and the tied Entry having the highest total score will be declared the winner.
- If two or more of the tied Entries remain tied based upon highest total points scored, then victory will be decided by the results of the Semi-final Judging.
- Judging will consist of seven rounds of thirty minutes duration. Not all Judges and Confederates will participate in every round. In each round, Judges will have 25 minutes to evaluate a pair. After the 25 minute evaluation period there will be a 5 minute break for scoring and reassignment.
The following table shows each round. In the first round J1 compares E1 with C1, J2 compares E3 with C2, and J3 compares E2 with C4. J4 and C3 are excused from the round and can rest. Excused Judges will be kept separate from excused Confederates.
Round ...... Participating ............Excused
1 .... J1E1C1 J2E3C2 J3E2C4 ......... J4 C3
2 .... J4E1C2 J3E3C1 J2E4C3 ......... J1 C4
3 .... J3E1C3 J4E3C4 J1E2C2 ......... J2 C1
4 .... J2E1C4 J3E4C2 J4E2C3 ......... J1 C1
5 .... J2E2C1 J1E3C3 ................ J3 J4 C2 C4
6 .... J1E4C4 ....................... J2 J3 J4 C1 C2 C3
7 .... J4E4C1 ....................... J1 J2 J3 C2 C3 C4