2006 Loebner Prize competition will be held Sunday, 17 September 2006 at Torrington Theater, University College, London. (Contest Management reserves the right to change the venue) $2000 and a Bronze Medal - In 2006, $25,000 and the Silver Medal will be at risk. Important Dates: June 1, 2006: Opening date for submission of entries. June 30, 2006: Closing date for entries. July 30, 2006: Semi-Final 8 announced Sept 1, 2006: Final Four announced Sept 17, 2006: Loebner Prize 2006 Contest The rules for 2006 are substantially the same as for 2005 with the following exceptions: a. Final Four entrants will each be awarded a USD 250 stipend. The winner of the Final Four round will receive the Competition Cash prize of USD 2000, making a total cash award of USD 2250 to the winner (unless there is only one compliant entry, in which case the cash award will be USD 2000. b. Final Four entrants must be present at the competition. c. Communications programs will be supplied by contest management. d. Judges will have 20 minutes to review transcripts of their interactions before scoring. e. Judges will have 27 minutes to interact with each pair of entities. Rules: Entrants should submit their entries on CD, DVD or USB Flash via a medium (a) requiring a receipt signature and (b) having a time/date stamp (E.g. Certified, Registered, FedEx, UPS, etc) To: Loebner Prize Contest c/o Crown Industries, Inc. 155 North Park St. East Orange, NJ 07017 If there is no compliant Entry for the 2006 Competition, the $2000 prize will be added to the 2007 Competition prize making the 2007 prize $4000, and the 2007 Competition will be held under these rules. If there is only one Entry compliant with these rules the submitter(s) of that Entry will receive $2000 and a Bronze Medal. If there are two or more Entries compliant with these rules a Competition will be held and the submitter(s) of the winning Computer Entry will receive $2000 and a Bronze Medal. Contest management reserves the right to enter one or more publicly available open source programs, The 2006 Competition will be a RESTRICTED CONVERSATION contest. Each male Confederate will begin his interactions with each Judge by asserting: "Hello, my name is John and I am a man." Each female Confederate will begin her interactions with each Judge by asserting: "Hello, my name is Joan, and I am a woman. Each Entry MUST make one of those two assertions. The choice is left to the entrant. This assertion must be made on the first interaction between the Entity and every Judge, but not necessarily more than once for each Judge. If a Judge re-questions an Entity it is not expected or required that the Entity reassert its name and sex (unless a line of question dictates this). Following the initial forced assertion each Entity should respond in such a manner as to attempt to convince the Judge that the Entity is a human responding truthfully. There are no other restrictions on conversations or topics. Confederates and the submitters of Computer Entries must understand that the transcripts of the interactions between the Entities and the Judges will be published. Judges, Confederates and submitters of Computer Entities are responsible for the content of their entries' utterances. It is the intent of the rules to provide a framework by which the Competition may be held in a frugal and simple manner while being meaningful. Judging and Selection of finalists will depend on the number of Entries according to the following Selection Rules. If there is only one compliant program, the submitter(s) will be declared the winner and awarded the medal and money. If there are 2 - 4 compliant Entries, all will be entered in the Final Four Round. The Final Four Round shall consist of paired comparisons between the Computer Entries and the Confederates. If there are 5 - 8 compliant Entries all will be entered into The Prelim Eight Round. The four entries having the highest scores in that round will enter the Final Four Round. The Preliminary Eight Round shall consist of real-time questions and answers between Judges and Computer Entries only. No human confederates will be employed. If there are 9 - 32 entries they will be prescreened down to eight. These eight will be entered in the Prelim Eight Round. Prescreening will consist of presenting all entries with a suit of test questions. The eight entries responding in the most human manner will be selected to enter the Preliminary Eight round. If there are more than 32 Entries the first 32 compliant Entries submitted will be entered in the prescreening round. HOWEVER, I reserve the right to invite into the prescreening round compliant Entries of outstanding merit. Scoring For the "Final Four". The Final Four Competition will be scored using the Method of Paired Comparisons. The task of each Judge will be to apportion 100 points between the two terminals based upon how "human" they seem. The judgment is relative, and ties are forbidden; one of the two terminals must be awarded at least 51 points. No judge may use the same apportionment twice. That is, if a judge apportions 100 points 40/60 for one Entry/Confederate pair, then he or she may _not_ use a 40/60 apportionment again. This will prevent ties if it becomes necessary to convert the judge's scores into rankings. We wish (a) each Entry to be compared with every Confederate; (b) each Judge to evaluate every Entry, (c) each Judge to evaluate every Confederate. Label the four Entries E1..E4, four Confederates C1..C4, and four judges J1..J4 The following matrix has Judges as rows and Entries as columns. The intersection of each row and column shows which Confederate is assigned to the combination of Entry and Judge. ....... E1 .... E2 .... E3 .... E4 ---------------------------------- J1 .... C1 .... C2 .... C3 .... C4 J2 .... C4 .... C1 .... C2 .... C3 J3 .... C3 .... C4 .... C1 .... C2 J4 .... C2 .... C3 .... C4 .... C1 For example, reading across the row 2 we see that J2 compares E1 with C4, E2with C1, E3 with C2, and E4 with C3. J2 will have scored every Entry and every Confederate, but in different combinations than J's 1, 3 and 4. Reading down the third column, we see in the first row that E3 is judged by J1against C3. Let us enter a 1 in that cell if E3 scored51 or higher against C3, and a 0 otherwise. We may continue down the column, entering a 1 in the second row if E3 scored 51 or higher against C2, etc. The sum of the column will be the number of times E3 was judged as "more human" than a Confederate. We may do this for each Entry. SCORING The Entry with the highest column total will be declared the winner. **** The Silver Medal and $25,000 will be awarded **** if the score is "2" or higher. (I believe that an Entry which has been independently rated by two different Judges as "more human" than two different Confederates in direct paired comparisons deserves the Silver Medal.) If two or more Entries tie for high column totals, each judge's scores shall be converted to rankings. That is, for each judge, the entity with the highest score will be scored with '1', the next highest with a '2', and so forth to the lowest score which will receive a '8.' These rank orders will assigned as scores. The rank scores that each Entry received shall be totaled and the Entry having the lowest total score of ranks will be declared the winner. If two or more of the tied Entries remain tied based upon rank order scoring, then victory will be decided by the results of the Semi-final Judging. Judging will consist of seven rounds of thirty minutes duration. Not all Judges and Confederates will participate in every round. In each round, Judges will have 27 minutes to interact with a pair. After the 27 minute evaluation period there will be a 3 minute break for reassignment. At the conclusion of the seven rounds the judges will have 20 minutes to review the transcripts of their interactions with the four pairs and to assign points based upon that review. The following table shows each round. In the first round J1 compares E1 with C1 and J2 compares E3 with C2. Judges J3 and J4 and Confederates C3 and C4 are excused from the round. Excused Judges will be kept separate from excused Confederates and both will be kept separate from the competition Round ...... Participating ............Excused 1 .... J1E1C1 J2E3C2 ................ J4 C3 J3 C4 2 .... J4E1C2 J3E3C1 J2E4C3 ......... J1 C4 3 .... J3E1C3 J4E3C4 J1E2C2 ......... J2 C1 4 .... J2E1C4 J3E4C2 ................ J1 C1 J4 C3 5 .... J2E2C1 J1E3C3 ................ J3 J4 C2 C4 6 .... J1E4C4 J4E2C3................. J2 J3 C1 C2 7 .... J4E4C1 J3E2C4................. J1 J2 C2 C3 Communications protocol : Each Entry Program must communicate with a perl "Judge Communications" program. The current version of the judge program is available at: http://loebner.net/Prizef/2006_Contest/Judge_Com.txt One can test the communications protocol by using the "Confedrate Communications program which is available at: http://loebner.net/Prizef/2006_Contest/Confederate_Com.txt It is recommended that the names of the files be changed to have a .pl extension so that the perl interpreter can recognize them. ______________________ Communications will be on a character-by-character basis. In order to simplify the communications procedure, communications will be accomplished by the creation/deletion of directories whose names will encode information regarding keypress. The entry program will, upon execution, request the name of a "Communications Directory" via a command such as the perl Tk command Tk::chooseDirectory() The Judge Communications program will communicate to the Entry by creating, within the specified Communications Directory, a sub-directory whose name is: