Correlating Computer Rankings

© Copyright 2006, Paul Kislanko

There are lots of ways to use Kenneth Massey's College Football Ranking Comparison to compare ratings. Another way to use it is to compare teams various different ways. We'll do both this year with some simple but useful views of the data.

Suppose what we want to do is come up with one ranking that represents all of the computers. The method used on Massey's comparison page is equivalent to the average rank for the most part, but there are other ways to define "consensus."

Round Robin

One way to rank the teams is to try to order them so that a majority of the computers rank a team better than all the teams that follow it (if this sounds like just the definition of ordering by "best", keep reading). It turns out that isn't always possible and in general, there won't be a way to do that if there are more than two teams and more than one computer.

What we can do is compare each team to every other team and for each pair count how many computers have each rated higher than the other. Usin the 23 computers available on August 23, the top 25 looks like this (the numbers are just identifiers for corresponding rows and columns):

Computer Rankings by Team Pair (Condorcet)
(23 rankings)

41 50 28 11 20 61 44 7 57 59 58 15 24 29 39 23 43 47 48 1 2 4 55 99 31
41 Texas   18 21 23 21 23 19 23 22 23 22 21 23 23 23 22 23 23 23 23 22 23 23 23 23
50 Southern California 5   17 21 18 22 17 21 21 22 21 21 21 22 21 21 22 21 22 21 22 22 22 23 22
28 Ohio State 2 6   17 19 22 17 20 21 21 22 18 22 22 20 20 21 20 22 22 20 23 23 23 22
11 Virginia Tech 0 2 6   12 15 13 15 17 17 18 17 19 16 16 17 21 15 22 21 20 19 22 20 23
20 West Virginia 2 5 4 11   13 12 14 15 17 16 13 16 13 17 15 16 20 22 22 19 17 23 21 21
61 LSU 0 1 1 8 10   12 11 14 19 18 18 16 17 15 19 22 20 20 20 20 21 22 19 21
44 Notre Dame 4 6 6 10 11 11   14 17 12 16 17 16 13 15 19 16 19 21 20 23 18 20 20 20
7 Miami-Florida 0 2 3 8 9 12 9   12 15 16 14 16 15 15 16 18 17 16 17 20 20 19 17 19
57 Auburn 1 2 2 6 8 9 6 11   13 15 12 13 11 14 14 17 20 17 16 18 20 17 17 19
59 Georgia 0 1 2 6 6 4 11 8 10   15 13 12 12 14 15 20 14 17 18 17 17 21 19 20
58 Florida 1 2 1 5 7 5 7 7 8 8   14 12 11 14 15 13 18 14 17 17 17 15 18 19
15 Louisville 2 2 5 6 10 5 6 9 11 10 9   12 13 12 16 13 15 15 18 17 15 19 19 18
24 Michigan 0 2 1 4 7 7 7 7 10 11 11 11   12 12 16 17 14 16 18 19 18 19 17 21
29 Penn State 0 1 1 7 10 6 10 8 12 11 12 10 11   14 13 16 12 18 17 16 13 18 19 19
39 Oklahoma 0 2 3 7 6 8 8 8 9 9 9 11 11 9   14 15 13 12 14 17 18 14 13 19
23 Iowa 1 2 3 6 8 4 4 7 9 8 8 7 7 10 9   13 14 14 16 18 12 16 17 20
43 Texas Tech 0 1 2 2 7 1 7 5 6 3 10 10 6 7 8 10   13 14 15 14 13 18 14 15
47 California 0 2 3 8 3 3 4 6 3 9 5 8 9 11 10 9 10   14 13 14 11 13 16 17
48 Oregon 0 1 1 1 1 3 2 7 6 6 9 8 7 5 11 9 9 9   14 14 12 14 14 16
1 Boston College 0 2 1 2 1 3 3 6 7 5 6 5 5 6 9 7 8 10 9   13 13 14 15 14
2 Clemson 1 1 3 3 4 3 0 3 5 6 6 6 4 7 6 5 9 9 9 10   13 14 14 11
4 Florida St 0 1 0 4 6 2 5 3 3 6 6 8 5 10 5 11 10 12 11 10 10   12 12 12
55 Alabama 0 1 0 1 0 1 3 4 6 2 8 4 4 5 9 7 5 10 9 9 9 11   12 12
99 TCU 0 0 0 3 2 4 3 6 6 4 5 4 6 4 10 6 9 7 9 8 9 11 11   16
31 Wisconsin 0 1 1 0 2 2 3 4 4 3 4 5 2 4 4 3 8 6 7 9 12 11 11 7
Texas is the clear number one because pairwise a majority of computers pick it compared to any other team (see the full results). When there is such a case it is said the winner is the Condorcet winner after the French mathematician who defined this method.

There are just a few differences between this list and the average, and we don't quite meet our objective. In just the top 25 there are five cases where a majority of the computers rank a lower-listed team better than one that came before it. For example, 12 of the ratings have "#8" Miami ranked higher than "#6" LSU and only 11 the other way 'round. So why are LSU and Miami so ranked? LSU is preferred to Notre Dame 12-11, and Notre Dame is ranked better than Miami 14-9, so there's a "cycle" with these three teams: LSU is better than Notre Dame is better than Miami is better than LSU. How to deal with this is up to us so we keep that 14-9 win because it's the "strongest".

Majority Rules

Of course, expressing even the top 25 as pairwise comparisons is complicated. For the whole field there are 14,042 counts involved. Another way to derive the list is to find the best ranking for which at least half the computers agree the team is as good as or better than. Instead of listing all the ranks for a team by rating, we list the count of ratings for each rank, and add them up until we get to one half the total ratings or more.

Maj Cnt TieBrk Best Worst Team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
1 13   1 9 Texas 13 6 2 1         1                              
2 15   1 45 Southern California 5 10 2 2 1   1                     1            
3 12   1 28 Ohio State 1 3 8 3 4 1             1         1            
7 12   1 24 West Virginia 1   2 1 2 4 2 1 1     2 1 1 1     2 1         1
8 15   5 31 LSU         2 4 5 4 2   1     1           1       2
8 14   3 33 Virginia Tech     3 4 3   1 3 3         2 1 1   1            
9 12   1 21 Notre Dame 3 2 1 3 1     1 1 1 1 1 2       2 1 1 1 1      
11 13   3 38 Georgia     1       2 3 1 2 4 1 1     1       1   2 1 1 1
11 12   5 31 Miami-Florida         2 5 2     1 2 1 3 1 2 1     1          
13 13   5 30 Auburn         1 1 2 2 4   1   2 3     2   2   2      
14 14   5 23 Michigan         2       2 5   1   4 1 1 3       2   2  
14 13   2 33 Louisville   2   3       1 1   1 3   2   4         3     1
14 12 4.73 5 41 Florida         2   1 2   1 1   4 1 3 1 1 2   1 1      
14 12 2.68 3 42 Penn State     1 1 2 1 3   1 1 1     1 1 1 2         1   1
15 13   3 39 Iowa     1 1     1     2 1 2   1 4 1 1 2 1 1     1   1
17 13   3 51 Oklahoma     2 3   1   1       1     3   2     1 2 4    
17 12   9 32 Texas Tech                 1 2 1 3 1 1 1 1 1 1 1 1     2 1
19 13 3.7 6 42 Boston College           1   1         1 1 1   1 3 4 2 2     1
19 13 2.56 10 44 Oregon                   2 1 1 1 2 1 2 1   2 1   1 1 2 2
19 12   7 38 California             2 1   3 1   2 1   1     1 2 1 2   1 1
22 13   4 34 Clemson       1         2   1       2         2   5 2 2 2
23 12 2.56 10 43 Alabama                   1   1 2       2 2 2       2   2
23 12 1.91 8 55 Florida St               1 2 1 2 3       1             2   1
24 14   6 46 TCU           3 1                 2     1 3   1   3 1
24 12   12 51 Wisconsin                       1 1 1   1 1 1 1   1 1 2 1 1
Here we've shown just the first 25 of all 119 rows and columns. We note that a majority of the ratings have both LSU and Virginia Tech in the top eight, but 15 ratings have LSU that high and only 14 have the Hokies as high as eight. This method of counting, known as Bucklin, is equivalent to taking the median rank instead of the average when there's an odd number of rankings. When there's an even number it will be the first ranking below (better than) the arithmetic median.

The tiebreaker isn't that important here, but is based upon the average ranking by the ratings that are not a part of the majority that assigned the team its rank. Note that we've not included the human polls here, just the computers. In general this year when we reference the computer rankings, it will be in the same summary format as the pre-season rankings.

Note: See the 30 August Update.