Meta-Ranking Construction

September 3, 2014

The debut of the USA Today College Football Computer Composite and its inclusion in Dr. Massey's College Football Ranking Composite prompts contemplation on the role and construction of composite rankings.

There are some good reasons to exclude it from my own computer ranking analysis, and some for including it.

• It does not coreespond to a rating system, as it uses the constituent systems' ranks as input. There is inevitable loss of precision in any such consolidation. This is balanced by the fact that the ranks are combined in an interesting and unique way (more on that in a bit.)

  Had I been tasked with construction of a composite rating, I think I would have done as I do with my baseball ratings comparisons: exploit the fact that the rating values for the systems are normally distributed to "scale" the team values to a distribution with mean zero and standard deviation one. The results for each team can then be combined in any arithmetical way while preserving the relative differences between team values in the different systems. I may take the trouble to do exactly that if I get time.

• Each of the constituent rating systems is already included in my analysis, so including a composite derived from them gives these more weight than the other ratings. At least in the first week's edition of the ranking, the composite appears to be different enough from each individual component¹ for that not to be a large concern.

I am including the ranking for now because part of the premise of my analysis is that a composite has the potential to be a better measure of the field, and including several such supports investigation of the premise's correctness. To facilitate this, I am also including the "Consensus Rank" from Dr. Massey's composite in the Weighted Retrodictive Ranking Violations (but not the Borda, Bucklin or Pairwise Ranking calculations.)

WHAT? Mean?

A reminder:

The arithmetic mean

is the usual average of the ranks.

The harmonic mean

is the inverse of the average of the ranks' inverses.

The geometric mean

is the antilogarithm of the average of the ranks' logarithms.

Having some kind of "decay" for rankings far worse than most without resorting to removing them entirely is desirable, and it can be argued that if one or some ratings are much better than most that is a measure of some desirable quality that the team demonstrates. Note that my "Majority Consensus" composite is a "step function", that assigns a zero weight to every rating worse than the majority rank.

Since the the UTCFCC has only five components and it is reasonable to expect that the spread of ranks would not be excessive, the geometric mean strikes me as a much much better method than the BCS and its predecessors applied.

Another benefit is that it renders true what was once a hilarious response to a serious question. In the earlier days of the BCS the ranking formula consisted of a linear combination of rational components. When asked how ties would be broken, a spokesmen demonstrated extreme innumeracy by responding "we'll just use more decimal places." Since the 5^th root of an integer that is not a 5^th power of an integer is guaranteed to be irrational (transcendantal, even) using more decimal places would actually work. Except there's still a tie when two teams have exactly the same number of the same ranks, as when team A has { 1 1 2 3 3 } and team B has { 3 3 2 1 1 }. In such cases, teams A and B would be tied in the UTCFCC, and that is reasonable and proper. Any "tiebreaker" would have to be arbitrary, in the sense that it cannot be derived from the input.

Comparisons

So what would a composite of all computer rankings look like if the geometric mean were used to form the composite? The values for all teams would be better than the average value, but not by the same amount so the ordinal ranking of the composite is different. For the 54 week 1 ratings published on Thursday, Sep 4 16:06:02 the top 25 by geometric mean was:

The "Derived Ranks" are the ordinals resulting from sorting the list according to the "Values" for each definition of "composite." A few notes:

• The derived rank for my Consensus majority is based upon a tiebreaker that depends upon the input but is "arbitrary" in the sense that it is only one of many possible choices. I only use it to determine a reporting sequence. Since the Value is itself an ordinal rank it is the value I use in all of my team-oriented reports.
• PWw (pairwise wins) has a derived rank that is just #teamsranked-#pairwise wins (#pairwise wins is the value.) There is no tiebreaker, although if one is needed the pairwise head-to-head results are available.
• Avg and MCon (the Massey Consensus) would be identical if I included the human polls. That the derived ranks differ at all indicates something about the human polls.

All of the possibilities are interesting, but I have an aesthetic preference for a composite ordinal rank that is derived from ordinal ranks without resorting to an average of any kind. I'll calculate any varieties we come across, but the ones I use will be based upon the simple (?) methods based only upon counting.