Top 25 Correlations

August 27, 2017

I've always wondered what the "best" way to correlate different ratings' "top 25" is. It is a non-trivial question because because there is more than way to define the "consensus." Do you start with the consensus ranking based upon all ranks or just use the top 25 ranks from each rating to form a "top 25 consensus" and compare to that?

I still don't know what the answer is, but I've always wondered how I would define "correlation to top 25" using the latter approach. Actually, I knew how I would do it, but was too lazy busy to implement it until I was prompted to by compare the human top 25s to the computers.

Presuming that Dr. Massey's "Top 25 Cor to Con" is a correlation of the teams in a rating's top 25 to the "consensus" ranking of all teams by all ratings

The "average" or "consensus" ranking for each team is determined using a least squares fit based on paired comparisons between teams for each of the listed ranking systems. If a team is ranked by all systems, the consensus is equal to the arithmetic average ranking. When a team is not ranked by a particular system, its consensus will be lowered accordingly.
I choose to calculate the "consensus of top 25s" by only considering ranks 1-25 for each of the computers, as reported in Computer Top 25. The notion of correlation is made more complicated by the fact that different ratings' top 25s can (invariably do) include different teams.

For rank-based correlations I use the distance metric to characterize the correlation between two ratings. This is the number of team-pair swaps it would take to re-order one list to match the other. That doesn't work for lists that don't have the same elements, as is usually the case with the computer rankings' top 25s. No number of swaps of teams in one list can add a team to it!

So we add percentage of total team-pairs that are "concordant" as a measure of how close two rankings are. Instead of -1 ≤ x ≤ 1 we have 0 ≤ x ≤ 1.

Kendall's tau:
τ = #Concordant pairs − #Discordant pairs

Goodman and Kruskal's gamma:
γ = #Concordant pairs − #Discordant pairs

#Concordant pairs + #Discordant pairs
Percentage of Pairs that are Concordant
%Concordant = #Concordant pairs

#Total pairs

I calculate the "top 25 consensus" by just summing 26-team's rank by rating over all ratings that rank the team. This is not a good method for counting votes when all ranks 1-130 are considered, but it has the same properties as the consensus described by Dr. Massey when applied to truncated rankings. Using the ranings from Massey Ratings College Football Ranking Composite as of Sun Aug 27 05:28:22 we have 64 teams in at least one of 40 ratings' top 25.

Top 25 Correlation to Top 25 Consensus

#Common#Teams    ConcDisc#Pairsγτ%Conc
#Common#Teams    ConcDisc#Pairsγτ%Conc

I've added a Top 25 Consensus page with three reports.

Top 25 Rank Correlations to Consensus
Lists the actual top-25 ranks for each rating, the consensus top 25, and what the top-25 list would be based upon average (Borda) and median (Mix) rankings using all team ranks (not just the top 25) from all ratings. The latter two are not used to determine the consensus.

For each team in any ratings' top-25:

Points is the usual sum of 25 points for each #1 ranking, 24 for each #2, and so on down to zero points for ranks worse than #25.
∑Dist is the sum of the number of discordant pairs over all ratings. Lower numbers indicate stronger agreement among the computers about the team's rank.
#Votes is the number of ratings that include the team in their top 25.
Team links to the list of ratings that rank the team at each rank that any does.

For each Rating

∑Dist is the sum of the number of discordant pairs when the rating is compared to every other rating. The lower the number the more representative the ranking is of all rankings.
Dist from consensus is the number of discordant pairs when the rating's top 25 is compared to the consensus top 25.
Common w/ Cons is the number of teams in both the rating's top 25 and the consensus top 25 (the intersection of teams ranked by each.) To find the number of teams ranked by either (the union) just subtract this number from 50.
%Concordant w/ Cons is (10000×) the ratio of concordant pairs to total pairs.

Computer Ratings Top 25 % Concordant
For any two ratings displays the percentage of team-pairs that are concordant, to four decimal places with the leading zero and decimal point removed. The column-rating that matches the best agreement with the row-rating is displayed as blue and those which least agree in red. Higher numbers indicate better agreement between the ratings.

Computer Ratings Top 25 # Common Teams
The order of the intersection of teams in row-rating's top 25 and column-rating's top 25 is displayed in row, column. If this value is c the number of pairs used to derive the previous report is
(50 − c) × (50 − c − 1) ⁄ 2
= ( c2 − 99×c + 2450 ) ⁄ 2

The "top 25" is not as significant as its popularity might suggest. It approximately selects the top quintile of the 1A field, but the main value in voting for a top 25 is to rank some smaller number of teams.

© Copyright 2017, Paul Kislanko
Football Home