Conference Ranks

© Copyright 2011, Paul Kislanko

I was asked to weigh in on the best way to characterize conference strength for any particular rating. (Really, somebody asked me to revisit this topic. Blame them!)

There are as many ways to compare conference strength as their are opinons. Simply averaging teams' ratings (or ranks) doesn't sufficiently account for the fact that a conference may contain one very good team or a few very bad teams that skew the average.

Here are a couple of methods I find most useful, along with the reasons why I do.

Team Ranks

Rankings for which we do not have the corresponding rating values present a special problem. Average rank is not very useful, because there's no way to take into account that the difference between #10 and #20 is much larger than that between #40 and #80 (typically.) A more appropriate measure is the median rank - how good is the top half of each conference.

I actually use a "weighted median" - which is a bad name because it's really a modified average. The adjustment takes into account how much better/worse the teams' ranks are than the field median. It gives a rough measure of how hard it is to have a good conference record in a given conference.

Rank Distribution by ConferenceNote 1
17 Oct 2011

ix #Teams WtdMed Best Med Worst Conf T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13
1 10 -76.30 3 18 83 B12 3 6 11 12 18 27 29 33 65 83      
2 12 -62.25 1 23 112 SEC 1 2 10 17 20 23 26 43 56 58 85 112  
3 1 -61.00 31 31 31 ND 31                        
4 12 -33.33 4 34 148 B10 4 13 15 22 24 34 37 52 94 95 130 148  
5 12 -31.42 7 38 115 P12 7 8 19 28 30 38 55 59 71 106 107 115  
6 12 -24.33 9 44 119 ACC 9 16 35 39 42 44 53 60 64 78 97 119  
7 8 -16.00 14 51 98 BigE 14 40 45 51 61 67 72 98          
8 3 35.00 57 75 117 Ind 57 75 117                    
9 8 42.38 5 73 196 MW 5 32 54 73 93 127 159 196          
10 12 53.83 21 79 204 CUSA 21 25 49 50 68 79 88 105 114 177 194 204  
11 8 62.00 47 89 149 WAC 47 82 84 89 90 96 131 149          
12 13 81.08 36 101 184 MAC 36 41 69 76 86 99 101 109 122 125 132 160 184
13 9 112.00 46 118 217 SoCon 46 48 87 102 118 128 153 154 217        
14 11 128.00 80 124 190 CAAF 80 92 100 103 116 124 134 137 141 180 190    
15 9 130.33 62 129 197 SBC 62 70 81 111 129 147 157 165 197        
16 5 154.60 121 140 146 GWest 121 138 140 143 146                
17 9 157.89 63 144 208 MVC 63 66 108 142 144 162 166 173 208        
18 9 163.33 77 135 233 BSky 77 110 126 133 135 156 174 218 233        
19 8 213.00 74 161 228 SLC 74 150 155 161 205 207 220 228          
20 4 219.50 139 158 229 1AAInd 139 158 212 229                  
21 7 226.71 104 179 225 Pat 104 136 176 179 186 189 225            
22 9 234.11 145 175 224 OVC 145 164 169 171 175 182 203 206 224        
23 8 241.63 91 185 240 Ivy 91 120 163 185 198 210 230 240          
24 11 276.73 123 200 243 MEAC 123 170 183 188 193 200 202 221 232 242 243    
25 7 279.29 113 209 241 BigS 113 151 167 209 235 237 241            
26 8 285.88 168 201 239 NE 168 187 192 201 219 223 234 239          
27 10 304.20 152 215 244 SWAC 152 172 211 213 215 222 226 231 236 244      
28 10 304.50 178 214 246 Pio 178 181 191 199 214 216 227 238 245 246      

I'm not fond of any meta-ranking that depends upon rankiings, but this one does make an easy distinction between AQ-conferences and the dregs of D1. Since this is based upon ordinal rankings, lower scores are better.

Ratings, not Rankings

When the ratings values are available, I prefer to use the ratings the way a teacher "grades on the curve." Any rating worth worrying about will necessarily have a normal distribution. Assign the teams "grades" based upon how many standard deviations they are from the mean, and then order the conferences by the grades their constituent teams get.

Rating Distribution by ConferenceNote 2
17 Oct 2011

Ix Srt Conf A+ A B C+ C C- D E F
1 107.20 B12 2 2 4 1 1 0 0 0 0
2 85.33 SEC 2 1 4 3 2 0 0 0 0
3 66.67 B10 1 1 5 1 4 0 0 0 0
4 64.00 ND 0 0 1 0 0 0 0 0 0
5 55.00 MW 1 0 1 2 3 1 0 0 0
6 54.67 P12 0 2 4 3 3 0 0 0 0
7 46.00 BigE 0 1 1 5 1 0 0 0 0
8 44.00 ACC 0 1 3 5 3 0 0 0 0
9 26.00 CUSA 0 0 2 3 4 3 0 0 0
10 24.00 MAC 0 0 2 1 9 1 0 0 0
11 21.33 Ind 0 0 0 1 2 0 0 0 0
12 18.67 SBC 0 0 0 2 6 1 0 0 0
13 18.22 SoCon 0 0 0 2 6 0 1 0 0
14 18.00 WAC 0 0 0 1 7 0 0 0 0
15 17.33 MVC 0 0 0 2 5 1 1 0 0
16 16.00 GWest 0 0 0 0 5 0 0 0 0
17 14.55 CAAF 0 0 0 0 9 2 0 0 0
18 12.22 BSky 0 0 0 0 6 1 1 1 0
19 11.11 OVC 0 0 0 0 4 4 1 0 0
20 10.50 SLC 0 0 0 0 4 1 3 0 0
21 10.00 1AAInd 0 0 0 0 2 0 2 0 0
22 9.71 Pat 0 0 0 0 2 4 1 0 0
23 9.00 Ivy 0 0 0 0 3 2 1 2 0
24 8.29 BigS 0 0 0 0 3 0 1 3 0
25 7.27 MEAC 0 0 0 0 2 5 1 1 2
26 5.78 NE 0 0 0 0 1 3 2 2 0
27 5.70 SWAC 0 0 0 0 2 0 5 2 1
28 4.80 Pio 0 0 0 0 0 4 3 1 2
                     
    SAG≥ μ+2σ μ+1.5σ μ+σ μ+0.5σ μ-0.5σ μ-σ μ-1.5σ μ-2σ -∞
    All: 6 8 27 32 99 33 23 12 5

While the weighted median gives more weight to the middle strength of the conference, this "grade summary" emphasizes how good the top teams (in the field) are. The sort sequence gives 256 points for each team whos rating is more than 2 standard deviations better than average, 128 for teams with ratings less than 2 but more than 1.5 SDs better, down to 0 for teams whose ratings are more than 2 standard deviations below average. The sort value is the sum of all those divided by the number of teams.

Averages on Steroids

A problem with each of the aggregations I've talked about is that the weights ultimately depend upon the number of teams in the conference and those aren't the same for every conference. It's a little more work to calculate, but there is a lot to be said for what I call the Team-Pairwise Conference Comparison.

The basic idea is embodied in the aggregation's name - compare each team from conference A to every team in every other conference based upon some particular rating/ranking. For example, to compare the Big 12's ten teams to the SEC's 12 teams requires 60 comparisons (12×10)/2.) If we are comparing teams' ranks the resulting table gives the probability that if that you choose a team at random from the conference listed in the row and a team at random from the conference listed in the column heading the rank is better for the row-conference than the column-conference.

Team-Pairwise Conference Comparison
Probability Rank(team from row-conf) < Rank(team from column-conf)

Pct WW LL Conf B12 ND SEC P12 ACC B10 BigE Ind MW WAC CUSA MAC SoCon SBC CAAF MVC GWest BSky Pat SLC Ivy OVC 1AAInd BigS MEAC NE SWAC Pio
0.9017 2128 232 B12   0.700 0.542 0.708 0.758 0.692 0.800 0.900 0.813 0.963 0.867 0.954 0.956 0.956 0.991 0.967 10.989 10.988 11111111
0.8776 215 30 ND 0.300   0.417 0.583 0.833 0.583 0.875 10.875 10.833 11111111111111111
0.8665 2433 375 SEC 0.458 0.583   0.632 0.660 0.604 0.698 0.861 0.771 0.885 0.813 0.885 0.907 0.935 0.962 0.954 10.972 0.988 0.979 0.990 1111111
0.7988 2243 565 P12 0.292 0.417 0.368   0.549 0.472 0.563 0.778 0.677 0.781 0.694 0.808 0.833 0.889 0.909 0.917 10.963 0.964 0.969 0.969 110.988 1111
0.7942 2230 578 ACC 0.242 0.167 0.340 0.451   0.424 0.563 0.750 0.667 0.833 0.715 0.821 0.852 0.907 0.947 0.926 10.963 0.988 0.969 0.979 110.988 1111
0.7899 2218 590 B10 0.308 0.417 0.396 0.528 0.576   0.594 0.722 0.667 0.750 0.708 0.782 0.815 0.843 0.856 0.889 0.900 0.907 0.964 0.958 0.938 0.991 0.979 0.976 0.985 111
0.7836 1492 412 BigE 0.200 0.125 0.302 0.438 0.438 0.406   0.792 0.672 0.844 0.708 0.837 0.847 0.917 0.977 0.917 10.986 10.984 0.984 1111111
0.6667 486 243 Ind 0.100 0 0.139 0.222 0.250 0.278 0.208   0.500 0.667 0.500 0.667 0.704 0.778 0.848 0.815 10.926 0.952 0.917 0.958 110.952 1111
0.6308 1201 703 MW 0.188 0.125 0.229 0.323 0.333 0.333 0.328 0.500   0.547 0.510 0.577 0.611 0.667 0.682 0.722 0.725 0.764 0.839 0.859 0.859 0.903 0.875 0.893 0.920 0.944 0.963 0.963
0.6150 1171 733 WAC 0.038 0 0.115 0.219 0.167 0.250 0.156 0.333 0.453   0.448 0.553 0.639 0.639 0.761 0.750 0.850 0.819 0.946 0.891 0.922 0.986 0.969 0.964 0.977 111
0.6097 1712 1096 CUSA 0.133 0.167 0.188 0.306 0.285 0.292 0.292 0.500 0.490 0.552   0.564 0.602 0.648 0.697 0.676 0.750 0.759 0.807 0.833 0.854 0.833 0.875 0.881 0.894 0.917 0.950 0.942
0.5877 1780 1249 MAC 0.046 0 0.115 0.192 0.179 0.218 0.163 0.333 0.423 0.447 0.436   0.590 0.632 0.678 0.701 0.800 0.795 0.890 0.856 0.865 0.940 0.923 0.912 0.958 0.991 0.977 0.985
0.5274 1125 1008 SoCon 0.044 0 0.093 0.167 0.148 0.185 0.153 0.296 0.389 0.361 0.398 0.410   0.568 0.556 0.642 0.644 0.704 0.810 0.806 0.806 0.877 0.861 0.841 0.899 0.938 0.922 0.933
0.4965 1059 1074 SBC 0.044 0 0.065 0.111 0.093 0.157 0.083 0.222 0.333 0.361 0.352 0.368 0.432   0.505 0.580 0.533 0.630 0.778 0.792 0.806 0.877 0.833 0.857 0.909 0.951 0.956 0.967
0.4847 1253 1332 CAAF 0.009 0 0.038 0.091 0.053 0.144 0.023 0.152 0.318 0.239 0.303 0.322 0.444 0.495   0.606 0.709 0.636 0.779 0.807 0.784 0.889 0.886 0.857 0.917 0.970 0.964 0.972
0.4435 946 1187 MVC 0.033 0 0.046 0.083 0.074 0.111 0.083 0.185 0.278 0.250 0.324 0.299 0.358 0.420 0.394   0.400 0.556 0.730 0.708 0.750 0.815 0.722 0.810 0.869 0.926 0.933 0.956
0.4415 532 673 GWest 0 0 0 0 0 0.100 0 0 0.275 0.150 0.250 0.200 0.356 0.467 0.291 0.600   0.511 0.743 0.875 0.750 0.978 0.850 0.857 0.927 111
0.3826 816 1317 BSky 0.011 0 0.028 0.037 0.037 0.093 0.014 0.074 0.236 0.181 0.241 0.205 0.296 0.370 0.364 0.444 0.489   0.667 0.639 0.653 0.728 0.722 0.746 0.778 0.840 0.822 0.856
0.3019 505 1168 Pat 0 0 0.012 0.036 0.012 0.036 0 0.048 0.161 0.054 0.193 0.110 0.190 0.222 0.221 0.270 0.257 0.333   0.554 0.589 0.508 0.607 0.653 0.740 0.810 0.800 0.843
0.2836 540 1364 SLC 0.013 0 0.021 0.031 0.031 0.042 0.016 0.083 0.141 0.109 0.167 0.144 0.194 0.208 0.193 0.292 0.125 0.361 0.446   0.563 0.514 0.563 0.661 0.636 0.681 0.775 0.738
0.2642 503 1401 Ivy 0 0 0.010 0.031 0.021 0.063 0.016 0.042 0.141 0.078 0.146 0.135 0.194 0.194 0.216 0.250 0.250 0.347 0.411 0.438   0.458 0.500 0.589 0.625 0.639 0.713 0.700
0.2527 539 1594 OVC 0 0 0 0 0 0.009 0 0 0.097 0.014 0.167 0.060 0.123 0.123 0.111 0.185 0.022 0.272 0.492 0.486 0.542   0.500 0.603 0.694 0.741 0.811 0.822
0.2479 240 728 1AAInd 0 0 0 0 0 0.021 0 0 0.125 0.031 0.125 0.077 0.139 0.167 0.114 0.278 0.150 0.278 0.393 0.438 0.500 0.500   0.607 0.614 0.667 0.725 0.725
0.2080 348 1325 BigS 0 0 0 0.012 0.012 0.024 0 0.048 0.107 0.036 0.119 0.088 0.159 0.143 0.143 0.190 0.143 0.254 0.347 0.339 0.411 0.397 0.393   0.532 0.524 0.586 0.629
0.1752 453 2132 MEAC 0 0 0 0 0 0.015 0 0 0.080 0.023 0.106 0.042 0.101 0.091 0.083 0.131 0.073 0.222 0.260 0.364 0.375 0.306 0.386 0.468   0.525 0.618 0.600
0.1500 320 1813 NE 0 0 0 0 0 0 0 0 0.056 0 0.083 0.009 0.062 0.049 0.030 0.074 0 0.160 0.190 0.319 0.361 0.259 0.333 0.476 0.475   0.589 0.567
0.1242 293 2067 SWAC 0 0 0 0 0 0 0 0 0.038 0 0.050 0.023 0.078 0.044 0.036 0.067 0 0.178 0.200 0.225 0.288 0.189 0.275 0.414 0.382 0.411   0.530
0.1186 280 2080 Pio 0 0 0 0 0 0 0 0 0.038 0 0.058 0.015 0.067 0.033 0.028 0.044 0 0.144 0.157 0.263 0.300 0.178 0.275 0.371 0.400 0.433 0.470

This conference rating (I sorted by the percentage of pairwise-comparisons in which the conference's teams came out better) including the conference-pairwise results is more useful than other "conference rankings based upon ratings" because it easier to see how even though conference B is lower on the list than conference A, it still might be ahead of conference A if they were the only two conferences. For example, the ACC does better against the entire field than the Big 10, but the Big 10 does better against the ACC than vice versa. (There's a math lesson here but I'll save that for a different audience.)

There's more to like about this method. If you save the intermediate results for each team you get a rough approximation of how each team would compare to teams in conferences other than its own. This is highly useful in an era when lots of teams are considering changing conferences.


Notes

  1. All of the examples in this essay are based upon Jeff Sagarin's "composite" rating as of 16 October 2011. The rating was chosen because the data is public and includes sufficient detail to illustrate each of the methods included in the article.
  2. μ is just the aritmentical average of all teams' ratings. σ is the standard deviation. The chart just places each team in a ½σ-wide bucket based upon its rating.