Grading the 2007 Conferences

© Copyright 2008, Paul Kislanko

10 January 2008

Now that another season is in the books we can get back to the interminable my conference is better than your conference arguments, pointless as they may be. There are nearly an infinity of comparitive metrics typically used in such arguments, most of which are subject to (mis)interpretation.

The problem is that conferences don't play the games. Any metric that aggregates team ratings into conference ratings is going to be more or less affected by extremes on either end of the "good-bad" spectrum as measured against any particular criterion. In this column we will apply a rating to all of the Football Bowl Subdivision teams and then use a simple counting stat to find out where the teams in each conference fall.

Grading "on the curve"

The first step is to find a suitable rating system that is normally distributed. The one for which I have rating values for that ranks highest in terms of Weighted Retrodictive Ranking Violations is the Iterative Strength of Victory.
The ISOV WRRV value is very slightly better than that for the Massey BCS rating. At this writing it is the 10th best of 84 published computer ratings. I'll describe the WRRV in more detail next time, but basically it is a function of the size of an "upset" in MOV and the difference in ranks that involve a violation, and the rank of the losing team.

Because of the way it is calculated the ISOV is approximately normally distributed - 65 percent of the values are within one standard deviation from the mean.
ISOV Grade Distribution - all FBS 2007
Grade distribution by ISOV

Here we've just used "grades" to indicate the distance from the mean (μ) in standard deviations (σ). If a team's rating is greater than or equal to μ − ½ × σ but less than μ + ½ × σ we assign it a "C"; the other grades are assigned higher or lower grades based upon (½)σ increments above or below "C".

We could just reproduce the chart above for each conference to count the number of conference teams that receivd each grade, but since there are different numbers of teams in various conferences it is more informative to tabulate the percentage of teams in each conference that received the specific grades.
Grade < All SEC P10 BigE B10 B12 ACC MW SBC CUSA WAC Ind MAC ND
F -∞ μ-2σ 0                        
E μ-2σ μ-1.5σ 10               25.0 8.3 33.3 33.3 23.1
D μ-1.5σ μ-σ 14         8.3 8.3   25.0 25.0 22.2 33.3 30.8
C- μ-σ μ-0.5σ 20     12.5 18.2 8.3 8.3 44.4 25.0 25.0 11.1   30.8 100.0
C μ-0.5σ μ+0.5σ 32 16.7 20.0 12.5 27.3 33.3 50.0 22.2 12.5 41.7 33.3 33.3 15.4
C+ μ+0.5σ μ+σ 26 50.0 50.0 37.5 36.4 16.7 25.0 22.2 12.5        
B μ+σ μ+1.5σ 10 16.7 20.0 25.0 9.1 8.3 8.3 11.1          
A μ+1.5σ μ+2σ 6 8.3 10.0 12.5   25.0              
A+ μ+2σ +∞ 2 8.3     9.1                
    % ≥ C+ 36.7 83.3 80 75 54.5 50 33.3 33.3 12.5 0 0 0 0 0
    % ≥ B 15 33.3 30 37.5 18.2 33.3 8.3 11.1 0 0 0 0 0 0

Why is that a better measure?

Good question. It probably isn't a very good metric, because the ISOV is a predictive rating, and without some meta-ranking help isn't all that good at rating teams' overall seasons.
A fascinating aspect of predictive ratings is that even when they're right they can be wrong. For instance, the ISOV correctly predicted Mississippi State's win over Auburn early in the year, but now rates Auburn significantly better than Mississippi State, so that game is now a "retrodictive violation" for the ISOV, even though the ISOV predicted that game correctly. Likewise ISOV ratings for teams that played games it incorrectly predicted have been adjusted to the point where those games are not retrodictive violations.

I'll address the problem of measuring rankings later.

To account for the "fuzziness" inherent to predictive systems we perform the same analysis using a retrodictive system, and the best one (by the WRRV criterion) for which we have rating values is the Massey-BCS. It also distributes rating values in an approximately normal manner, with 64.2 percent of observed values falling within one standard deviation from the mean:
MB Grade Distribution - all FBS 2007
Grade distribution by Massey-BCS

Again we count the percentage of teams in each conference that receive a given grade:
Grade < All SEC P10 B12 ACC BigE B10 MW WAC ND SBC CUSA Ind MAC
F -∞ μ-2σ 2               11.1     8.3  
E μ-2σ μ-1.5σ 8               22.2   25.0 25.0   7.7
D μ-1.5σ μ-σ 12       8.3   9.1 11.1   12.5 16.7 66.7 30.8
C- μ-σ μ-0.5σ 16     8.3   12.5   22.2 33.3   25.0 16.7   38.5
C μ-0.5σ μ+0.5σ 41 16.7 40.0 33.3 41.7 37.5 45.5 44.4 22.2 100.0 37.5 33.3 33.3 23.1
C+ μ+0.5σ μ+σ 20 33.3 20.0 25.0 33.3 37.5 27.3 11.1          
B μ+σ μ+1.5σ 16 33.3 30.0 16.7 16.7 12.5 18.2 11.1 11.1        
A μ+1.5σ μ+2σ 5 16.7 10.0 16.7                  
A+ μ+2σ +∞                          
    % ≥ C+ 34.2 83.3 60.0 58.3 50.0 50.0 45.5 22.2 11.1 0.0 0.0 0.0 0.0 0.0
    % ≥ B 17.5 50.0 40.0 33.4 16.7 12.5 18.2 11.1 11.1 0.0 0.0 0.0 0.0 0.0

In general I don't think there's a valid way to sum up team performance into conference "performance", but if you must order conferences by which had the best teams this year, this is as good a measure as any. You could give the SEC and Pac 10 an "A", the Big 12, ACC, Big East and Big Ten a "B", the Mountain West and WAC a "C" and every other conference an "incomplete" if you're an easy a grader as I am.

More interesting metrics

As I said above I don't believe there's an accurate way to aggregate teams' success into conference-level equivalents, but there probably is something to characterizing conferences by "style." It doesn't take sophisticated analysis to notice that in 2007 the Big 12 was dominated by offenses and the ACC either by defenses or the lack of offenses.

Dirk Chatelain of the Omaha Post Herald wrote College Football: Big 12 scoring was tops in nation pointing out that

Big 12 teams averaged 33.4 points per game in 2007, a mark believed to be the highest conference average in Division I-A history, according to an NCAA official.

Five of the nation's top nine offenses came from the Big 12 Nebraska ranked ninth. Oklahoma was 19th in the country, but couldn't even crack the Big 12's top half.

about which one smart-aleck SEC fan (is that redundant?) remarked on a message board:
That makes perfect sense. When your opponents don't play defense, you're going to score more.
So the question is, "which is it, great offenses or mediocre defenses?"

When the question came up in 2005 (regarding Western offenses vs southeastern - ACC more than SEC in that year) I came up with a way to mormalize scoring stats. Basically you just use the available game data to define a model that can be used to calculate what teams' average scores might be if they'd played every other team in division 1A.

The answer this year turns out to be the same as then - it's some of both. Even if every team had played the same defenses, the Big 12 in aggregate had better offenses than any other conference.

Rank Distribution - Normalized Scoring Offense
16 Jan 2008 1:50pm (Central)

ix WtdMed Best Med Worst Conf T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13
1 -25.58 4 18 111 B12 4 6 7 11 15 18 23 27 45 51 95 111
2 -23.50 5 29 82 P10 5 9 14 25 29 40 41 56 64 82    
3 -23.42 1 19 87 SEC 1 2 10 13 17 19 44 49 52 72 73 87
4 -21.88 3 28 105 BigE 3 8 16 28 32 47 66 105        
5 -15.82 12 38 89 B10 12 22 24 30 36 38 39 57 69 70 89  
6 4.17 21 62 103 ACC 21 34 46 55 58 62 63 68 83 84 93 103
7 8.67 37 77 99 MW 37 48 50 53 77 79 81 94 99      
8 15.58 33 75 115 CUSA 33 43 59 65 74 75 76 85 88 90 104 115
9 19.11 20 107 114 WAC 20 35 42 67 107 108 109 110 114      
10 24.13 26 92 119 SBC 26 61 80 92 97 98 100 119        
11 28.00 31 116 117 Ind 31 116 117                  
12 32.85 54 96 120 MAC 54 60 71 78 86 91 96 102 106 112 113 118 120
13 41.00 101 101 101 ND 101                      

The "weighted median" combines how much better or worse the best (worst) teams in the conference are than the middle of the whole field. A larger negative number means the conference is "top-heavy" and a large positive number means it is "bottom-heavy."

There are still six Big 12 teams in the top 20, but there are also six SEC teams, and while the Big 12 overall sorts into 1st, it also has two teams with offenses worse than the worst offenses in the Pac 10 and SEC. A more enlightening view of the same data emphasizes that it is team performance that matters more. In this table we list the teams by rank and conference, with conference champions indicated by a †:

N_PF Rec B12 P10 SEC BigE/ND B10 ACC nBCS
1 9-4     Florida      
2 12-2     LSU†      
3 11-2       West Virginia†    
4 12-2 Missouri          
5 9-4   Oregon        
6 11-3 Oklahoma†          
7 12-1 Kansas          
8 9-4       South Florida    
9 11-2   Southern California†        
10 8-5     Arkansas      
11 10-3 Texas          
12 11-2         Ohio State†  
13 8-5     Kentucky      
14 10-3   Arizona St        
15 9-4 Texas Tech          
16 10-3       Cincinnati    
17 11-2     Georgia      
18 7-6 Oklahoma St          
19 10-4     Tennessee      
20 10-3             Boise St
21 9-4           Clemson
22 7-6         Michigan St  
23 5-7 Nebraska          
24 9-4         Penn State  
25 4-9   Washington        
26 8-4             Troy
27 5-7 Kansas St          
28 6-6       Louisville    
29 7-6   California        
30 9-4         Illinois  
31 8-5             Navy
32 8-5       Rutgers    
33 10-4             Tulsa
34 11-3           Virginia Tech†
35 12-1             Hawaii†WAC
36 9-4         Michigan  
37 11-2             BYU†MW
38 9-4         Wisconsin  
39 8-5         Purdue  
40 9-4   Oregon St        
41 5-7   Arizona        
42 9-4             Fresno St
43 10-4             UCF†CUSA
44 9-4     Auburn      
45 7-6 Texas A&M          
46 11-3           Boston College
47 10-3       Connecticut    
48 9-4             Air Force
49 6-6     South Carolina      
50 9-4             Utah
51 6-7 Colorado          
52 7-6     Alabama      
53 8-5             TCU
54 8-6             Central Michigan†MAC
55 8-5           Wake Forest
56 5-7   Washington St        
57 7-6         Indiana  
58 7-6           Florida St
59 8-5             East Carolina
60 7-6             Ball State
61 8-5             Florida Atlantic†SBC
62 9-4           Virginia
63 6-7           Maryland
64 6-7   UCLA        
65 8-5             Houston
66 5-7       Pittsburgh    
67 6-7             Nevada
68 7-6           Georgia Tech
69 6-6         Northwestern  
70 1-11         Minnesota  
71 5-7             Toledo
72 5-7     Vanderbilt      
73 8-5     Mississippi St      
74 7-6             Southern Miss
75 4-8             UTEP
76 3-9             Rice
77 9-4             New Mexico
78 8-5             Bowling Green
79 3-9             Colorado St
80 5-7             Middle Tenn St
81 4-8             San Diego St
82 4-8   Stanford        
83 5-7           Miami-Florida
84 4-8           North Carolina
85 7-6             Memphis
86 6-6             Ohio
87 3-9     Mississippi      
88 3-9             Marshall
89 6-6         Iowa  
90 1-11             SMU
91 5-7             Western Michigan
92 6-6             UL Monroe
93 5-7           North Carolina St
94 5-7             Wyoming
95 3-9 Iowa State          
96 5-7             Buffalo
97 5-7             Arkansas St
98 3-9             UL Lafayette
99 2-10             UNLV
100 2-10             North Texas
101 3-9       Notre Dame    
102 4-8             Eastern Michigan
103 1-11           Duke
104 4-8             Tulane
105 2-10       Syracuse    
106 6-7             Miami-Ohio
107 2-10             Utah St
108 4-9             New Mexico St
109 5-7             Louisiana Tech
110 1-11             Idaho
111 3-9 Baylor          
112 3-9             Kent St
113 4-8             Akron
114 5-7             San Jose St
115 2-10             UAB
116 3-9             Army
117 7-5             Western Kentucky
118 2-10             Northern Illinois
119 1-11             Florida Intl
120 4-8             Temple

Among the BCS autobid conferences, only the Big East and Big Ten were won by the teams with the best scoring offenses. Even those might have been coincidences, as we see from the scoring defense data.

The SEC is caricatured (not characterized) by the talking cliche-spewing heads as being the toughest conference from a defensive standpoint. That is only true to the extent that the worst defense in the SEC was still better than over half the defenses in the Bowl Subdivision - the best defenses weren't in the SEC - the best offenses were!

In fact, five BCS autobid conferences plus the Mountain West had exactly three teams in the top 20. The ACC only had two, which somewhat belies the notion that the offenses looked so bad because of the good defenses they faced.

Rank Distribution - Normalized Scoring Defense
16 Jan 2008 1:45pm (Central)

ix WtdMed Best Med Worst Conf T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13
1 -28.25 4 33 59 SEC 4 5 10 21 32 33 36 39 40 48 54 59
2 -25.40 2 26 61 P10 2 17 18 24 26 43 44 53 58 61    
3 -23.75 3 23 85 BigE 3 12 16 23 42 45 64 85        
4 -21.92 6 30 83 ACC 6 11 22 28 29 30 34 38 52 56 68 83
5 -18.36 1 31 93 B10 1 8 13 25 27 31 47 62 70 81 93  
6 -14.44 9 41 84 MW 9 15 19 37 41 60 72 73 84      
7 -9.00 7 51 103 B12 7 14 20 35 50 51 55 57 66 75 79 103
8 5.00 65 65 65 ND 65                      
9 29.13 46 82 119 SBC 46 74 80 82 87 110 115 119        
10 31.11 49 96 118 WAC 49 69 77 95 96 98 102 116 118      
11 35.77 71 99 112 MAC 71 78 86 89 94 97 99 100 101 104 105 109 112
12 36.17 63 92 120 CUSA 63 67 76 88 91 92 106 107 113 114 117 120
13 43.00 90 108 111 Ind 90 108 111                  

Again, the list by team is telling. In the two BCS conferences won by the best scoring offense, the champions were also the best scoring defense. It was only in the Big 12 where neither the best offense nor the best defense won, and in that case the champion was second in both categories.

And it must be conceded that the smart-aleck had a point: in scoring defense adjusted for strength of schedule, not only did all other BCS conferences rank higher than the Big 12, so did the Mountain West. (Aside: It is not a coincidence that the MW had the best bowl season of any multi-bowl conference, at 4-1. Their games were mostly defensive mismatches - their teams played D, their opponents mostly didn't.)

N_PA Rec SEC P10 BigE/ND ACC B10 B12 nBCS
1 11-2         Ohio State†  
2 11-2   Southern California†        
3 11-2     West Virginia†      
4 12-2 LSU†          
5 9-4 Auburn          
6 11-3       Virginia Tech†    
7 12-1           Kansas
8 9-4         Penn State  
9 9-4             Utah
10 11-2 Georgia          
11 9-4       Clemson    
12 10-3     Cincinnati      
13 9-4         Michigan  
14 11-3           Oklahoma†
15 8-5             TCU
16 10-3     Connecticut      
17 10-3   Arizona St        
18 9-4   Oregon        
19 11-2             BYU†MW
20 12-2           Missouri
21 9-4 Florida          
22 9-4       Virginia    
23 9-4     South Florida      
24 9-4   Oregon St        
25 9-4         Illinois  
26 6-7   UCLA        
27 9-4         Wisconsin  
28 11-3       Boston College    
29 7-6       Florida St    
30 6-7       Maryland    
31 6-6         Iowa  
32 7-6 Alabama          
33 5-7 Vanderbilt          
34 7-6       Georgia Tech    
35 10-3           Texas
36 6-6 South Carolina          
37 9-4             New Mexico
38 8-5       Wake Forest    
39 10-4 Tennessee          
40 8-5 Mississippi St          
41 9-4             Air Force
42 8-5     Rutgers      
43 7-6   California        
44 5-7   Arizona        
45 5-7     Pittsburgh      
46 8-4             Troy
47 7-6         Michigan St  
48 8-5 Arkansas          
49 10-3             Boise St
50 7-6           Texas A&M
51 9-4           Texas Tech
52 4-8       North Carolina    
53 4-9   Washington        
54 8-5 Kentucky          
55 7-6           Oklahoma St
56 5-7       Miami-Florida    
57 6-7           Colorado
58 4-8   Stanford        
59 3-9 Mississippi          
60 5-7             Wyoming
61 5-7   Washington St        
62 8-5         Purdue  
63 7-6             Southern Miss
64 6-6     Louisville      
65 3-9     Notre Dame      
66 5-7           Kansas St
67 10-4             UCF†CUSA
68 5-7       North Carolina St    
69 9-4             Fresno St
70 7-6         Indiana  
71 7-6             Ball State
72 2-10             UNLV
73 3-9             Colorado St
74 5-7             Middle Tenn St
75 5-7           Nebraska
76 8-5             East Carolina
77 12-1             Hawaii†WAC
78 6-7             Miami-Ohio
79 3-9           Iowa State
80 6-6             UL Monroe
81 6-6         Northwestern  
82 8-5             Florida Atlantic†SBC
83 1-11       Duke    
84 4-8             San Diego St
85 2-10     Syracuse      
86 5-7             Buffalo
87 5-7             Arkansas St
88 8-5             Houston
89 5-7             Western Michigan
90 8-5             Navy
91 3-9             Marshall
92 10-4             Tulsa
93 1-11         Minnesota  
94 8-5             Bowling Green
95 6-7             Nevada
96 5-7             San Jose St
97 4-8             Temple
98 5-7             Louisiana Tech
99 8-6             Central Michigan†MAC
100 4-8             Akron
101 4-8             Eastern Michigan
102 2-10             Utah St
103 3-9           Baylor
104 3-9             Kent St
105 6-6             Ohio
106 4-8             Tulane
107 7-6             Memphis
108 3-9             Army
109 2-10             Northern Illinois
110 3-9             UL Lafayette
111 7-5             Western Kentucky
112 5-7             Toledo
113 4-8             UTEP
114 2-10             UAB
115 1-11             Florida Intl
116 1-11             Idaho
117 1-11             SMU
118 4-9             New Mexico St
119 2-10             North Texas
120 3-9             Rice

If we have to...

Rank Conf SD SO
1 SEC 1 3
1 P10 2 2
3 BigE 3 4
4 B12 7 1
5 ACC 4 6
5 B10 5 5
7 MW 6 7
8 SBC 9 10
8 WAC 10 9
10 CUSA 12 8
11 ND 8 13
12 MAC 11 12
13 Ind 13 11
I hope I've convinced you there's not even a theoretical way to rank conferences based upon one being "better" than another. But that doesn't mean we can't measure which of two "had a better year in terms of its teams' performances."

In all of the above analysis we've been aggregating team performance by conference. In the left sidebar we have the first comparison by conference metrics, namely the order defined by the rank distribution of its teams in scoring offense and scoring defense.

I didn't specifically break "ties", but ordered tied conferences by the normalized scoring defense metric, since from what we described above it appears that defense played the larger role in determining conference champions.

We're still not going to say "the SEC is better than the Big 12", or "the Pac 10 is better than the ACC." But whether by plain dumb luck or clever scheduling or, maybe, having better teams, we can safely say that the SEC and Pac 10 had better years than any of the other collections of teams called "conferences."

And there's lots of team-oriented data in these results that can be used to explain or analyze the crazy 2007 season.

But that can wait for the offseason, since this is too long already.