Rankings
Rankings are a big topic around here, as I suspect they are at any top-rated department. On the other hand, I’ve never noticed that we do anything to influence how we are ranked, other than do great research and teaching. Probably that’s how it should be, though at times I’m sure many of us wonder if there is more to the “game” than that. My guess is that all the good places are simply too busy doing more important things to worry about this too much.
The topic of rankings comes to mind right now because I recently attended a talk by Ralph Cicerone, the President of the National Academies. The topic of the talk was the Assessment of Research Doctorate Programs, which was conducted by the National Research Council and scheduled to be released in the spring of 2008. The talk was presented at the annual meeting of the Council of Graduate Schools, which brings together the nation’s Graduate Deans and VP’s for Research to share ideas on administration of graduate programs. I am no longer the vice provost, but I had committed to giving a talk on interdisciplinary research at this meeting almost a year ago. And while vice provost, I spearheaded (jointly with Indira Nair and others) our university’s response to the NRC’s assessment.
The NRC’s assessment, widely considered to be the “gold standard”, is due out in the spring of 2008. There have been two previous NRC assessments. The last one was published in 1995, and while it provided perhaps the most credible ranking of the nation’s Ph.D. programs, it was also fraught with problems. Chief among the many problems was an over-dependence on a reputational survey of Deans, as opposed to more objective measures of excellence and productivity. Some of you might recall, also, that some of our academic units — most notably the Robotics Institute — didn’t quality for the assessment. This meant that, according to the 1995 NRC assessment, CMU had almost no research activity in the robotics area!
For the new assessment, the NRC was extremely ambitious. It scrapped the dependence on reputational surveys, and instead decided to gather large amounts of data from faculty and administrators, as a basis for objective measures. They collected, for example, CVs from every faculty member and hired a staff to go through each publication, count citations, etc.
The NRC also took steps to avoid what it calls “spurious precision” in the rankings. In other words, instead of trying to make fine-grained distinctions (e.g., is the 11th ranked department really better than the 12th ranked department?), the new assessment will provide ranked groups of departments/programs. Departments will be ranked on many dimensions, some scholarly (e.g., citation counts, impact factor, etc.), some administrative (such as tenure rates, facilities, etc.), and some education-oriented (e.g., time to degree completion, etc.). Finally, some of the advantages of larger departments will be diminished by a greater dependence on per-capita measures.
Gathering all the data for this level of assessment, for over 5,000 programs, was a daunting task. Even getting the questionnaires right was non-trivial. And, of course, there were lots of problems. And these problems have led to quite a bit of grumbling and controversy by many universities.
Despite the problems, I believe that the new assessment will be, for the most part, sound. I think it will be considered definitive by almost everyone who reads it. So, the obvious question is, how will we make out in it? Well, it’s impossible to know for sure. We are most likely the largest computer science Ph.D. program in the assessment, with about 140 faculty responding to the survey. That’s massive. But size doesn’t count for much, because we expect per-capita measures to dominate (and we must admit that not all 140 faculty responding are uniformly productive). On the other hand, the coverage of areas, the number of “stars” on the faculty, the completion rates for our Ph.D. students, the amount of research expenditures per faculty member, etc. all should look very good. We’ll see.
At a minimum, this time the NRC assessment will show that we have a ton of robotics research going on here!
Peter Lee @ December 18, 2007