Wed 6 Oct 2004
I reckon the readers of Ephblog would like a market based solution to college rankings rather than US News & World Report’s rankings. Why base ranking on arbitrary criteria when you can examine what consumers actually select?
A group of economists (headlined by Harvard’s Caroline Hoxby, whose work is top notch) decided to test out the idea (in a paper that can be found here). They selected 510 high schools that typically send many students to competitive colleges. Guidance counsellors at these schools then randomly selected students to participate in the study. A host of questions were asked, but the most important issue was which schools were they accepted by and which school did they decide to attend?
The schools were then ranked using an algorithm for ranking chess players developed by a Hungarian mathematician named Elo (Jeff Sagarin uses the algorithm in his USA Today rankings of sports teams). Schools that accepted the same student were compared head to head. The school that was selected by the student was deemed the “winner” and the schools that were turned down by the student were deemed the “losers.” In this way, Hoxby and crew were able to come up with rankings (I loaded up the top 50 Download file).
Williams places 18th overall.
Among liberal arts schools, it placed 4th behind Amherst; Wellesley; and Swarthmore.
A couple of quick thoughts:
1) The survey of students is obviously incomplete. Hoxby and crew had limited resources and put together a very interesting dataset with what they had (and the difficulty in execution should not be under-estimated);
2) Self-selection is an issue for the design. Students pick to which schools they apply. Specialty schools (like Caltech and MIT) might receive a boost as a result of this selection process. It isn’t obvious to me where else bias from self-selection might creep into the estimates.
3) Selection on the part of schools is also a slight problem. Harvard accepts a small percentage of people. Presumably a large number of people who applied to Harvard would have accepted had they been accepted. Harvard cheats itself out of “wins” against other schools — not that it needs much help.
4) Oddly enough, Williams did best in Region #4, which contained Minnesota, Nebraska, Missouri, and Kansas. I’d like to think that my promotion back home was the cause, but I haven’t done any, so I guess that can’t be it.
5) In general, big schools were more of a draw than small schools. Given the prestige that comes with brand name schools, it should not be surprising that a boutique school like Williams is snubbed in favor of the Ivies.
6) I was a little surprised to see that my new home institution, Notre Dame, ranked more highly than Williams. Go Irish! Notre Dame would beat Williams at football, too.
7) I wonder if this ranking system sets up perverse incentives for admissions offices. US News and World Report definitely provides incentive to keep yields high, but this ranking system sets up LARGE incentives to not lose out to other schools. In order to game the system, admissions should not admit people who look destined to land at MIT or Caltech.
8) Contrariwise, this ranking punishes early admission because you don’t go head to head with other schools. Williams would like your most enthusiastic applicants to get into other schools and then pick Williams over Princeton and te other Ivies.
9) Markov Chain Monte Carlo is cool. I need to get better at WinBugs so I can do stuff like this.
October 6th, 2004 at 7:46 pm
Ugh. This methodology is about as sloppy as any other, with the added benefit of pretensions to being scientific. note that the appropriate analogy is Sagarin’s college football rankings. ‘Nuff said. For one thing, a whole lot of us did not go to these elite feeder high schools. for another, there may be lots of reasons to choose between two elite schools, and if we do not know those, the market explanation is only one possibility. US News is flawed, but surely it takes into account more variables that make for a “good college’ than does this one, and thus must be considered more valid in many ways, especially as the only input comes from people who have not yet been students at their schools for one day. this is akin to crowning a national college football championship based on what some computer geek says about the incoming recruiting class. It seems to me to get it all backward. “Precision” at the expense of accuracy is no virtue.
dc
October 7th, 2004 at 8:16 am
I agree with Derek — especially with his observation that the article has pretensions of being scientific.
October 7th, 2004 at 9:51 am
Let me get the four main objections straight:
1) A paper by economists from Harvard, UPenn, and BU shouldn’t have pretensions of being scientific;
2) Because a mathematics major from MIT (who later received an MBA from Indiana) recognized that the work of a Hungarian mathematician/physicist could be applied to sports rankings, automatically invalidates its use for academic questions;
3) Because their sample does not include every type of high school in the country, we can glean nothing from the choices made by seniors deciding between colleges;
4) The arbitrary weightings of the editors of US News & World Report are preferable to the average high school senior weighing options.
R#2: The fact that Sagarin was smart enough to adapt Elo’s algorithm from chess to football (and other sports) in no way invalidates its use in other venues. The same tool can be used in different venues to great profit. ‘Nuf said.
R#3: Yes, a more complete survey of high schools would have been preferable. That would have allowed ranking a wider range of schools. However, with limited resources, the researchers concentrated their efforts on schools where students might be making choices between several different options (and the brand name schools that people care about). But I don’t see how this invalidates the ranking completely. You would have to make an argument that the students from poor performing high schools would have a systematic different ranking from kids attending schools that regularly send students to elite colleges (and trust me, there is no such thing as an elite feeder school in Nebraska — their sample consisted of high schools that regularly send students to competitive colleges).
R#4: How do we know that the weight the editors of US News place on variables are the correct weights? How do we know that they even consider the correct factors? In contrast, high school students are making decisions based on: a) price; b) geography; c) campus culture; d) desirability of metro setting; e) job prospects; f) education quality; g) quality of dorms; h) particular programs available; i) what they hear from alums or the grapevine more generally; j) what they read in places like US News & World Report. The individual consumer then makes an idiosyncratic decision based upon her particular circumstances and tastes. However, averaging across these idiosyncratic tastes and financial circumstances, on can come up with a good idea of which schools are more desirable than others.
Thus, US News & World Report is a much more limited ranking than Hoxby’s consumer preference model. US News is a one-size fits all model. It decides what criteria is important and then weights them. Hoxby et al. allow consumers to decide which variables are important and how much each variable should be weighted.
Perhaps Derek is right in suggesting that high school seniors are pathologically uninformed consumers and that the rankings are therefore skewed. However:
1) Hoxby and crew only claim to be measuring consumer preference — students may not value pedagogy very highly;
2) What is the basis for this notion that high school student can’t make informed decisions about college? Yes, they do not know as much about the school as people who attend it, but surely some of that wisdom from attending the university filters down to the high school students. Student dissatisfaction would become evident during campus visits, reading through the student newspaper on-line, or speaking with alumni.
I’m not entirely satisfied with the paper. I offered a couple of critiques in my original posting. However, the complaints that because the system is used by USA Today for ranking sports teams and that they don’t consider enough criteria seem to be wrongheaded. Yes, economists have pretentions of making the “dismal science” into a real science. This paper doesn’t strike me as a bad step in that direction. At the very least, using consumer preference to rank colleges is an interesting addition to the sea of literature on the subject.
October 7th, 2004 at 3:04 pm
“You would have to make an argument that the students from poor performing high schools would have a systematic different ranking from kids attending schools that regularly send students to elite colleges.”
Done and done. Two books since 1990, just for example, “the high status track” and “Choosing colleges”.
The problem here is that college as a relatively non-quantifiable experience (yes, you get a BA but was it a “good learning experience”?) cannot be properly ranked. But we really like to rank things. But because the actual learning (both inside and outside the class) cannot be tested in a standardized tests, we instead have these bastardizations. Fun and interesting, but it tells me nothing about which school is “better” except at convincing high school kids to choose it over other schools, just like US News tells us nothing except that other schools think that one school is doing well and that it is financially stable. wow. That’s the stuff that made college special, not the late night conversations in the snack bar (i heard its not going well in mission) or the opportunity to know the college president as a colleague or the brooks late night par…wait, no, that wasn’t always such a good thing about college…;)
October 7th, 2004 at 5:06 pm
Dave –
You so warp and misrepresent my points that I am not certain that it is worth going much further, but I’ll bite.
1)You get point one exactly backward : It isn’t that economists (economics as a science does make me laugh though; pretension indeed) should not aim toward a sort of precision when they are engaging in things that are within that ambit. This simply is not. What makes a school great or good or important or valuable cannot be reduced to formulas. This is like trying to quantify sex, or fun at amusement parks, or why we like baseball. It is just irredeemably dumb.
2) No, you see, the rankings that Sagarin comes up with are just not that good. They are incredibly flawed. Sports are played on the fields, not on a spreadsheet. So yes, if the analogy that makes these rankings valid is with the Sagarin rankings, as I said, ’nuff said.
3)No, because the sample intentionally leaves out a very specific category it is incredibly flawed. This is social science 101 type stuff. I’m not going to give a primer on sample groups here.
4) US News is flawed. But it takes into account a whole lot more factors than this dopey concept does. I’d prefer the flawed flesh and blood factors that US News considers, covering a range from incoming student scores, satisfaction levels, retention and grad rates, data on professors, and so forth. Yes.
David — when you misrepresent what someone says, that is a form of lie. So when you say that I say that high school students are pathologically uninformed, and when I say no such thing, you are lying. It is slimy. I said no such thing. But what I did say, and I stand by this, is that when the sole method of assessment, the sole input, comes from students who have not spent one second as students on the campuses in question, it is incredibly flawed. So if your personal integrity does not matter to you, perhaps your intellectual integrity does. Come after my ideas if you’d like. Do not turn them into straw men so that you can win a debate. Is that too much to ask?
dc
October 7th, 2004 at 6:15 pm
Derek, I’m sorry if you felt that I misrepresented your points. My intention was to summarize for purposes of clarity. I apologize if my reading comprehension was not up to snuff (amazing how the SAT can be wrong about stuff like that).
On #1) Why shouldn’t economists try to use price theory to analyze a range of subjects? Pretty much every topic you listed (amusement parks, baseball, or even sex) has a price attached to it that has been determined by a market. That is, individuals apply personal cost-benefit analysis to a good (e.g., an amusement park) and attach a value to it (i.e., a price past which they will not purchase the good). The market price indicates the value consumers place upon the good given supply and effective demand (note: that is not to imply that markets are democratic or do not create winners and losers).
Why can’t the same type of analysis be applied to education? Is a Harvard education costing $35,000 a year better than the education a person would receive at her/his state school for $10,000? There is a price attached to the good. Students who are accepted by Harvard, Williams, and the University of Minnesota have a choice to make. What Hoxby et al tell us is that students systematically select Harvard over the other two.
on #2: Why is Sagarin incredibly flawed? Looking over his rankings for the 2003 NFL season (http://www.usatoday.com/sports/sagarin/nfl03.htm) they look pretty good to me. Same goes for college basketball (http://www.usatoday.com/sports/sagarin/bkt0304.htm) and pretty much any sport. The rankings don’t tell you who will win any particular game (as you point out, there is a reason they play the games). The rankings tell you which team is more likely to win.
And furthermore, your criticism should be focused on the case at hand. Why is it wrong to determine consumer preferences by the choices made by students who have the options between several separate schoosl? I don’t think the problem lies in the Elo algorithm. If there is a problem, my guess is that it lies in the selection process (but I can’t articulate why and am not certain that I am correct).
#3) The concern is external validity here, not internal validity. The specific sampling algorithm is only a problem if the preferences of students from “feeder” high schools (a bit of a misnomer, since the standard is sending several students to competitive colleges) differ from those at “non-feeder” schools. The former admissions counsellor claims this is the case and offers two books (I haven’t read the books). If this is the case, then it would suggest that the ranking system only applies to students from these feeder schools. It does not suggest that the entire enterprise is illegitimate. [Besides, given finite resources, it isn't clear to me why the authors should have sampled schools that rarely send students to competitive colleges. They decided to target the students who would be in a position to make choices between competitive colleges. The authors say they would like to expand the sample in future studies.]
#4: US News takes into account a handfull of factors. Presumably, the consumers making the choices take into account every factor that might be important to them. US News derives a weighting system based on some secret formula. Why should we believe the weightings that US News uses rather than the equilibrium of the prices?
Now, one reason not to believe the market’s selection is if the consumers are systematically misinformed. Individual consumers will make uninformed solutions, but such uninformed decisions will balance out and informed consumers will set the price. The problem is if there is a systematic pattern (pathology) of misinformation that creates market distortions (think stock market bubbles or purchasing bogus nutritional supplements). I interpreted your statement that “the only input comes from people who have not yet been students at their schools for one day” to be a statement about the nature of the consumers making the decision. I think that is a genuinely interesting critique and don’t think it is slanderous to attribute it to you. It represents my honest effort to categorize that clause in your post. [And I still think that the reasons I outlined in the earlier comment for believing that high school student can draw upon the experience of current and past students are worth considering.]
The point of the Elo college rankings is to indicate which schools students systematically select when given a choice. It does not say what a particular student will choose nor which college she should choose. The take away message is that on average Harvard is selected more often than any other school when in head-to-head competition. A particular student may choose differently. The ranking also let us know that an average student (among the type of schools sampled) were more likely to select Yale over Harvard than Williams over Harvard or Michigan over Harvard.
So I fail to see why the idea is “dopey.” I guess the crux of my question to you, or anyone else, is why/how does US News take into account more factors than consumer preferences? Individuals surely take into account a vast array of information when making their college decision (or if not, how are we to know what factors are decisive?). Why is it uninformative to aggregate their choices between schools?
I’m not trying to win a debate. I didn’t write the paper and have no stake in it one way or another. I just think this is an interesting new/different approach to ranking colleges.