Mathematicians Deconstruct US News College Rankings

An anonymous reader writes "US News makes a mint off its college rankings every year, but do they really give meaningful information? A pair of mathematicians argues that the data the magazine uses is all likely to be at least somewhat relevant, but that the way the magazine weights the different statistics is pretty arbitrary. After all, different people may have different priorities. So they developed a method to compute the rankings based on any possible set of priorities. To do it, they had to reverse-engineer some of US News's data. What they found was that some colleges come out on top pretty much regardless of the prioritization, but others move around quite a lot. And the top-ranked university can vary tremendously. Penn State, which is #48 using US News's methodology, could be the best university in the country, by other standards."

Comments

Reputation

[Reality Master 201]

A college degree is an education, and that should be of paramount concern. It's also nice to be in a place you'd enjoy living, etc.

But then there's reputation. You might get the same education at CMU and MIT, but if you're looking for jobs, all other things being equal, someone's gonna pick the MIT grad because it'd a bigger name. I realize it's variable across fields and with individuals, but names mean something to a lot of people, particularly when they're not really qualified to judge on merits.

Re:Reputation

[the_weasel]

And he became president. That's a pretty clear success for him. :-)

Re:Reputation

[jd]

Yes, general reputation matters. Cambridge and Oxford in the UK have enormous street cred with employers and it doesn't matter if they're the best in a given field. They'll always be highly regarded. Specific reputation also matters. A university known to have brilliant students and produce top-notch wizards (Hogwarts?) in that specific field will also count highly with an employer. (It'll be the name HR becomes extremely familiar with, if HR is bothering to track such things - and top employers are more likely to than bottom-of-the-barrel types.)

However, once you're past HR, you've the real professionals to contend with. Again, they'll recognize the big-general names and the big-in-field names if they're any good, but they'll also recognize heavily-published places because it's the published stuff that helps the professionals stay on top. HR doesn't read (and in some places I've applied, I wonder if they could read), so major research universities (where students are likely to be more current but not necessarily more educated) won't rank as high with them, but it should matter a great deal more to those in the field.

Since higher degrees are where people get deep into the research, such places should matter more to those going for a masters or PhD than for the first degree, but it is one of those things that does matter to some extent no matter what degree you're going for. After all, the lecturers can't be any better than the information they have available to them, so non-research places may be 5-10 years behind the curve, as they'll rely on secondary sources of information (such as textbooks) which will never be truly current. On the other hand, if you're the one defining what is current, you aught to be able to teach what is current.

Even then, it's no good if the lecturer knows something if they can't present it, so you still need more feedback. Ideally, you'd know how students change in ability from the start to the end of the course. Even a dumb lecturer can teach students who already know everything they need. The rate of change of student ability in relation to the rate of change expected for students of that level of ability is the magic number you need to tell if the lecturers are any good at imparting information. As I don't think any existing tests currently give you the data you'd need to calculate this, since you can't standardize a test that is inherently much more specialized and all tests these days are standardized (fools that they are), such a calculation is impossible. As such, you either have to pick a number out of thin air, or use surveys to get the students to pick the number out of thin air for you. Either way, it's not the number you want and there's no possible way of determining how close or distant it is. Without that number, the rest of the data only tells you what the upper theoretical limit is on quality, not what the practical day-to-day reality is.

Re:Reputation

[johnlcallaway]

Bzzzzt!!! Not always right.

A college degree is a piece of paper. But I'll concede that the education should be the paramount concern. I agree with a concept I read in Money magazine a few years ago when they analyzed how much a difference in salary people got depending on what school they went to and how much they spent. The gist of what they recommended was to get the basics from a community college that can transfer credits, then enroll in the more expensive places. Math is math, science is science, IT is IT up to a point. That way you don't spend two years figuring out that you suck at IT and spending a crap load of money doing it.

Not everyone puts a lot of value in a school's reputation. I'd rather work for a place that hires people based on their abilities instead of on a sheepskin. I worked with a VP of development that had a PhD in neuro-networks from MIT. Smart guy ... lousy to work with. Ego the size of Massachusetts, and the personality of a penny.

I don't even pay attention to whether someone has a degree or not when hiring admins or development staff. In fact, 'professional students' will probably fall down lower on my list than someone who has been attending local colleges taking specific courses. All I care about is how smart and curious they are, and lots of smart, curious people don't go to school. Anybody can learn to code, but the smart and curious people are really good at it. Some of the best IT people I have worked with in the last 25 years had very little college education.

You want to be a doctor or a lawyer or a consultant and have your own business?? Pay for the degree, many people put stock in it.

Aren't that smart?? Pay for the degree, it fools some people.

Otherwise, save your money. Learn what you need, go to a tech school or get a 4 year at a state school if it's that important to you.

If you are smart, curious, and have a strong work ethic, you will do fine.

Re:Reputation

[fakeplasticusername]

I think if you follow the point of the article to it's ultimate logical conclusion, it makes the point that perhaps the reputation that is such a giant draw isn't earned in all cases. Part of the reputation is probably a result of the US News's reports, I know when I was picking a school all of my contemporaries certainly viewed it as the bible from which was pronounced the word.

If the rankings were more statistically driven instead of by the whims of "experts", these unearned reputations might start to evaporate. The reputation of any social institution could be described as being based on the people that are produced as a result of it, the hype surrounding it, and of the network of people attached to it. Make the hype approximate the actual educational result produced by the school (based on your own personal priorities) and maybe the reputations will change on a long enough timeline.

Also, from the article:

The pair point out that their methods can't address another of the fundamental criticisms of the U.S. News evaluations, that the magazine chooses the wrong factors to base their evaluations on in the first place.

If there are serious criticisms about the factors U.S. News uses to rank colleges by even the flawed methods of these "experts", maybe a more statistical method is even more superior than the results of this preliminary study. If the rankings could be done in a superior fashion by an algorithm that accounts for ALL qualities of interest, configured to reflect personal priorities, there may be even more room for improvement. Not that there isn't a place for a personal touch, but the data should come first...

This is news?

[afidel]

Doesn't everyone know that the front page rankings are worthless and it's the per area/major rankings and the detailed information that's important? Also the rankings are only a place to start, you need to do an extended visit to your top 5 schools to see how likely you are to be compatible with the school.

So the best college then

[Mesa MIke]

... is the one that comes out on top after taking an average of all the different ranking methods.

When will this be publically available?

[agent4256]

So when will there be a site available so I can see how my college ranked based upon what I deem to be the most important?

US News could take this, print their magazine, then offer this "service" on their site, run by ad revenue to really give the student a run for their money when applying for a college.

... that is if they can afford it.

Blindingly obvious stuff makes headlines... again

[Geoffrey.landis]

Amazing how the blindingly obvious can get headlines.

If all the different criteria all gave the same result, then there would be no need to make a weighted average; you could just look at any single one. If they give different results, then of course the result will depend on how you weigh them. In fact, if a college ranks number one on any of the criteria, clearly a weighting exists to rank that college number one overall (just rate that one factor 100%...)

You don't need "a pair of mathematicians" to show that. A pair of high-school freshmen could do it.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Our university didn't feature in the top 10...

[syousef]

...so we made up our own top 10! We even made it sound more authoritative with better hand waving mathematics. Take that US News!

Just as SAT is useless after you're in college

[Overzeetop]

Once you get your first job, where you graduated from (name recognition) is less important than the intelligence of the student and what you're really done. Don't get me wrong, you should probably consider one of the "top 20" in your field, but you're just as likely to get a good (or better) answer from people in your future industry than from a magazine. There are exceptions to the rule, of course, but unless you happen to be in one of the few snobby professions it doesn't matter. Finding a good "fit" for college is almost as important as the curriculum itself.

Now, if you're going on to do something great (and almost all of you can put your hands down - you either weren't born with the brain or the parents; I'm included in that class, too, fwiw) you should consider finding the top graduate program in your field. Not one of the top, THE top, as judged by your peers. Then school will matter, because when you get near the top, snobbery is almost everything. Your parents, your intellegence, your charisma, and your degree for the "three of four" ticket to stardom. You can need at least three and get to the top. Actually, I think you can only have three - if you get all four your competition will be jealous and cut you down like a dog.

Less than useless....

[Afforess]

What does it matter how good a particular college is? What are they basing it off? Satisfaction polls? Tuition cost? Income? Alumni? I find that it is best to compare departments to find the best college (Compare Engineering Departments, Compare Math Departments... etc...) Who cares if a University makes #8 at US News and Reports, if it is because of its fine arts programs and you want a computer science major?

Re:Blindingly obvious stuff makes headlines... aga

[argiedot]

I thought the bit where the best result would be the point nearest the line from the origin to the polytope (containing the bunch of points representing universities' parameters) passing through the point representing weights was the important bit not that different weights give different results. While what you said _is_ blindingly obvious, the other thing isn't. It might be more useful to use this method than to simply recalculate all scores using the new weights.

And next.....

[Gat0r30y]

Huggins and Pachter are now applying their methods to voting in elections with more than two candidates.

Elections have more than two candidates?

damn I gotta get out of the US for a while

Rankings

[Anonymous Coward]

I've actually played with ranking data quite extensively, and usually for reasonable weighthings of the parameters the movement in position is in the order of plus/minus 5 places. Sure, Penn State would be number one if all one cared about is retention rates, but really nobody does. Instead we can define a range of reasonable weights for retention rates (say between 7% and 35% of the total weight) and test all possible combinations in that space, suddenly Penn State place goes up and down a fairly small amount.

A bigger concern is what is the value of selecting a school based on the ranking as a whole, without paying attention to the your likely area of major. Say, Yale is a great school but in CS is a non-entity. If you are positive CS is your thing, MIT, Stanford, Harvard and Princeton are far better choices.

Re:Playing the numbers

[blair1q]

What I want to know is why U.S. News considers itself qualified to rate colleges in the first place.

depends heavily on the field

[Trepidity]

In technical areas (e.g. engineering), reputation within the field matters a lot more than generic reputation. People at Boeing know what the good aerospace engineering places are, and hire accordingly. If you graduated from an Ivy with an unknown engineering program, you're more likely to get responses like, "huh, I didn't even know Yale had an engineering program". Meanwhile, if you graduated from a generally lesser-known school with a top-rated engineering program (e.g. Rose-Hulman or Harvey Mudd) you're going to get plenty of offers.

Re:and Harvard Business School

[ScreamingCactus]

Maybe he really is a genius... I think it would take someone very clever to appear that inane and yet still get elected. Twice.

Re:Blindingly obvious stuff makes headlines... aga

[Anonymous Coward]

The take-home message might have been obvious, but what the "pair of mathematicians" did was study exactly how the rankings depend on the weights. For example they figured out all the possible rankings for each university, and computed the regions of weights which gave each ranking. You can't just read off that kind of info from the university measurements. If a university is ranked #10 in all the different categories, then you might be able to make that university #1, or maybe not

Re:Anecdotal

[Icarium]

Over 6 years, the school graduates 60 full-time students (let's say they're spread out evenly at 6 per year)

Your law school is teaching you some strange maths...

Re:That's actually pretty sad...

[Mgccl]

It's like racism. Instead of race, it's college names.