Slashdot Mirror
Mathematicians Deconstruct US News College Rankings
An anonymous reader writes "US News makes a mint off its college rankings every year, but do they really give meaningful information? A pair of mathematicians argues that the data the magazine uses is all likely to be at least somewhat relevant, but that the way the magazine weights the different statistics is pretty arbitrary. After all, different people may have different priorities. So they developed a method to compute the rankings based on any possible set of priorities. To do it, they had to reverse-engineer some of US News's data. What they found was that some colleges come out on top pretty much regardless of the prioritization, but others move around quite a lot. And the top-ranked university can vary tremendously. Penn State, which is #48 using US News's methodology, could be the best university in the country, by other standards."
A college degree is an education, and that should be of paramount concern. It's also nice to be in a place you'd enjoy living, etc.
But then there's reputation. You might get the same education at CMU and MIT, but if you're looking for jobs, all other things being equal, someone's gonna pick the MIT grad because it'd a bigger name. I realize it's variable across fields and with individuals, but names mean something to a lot of people, particularly when they're not really qualified to judge on merits.
At best, they provide a filter for individuals of a certain level of ability of competence, e.g. the average graduate from a school #1 is going to be more capable than the average graduate from school #100.
Doesn't everyone know that the front page rankings are worthless and it's the per area/major rankings and the detailed information that's important? Also the rankings are only a place to start, you need to do an extended visit to your top 5 schools to see how likely you are to be compatible with the school.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Penn State, for example, was 48 according to the magazine's criteria, but it could also be as high as 1 or as low as 59. That variability evolves because Penn State is the best at making sure students graduate
which is the best university to meet pretty and easy girls?.
Because knowledge is power! Hourray!
Penn State, which is #48 using US News's methodology, could be the best university in the country, by other standards.
Was that standard "name which sound most like a Prison". Its good to get some measure of how good a team is but there are of course other approaches, one would be to have a league system with a set play-off format (rather than 100 "bowl" games) with a number if tiers, bottom few teams drop down a tier, winners of the various tiers below move up.
The whole point of the US News figures is that they are arbitrary, this isn't about really working out who is best over the course of the season its about having something to talk about around the water cooler, it would be miles more boring if you know that winning a game by 4 points when someone else loses by 2 means that your ranking goes up. You'd have commentators talking all the time about the "real time change" in the figures, it would be mind-numbingly boring.
Keep the arbitrary figures lets just have a proper league system instead rather than a flat "randomly play teams" format.
An Eye for an Eye will make the whole world blind - Gandhi
... is the one that comes out on top after taking an average of all the different ranking methods.
And even if he weren't a member of one of the more powerful families in the US, he probably would have done pretty well for himself having those names under his belt.
Whether or not he actually learned anything of value, though, is a matter we must pass over in silence.
So when will there be a site available so I can see how my college ranked based upon what I deem to be the most important?
... that is if they can afford it.
US News could take this, print their magazine, then offer this "service" on their site, run by ad revenue to really give the student a run for their money when applying for a college.
If all the different criteria all gave the same result, then there would be no need to make a weighted average; you could just look at any single one. If they give different results, then of course the result will depend on how you weigh them. In fact, if a college ranks number one on any of the criteria, clearly a weighting exists to rank that college number one overall (just rate that one factor 100%...)
You don't need "a pair of mathematicians" to show that. A pair of high-school freshmen could do it.
http://www.geoffreylandis.com
Comment removed based on user account deletion
...so we made up our own top 10! We even made it sound more authoritative with better hand waving mathematics. Take that US News!
These posts express my own personal views, not those of my employer
Percentage female. If you are going into engineering (face it, you are) try to get in to a liberal-arts dorm.
More obvious insight which should get me some money somehow: even if a college is ranked #1, you MIGHT get a better educational experience somewhere else!
Where's my article and money?
Once you get your first job, where you graduated from (name recognition) is less important than the intelligence of the student and what you're really done. Don't get me wrong, you should probably consider one of the "top 20" in your field, but you're just as likely to get a good (or better) answer from people in your future industry than from a magazine. There are exceptions to the rule, of course, but unless you happen to be in one of the few snobby professions it doesn't matter. Finding a good "fit" for college is almost as important as the curriculum itself.
Now, if you're going on to do something great (and almost all of you can put your hands down - you either weren't born with the brain or the parents; I'm included in that class, too, fwiw) you should consider finding the top graduate program in your field. Not one of the top, THE top, as judged by your peers. Then school will matter, because when you get near the top, snobbery is almost everything. Your parents, your intellegence, your charisma, and your degree for the "three of four" ticket to stardom. You can need at least three and get to the top. Actually, I think you can only have three - if you get all four your competition will be jealous and cut you down like a dog.
Is it just my observation, or are there way too many stupid people in the world?
Several of the metrics that U.S. News uses do seem to be arbitrarily weighted, leading to some bizarre contortions on the part of the various schools to enhance their ratings. Most of the data is self-reported by the universities, which clearly provides a powerful motivation to spin or "enhance" the numbers to one's advantage. I have no doubt that several colleges fudge the numbers to raise their rankings, leading to a lot of frustration at other schools that are playing by the rules but feel that they're being cheated in the rankings.
And some of the metrics make little sense. For example, engineering schools can raise their rankings by several places just by having one or more faculty members in the National Academy of Engineering. Yet NAE membership is essentially meaningless in terms of research and teaching, and hardly more prestigious than having faculty members who are Fellows in other established engineering societies. Yet U.S. News ignores the number of Fellows in IEEE, ASCE, ASME, etc., and focuses on NAE membership. So why the emphasis on NAE? Probably because the NAE told U.S. News that they were the most important engineering society, and U.S. News never questioned it, when in fact the NAE has almost negligible impact on higher education.
What does it matter how good a particular college is? What are they basing it off? Satisfaction polls? Tuition cost? Income? Alumni? I find that it is best to compare departments to find the best college (Compare Engineering Departments, Compare Math Departments... etc...) Who cares if a University makes #8 at US News and Reports, if it is because of its fine arts programs and you want a computer science major?
If our elected representatives no longer represent us, do we still live in a Democracy?
is that in order to sell more magazines every year(and not have people read the rankings from years previous) they radically and arbitrarily change the criteria. Ever actually try to track the rankings? A school will gain 10, lose 5, gain 2, lose 7 over the course of 4 years. Now obviously schools do change, but lets face facts, Universities, esp. ones that have been around for a few centuries, are not highly dynamic beasts. How different really is the Harvard of this year compared to last? Or even compared to Harvard of 1997?
Maybe if US News actually kept relatively static criteria it would be interesting, but that doesn't sell very many magazines, does it?
Monstar L
In recent news, a pair of mathematicians from Penn State, have re-weighted US News analysis of top universities so people will think they went to a better school.
Up next, a quartet of Yale theologians have reinterpreted the bible and discovered that "the promise land" is actually New Haven, Connecticut.
I think this problem could have been easily solved by what Information Retrieval community has been practicing for decades now: Vector Space Model. In fact just going by the description of the method provided by the news article, it seems that the their method is not much different than the VSM model and simple cosine similarity could have been applied between the priority vector ("query vector") and each university's score("docuement vectors") along the 7 dimensions. Then all universities could be ranked in the descending order of the cosine, 1 being the perfect match. Am I missing something or this is a reinvention of the wheel?
Selecting universities: personal preference and rankings
Having been to Stanford as a visiting scholar recently, I have to say I am very glad I never went there (tuition and housing orders of magnitude out of my reach or not.)
Things I take for granted on a university campus, such as being able to walk into a library, or to use public wi-fi while sipping coffee somewhere on the grounds... These things are actually difficult or impossible for a visitor to do -- even a visitor with credentials who is there on academic business! I was *amazed* at the difficulty of getting into the Green Library for instance, and my week was pretty much destroyed by the fact that if you want to use on-campus wi-fi, "you can't", simple as that.
At every turn, everything that could have been convenient for a visitor was hostile. I ended up rushing through my research and spending all my time at a coffee shop in Palo Alto (where the wi-fi was free, and nobody minds if you hang out and work.)
Thanks Stanford, you're awesome.
-fb Everything not expressly forbidden is now mandatory.
I thought the bit where the best result would be the point nearest the line from the origin to the polytope (containing the bunch of points representing universities' parameters) passing through the point representing weights was the important bit not that different weights give different results. While what you said _is_ blindingly obvious, the other thing isn't. It might be more useful to use this method than to simply recalculate all scores using the new weights.
...which means they'll run afoul of Kenneth Arrow.
Huggins and Pachter are now applying their methods to voting in elections with more than two candidates.
Elections have more than two candidates?
damn I gotta get out of the US for a while
Prediction: The real iPhone killer is going to be sex robots from Japan. Think about it.
Where beer is Job 1
I guess it's too much to ask for the article to give a link to the actual paper... http://arxiv.org/abs/0805.1026
For graduate schools one can get customized rankings at this website.
now everybody wins!
gives new meaning to "oh yea well I did attend the Harvard of the --insert region here--" :D
I've actually played with ranking data quite extensively, and usually for reasonable weighthings of the parameters the movement in position is in the order of plus/minus 5 places. Sure, Penn State would be number one if all one cared about is retention rates, but really nobody does. Instead we can define a range of reasonable weights for retention rates (say between 7% and 35% of the total weight) and test all possible combinations in that space, suddenly Penn State place goes up and down a fairly small amount.
A bigger concern is what is the value of selecting a school based on the ranking as a whole, without paying attention to the your likely area of major. Say, Yale is a great school but in CS is a non-entity. If you are positive CS is your thing, MIT, Stanford, Harvard and Princeton are far better choices.
In technical areas (e.g. engineering), reputation within the field matters a lot more than generic reputation. People at Boeing know what the good aerospace engineering places are, and hire accordingly. If you graduated from an Ivy with an unknown engineering program, you're more likely to get responses like, "huh, I didn't even know Yale had an engineering program". Meanwhile, if you graduated from a generally lesser-known school with a top-rated engineering program (e.g. Rose-Hulman or Harvey Mudd) you're going to get plenty of offers.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
That's actually a pretty sad indication of how much a college name matters over what you do. I had a friend who did his undergrad at MIT, and when applying for jobs, he was insta-accepted to various tech jobs. No interview. No background checks. Just an open door. Given that, he refused those jobs because that easy entry gave him some indication as to who the companies hired and on what criteria. On the other extreme, were a couple of people who had to work twice as hard because they had to sell the college they attended. It's a little sad, but it's the reality.
I'm a law student. I also attend one of the most maligned law schools in the country. Not entirely by choice.
Oh sure, I wanted to go to the University of Michigan. I wanted to go to Georgetown. I applied to a number of elite law schools, and was surprisingly accepted by most of those that I applied to. The problem was money. Law school, as you can imagine, is pretty expensive. It's typically a 3-year program that runs anywhere from $25-50k/year for tuition alone. Build in the cost of books, rent, food, etc. and you're looking at another $15-20k/year. Federal student loans aren't that generous, and the terms on the private loans make them rather detestable. While my grades were good enough to get me into those high-end schools, hey weren't good enough to make be stand out enough to get much in the way of scholarships. And since I was paying for school by myself, I had to take a look at my safety schools. So I started researching the various ranking systems and what criteria they used.
One of the major ranking indexes I looked at, for example, heavily weighted entrance requirements as well as the attrition rate. The result was that the schools who only accepted people with the best GPAs and LSAT scores ranked high. That was expected. But the attrition rate? By its rankings, if two schools accepted students with the exact same criteria, the one with fewer failures/drop-outs after the first year ranked higher. That struck me as being really odd. A more rigorous program is desirable, and will likely result in more failures. Meanwhile, the school I go to will take in very average students the first year, and has a huge failure rate; anywhere from 20-50%, depending on who you ask. The first year professors are brutal, and the whole year is designed not only to teach you, but to weed out the people who don't really want or deserve to be there. Consequently, they get hammered in almost every ranking except for "most competitive students," where it's in the top 10 in the country.
Then I started noticing some other oddball problems. That same ranking service said that the average undergraduate GPA and LSAT score were below the school's minimum requirements. At several schools, I noticed that, if they offered part-time programs, it looked like an incredibly low portion of the students were enrolled full time. Then I realized how they were figuring that out: it wasn't by graduation, it was by sampling year-to-year enrollment.
Example: Say a normal student graduates in 3 years. A part-time student graduates in 6. Over 6 years, the school graduates 60 full-time students (let's say they're spread out evenly at 6 per year) and 10 part-time students. The thing is, because of their sampling, those part-time students wind up being counted for twice as long. So at any given time in that 6-year period, you have 18 full-time students, and 10 part-time students. The sample is going to show that more than 1/3 of the student body is part-time, even though the school is graduating six times as many full-time students. It's rather misleading.
I noticed a number of other glaring issues, too. For example, prestigious schools have loads of information published, while the less prestigious schools usually have little more than a few out-of-date statistics. Self-reinforcing, no?
In the end, it felt like the ranking systems were a complete waste. They rank everything but the quality of the education. And while I don't mean to play to the cliche, because I know it's not universally true, but I actually flew around the country and visited a couple of those "elite" schools that I was accepted into. They don't let you forget how "elite" they are. At all. The snobbery was utterly overwhelming. One of them told me that their students were "the Maseratis of law school." /gag
I wound up going to the school that offered me the biggest scholarship.
At unnamed university where I've spent some time, we basically keep around a wireless router to power on during visits so our visitors can get online from our lab. (It gets powered down the rest of the time to avoid getting spotted by the network police.)
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Yeah, but could a caveman do it?
The part where MIT manages to squeeze into 3rd seems to come up only if you completely de-emphasize freshmen retention. (Unless I'm reading the "plot using higher-dimension mathematics" wrong... which is possible since I wasn't good enough for MIT ;)) Or in other words, it comes in #3 only if you're so supremely confident in your abilities that you're certain you won't be one of THOSE GUYS who flunks out in the first year. And that sort of over-confidence sounds just like MIT students, doesn't it? :)
Those who fail to understand communication protocols, are doomed to repeat them over port 80.
Facts are meaningless. You could use facts to prove anything that's even remotely true!
-- Homer Simpson
It's like me leaving of the moronic in the title ... freetard doesn't quite cover it. Moronic freetard is right on the money.
Comment removed based on user account deletion
If you understand Spanish, just take a look at any classifieds. They will outright print in the ad that they don't want certain types, like gender and age. There might be anti-discrimination laws, but if so, they certainly are not top priority. If you think discrimination is bad in the USA, then you really don't know what that word means. Try going to Mexico, where outright discrimination is the rule, not the exception.
From the posts I've been reading, it seems like everyone agrees that a college or university matters in name only; it seems like some people actually think you'll receive the same undergraduate education at the University of Vermont as you would at Harvard.
It's simply not true. If you're going to college for a *real* reason--not for your career or because your parents told you to--your main pursuits are intellectual ones. A college like Harvard or Princeton offers an intellectual experience far greater than those offered by lesser schools.
These aren't called the best schools in the country just because they have fancy names. They're the best schools in the country because they have the best faculty and students. People who say otherwise are just angry that they weren't smart enough to matriculate.
The take-home message might have been obvious, but what the "pair of mathematicians" did was study exactly how the rankings depend on the weights. For example they figured out all the possible rankings for each university, and computed the regions of weights which gave each ranking. You can't just read off that kind of info from the university measurements. If a university is ranked #10 in all the different categories, then you might be able to make that university #1, or maybe not
This math should then also work to analyze candidates and propositions or measures on the ballot according to which issues and positions an individual feels is a priority.
When I went to college I looked at the best schools according to publications like US News. In the end I selected a school in Michigan that put me back about 40 grand a year. I hated the place. Sure the academics were good (though hardly amazing) but the attitude of the students and administrators was simply put, piss poor. It was a very unhappy place in a town with very little to do. Often these organizations that classify schools seem to forget the importance of happiness. Yes I know it is a factor they measure, but to me it the most important thing, if students are miserable as myself and many of my classmates were, the quality of the education is hardly relevant.
FWIW, that use of methodology in the summary should be method, as in "Penn State, which is #48 using US News's method..."
I thought it was OK to be an anal language grinch on Slashdot!
Languages aren't inherently fast -- implementations are efficient
Did they forget to mention they were from Penn State? Biased bastards. Disclaimer: This post funded in part by the University of Nebraska
Many schools that are not too famous have very good departments in particular fields. The college guides focus almost exclusively on the undergraduate experience, but from my grad student point of view, undergrads are just so much background noise.
...a college degree should mean something, and some colleges invest considerably in the branding of their name by having a rigorous admissions standard and rigorous grading standard.
If 10 of the past 10 people hired from college X have turned out to be great employees, and 5 out of 10 from college Y have turned out to be great employees, then there's nothing irrational about having a less-rigorous interview process for people hired from the college where your experience has been 10 out of 10.
Note that PERSONALLY, I wouldn't assume that someone who went to Yale or Harvard was qualified (too many legacies), but *IF* I wanted to pay them the additional salary they command, I wouldn't have any trouble hiring someone from MIT and assuming they're qualified. The risk that they turn out to be unqualified is so small that it's not worth the expense of additional filtering.
paintball
A standard optimization problem got them on slashdot? I should have tried to get the analysis of my warcraft character's stats published.
I assume these rankings are for undergraduate programs? The grad program landscape can be quite different.