Slashdot Mirror

There is no "US program"

That's not entirely true.

The ACM, in conjunction with the IEEE publishes a set of common curriculum standards for post-secondary Computer Science programs. Many (most?) colleges and Universities in North America that have Computer Science programs will, at least in part, base their curriculum off the ACM/IEEE recommendations.

Being a US-centric test, it's possible that the test presumes the use of the ACM/IEEE curriculum. From what I can tell, the Major Field Test for Computer Science is designed with input from professors from several US universities, and presumably they design it around the curricula they teach and feel is important for a student to know -- and chances are very good that curricula is the ACM/IEEE curricula. If Chinese, Indian, and Russian Universities are using different curricula standards, it would absolutely introduce some form of bias.

Yaz

The summary does say the exam was translated. There's a set of example questions for the exam available online from the group that creates the test: https://www.ets.org/Media/Tests/MFT/pdf/mft_samp_questions_compsci.pdf

In looking at some of the questions, it doesn't appear as though there should be any cultural barrier, unless they don't have stacks in Russia or recursion in China or some such nonsense. Maybe you could argue that the test is designed with the assumption that the ACM or IEEE curriculum guides are followed, but those aren't just some standard for or set by the U.S., though they may be more likely to follow it.

I won't assume that the U.S. has better students, or we'd be kicking ass in other fields as well, unless all of the really smart kids are going into CS, whereas this isn't the case in other countries. There's probably a very good explanation for the results, but I don't believe that it's a result of the test design. The U.S. is generally recognized as having some of the world's best universities. It's entirely possible that this is especially true of computer science. Maybe the standards are higher and the weaker students are culled before their fourth year. It could be a lot of things, but I doubt culture or language barrier is one of them.

I wouldn't put much stock into your common on hacking either. The reasons for that have more to do with economics and law enforcement than anything else. The authorities in Russia won't give a fuck if someone there is scamming wealthy Americans or Europeans. The same isn't true for people within those countries (and they're much easier to arrest if operating within their own country) and there's less financial incentive to try hacking people in a country a lot less well off than your own.

I tried to digest the PIACC data. What a hot mess. Here's one page that initially seemed promising:

Overall Results — Millennials

This covers 2012. I had to look elsewhere for descriptive text around the performance levels (Description of PIAAC literacy discrete achievement levels).

Table 1 excludes a row for the United States, but includes asterisks for whether each value for "Significantly different (p < .05) from United States". (If there was a Level 6 it would be this: ability to glean primary information from texts where conflicting information fills all the tables.)

Not once is the question of language addressed. Here is what I presume: "Subjects are tested in their official language of choice." How hard was that to blurt out? And what about subjects whose preschool home environment was "none of the above"?

This is all gussied up to mobilise action as a future employment catastrophe, yet it's not at all clear that future America has any desire to employ the bottom 20% of its millennial generation. Their contribution to America's employment landscape might be effectively zilch.

By now I'm firmly of the opinion that these numbers have been sifted, folded, and mutilated to sell something. What might that be? Then I look up and see this URL has the domain name ets.org. Bingo!

Still, it's hard to believe how much of the millennial generation in Canada falls below level 3. 40% in the 2012 survey. (I trust this number more because we don't have the Spanish problem—not yet an official language in America, so probably not tested).

Then I look at the requirements for level 3 and see that this is well above merely "getting the news". Level 2 is more than sufficient for reading the first two paragraphs of what generally qualifies as journalism these days. Level 3 requires elementary synthesis from two or more sources at the same time. Ah yes, I can see how the mayfly generation might struggle with this.

You can tell from any election cycle that half the population can't successfully meld two quantitative ideas (e.g. spend more here, spend less there). I recall a recent John Oliver-esque clip where Americans on the street professed to want "smaller government" but then you go through a list of entitlement programs and one by one the answer is "don't you dare cut that one!" Many of these people appear adequately employed. Small government. Mmmmm, donut! Program cuts. Ugh, no donut!

Sorry, rationality, Americans are just not that into you.

Here's the requirements for level 5:

At this level, tasks may require the respondent to search for and integrate information across multiple, dense texts; construct syntheses of similar and contrasting ideas or points of view; or evaluate evidence based arguments. Application and evaluation of logical and conceptual models of ideas may be required to accomplish tasks. Evaluating reliability of evidentiary sources and selecting key information is frequently a requirement. Tasks often require respondents to be aware of subtle, rhetorical cues and to make high-level inferences or use specialized background knowledge.

Did spotting ets.org in the source text URL immediately set off a blinking radiation hazard light. Full marks.

Still no mention of the language question. Time to pull out the big guns. Use the FAQ Luke. (After all, why on earth would language status be up-front information in a survey on reading proficiency?)

Frequently Asked Questions

6. Why does PIAAC assess literacy only in English?

PIAAC assesses adults in the official language or languages of each participating country. Based on a 1988 congressional mandate and the 1991 Nati

What if you broke out United States statistics by race? I wonder what you'd find.

The Fine ETS Study discusses that starting at page 37.

things in Japan tend to happen in Japanese despite the speaker's English ability

This is an interesting use of "despite" - Japanese, as a rule, speak abysmal English. See e.g. page 6 of 2012 TOEFL scores

All your GRE essays are evaluated by a machine and have been for years -- the e-rater. http://www.ets.org/research/topics/as_nlp/writing_quality/
The rating is also done by humans. It works well in practice and ensures that essays are graded fairly. If there is a significant discrepancy between the two ratings for a essay, that essay is examined further by another specialist. It prevents students from being victims of someone having a bad day at the office, and also does not encourage writing an essay to beat a machine.
The significance of the EDX news is not the concept of automated grading, it is that that such software is now free and opensource.

When teachers didn't want to be tested as they claimed that testing was a poor indicator of someone ability. Go Figure.

Most teachers don't complain about being tested on their subject matter -- maybe a few, but not most. Testing on subject matter is standard practice for getting a secondary certificate these days... not just in the context of the dual education/subject degree you generally earn while you're working towards certification, but there's actually tests at the end to certify. Heck, in some states, you have continuing education requirements for a long while afterwards. This is all par for the course.

What teachers do complain about is having how students fare on standardized tests serve as a metric for their performance. Everyone knows standardized tests are somewhat problematic metric of even student ability, but most people are willing to accept it as a starting point while trying to work with varying cases. So, just like you sometimes see higher grades than test scores would indicate awarded to students who diligently complete their homework, take extra credit assignments, consistently participate in classroom discussion, and in general work hard, you'll also see colleges accept students with lower standardized test scores who show a similar pattern in their schoolwork and extracurricular activities. (And you see people succeed in life that way, too -- my girlfriend says her rocket scientist father actually struggled quite a bit with math, but he's know since he was a kid he wanted to be freakin' rocket scientist, and he worked hard and he's a highly respected guy at Aerojet who's worked on stuff from the NASA New Horizons project to fielding calls from the Mythbusters team).

But when you take something that problematic and then use it as an indirect metric for something else, the problems are magnified. There are too many confounding factors. What the student population brings to the table is quite simply as important as what the teacher brings, and what the larger system does to support or work against teachers is a big factor as well.

You might be able to use tests that measure only aggregated student improvements as a minor part of an overall program including human assessments from other professionals, continuing education/training, feedback from students and parents, and organizational reviews for schools and districts. But any teacher who complains about a merit program that focuses on standardized testing is only acting on good sense.

Why is western society obsessed with mathematics, deluded into thinking it's useful in general, and why are people so stressed over learning this useless and dryly-presented subject?

Math is useful in general. And western society doesn't just stress about learning math. An even greater number are probably stressed about passing english tests. Society thinks language and math are important to education; your basket-weaving and sculpture not so much. I personally don't see the problem with this.

No, whether the Earth is flat or round is a matter of fact, not opinion. You need to understand the difference between fact and opinion. Take a look at the GRE issue and argument tasks, where you are allowed to present your perspective on an issue, and you argue how well reasoned an argument is. Those arguments also have nothing to do with "right" or "wrong", but how well you are able present a well-reasoned argument. Those rhetorical arguments are completely different from scientific evidence supporting a hypothesis or a mathematical proof.

No, whether the Earth is flat or round is a matter of fact, not opinion. You need to understand the difference between fact and opinion. Take a look at the GRE issue and argument tasks, where you are allowed to present your perspective on an issue, and you argue how well reasoned an argument is. Those arguments also have nothing to do with "right" or "wrong", but how well you are able present a well-reasoned argument. Those rhetorical arguments are completely different from scientific evidence supporting a hypothesis or a mathematical proof.

Better teachers and better education is a problem that has lots of factors of which I'll only address the ones that I care about. This mostly pertains to elementary education majors.

1. The average education major is less academically capable than your average college student.

    I'm sort of bending the findings of a study from about 15 years ago:

The Academic Quality of Prospective Teachers: The Impact of Admissions and Licensure Testing (warning this is a link to a pdf).

    There are exceptions, those going into to teaching science or math have just as good a scores as math or science major. If you start off with poor talent it won't get much better no matter how good the training.

2. We pay teachers not nearly enough money.

    If we really value education we need to pay them more. We need to be willing to pay the taxes to support the important job they do. Every good engineer, scientist or mathematician probably had a good teacher some time in their life. Too bad there aren't more.

3. We need better metrics to define what a good teacher is.

    Don't get me started with the fiasco that is No Child Left Behind. Poor testing, poor accountability and poor funding.

    How about to test a teacher's effectiveness we compare apples to apples. Let a teacher stay with a group of students for 2 to 3 years. They we can better tell if it's the student or teacher. If that doesn't work how about comparing the student's progress instead of the group's progress (which my wife thinks is a suggested change for NCLB), you will also need to control for similar groups (smart kids vs. smart kids).

4. Get rid of bad people earlier in the cycle (mostly at the college level).

      I think this applies to all majors. Weed-out courses earlier. My major back in school (aero engineering) had to take an electrical engineering weed out course our sophomore year (don't ask me why). It will make you think twice if you want to pursue a major.

      I think for teachers they need to take a public speaking course early on. If you can't talk in front of a class of 20-30 peers you certainly can't do that in front of a bunch of unruly kids. I get this idea mostly from my wife's experience as an instructor in a school of education (teaching teachers how to teach basically). Most of the kids have a horrible time teaching a lesson and this is as juniors/seniors.

      Hell, even better give them a taste of teaching no later than their sophomore year. Most don't get that until their junior year. By then it's too late for them to do anything but finish their degree. This means they either will go into the system as a lousy teacher or flail around with a degree they don't like or can't use.

Extra bonus crazy idea.

      Treat teachers like doctors/trade crafts. Extra training and lots of practical experience before we unleash them by themselves. Basically after they get their initial degree/license they will need to work with another teacher (like a residency/apprenticeship) before they get to pass another examination and get to teach on their own. The downside this would be rather pricey. Depends if you think education is important or not.

Extra bonus rant:

      I think students, college students at least since I work on a university, are less capable than 15-20 years ago. The top 10% are amazing probably better than the top 10% of 15-20 years ago. The bottom 10% are the bottom 10% and it doesn't matter too much if they are better or worse. The middle 80% just seem less able to do the work and understand the content of most college level degrees. I've asked many people about this observation (from professors that have been doing this for decades to students themselves) and their answer has generally been yes. I do submit the caveat that the plural of anecdote is not data. So take all of this with a block of salt.

The GRE Writing portion is already using it.

From http://www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=ebd42d3631df4010VgnVCM10000022f95190RCRD&vgnextchannel=54c846f1674f4010VgnVCM10000022f95190RCRD

"For the computer-based Analytical Writing section, each essay receives a score from at least one trained reader, using a six-point holistic scale. In holistic scoring, readers are trained to assign scores on the basis of the overall quality of an essay in response to the assigned task. The essay score is then reviewed by e-rater, a computerized program developed by ETS, which is being used to monitor the human reader. If the e-rater evaluation and the human score agree, the human score is used as the final score. If they disagree by a certain amount, a second human score is obtained, and the final score is the average of the two human scores."

If you find a way on what the algorithm look for, even a software-generated essay can get 6's.

What else did you expect from EST? That's the same people who give GRE tests. Beyond the general test (which itself is not without problems), they seem to be clueless about what exactly they are evaluating. Take the math test: 66 questions over 170 minutes, 2.7 minutes per question.

Scores on the tests are intended to indicate knowledge of the subject matter emphasized in many undergraduate programs as preparation for graduate study.

I am not sure that "indicate" means what they think it does. Anyway, this should read: "to assess the ability to make quick guesses about an assorted collection of exercises and to retain in one's memory the entire arsenal of shortcuts one was taught during the sophomore year". That would hit closer to the mark. Of course, more able mathematicians will tend to do better on average, but what about all of the freaks who spent the last year or two concentrating on advanced topics, while looking forward to the graduate study? Topics like logic (not covered) or topology (barely touched). Some of them could probably teach calculus, just not off the top of their freaking heads.

The problems with "testing" people is that the people who write the tests have their own biases and opinions about what is "better" or "bad". And since they write the tests, their opinions are naturally considered to be more "correct" than the people they're testing.

Not unlike the GRE Subject examination for Computer Science in the U.S.?
And by the way, it looks as though the cheating has been going on for a while now...

I'm all for teaching using practical applications that require integrating a grreat deal of different areas of knowledge, but I think that there is a bit of bashing of CS students on the basis of matteres that are really more related to engineering than the what CS is really about.

Look at the problems of about average difficulty on the CS GRE practice subject test (the percentages of test-takers who answered each question correct ly are listed on page 48). I think you'll find many of them quite challenging, and the real test has gotten even harder.

Here, with the logical symbols translated into standard ASCII is the easiest question on the test, 96% got it right:

6. Suppose that P(x, y) means "x is a parent of y" and M(x) means "x is male". If F(v,w) equals
M(v) & Exists x Exists y(P(x,y) & P(x,v) & (y != v) & P(y,w)),
what is the meaning of the expression F(v,w)?
(A) v is a brother of w.
(B) v is a nephew of w.
(C) v is an uncle of w.
(D) v is a grandfather of w.
(E) v is a male cousin of w.

The GRE® General Test measures critical thinking, analytical writing, verbal reasoning, and quantitative reasoning skills that have been acquired over a long period of time and that are not related to any specific field of study. The GRE® Subject Tests gauge undergraduate achievement in eight specific fields of study.

"I'm a conservative and don't think that there should be any dissent on racial equality of abilities."

The evidence is against you here. Clearly Asians have a distinct disadvantage when it comes to the physical requirements of professional basketball, and the appearance of a Yao Ming does not refute the difference in the means of the height distributions and the resulting extreme disparity in the number of Asians as compared with Blacks who are three or more standard deviations out from the overall population mean height.

Just as an example, say one minority population (e.g. Blacks) has a wider spread of heights and a greater mean heigh than the overall population, so that someone two SD out on their disribution is at the third SD for the general population , and another minority population (e.g. Asians) has a tighter distribution and a slightly lower mean height than the overall population, so that a member of this population at the fourth SD in height is at the third SD of height for the overall population. Then the relative proportion of people in the first group that have the height requred to play pro basketball will be, IIRC, around 1 in 50 compared to 1 in 30,000 in the second group. So the expected proportion of the first to the second group among pro basketball players will be 600 to 1, assuming that the two minorities are of equal numbers in the overall population. There is no discrimination going on, yet modest differences in groups lead to extreme disproportions out in the tails.

This happens with intellectual differences between groups, too. There is no doubt given the massive amount of research that groups differ substantially, measurably and repeatably in their ability to do any cognitively loaded test and the amount of the disparity between groups depends directly and positively on the degree of cognitive loading of the test. Both race and sex are predictors of performance on cognitive tests: men do better on mathematical and most spatial tasks, women on verbal and certain types of visual tasks. (This difference seems to show up at puberty and to be linked to testosterone.) The difference is fairly mild, but men also have a wider distrbution of abilities, so that there are far more men at the top and bottom of the distribution than women. Asians do better than Whites at mathematical/spatial reasoning by about half a standard deviation, Northern- and Eastern- European-descended Jews do better than other European-descended people on all major cognitive measures by about 1 - 2 SDs, American Latinos do worse than Whites on all major cognitive factors by about half of an SD, and American Blacks trail Whites by about a SD. Herenstein and Murray's book The Bell Curve is the most convenient place to get references (hundreds of 'em!) to studies demonstrating these approximate figures. Many have attempted to refute the arguments of The Bell Curve, but the facts of group differences have never really been in dispute, only the causes. Fine sentiments such as reverence for the principles of legal equality and the innate worth of people will not by themselves change the facts.

At the tails of the distributions there is very little overlap between the abilities of certain groups. Look at the asians just coming out of college - their mean score was 674 on the GRE quantitative vs. 483 for the same-age blacks - the 25th percentile cutoff for the asians is 610, while the 75th percentile score for the blacks in that age group is 580. If you were to compare the quantitative scores of young male asian with older black females you would find virtually no overlap in mathematical abilities at all. [2002-3 p.48, 51. http://www.ets.org/Media/Tests/GRE/pdf/gre_factors _02-03.pdf%5D

No viewpoint supported by the overwhelming preponderance of the factual evidence should be suppressed in favor of a fond wish that is refuted by the evidence.

Are you discussing latent semantic analysis by any chance? ;^)

It performs well in certain areas (for example, completing certain MCQ's to the same level as humans), automatic essay marking (but read the Powers et al study for more), and other things. It's surprising how well it does despite there being a complete absence of grounding (grounding in artificial intelligence terms).

My prep (aside from 3.5 years of CS classes) consisted solely of getting the practice test [PDF] (it wasn't available online at the time {smile}) and making sure I understood the questions it asked. In particular I recall the regular expressions stuff puzzling me (I got the concept, but didn't grok their syntax), so I compared the questions with the answers and reverse-engineered it. If you can figure out the practice test, you'll probably do OK on the real one. And if you can't... maybe the exam should identify that? Sorry, but I'm just not a big fan of teaching-for-the-test education.

I only scored in the 92nd percentile (compared to 98-99th for the general test sections), so maybe more prep would've been helpful. But I prefer to blame the fact that I had traveled from Aberdeen to the wrong university in the wrong part of Glasgow (clueless American exchange student that I was) and didn't discover this mistake until late the night before the exam, after I'd gotten a room nearby and was trying to figure out where I had to go in the morning. So I had to get up early, catch the very first bus of the day, and finally sprint a couple blocks to get there before they closed the doors.

OK, in that case I must add to my original list:

• require high-school teachers to have at least an undergraduate degree in the topic they want to teach (e.g. a math teacher would be required to have a BA/BS in math, a physics teacher an equivalent in physics, etc) - like they do in most other countries.

  And after that:

• don't homeschool

With firefox I receive: Welcome to TOEFL: The Test of English as a Foreign Language No Javascripts or Redirects.

Google Mobile already does this to an extent, though I don't know about the compression part. It seems to take ordinary sites and condense them down to just the text delivered in XHTML. Check out this page (the first result for "test") then check out the full version. I actually kind of liked the stripped down version better, it communicates what it has to communicate and doesn't get in the way.

Actually, ETS claims to be a nonprofit organization: http://ets.org/aboutets/

Yes, this is true.

Here is an academic paper describing e-rater:
http://acl.ldc.upenn.edu/P/P98/P98-1032.pdf

Here is a site where you can try it for yourself:
http://www.ets.org/scoreitnow/

According to ETS, the e-rater agrees with the human grader 98% of the time.

Keep in mind that those are the people that brought you the AP high school tests, GRE, SAT, CLEP etcetera http://www.ets.org/tests.html

Since when did the law of averages have anything to do with programming ability? I would say that Indians generally are not only better programmers, on average, but better theoretical computer scientists, too.

Go look up some of the premier computer science departments in the country (or even around the world) and take a hard look at the number of Indians (and Chinese) PhD students. Then go look at the average quantitative GRE scores of Asians and compare them to other races. Seeing anything interesting?

The reality is that the education system in India is generally more rigorous, especially when it comes to math. I doubt that Microsoft's primary motive for outsourcing is money -- they're not hurting for cash -- but instead is simply an effort to try and find a large number of great computer scientists. Unless we start increasing the effectiveness of our own elementary and secondary school systems, we're going to be slowly left behind.

Try e-rater

To see how e-rater works, select a sample essay on a demo topic and submit it to e-rater for scoring. Several seconds later, you will see the e-rater score assigned to that essay, along with descriptive text associated with that score point.

If you would like to try out e-rater, you can obtain an ID and password and submit an original essay for scoring on the CriterionSM Web site.
Using e-rater

The following essays are samples that reflect various scores on a six-point scale (6 is the highest score; 1 is the lowest). If you wish, you may submit these essays to e-rater for scoring on the Criterion website to see whether e-rater 's score reflects your own judgment. These essays are reproduced exactly as submitted by the writers. They are randomly presented and are not in score point order.

Essay Topic: Often in life we experience a conflict in choosing between something we want to do and something we feel we should do.

In your opinion, are there any circumstances in which it is better for people to do what they want to do rather than what they feel they should do? Support your position with evidence from your own experience or your observations of other people.

Essay Sample A:

In every situation, people have a choice to make. Sometimes, one chooses what they want to do and other times they choose what they feel they should do. In most circumstances, it is better for the individual to put away their wants, and do what they feel should be done.

Going to college may not be what some want to do, but it is definatly what they should do. It is a serious choice to make. The decision they make is vital and will affect the individual's life forever. The most important thing to remember in this situation is that, what one wants now, may not be what they want later in life. By going to college they can make their decision later. The person would have an education to fall back on and have more of an ability to get a good job. If one chooses not to continue their education because of what they want, they may be faced with difficulties down the road.

A choice many teenagers are faced with is whether to listen to their parents. Teenagers are filled with things they want. The problem is that those wants and desires change daily. A smart individual will set priorities for themselves and do what they feel they should. many parents try to guide their children in the right direction. Although most know that they should listen to their parents advice, they don't want to. In the long-run it would benefit them to set their desires aside and do what they feel they should.

It is a tough decision to make; what you want to do and what you should do. The decision is left up to the individual but in most circumstances it is better to do what should be done.

Essay Sample B:

Many people are different because they are raised however their parents want to, but in my occation my parents raised me to respect other's especially your elders. This experience I had had to do with and old lady, she wanted to cross the street but she could not get down to the street because the concrete was to thick and she could not step down.

She had some trouble, especially since she was carring her big bag that old people carry, I looked at some people that past right by her and didn't even pay attention to her and since I was there I decided to help her even though I didn't want to, but I felt that I should because that was the way I was raised.

So I went up to her and asked her if she needed some help, she said yes, took her hand to help her down and walked her across the street after all doing this felt good even though I didn't want but I did. In that occasion, I know that many people should of done it but they didn't want to, why because probably they didn't care, but in my opinion peo

I'd love to see somebody with access to the trial submit that and see what it gets. But if I were a computer, you'd lose points for lack of an attention-getter and transitions. Add "Did you ever wonder of what a good essay always consists?" to the first paragraph, and "First," "In addition," and "Finally," to the second, third, and fourth paragraphs respectively.

If it doesn't already, I would expect a service like this will eventually include plagiarism detection, due to marketing pressure if nothing else. This is something that human graders do, at least over the space of papers they grade and works they remember.

But if plagiarism detection is added, then the grading service would have to make and retain some encoding of each graded paper, a derivative work, in its database.

Once that happens, the grading service also becomes subject to all of the issues already raised with services like TurnItIn.com, already discussed here.

I also found this comment from ETS's site rather strange, to say the least:

It is important to remember that e-rater is an embedded real-time application; it is not software.

If you're interested to know what their own research shows, you may find this paper an eyeopener.

The paper is a PDF called "Stumping e-Rater" commissioned by ETS, the developers of the product.

A small correction, maybe, but the GRE Computer Science subject test is not the entire GRE.

A small correction, maybe, but the GRE Computer Science subject test is not the entire GRE.

I agree; haven't written the test so far, but if my SAT (I & II) experience is any indication, then practising with old tests is always a good choice.

Only problem is, I've found only this practice book published by ETS for the CompSci Subject Test. Any other *official* test-prep material you'd like to suggest?

Find a Chinese or Indian CS grad student who took the subject test in one of the previous years. Take a look at this article. It works exactly as described in the second item on that page. I'm surprised that it took the ETS so long to catch on to this.

ETS uses their "e-rater" system to score essays in the GMAT.

This link is a summary of some work done here at Educational Testing Service .

Basically what researchers here have decided is that the use of computers to teach mind numbing things like multiplication tables or addition tables is negatively correlated to performance on national tests. That is if you use a computer like flash cards, kids are not helped....they are done a diservice.

However, the use of computers to play educational games (e.g. Math Rabbit or Treasure Math Storm or any other math game that requires a child to think about what they are doing) is positively correlated with performance on national tests.

So its not just that they have computers in the classroom, its what they use them for.


ps: I hope ets.org can handle being /.'ed!!!