Canada's 'Random' Immigration Lottery Uses Microsoft Excel, Which Isn't Actually Random (gizmodo.com)

An anonymous reader writes: Last year, Canada introduced a new lottery system used to extend permanent-resident status to the parents and grandparents of Canadian citizens. The process was designed to randomly select applicants in order to make the process fairer than the old first-come, first-served system. There's just one problem: the software used to run the lottery isn't actually random. The Globe and Mail reported the Immigration, Refugees and Citizenship Canada (IRCC) uses Microsoft Excel to run the immigration lottery to select 10,000 people for permanent resident status from a field of about 100,000 applications received each year. Experts warned that the random number generating function in Excel isn't actually random and may put some applicants at a disadvantage.

First, it's best to understand just how the lottery system works. An Access to Information request filed by The Globe and Mail shows that IRCC inputs the application number for every person entering the lottery into Excel, then assigns them a random number to each using a variation of the program's RAND command. They then sort the list from smallest to largest based on the random number assigned and take the first 10,000 applications with the lowest numbers. The system puts a lot of faith in Excel's random function, which it might not deserve. According to Universite de Montreal computer science professor Pierre L'Ecuyer, Excel is "very bad" at generating random numbers because it relies on an old generator that is out of date. He also warned that Excel doesn't pass statistical tests and is less random than it appears, which means some people in the lottery may actually have a lower chance of being selected than others.

Comments

Irrelevant

[XXongo]

The story is about an issue that is completely irrelevant.

It doesn't matter whether the "RND" function is ideally random in a mathematical sense. It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.

Re:Irrelevant

[geschbacher79]

Exactly. And if the order in which the data is inserted into Excel is essentially random, that's good enough. There is no fair or unfair if the number of applicants exceeds the quota and there is no opportunity to game the system (such as naming yourself Aaron A Aaronson to appear first, or something)

Re:Irrelevant

[Anonymous Coward]

such as naming yourself Aaron A Aaronson to appear first, or something

[Tears up application]

Re:Irrelevant

[Obfuscant]

While that's true, if Excel's RNG results in a pattern (e.g. cell A2534 is always assigned a low number and thus selected),

If you had bothered to read the finr article, you'd have learned that the tempest in the teapot is that Excel uses a pseudorandom generator. That's just like what a lot of systems use. It costs time and money to do real random generation, and it requires some hardware. PGP, IIRC, requires someone to type at the keyboard and it times the characters to generate a random number. There are radiative decay RNGs. There's even a lava lamp based RNG.

But MOST of the "random" for most software is pseudo. And the algorithms are published.

The fine article talks about how bad some RNG is because you could "reverse engineer" the algorithm. In 1980 I tested the RNG in DEC RSX-11, and I simply looked up the algorithm in the manual to see if the results from testing matched theory. In a manual. "This is how the RAND function works..." As I recall, it was based on doing a simple calculation on a double precision number and pulling the middle 32 bits out of the number as the random output. You could actually write the code to do this yourself.

it could result in immigration employees who know of this gaming the system, to do an immigrant friend a favor or even auctioning the spot to the highest bidder.

Oh for pete's sake. If an immigration employee has this much access to determining who wins and who loses, then even if the RNG is a true, completely unpredictable, physical random process based number, all he'd have to do is run the random generation process over again until his chosen winner "won".

The bigger question to me is why are they using Excel for this? Spreadsheets are for calculating things.

Yeah, calculating things. Like random numbers. They're using Excel because it works for this and didn't cost them $1 million to pay a consultant to write something in python to do the same thing.

An immigration employee could take the spreadsheet after the random numbers were assigned, and copy-paste names or random numbers around to move people to/from the selected and denied categories,

They could do this even if the RNG is truly random, so the RNG has nothing to do with the problem.

even weaker

[goombah99]

The story is about an issue that is completely irrelevant.

It doesn't matter whether the "RND" function is ideally random in a mathematical sense. It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.

It isn't even that. Just because the distribution of random numbers isn't random it doesn't mean the sort order based on that isn't random. For example, suppose my random number generator only put out numbers divisible by 1/(2^16) which is what a finite precision binary based system is going to do. This distribution isn't random because it's zero density at many possible floating point values. Yet the sort order might be perfectly random.

Re:So...

[alvinrod]

I think you're confusing libertarians with anarchists. Libertarians believe in strong property rights, which naturally implies borders and a need for some entity to enforce them. Some would argue that doesn't necessarily need to be a government, but historically that has been a major role of governments.

Most libertarians probably wouldn't care about immigration if the U.S. had no welfare state as was historically the case throughout most of the country's history. Once you got off the boat you were on your own, but plenty of people were perfectly happy with that arrangement. This generally worked quite well with the homestead system since anyone who wanted to move west could do so and a large number of immigrants did. You had the freedom to make something of yourself and if you couldn't that was your own problem.

However, the reality in modern America is that the government is expected to care for anyone who shows up and pay for their children's education and healthcare. That necessarily means taxation, which libertarians are generally loath to agree to outside of the limited government functions that they're in favor of, so they'll tend to be less in favor of immigration with those kinds of strings attached to it. If everyone immigrating from Mexico were a doctor, engineer, or otherwise highly skilled such that they'd pay more into the system than they get out of it, I suspect there'd be no disagreement with letting in as many people who fall into that category either.

Translation:

[gman003]

Gizmodo just discovered what a PRNG is