Slashdot Mirror
Disaster Strikes Norwegian Government Web Portal
An anonymous reader writes "Altinn.no is a web service run by the Norwegian government, on which citizens can find, fill out and deliver forms electronically. Every year Norwegian citizens can also log in to check their tax results. This year, as every year, the site was unable to cope with the traffic generated from everyone wanting to check their taxes at the same time. New this year, however, was that once people were finally able to log in, a significant amount of people were logged in as someone else. Users then had access to all financial data of this unfortunate person over two years back in time, in addition to the financial information of his wife and the company he worked for. Altinn shut down some 15 minutes later, and has been down since."
by the government sending them a letter saying how much is owed.
The government does all the calculations.
Really they need a staggered ticket system to distribute the load over time. Issue each citizen a ticket that indicates a period when they can log in to check data, both a soonest and latest date (stragglers not tolerated). This is no different than physical scenarios where people are grouped by first letter of last name, etc. in a crowded office and then each group served sequentially to lighten the load.
Wanna guess how the norwegian government decided how traffic shoul be scaled? Come on, guess They made a limit of 300 000 logins, before making the main web page redirect to a page saying "sorry the lines are full pleas pick a number" - it, apparantly , seemed more logical than scaling the hardware :P
I foresee a large lawsuit settlement in his future
It's been very briefly reported that this was related to a caching error. This guy's information was apparently cached and then served to everyone.
wait, what?? I don't even get how that happens. Someone care to enlighten this rock?
Operation Guillotine is in effect.
If you had read the summary you would have seen that this wasn't the case, everyone was logged in as the same person, not as someone with a similiar SSN. (SSN isn't really correct, but there isn't really a suitable word in the English language here.)
The system they have in use also requires a personal password (According to TFA) so the scenario you are suggest couldn't happen here.
From your post it seems like you think it is normal with login-systems without passwords, please tell me that you don't program anything network-related.
This kind of thing doesn't need a server side cache system. This isn't Facebook.
now we need to go OSS in diesel cars
threaded app server + global who_is_logged_in variable = big mess
now we need to go OSS in diesel cars
* The government has spent on the order of $200 millions on this system
* Accenture is the main developer
* Every year the systems go down because it doesn't scale
* This year a queueing system was put in place to "fix" scalability
* From an outsider's view at least, it would seem like some cowboy decided to put up a Varnish-type frontend cache as a desperate measure to handle traffic with no thought given to sessions
* An independent report basically slaughtered most of the systems with criticism of flaws last year, which was kept secret until a week ago
* Also yesterday someone found several flaws which allowed any website to grab a json(?) script and steal userinfo if the browser had a valid session
"This person visited 18:17 and checked his tax return, and for some reason or another we had a caching system hooked up to this site, which didn't belong there".
There, fixed it for 'em.
now we need to go OSS in diesel cars
From the people in charge: "This person visited 18:17 and checked his tax return, and for some reason or another there was an error in the system, and this page entered the so-called cache memory of our servers, where it doesn't belong". You can try to decipher from that what you will.
In other words, either the person who wrote that didn't know what he/she was doing, or else a manager got involved in the software design decisions and forced the programmer to incorporate a blazingly stupid idea.
In either case, someone probably said something vague about "saving cycles" and everyone else nodded.
#DeleteChrome
Or maybe left hand vs. right hand?
now we need to go OSS in diesel cars
It's simple. They got slashdotted last year. So, this year they did all they could to end the problem. Likely, they used SSL for security. And for anything high-traffic, you put an SSL proxy in front of the servers. Servers, be they Linux or otherwise, take a much bigger hit with encryption than dedicated security boxes, like F5. So they had some proxy in front of the servers. I've put similar in place in New Zealand for the IRD, and I'd expect that the IRS uses F5 in front of their secure web sites. And dedicated proxy devices, like Blue Coat, also do SSL offload. So, mis-configuring a proxy used for SSL offload would easily serve a cached page, after all, that's its primary purpose, the SSL offload was an afterthought.
That's what happens when you have a problem one year and throw money at it to fix it without a full understanding of the problem and the fix. I'd bet it was outsourced. And I bet they outsource it again next year. I could do better for a lower cost, wouldn't be hard to do better than their performance the last two years.
Learn to love Alaska
Oh we do that here in the US too, for most salaried jobs. But then we *also* tax your property, your spending, your savings and then every year we also make you fill out forms that tax you more.
How, from a technical POV can this even happen? Dirty cache? Corrupted pointers?
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
I used to build HPCs. Doesn't require secured logins from the nodes, does when I incorporate remote admin for the head node, but that's to named accounts with passwords from the off. Those admin accounts are created locally from a Master account which is specifically excluded from remote access.
Operation Guillotine is in effect.
A 16 bit session id should be enough for everyone...
This is what happens when login credentials are based on the SSN, which is a serialised integer system. One wrong digit doesn't throw an error - it fuckin' logs you in as someone else!
If they didn't have a password, this might possibly do what you have suggested above. I highly doubt access was given without a password, so there's no way one wrong digit would do anything other than 'throw an error'. The problem here does not lie in using integers as user keys.
If it was a caching issue, possibly a page was cached when it shouldn't have been (including someone's account details), and the server returned that single person's page to everyone requesting /my_account or whatever, regardless of their logged in status - that's more likely, and actually quite an easy mistake to make if they turned on caching without properly checking the implications and disabling it for logged in users.
What they would want to do with caching is cache all public pages for everyone (which is fine, as they contain nothing but public information), and it sounds like they also cached a few (or one) private page, and served that instead of the individual private pages for logged in users as intended. I'm sure the details will come out in time.
That's not entirely accurate, at least not for foreigners working in Singapore like myself. We don't get taxed at all from our normal paychecks, but instead receive a consolidated tax bill around this time of year (I'm waiting for mine) based on our company's reporting of our income earned. Once you receive your bill you can elect to pay it all at once or allow the government to automatically deduct a portion from your account on a monthly basis. This system works quite well IMO, especially considering the taxation in Singapore is considerably less than that in the US. Unfortunately, as a US citizen you're required to pay taxes back to our money-hungry government as well. Luckily with some creative book keeping and the Foreign Tax Credit, that tax is slim to none. I haven't had to pay a dime to the US govt for 3 years, and am taxed only on roughly 15% of my income in Singapore.
When everybody's money is 'stored' in a government computer somewhere saying how much money you have, imagine what happens when there's a glitch putting your money in someone else's account.
Yeah, I know, bank accounts.
But, glitches happen there, too. At least you have a little cash to get to and from the bank to pursue the matter. When it's digital all the way down, what will you do?
I'm not a lawyer, but I play one on the Internet. Blog
That's not true. There's a checksum on our SSNs, and the checksum is constructed in such a way that the two most common mistakes in entering SSNs (double one digit, forget another, and transpose two digits) always results in a invalid SSN.
But yes, it's still possible to hit someone elses SSN by accident, but it takes more than one digit wrong. (it takes multiple wrong digits in such a way that the new SSN happens to pass the checksum-test, *and* match an actually used SSN)
It used to be like that, but the tax records are not available on the internet like it used to be anymore.
You can still check other peoples tax record, but not anonymously.
Ok - so the deal is this: For everyone in Norway, you can check 3 vital numbers: Amount earned, amount taxed and amount owned of every year. The number are skewed somewhat since they do not cover the full value of your house, it is after certain deductions on your salary, it is with your loans deducted from what you own, etc, but in essence it can give you a ballpark on how much money someone earns.
So, why is this? One of the major reasons is to ostracize anyone that pay little tax as compared to what they earn/own. So you would not need to ask your presidential candidate for his tax record - it is already online: http://skatt.bt.no/skattelister/9397621/Jens%20%20Stoltenberg *. You would also at once see it if your palace-owning neighbour had millions in earnings but payed nothing in taxes.
* This number is from 2009, you now have to login to a governmental site to be able to look up taxes for people. This is to stop malicious use of the numbers.
You are completely wrong. SSN like credit card number have control checksums. Up to 2 errors in the SSN could be detected with 100% accuracy, more errors could still be detected with a good probability.
From the people in charge: "This person visited 18:17 and checked his tax return, and for some reason or another there was an error in the system, and this page entered the so-called cache memory of our servers, where it doesn't belong". You can try to decipher from that what you will.
This is quite easy to interpret. They turned on caching to speed up page loads, but without disabling it for logged in users or sensitive pages, so one user logs in, visits /my_account or whatever, and the page is cached, then when the next 100,000 users visit /my_account the cached page (containing the first user's details) is served without authentication (!). Page caching works great for public pages like / which are served the same to everyone, and doesn't work so great for pages which require authentication.
It's the sort of mistake you wouldn't normally see on a site this size as it's a rookie error and ANY sort of testing of caching would catch it, but apparently that's what they did. Probably they only intended to cache public pages or something and managed to extend it to private pages by mistake. Their server could be properly configured and secure but then this mistake triggered by one small change to their caching config by someone who didn't know the implications.
I believe Norway has similar identification numbers as Sweden, i.e. birthdate, a few other digits and a control digit, if you throw some of the other digits off, it likely won't be a valid number. Besides, these numbers are not secret and you usually need some other form of authentication than just the number, electronic identification, number printed on tax form, etc.
Improper caching could have happened if the URLs were not unique. But caching in this case is just so wrong. And rarely is it even right. Static data can simply be preloaded in a server as streamlined as a cache would be, and those get delivered at cache speeds. Dynamic data should not be cached except in the browser, and even that with a short expire (5 minutes max).
now we need to go OSS in diesel cars
I'd be willing to bet that it was something turned on, because they needed to lighten the load on the servers. IT could have been a front end caching machine, or on the web server itself in code. In either case, it clearly wasn't tested as well as it should have been.
You *can* cache authenticated pages. Really, the /my_account (your example) only needs to be generated once a year. If that happens to be the main page to view from, you'll keep ending up back on it, to go to other pages. Generating it once is a whole lot more efficient than generating it 15 to 30 times. You'd have to get a bit creative with how you ensure no user can look at another users results. For example, if you happened to save the page as /cache/my_account_[userid].tmp, that's all fine and dandy, unless the code forgets to actually populate [userid]. :)
So may ways to screw this up, and they all should have been caught in testing.
Serious? Seriousness is well above my pay grade.
LoL and to imagine some countries (like Greece for example) are actually collecting your next years
tax as a sort of down payment. Yep, when paying taxes in 2012 the Greeks are asking taxpayers to pay
upfront for what they are going to earn untill the end of the year.
No wonder that country is head first into debt.
-- no sig today
I normally wouldn't care about this, but since the Norwegian government (i.e. the people, myself included) paid 1 billion NOK for this solution, I expect it to WORK. Mind you, this is not the first time we've had problems with Altinn, this has been a recurring drama the past few years. As the article states; every year they claim to be prepared, and every year they are unable to deliver.
We're not *that* many people in Norway (recently hit the 5 million mark), and certainly not that many adults checking their tax returns online. Guesstimate: 1 million? And how many checks it simultaneously? Let's be generous and say half.
So how the hell can a 175 million USD project not be able to deal with 500k visitors? It's a fucking joke.
www.6502asm.com - Code 6502 assembly or.. DIE!!
All Norwegian tax returns are published publicly on the Internet, so Kenneth's information was already available to anyone who cared to check it. There's been no privacy violation here that I can see.
Mod parent Informative. They are actually using F5's Big Ip solution, from my snooping before it went down. And it was outsourced, to Accenture, who has such a good track record producing stable, efficient, Microsoft-based solutions.
What is even more funny, just last week, a report leaked in the Norwegian press about this very system being hastily implemented, poorly tested and perhaps insecure.
for i in `facebook friends "=bday" 2>/dev/null | cut -d " " -f 3-`; do facebook wallpost $i "Happy birthday!"; done
It is done similarly in über-effective, ultra-efficient Singapore:
1) Let's say I'm employed by company C. Company C will send to taxman my identity card number and the amount they have paid me for the tax year.
2) Taxman will do the calculation of tax. Taxman will also consider the recurring tax claims/rebates I am likely to have (spouse/parents-related rebates, for example).
4) Taxman sends me a reminder to confirm their calculations on their website.
5) I will adjust the calculations if needed and submit the final figure.
6) Taxman sends me the final amount of tax I need to pay with payment options including a 12-month instalment plan deducted from my bank account.
7) If I'm audited, I will have to provide documents for the claims/rebates.
Total time spent: about 1 hour (including claims for private insurance, education expenses, donations)
Total $$$ spent: ZERO, ZILCH, NADA!
A casual stroll through the lunatic asylum shows that faith does not prove anything.
Seems relevant http://accidenture.com/
What the submitter wrote is not entirely accurate. All this person's financial data were not available. What was made available was his inbox, containing the full names and personal number (SSN) of this guy and his wife, and some information on a company he was working for.
The officials say that while they do not consider the information that was revealed to be sensitive, they take any information leak very seriously, and therefore the site will stay down until they find the error and correct it.
Altinn has had problems handling the load on these dates (when people do their taxes) for years.
My guess it's that a caching solution has been hurriedly pushed onto a system poorly set up for it, and accidentally set up to cache login credentials. When the credentials storage method is the right(wrong) type, a single-character typo in Varnish can be enough to do that, causing disaster.
xkcd is not in the sudoers file. This incident will be reported.
It is certainly very convenient, when it works. It feels kinda strange to trust every financial detail of my life to the government, so whether it is good in a real sense is a question I'm very open to debate. It does allow some very useful applications to be developed, with a very nice potential for streamlining interaction between government, citizens and private sector. This is actually very high on the government's agenda, which I'm happy about, because the bureaucracy is sometimes both heavy and heavy handed. If it is done well, it could potentially enable citizens to simulate possible choices in their lives before they make a decision: "If I do $that, the taxes will be $this". It would also enable an improved public debate: now it is a lot of bickering of the style "if you raise $that_tax, it will adversly effect $that_group" "no, it won't, but not doing it is required by $that_group". They're just making things up, of course, the debate is usually completely devoid of facts. Soon, it might be possible to simulate those scenarios on a regular basis, so we get real facts on the table before making a decision. Unfortunately, there's a long way from good ideas to actual implementations. I've been in meetings with the people who actually order these systems, and what can I say... Heads gotta roll to go anywhere... They're easily blinded by suits, and they have no idea what makes a robust system. So, for now, I'm not too confident it will happen, even though there are some very interesting ideas around.
Employee of Inrupt, Project Release Manager and Community Manager for Solid
Accidenture living up to its nickname.
Yes, it seems the project audit by Veritas found insufficient testing as one of the criticisms raised. Does .Net/Sharepoint have any serious tools for systems testing, like you have a plethora of for Java?
Mostly because whiny rich people will start screaming about a 45% tax rate. so it's spread out across things.
we are taxed as heavily as many Europeans, but we dont get the good healthcare or infrastructure that works well.
Do not look at laser with remaining good eye.
> your property
Norway taxes that too, on the municipal level.
> your spending
Norway taxes this too: a sales tax (VAT) on the national level, at 25%. No, there is no decimal point missing there.
> your savings
Yup.
Silly Americans complaining about taxes, you haven't seen nothing!
(But actually, I don't think the overall taxation level in Norway is too high, though some of it is pretty regressive, e.g. the VAT)
xkcd is not in the sudoers file. This incident will be reported.
As part of the military-industrial complex I just want to say, "Thanks for forking over all that money!" Oh, the Gulf Arab states and Israel also owe the US taxpayer big time, but they're too arrogant to say "Thank you".
Improper caching could have happened if the URLs were not unique. But caching in this case is just so wrong. And rarely is it even right. Static data can simply be preloaded in a server as streamlined as a cache would be, and those get delivered at cache speeds. Dynamic data should not be cached except in the browser, and even that with a short expire (5 minutes max).
Most pages now are not static in any meaningful sense - consider the homepage on almost every website. They have some dynamic data like news, but don't change every second, but may do every few minutes, and thus are cached, and often even on dynamic pages you can cache fragments if not the whole page. Server-side caching is almost always the right thing to do (in conjunction with browser-side caching), if it's done correctly and massively reduces the load on the server, so not sure why you feel it is wrong?
No doubt saves time on renting an apartment or getting a loan too -- they can verify your income without a pile of bank statement and tax form printouts.
This space intentionally left blank
They would be up in arms about 45%... currently they pay around 35% if they have bad accountants. Some rich people brag about paying less in taxes than their employees and are screaming and kicking about restoring their tax rate to what it was under Bush (39ish%).
Bunch of useless egotistical idiots the lot of them. The know-nothings they hire seem to think they're gods gift because they work for this piss poor company , but most of them are clueless. Many a time I've had to sort out the mess they've created.
The queuing system in Denmark was one provided by a company selling out-sourced queuing systems operated in the cloud. From someone who obviously knew what they were doing.
Ah, makes sense. My first guess was "the ever-dangerous auto-increment ID column strikes again!"
But of course I didn't RTFA.
Should still have been picked up if an adequate amount of load, spike & endurance testing had been performed.
Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.
I believe Norway has similar identification numbers as Sweden, i.e. birthdate, a few other digits and a control digit,
Two digits, actually.
The ID number is of the form:
DDMMYYXXXYZ
The Y and Z are moduli 11 numbers calculated based on fixed multipliers for each preceding digit. There is no way to change one or even two digits without it becoming invalid.
That was not the problem here, of course. If I were to venture a guess, they pull the data, store it, then display it. Without checking well enough whether the data pull succeeded. So if it always fails, everybody will get the last successfully pulled data.
But the real problem is the government (by pressure from the right) farming out important work like this to the lowest bidder. It ends up more expensive, less thought through, and ill maintained. It's written in Java in India, ffs.
The queue system was running on Amazon EC2.
I'd say Greece has got more of a problem with the fact that in 2011 the total tax paid was USD $1.2 billion, while unpaid taxes amount to USD $77 billion...
I'm a dreamer, the world is my playpen. But hey, I'm a serious person, I can't dream all the time.
I betcha some contractor decided to use a singleton in the authentication code or something like that! Probably worked great in single-user testing! And they probably never did any multi-user testing. I saw a very similar thing happen a company I worked for a few years back. They had to push back a release date because of static methods and members in a bunch of the auth code. Whole system worked great as long as only one user ever logged on at a time. Too bad we'll probably never know, because if there is an investigation the results will quickly be filed in a cabinet in the basement behind a sign that reads "Beware of leopard."
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
That depends on where you live. Your local and state taxes vary greatly. In Florida, the tourists pay most of the taxes, in Alaska the oil companies do. As to Federal taxes, they're lower than they've been in my lifetime, and I turn 60 in a couple of weeks. But the Goddamned Illinois state income tax doubled last year. I may move to Missouri when I retire.
Free Martian Whores!
Yeah we have a similar problem, everything is outsourced to consultants who overcharge for crappy solutions instead of hiring a few competent people to develop and run the systems. A few months ago, some glitch in Tieto's datacenter caused problems for getting prescription drugs, vehicle inspection as well as several commune services, apparently they didn't have any redundancy. IMO, the state should create a public "cloud" service with built-in redundancy which all government services can use, the companies hired by individual agencies certainly can't handle it.
Sounds like it was just a bug that didn't get caught by QC. An unexpected exception caused a page to get cached incorrectly. Perhaps the page wasn't supposed to be cached, or perhaps the cache key was calculated incorrectly.
I hope F5 takes a big hit from this. They aren't a proxy solution, so to mess it up in that way takes extra effort. Though I'm feeling all full of myself for having guessed right. Though it was an educated guess. Why yes, I do hold an F5 certification as well and secure/accelerate web sites for a living, but I haven't run across that specific problem before.
I'd put it down to Accenture being a MS shop and screwing up the F5 part because they either said "how hard can it be" or they subcontracted out the F5 config, and didn't manage that well. Having subcontracted for companies of that size, I'd guess subcontracted, but that's even more of a guess than my first guess they were F5s misconfigured.
Learn to love Alaska
You could say that about any software issue ever
Why did you throw the word "socialist" in there as a cheap insult? Do other sorts of government make less of a hash of IT projects?
Rgds
Damon
http://m.earth.org.uk/
I may be missing something here, but what do they have to do with teeth?
"If a nation expects to be ignorant and free in a state of civilization, it expects what never was and never will be."
The problem, at least in the U.S., isn't that taxes are too high. It's that the tax money gets wasted, whether it's through kickbacks, fat contracts, or just useless pet projects (and all three are not mutually exclusive categories).
That's the real thing that's making everyone so upset. But the ignorant get swindled by the same people receiving the kickbacks into thinking that taxes are bad in general. They don't realize that abolishing taxes makes things worse than a little bit of waste (though the waste is increasing as companies try to milk more and more out of the government).
Taxes are not bad in and of themselves. They're just not being used properly.
"If a nation expects to be ignorant and free in a state of civilization, it expects what never was and never will be."
This seems very likely. The F5 solution was tacked on after the entire site went down last year due to traffic overload.
for i in `facebook friends "=bday" 2>/dev/null | cut -d " " -f 3-`; do facebook wallpost $i "Happy birthday!"; done
Someone made a pretty funny spoof site (half decent google translation). The translation will give you the gist of it if you can't read Norwegian. I especially like "Login as Kenneth (does not require password)". They missed the chance to misspell Buypass (an authentication service) as Bypass, though :)
Are you a grammar Nazi? I'm trying to improve my English; please correct my errors!
They could, but they don't. They usually demand a printout of last month's paycheck.
xkcd is not in the sudoers file. This incident will be reported.
Though I'm feeling all full of myself for having guessed right.
It might please you to know this info from a norwegian article:
Not only did they use "Big IP" from "F5 Networks", but it seems to have been a previously unknown bug in the cache system. They reportedly managed to reproduce it in the lab, and have worked with Altinn to solve the problem. Right now they're running without caching, with the extra load problems that causes.
It also seems like they applied a hotfix to OpenSSO (which they also use) that made it less prone to garbage collection, which increased the overall performance to a level slightly above what it was earlier with caching.
It's The Golden Rule: "He who has the gold makes the rules."
and how much money he/she has in the bank
Rubbish
MS-based web application basket
This should not be an issue. Massive scaling with IIS and .NET is not an issue. Even on rather modest hardware. Azure has had some availability issues, but it does scale. Massively. Just ask Apple.
As for Accenture, sure, if you give those guys that much money to develop something, you're an idiot.