Slashdot Mirror
Disaster Strikes Norwegian Government Web Portal
An anonymous reader writes "Altinn.no is a web service run by the Norwegian government, on which citizens can find, fill out and deliver forms electronically. Every year Norwegian citizens can also log in to check their tax results. This year, as every year, the site was unable to cope with the traffic generated from everyone wanting to check their taxes at the same time. New this year, however, was that once people were finally able to log in, a significant amount of people were logged in as someone else. Users then had access to all financial data of this unfortunate person over two years back in time, in addition to the financial information of his wife and the company he worked for. Altinn shut down some 15 minutes later, and has been down since."
by the government sending them a letter saying how much is owed.
The government does all the calculations.
Really they need a staggered ticket system to distribute the load over time. Issue each citizen a ticket that indicates a period when they can log in to check data, both a soonest and latest date (stragglers not tolerated). This is no different than physical scenarios where people are grouped by first letter of last name, etc. in a crowded office and then each group served sequentially to lighten the load.
Wanna guess how the norwegian government decided how traffic shoul be scaled? Come on, guess They made a limit of 300 000 logins, before making the main web page redirect to a page saying "sorry the lines are full pleas pick a number" - it, apparantly , seemed more logical than scaling the hardware :P
It's been very briefly reported that this was related to a caching error. This guy's information was apparently cached and then served to everyone.
threaded app server + global who_is_logged_in variable = big mess
now we need to go OSS in diesel cars
* The government has spent on the order of $200 millions on this system
* Accenture is the main developer
* Every year the systems go down because it doesn't scale
* This year a queueing system was put in place to "fix" scalability
* From an outsider's view at least, it would seem like some cowboy decided to put up a Varnish-type frontend cache as a desperate measure to handle traffic with no thought given to sessions
* An independent report basically slaughtered most of the systems with criticism of flaws last year, which was kept secret until a week ago
* Also yesterday someone found several flaws which allowed any website to grab a json(?) script and steal userinfo if the browser had a valid session
I foresee a large lawsuit settlement in his future
This isn't the USA
http://blog.nexusuk.org
From the people in charge: "This person visited 18:17 and checked his tax return, and for some reason or another there was an error in the system, and this page entered the so-called cache memory of our servers, where it doesn't belong". You can try to decipher from that what you will.
In other words, either the person who wrote that didn't know what he/she was doing, or else a manager got involved in the software design decisions and forced the programmer to incorporate a blazingly stupid idea.
In either case, someone probably said something vague about "saving cycles" and everyone else nodded.
#DeleteChrome
It's simple. They got slashdotted last year. So, this year they did all they could to end the problem. Likely, they used SSL for security. And for anything high-traffic, you put an SSL proxy in front of the servers. Servers, be they Linux or otherwise, take a much bigger hit with encryption than dedicated security boxes, like F5. So they had some proxy in front of the servers. I've put similar in place in New Zealand for the IRD, and I'd expect that the IRS uses F5 in front of their secure web sites. And dedicated proxy devices, like Blue Coat, also do SSL offload. So, mis-configuring a proxy used for SSL offload would easily serve a cached page, after all, that's its primary purpose, the SSL offload was an afterthought.
That's what happens when you have a problem one year and throw money at it to fix it without a full understanding of the problem and the fix. I'd bet it was outsourced. And I bet they outsource it again next year. I could do better for a lower cost, wouldn't be hard to do better than their performance the last two years.
Learn to love Alaska
Oh we do that here in the US too, for most salaried jobs. But then we *also* tax your property, your spending, your savings and then every year we also make you fill out forms that tax you more.
When everybody's money is 'stored' in a government computer somewhere saying how much money you have, imagine what happens when there's a glitch putting your money in someone else's account.
Yeah, I know, bank accounts.
But, glitches happen there, too. At least you have a little cash to get to and from the bank to pursue the matter. When it's digital all the way down, what will you do?
I'm not a lawyer, but I play one on the Internet. Blog
That's not true. There's a checksum on our SSNs, and the checksum is constructed in such a way that the two most common mistakes in entering SSNs (double one digit, forget another, and transpose two digits) always results in a invalid SSN.
But yes, it's still possible to hit someone elses SSN by accident, but it takes more than one digit wrong. (it takes multiple wrong digits in such a way that the new SSN happens to pass the checksum-test, *and* match an actually used SSN)
This being Norway, you should have written "I smell pagan shouting and blood and entrails in his future."
Ezekiel 23:20
Ok - so the deal is this: For everyone in Norway, you can check 3 vital numbers: Amount earned, amount taxed and amount owned of every year. The number are skewed somewhat since they do not cover the full value of your house, it is after certain deductions on your salary, it is with your loans deducted from what you own, etc, but in essence it can give you a ballpark on how much money someone earns.
So, why is this? One of the major reasons is to ostracize anyone that pay little tax as compared to what they earn/own. So you would not need to ask your presidential candidate for his tax record - it is already online: http://skatt.bt.no/skattelister/9397621/Jens%20%20Stoltenberg *. You would also at once see it if your palace-owning neighbour had millions in earnings but payed nothing in taxes.
* This number is from 2009, you now have to login to a governmental site to be able to look up taxes for people. This is to stop malicious use of the numbers.
From the people in charge: "This person visited 18:17 and checked his tax return, and for some reason or another there was an error in the system, and this page entered the so-called cache memory of our servers, where it doesn't belong". You can try to decipher from that what you will.
This is quite easy to interpret. They turned on caching to speed up page loads, but without disabling it for logged in users or sensitive pages, so one user logs in, visits /my_account or whatever, and the page is cached, then when the next 100,000 users visit /my_account the cached page (containing the first user's details) is served without authentication (!). Page caching works great for public pages like / which are served the same to everyone, and doesn't work so great for pages which require authentication.
It's the sort of mistake you wouldn't normally see on a site this size as it's a rookie error and ANY sort of testing of caching would catch it, but apparently that's what they did. Probably they only intended to cache public pages or something and managed to extend it to private pages by mistake. Their server could be properly configured and secure but then this mistake triggered by one small change to their caching config by someone who didn't know the implications.
I believe Norway has similar identification numbers as Sweden, i.e. birthdate, a few other digits and a control digit, if you throw some of the other digits off, it likely won't be a valid number. Besides, these numbers are not secret and you usually need some other form of authentication than just the number, electronic identification, number printed on tax form, etc.
All the URLs look alike because the login ID is in cookies, and the cache wasn't set to figure in the cookie state.
now we need to go OSS in diesel cars
I'd be willing to bet that it was something turned on, because they needed to lighten the load on the servers. IT could have been a front end caching machine, or on the web server itself in code. In either case, it clearly wasn't tested as well as it should have been.
You *can* cache authenticated pages. Really, the /my_account (your example) only needs to be generated once a year. If that happens to be the main page to view from, you'll keep ending up back on it, to go to other pages. Generating it once is a whole lot more efficient than generating it 15 to 30 times. You'd have to get a bit creative with how you ensure no user can look at another users results. For example, if you happened to save the page as /cache/my_account_[userid].tmp, that's all fine and dandy, unless the code forgets to actually populate [userid]. :)
So may ways to screw this up, and they all should have been caught in testing.
Serious? Seriousness is well above my pay grade.
LoL and to imagine some countries (like Greece for example) are actually collecting your next years
tax as a sort of down payment. Yep, when paying taxes in 2012 the Greeks are asking taxpayers to pay
upfront for what they are going to earn untill the end of the year.
No wonder that country is head first into debt.
-- no sig today
I normally wouldn't care about this, but since the Norwegian government (i.e. the people, myself included) paid 1 billion NOK for this solution, I expect it to WORK. Mind you, this is not the first time we've had problems with Altinn, this has been a recurring drama the past few years. As the article states; every year they claim to be prepared, and every year they are unable to deliver.
We're not *that* many people in Norway (recently hit the 5 million mark), and certainly not that many adults checking their tax returns online. Guesstimate: 1 million? And how many checks it simultaneously? Let's be generous and say half.
So how the hell can a 175 million USD project not be able to deal with 500k visitors? It's a fucking joke.
www.6502asm.com - Code 6502 assembly or.. DIE!!
That's not correct. Only the final sums are/were published after the affected person has had a chance to verify and correct the information. Here all his details were published, which is a severe violation of his privacy.
Employee of Inrupt, Project Release Manager and Community Manager for Solid
Mod parent Informative. They are actually using F5's Big Ip solution, from my snooping before it went down. And it was outsourced, to Accenture, who has such a good track record producing stable, efficient, Microsoft-based solutions.
What is even more funny, just last week, a report leaked in the Norwegian press about this very system being hastily implemented, poorly tested and perhaps insecure.
for i in `facebook friends "=bday" 2>/dev/null | cut -d " " -f 3-`; do facebook wallpost $i "Happy birthday!"; done
It is done similarly in über-effective, ultra-efficient Singapore:
1) Let's say I'm employed by company C. Company C will send to taxman my identity card number and the amount they have paid me for the tax year.
2) Taxman will do the calculation of tax. Taxman will also consider the recurring tax claims/rebates I am likely to have (spouse/parents-related rebates, for example).
4) Taxman sends me a reminder to confirm their calculations on their website.
5) I will adjust the calculations if needed and submit the final figure.
6) Taxman sends me the final amount of tax I need to pay with payment options including a 12-month instalment plan deducted from my bank account.
7) If I'm audited, I will have to provide documents for the claims/rebates.
Total time spent: about 1 hour (including claims for private insurance, education expenses, donations)
Total $$$ spent: ZERO, ZILCH, NADA!
A casual stroll through the lunatic asylum shows that faith does not prove anything.
Altinn has had problems handling the load on these dates (when people do their taxes) for years.
My guess it's that a caching solution has been hurriedly pushed onto a system poorly set up for it, and accidentally set up to cache login credentials. When the credentials storage method is the right(wrong) type, a single-character typo in Varnish can be enough to do that, causing disaster.
xkcd is not in the sudoers file. This incident will be reported.
It is certainly very convenient, when it works. It feels kinda strange to trust every financial detail of my life to the government, so whether it is good in a real sense is a question I'm very open to debate. It does allow some very useful applications to be developed, with a very nice potential for streamlining interaction between government, citizens and private sector. This is actually very high on the government's agenda, which I'm happy about, because the bureaucracy is sometimes both heavy and heavy handed. If it is done well, it could potentially enable citizens to simulate possible choices in their lives before they make a decision: "If I do $that, the taxes will be $this". It would also enable an improved public debate: now it is a lot of bickering of the style "if you raise $that_tax, it will adversly effect $that_group" "no, it won't, but not doing it is required by $that_group". They're just making things up, of course, the debate is usually completely devoid of facts. Soon, it might be possible to simulate those scenarios on a regular basis, so we get real facts on the table before making a decision. Unfortunately, there's a long way from good ideas to actual implementations. I've been in meetings with the people who actually order these systems, and what can I say... Heads gotta roll to go anywhere... They're easily blinded by suits, and they have no idea what makes a robust system. So, for now, I'm not too confident it will happen, even though there are some very interesting ideas around.
Employee of Inrupt, Project Release Manager and Community Manager for Solid
Accidenture living up to its nickname.
Yes, it seems the project audit by Veritas found insufficient testing as one of the criticisms raised. Does .Net/Sharepoint have any serious tools for systems testing, like you have a plethora of for Java?
> your property
Norway taxes that too, on the municipal level.
> your spending
Norway taxes this too: a sales tax (VAT) on the national level, at 25%. No, there is no decimal point missing there.
> your savings
Yup.
Silly Americans complaining about taxes, you haven't seen nothing!
(But actually, I don't think the overall taxation level in Norway is too high, though some of it is pretty regressive, e.g. the VAT)
xkcd is not in the sudoers file. This incident will be reported.
As part of the military-industrial complex I just want to say, "Thanks for forking over all that money!" Oh, the Gulf Arab states and Israel also owe the US taxpayer big time, but they're too arrogant to say "Thank you".
No doubt saves time on renting an apartment or getting a loan too -- they can verify your income without a pile of bank statement and tax form printouts.
This space intentionally left blank
Ah, makes sense. My first guess was "the ever-dangerous auto-increment ID column strikes again!"
But of course I didn't RTFA.
Should still have been picked up if an adequate amount of load, spike & endurance testing had been performed.
Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.
Yeah we have a similar problem, everything is outsourced to consultants who overcharge for crappy solutions instead of hiring a few competent people to develop and run the systems. A few months ago, some glitch in Tieto's datacenter caused problems for getting prescription drugs, vehicle inspection as well as several commune services, apparently they didn't have any redundancy. IMO, the state should create a public "cloud" service with built-in redundancy which all government services can use, the companies hired by individual agencies certainly can't handle it.
I hope F5 takes a big hit from this. They aren't a proxy solution, so to mess it up in that way takes extra effort. Though I'm feeling all full of myself for having guessed right. Though it was an educated guess. Why yes, I do hold an F5 certification as well and secure/accelerate web sites for a living, but I haven't run across that specific problem before.
I'd put it down to Accenture being a MS shop and screwing up the F5 part because they either said "how hard can it be" or they subcontracted out the F5 config, and didn't manage that well. Having subcontracted for companies of that size, I'd guess subcontracted, but that's even more of a guess than my first guess they were F5s misconfigured.
Learn to love Alaska