Slashdot Mirror
Cryptocurrency Exchange Kraken Suddenly Goes Dark For Two Days (sfchronicle.com)
An anonymous reader quotes the San Francisco Chronicle:
One of the biggest cryptocurrency exchanges was down more than 40 hours this week, causing clients to freak out... San Francisco's Kraken went offline at 9 p.m. on Wednesday for maintenance that was initially scheduled to last two hours, plus an additional two to three hours for withdrawals, according to an announcement on the company's website. "We are still working to resolve the issues that we have identified and our team is working around the clock to ensure a smooth upgrade," according to a status update on Kraken's website posted early Friday. "This means it may still take several hours before we can relaunch." Shortly after noon, the company said it was "still working to track down an elusive bug which is holding up launch." It promised customers "a substantial amount of free trading" after the problem was resolved. In previous updates, Kraken mentioned it is working on "unexpected and delicate issues" and assured clients their funds were secure, adding that "Yes, this is our new record for downtime since we launched in 2013. No, we're not proud of it."
It's 53 hours after the downtime began, and their web page is still showing the same announcement.
"Kraken is presently offline for maintenance."
It's 53 hours after the downtime began, and their web page is still showing the same announcement.
"Kraken is presently offline for maintenance."
It's amateur hour stuff like this that keeps fucking up the market. Coinbase and their sketchy release of BCH trading just as Bitcoin had peaked, tanked the market a month ago. Now I fully expect this news to do the same once it becomes widespread.
If you're going to be running a major currency exchange, why the hell are you taking down and deploying into production systems?
With the world of VMs these days, duplicate your entire fucking production environment over to new VMs. Leave the original environment up and running to keep serving customers. Deploy into the new duplicated VM environment. If all passes sanity testing after deployment, cut the traffic over from the old production environment to the new production environment. With VMs, all they should need is a shitload of RAM and disk space. CPU usage will be low on the new environment that deployment is being done on, and once customers cut over that usage will just drop on the old VMs and pickup on the new VMs.
If the whole thing goes tits up, you still have your old prod environment you can cut all the traffic back over to. if it all goes well, after some amount of time you can decommission the whole old VM environment.
Aside from being able to safely deploy, having a duplicated prod environment, hopefully in a separate datacenter gives you redundancy during the time between deployments
Citibank was deeply involved in the subprime morgage crisis, and had to pay fines of roughly 7 billion dollars. If you think that major banks are immune from large scale theft, I'm afraid you'll need to rethink that.
Or it could be something like "Shit, we just fried the drive controllers, the RAID array is toast, where are the backups? What do you mean it takes three days to ship them back, build new arrays and restore the data? Get on with it!"
If you're not running fully resilient (and that's expensive, so most people don't) then it's quite easy to lose a couple of days.