Cryptocurrency Exchange Kraken Suddenly Goes Dark For Two Days (sfchronicle.com)

An anonymous reader quotes the San Francisco Chronicle: One of the biggest cryptocurrency exchanges was down more than 40 hours this week, causing clients to freak out... San Francisco's Kraken went offline at 9 p.m. on Wednesday for maintenance that was initially scheduled to last two hours, plus an additional two to three hours for withdrawals, according to an announcement on the company's website. "We are still working to resolve the issues that we have identified and our team is working around the clock to ensure a smooth upgrade," according to a status update on Kraken's website posted early Friday. "This means it may still take several hours before we can relaunch." Shortly after noon, the company said it was "still working to track down an elusive bug which is holding up launch." It promised customers "a substantial amount of free trading" after the problem was resolved. In previous updates, Kraken mentioned it is working on "unexpected and delicate issues" and assured clients their funds were secure, adding that "Yes, this is our new record for downtime since we launched in 2013. No, we're not proud of it."
It's 53 hours after the downtime began, and their web page is still showing the same announcement.

"Kraken is presently offline for maintenance."

Comments

Check their actual status page

[wyoung76]

Their status page (https://status.kraken.com/) claims that they're preparing to relaunch the site soon.

FUD

[Anonymous Coward]

While the facts seem to be in order, this is written as if it was to fuel FUD. Kraken planned to change/upgrade their trading platform for a long time; now that they do, they ran into some technical glitch that they are trying to solve before relaunch.

The relevant link is status.kraken.com

Re:FUD

[Cederic]

Or it could be something like "Shit, we just fried the drive controllers, the RAID array is toast, where are the backups? What do you mean it takes three days to ship them back, build new arrays and restore the data? Get on with it!"

If you're not running fully resilient (and that's expensive, so most people don't) then it's quite easy to lose a couple of days.

I hate to disclose this but...

[slashmydots]

Recently the third party that does domestic wire transfers to a bank account made a mistake communicating with Kraken's systems and they paid me around $400 twice. I told them about it and offered to reimburse them but their support staff never got back to me about how to do so and basically just locked one of my latest deposits and most of my accounts. I even offered to straight up paypal them the funds to get my account unlocked. No response. So if this happened on a larger scale to more people, they just lost A LOT of money. Btw wire transfers are expensive at most banks so just use Coinbase, they do it as a free direct deposit. Oh and also their system actually works.

Most Interesting Coder

[mentil]

I don't often test my code; but when I do, I do it on production server.

Re: Most Interesting Coder

[Reverend Green]

Sounds like you're doing it the Agile (tm) way!

Way to keep fucking up the market

[Anonymous Coward]

It's amateur hour stuff like this that keeps fucking up the market. Coinbase and their sketchy release of BCH trading just as Bitcoin had peaked, tanked the market a month ago. Now I fully expect this news to do the same once it becomes widespread.

If you're going to be running a major currency exchange, why the hell are you taking down and deploying into production systems?

With the world of VMs these days, duplicate your entire fucking production environment over to new VMs. Leave the original environment up and running to keep serving customers. Deploy into the new duplicated VM environment. If all passes sanity testing after deployment, cut the traffic over from the old production environment to the new production environment. With VMs, all they should need is a shitload of RAM and disk space. CPU usage will be low on the new environment that deployment is being done on, and once customers cut over that usage will just drop on the old VMs and pickup on the new VMs.

If the whole thing goes tits up, you still have your old prod environment you can cut all the traffic back over to. if it all goes well, after some amount of time you can decommission the whole old VM environment.

Aside from being able to safely deploy, having a duplicated prod environment, hopefully in a separate datacenter gives you redundancy during the time between deployments

Re: Way to keep fucking up the market

[Reverend Green]

PHB: How long have you been working with Crypto(tm)?

Devops nerd: cryptocurrency processing is not especially different from other workloads, and benefits from the same modern best practices.

PHB: get out of here, nerd. Next!

Wannabe: Hi! I love Agile(tm)! I'm willing to sell my soul for cheap!!

PHB: Awesome! How long have you been working with Crypto(tm)?

Wannabe: I've been doing CRYPTO for five years! I'm CRYPTO Certified(tm)!

PHB: double awesome! Now a technical question: describe the basics of how to set up a resilient production environment for a web application.

Wannabe: huh? What's resilient mean??

PHB: I don't know either! You're hired!

Re:Not So Bad: It's 99.5% Service Availability!

[Antique Geekmeister]

Citibank was deeply involved in the subprime morgage crisis, and had to pay fines of roughly 7 billion dollars. If you think that major banks are immune from large scale theft, I'm afraid you'll need to rethink that.

The blockchain tells me

[perpenso]

It's 2AM And do you know where your BTC is?

Yes, the blockchain tells me. ;-)