Fire Takes Azerbaijan Offline (datacenterdynamics.com)

judgecorp writes: On Monday, 90 percent of Azerbaijan lost Internet access, due to a fire at one data center in Baku, the capital of the former Soviet Republic. Cables caught fire at the Delta Telecom facility, and international providers including NTT and Telecom Italia all lost service for nearly eight hours. Some interesting snippets: Azerbaijan is a former Soviet republic that has seen rapid development thanks to its rich oil and gas reserves. The country has been running several projects aimed at modernizing its communications infrastructure, including participation in Trans-Eurasian Information Highway (TASIM). ... At about 16:10 on Monday, consumers, businesses and government agencies across Azerbaijan suddenly lost their connections to the Internet. Banks couldn’t make domestic money transfers, and even Point-of-Sale terminals were not working. ... Interestingly, no international traffic flowing though Azerbaijan was affected by the outage. “Transmission channels to Georgia, Iran, and the Middle East were working at full capacity,” Iltimas Mammadov, the minister of communications, told AzerNews.

Comments

Shaming a country

[michaelmalak]

the former Soviet Republic

Might as well just say pwned by Lenin

Goes through one spot for a reason

[rahvin112]

The entire countries internet goes through a single building for a reason and it's not cost. They've not forgot their KGB roots.

Re:Goes through one spot for a reason

[Aaden42]

Single point to take the entire country offline?

CLOSED: Works as designed.

Re:Goes through one spot for a reason

[Anonymous Coward]

It is because of cost. The US has more sites and the monitoring works just fine. China is probably the same.
Not that there's anything wrong with keeping costs down. I don't know Azerbaijan, but maybe 99,9999% uptime is not their top priority.

Re:Goes through one spot for a reason

[bobbied]

Actually, it's most likely because they didn't think of physical location redundancy... I've seen "redundant" set ups which where strictly maintained within a building get totally undone the second the fiber runs get mounted on the same poles to cross the street before they split and when their separate ways. Nobody was paying attention until a dump truck accidentally snagged both fiber runs and yanked them down the street, disconnecting both redundant fibers in one accident.

So, where it may have started as a KGB thing, nobody really thought though how their redundant design on paper, fell to a single physical event because they didn't maintain enough separation. Or, maybe it just was the NSA, providing low cost engineering services to keep them under their thumb?

Re:Goes through one spot for a reason

[goombah99]

I though the internet was supposed to route around damage. Something is too centralized.

Re:Goes through one spot for a reason

[schnell]

I though the internet was supposed to route around damage. Something is too centralized.

At an IP routing level, sure. But that doesn't do any good if there's only one physical cable or set of cables linking you to the rest of the world.

Re:Goes through one spot for a reason

[AK Marc]

The Internet doesn't exist within most countries. There are a few POPs to the Internet, one or two, but two people on different networks usually don't change carriers at the tiny POPs. NTT and TI might not peer at that POP, so two people would have their traffic leave the country to come back. I've seen that happen with Pacific islands with a single POP. That means that the Internet starts one hop out of the country.

The Internet routes around the damage. Nobody outside Azerbaijan noticed the outage. But the people inside are on the Internet the same way as someone on Comcast is on the Internet when Comcast goes down.

When I worked for a US company with an office there, we got training on how to bribe. The issue is that bribing a foreign government is illegal. And in Azerbaijan, so much is still owned by the government. The government workers are so poor that they use their power to work for tips. In my case, we had a microwave link in a government owned tower. The equipment was up on the top floor. If we wanted access, we had to pay an "elevator fee" bribe paid to a government worker for access to a government building. It was illegal, but the options were down to "don't do business in Baku, or do so in violation of US law".

Re:Goes through one spot for a reason

[AK Marc]

99% of redundancy I've seen comes down to a single line of config in a single box. Redundancy is great, but a delicate cluster of a single type of gear that requires that single vendor proprietary redundancy protocol work perfectly 100% of the time is redundant in theory only.

That and I've seen great cost for redundancy for redundancy's sake where the complexity added to get redundancy made the whole network less stable. 1 box with a MTBF of 10,000 hours means one outage every year. 4 boxes with the same MTBF means 4 outage incidents a year, though hopefully without a network down incident. Replacing the 1 box with another 1 box with a 100,000 MTBF would see a better increase in MTBF-related downtime, and the 4-box solution has full outages on a regular basis from the delicacy of the clustering used.

So I wouldn't put it down to malice and KGB. All the population is in Baku, and it is easier to have one place to go. They had lots of redundancy, just not physically separate. Every place I've seen with multiple power feeds all had a single point of failure. One went so far as to have 3 power feeds from 2 carriers at 3 building entry points, 2 generators, 3 UPSs, and all that, with the 3 UPSs in the same small room, and the power from them out to the facility through a single power panel. Though the feeds alternated A B and C, a single event at that one panel could bring down the entire facility, regardless of the state of the 3 UPSs and 2 generators.

Re:Goes through one spot for a reason

[bobbied]

Yep, that reminds me of a "redundant" system we purchased once. It had two processors that ran in locked step so you could fail over seamlessly from the primary to the secondary at the slightest indication of a fault (say an ECC error, or some noise on some address line). Sounds great on paper... Problem was, when this happened, you where no longer redundant until you could reboot the system which took something like 20 min to cleanly shut down, reset the redundancy and restart everything. Starting the system in non-redundant mode took like 5 min because it didn't have to re-sync everything. The failure rate of the individual systems was such that it failed about once a quarter or so and we'd process on w/o a break, but it took us 20 min a quarter, 80 min a year to get it back into redundant mode. The thing was actually *better* availability when we didn't run it fully redundant because the amount of down time was much less to just reboot the one side... So we'd go fully redundant when we could afford the boot time, like in planned maintenance windows which where at the customer's request, but then once it failed over, we'd leave it in non-redundant mode and just boot the one side....

Sometimes redundancy is just more complicated and depending on the MTBF's involved, not a good idea...

Re:Goes through one spot for a reason

[AK Marc]

I'm full of shit because why? You think I'd lie about it if I were given training in how to bribe? Or are you claiming the situation where government employees demand bribes to do their job is impossible?

And the whole country says..

[the_skywise]

Network's down... bagel run!

And this friends, is a real problem

[Anonymous Coward]

If terrorists were halfway competent, this is the kind of thing they'd do rather than trying to frighten the population with useless displays of violence.

And it can happen by accident.

Or was it?

Re:Route around problems

[RavenLrD20k]

Can't route around a horrible network design. 90% of the country's network was all routed to a single building with evidently no redundant links. Great for spying...not so great for disaster recovery and avoidance. But there's good news on multiple levels: The rest of the world's network didn't go down because of it!

Re: They should have used APPS!

[ModernGeek]

Yeah, why use these data centers when you can just put your data in the cloud?

Re:why does this happen?

[Anonymous Coward]

I thought it was to send nekkid ascii pics back and forth

For Context

[avandesande]

Pennsylvania:
46,055 sq mi
12,787,209 population

Azerbaijan:
33,436 sq mi
9,624,900 population

Re:For Context

[Anonymous Coward]

Pennsylvania:
12 letters
Starts with a P

Azerbaijan:
10 letters
Starts with an A

Cold War much?

[Crimey McBiggles]

So what you're saying, is that Azerbaijan was a former soviet republic; seriously? Let's move on from the cold war and call it what it is: part of Europe. We don't commonly refer to the Ukraine as a former soviet republic, nor Belarus, nor Estonia. Get with the times and stop posting ignorant propaganda.

Re:Cold War much?

[tsqr]

So what you're saying, is that Azerbaijan was a former soviet republic; seriously? Let's move on from the cold war and call it what it is: part of Europe.

Part of Europe? Oh, yeah -- the part of Europe that is called the Russia and Caucasus region of Asia, though some might consider it part of the Middle East region of Asia.

Re:Cold War much?

[thegarbz]

At this rate it will only be a matter of years before Australia is considered part of Europe too.

Re:Cold War much?

[AK Marc]

Since all that's so complicated, let's just call it a former soviet republic.

Part of WHERE?

[Hognoxious]

Let's move on from the cold war and call it what it is: part of Europe.

Azerbaijan
Ukraine
Belarus
Estonia.

One of those is not like the others.

(Hint: it borders Iran).

Re:Part of WHERE?

[unixisc]

The thing about Azerbaijan, Turkey and the stans is that the secular elements of those countries see themselves as a part of Europe, while the Islamic elements see them as a part of the medieval Timuride, Seljuk or Ottoman empires. (I'm not including Tajikistan in this analysis, since that country, despite being Sunni, is culturally more allied to Iran).

Azerbaijan is Shi'ite like Iran, but Turkic, like 4 of the stans and Turkey. Kazakh, Kyrgyz, Turkmen and Uzbek are all Sunnite and Turkic. Turkey was a secular country since Kemal Ataturk, but has reverted to being an Islamic country under Erdogan, and is past the point where it considers itself a part of Europe. However, Azerbaijan and the other stans still have secular Soviet era quasi-democracies, and those parts still consider themselves a part of Europe, as opposed to the Middle East or Central Asia.

Personally, I group Turkey along w/ Kazakhstan, Kyrgyzstan, Turkmenistan and Uzbekistan as the Turkic world, just like the Arabs have the Arab world - everything from Iraq to Morocco. Azerbaijan and Tajikistan are oddballs - Azerbaijan being Turkic but Shi'ite, while Tajikistan is Farsi but Sunnite. So these 2 could either be influenced by Iran or Turkey, but at this point, both those countries are Islamic, and the differences just theological

Re:Part of WHERE?

[AK Marc]

I always lumped Kazakhstan in with Mongolia. They don't even border, but they are similarly rural with nomadic roots.

Part of the issue with the former soviet republics is that Russia shuffled the locals around. Russia liked the idea of a winter retreat Crimea, so they shipped in Russians, and move the others out.

Re:Part of WHERE?

[unixisc]

But that's been a process that's been undone since the Cold War. Like Kazakhstan was 50% Kazakh and 45% Russian in 1991, when the Soviet Union came apart. Today, it's 63% Kazakh, and 23% Russian: since that time, most Soviet people have returned to their native lands - like Russians returning to Russia, Ukrainians returning to Ukraine, Uzbeks returning to Uzbekistan and so on. As for Crimea, Stalin saw Crimean Tatars as a potential threat, and shipped them out to Central Asia. Culturally, the Crimean Tatars have more in common w/ the people of the stans than w/ Cossacks, who are the people native to the Crimea.

While Kazakhstan and Mongolia are similar, Turkic and Mongol are 2 related but distinct ethnicities. Kazakhs are Turkic, like their neighbors to the south. And there's religion as well - Mongols being predominantly Tibetan Buddhist (Shamanism having been stamped out during Chinese rule) while Turks/Kazakhs/Kyrgyz et al are predominantly Muslim.

Re:95% muslim country can't keep basic infrastruct

[unixisc]

I am the first person to criticize Muslims, but people of the ex Soviet Republics are just nominal Muslims - nothing like the Quran thumpers that you'll find in Saudi Arabia, Pakistan, Afghanistan, Palestinian Authority, Syria, Egypt, Iraq or Iran. In fact, most of them also recognize Israel and have full diplomatic relations w/ Israel - something that's unthinkable for most Muslim countries.

Also, while I'm no fan of Communism either, that's the one thing that has kept the stans - particularly Uzbekistan - from becoming an Islamic regime like Afghanistan.

Bummer

[nospam007]

Seems like their data center didn't have a firewall.

Re:Route around problems

[AK Marc]

Can't route around a horrible network design. 90% of the country's network was all routed to a single building with evidently no redundant links.

The Internet routed around Baku without issue. And there were redundant links. They just all went into the same building.

Someone took this article too seriously

[Ukab the Great]

http://gizmodo.com/5833267/why-the-internet-should-die-in-a-fire

90%?

[smoot123]

That means they have more than 10 users?

Azerbaijan - Land of Fire

[Dr. Spork]

I always thought that it was weird for a country to advertise on the jerseys of Atletico Madrid, and I thought that "Azerbaijan - Land of Fire" was always a weird motto. I think they were trying to indicate passion, but really, who would want to live in, or even visit, the Land of Fire? Especially now that the fire took out their internet?

If you think I'm kidding, click here.

Re:95% muslim country can't keep basic infrastruct

[unixisc]

The Syrian regime is 'secular' and socialist. The Syrian people - the vast majority of them are Sunni fanatics, indistinguishable from their comrades in Saudi Arabia and Egypt. The reason ISIS has the support it has is that an overwhelming majority of Syrian Sunnis support it. They are 80% of the Syrian population

As for the Baath regimes in both Syria and Iraq, they are/were 'secular' in the following aspect. Whenever a country is under pressure to be declared 'Islamic', what that means is that the majority sect would be the one whose practices would define Islam in that country. So it's Wahabi for Saudi Arabia, Shia for Iran, Iraq, Bahrein and Azerbaijan, Muslim Brotherhood in Syria or Egypt and so on. Given this standard, it was obvious that Sunnis in Iraq would be superficially opposed to an 'Islamic' state, as would Alawites in Syria, while Sunnis in Syria and Shias in Iraq would both be supportive of it.

However, just b'cos Sunnis in Iraq didn't support the Islamic state of Iraq didn't mean that they are secular: they were happy to join ISIS, which is explicit about having an Islamic state that is Sunni. That they support. Conversely, in Syria, the Baathists, while in an Alawite coalition w/ Druze and Christians, would be perfectly happy w/ an Islamic state where the standard was Alawite. And thanks to their power, at least 1 seminary in Iran recognizes them as Shia Muslims.