Slashdot Mirror
Fort N.O.C.'s Security in Obscurity
penciling_in writes "Brock N. Meeks of MSNBC reports
on his recent visit to VeriSign's secret location: 'The unassuming building
that houses the "A" root sits in a cluster of three others; the architecture
looks as if it were lifted directly from a free clip art library. No signs or
markers give a hint that the Internet's most precious computer is inside
humming happily away in a hermetically sealed room. This building complex could
be any of a 100,000 mini office parks littering middle class America.' The
report goes on to say: 'Access to the Network Operations Center, the "NORAD"
of the Internet's traffic monitoring, requires the electronic badge and then a
double biometric hand print scan.' And here are Karl
Auerbach and Robert
Alberti offering their interesting analysis of this report on CircleID."
Sure, the
I'm still fuming about that.
Trolling is a art,
This could actually be dangerous. Whenever I hide something I seem to inevitably lose it...
Although the article says that the location is a secret, a link from the article to www.root-servers.org happily tells you that server A is in Dulles.
Sigh. Deep Sigh.
There's more than the 'A' root server. Taking "it" down leaves a whole hurd of other root servers alive. Located all around the world.
The above linked articles are full of that which promoteth growth.
This is also the building that has the big red button labeled "Hijack Internet Traffic"
One bad monkey spoils the whole barrel.
It's cool to see someone write about the building you used to work in! I worked in this building, a bit more than 2 years ago. I was in Network Solutions' consulting arm, whose DC office was in that building, two floors under the NOC. The security really is as spectacular (and low-key) as you'd expect. You would NOT believe the camera surveillance they have facing outwards...you can see some of it, but you can't see some of them at all. And the cameras themselves are startlingly cool...there's a small strip mall across a major highway from the facility, with a clear line of sight. One of the security guys showed me how far the zoom worked, as he zoomed in on a guy smoking in front of a bookstore in the strip mall...about half a mile away. It was still a clear picture.
When 9/11 happened, we were not allowed back into the building for a couple of days, but all they had to stand up as barriers were road cones. Luckily, they're finally moving to a location that isn't just obscure and secure, but armored, as I hear their Mountain View, CA location is.
For your security, this post has been encrypted with ROT-13, twice.
If this building were destroyed by a nuclear weapon, what would be the impact on the Internet?
you brought their server to a crawl by posting that...
and im not sure which is worse to look at... the goatse man, or rhonda...
The temple from Tron?
Approch, Program, and speak to your User...
Never answer an anonymous letter. - Yogi Berra
This story is news, but I kept expecting some point of contention in the article, rather than some musings on decorating schemes that were compared to clip art.
I found my point here:
The root server operators "have no contract with anyone, no guarantee of level of service, they could turn [the root servers] off tomorrow with no consequences at all because they are doing it out of the kindness of their heart," said Internet consultant Ambler. "ICANN needs contracts with the root server operators that specify minimum levels of service and minimum levels of security and the root servers need to be paid for that," he said.
Why is it so confusing to imagine that (a) People do like to do things out of the "kindness" of their collective hearts, and (b) security is not always "secured" by either contracts or money? I understand the legal protections associated with contracts, but I think there's a chance that the root server operator system, as it stands, could alternatively be viewed as something successful - something, much like the open source software movement, that works, not because of contracts or restrictive covenants, but because people enjoy contributing to something useful for their own and others' use.
I guess amazon.com which went public in 1997 must have been frequented only be researches and nerds for the first 5 years of operation.
Digex, along with other major hosting and co-lo facilities, has had these kinds of systems in place for their datacenters for many a year. And yeah, most of them look like very non-descript office buildings - a great many I've seen are in warehouse-style industrial complexes, far off the beaten path of regular office space and retail properties.
You have to wonder if they're a little overboard, though; the military doesn't typically have checks that secure to get into specific rooms - not even TS/SCI environments. Though, to be fair, the military certainly has an edge on physical security.
I guess if you're really concerned about your data being physically secure, you could always co-lo out at Sealand, too.
Back in the good old days, if you had a recent copy of hosts.txt all this was irrelevant :-).
But it's been most of a decade since just anyone could download it.
If you really wanted to hide it, disguise the building as a whore house next door to a police station.
The hookers and the johns could really be Verisign employees running the root server.
In case a real customer showed up and was unfazed by the police station next door, tell him that most of the girls are at the doctors office for their tuberculosis test and the rest are being treated for various venereal diseases.
Or you could disguise it as a crack house. The neighbors would assume that everyone running around with machine guns were drug smugglers.
Or just disguise it as a police station. When someone comes in seeking assistance, tell them "We don't handle those kind of cases any more."
I'd hate to think the internet depends on SCO UnixWare running on an old 486 ;)
Jonathan
I'd like to see some statistics on how many people attempt to invade/evade the physical security checks at Netsol's NOC that require and necessitate facilties on that level. The same goes for most any datacenter - your physical security is awesome, but why?
:)
Aren't most attacks against servers launched over that intarweb thing?
I can't recall the last time someone tried to suicide bomb a root server.
I can only hope that their NOC has multiple fibers coming to the building and that those fibers aren't in the same trench.
The other potential source for a single-point of failure is the OS that the root server uses. If Verisign uses any kind of monoculture, they will not be as secure as we might hope. A hacker or botched OS patch could hose the thing.
Two wrongs don't make a right, but three lefts do.
Bah! That's nothing. You need to traverse a gauntlet of obsolete motherboards, dead power supplies, empty CD cases and soda cans as well as a floor mined with tiny machine screws to get to my NOC. That's assuming you got past my wife at the front door.
Nope, VeriSign was never in Palo Alto. It was dotCom era, rents in Palo Alto were way high by that time. VeriSign started in Redwood Shores and then moved to Mountain View. These days they own the old Netscape campus.
The operations center is another matter, those are in unmarked buildings at several locations. If you look at some of the displays of root server locations you will see blobs in the San Francisco and Washington D.C. areas. Well duhh! Who would have guessed that the DNS servers would be so close physically to MAE West and MAE East?
The Circle ID stories are both slashdotted. So we can't hear if Karl and co are saying 'nah, we don't need high bandwidth roots capable of a good slashdotting' which if they were would be somewhat ironic.
The point that the article does not really mention is that at the moment running the DNS roots is done on a voluntary basis. ICANN is getting a free ride here. After the DDoS event in 2002 it was clear that 1) the roots were a major target 2) There was a big difference in the quality of service.
Given the importance of the roots shouldn't we actually invest something so the people running them can afford to do the job well? VeriSign can afford to run its systems the way it does because it has revenue from other sources. How do you justify the cost of a high end four way server to be dedicated to root ops if you are a non-profit? ICANN could at least pay for hardware and bandwidth.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
The design documentation of the Internet is globally available... wait for it.. on the Internet!
If you examine it, you will notice that
a) DNS is not part of the original design
b) as designed, it WON'T survive a nuke
c) nobody intended it to.
What it *was* designed for was a limited fault tolerance - based on the idea that phone companies suck and the guy that runs the next node is an idiot who can't be trusted to tie his own shoes.
Turns out they were right about those last two points, incidentally.
I'll bite.
The Domain Name System works by sending out a verified master list to other servers on a graduated time scale. This way no one, two, or twelve servers gets nailed with lookups from THE ENTIRE INTERNET....
Those Primary and Secondary DNS number you're asked to enter when doing network setups are for the partial copies stored on the (insert any number of levels) nth server from the master.
If it can't find the match on one of those, it'll ask others, until a timeout occurs.
There is nothing to stop you from setting up your own DNS, if you're willing to donate the time and hardware to the cause.
all you need to access it is a bomb, or, pretty much anything that explodes spectacularly.
I'm glad it's down. Good on her for getting it done. Of course, the picture will live on elsewhere but at least she did what she could.
Just because you can post something doesn't mean you should post something. Redeeming value of that picture? None.
Yeah, baby, I'm using my real nick...unlike all the cowards who will doubtlessly reply.
Many of the root server operators have deployed mirrors of their machines using "anycast".
.com, .net, .org, and .in-addr.arpa. The roots are heavily cached and easily replicated. It isn't quite so easy to handle a loss of connectivity to the big top level domain servers.
Anycast is a way of using routing information so that a single IP address appears at many locations on the net. Packets flowing to an anycast IP address tend to go to the nearest instance of such an address.
Physical security isn't the risk that the roots face - the issue is damaged connectivity to those 13 addresses on which those root machines are to be found.
As I mentioned in my note on Circle-ID, the biggest risk isn't to root servers but rather to the set of servers that deliver
I've suggested a "DNS on a CDROM" (which I guess should be updated to "DNS on a DVD") in which all the stuff needed to get a local but limited DNS running in cases when a community has been cut off from the main body of DNS services.
I've had a few guys point it out to me before. Many DC / Dulles Toll Road-types know where it is.
Now, there are other buildings in DC that's are much more cool. Like the one on the Toll Road with green "windows" that are merely for appearances as the entire building is solid concrete. Or the stuff in Crystal City that is bathed in electronic white noise to prevent eavesdropping.
Unless the NOC was ordered at this place, I'm not impressed.
Hate me!
ROOT-A /--
--\
)(
--/ \--
20 MBs
but here is the /. thread on this facility from March, 2002. http://slashdot.org/article.pl?sid=02/03/29/144922 8&mode=thread&tid=95
To be honest it is kind of embarassing that I immediately thought- "I just saw something just like this on slashdot not long ago" to find out it was almost 2 years ago. I didn't look at the new article close enough to see if there were any big differences over the years. To be honest the articles are spooky similar. Hmmmmm.
It's hard to believe that's how Micronians are made. Why don't we see it right now by having you both kiss one another?
Visitors are "tagged and bagged" and made to sign de facto non-disclosure agreements before being lead to an elevator.
"Tagged and bagged"? Really? Visitors are killed, inventoried, and their remains placed into a body bag? And then they're asked to sign an NDA?
That really is tight security!
- Distributing the database to major servers (at least one machine from each of the 13 often-virtual root servers, plus the master DNS servers at the Tier 1 ISPs, the CCTLD servers, and some small number of other sites
- Answering DNS queries from the major servers
- Answering DNS queries from any random machine on the Internet
The system becomes performance-critical to lots of people because too many machines send queries to the root servers (or theThe root zone itself is probably under 10KB of data that doesn't change every day - if you provide a separate server for zone transfers and let 1000 other DNS servers have access to it (firewalled to prevent any other IP traffic), that's about half an hour on a 56kbps modem. Remember that all it's doing is answering good questions like "Where are .com's name servers?" "Where are .za's name servers", bad questions like "Where are .example,com's name servers?", "Where is 10.in-addr.arpa?" and ugly questions like "Where is Ping of Death?". Let the major servers handle most of the work, absorb the ugly packets and do some queries for bad packets, and let the general public query those anycast machines - they should be querying their ISPs' servers, or their upstreams', which cache the real information, and even when their queries aren't bogus, they shouldn't be blocking the internet-stability-critical traffic.
The .net, .com, and .org domains are a similar problem, except of course they aren't served by the root servers. The zones are much bigger, a few gigabytes size, but probably only 10% of it changes in any given month, or 99.9999% of the existing domains, which ought to be enough to call the Internet stable, using about 1 Mbps (10GB * 1%/day * 8 bits/byte / 24*60*60 ), and again, keep the public query traffic separate from the zone transfer traffic, and maybe offer a third set of DNS servers to answer queries from the big ISPs to handle things like newly created domain names. The reason to keep that kind of query traffic separate is to avoid attacks like "query bogus00001.com" "query bogus00002.com" ... etc.
Obvious flame-attracting discussion points:
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
According to an October 2002 study, 98% of queries to the F Root Server (and therefore probably to the other root servers) are unnecessary. Either they're duplicates (75%) or they're for bogus TLDs (.localhost, .elvis, .corp, etc.) or they're in-addr.arpa queries for RFC1918 addresses, or they're some other bogus query, and they should have been served out of cache or handled by some ISP's DNS instead of bothering the roots. Maybe the A Root has some important functions, but they aren't what it spends its time on. And 50% of the queries come from about 220 servers - they should either be caching responses, or be shuffled off to some server that handles them (I guess anycast will help with this...) as well as cleaning up their act if they're broken, which some of them are.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Back in the good old days when her serene highness the Dalai Lauren worked there and Dave Holtzman was still VP I took the e-ticket tour. The facility is in a nondescript industrial mall a few miles from the NSI mothership.
"oh, you'll want to see this"
"what is it"
"A-ROOT"
"THAT tiny little thing?"
"Yup. Go ahead and touch it, everybody that comes here wants to do that. See where the paint has worn off the case?".
"Uh, ok"
"You use this thing Dave"
"Nah, I download the root zone from you".
"Cool, for that you can buy me lunch".
"Good idea. Thai okay?"
NSI was fun once and there's lots of good stories. When the FNCAC made the NSF tell NSI to start charging for domain names none of the freaks working at NSI could believe you could charge for this and lots of checks were just pinned up to a bulletin board in a "wait and see" holding pattern for a few months. There weren't so many domains back then.
Karl Aurbach also downloads the root zone from me and you should too. Or use OpenNIC's root or even *cough*ICANNs*cough* (ftp://internic.net/domain/root.zone.gz, or any root.zone you want but if you know what's good for you you won't rely any anybody but yourself to serve up the root zone so your computer can find pointers to the various TLD servers: primary the root for yourself and don't worry about DOS attacks on other peoples computers taking your machine off the air.
That really was the dumbest part of the change from hosts.txt to the DNS - it changed the paradigm from your computer knowing where everything was to making your computer rely on the "." zone to be able to find the computers that know where all names can be found and there's really no reason for it.
Certainly it does not scale for everybody to grab a copy of the root from one place, and Dan Bernstein has suggested a cryptographically signed root be distributed via usenet. To this end I've created news:alt.root.orsc and will begin doing just that this quarter.
Need Mercedes parts ?
At the beginning of the article:
... VeriSign isn't shy about touting the $150 million it has invested in various security measures.
...
A bit later
"Can you pull that door closed? I didn't hear it click," he asks of the person standing nearest to the first door.
"Click."
Sheesh, for $150 million you'd think a robot would double check the door for them.
(Score:-1, Wrong)
You raise a number of really good points.
.biz there was already an operational .biz. I had some machines that were using the ICANN version and some using the pre-existing version. And yes, there were some confusions. The point to draw is not that the idea is thereby necessarily bad, but rather that consistency is important. But DNS never operates with perfect consistency - for example for years Taiwan (.tw) was operating with its own roots that were hacked into the system in a really strange way. I was the only one who noticed. (The situation was corrected last year after we [ICANN] pointed it out to them - it turned out that it was an experiment that they forgot to turn off.)
.com). Well, the folks at Verisign, much as we like to dislike 'em, are smart and have more than a lot of "clue". Yes, for a while two root servers sat in the same room, but things like that are past history. No, I do not know the actual locations (I intentionally chose not to use my position at ICANN to try to learn that information), but I can assure you that the concept of physical separation has become an article of faith. And with the increasing use of anycast, replica servers are getting easier to deploy.
Let's see if I can deal with at least some of 'em.
First, regarding use of data on a CD/DVD to recover locally - this is for use when a community is cut off, as happened here in Santa Cruz in 1989 when we have a medium sized earthquke. There were enough folks here with enough gear that we could rebuild a local, usable net to assist with recovery even though the links over the mountain to the rest of the world took a while to be restorred. In that situation the folks who risked any bad information that might be introduced were those who knowingly changed the hints addresses, and if they knew enough to do that they also probably knew enough to clear things out (i.e. reboot named) when they changed the hints file back to the global values.
I've actually experienced the introduction of bad DNS data. Before ICANN permitted its version of
As for the location of the big TLD servers (such as those for
As for the reputation value of an attack - yup, some perverse folks would feel their reputations enhanced if they brought down DNS. And for that reason I feel that all the armor plating is good. But we need to recognize the gaps in that armor, which are things like routing or mindless belief that there must be one catholic system of DNS root servers. And we have to remember that a lot of bad things are caused by mother nature and Murphy's law rather than folks who have abandoned reasoned discourse and moved to techno-mayhem.
The equivalent for .com is obviously much bigger - I think there are ~35 million names (maybe that includes .net). But that's still about 5GB of highly compressible data - probably about 1GB if you sort it appropriately first. That's about the size of a Linux distribution - use BitTorrent. That's about 3 hours on a T1 line, and most of the people who need it are ISPs anyway (so it's about 10 minutes on a T3.) Probably doesn't change by more than 20% a month, or 1% a day.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks