Building an IT Infrastructure Today vs. 10 Years Ago

rjupstate sends an article comparing how an IT infrastructure would be built today compared to one built a decade ago. "Easily the biggest (and most expensive) task was connecting all the facilities together. Most of the residential facilities had just a couple of PCs in the staff office and one PC for clients to use. Larger programs that shared office space also shared a network resources and server space. There was, however, no connectivity between each site -- something my team resolved with a mix of solutions including site-to-site VPN. This made centralizing all other resources possible and it was the foundation for every other project that we took on. While you could argue this is still a core need today, there's also a compelling argument that it isn't. The residential facilities had very modest computing needs -- entering case notes, maintaining log books, documenting medication adherence, and reviewing or updating treatment plans. It's easy to contemplate these tasks being accomplished completely from a smartphone or tablet rather than a desktop PC." How has your approach (or your IT department's approach) changed in the past ten years?

Comments

You don't build it

[dyingtolive]

You just put it all in the cloud brah. My boss assured me it'd be okay and he got his MBA from

Re:You don't build it

[Archangel Michael]

The cloud is fine and dandy until Microsoft Azure is unreachable for several hours ... again ...

http://www.theregister.co.uk/2013/11/21/azure_blips_offline_again/

expect to allow intrusive oversight

[Anonymous Coward]

not much else has changed

Typing will always be better on a PC

[tepples]

The residential facilities had very modest computing needs -- entering case notes, maintaining log books, documenting medication adherence, and reviewing or updating treatment plans. It's easy to contemplate these tasks being accomplished completely from a smartphone or tablet rather than a desktop PC.

And by the time you've paired an external keyboard in order to key in all that stuff, you might as well just use a laptop PC.

In addition, some cloud solutions make dedicated desktop application suites or specific configurations unnecessary today. Browser-based options or virtual desktops have added appeal in health organizations because data is less likely to be stored locally on a device.

That'd double an organization's spending on operating system licenses because a Terminal Server CAL for Windows Server costs about as much as a retail copy of Windows for the client.

Same now as it was back then . . .

[mmell]

FIrst, consult the stars to ensure that the project will be done at the right time. Then, after arranging the entrails of a rooster in a circle under the full moon cast the bones into the pit and invoke the augury which will allow me to see the hardware, software stack, network stack and end-user facilities all magically "come together".

Really - I'm pretty sure my boss in the Midwest thought that was how I did it. Why would I change success?

Re:Same now as it was back then . . .

[Anonymous Coward]

That's good, but reality is more like...

Determine the deadline, if at all possible, don't consult anyone with experience building infrastructure.

Force committal to the deadline, preferably with hints of performance review impact.

Ensure purchasing compliance via your internal systems, which minimally take up 30% to 40% of the remaining deadline.

Leave the equipment locked in a storage room for a week, just to make sure. Or, have an overworked department be responsible for "moving" it, that's about a week anyway.

Put enormous amounts of pressure on the workers once the equipment arrives. Get your money's work, make them sweat.

When it's obvious they can't achieve a working solution in 30% (due to other blockers) of the allotted time, slip the schedule a month three days before the due date; because, it isn't really needed until six months from now.

That's how it is done today. No wonder people want to rush to the cloud.

actualy 10 years ago

[mjwalshe]

You'd be doing what we do now except maybe some types of networks that use leaf and spine rather than a tree design.

Re:HIPAA Privacy Rules

[Joehonkie]

If only there were some way to look this up:

• http://aws.amazon.com/compliance/#hipaa
• https://support.google.com/a/answer/3407054?hl=en

Well.... Quite a bit has happened.

[Chrisje]

We've consolidate all office application servers to 5 data centers, one per continent. Then we've rolled out end-point backup for some 80.000 laptops in the field and some 150.000 more PC's around offices across the world which includes legal hold capabilities. Each country in which we're active has a number of mobile device options for telephony, most of them being Android and Win8 based nowadays since WebOS got killed.

Then we're in the process of building a European infrastructure where we have data centers for managed customer environments in every major market in Europe. I am currently not aware of what's going on in APJ or South America. This is important in Europe however, because managed European customers don't want to see their data end up in the States, and the same goes for those that use our cloud offerings.

physical local IT staff presence in all countries has been minimized to a skeleton crew, not only because of data center consolidation but also because of the formation of a global IT helpdesk in low cost countries, and the rise of self-service portals.

The plethora of databases we had internally has been Archived using Application Information Optimizer for structured data archiving. We are our own biggest reference customer in this regard. On top of that we've beefed up our VPN access portals across the world so as to accommodate road warriors logging in from diverse locations.

Lastly, we use our own Records Management software suite to generate 8.000.000. unique records per day. These are archived for a particular retention period (7 years I believe) for auditing purposes.

Re:Well.... Quite a bit has happened.

[cusco]

In the field of physical security, I've seen customers with 10 independent access control systems scattered around their various facilities condense into a single centralized and monitored system. Access control system panels used to be connected serially to a "server" which was a cast-off desktop PC shoved under a janitor's desk, but now are actual servers in server rooms, monitored and backed up by IT staff, communicating with panels that might be on the other side of the planet.

Security video was analog cameras connected with R-59 coax cable and plugged directly into a DVR or (gods help us) a VCR. The only way to view live or recorded video was to go to the site where the cameras were physically located, and with many systems the act of viewing recorded video would stop the system from recording until you were done. Recording capacity was measured in days or sometimes hours, and casinos had people whose jobs were to just walk from one VCR to another changing tapes all day. Failures to record were more common than actually capturing an incident. Today's IP cameras record to NVRs that have terabytes of capacity in RAID 10 arrays, often with redundant servers, sometimes recording across the WAN or across the Internet to data centers in other countries.

Integration between card readers, alarm points, monitoring points and cameras are common today. A motion detector set off in a data closet in Mumbai may raise an alarm and pop up video on a guard's computer in Dublin while sending an email with a video clip to smart phones in Los Angles and Sydney. The guard may dispatch local staff in Mumbai on his handheld phone across the IP telephony system, and they may reply on their walkie-talkie. Access to the site might be granted by staff in the SOC in Phoenix, and repair crews may be dispatched by the Facilities department in Houston.

In the Physical Security industry the future is now, and it's exciting.

Re:Well.... Quite a bit has happened.

[Chrisje]

Yes they are. I work in the Information Management software division as a pre-sales, and I'm pretty much paid to tell subsets of the above to customers.

- We are our own reference customer for Connected backup for end-points.
- We are our own reference customer for TRIM, now known as HP Records Manager 8.0
- We are our own reference customer for Database Archiving, now known as HP Application Information Optimiser

So all of that is publicly available in white-papers and case-studies.

The fact that we're building a public cloud infrastructure per country in Europe is also very much not a secret. If we want to get or retain EU based cloud customers, we need to be able to guarantee that their data remains their data and that it won't fall prey to third parties, chiefly amongst which the US government.

In terms of data center consolidation and cost savings associated with that, the strategy internal IT is following is largely in line with the Data Center concept we sell as Converged Infrastructure, Cloud System Matrix and Cloud System One.

Moreover our external web presence is run on the newly launched project Moonshot, in which you can currently cram some 45 servers in 5U rack space, which will soon get uplifted to 180 servers in 5U rack space.

All of this is a clean cut case of eating your own cooking, and then using that fact to market the underlying technologies.

So yes, I am very much convinced HP is comfortable with me sharing this publicly.

Virtualization

[Jawnn]

For good or bad (and yes, there's some of both), virtualization is the single biggest change. It is central to our infrastructure. It drives many, if not most, of our other infrastructure design decisions. I could write paragraphs on the importance of integration and interoperability when it comes to (for example) storage or networking, but let it suffice to say that it is a markedly different landscape than that of 2003.

Re:Virtualization

[GodfatherofSoul]

Amen to this. I'd say it's the single most important change for network admins in the past 15 years. Our server farm went from a 7 foot stack of pizza boxes with disparate hardware and OSs that we were paying oodles to be parked in a server farm; to one public VM host on the cloud and one private VM host running on my boss's desktop.

Re:Not much difference

[mlts]

In 2003, Sarbanes-Oxley was passed, forcing companies to have to buy SANs just to stick E-mail for long term storage/archiving.

For the most part, things have been fairly static, except with new buzzwords and somewhat new concepts. A few things that have changed:

1: Converged SAN fabric. Rather than have a FC switch and a network switch, people are moving to FCoE or just going back to tried and true iSCSI which doesn't require one to fuss around with zoning and such.

2: Deduplication. We had VMs in '03, but now, whole infrastructures use that, so having disk images on a partition where only one image is stored and only diffs are stored for other machines saves a lot of space.

3: RAID 6 becomes necessary. I/O hasn't gone up as much as other things, so the time it takes to rebuild a blown disk is pretty big. So, RAID 6 becomes a must so degraded volumes rebuild.

4: People stop using tape and go with replication and more piles of hard disks for archiving. Loosely coupled SAN storage in a hot recovery center becomes a common practice to ensure SAN data is backed up... or at least accessible.

5: VMs use SAN snapshots for virus scanning. A rootkit can hide in memory, but any footprints on the disk will be found by the SAN controller running AV software and can be automatically rolled back.

6: We went from E-mailed Trojans, macro viruses, and attacks on firewalls and unprotected machines to having the Web browser being the main point of attack for malware intrusion. It has been stated on /. that ad servers have become instrumental in widespread infections.

7: The average desktop computer finally has separate user/admin access contexts. Before Vista, this was one and the same in Windows, allowing something to pwn a box quite easily.

8: The OS now has additional safeguards in place, be it SELinux, Window's Low security tokens, or otherwise. This way, something taking over a Web browser may not be able to seize a user's access context as easily.

9: BYOD has become an issue. Ten years ago, people fawned over RAZR-type devices and an IT person had a Bat Belt of devices, be it the digital camera, MP3 player, the PDA, the pager, the cellphone, and the Blackberry for messaging. Around -05, Windows Mobile merged all of this into one device, and '07 brought us the iPhone which made the masses desire one device, not a belt full.

10: Tablets went from embedded devices to on desktops and big media consumption items.

11: Music piracy was rampant, so one threat was people adding unexpected "functionality" to DMZ servers by having them run P2P functionality (AudioGalaxy, eMule, etc.)

12: We did not have to have a Windows activation infrastructure and fabric in place, where machines had to have some internal access to a KMS box to keep running. XP and Windows Server 2003 had volume editions which once handed a key would update and were happy for good.

13: UNIX sendmail was often used for mail before virtually everyone switched over wholesale to Exchange.

14: Hard disk encryption was fairly rare. You had to find a utility like SafeBoot or use loopback encrypted partitions on the Linux side for data protection. This was after the NGTCB/Palladium fiasco, so TPM chips were not mainstream.

15: One still bought discrete hardware for hosts, because VMs were present for devs, but not really "earned their bones" in production. So, you would see plenty of 2-3U racks with SCSI drives in them for drive arrays.

Things that have stayed the same, ironically enough:

1: Bandwidth on the WAN. The big changes came and went after initial offerings of cable and DSL. After that, bandwidth costs pretty much have not changed, except for more fees added.

2: Physical security. Other than the HID card and maybe the guard at the desk, data center physical security has not changed much. Some places might offer a fingerprint or iris scanner, but nothing new there that wasn't around in 2003. Only major di

Re:HIPAA Privacy Rules

[PlusFiveTroll]

The effective compliance date of the Privacy Rule was April 14, 2003 with a one-year extension for certain "small plans"

Or pretty much 10 years ago.

abstraction

[CAIMLAS]

The biggest difference in the past 10 years is that everything has been abstracted and there's less time spent dealing with trivial, repetitive things for deployments and upkeep. We support more users now, per administrator, than we did back then by many a massive amplitude.

No more clickclickclick for various installations on Windows, for instance. No more janky bullshit to have to deal with for proprietary RAID controllers and lengthy offline resilvers. These things have been abstracted in the name of efficiency and the build requirements of cloud/cluster/virtualization/hosting environments.

We also have a lot more shit to take care of than we did a decade ago. Many of the same systems running 10 years ago are still running - except they've been upgraded and virtualized.

Instead of many standalone systems, most (good) environments at least have a modicum of proper capacity and scaling engineering that's taken place. Equipment is more reliable, and as such, there's more acceptable cyclomatic complexity allowed: we have complex SAN systems and clustered virtualization systems on which many of these legacy applications sit, as well as many others.

This also makes our actual problems much more difficult to solve, such as those relating to performance. There are fewer errors but more vague symptoms. We can't just be familiar with performance in a certain context, we have to know how the whole ecosystem will interact when changing timing on a single ethernet device.

Unfortunately, most people are neither broad or deep enough to handle this kind of sysadmin work, so much of the 'hard work' gets done by support vendors. This is in no small part due to in-house IT staffing budgets being marginal compared to what they were a decade ago, with fewer people at lower overall skill levels. Chances are that the majority of the people doing the work today are the same ones who did it a decade ago, in many locations, simply due to the burden of spinning up to the level required to get the work done. In other places, environments simply limp by simply on the veracity of many cheap systems being able to be thrown at a complex problem, overpowering it with processing and storage which was almost unheard of even 5 years ago.

The most obnoxious thing which has NOT changed in the past decade is obscenely long boot times. Do I really need to wait 20 minutes still for a system to POST sufficiently to get to my bootloader? Really, IBM, REALLY?!

Re:Not much difference

[CAIMLAS]

Another big difference which relates to the list you mentioned: almost nobody runs their own in-house mail anymore. It's too expensive (in time and experience, mostly) to maintain efficiently and effectively, in no small part due to spam. Even larger organizations have decided it's not worth the headache.

If there is in-house hosting of mail, it's due to complex requirements and the headache that migration would be to another system. Many of these have also put in place either Google or Microsoft frontend filtering to their mail systems.

Ten years? Bah, humbug.

[Slartibartfast]

10 years ago really wasn't that big a deal. By 2003, VPN (IPSec and OpenVPN) was fairly robust, and widely supported. PPTP was on the way out for being insecure. Internet was most everywhere, and at decent-if-not-great throughput. Go back five or ten years before *that*, and things were much more difficult: connectivity was almost always over a modem; remote offices *might* be on a BRI ISDN connection (128 kb/s), probably using some sort of on-demand technology to avoid being billed out the wazoo due to US telcos doing this bizarre, per-channel surcharge for ISDN. PPP was finally supplanting (the oh, so evil) SLIP, which made things better, assuming your OS even supported TCP/IP, which was not yet clearly the victor -- leading to multiple stacks to include MS and Novell protocols.

All in all, 2003 was about when things were finally getting pretty good. Leading up to 2000 had been a tough row to how. And let's just not even go before that -- a mishmash of TCP/IP, SNA, SAA, 3270, RS-232, VT100, completely incompatible e-mail protocols, network protocol bridges, massive routing tables for SAPpy, stupid protocols... a 100% nightmare. Very, very glad to have left those days behind.

Re:Not much difference

[cusco]

I work in physical security, so will mention some changes that your site may not have implemented but which many larger sites have.

1) Granularity of access - Formerly if you had an access card it got you into the data center and from there you had free range. Today the data center is (or should be) compartmentalized and access to each area dependent on need.

2) Rack Access - There are now several brands of hardware that control technicians' access to individual racks, including front and/or rear rack door.

3) Video Monitoring - Data centers are now full of cameras, often linked to readers or door contacts on individual racks (especially Global Payment System racks).

4) Facility Monitoring - Temperature, power status, UPS state, water sensors, smoke detectors, etc. all come into the alarm system, where they are monitored by guard staff.

5) Computing Pods - Access to container-based computing centers has not only changed power and cooling management but access control as well.

6) Key Tracking - Systems like Traka and Keywatcher can be integrated into the access control system so that hard keys for individual racks/room/pods/equipment can be checked in and out by strict controls.

7) Procedures - Data center staff now have (or should have) documented procedures to follow to grant/allow access, generate/revoke cards, tracking and automated expiration for temporary access cards, etc.

Re:Well...

[AK Marc]

2000 managed all sorts of problems with hardware. Drivers lagged, so USB support was crap. Blue screens for plugging in a USB device wasn't just saved for press conferences. 2000 was good so long as all you did was Office. For the marketing department, they all went back to macs. Where they had a variety of monitor sizes and commercial editing packages that Just Worked. Ah, making fun of my slashdot number, when you don't even have one. 2000 was "supposed to be" the first converged OS (95/NT), but failed because it wasn't home-user friendly (not just games). XP managed it, and was really an SP of 2000, but with new OS name, pricing, and marketing.

Re:Not much difference

[hairyfish]

almost nobody runs their own in-house mail anymore.

My experience is different from yours. I work for an IT service consultancy and we're trying to push a lot of customers to cloud based email but they're all sticking to their guns. No-one around here likes the cloud for key business functions, and the NSA press is keeping them firmly entrenched in their views. For most companies (less than 1000 users) Exchange is trivial to setup and maintain, and can be supported part-time or by outsourced support. Over 1000 users then you have a big enough IT team to look after it properly.