Businesses Moving From Amazon's Cloud To Build Their Own

itwbennett writes "There are rumblings around this week's OpenStack conference that companies are moving away from AWS, ready to ditch their training wheels and build their own private clouds. Inbound marketing services company HubSpot is the latest to announce that it's shifting workloads off AWS, citing problems with 'zombie servers,' unused servers that the company was paying for. Others that are leaving point to 'business issues,' like tightening the reins on developers who turned to the cloud without permission."

Comments

Nor surprising and won't matter.

[serviscope_minor]

It doesn't surprise me and I don't think it will matter much.

Amazon is not particularly cheap. If you host your own, even with power, cooling and hardware, the payback time is about 4 to 6 months.

If you have a lot of load then it is going to be cheaper to host it yourself, so it's worth doing for big companies.

With Amazon of course you can start as a one man band and still have potential to grow without it getting painful from an administrative point of view.

Re:Nor surprising and won't matter.

[CastrTroy]

The only case where it really made sense was when you had extremely variable load. It's nice for scientists that need to rent 100 computers for use with one project, but if you're going to be using the same resources on a day-to-day basis, then it makes much more financial sense to just own your own hardware, and rent space in an existing data center. It also makes sense if you use less than a whole server in resources, but VPS was already filling that need quite well before Amazon came along.

Re:Nor surprising and won't matter.

[Dancindan84]

Exactly. This is no different than anything else. Companies reach a certain point and hosted X becomes less viable than doing their own solution depending on the pricing model and service level provided. Email, website, call center, payment processing...

Re:Nor surprising and won't matter.

[Albanach]

Does your payback time include any costs for hardware administration/maintenance?

It costs money through time in sourcing and installing hardware. It costs you to keep spare equipment that can take over in the event of hardware failure. These all need factored in. It's common when buying a box to overspec, anticipating future growth, whereas on a service like Amazon you can click and upgrade your hardware capacity when you need it.

I think there are also fewer well managed co-location sites that have good connectivity and who are interested in customers needing only a few U of space.

I'm not saying that there aren't advantages to hosting yourself but in my experience there are lots of costs saved by using cloud based servers that often get overlooked from the accounting equations.

Re:Nor surprising and won't matter.

[serviscope_minor]

The only case where it really made sense was when you had extremely variable load.

Indeed, or if you're expecting to scale. The thing is, as you scale up, you can always move the baseload to dedicated servers and just do the variable part on Amazon.

Re:Nor surprising and won't matter.

[thereitis]

If you're just using Amazon for compute power then perhaps, but then you've got no geographic redundancy with that single data center. Whether it's worth rolling your own solution really depends on your needs (lead time, uptime requirements, budget, IT skill/availability, etc).

Re:Nor surprising and won't matter.

[CastrTroy]

I guess it really depends on what business you are in though. Take for instance a large company like Ford (picked because they aren't a computer/technology/web based, but large company). Their expertise has nothing to do with computers. Now, the question becomes, would it be cheaper for an organization of this size to host their own email? Most likely it would. But the real question is, do they want to devote any corporate time to even dealing with this kind of thing. Basically they would have to have a whole new division added on to their company to handle IT management, and they'd have all the fun stuff that goes along with it. It makes much more sense for someone like Amazon to host their own email because they already have a bunch of people managing servers anyway, so adding a few more servers isn't getting into uncharted territory. I mean, Ford probably uses enough keyboards that it might make sense to have their own keyboard factory. But that doesn't mean its a good business decision or that it's something they want their company to be doing.

Re:Nor surprising and won't matter.

[Anonymous Coward]

we use Azure for a lot of things in my particular department because it helps us bypass our IT department. Sometimes we need to set stuff up really fast and only have it last for a short amount of time. It takes our IT department about a week to open ports on our firewall or map a machine to an IP....when we have 2 days to get something working this doesn't work. As far as cost goes...it isn't all that much more expensive than handling the hardware ourselves. I can also, on the fly, scale things up as I need to. It's a lot easier than buying ram...shutting down the server...getting someone to put the RAM in since I don't have access to NOC. Works perfectly for our needs. I can also run a ton of test web sites for free with Azure and then move them to production as I need to. If they stay under a certain barrier then I don't get charged for them at all since the first like 5 gig of traffic is free once I move it to a dedicated resource. Trying to do this traditionally wouldn't work for us at all...and would make things even worse.

Your mileage may vary.

Re:Nor surprising and won't matter.

[Dancindan84]

The thing is, when a company reaches a certain size they likely have a enough computer infrastructure to have an IT department anyway, even if they aren't an IT company. With your example of Ford, they have offices for managers, sales etc. All of those people likely have desktop computers, so they likely have dedicated desktop support. Additionally they probably have some kind of centralized authentication like active directory, which means they'll need a server and some sort of sys admin/IT infrastructure already. They likely wouldn't be adding an IT division in order to host their own email, they'd be adding an email server/management to the load of the existing IT department, which is obviously not as big an upfront overhead cost, making it more attractive.

Re:Nor surprising and won't matter.

[serviscope_minor]

Take for instance a large company like Ford (picked because they aren't a computer/technology/web based, but large company). Their expertise has nothing to do with computers.

Are you sure about that?

A large company must have many many areas of expertise. Obviously their goal is to make cars. But have expertise in cars, large scale manugacturing, logistics, marketing, engineering, anything required to support engineering including simulation running on supercomputers, human resources and probably a whole bunch I haven't thought of.

The point is that many of them will involve computers to a large degree, so although a company like Ford makes no money with computers per-se every area of their operation will involve computer systems. As a result they will have a huge amount of computer expertise.

Re:Nor surprising and won't matter.

[Jawnn]

It If you host your own, even with power, cooling and hardware, the payback time is about 4 to 6 months.

That depends a great deal on the scale and availability demands placed upon your infrastructure. One can deploy a "private cloud" on one or two cast-off PC's, but that will be little more than a toy. If you want to support a serious deployment (dozens or hundreds of nodes) with anything approaching usable performance, you're going to be investing in some serious network and shared storage hardware, not to mention host servers. Want HA? Still more (bigger) bucks. Still, it doesn't take much to make those investments pay. I just think that 4-6 months is a bit too optimistic in all but the most trivial installations. YMMV, of course, but 12-36 months is more like it.

Re:Nor surprising and won't matter.

[alen]

neither does amazon unless you pay them a lot more $$$

Re:Nor surprising and won't matter.

[drinkypoo]

The defining factor is whether you can keep more than one IT guy busy full time. If you can, then you hire at least two, one senior and one junior to at least fight fires when he's sick. If you're keeping at least two IT buys busy full time, you're going to be paying for them whether they work for you or not, but if they do work for you then you can fire them, so you have some control over what they do. If they'll just be placed with someone else if you don't like them, they're not going to work as hard for you. You need as much control as you can get over your own IT department. It's daft to contract out anything so critical when you're only adding to the likelihood of leaks and malfeasance.

Re:Nor surprising and won't matter.

[afidel]

Right now AWS compute costs about 2-3X as much as an in-house VM for me given a 5 year lifetime (we buy storage with 5 years support and hosts last 4-5 years with upgrades), it's when you need anything that needs serious storage performance that the ROI time starts to decrease sharply. Where AWS rocks is peak shaving, if you have a workload that only needs a few hours a day of powered on time then it's really easy to justify it, but for your run of the mill corporate IT systems that just kind of chug along and need to be available whenever it's definitely more expensive. The other one I can see is if you want geographic distribution and can't easily use a CDN.

Re:Nor surprising and won't matter.

[AlphaWolf_HK]

It's often the case that you'll just need one, plus a support contract. The support contract will handle any issues that the IT guy might not be able to do on their own, such as speedy hardware replacement.

Flexpod is pretty neat in that regard; it has automatic monitoring that will notify the vendor in the event of a perceived imminent hardware failure. They'll begin the process of sending a technician out with the replacement part in hand often times before the admin is even aware that anything is wrong. Doesn't cost anything when it happens, and because of the HA features you have zero downtime while the hardware is being replaced.

Flexpod (or its competitor Vblock) aint cheap, but you do get what you pay for if you want an in-house HA cloud service.

Re:Nor surprising and won't matter.

[ebno-10db]

Additionally there is the matter of control. You have a lot more of it if you do your own stuff. However, business often mindlessly follows the latest fashions (this explains why the execs get the big bucks - like fashion groupies). For years the trend has been to move everything out-of-house and buy "services". There recently seems to be some trend away from this, because it turns out the people who warned of the problems were often right (what a surprise). It doesn't help that many companies that offer "services" have become little more than overpriced front companies. For example, much of IBM's US operations have become little more than an overpriced front and unnecessary for India Business Machines.

Re:Nor surprising and won't matter.

[7213]

Regarding Ford specifically.

You'd be surprised at the scale of their IT organization (as someone who once worked in Ford's datacenter).

They already have their own 'internal cloud' and have for some time (before 'cloud' was a 'thing'). The only thing different here is internal provisioning processes vs. Amazons credit card & go plan.

The cost of Amazon doesn't make sense, when you already have a pair of tier 1 datacenters and an IT organization more then capable of maintaining it.

Ford already HAS servers that won't be 'clouded' any time soon, so they have every bit as much justification to keep on doing things internally as Amazon would. And doing things for themselves gives them more control & likely better costs.

Re:Nor surprising and won't matter.

[nine-times]

But the real question is, do they want to devote any corporate time to even dealing with this kind of thing. Basically they would have to have a whole new division added on to their company to handle IT management, and they'd have all the fun stuff that goes along with it.

Obviously you've never done this kind of thing before. Ford needs IT management no matter what. Even if they use Amazon for hosting and Gmail for email and whatever else, that decision first implies that you have someone who understands the benefits and drawbacks to hosting your own services vs. going with a hosted service. People who don't understand it think that the benefits/drawbacks are obvious, but that's only because they don't understand it.

Once you've made a decision, you need to choose a vendor. You need someone on your staff to manage the setup process to make sure the solution is meeting business needs. You'll need someone overseeing the vendor for as long as you use them, and you need someone investigating new solutions to know when a superior service is being offered by someone else. Then you need people to help roll out the solution to users, provide training and documentation, and provide desktop support.

A company that size needs dedicated IT resources and management no matter what. Some MBA might think, "Oh, you just outsource it," but unless you have a knowledgeable staff managing the outsourced services, it'll be a disaster. Especially so with something like AWS, where Amazon isn't really providing a deep level of support. AWS isn't exactly a point-and-click solution for the masses.

Re:Nor surprising and won't matter.

[hawguy]

neither does amazon unless you pay them a lot more $$$

Depending on your needs, setting up geographical redundancy with Amazon can be extremely cheap -- if you just want a cold or warm site to fail over to, you don't need to keep your entire infrastructure running at the secondary site, just replicate the data, and then spin up the servers over there when you need to fail over.

That's what my company does - we have about a dozen servers to run our website, but the secondary site has only a couple micro instances to receive data. When we need to failover, we just tell one of those servers to wake up the rest of the infrastructure and update the databases from the snapshots that have already been transferred over, including repointing DNS to the backup site. We could make the failover fully automatic, but are afraid of "split brain syndrome" leading to the failover site taking over when the primary is still fine so it's still a manual process. Our backup site is never more than 15 minutes out of date from production.

This has worked well in testing - we've done some "live" late-night failovers and it's relatively seamless -- since it's so cheap to set up the backup site (essentially we just pay for the cost of storage at the backup site), we're going to set up another region overseas for extra redundancy.

Re:Nor surprising and won't matter.

[hawguy]

Depending on your needs, setting up geographical redundancy with Amazon can be extremely cheap

And history has shown that you pay for what you get.

Right, if you cheap out and pay for a single availability zone in a single region, when that AZ or Region goes down, your site is down.

If you pay for multi-AZ and Multi-region deployments you get much better availability.

Just like Amazon says.

Over the past 2 years, Amazon has been more reliable than the coloc we moved away from, mostly due to the triple (!) disk failure that took out our SAN RAID array - one disk failed, and while we were waiting for the replacement, another disk went down, after we replaced those two, another disk went down while rebuilding the RAID-6 array.

With AWS, an entire region can go offline and we can bring up the backup site on the other side of the country (or, starting next month, we could bring up our Ireland region).

All this for less than half the cost we were paying for the coloc + equipment maintenance.

Re:Nor surprising and won't matter.

[Demonantis]

Your example is so true though. I have seen several professors use the service to complete a simulation over the weekend when they are on a deadline. Its a tool. You use it when it makes sense. The migration away is from inept companies that didn't do a real business case for the procurement, but have woken up now to the real cost.

Re:Nor surprising and won't matter.

[Patch86]

It is the typical "outsourcing cycle". The executives who could show on paper that it would be cheaper to move all hosting to a third party cloud provider have now all moved on from their posts of a couple of years ago, and a bunch of new executives can now show on paper that it will be cheaper to host it in-house.

I've always held the view that outsourcing never makes sense on a large scale- if you're a big company with a lot of hosting needs, it's probably cheaper to hire a team of full time employees and purchase the hardware yourself- after all, Amazon doesn't have a magic machine which makes salaries or hardware cheaper than they are for everyone else. If you a small shop, where your requirements would be for a tiny team and underused hardware, that's when it makes sense to pay the experts to do it for you.

Re:Nor surprising and won't matter.

[thetoadwarrior]

With the outages in their east centre it has shown a lot of people on Amazon don't have redundancy either.

Re:Nor surprising and won't matter.

[Vrtigo1]

If you're talking about virtualizing, then yes it's less expensive to purchase your own equipment and run your own VMs on it than it is to pay for instances in EC2. If you're just talking about purchasing a physical server to run a single application, then it's cheaper to rent from AWS than it is to purchase.

Even for companies that have relatively static compute needs, one area where AWS still really shines is storage. Take Glacier for instance...unlimited storage for a cent per GB per month. $120 a year to store a terabyte of data is pretty darn cheap. Sure you could buy some cheap SATA hard drives, but then you have the overhead of replacing them when they fail, powering them, keeping them cool, etc.

Another area where AWS is really handy is if you're doing a migration or upgrade and you temporarily want to spin up a complete replica of your production environment for test purposes. You can do that in AWS very easily and all you're paying for is the compute resources you're using. Most shops would find that very difficult if not impossible to do on their own hardware in their own datacenter.

Re:Nor surprising and won't matter.

[radarskiy]

"The point is that many of them will involve computers to a large degree, so although a company like Ford makes no money with computers per-se every area of their operation will involve computer systems. As a result they will have a huge amount of computer expertise."

They will consume a huge amount of computing. That does not equal expertise.

For example, you can consume a huge amount of beer. That does not grant you any expertise in brewing. That doesn't even mean you can pick good beers to drink.

Re:Nor surprising and won't matter.

[jp10558]

This is why RAID is not Backup, also, why if I had such a SAN, I'd either have redundant SAN nodes, a hot spare disk, or at least a spare cold disk to swap in on site.

The obvious next step...

[lxs]

...will be to give every user their own personal cloud housed in a box under their desk.
At which point the cycle will begin again.

Re:The obvious next step...

[benf_2004]

...will be to give every user their own personal cloud housed in a box under their desk. At which point the cycle will begin again.

That sounds like a great idea! We can call it a Personal Cloud, or PC for short.

Re:The obvious next step...

[Richard_at_work]

Why is there *always* a snarky comment along these lines whenever someone talks about not using a "public" cloud provider - cloud when talked about in these ways does not mean "someone else owns the hardware", it means "an infrastructure setup which means I do not have to care about the infrastructure when deploying applications", whether that be owned by someone else or an internally provided solution.

The old manner of inhouse application infrastructure involved one or more application server, one or more database server, and the related network and service architecture specifically required to handle redundancy and failover - but the point is, you had to care about that service architecture when dealing with your app! Which server had spare resource to act as a failover for another application (which invariably meant you ended up with two servers allocated for the job anyway, the main and a dedicated backup or two servers which took requests on a round robin manner), which server was not to be used for these purposes, which applications do not live together etc etc.

Today, the goal is to have a "large number of essentially commodity hardware servers" acting on a level which you can forget about for most solutions (there are always going to be situations where heavily tailored hardware solutions will still exist) - where you can treat the hardware as what it should be, a resource to be used and allocated as required.

Virtualisation was the first step (in modern terms, not talking about mainframes here), and cloud takes the aspect of virtualisation several steps down the road.

This has got sod all to do with the "cycle", and everything to do with "computing as a resource".

Re:The obvious next step...

[roman_mir]

Yo Dawg, I heard you like a PC, so I put a PC in your PC so you can PC while you PC.

Re:The obvious next step...

[nojayuk]

Followed shortly thereafter by "thin clients"...

Re:The obvious next step...

[CastrTroy]

With the specs of some of the desktops coming out, they almost are a personal cloud. For about $2000, you can get a machine with 64 GB of RAM, 6 cores (12 if you count hyperthreading), dual SSDs for some speed with redundancy, + 2x1 TB hard drive for large capacity storage, and a pretty decent video card. I remember when $2000 would buy a modest computer, and I'm not even that old.

Re:The obvious next step...

[egcagrac0]

Laugh as you will, but spinning up a VM on each desktop that can contribute to the central processing pool has intrigued me for some time.

Security is a bit of a hangup, and it would have to be cleverly configured to only use the extra cycles... but for some applications, where you just need "a little more oomph", I think it's got merit.

Re:The obvious next step...

[lxs]

I think somebody needs a hug.

Re:The obvious next step...

[drinkypoo]

If you drop the 64GB of RAM to 16GB you can get all that for about $800. That's still loads of headroom.

Re:The obvious next step...

[SimplyGeek]

I agree. It's about increasing the layers of abstraction.

Re:The obvious next step...

[ebno-10db]

cloud when talked about in these ways does not mean "someone else owns the hardware", it means ...

Congratulations, you've passed the Rorschach test!

Inkblots are so messy though. Clouds are pretty and fluffy. You can see anything you want in them. They're perfect for marketing. I detest that old-fashioned anal retentive precise language that used to be popular in technology. This new through-the-looking-glass stuff leads to so much more fun debate.

'When I use a word,' Humpty Dumpty said, in rather a scornful tone, 'it means just what I choose it to mean — neither more nor less.'

Re:The obvious next step...

[Attila Dimedici]

The thing is that the term "cloud computing" derived from the way that network architectures were drawn out for visualization purposes. You drew out a diagram that consisted of boxes of different descriptions representing devices on your company's network and how they connected to one another. Then you had a line that went out to the Internet, which was represented by a cloud because you had no idea what devices your communication passed through and you did not care. I have never really liked that metaphor for putting virtualized computer services on someone else's hardware, but I understand it. If I am hosting virtual machines on your hardware, I don't care how they communicate because if I run into a problem at that level it is your responsibility to fix it. On the other hand, if I am running virtual machines on my hardware and I run into a communication problem because of a hardware level problem I need to know how the various virtual machines connect to each other as well as how the actual hardware connects together if I am going to successfully troubleshoot the problem. That means, if it is running on my hardware it is not running on some vague "cloud" but on specific hardware which is connected in a specific manner. What makes the cloud metaphor useful is that nobody in my organization knows how those machines connect and we don't really care.

Re:The obvious next step...

[Anonymous Coward]

The issue isn't the hardware, the problem is getting a decent internet connection to make your cloud available. If they start releasing apple/windows operating systems with DNS supporting software, default domain name registration and push the 'always on/always connected' everyone could have their own personal cloud, accessible from anywhere.

WebDesktop/Cloud free stuff that works??

[RobertLTux]

So what is around for a SoHo type outfit that wants to do the Self Hosted Cloud thing but can't waste money? EyeOs would work if it
1 was a still being developed project
2 hadn't gone Closed Source

business forecast: cloudy

[OffTheLip]

Businesses don't want to miss the next big thing but like most decisions, time will tell. "I've looked at clouds from both sides now, From up and down, and still somehow It's cloud illusions i recall. I really don't know clouds at all"

Re:business forecast: cloudy

[dkleinsc]

On the upside, it makes it now possible for a business to say "Hey (hey), you (you), get offa my cloud!"

Maybe not completely

[Gripp]

I work at one such company. We recently setup openstack and plan to eventually use it for our production environment. But ec2 will still stay in the picture. Both for services were the end user needs more direct access to the machine and for failover purposes. I just don't know that openstack means the end of ec2.

Computing is going back to its roots

[DoomSprinkles]

Thanks to initiatives such as OpenStack and Hadoop and MapReduce (etc) and the countless contributors who commit to the many projects that allow companies (and individuals with commodity hardware in their garage!) to do these amazing things for cheap, this is all possible and should be the trend! The ROI is well within acceptable margins and well.. it's just fun for us computer geeks! Computing really is moving back to it's roots and we're getting to play with amazing software projects.

Tightening reins on developers?

[cryfreedomlove]

From this article: "like tightening the reins on developers who turned to the cloud without permission"

Let me state this in other words: "Insecure IT guys are afraid for their own jobs if they can't lord it over developers". Seriously, developers working in an API driven cloud just don't need a classic IT organization around to manage servers for them. Cloud is a disruptive threat to classic IT orgs.

Re:Tightening reins on developers?

[h4rr4r]

Let me state this in other words: "Developers know jack and shit about security and business requirements, they will now be able to not meet either of those even faster". Developers are afraid that if the cloud thing does not replace all classic IT they might still have to explain to someone in a meeting why their code falls over all the damn time and admit that maybe more hardware is not always the best answer.

Cloud is what traditional IT orgs manage for you slick. You think it is just developers all the way down?

Re:Tightening reins on developers?

[Sarten-X]

As an ex-developer IT admin in a financial company with history in medical data, let me state this in other words: "IT guys who have to deal with information security are afraid for their own jobs when the company is slapped with a fine for letting confidential information leak out on some cloud service that got hacked, or when the vital business process doesn't work because of a power outage in another country, or when a minor connectivity disruption shuts down every business process everywhere."

Developers want to make things work. IT admins want to make things work reliably, securely, and perpetually. In a well-functioning organization, this means the developers ask the IT guys for whatever they need, and the IT guys either deliver it or give a reason why they can't. Security and reliability requirements aren't always the domain of the developers, but they do weigh into IT's operations. Both groups have to be aware of the others' needs, and work together to meet them. Having a developer undermine IT restrictions is just as bad as having an admin placing unnecessary restrictions.

Re:Tightening reins on developers?

[h4rr4r]

Terrible developer is probably 90% of them. It's a rare day I don't get one in my office asking me some question that a simple look in the log files does not fix.

Caring about security and business needs is there damn jobs. If you don't care enough to do your own job go find other work.

Re:Tightening reins on developers?

[alen]

i've seen crap deployed by developers outside of IT input
it gets put on the oldest and crappiest server just because that's a name they have known for years
no backup gets done on the databases because IT has no idea they exist
half the time there is no DR or any kind of redundancy in case of hardware failure

and when it goes down they run to IT and scream how it's IT's responsibility to make it work

Re:Tightening reins on developers?

[mordred99]

Yes and no. There are many reasons for that. Information Security Laws, and control over costs would be two of the biggest things. Without control over the API, development, etc. of applications, how do you know you are running efficiently? How do you know you don't need only 2 server but are paying for 5 because of some coding mistake? Most professional IT organizations have architecture and capacity planning people who do this stuff and when a dev can do something unilaterally, irrespective of costs to the company, that makes IT managers not happy. Also who gets in trouble when private information is stolen, hacked, etc. from something that is not managed by them. My former company got fines of up $1 million an incident for not having audit-able proof (not that it did not occur, did not have screen shots or what not) for an account being turned off. When you don't manage a service, how do you do background checks on all AWS employees, etc? I am just saying there are more and more regulations coming down the pipe, and this is how they look. Cloud services while cool for some social media app or some basic apps, if you want to follow any modicum of strict control over costs, or security, will never fly.

Re:Tightening reins on developers?

[cryfreedomlove]

i've seen crap deployed by developers outside of IT input it gets put on the oldest and crappiest server just because that's a name they have known for years no backup gets done on the databases because IT has no idea they exist half the time there is no DR or any kind of redundancy in case of hardware failure

and when it goes down they run to IT and scream how it's IT's responsibility to make it work

Bad developers are bad developers, whether they are supported by classic IT or using the cloud. Great developers, however, don't do the nonsense you are referring to. They care a lot about security, DR, performance, availability, etc. It is this top tier developer that, given an API that procures new hardware, does not really need classic IT support.

Re:Tightening reins on developers?

[Guido von Guido II]

Let me rerephrase that for you. "Developers don't care jack shit..." Show me a developer who is incapable of being a successful sys. admin and I will know you a terrible developer. It's all about time and interests.

Absolutely. I know plenty of people who've been both good sysadmins and good developers at various points in their career.

Like you say, developers aren't necessarily interested in the things that make for good administration, though. The ability to create virtual machines at will, for instance, means that developers can create more virtual machines than are needed, which results in greater administrative overhead and greater costs. They can also sidestep normal administrative procedures.

We used to have enough problems with this back in the day when everything ran on physical machines. A developer would slip a box into the data center and not tell anybody or document anything. Later we'd have to clean up the mess when we discovered that nobody had been patching the machine and it had a security problem, or they hadn't configured something properly so that it didn't start up properly on reboot. My favorite was the time the developer (who was in my opinion very otherwise very good) forgot to configure the IP addresses of a machine with anything more than ifconfig. The machine came back up after its first reboot without IP addresses and a good chunk of the customer's site was down for an extended period of time.

With virtual machines, this is so much easier.

Re:Tightening reins on developers?

[SimplyGeek]

That's not always the case. Look at workplaces that fall under HIPAA regulations. That last thing IT wants is for developers to start up their own app projects in the cloud, whereby their apps start accessing PHI/PII. The moment that PHI goes from the local intranet and those bits go onto a 3rd party cloud service, the company will shit a brick because the developer's just violated regulations. There's a reason IT and security requires oversite of app development.

Re:Tightening reins on developers?

[ebno-10db]

In a well-functioning organization

Huh?

Re:Tightening reins on developers?

[jon3k]

50%+ of developers I've worked with are terrible. But that's true in any field. 50% of the admins I've worked with were bumbling idiots, too. Welcome to reality.

Re:Tightening reins on developers?

[datavirtue]

Yes, I work at a college and run into many admins who are openly hostile towards cloud services. None of their arguments have anything to do with any type of real concern. None of them can code ("I hate programming") or understand networking or databases though, so I assume they are afraid of being marginalized. They marginalize themselves by operating with limited skill sets. One day management will learn how much a liability they are, and their arguments which once had sway will be reveled as the ridiculous tripe it really is. The cloud isn't the answer to everything or an automatic cost savings, but these cloud services do offer solutions to once daunting problems. I just got off of Azure after checking it out--prompted by an above comment--and I'm pumped. As a developer it makes my life easier, and when developers start mocking up test apps and demonstrating them to management on services like Azure or EC2 the momentum will increase. BTW, Azure has an amazing interface and already seems much easier to digest and deal with than EC2.

Re:Tightening reins on developers?

[nine-times]

Sounds like you're a developer who's pouting because your IT department won't let you run amok.

As an IT pro, the cloud doesn't scare me. "The Cloud" just pushes the IT needs to different places. Sure, it might eliminate a few jobs here and there, but you're horribly misinformed if you think it removes the need for network engineers and support personnel. I've run into too many people who sign up for cloud services imagining that it will be an IT panacea, only to find that they now need someone to help them manage the cloud service.

The problem is that developers don't always think things through in the way that a good IT pro might. You throw your cloud application up on hosted solution because it was convenient, but did you think about security? Or did you just dump a bunch of confidential data on a poorly secured server? Did you consider networking topography and bandwidth requirements? Did you do a real cost/benefit analysis, or did you just pick a solution arbitrarily because you thought it was cool? Did you think about how you were going to keep your data backed up and redundantly accessible, or are you just relying on Amazon to never have an outage?

Some developers will think about everything they need to, but in my experience, even great developers sometimes make for poor IT professionals.

Re:Tightening reins on developers?

[ranton]

Its usually not that the developer thinks he is a better sysadmin than the actual sysadmin, it is that the developer is frustrated that the sysadmin won't help because it is a distraction from day to day operations. It is much easier to support the needs of standard users than it is to properly support the needs of developers (and testers). We often need to create our own private networks, create and destroy computer instances constantly, have virtual machines with dozens of OS/software configurations for testing, have servers with externally routable IPs, etc. A sysadmin would probably do a better job than a developer at doing these tasks, but usually they just don't want to deal with it. So the developer does it (often poorly) and then the sysadmin and developer fight over the results.

That is the only cause of developer / IT conflict that I have ever seen.

Re:Tightening reins on developers?

[onyxruby]

Your sort of on track. What you really need is a properly set up lab environment that you have administrative access to. Your needs are perfectly legitimate, and you certainly need to be able to do the things you do in a timely manner. This is not an issue with administrators, but an issue with not having the lab set up correctly. This is what you need, I speak from the experience of setting these things up when I used to travel.

This is what your IT staff needs to set up for you for you in a proper enterprise class lab.

1. Your lab needs to be identical to production. Rights, operating systems, software, versions, everything except hardware needs to be identical.
2. Labs should never have special privileges to make things easier or faster, if you have done this you have failed.
3. Your lab needs the ability for you to reset any given machine on demand without going change control.
4. Your lab must have enough RAM to be usable, ESXi or equivalent lab servers with inadequate RAM will spend far more in labor than the cost of RAM in days.
5. Your lab must support the number of users that are needed for use simultaneously.
6. Your lab needs to be part of change control. Nothing should ever happen in production until it has been tested in the lab.
7. The use of snapshots to maintain parity with production is critical.
8. Your lab must be patched as the test environment before production. It is never acceptable to have an unpatched lab.

Remember that lab license costs are almost always allowed to be significantly offset by almost any vendor as long as it is not production.

Re:Tightening reins on developers?

[ahodgson]

I love the cloud. I don't have to screw around with hardware and most of the real sysadmin tasks are harder in the cloud than on a real server, at least to do them right. It's perpetual job security. A developer wouldn't know a good configuration management system if it bit him. I take over projects all the time deployed by developers. Hand-crafted config files, no redundancy, no backup. It's hilarious.

Re:Tightening reins on developers?

[h4rr4r]

Why are they not in the server room more often?

The daily check should have found the undocumented machine. At that point you unplug it, remove from rack and move it to your desk. You will find the culprit when he comes looking for it.

Also no developer should have keys/fob access to the server room.

What the hell is a "private cloud"

[LordLucless]

So the idea of "cloud computing" is that out there somewhere, a company has a helluva lot of computing resources (processing, disk, network). There's an abstraction layer between the physical hardware and the user, that lets you spin up virtual machines that consume fractions of this capacity. Because the cloud provider operates at such a large scale, it can guarantee that when you want to spin up a new virtual machine, there's the physical capacity there to back it.

But that depends on scale. Ok, so an individual company buys a bunch of hardware, runs some abstraction stuff on top, and starts spinning up virtual machines. How is this different from the what they were doing pre-cloud - that is, running their own cluster of physical machines? Oh sure, you can probably make your physical machines a little bit more flexible by running arbitrary virtual machines on them, but the main benefit of the cloud is that you can utilise the provider's scale to quickly ramp up if needed. The only way you could do that in a private cloud is if you massively over-invest in the physical machines your cloud's running on. What company's going to do that? Why run a "private cloud" over a cluster?

Also, the "zombie machine" argument is pretty hilarious. I'm sure we've all heard of the infamous drywalled server - and that's just an extreme example of a common issue. How many places have you worked were there's random machines running that people are too afraid to turn off because nobody knows what they do anymore? Zombie machines hardly seem to be a cloud-specific issue. At least cloud providers give you an itemized list of every server you're paying for, and you can decomission them with the click of a mouse.

Congrats

[MickyTheIdiot]

You just did a story about businesses creating server rooms.

Ooooh... the cloud!

Different needs for different scales

[slim]

How hard is it to understand that the cost/benefit depends on your size?

Car analogy: If you're an individual who needs a car a couple of times a year, you rent one on those occasions. If you drive almost every day, you buy a car and you get it insured. If you're a small company, you give your travelling staff a car allowance. If you're a big company, you buy a company car scheme and insure all the cars under one policy. If you're a gigantic company, you self-insure all your staff's company cars.

Draw a graph of the cost vs scale of a third-party cloud, versus your own datacentre. At some point the graphs will cross. That's where you switch.

Re:Different needs for different scales

[Attila Dimedici]

Exactly right. Of course for a short period of time there was this idea that somehow, magically, vendors would be able to offer limitless scalability where if I needed 5 servers today and 500 servers next week I could get that in the cloud for less than the cost of maintaining 500 servers. It sounds good. That outside vendor can rent the space on those additional servers to someone else when I don't need them. The problem is that it turns out that the people who suddenly need a 100 fold increase in servers all tend to need it at the same time. So the outside vendor needs large numbers of servers and doesn't have anybody to rent them to when the people who need them don't need them.
The end result being that if you are a small company, there are significant chances that an outside vendor will be able to rent you server space cheaper than you can supply it for yourself, but once you get bigger than a certain size that no longer is true.

BS summary and title

[andy1307]

From TFA

It’s impossible to know whether a significant number of businesses are deserting AWS and public clouds in favor of private. My guess is there’s some movement as businesses get more experience in the cloud but certainly not enough to dent the potential of the public cloud. Still, the murmurs are an indication that AWS competitors are starting to get more aggressive.

That's exactly the kind of hard data nerds use to arrive at conclusions...

Regarding zombie servers

[jmak]

There are tools to deal with them, and were even recently featured on Slashdot: http://news.slashdot.org/story/13/01/07/1551231/netflix-open-sources-janitor-monkey-aws-cleanup-tool

So . . . what?

[Kimomaru]

So, in other words, companies are leaving cloud comuputing to set up co-los? This is an option that's been available for, like, at least 15 years.

You ever get the feeling the term "cloud computing" was coined because people were desperate for something new while the economy was getting its legs back?

Re:So . . . what?

[Kimomaru]

No . . . it's not a real thing. It, "Let's host a ton of virtual servers on very expensive machines and let people rent space on it." Wow. Not new.

Pretty blatant ad posing as blog entry....

[technomom]

....for OpenStack. C'mon, can you be a little less obvious next time?

Re:For one thing...

[slim]

We need to get away from the term "cloud" -- it's a misnomer.

No, it's just frequently misused.

One remote server, is not a cloud. Two load-balanced remote servers is not a cloud.

Dozens, or hundreds, or thousands of remote servers, configured such that data is stored redundantly and the software routes around a failed node; controlled by infrastructure such that adding or removing nodes is negligible effort -- that's a cloud.

Of course the marketers misuse it, because they want their non-cloud product to bask in the halo effect of the buzzword.

Instances are ...

[Skapare]

... last decade's processes. It's just a different environment to make ever more use of computers.

cowboys like you

[onyxruby]

I've reined in cowboys like you for years, from one fortune 500 to another. Arrogant jackasses that can't be bothered with change management, best practices, version control, documentation, pesky things like policies, regulations and laws. Self righteous developers that can't see past their own nose too see how thier actions or inactions affect those around them.

Every single time they think they are above these things and that they know better than the industry around them. They never realize why something that works in their special environment works perfectly fine where they have the rights of a God but has all kinds of mysterious errors in production where there they are brought back down to earth. They then chafe when their development environment is set up identical to production, yet it is amazing how quickly previous mysterious bugs that plagued production and caused incredible operational costs suddenly get fixed. They of course never have to clean up multi-million dollar messes, talk to regulatory agencies, sit down with lawyers to plan how to mitigate their mess or have a face to face with an angry Attorney General.

I've only won this argument and helped companies save millions by reining in the cowboys like yourself a couple dozen times. Probably something to do.with cleaning up large multi-million dollar messes more than once.

Re:cowboys like you

[ebno-10db]

I've only won this argument and helped companies save millions by reining in the cowboys like yourself a couple dozen times.

Sounds like you should get paid pretty well for that. So instead of complaining, you should thank the OP and his ilk for helping to provide your paycheck. Next cops will complain about there being crooks. Some people don't understand where their bread is buttered.

Re:cowboys like you

[onyxruby]

My point of fact I have been paid well for reining in cowboys like him. I traveled for years as a consultant and while that wasn't my job as such, it was something that kept coming up. Point being that every time I have to deal with a cowboy it takes up time and energy to rein them in and bring them back town to earth.

I've never lost the argument, and I've never failed to rein in any department of developers, no matter how much they thought they were big shots. It's not about my ego though, it's about keep cowboys from bringing down the house when their ego get's out of control.

Re:cowboys like you

[onyxruby]

I feel your pain, that's a large part of what I did was implementing the type of things your talking about needing. In fact it was setting up that kind of process and environment where I would run into the cowboys. IT is supposed to support and facilitate the business in meeting their needs. It should never be an obstacle or an impediment.

In order to do that you have to responsive to what the business needs in a timely manner. You can't take so long to implement something that the business works around you. If your environment wants to use Mac's and you don't have a solid reason not to do so than you need to figure out a way to do so. For example I've got 8500 Mac computers where I work that were not being managed so I created a steering committee to oversee them and drove the process to bring in an enterprise management platform because that is what a large number of users want to use - not what I want.

You also have to implement enough change management that your retaining control of your organization. Change management should never involve making secondary changes like you've experienced, that is the result of poor change control. Rollback plans are important though, the idea is that if something fails you should have a plan for how you would put things back.

Things like setting up labs that are identical between production and development are critical for success. If development doesn't mirror production (rights, different OS's, different versions of software etc) than all of your testing is meaningless. /rant off

Re:For one thing...

[tubs]

Sounds like Web 2.0 to me. Or was that just a marketing term too ... anyway according to Nist the essential characteristics of "cloud computing" are : On-demand self services, Broad network access, resource pooling, rapid elasticity and measured service. No mention of redundancy or routes.

Ahhh, ha ha ha. last square on buzzword bingo!

[swschrad]

get off the cloud, build our own cloud. also known as bringing the server room back into your own hands.

also known as BOFH never dies.

Random pricing

[EmperorOfCanada]

One thing that has kept me away from Amazon's cloud is the unknowns with its pricing. I have visions of a DDOS either clearing out my bank account or using up my monthly budget in the first 2 days of the month. Plus if I mis-click on something I might get an awesome setup that cleans me out. I am not a large corporation so one good bill and I am out of business. But even larger companies don't like surprises. So regardless of the potential savings I am willing to spend more if the price is fixed in stone instead of chancing being wiped out. I like sleeping through the night.

Plus as a human I really like being able to reach out and touch my machines, even if I have to fly 5 hours to do it. So the flexibility of the cloud sounds really cool where the pricing is not so flexible. It would be nice to spool up an instance of a machine that isn't going to do much most of the time that doesn't actually use up a whole machine. But then when one machine starts to get pounded to give it some more juice. Plus upgrading your hardware would be much more of a dream. You move your most demanding servers to your hottest hardware and slide the idle servers over to the older crap. Plus restores and redundancy are a dream.

Then you still have the option to fully dedicate a machine in "realspace" to a demanding process. While VM does not have much overhead it does have some. So taking a server(s) that is being pushed to the maximum and sliding it onto bare metal will then allow your hardware to be used to maximum efficiency.

Then by having no real cost overhead to having more near idle machines spool up your developers can play interesting games. Maybe they want to see what your software will do with 20 MongoDB servers running instead of the current 3; or 200.

This all said, I am a fan of Linode; where I can predict my pricing very well.

Re:Random pricing

[bored]

While VM does not have much overhead it does have some.

This is the common view, but its a gross generalization. For some problems you won't be able to measure much of a difference. These problems tend to be problems that are low thread count (1-2 loaded threads) and have a high cache/TLB hit rate with few kernel interactions. On the other hand, applications pegging out more than 8 CPUs, or doing a lot of cacheline ping-ponging, etc tend to take a noticeable hit. Furthermore, applications that are doing a crapload of IO through adapters that are neither pass-through, nor have SR-IOV or similar high speed IO support and appropriate hypervisor support will be hit with orders of magnitude performance problems. Even with support, picking the wrong set of adapter vendor/hypervisor/OS can have pretty serious performance impacts.

So as you said, put it on the bare metal if your having performance problems. But this is sort of my problem with many of the modern developers mindset. The lets just throw another VM at the problem attitude is so wrong its amazing. I think its founded on the fact that a huge number of people in the IT industry don't understand the difference between throughput and latency when it comes to performance problems. I see it all the time with the "we just need another VM" mindset. Half the time the problem is that the attached disk subsystem is doing 300IOPS cause its running on some internal RAID6 with 4 disks and each IO is doing a read/modify/write cycle. Plug in a SSD and reduce the physical server count from 10 machines to 1. Or for that matter move a table out of the database to a memory mapped file, and return the results directly from the mapped page rather than bouncing it through a socket, and then copying it a couple times through some interpreted scripting language before before the ethernet adapter finally DMA's it out. Its amazing when you show someone that the 1.5k piece of data they are returning was copied 30 times around in memory 20-50 times and that is what their CPU is spending all its time on.

OF COURSE people at an OPENSTACK conference....

[Thrill Science]

OF COURSE people at an OPENSTACK conference will be talking about alternatives to AWS. That's the point of the conference. What did you expect?

Amazon has convinced many people they are cheap

[stefancaunter]

I'd love to see more people taking on scale themselves, but unless the perception that Amazon is a good deal changes, this won't change much in the way of their dominance. Unless you've actually been taken to the cleaners by them on a project, and can convince your boss that owning/renting gear is a better plan, they will still be a first choice vendor. Decision makers read magazine articles (when they aren't playing games on their phone) that tell them Amazon saves them money. Everyone sits around in a meeting and nods their head.

Legal issues as well

[davidwr]

It's usually "cleaner" if you either don't out-source sensitive data or if you out-source it in a way that is either 100% encrypted and you hold the only keys or if it's stored in an "identifiable" physical place ("it's on THAT set of hard drives, and it's being processed on THAT set of CPUs" etc.) that isn't shared with other users.

need one button click move feature

[cellurl]

I for one am afraid to move away from Amazon. It took 6 months to understand it. Someone needs a simple exit button for us scaredy-cats.

Help eliminate stupid speeding tickets