Slashdot Mirror
Amazon Wants To Run Your High-Performance Databases
jfruh (300774) writes "Amazon is pushing hard to be as ubiquitous in the world of cloud computing as it is in bookselling. The company's latest pitch is that even your highest-performing databases will run more efficiently on Amazon Web Services cloud servers than on your own hardware. Farming out your most important and potentially sensitive computing work to one of the most opaque tech companies out there: what could possibly go wrong?"
Due to my high performance AWS posting station.
Amazon Wants Your Money FTFY
Be fair, Think of all the taxes the have to pay and the living wages for staff ... oh wait
Amazon Web Services can be found on the list of Linux Foundation patrons, which means that they help to assure that open source projects get the appropriate funding that they sorely need. I don't know about you, but that's a big plus in my book.
Needless, inane, editorializing in the summary, as usual. So sad. Especially when the article itself is concise, factual, and free of such nonsense.
People, "cloud computing" is nothing but a rather thinly veiled mix of software as a service and server hosting, ok? The reason why we needed a new word for it is that the former had a very bad rep by now (and it fully earned that rep), and the latter is anything but edgy and cool anymore.
Could we, at least here, avoid the whole marketing lingo? It may be "cloudy" to markedroids and management, but I guess we DO know here that the data is not just put "somewhere in the cloud", right?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Get real AWS...
A m3.2xlarge costs 4905.6 per year. You can buy a 32GB RAM 8 CPU core Dell R320 system for $2,666.80 in it's entirety. Literately you are spending nearly twice as much to use AWS. And this is before even taking into account the cost Amazon charges for bandwidth.
You are omitting the cost to admin, care and feed the hardware. That is AWS's selling point - what happens if you want to use it for your program / project that only lasts a short period of time? What if you got the scale wrong? Reliability and Redundancy? There is a price point for leasing services especially when there are unknowns in the scope of your venture.
And I mean "my", "fucking", "dead" and "body" literally. I already imagine telling one of my customers that I host their data on Amazon AWS. I will never have had that good an opportunity to study people's backs.
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
Seems Amazon and Google see the writing on the 'internet wall'.
Their core products/services are not going to bring them anymore revenue than what they get now, and can shrink further when nimble competitors or new ideas happen. So the only way is to branch out.
Google thinks it will be driver-less cars, automation, internet balloons, thermostat etc., while Amazon thinks it will be AWS, cloud and so on.
Surprisingly both these behemoths are not branching into life sciences. May be no has made good impressive power points yet.
The one company terribly lost is Apple. They are buying into an arthritic rapper!!!
Tat Tvam Asi
Dear America,
Following the Snowden revelations your NSA inspired dream of cloud computing and total social networking (i.e. full access too all the data in the world) is dead.
Nobody with a brain would even think of storing their data on an American computing resource.
Sincerely yours,
The rest of the world.
Sky subscribers are morons. They pay to be advertised at !
At first glance, I red "Ruin" instead of "run". I must be biased.
Yes, I'm sure Amazon can run my database more efficiently than I can. But what are they going to do when I need to fetch 100 megabytes of data from a table and I want it in less than 30 seconds over my 20 megabit/s internet connection? Hmm?
systemd is Roko's Basilisk.
A couple of questions for you:
1) What happens when your single server goes down? How long does it take you to get back up and running?
2) What happens if your demand is spiky?
If you're going to use an instance for a year constantly, you need to look at reserved instances. That brings the price down to $3054 for the year which is not bad as you don't pay for electricity or cooling.
Get real AWS...
A m3.2xlarge costs 4905.6 per year. You can buy a 32GB RAM 8 CPU core Dell R320 system for $2,666.80 in it's entirety. Literately you are spending nearly twice as much to use AWS. And this is before even taking into account the cost Amazon charges for bandwidth.
AWS is best used in time frames of weeks, not months or years.
Try justifying a few thousand in hardware expenses for a 2-week project. Then you might see where the value lies in spinning up hardware within hours at AWS for short-term demands.
OS upgrades, backups, hardware support contracts (which aren't cheap for good response time), and of course the SysAdmin to keep it all running. Gee, I just love how you sell your Dell solution as "cheap" while omitting all of these expenses. Reminds me of Oracle sales pitches.
If you're using AWS as a replacement for a dedicated machine, then you're doing it wrong. Even then, your comparison is disingenuous, because you're not costing in rack space, cooling, power, and so on in your purchase price.
The point of AWS is if you need to spin up a few (or a lot of) instances quickly and use them for short periods. How long does it take you to buy and rack that Dell system? If you can get it in under 2 days then your admin staff are incredibly impressive. If you can get it in a week, then they're doing pretty well. Amazon can get you the instance in a useable state in a few minutes. They can also dispose of the machine in a few minutes. That's a lot cheaper than buying those Dell instances to support your peak load, if you only ever hit that load once a year and are at 10% of that the rest of the time.
I am TheRaven on Soylent News
Yes, but what if you actually want to turn it on? Then you have to pay for electricity and batteries and a generator and redundant A/C. Or if you want to connect it to the Internet reliably? Then you have to pay for multihomed Internet service. If you want it to be highly available then you have to pay for a data center to house it and probably another one for DR. If you want disk to go with it then you have to buy disk, and you have to figure out exactly how much you're going to need because if it turns out in a year or two that you need more than you thought you probably have to get another CapEx purchase approved. What if you bought too much hardware or not enough? Then you have to start over!
Not to say that AWS isn't more expensive in some respects, just highlighting the fact that other things are included in their pricing and the flexibility they provide.
Of course Amazon wants your money. They're a business trying to make a profit. There are plenty of things people can complain about when it comes to Amazon or to any of their competitors in this arena, so why do people keep complaining that Amazon is trying to make a profit for itself and its shareholders?
1) I guess it goes down until it can be fixed under warranty (same or next day depending on purchase option). Redundancy is expensive. What happens when your single instance of AWS goes down with an "oops amazon is having problems with a datacenter" message?
2)Good job, you have identified why Netflix uses AWS.
3) Reserved instance is cheaper, but at that price still more than a dedicated server and the server typically comes with a 3 year warranty and will likely last past that (Dell will warranty for 6 years). Assuming it only lasts 3 your cost for running on AWS is nearly 3 times higher even when figuring in an improved warranty and OS licensing. I concede that short duration projects or very spiky loads are a great use for the cloud, but long running relatively even loads simply don't make sense form a cost perspective, nevermind the fact that you now lose access to your database if your wan connection goes down (unless you build out multi-wan, but there is yet another expense).
Get a web developer
That is AWS's selling point - what happens if you want to use it for your program / project that only lasts a short period of time?
I used AWS for 3 weeks for a Data Science class. My bill was $1.92.
What if you got the scale wrong?
Buy more bandwidth processing power? You start off small and as you need more, you buy more. And if you need less, change you plan. It's nothing like buying hardware where once you buy it, you have to deal with equipment.
Reliability and Redundancy?
It'd be hell of a lot better than what I could do with Dell's in my own space.
Everybody that uses external hosting for applications that use databases (I refuse to use the c-word any more) also uses it to host the databases they rely on, as it is a basic principle that you need to minimize network latency in this scenario. Having an application talk to databases thousandss of miles away running over anything but very expensive dedicated fiber would be a Very Bad Idea.
OK, Amazon probably handle the install and basic configuration for you, but how difficult is this for a DBA? In my experience, these type of externally hosted services don't reduce the amount of administration required at all, all they really save is the initial installation costs. (And that's only installation, not configuration which you still need to do plenty of yourself)
Okay, maybe your post (if it was yours) wasn't a complaint, but I have seen this complaint time and again when it comes to companies like Amazon, as if we should expect big companies not to try to make a profit for their shareholders (which would be considered negligent). I think there are many more important things to be worried about (privacy being the most obvious) before people are concerned about capitalism being capitalist.
I am glad to see companies and individuals succeed, be it monetarily or otherwise, and I don't think we should let their success, or perhaps our own avarice, get in the way of asking the actually important questions about things like ethics, morals, and integrity.
same with AWS
the storage is extra
the networking you pay for data per MB or GB in and out
you pay for backups
you pay for DR in different availability zones
you have to pay more in your office internet access to get the faster access to access your data
all that is extra on top of that instance charge
If you are comparing with a fixed purchase, you should use the 3-yr reserved price for the M3.2XL, which is $162/month ( includes the initial payment ). This gives you a yearly cost of $1944. And that includes all NOC costs.
If you do not factor in NOC costs in your estimate then you clearly haven't been doing this very long.
Source http://calculator.s3.amazonaws.com/index.html
AWS has been a blessing for our company. It is WAY cheaper and way more reliable than running something in house. We have 5 servers running that ends up being only a little more than what we were paying to run 2 servers internally (factoring in time to manage hardware, outages etc). We had way more outages running locally than we do with amazon. We do have quite a few "short" projects that is great for spinning up a server for a couple months then killing it
We are using AWS for our startup. Our bills are around $2200 a month. $1700 of that is a charge to have dedicated instances instead of shared. this gets us 6 servers - 4 small 2GB RAM web servers and 2 4GB ram DB servers (in reality what we need at the moment, we can scale the DB later when we bottle neck).
We've done the math ourselves and in reality we could probably save some money (face value) or buying servers ourselves and colocating them. But, then you have to add in the maintenance costs, a part time infrastructure support person, downtime, replication, etc. Plus, things like good firewalls, load balancers, are terribly expensive which we didn't factor into the equation.
In reality, it boils down to convenience for us to stick with AWS, though it maybe a slight premium in the end. It simply cuts out a lot of costs having to deal with infrastructure.
Is AWS perfect? No, not really. Any big storm you pray to baby Jesus US-EAST stays up. But, we've been happy so far with it. I'd say all in all it isn't even a premium, factoring in all the costs it is probably break even at this point. But for us we were able to scale from $200 in server costs a month to $2000 a month easily and we can scale to $10000 a month easily. it has its value.
Why use Amazon when its competitors are offering SLAs?
We just did a 3 year reserve on a few instances, it makes the cost about 1/3 the price of their on demand. Sure the cost on paper is still a bit more than dedicated for hardware alone but the chances of hardware failure, internet outages, etc has a way higher chance than AWS going out. We ran servers internally for years and switched to AWS 3 years ago, we would never go back to running internally again.
Basically I think it boils down like this - small to medium sized businesses "cloud computing" is probably more viable. When you start getting to large businesses and corporations it may be more viable to run internally.
Shhh, don't tell anyone the cloud is just an overpriced money grab. Ride the wave kid, make the cash while it's on the table.
Cheap storage VM.
1) I guess it goes down until it can be fixed under warranty (same or next day depending on purchase option). Redundancy is expensive.
With Amazon's multi-az, redundancy just costs double whatever your non-redundant primary server costs (or less). Less if you don't mind your backup server being a bit smaller. Getting true hot-spare failover working properly on a database server is a royal PITA. The fact that Amazon will happily sell it to you for $250/mo. is a steal, and far cheaper than actually doing it yourself would be for any significant project.
You're special forces then? That's great! I just love your olympics!
Um, actually Amazon tries not to make a profit. I'm not sure they've ever made more than 2% profit in a quarter. Typically, closer to 0%.
https://www.google.com/finance...
There are two types of people in the world: Those who crave closure
As someone managing a farm of VMware servers where RAID6 SSDs don't have enough IO, that's good to hear.
There are two types of people in the world: Those who crave closure
They are definitely trying to make a profit, just not in the short term. Bezos has always had a very long term outlook when it comes to profit, and apparently his shareholders are okay with that.
No.
1) I guess it goes down until it can be fixed under warranty (same or next day depending on purchase option). Redundancy is expensive. What happens when your single instance of AWS goes down with an "oops amazon is having problems with a datacenter" message?
Well i guess the same thing that happens when the datacenter that my 1U server is colocated in goes down -- I either bring up the server n a DR region (which I can set up nearly for free with AWS), or wait until the datacenter problem is fixed. In the past 2 years, haven't experienced a single multi-AvailabilityZone outage with Amazon, and only 2 short single AZ outages that resulted in no loss of service since my servers are split across multiple AZ's. I've never had to fail over to the warm-spares in a separate region (other than during testing).
3) Reserved instance is cheaper, but at that price still more than a dedicated server and the server typically comes with a 3 year warranty and will likely last past that (Dell will warranty for 6 years). Assuming it only lasts 3 your cost for running on AWS is nearly 3 times higher even when figuring in an improved warranty and OS licensing. I concede that short duration projects or very spiky loads are a great use for the cloud, but long running relatively even loads simply don't make sense form a cost perspective, nevermind the fact that you now lose access to your database if your wan connection goes down (unless you build out multi-wan, but there is yet another expense).
Coloc space is not cheap, so don't forget to factor that into the costs. Running a datacenter in the office is even more expensive due to the costs to add the needed redundancy (power, cooling, internet) to an office tower.
If you do the math, you are going to be paying for that server, CPU, RAM, OS, disks, and such, either by having the box physically located at your site, or you will be paying for a similar server in one of Amazon's racks. For ranges of weeks, AWS is fine. For constant load, might as well have the server in house and at least pay lip service to Sarbanes-Oxley or PCI-DSS3.
Yeah, once they have enough of a monopoly to abuse it.
You know like with Hachette.
There are two types of people in the world: Those who crave closure
Don't feed the trolls.
Don't feed the trolls.
I used AWS for a few projects for my research. I would upload a data set to AWS run the MapReduce jobs and then analyze the output in AWS. Once complete, I'd download the results and shutdown the whole environment. I could programatically spin up the whole environment in about an hour or so.
Whole thing cost me about $350.
Um, actually Amazon tries not to make a profit. I'm not sure they've ever made more than 2% profit in a quarter. Typically, closer to 0%.
https://www.google.com/finance...
That's excess profit. That's like saying you didn't make any money last year because you spent it all on a house and a boat.
Amazon is making plenty of profits. It's just spending them on expanding so it doesn't actually post profits but if you look
at it's total net worth you can see that it is still growing every year.
The R3 is just an "instance". Sure it's a memory optimized instance, but it's not even their relational instances or mapreduce databases.
We also had no problems with AWS pricing. Our problem was with their performance.
They are not set up well for high io database applications.
We switched to solid state drives on stormondemand(aka liquidweb) and have seen a 10 fold increase in performance.
I prefer liquidweb's model as I can even opt to pick the exact specs of my machine but I still have all the same
cloud features like spinning up a new instance or changing the size of an instance with a click of the button.
To me stormondemand is the best of both worlds. Oh, and the best part is that I can actually talk to someone if
there is a problem.
> It is WAY cheaper and way more reliable than running something in house.
No, it is not. Either you're a terrible shopper, do not do metrics, or are mischaracterizing your usage. AWS is about twice as expensive as alternatives with instability at the node and service level. They are ok for elastic needs and decent for prototyping (when you make a new account you get that free year of micros).
I am curious on using SSDs for high IO databases. Do you worry about the massive amounts of read and write on more full drives or does stormondemand balance your data so that drives are not writing over the same block enough to hit the upper limit? Increase in performance makes perfect sense but integrity of the databases is something we always debate (I work on multiple large, high IO RDBMS) whenever talk of using SSDs comes up.
They are mirrored drives and we use a journalling filesystem and take good backups.
We haven't seen any issues yet. Unfortunately that's one place where cloud is lacking.
Even though stormondemand offers mirrored drives and bare metal servers, to my
knowledge there is no way for me to actually check the raid status of the server.
Okay. That makes sense I figured there would be some redundancy to protect the data but I wasn't sure how easily it was managed. It sounds like it is definitely in between. Thanks for the response.
For us, I knew the game was up when the new (from a hot AWS-only company) data center director moved the goalposts for his AWS eval to 'only 20% more expensive than physical'.
If you want it to be highly available then you have to pay for a data center to house it and probably another one for DR.
AWS US East (Virginia) keeps going down and taking parts of the internet with it.
Far too many *large* AWS customers aren't setting up any redundancy.
Vine and Instagram both got zapped in the last US East outage.
[Fuck Beta]
o0t!
Yes I can buy it, but then you need to include hardware installation price and hosting price.
And if the Amazon computer hosting my vm dies, the system will just boot up on a new computer and restart the vm, without me even knowing it. With the dell I would have next day business service aka far to much down time.
The reason the company I work for uses vms(Not from amazon however) is that if there are hardware problems we just reboot, and if there are to many problems with our host, all we have to do is a dns change, a scp, and a database copy(Or restore from backup) and then we can be online in a new data center.
I would however be cautious with a big database server at amazon, because I don't think their io performance is that good, and than is the thing a huge database server really need.
yeah u r right... 99% of aws customers need scale...? NO
Scalability is not just for the enterprise. In AWS you can feel safe starting with a $15/month setup, knowing that you can always move to the $100/mo plan if the workload changes. When buying hardware you do not have a similar liberty. It is either buy more than you need, or risk having to buy again.
You can also deduct the full cost of your "lease" as opposed to depreciating the hardware investment over 5 (3?) years. Because we have an outsourced, throwaway, service-based economy/society, the government makes it compelling NOT to invest in infrastructure.
I want to delete my account but Slashdot doesn't allow it.
Thanks, but no thanks. I'd rather not eat into my bandwidth limits each month.
If you're truly in a PCI-DSS situation, the last thing you want is to have your server in house. Now your datacenter has to be fully compliant with the PCI physical access guidelines. It'd be a lot easier to lease one somewhere like Rackspace, which is even more expensive than AWS (who provides one of the few cloud environments that's PCI level 1 certified at this point).
You're special forces then? That's great! I just love your olympics!
Re Hachette: Nice background http://www.hughhowey.com/more-...
you had me at #!
https://aws.amazon.com/premium...
you had me at #!
A software RAID implementation like ZFS is superior to hardware RAID (it has a far higher standard of integrity and flexibility designed in), and you can monitor it however you wish. You can run it on cloud services.
you had me at #!
Yeah, in theory, "the cloud" is suppose to handle the failures transparently (and/or never fail) but I
find it harder to trust hardware and backups procedures that I can't physically see or test.
It would be very easy to create a virtual server which falsely claims to have full raid and geographic
redundancy, etc... but without the ability to actually test a drive failure or a datacenter failure you are
really just trusting the 3rd party to be honest and to perform due dilligence as usually their contracts
state their maximum liability is limited to just giving you your hosting fees back.
That's partially true...US East is actually comprised of many different availability zones, each of which is self contained and housed in a physically separate data center. So, US East isn't going down, particular AZs go down from time to time and the way Amazon names them makes it exceedingly confusing to narrow down what's actually failing. us-east-1a for me may not be the same AZ it is for you. AZs are defined on a per account basis, so it just makes it easier to say the region has issues when technically that's incorrect.
But you're spot on when you say lots of customers aren't setting up proper redundancy. AWS has made it very clear from the beginning that for proper redundancy you want resources in multiple AZs. I still don't fully trust a multi-AZ setup because although I wouldn't expect Virginia to get many hurricanes, if they did it could impact multiple AZs. I'd really like to see AWS improve support for multi-region deployments. It's an area they haven't historically been very good at. A year or so ago they finally introduced the ability to copy EBS snapshots between regions. I was one of the customers that was pounding on their door to get them to do that so they let me in on the beta testing for it.
It's a shame too because it's really pretty easy to architect for HA in AWS if you're building something there, it's not as easy if you're transplanting something though.
the storage is extra
You're paying for storage either way but in AWS you don't have to try to figure out the max storage you'll need. Figure you design a system with a 3-5 year lifetime, so you need to know how much storage you'll need over that period. I don't know about you, but that's pretty hard for me to do. Plus, what happens when you do a major software upgrade and you want to retain full offline backups of everything for a couple weeks, effectively doubling of tripling your storage footprint during that time? The flexibility to get as much storage as you need and only pay for it while you use it is hugely useful.
the networking you pay for data per MB or GB in and out
You rarely if ever pay for data transfer into AWS., but for transfer out you'd have the same costs if you self hosted anyway. They wouldn't be per MB, they'd be flat rate, but you're getting a lot more for your dollar at AWS. Let's say you average 10 Mbps 24x7. That's about 2.96 TB of data transfer a month which costs $346 at AWS. A symetrical 10Mbps business connection for your office is probably going to cost you more than that. In addition to that, you're also getting much better redundancy and scalability at AWS and you don't have to pay for the network equipment (switches, routers, etc).
you pay for backups
You pay for backups if you self host too.
you pay for DR in different availability zones
You might. You can create an application in an autoscaling group and tell it to span multiple AZs but only deploy one instance to the group. If that instance in AZ 1 crashes the system will automatically launch it in another AZ and you're only ever paying for 1 instance. You can also create a DR instance in another AZ and just leave it powered off until you need it, in which case you're only paying for the EBS storage, no compute costs.
you have to pay more in your office internet access to get the faster access to access your data
That's true. Many apps these days are just web apps which are not super bandwidth intensive. A lot of times if you have big data it's not as easy to make a financial case for going to the cloud. You can use direct connect to get 1Gbps or 10Gbps access directly to your data in AWS but it is fairly expensive. If you've got terabytes of data though, you probably can make up the cost in other areas.
1) I guess it goes down until it can be fixed under warranty (same or next day depending on purchase option). Redundancy is expensive. What happens when your single instance of AWS goes down with an "oops amazon is having problems with a datacenter" message?
If your acceptable downtime is measured in days, then yeah, you should buy your own hardware. For me, if Amazon sends me an email that my instance is running on degraded hardware, I can clone the instance and remap the IP to the new one. Total downtime is a few seconds.
Bringing up a new instance in a different AWS availability zone is always an option, or a different region if an entire region is down (rare).
Also, I do have spiky loads for the project that I'm working on and I shut down the instances when I'm not using them. I know how long my analysis is going to take, and so I launch however many instances that I need to complete the work in an hour, since AWS bills by the hour. There have been times where that's been around 150 instances. I don't have room in house for 150 machines at any price, so there is that.
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
As someone managing a farm of VMware servers where RAID6 SSDs don't have enough IO, that's good to hear.
Am I being unfair or overly unrealistic by calling VMware's IO performance "less than shitful"?
Granted, tuning is always going to be an exercise best left to the reader, however even with that in mind I've found VMware ESXi 4.0 through 5.5 to be out-of-the-box a fairly porcine experience. I'm interested to hear what I suspect may be an echo of someone with a similar opinion!
..Mullah or Pope, Preacher or Poet, who was it wrote: "Give any one species too much rope and they'll fuck it up"?