Slashdot Mirror


Why PayPal Chose OpenStack

AlbanX writes in with this story about Paypal's use of OpenStack. "PayPal's IT team has taken control of its technology release cycle by shifting key components of its IT infrastructure onto OpenStack. For PayPal, the decision to use components of OpenStack was based around speed to market. It allows the payments provider to untether its release cycle from those of vendor partners. 'PayPal has not historically been known for its fast reactions,' PayPal senior engineer Scott Carlson conceded to attendees at the VMworld conference in San Francisco this week. 'It has taken us six to nine months sometimes to react to our competitors.'"

19 of 64 comments (clear)

  1. Openstack bias by Drewdad · · Score: 5, Interesting

    No articles on Credit Suisse's decision not to use OpenStack and their rationale for not doing so? On my own head be it, I suppose....

    1. Re:Openstack bias by Anonymous Coward · · Score: 3, Informative

      http://diginomica.com/2013/08/27/credit-suisse-cio-perspective/

      "I’ve got 15,000, 20,000 VDI systems running on VMware. I’ve got probably another 30,000 virtual servers, 25,000 to 30,000. ... The money we spend with integrated stack like VMware pays for itself, magnitudes, like hundreds of times when it comes to my people’s cost to run it."

    2. Re:Openstack bias by mzito · · Score: 4, Informative

      Well, VDI is not an openstack use case. But even the traditional use case - it's a purely business decision. They believe that the cost to run and operate VMware is lower than the cost to run and operate Openstack. I know a couple of very large enterprises who have come to that conclusion. I also know one or two that went with openstack and wish they hadn't. Then I know a couple who love their openstack deployment.

      The key thing here is that VMware and Openstack are not really 1:1 comparison points. You can run Openstack on top of VMware vsphere. Why would you do this? You want amazon-like APIs, a real storage service, DRS, and so on. Or you could run it on top of Xen or KVM and save money, but lose functionality. Or you could go out and buy RH's Openstack implementation.

      This is a very complex series of decisions, and it's not really easy or possible to say, "Well, we didn't decide to do Openstack because VMware is better"

      --
      me@mzi.to
  2. Re:So, what the hell is Open Stack? by afidel · · Score: 5, Informative

    It's a collection of open source API's for the management and automation of virtual machines at scale. It can be used with a number of hypervisors including VMWare vsphere, Xen, KVM, Hyper-V and a few others.

    --
    There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
  3. Re:Uh.... is Google broken? by Vernes · · Score: 2

    Both AC's fail to notice the 2nd line explaining Open Stack.

  4. Why does Paypal need "cloud" ? by Gothmolly · · Score: 2

    They have a single business, with a relatively constant, yet probably growing number of users. If they need 100 servers today, next year it will be 120. They don't need massive content or application distribution, they don't need to rush new products to market, they don't need all that cloudy stuff. If "OpenStack" just means they virtualized some old iron that's another story, but a far less interesting one.

    --
    I want to delete my account but Slashdot doesn't allow it.
    1. Re:Why does Paypal need "cloud" ? by afidel · · Score: 2

      FTFA:
      "From the minute developers finish final QA in our dev space, they have 15 minutes to be live in front of customers on the PayPal website," Carlson said. "That means they put their innovation on the VM, they put the operating system on it, they put the app on it, they put on a load balancer, they fix the firewalls and its up."

      and

      The scalability of the OpenStack platform also offered a crucial advantage for when PayPal is forced to cope with sudden spikes in traffic.

      "When [parent company] eBay has a free listing week, they don't necessarily pre-warn us at PayPal, but by the end of that week all of the payments suddenly flood in," said Carlson. "So our volume scales up in one day and goes away the next day."

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    2. Re:Why does Paypal need "cloud" ? by mzito · · Score: 4, Interesting

      So - there's a couple of reasons why they would want their own cloud:

      - Their business isn't actually very static. As you might imagine, they have daily spikes of traffic at particular times of day, likely early evening across the US. It might not be worthwhile for them to do elastic computing for that, but think about holiday times, like Christmas, and their purchase volume certainly goes up.
      - Development environments - very often, developers will want a sandbox environment to use for a few weeks or months and then get rid of them. Or, they might want to run some analytics on 50-100 nodes and then tear them down
      - Easier infrastructure lifecycle management - abstracting the running OS into a virtual machine makes it much easier to archive out old hardware and onboard new - just migrate the VMs over to a new machine, pull out the hardware, throw it away.
      - Rightsizing hardware - cloud allows them to buy a small number of predictable builds and then size their compute to their needs - no need to dedicate an 8-core machine with 8 GB of RAM for an internal email server, or a sandbox to play with MySQL

      PayPal actually has a very complex business, huge infrastructure, crazy security requirements, tons of applications and people, and is generally a technology heavy company.

      --
      me@mzi.to
    3. Re:Why does Paypal need "cloud" ? by mzito · · Score: 2

      Some fair points, but here's my responses:

      - "Deployments" doesn't only mean "rolling out new applications" wholesale. It could mean, "I want to test my new fraud analytics algorithm on our last six months of transaction history", or "We're adding a new feature that might be very popular", or "turns out our application is hitting a database bottleneck, we're working on figuring out why that is, till then, spin up three more read-only slaves to see if we can alleviate it"
      - "Time to Market" covers a lot of scenarios - sure, yes, in the year-long development cycle for an entirely new application hardware procurement is a small percentage of the overall lifecycle. But how much hardware to procure? What datacenter should it live in? Can you accurately predict which portions of the application will be the busiest? Or slowest? In effect, in the same way a lot of organizations are using agile development methodologies to partition their work into smaller, faster, more responsive units, cloud computing environments allow for more agile infrastructure
      - You say you can Ghost a server in an hour. Which is great. In more traditional environments though, you need to also configure network switches, firewall rules, load balancers, configure virtual IPs, backup schedules, provision storage, configure replication (possibly), add the node to monitoring, load application stacks, load code, adjust the app configuration, QA it, and then go live with it. In one of my customer environments, there are 13 separate agents that may have to be configured to run depending on what kind of server it is, and in which datacenter it's going to live, each with its own requirements, dependencies, and so on. In most environments, building out a server is going to take a lot longer than an hour.

      BTW, I never said explicitly that "time to market" was one of the fundamental cloud benefits - I wouldn't use that term, as it's a little loaded. But I do think it gives you agility. Not everyone needs that - I was talking with a mining company a few years ago that had a decent enough infrastructure, but it was completely predictable. For every mine that opens, they need X servers. When a mine closes, they decommission X servers. They might still get *some* benefit from cloud, but it certainly wouldn't be worth the hassle of dealing with Openstack.

      --
      me@mzi.to
  5. You have to understand the technology by Stu101 · · Score: 2

    Those involved in Virtualisation probably (or should have) known this anyhow.

    The Hypervisor war is done. Pretty much everyone (VMware, MS, Citrix) have their new cloud based offerings that are agnostic towards the hypervisor that runs on the tin. If you have played with vClould Automation Center for example, there are multiple options for the hypervisor types including Citrix. The bottom line is there is not much more to add to to hyervisor and there is also less money in the hypervisor. It is an old (mature?) technology.

    The new hot button is the tools to manage the infrastructure and that is where the real war is going to be won or lost.

    --
    http://www.writeitfor.us - Writing IT for the IT generation.
  6. Re: Reasons not to use PayPal by Anonymous Coward · · Score: 4, Funny

    Anonymous - check
    "Shitload" - check
    Not afraid of excessive punctuation - check

    Congrats, you are my new financial advisor, call me.

  7. Re:So, what the hell is Open Stack? by afidel · · Score: 4, Informative

    In the paypal use case they could use it to take an internal image and burst it quickly into a cloud provider to scale up their capacity as they see demand spiking beyond what their internal resources can accommodate (not saying they are doing this, just that it's a possible application). For a typical enterprise it's useful to allow on demand lab creation for developers, snapshot the current production machines and generate an isolated sandbox that accurately mirrors the production environment. Automated unit testing is another popular use of API driven provisioning. If you can't find a use for automation in your environment it's either too small to qualify (I'm in this boat, we're fairly big at over 300 VM's at our main site but still too small for much automation) or you're not thinking outside your current box.

    --
    There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
  8. Re:So, what the hell is Open Stack? by Charliemopps · · Score: 4, Interesting

    Not if you want things to scale easily. I think a good example is SOE. They have a bunch of MMOs and years ago they virtualized their servers. Now it's completely irrelevant how many players are playing any particular game they have. If they have even 1 paying customer a limited amount of resources is dedicated to the server side of that players gameplay. If the population suddenly shoots up to 100,000 it just scales up assets dedicated to that game. It's brilliant really and is why SOE has been able to keep decades old games going for so long. None of their hardware is application specific.

    In a non-virtualized environment once the player population fell bellow a few thousand it would not longer be profitable for them to keep the game running and they've have to shut it down... and lose all those customers.

  9. Re:So, what the hell is Open Stack? by Anonymous Coward · · Score: 2, Informative

    Because many companies have many products and development efforts going on at any given time. Because many companies have weeks- or months-long provisioning cycles for new physical servers.

    Because as the summary says, having a platform allowing for rapid (consistent) deployment of infrastructure like development, test, and production servers allows companies faster time to market on their major development efforts.

    Because as you may have heard, software-defined infrastructure allows you to leverage automation to a much greater extent than with physical servers, allowing you to literally provision 100 identically configured servers in a couple hours for a server farm with just a couple commands.

    Because if you can't see the benefit in allowing that, you aren't qualified to express an opinion about the wisdom of the approach.

  10. Re:Your bias is showing. by mzito · · Score: 2

    Why would I be biased about OpenStack? I have no horse in this race. If VMware is "orders of magnitude" cheaper, good for them, go nuts. My point is just that there's a lot of moving parts in these decisions. What is true for one organization is not true for another. And look, those TCO calculations are messy.

    Here's a story I like to tell. A friend of mine is head of server architecture at a very large company. They're a huge IBM shop, and they were deciding where to make their primary server investments over the next three years. They can do linux on the mainframe, AIX on P-series, or Linux on X-series. Which one is going to be the cheapest? Best ROI for their workloads? Best TCO over five years? He's got three IBM server reps, one for each platform. He tells each of them that he wants an in-depth TCO study done for their respective platform, based on their current skillsets, investments, technologies in use, etc. They do interviews, asset inventories, it takes a month and a half. At the end, he gets three reports back, one from each sales rep, each conclusively proving that *their* platform would be the cheapest . The numbers worked out, everything was written out clearly, nothing was made up or fudged. It's just that if you make a couple of slightly different assumptions, or weight things differently, you can come up with a totally different result.

    Again, that doesn't mean that CS is wrong, it's just a question of where you want to spend your costs, and that TCO means different things to different people.

    One thing I will quibble with - $1b is their capex spend annually, that's capital outlays on hardware and software for their entire IT organization. He said they saved orders of magnitude on their people costs to operate. A fraction of a fraction of their capex spend.

    --
    me@mzi.to
  11. Re:So, what the hell is Open Stack? by Richard_at_work · · Score: 2

    Not really, there's a world of difference between the two. It's really a case of bringing portability, redundancy and scalability to a whole raft of applications which didn't have it before, and simplifying it dramatically in the process.

    I don't think any OS below the million dollar mark has supported moving running processes and environments between actual independent hardware nodes with little or no interruption, and now you can have that with a free OS and a little configuration.

    No longer do you have to wait for your vendor to send over a part in order to stop a critical system running in degraded mode - just allocate that VM to a sdifferent node in your cloud cluster, if it wasn't done automatically for you...

  12. Re:Uh.... is Google broken? by jekewa · · Score: 2

    Oh...you mean they didn't RTFA?

    Here, you lazy bums:

    OpenStack is an IT infrastructure framework made up of several components contributed under an open source license by multiple technology providers.

    --
    End the FUD
  13. Re:Uh.... is Google broken? by michrech · · Score: 2

    Thanks, Balmer! I'll keep that in mind next time...

    --
    bork bork bork!
  14. Re:So, what the hell is Open Stack? by tnk1 · · Score: 2

    I agree. When I read about their 15 minute rule I was like: "Well that's nice, but are deployments really where they are losing time in reacting to competitors?"

    Having been involved in production deployments for over 15 years now, I can tell you that deployment time is pretty much a non-issue with speed to market. Things like VMs and storage options and all of that make for easier, and less risky deployments, I agree. However, I have never heard of a piece of software that was ready for prod that was waiting for more than a simple scheduled maintenance window in order for it to go out. Maybe some of the more complex deployments might have required a week at most to practice and get the scripts right. That's it.

    Compare that to the weeks and months that marketing, product management, development, QA testers are working on features and it is insignificant.

    When I do a deployment, I'm perfectly willing to do 8 hours of prep work to make sure that my customers could get 5 minutes less downtime, and preferably no downtime. Rushing things into production from test is like spending your R&D budget on improving fuel efficiency from 50 to 51 MPG while you spend nothing on getting 12 MPG trucks up to a respectable 18 or something. Where is the real time savings?

    If this is Paypal's solution by itself, I predict that they will remain behind the curve going forward as well. Maybe they should figure out how to improve their development process?