Slashdot Mirror

← Back to Stories (view on slashdot.org)

Google Launches Cloud Dataproc, a Managed Spark and Hadoop Big Data Service

Posted by samzenpus on from the new-player dept.

An anonymous reader writes: Google has a new cloud service for running Hadoop and Spark called Cloud Dataproc, which is being launched in beta today. The platform supports real-time streaming, batch processing, querying, and machine learning. Techcrunch reports: "Greg DeMichillie, director of product management for Google Cloud Platform, told me Dataproc users will be able to spin up a Hadoop cluster in under 90 seconds — significantly faster than other services — and Google will only charge 1 cent per virtual CPU/hour in the cluster. That's on top of the usual cost of running virtual machines and data storage, but as DeMichillie noted, you can add Google's cheaper preemptible instances to your cluster to save a bit on compute costs. Billing is per-minute, with a 10-minute minimum."

18 comments

  1. Yet... by Anonymous Coward · · Score: 0

    They throttle my dart reports. Big data... yeah... right.

  2. Big data... by Anonymous Coward · · Score: 0

    Yeah, right... this is the same company that throttles dart reports.

  3. How to max out someone's billing by msobkow · · Score: 1

    So in order to max out someone's billing, just run a query that will take half a second or few once every ten minutes to make sure that "ten minute minimum" is applied throughout every hour of the day. :(

    --
    I do not fail; I succeed at finding out what does not work.
    1. Re:How to max out someone's billing by Anonymous Coward · · Score: 0

      The Cloud is the best way to save money! It's the commodification of the IT infrastructure (and staff)! Cut cut cut!

      Until you get the first bill.

      Or the Cloud goes down, Titanic style.

      2013: http://www.zdnet.com/article/amazon-web-services-suffers-outage-takes-down-vine-instagram-others-with-it/
      2015: http://www.theinquirer.net/inquirer/news/2426828/aws-goes-down-takes-netflix-and-reddit-with-it

    2. Re: How to max out someone's billing by Anonymous Coward · · Score: 1

      And we alll know that self-hosted services never go down.

    3. Re:How to max out someone's billing by Anonymous Coward · · Score: 0

      Take your FUD and shove it. Of course the cloud had issues early on. If we'd applied that standard, nobody would run Linux either. It had plenty of issues with its early versions and, honestly, still does.

    4. Re:How to max out someone's billing by msobkow · · Score: 1

      The "cloud" still has the same issues people have complained about for years:

      - Usually no effective way to back up large data sets to local media
      - Usually no effective way to restore large data sets from local media
      - Unpredictable costs
      - Single point of failure: one service provider
      - "All or nothing" failures
      - Virtually impossible to switch providers
      - Performance limited by network bandwidth
      - Difficulty loading large data sets for initial deployment
      - At the mercy of the provider; often no way to prioritize deployments if repair/recovery are necessary
      - Security at the discretion of the provider
      - Nothing more than good old fashioned clusters, just hosted off-site with fancy new buzzwords

      --
      I do not fail; I succeed at finding out what does not work.
    5. Re: How to max out someone's billing by Anonymous Coward · · Score: 0

      I've never, ever, ever seen a self hosted service go down. Never.

      I'd trust Google to keep their machines running more than I'd trust most of the IT departments I've worked with over the years!

    6. Re: How to max out someone's billing by Anonymous Coward · · Score: 0

      Apparently slashdot strips tags now.... for those who were wondering... first sentence is a sarcasm element

    7. Re: How to max out someone's billing by Anonymous Coward · · Score: 0

      Apparently even if you separate the tag containers it gets stripped.... .... *first sentence goes here*.... ......

      Honestly now I'm just curious to see if this works....

    8. Re: How to max out someone's billing by Anonymous Coward · · Score: 0

      nup... didn't work.

    9. Re: How to max out someone's billing by msobkow · · Score: 1

      <rolleyes>Of course you could try using the &lt;/&gt; explicitly...</rolleyes>

      --
      I do not fail; I succeed at finding out what does not work.
    10. Re: How to max out someone's billing by Anonymous Coward · · Score: 0

      Apparently slashdot strips tags now....

      If by "now" you mean "for more than 20 years", then yes, you can't directly type < or >. You're presumed to know how to do it, and once upon a time all slashdotters did.

    11. Re:How to max out someone's billing by Anonymous Coward · · Score: 0

      This is why container systems are becoming so popular. Distributed containers help resolve the following issues:

      Single point of failure: service provider
      "All or nothing" failures
      Virtually impossible to switch providers
      Performance limited by network bandwidth (yes, always, but more distributable)
      At mercy of provider -- distribute over providers

      Things that are potentially exacerbated:
      Difficulty loading large data sets (with distributed containers data needs to be sharded or duplicated)
      Security at discretion of provider -- problems on the bare metal system of any of the providers would expose your containers

      I agree that "the cloud" is buzzword-stupid, but containers (combined with a distributed software design) can provide significant improvements over a local cluster.

    12. Re:How to max out someone's billing by jonbally7631 · · Score: 1

      Nowadays Billing is tough and old way to pay on hand. You should have to pay online via card or something else. Pay for securing the data in the drive.

  4. dataproc by robi5 · · Score: 1

    How is it pronounced?

    1. Re:dataproc by Anonymous Coward · · Score: 0

      The term "dataproc" is an abbreviation for "data processing", so it's pronounced "Throat Warbler Mangrove".

  5. Hey quiet now by Anonymous Coward · · Score: 0

    you are angering amazon's/rackspace's management