Microsoft Goes In For Hadoop

← Back to Stories (view on slashdot.org)

Posted by timothy on Thursday October 13, 2011 @03:02AM from the well-it-is-a-nice-name dept.

Frankie70 writes that after more than three years, Microsoft has "finally learned to stop worrying and love Hadoop." Frankie70 excerpts from the linked Wired article: "Any aversion to Hadoop disappeared on Wednesday, when the company announced that it will integrate the platform with future versions of its relational database, SQL Server, and its platform cloud, Windows Azure, an online service for hosting and readily scaling applications. The company is now working to port the Hadoop platform to Windows."

44 of 67 comments (clear)

Min score:

Reason:

Sort:

Hadoop is written in Java by binarylarry · 2011-10-13 03:06 · Score: 5, Funny

So what they mean is, they're going to do a search and replace to make it compile as a C# application.

--
Mod me down, my New Earth Global Warmingist friends!
1. Re:Hadoop is written in Java by allenw · 2011-10-13 08:19 · Score: 1
  
  Actually, there is an ever increasing amount of JNI (read: C) code in Hadoop that is in the critical path for security and performance features. Most of that code is not very portable. So either MS is going to pay for some major overhauling of that code, completely new code/branch to replicate that functionality or MS Hadoop is going to be severely lacking in features/performance.
2. Re:Hadoop is written in Java by shutdown+-p+now · 2011-10-13 10:15 · Score: 1
  
  If that C code is well-written, it shouldn't be hard to port it over - the porting would have to be done at the Java/C boundary, and .NET actually has it much simpler thanks to P/Invoke.
3. Re:Hadoop is written in Java by allenw · 2011-10-13 11:21 · Score: 1
  
  It isn't. There is an incredible overuse of glibc/Linux-isms to the point that even porting it to another UNIX is difficult.
4. Re:Hadoop is written in Java by shutdown+-p+now · 2011-10-13 11:50 · Score: 1
  
  Well then, perhaps those guys will clean it up architecturally while they're porting it, and submit the changes upstream.
5. Re:Hadoop is written in Java by Ed+Avis · 2011-10-13 21:48 · Score: 1
  
  Since it's written in Java, they can just run it on the .NET virtual machine using IKVM.
  
  --
  -- Ed Avis ed@membled.com
Re:We know, we know... by Tsingi · 2011-10-13 03:12 · Score: 1

How about Spiteful?
Does taking the source from an Apache licensed product and converting it from java to .net mean you no longer have to observe the license?
I'd say no, I wonder what M$ will say.
Not wanting to put a dampener on things... by bernywork · 2011-10-13 03:13 · Score: 1

I think MS getting involved with Open Source is great, but....
We've seen the way that they work before, embrace and extend... This hasn't worked out that well for them before, but you have to ask if there is an alterior motive in there...

--
Curiosity was framed; ignorance killed the cat. -- Author unknown
1. Re:Not wanting to put a dampener on things... by Attila+Dimedici · 2011-10-13 03:33 · Score: 1
  
  I am not sure what an alterior motive is, but I am quite sure that MS has an ulterior motive for this. The only question is whether or not that ulterior motive is detrimental to the Open Source community.
  
  --
  The truth is that all men having power ought to be mistrusted. James Madison
2. Re:Not wanting to put a dampener on things... by bernywork · 2011-10-13 03:41 · Score: 1
  
  > alterior
  bad spelling, ulterior. Sorry, my bad.
  
  --
  Curiosity was framed; ignorance killed the cat. -- Author unknown
3. Re:Not wanting to put a dampener on things... by jbolden · 2011-10-13 04:16 · Score: 1
  
  They are quite publicly indicating their intention is to embrace and extend:
  1) Get Hadoop to work on Windows servers
  2) Create a Windows server management interface for Hadoop
  3) Create SQL Server extension to manage Hadoop.
  So we don't have to speculate, that's what they say they are doing. That being the case all that stuff might be useful for Hadoop.
4. Re:Not wanting to put a dampener on things... by Forbman · 2011-10-13 04:16 · Score: 1
  
  Sell more Windows Server & SQL Server Enterprise/Data Center licenses?
5. Re:Not wanting to put a dampener on things... by jbolden · 2011-10-13 04:17 · Score: 2
  
  I think they are telling the truth about their goals:
  1) Get Hadoop to work on Windows servers
  2) Create a Windows server management interface for Hadoop
  3) Create SQL Server extension to manage Hadoop.
  And the motive is:
  a) Sell server licenses
  b) Sell SQL Server licenses
6. Re:Not wanting to put a dampener on things... by mandelbr0t · 2011-10-13 04:44 · Score: 1
  
  This is a smart move by MS. Microsoft is not working internally on any sort of NoSQL server, so they support an existing project that complements their own product. The very obvious goal is to integrate Hadoop with SQL Server management tools. The upshot is that Hadoop gets a leg up on their competitors (Yahoo! PNUTS, Google BigTable) and Microsoft sells more SQL server licenses. Seems to me to be a win/win.
  
  --
  "Please describe the scientific nature of the 'whammy'" - Agent Scully
7. Re:Not wanting to put a dampener on things... by jimicus · 2011-10-13 04:58 · Score: 1
  
  You would be amazed how many people go nuts over the latest F/OSS platform du jour... and then complain that it runs first and foremost under Linux.
  Even if they're never going to go anywhere near the underlying OS anyway, still that gets brought up.
  Windows Server licensing is quite lucrative for Microsoft. So if they can now announce "Hadoop: Now certified for Windows (TM) Server" they can sell more licenses for Windows Server.
8. Re:Not wanting to put a dampener on things... by aztracker1 · 2011-10-13 05:21 · Score: 1
  
  I think it's to integrate a map/reduce structure into SQL server... I haven't RTFA, but that is about it... I wouldn't necessarily expect them to use Hadoop directly, but to support Hadoop's interfaces. My $0.02 on this. I know a lot of people are using MongoDB, and other document centric datastores lately, and MS is moving to compete in their tool space. More power to them, doesn't mean it'll be my first choice.
  
  --
  Michael J. Ryan - tracker1.info
9. Re:Not wanting to put a dampener on things... by Anonymous Coward · 2011-10-13 06:51 · Score: 1
  
  NASQAQ uses sql server http://www.computerworld.com/s/article/106050/Microsoft_unwraps_flagship_database_SQL_Server_2005
  DirectEdge - 4th largest stock exchange uses sql server
  http://blogs.technet.com/b/dataplatforminsider/archive/2011/06/03/fourth-largest-us-stock-exchange-direct-edge-looks-to-sql-server-parallel-data-warehouse-for-big-data-needs.aspx
10. Re:Not wanting to put a dampener on things... by allenw · 2011-10-13 08:29 · Score: 1
  
  This isn't about Microsfot getting involved with open source. This is about Microsoft not getting left out. Beyond the countless startups, Apache Hadoop already has major players like Amazon, Dell, EMC, HP, IBM, NetApp, Oracle, VMware, ... trying to make a dent in the community in some form or another. Hell, I have a SuperMicro catalog on my desk emblazoned with the Apache Hadoop logo all over it. Like Oracle, they are coming in very late to the party and now need to play catch-up. Buying off Hortonworks is a very fast way to do that.
11. Re:Not wanting to put a dampener on things... by Thing+1 · 2011-10-15 04:00 · Score: 1
  
  I am not sure what an alterior motive is, but I am quite sure that MS has an ulterior motive for this.
  My ex-girlfriend was an ulterior decorator.
  
  --
  I feel fantastic, and I'm still alive.
Re:We know, we know... by MozeeToby · 2011-10-13 03:13 · Score: 1

I have to wonder where the first line of the article came from. If it's from the author, nothing to see here, but if it's a near direct quote from the source it sounds more like a shoutout to Dr. Strangelove, which could indicate that they are still terrified of it but are embracing it because they don't feel they have any choice in the matter.
First... by knuthin · 2011-10-13 03:17 · Score: 1

First they ignore you.
Then they laugh at you.
Then they port you on their platform.
Then you win.

Original: http://bit.ly/o3V3cA [Google Books]

--
Some apps are WYSIWYG. Some others are WYSIWTF.
1. Re:First... by camperdave · 2011-10-13 03:56 · Score: 1
  
  First they ignore you.
  Then they laugh at you.
  Then they port you on their platform.
  Then you win.
  
  First they ignore you.
  Then they laugh at you.
  Then they port you on their platform.
  Then they add some convenient feature that they only make available on their platform
  Then they win
  
  FTFY
  
  --
  When our name is on the back of your car, we're behind you all the way!
2. Re:First... by knuthin · 2011-10-14 03:22 · Score: 1
  
  That was one another depressing possibility.
  
  Think I will go into a corner and cry for a minute now. :/
  
  --
  Some apps are WYSIWYG. Some others are WYSIWTF.
Heh by Hatta · 2011-10-13 03:27 · Score: 4, Funny

Someone should trick Timothy into reposting this article. Then he'd be duped into posting a dupe about hadoop.

--
Give me Classic Slashdot or give me death!
1. Re:Heh by darthdavid · 2011-10-13 03:36 · Score: 1
  
  Someone should trick Timothy into reposting this article. Then he'd be duped into posting a dupe about hadoop.
2. Re:Heh by Anonymous Coward · 2011-10-13 03:53 · Score: 1
  
  Congrats, you've demonstrated enough brain-power for kindergarten. If a kid stacks a blue lego block on top of another, you are able to imitate and stack a 3rd blue lego block.
3. Re:Heh by Gilmoure · 2011-10-13 05:10 · Score: 1
  
  *golf clap*
  
  --
  I drank what? -- Socrates
4. Re:Heh by robot_love · 2011-10-13 05:31 · Score: 1
  
  And then I'd file a false complaint against it being a copyright violation in France.
  It would be a Hadopi dupe dupe Hadoop.
  But I digress...
  
  --
  .there is enough of everything for everyone.
5. Re:Heh by Anomalyst · 2011-10-13 05:43 · Score: 1
  
  *golf clap*
  Something you get from being in the showers with Tiger Woods?
  
  --
  There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
Embrace. Extend. Extinguish. by mrflash818 · 2011-10-13 03:33 · Score: 3, Interesting

"Those that do not learn from history are doomed to repeat it."

"Embrace, extend and extinguish,"[1] also known as "Embrace, extend and exterminate,"[2] is a phrase that the U.S. Department of Justice found[3] was used internally by Microsoft[4] to describe its strategy for entering product categories involving widely used standards, extending those standards with proprietary capabilities, and then using those differences to disadvantage its competitors.
http://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish

--
Uh, Linux geek since 1999.
1. Re:Embrace. Extend. Extinguish. by mrflash818 · 2011-10-13 03:41 · Score: 1
  
  Microsoft allowed the pair to continue their contributions to the open source project, and Powerset, which was rolled into Redmond’s Bing search engine, continued to run atop Hadoop.
  This made Bing one of the first “shipping” Microsoft products to actually include open source code. But somewhere along the way, Microsoft moved the engine onto a proprietary platform...
  "Microsoft allowed the pair" -- Here
  "which was rolled into Redmond's Bing search engine" -- It
  "But somewhere along the way, Microsoft moved the engine _onto_a_proprietary_platform_" -- Comes *sigh*
  
  --
  Uh, Linux geek since 1999.
as usual, the summary here is lacking by tomzyk · 2011-10-13 03:34 · Score: 4, Insightful

its relational database, SQL Server, and its platform cloud, Windows Azure, an online service for hosting and readily scaling applications
That's wonderful that the summary mentions what "SQL Server" and "Azure" are... but why no mention of wtf "Hadoop" is?
Why do I need to RTFA just to find out what we're talking about here?

Hadoop — an open source platform for crunching epic amounts of a data across an army of dirt-cheap servers

--
Karma: NaN
1. Re:as usual, the summary here is lacking by slim · 2011-10-13 03:46 · Score: 1
  
  The Wired article tells you what Hadoop is, because it's written by journalists, and aimed at a broad readership.
  The /. summary tells you what SQL Server and Azure are, because the descriptions arbitrarily happen to occur in the Wired paragraph that's been quoted.
  The /. summary does not tell you what Hadoop is, because (unlike Wired readers), /. readers are expected to have some basic knowledge of the software world. If you think Hadoop is obscure, maybe this isn't the site for you?
2. Re:as usual, the summary here is lacking by camperdave · 2011-10-13 04:04 · Score: 1
  
  Me too! I had to double click on Hadoop, then right click and choose 'Search Google for "Hadoop"' from the context menu. Then I had to switch tabs and scan for the wikipedia site, click on that, and then skim through the wiki for a few seconds just to figure out whether it was worth it to read anything beyond the summary.
  
  Sadly, this is happening more and more with Slashdot.
  
  --
  When our name is on the back of your car, we're behind you all the way!
3. Re:as usual, the summary here is lacking by Anonymous Coward · 2011-10-13 04:15 · Score: 1
  
  I'm with the OP. Quoting an article is fine, even with descriptions one should know... but if an article is primarily about something that isn't common, then a nice little blurb about it would save AT LEAST TWO PEOPLE a wiki call. Hadoop isn't as ubiquitous as you believe. Coming from a heavy consulting background in coding, I've never seen this.
  Oddly, by your logic, /. readers don't know what SQL Server and Azure are. I'll put dollars to doughnuts that more people USE SQL Server than KNOW what Hadoop is. Looking at the WIKI, I'm also willing to bet that you aren't pronouncing Hadoop correctly (not unlike the original GIF pronunciation).
4. Re:as usual, the summary here is lacking by CannonballHead · 2011-10-13 06:46 · Score: 1
  
  The /. summary tells you what SQL Server and Azure are, because the descriptions arbitrarily happen to occur in the Wired paragraph that's been quoted.
  So what you're saying is .... the person who wrote the summary doesn't know how to write a good summary or pick a good quote. :) (no, I'm not new here...)
5. Re:as usual, the summary here is lacking by slim · 2011-10-14 02:03 · Score: 1
  
  Oddly, by your logic, /. readers don't know what SQL Server and Azure are.
  Nope. As I said, the definitions for those arbitrarily happen to be in the quote chosen for the summary. But those definitions aren't the pertinent part of the quote.
Dont look behind that curtian! by Lumpy · 2011-10-13 03:41 · Score: 1

"the company plans to eventually release its work back to the open source community."
That is a bit too vague... because 100 years from now is "eventually"

--
Do not look at laser with remaining good eye.
1. Re:Dont look behind that curtian! by jbolden · 2011-10-13 04:20 · Score: 2
  
  If you look at their todo list they could release it constantly it won't matter. What they are doing is essentially creating extensions for their commercial products that work with Hadoop. I think they have every intention of trying to get the small parts that need to be in Hadoop back into the main tree.
Dryad by Utopia · 2011-10-13 04:31 · Score: 1

Microsoft has a equivalent to Hadoop known as Dryad.
They should have open-sourced Dryad a long time ago.
I wonder what is going to happen to Dryad with this focus on Hadoop.
1. Re:Dryad by mandelbr0t · 2011-10-13 05:07 · Score: 3, Informative
  
  Dryad is not quite Hadoop. From their whitepaper:
  
  We can map the whole relational algebra on top of Dryad, however Dryad is not a database engine: it does not include a query planner or optimizer; the system has no concept of data schemas or indices; and Dryad does not support transactions or logs
  I can see how Hadoop would supplement their own research in this field.
  
  --
  "Please describe the scientific nature of the 'whammy'" - Agent Scully
Denver Open Source Group meets at MicroSoft by peter303 · 2011-10-13 05:14 · Score: 1

In the beginning I thought it was ironic. But its very generous of them to provide a free meeting room to our open-source computer study group. And MSFT people attend, but dont speak often.
Microsoft Goes Hadooken by Spy+Handler · 2011-10-13 09:18 · Score: 1

that's what i thought it said...... MS going all street fighter
The cool parts by davevr · 2011-10-13 15:24 · Score: 1

The two cool parts of this announcement:

1) They are contributing the bits needed to make it work on windows back to open source (Hortonworks is helping that make sure that goes smoothly)
2) They are making JavaScript a first-tier language for writing map/reduce jobs, and contributing THAT work back to the community.

That is awesome.