Slashdot Mirror
Spark Advances From Apache Incubator To Top-Level Project
rjmarvin writes "The Apache Software Foundation announced that Spark, the open-source cluster-computing framework for Big Data analysis has graduated from the Apache Incubator to a top-level project. A project management committee will guide the project's day-to-day operations, and Databricks cofounder Matei Zaharia will be appointed VP of Apache Spark. Spark runs programs 100x faster than Apache Hadoop MapReduce in memory, and it provides APIs that enable developers to rapidly develop applications in Java, Python or Scala, according to the ASF."
Generally when Spark advances you get engine knock.
Only thing -- where do I get my big data?
Solr claims to be yet strictly fails to be a drop-in search engine for your website.
A former employer of mine, who didn't have a clue about Linux, Java or Open Source, bet the farm on Solr speeding up the report generation for his online service.
I don't want to tell you who this employer is because they provide a valuable service to the business community. But the owner of the company is a raging alcoholic, who devoted at least an hour at the end of each day for not having gotten Solr up and running yet, despite his not having lifted a finger to evaluate it before committing to the project.
If there is the slightest error in Solr's configuration, and you have logging enabled, it spews Java exception stack traces, but does not give you the first clue as to what you did wrong.
Stack traces are for developers not end-users, M'Kay? How about a diagnostic message?
I repeatedly asked for help on Stackoverflow but no one ever answered my questions. All I ever found were questions from other desperate Solr users, for the most part unanswered.
Before you commit to a technology, or your new bride, or your vote for a political candidate, put it or their name into google along with "sucks" just for grins.
for example, at the time "Solr sucks" got 600,000 hits.
It is appalling that software like that would be released to end users with the claim it is production quality software.
From time to time I see Solr coding gigs on the job boards. I never apply, I just say "You are doomed" to myself. Perhaps I should do the right thing, by sending a polite email to the hiring manager, suggesting he select some other solution to his problem.
Please mail me URLs of software employers.
enable developers to rapidly develop applications in Java
That's a contradiction.
And Tachyon, another component of Matei's Berkeley Data Analytics Stack, boosts Spark another factor of 2-8x by sidestepping JVM garbage collection issues.
Or after? Which is advanced? Anybody got a timing light? Anybody know what a timing light is? Does anybody really care? Know what time?
On one carefully selected benchmark, discounting a lot of things that matter (like data movement) spark performs better than Hadoop. Tech reports generated by the authors suggest that this is a corner case and that the variance in spark performance is wildly variable. Don't believe the hype.
Solr speeding up "report generation"? That's completely stupid. It's meant to search text and then link back to the "source" documents, it's not meant for report generation.
I am a HUGE solr fan and I think it's one of the most impressive open source projects I've ever seen. Yeah there are a ton of knobs, switches, and things for you to fiddle with but that's what makes it so great, you can do so much with it. At my job, we needed to do "on demand" indexing because it was basically impossible to index all of the stuff in the database all at once, so we wrote a custom "data import handler" to index people's information when a user picks them and it works unbelievably well. Solr handles over 100,000 "build this person's index" requests a day while also handling the search load without blinking. I'm a programmer and I think it's absolutely great.
Spark is a seriously awesome project. I have used it quite a lot in the last 3 months and I have to say, I love it. I currently use the spark shell as I would use awk. I have input files and I want to process stuff in it and output stuff out. That is probably the best thing about spark. Almost everyone I talked to is doing something different with it. Machine Learning with MLLib, graph processing with graphX.
Shh, or they'll add horizontal scrolling too!
his real problem was that his Microsoft SQL Server database had three hundred tables, many of which had three hundred columns.
It, as they say, "Grew Organically".
I never got a look at any of the C# .net code for their web application, but I was told that it had grown organically too.
The long-term plan was to scrap all the C#, then rewrite the entire thing in Java.
However, the company owner never seemed to actually clue in to that there was something wrong with the database.
I resigned in protest when I concluded he was a raging alcoholic. I've seen this many times before; he was never drunk during the work day but there were vast quantities of empty beer bottles, as well as ten lovingly preserved, quite large and empty hard liquor bottles.
the only room in the office that didn't look like a bomb went off in it, was a pristine, spotlessly clean room with a real nice pool table, as well as the kinds of liquor advertisement mirrors that one commonly sees in bars.
To top it off, the office did not have sufficient cooling. While I was right next to a door, my cubicle wall was right against the door. I'm pretty sure that's a fire code violation.
I am the 19th comment. *tips hat*
How does Spark compare with H2O http://0xdata.com/ from Cliff Click