Slashdot Mirror

Without sociology skills (my blog) on a data science team, hypothesis formation and ability to model clients will suffer. It would seem particularly important for a people-focused company like Dice.com.

My blog post today argues that it takes as much or less time to train an existing employee on new skills than it does to train a new employee on the company's domain knowledge.

I.e., yes, companies should be training instead of churning. And training doesn't even cost anything any more except for the paid time to do it -- everything is online now.

Statistically, women are bad at spatial reasoning. There are many sociological and political reasons for this, of course, and there is even a natural component. Even the same woman, when at a point in her cycle where testosterone is low, performs worse at spatial reasoning than when her testosterone is high.

But regardless of the source, the good news is that spatial reasoning can be taught.

For 20+ years, HPC systems have relied on the same conservative design of compute separated from storage, connected by Infiniband. Hadoop kind of shook up the HPC world with its introduction of data locality, especially as scientific use cases have involved larger data sets that distributed data storage is well-suited for. The HPC world has been wondering aloud how best and when to start incorporating local data storage for each node. Summit introduces some modest 800GB non-volatile storage per node for caching (which they call a "Burst Buffer"), but no bulk data storage.

I blogged about how the Summit design seems very conservative, especially for a system to be delivered in 2018, and especially for a supercomputer that is billed to be the most powerful in the U.S. if not the world.

From the linked piece:

In hindsight, his remark was a clear sign that the marketing hype around "big data" had peaked.

This is true, and it provides the context missing from TFS: "Big Data" is over as a marketing term. But as technological term and as far as actual implementation, it is the status quo and forevermore will be.

From a technological perspective, "Big Data" has a simple definition: more data than can be stored on a single machine. And this need will only grow as hard drives and maybe even SSDs plateau while of course enterprise data only grows.

Indeed, TFA itself states (that TFS omitted):

A particularly hot sector has matured around Hadoop, an open-source analytics software platform. Many tech companies are writing software to make Hadoop industrial strength and integrate it with new and existing types of databases.

So, from TFA itself: Hadoop is hot, but the term "Big Data" is not.

Data Science is not "dead", but I've blogged my response: The End of Data Science as We Know It

Spark runs programs 100x faster than Apache Hadoop MapReduce in memory

And Tachyon, another component of Matei's Berkeley Data Analytics Stack, boosts Spark another factor of 2-8x by sidestepping JVM garbage collection issues.