Slashdot Mirror

Website: http://sortbenchmark.org/
PDF: MinuteSort with Flat Datacenter Storage

The sorts were accomplished using a heterogeneous
cluster consisting of 256 computers and 1,033 disks, di-
vided broadly into two classes: storage nodes and com-
pute nodes. Notably, no compute node in our system
uses local storage for data; we believe FDS is the first
system with competitive sort performance that uses re-
mote storage. Because files are all remote, our 1,470 GB
runs actually transmitted 4.4 TB over the network in un-
der a minute. No strong assumptions are made around
key or record lengths; keys and records of other lengths
can be handled with only a performance-neutral config-
uration change.

Summary
FDS is a general-purpose scalable parallel blob store
that exploits a full-bandwidth interconnect to expose the
entire cluster’s disk bandwidth to remote clients. The
sort performance results in this paper demonstrate the
power of the architecture: in both Daytona and Indy
sorts, the system reads the data remotely to the sort ma-
chines, sorts the data across the network, and writes it
remotely back to storage.
Performant remote file access imparts a flexibility ab-
sent in contemporary distributed storage systems. Be-
yond sort, FDS supports a broad variety of scalable large-
data applications. It does so without demanding that
cluster nodes balance compute and disk performance;
more importantly, it does so without demanding that ap-
plications observe locality constraints.

Website: http://sortbenchmark.org/
PDF: MinuteSort with Flat Datacenter Storage

The sorts were accomplished using a heterogeneous
cluster consisting of 256 computers and 1,033 disks, di-
vided broadly into two classes: storage nodes and com-
pute nodes. Notably, no compute node in our system
uses local storage for data; we believe FDS is the first
system with competitive sort performance that uses re-
mote storage. Because files are all remote, our 1,470 GB
runs actually transmitted 4.4 TB over the network in un-
der a minute. No strong assumptions are made around
key or record lengths; keys and records of other lengths
can be handled with only a performance-neutral config-
uration change.

Summary
FDS is a general-purpose scalable parallel blob store
that exploits a full-bandwidth interconnect to expose the
entire cluster’s disk bandwidth to remote clients. The
sort performance results in this paper demonstrate the
power of the architecture: in both Daytona and Indy
sorts, the system reads the data remotely to the sort ma-
chines, sorts the data across the network, and writes it
remotely back to storage.
Performant remote file access imparts a flexibility ab-
sent in contemporary distributed storage systems. Be-
yond sort, FDS supports a broad variety of scalable large-
data applications. It does so without demanding that
cluster nodes balance compute and disk performance;
more importantly, it does so without demanding that ap-
plications observe locality constraints.

A paper describing the test is here. TritonSort is the abstract method; GraySort and MinuteSort are concrete/benchmarked variations of TritonSort.

As TFA states, this is more about balancing system resources than anything else. The actual sort method used was "a variant of radix sort" (page two of the linked PDF). Everything else was an exercise in how to break up the data into manageable chunks (850MB), distribute the chunks to workers, then merge the results. That was the real breakthrough, I think. But I wonder how much is truly a breakthrough and how much was just taking advantage of modern hardware, since one of the major changes in MinuteSort is simply avoiding a couple of disk writes by keeping everything in RAM (a feat possible because that much RAM is readily available to a single worker).

Regardless, this kind of work is very cool. If web framework developers (for example) paid greater attention to balancing data flow and work performed, we could probably get away with half the hardware footprint in our data centers. As it stands now, too many COTS -- and hence, often-used -- frameworks are out of balance. They read too much data for the task, don't allow processing until it's all read in, etc.. This causes implementors to add hardware to scale up.

• 10Gbps Networking
• 52 servers x 8 cores each = 416 CPUs
• 24 GB RAM per server = 1,248 GB
• ext4 filesystem on each, presumably with hardware raid

I think this is cool, but.... how fast is it in a more practical situation?

source

When I read the summary I thought what's the big deal if the 1 TB of data only contained two records 0.5 TB each. Then I saw that kdawson wrote the summary. So I browsed over here and saw that the impressive thing is that they sorted 1,000,000,000,000 records of 100 bytes each with 10 byte keys.

The system is based on a Zotac IONITX-A board, equipped with an Atom 330. This processor consumes more than three times the power of an N270 (8 W TDP) but supports two cores and four hardware threads. The main advantage of this system is that its nVidia Ion chipset provides four SATA ports that can handle the SSD transfers at full speed. Moreover, it allows two DIMMs for a total of 4 GiB of RAM. The 64 bit logical address space is less prone to fragmentation, which we experienced on the 32-bit Atom N270.

Which really shows that they got a 3 times improvement by going with sub-optimal choices because that's all that was available. If there existed a two or 4 core N270 64 bit that had the necessary RAM and SATA interfaces the results would have been even better. Just think if there was an ARM chipset and motherboard that had the needed interfaces. So there is quite a ways to go with power efficient computing and that's a good thing. I'm still waiting for a low power desktop Cortex A8 or A9 system to be widely available with all the standard PC parts. It's just going to take someone putting all the pieces together right.

More details are here. Looks like it's a tweaked merge-sort.

Java is faster then C++ for any non trivial use case. This is because of 2 reasons:
- It compiles more natively then C++: C++ is compiled for just "Windows", while Java is hotspot compiled at runtime (where necessary, after a startup delay) to "Windows XP SP3 Intel processor".
- A garbage collector is faster for non-trivial use cases then malloc because it can free entire chunks of memory at once. When someone walks into your living room with muddy shoes, it's more efficient to mop the entire room then it is to clean only the footprints with a toothbrush.

As a planning solving programmer, I 've seen this in practice.
Planning problems are problems that run for hours and do virtually no IO, they simple take up all the CPU power and evaluate an exponential of billions solutions.

Real-world use case performance competitions prove that Java implementations are faster then C++ implementations:
- Terrabyte sort benchmark: http://sortbenchmark.org/
- International timetabling competition: http://www.cs.qub.ac.uk/itc2007/

Java does have downsides over C++: it has a terrible warmup time and it loves to grab and hold lots of RAM, because garbage collectors work faster with more RAM. So in use cases like gedit/kate/jedit where startup time is more important than performance, I'd stick with C++.

Google's sorting results from last yeat (link) are much faster; they did a petabyte in 362 minutes, or 2.8 TB/sec. They minute sort didn't exist last year, but Google did 1TB in 68 seconds last year, so I think it may be safe to assume that they could do 1 TB in under a minute this year. Google just hasn't submitted any of their runs to the competition.

From the sort benchmark page, the list the winning run as Yahoo's 100TB run, leaving out the 1PB run; that implies the 1PB run didn't conform to the rules, or was late, or something.

People have commented that this is a "who has the biggest cluster" competition; the sort benchmark also includes the 'penny' sort, which is how much can you sort for 1 penny of computer time (assuming your machine lasts 3 years), and 'Joule' sort, how much energy does it take you to sort a set amount of data. Not surprisingly, the big clusters appear to be neither cost efficient nor energy efficient.