Flink and Spark address similar use cases and have similar APIs, but the technology under the hood differs a bit.
Flink serializes data to memory and works a lot on binary representations. This also makes (partial) spilling to disk easier if the system runs out of memory. Flink's processing engine uses pipelined processing (like many DBMS) which allows for true streaming and batch processing in a single engine. Iterations are build-in operators in Flink which means that data continuously flows in cycles. A special iteration operator reduces the number of computations as iteration count increases which gives very good performance for certain use cases such as some ML algorithms.
Flink is a parallel data processing engine similar to Hadoop and Spark with some unique features: 1) combines realtime stream and batch processing, 2) features an DBMS-style optimizer, 3) in-memory processing which goes gracefully to disk if memory is scarce, 4) provides special operators for iterative processing,...
Check out http://flink.apache.org/ for details.
Slashdot Mirror
:-) The original intention was the German meaning, which is agile or swift (Flink was started in Berlin). But clever is great as well ;-)
Flink and Spark address similar use cases and have similar APIs, but the technology under the hood differs a bit. Flink serializes data to memory and works a lot on binary representations. This also makes (partial) spilling to disk easier if the system runs out of memory. Flink's processing engine uses pipelined processing (like many DBMS) which allows for true streaming and batch processing in a single engine. Iterations are build-in operators in Flink which means that data continuously flows in cycles. A special iteration operator reduces the number of computations as iteration count increases which gives very good performance for certain use cases such as some ML algorithms.
Flink is a parallel data processing engine similar to Hadoop and Spark with some unique features: 1) combines realtime stream and batch processing, 2) features an DBMS-style optimizer, 3) in-memory processing which goes gracefully to disk if memory is scarce, 4) provides special operators for iterative processing, ...
Check out http://flink.apache.org/ for details.