Why Reactive Programming For Databases Is Awesome

Nerval's Lobster writes "'Building on many lessons learned from spreadsheets, functional languages and model-driven application development environments, Reactive Programming (RP) is a recent approach to creating applications,' Val Huber, CTO of Espresso Logic, writes in a new column. 'In the RP paradigm, you can create complex applications from a series of simple declarative statements. Many of the details of implementation and work of gluing together various sorts of application constructs can be hidden because of the declarative approach.' He goes on to argue that RP makes maintenance easier and applications more adaptable, because RP offers automatic dependency management and reuse; at the same time, he also acknowledges that RP has some shortcomings. What say you?"

Comments

I for one would love to see DBs be more like Excel

[idioto]

I think excel is one of the greatest programs ever and reactive programming sounds great. This article isn't very specific as to how to get started though. Is it a concept, a product or something I have to implement myself?

Sounds like VHDL and similar approaches...

[Lisias]

Interesting.

Code modeled in RP sounds like VHDL, Verilog et all - the languages used to model modern Programmable Logic Arrays, Programmable Gate Arrays and Programmable Logic Devices - aka, hardware.

No doubt this approach has its merits, but this kind or "programming" has its gotchas - a lot of gotchas.

I stand aside for now, looking with great interest.

Spreadsheets?

[Animats]

Did this guy just reinvent spreadsheets? There's something to be said for this, but having written in Prolog, which works that way, the 'reactive programming' people have to make a better case than the article does.

On the other hand, one of the big problems in databases is change notification. Microsoft at one point had a research project on this. The concept was that you could make a request to be notified when something of interest changed in the database. This was expressed as an SQL SELECT statement. When some change was made that would affect the output of your pending SELECT, you'd be notified. The problem was to figure out some efficient way to do this with thousands or millions of pending SELECT statements.

Finance people use notifications like that all the time. Limits on stocks, limits on combinations of stocks, changes in key ratios, that sort of thing. But the systems for that are specialized, a special case of a more general problem. The most general form of the problem is that B wants to know when something has changed at A. Most of the solutions for this have problems. Polling is either inefficient or slow. B relying on A to message them on a change has a failure problem. Both approaches can result in overloading A with checks for events that aren't happening.

Re:Spreadsheets?

[Anonymous Coward]

A trigger on INSERT or UPDATE of a watched entity should kick off an INSERT into a message queue, and a trigger on INSERT in the message queue should run a script to send the message where it needs to go. Now you have a self-managing, self-logging, asynchronous system that only uses system resources when needed.

Don't reinvent the wheel. Do learn your craft and become a master at it.

This was solved decades ago. Next question, please.

Re:Spreadsheets?

[sribe]

A trigger on INSERT or UPDATE of a watched entity should kick off an INSERT into a message queue, and a trigger on INSERT in the message queue should run a script to send the message where it needs to go. Now you have a self-managing, self-logging, asynchronous system that only uses system resources when needed.

Yes, exactly. OK, sometimes it can actually be a little tricky to figure out who needs notification and what data they need, but that's application/business logic that just needs to be worked out. (Example: client was not interested in a particular "master" record, and therefore has not received it, but the addition of a particular "detail" record means that client is now interested, so sending the changed record is not enough, the update needs to include the master and all related details. And so on and so forth, for a really rich app with rich workflow, there winds up being a surprising amount of logic required to get the right updates but no more than needed out to each client.)

This was solved decades ago. Next question, please.

Well... No, not really, at least not that I know of. The ability for "trigger on INSERT in the message queue should run a script to send the message where it needs to go" has actually *not* been around for decades. For way too long, this had to be a process *polling* the message queue. Triggers were limited to whatever "procedural SQL" flavor your db had, and had no access to file or network resources. Now you can write triggers in many languages, sometimes with full access to the system. Now you can from a SQL trigger send a notice, which a client can listen for via select/kqueue/libevent etc. But that's all much more recent.

Re:I for one would love to see DBs be more like Ex

[rev0lt]

Usually database manuals calls them triggers.

Spreadsheet programming

[ilsaloving]

I can see this being useful for problems that are extremely linear and require extreme parallelization on large quantities of data, but that's about it.

I've done this 'methodology' many times using Excel. I did it because I I needed to give the spreadsheet to other people, and wanted to avoid having Office nag about 'potentially dangerous VBA'.

It makes it very easy to see *exactly* how your data is flowing, which is a bonus. It also uses a *lot* more ram because you are now maintaining a permanent block of memory for every single operation, for every different piece of data you are coding against. Of course, the second you get a cyclic dependency, the whole thing blows up.

But this method of 'programming' was a natural and convenient extension based on how Excel (or any spreadsheet) operates. Nothing more.

Is it just me, or does it seem like everyone now-a-days is trying very hard to come up with new methodologies and paradigms and web 6.5isms, so they can get their 5 minutes in the lime light?

Re:Spreadsheet programming

[benjymouse]

I can see this being useful for problems that are extremely linear and require extreme parallelization on large quantities of data, but that's about it.

That's the danger: Relying on your own lack of imagination to support your argument. If you cared to google a little you would realize that there are many other areas where RP could be relevant.

Erik Meijer who pioneered the RP push at Microsoft and which lead to the Reactive Extensions for C# and JavaScript (on which the current implementations in Java are based) wrote an article with a thoughtful title: "Your mouse is a database". I could as "so is your finger".

The idea is that a mouse is a series of "rows" with mouse positions and button states. Instead of viewing them as series of *events*, RP views them as streams of objects. That allows you to apply filters, actions, transformations etc.

Suddenly, the events become *composable*.

Think of a touch interface. The touch surface fires events. But if you view the events as streams of objects you can define filters and transforms which let only certain "events" through, like a stream which accepts objects starting/resetting at the "touch" but only accepts messages as long as they describe an ever longer diatance in a certain direction until the finger is lifted again. Now you have a composable stream of "flicks". You can define similar streams which will filter/generate circular moves, pinching with 2, 3 or 4 fingers.

Think of the problem of cache eviction. You can define sweeping/eviction algorithms as (simple) streams.

Think of asynchronous programming in a web browser. Parred with web sockets (or in .NET the awesome SignalR which uses web sockets but automatically falls back to other channels when web sockets are not available) you can build responsive, asynchronous interfaces where you *declare* what should happen as server messages are received.

Think of a dealer system where the dealer wants to track certain papers, positions, rates. RP enables you to build a chained, composable filter where alarms are raised (alarms being messages on streams) when a rate changes more than a certain percentage within a given time period.

I've done this 'methodology' many times using Excel.

Sure. Haven't we all?

It also uses a *lot* more ram because you are now maintaining a permanent block of memory for every single operation, for every different piece of data you are coding against.

Not neccesarily . In .NET, Reactive Extensions are integrated with LINQ. As a supplement to IObservable, Reactive Extensions also define IQbservable (a *queryable* observable). Analogous to LINQ for IQueryable, LINQ applied to a IQbservable is actually capable of combine criterias and transformations "up the chain" all the way to the source. I.e. if the "event" source is capable of filtering/transforming - like a SQL database or a RSS based web feed is - LINQ for Reactive Extensions can apply the filter very early in the chain and avoid uneccesary progression of objects which would be masked later anyway.

Is it just me, or does it seem like everyone now-a-days is trying very hard to come up with new methodologies and paradigms and web 6.5isms, so they can get their 5 minutes in the lime light?

It is not just you. There a scores of other people out there who have always done it before 20 years ago.

Grow up. Sometimes Computer Science does see advanced. Realizations. Erik Meijer did an awesome job with Reactive Programming. He found a *duality* between actively "pulling" objects from sequences and reactively receiving "pushed" objects from a source. Given the mathematical duality he even managed to find places where the LINQ pull model was not complete yet.

Reactive Programming is not new. Nor was OO when it really took off. OO actually was don back in the 1970ies with Simula (IIRC). But now RP has *matured* as

Re:I for one would love to see DBs be more like Ex

[blue trane]

In Coursera's Reactive Programming MOOC, the difference between reactive programming event-handling and traditional event-handling is described in two slides from the introductory lecture:

http://subbot.org/coursera/reactive/callbacks.png
http://subbot.org/coursera/reactive/howtodobetter.png

A traditional Java event-handler is first presented, and the problems enumerated: it relies on a side-effect (the variable "count" in the example), which involves shared mutable state; events and their handlers are not first class. Reactive programming tries to do better so that complex handlers can be composed from primitive ones.