Slashdot Mirror
Why Reactive Programming For Databases Is Awesome
Nerval's Lobster writes "'Building on many lessons learned from spreadsheets, functional languages and model-driven application development environments, Reactive Programming (RP) is a recent approach to creating applications,' Val Huber, CTO of Espresso Logic, writes in a new column. 'In the RP paradigm, you can create complex applications from a series of simple declarative statements. Many of the details of implementation and work of gluing together various sorts of application constructs can be hidden because of the declarative approach.' He goes on to argue that RP makes maintenance easier and applications more adaptable, because RP offers automatic dependency management and reuse; at the same time, he also acknowledges that RP has some shortcomings. What say you?"
is it webscale?
I think excel is one of the greatest programs ever and reactive programming sounds great. This article isn't very specific as to how to get started though. Is it a concept, a product or something I have to implement myself?
Complex applications from simple declarative statements?
I have a button you can push and it will create an app for you.
How does this differ from what the rule engines Drools or Clips provide? They are reactive systems that update state by monitoring changes in the underlying data.
Interesting.
Code modeled in RP sounds like VHDL, Verilog et all - the languages used to model modern Programmable Logic Arrays, Programmable Gate Arrays and Programmable Logic Devices - aka, hardware.
No doubt this approach has its merits, but this kind or "programming" has its gotchas - a lot of gotchas.
I stand aside for now, looking with great interest.
Lisias@Earth.SolarSystem.OrionArm.MilkyWay.Local.Virgo.Universe.org
Did this guy just reinvent spreadsheets? There's something to be said for this, but having written in Prolog, which works that way, the 'reactive programming' people have to make a better case than the article does.
On the other hand, one of the big problems in databases is change notification. Microsoft at one point had a research project on this. The concept was that you could make a request to be notified when something of interest changed in the database. This was expressed as an SQL SELECT statement. When some change was made that would affect the output of your pending SELECT, you'd be notified. The problem was to figure out some efficient way to do this with thousands or millions of pending SELECT statements.
Finance people use notifications like that all the time. Limits on stocks, limits on combinations of stocks, changes in key ratios, that sort of thing. But the systems for that are specialized, a special case of a more general problem. The most general form of the problem is that B wants to know when something has changed at A. Most of the solutions for this have problems. Polling is either inefficient or slow. B relying on A to message them on a change has a failure problem. Both approaches can result in overloading A with checks for events that aren't happening.
I hope this silver bullet is better than the last one.
From the article description I couldn't figure out a) what is actually reactive programming; b) how will that help me build better applications. Complex applications are already built from a series of simple declarative statements. In fact, we call the most simple declarative statements language available "assembly". We call the machine representation of those very complex applications built with simple declarative statements "binary". On the other hand, Brainfuck also allows you to build complex applications with very simple statements, and I'm not migrating to it anytime soon.
Not cool, Slashdot, I'm going to have to read TFA!
I had never heard of Reactive Programming before, but after reading the article, it seems that Reactive Programming is the same thing as using views, which have been a part of mature RDBMS's for several decades now. Am I missing something obvious?
I don't respond to AC's.
A = B + C;
B = A + C;
C = A + B;
C = 42;
*POOF* and magic smoke and stuff.
Usually database manuals calls them triggers.
I can see this being useful for problems that are extremely linear and require extreme parallelization on large quantities of data, but that's about it.
I've done this 'methodology' many times using Excel. I did it because I I needed to give the spreadsheet to other people, and wanted to avoid having Office nag about 'potentially dangerous VBA'.
It makes it very easy to see *exactly* how your data is flowing, which is a bonus. It also uses a *lot* more ram because you are now maintaining a permanent block of memory for every single operation, for every different piece of data you are coding against. Of course, the second you get a cyclic dependency, the whole thing blows up.
But this method of 'programming' was a natural and convenient extension based on how Excel (or any spreadsheet) operates. Nothing more.
Is it just me, or does it seem like everyone now-a-days is trying very hard to come up with new methodologies and paradigms and web 6.5isms, so they can get their 5 minutes in the lime light?
I remember experimenting with a C++ framework that would work somewhat similar. Lazy evaluation and such. http://sourceforge.net/projects/ditto, it's long since been abandoned because, well... the only thing this way of programming solves is a small part of the "R" in "CRUD". It worked brilliantly for showing information on screen and automatically updating all kinds of stuff whenever some data changed, but in the end it only solved the least difficult part of creating an application, and doing so at a relatively high CPU% cost.
It was also supposed to have lazy/dynamic/late-binding expressions (i.e. lazy_int a = b + c;), but it turns out you only very rarely have any use for that type of expressions. At the time, there were also far superiour C++ frameworks being developed which could do these things as a side effect of supporting more useful programming models.
This RP thing doesn't really seem to do anything more than that.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
The article may be bullshit, but reactive is really interesting. And yes, this is similar to views, but with networking in the middle. Take a look at what meteorjs guys are doing: http://meteor.com/
Excel is the swiss army knife of applications. But if you want a DB to be more like Excel, let me introduce you to MS Access ...
This won't create an application. It just automates calculations that populate form fields. That's the easy part of most applications. The hard parts are the business logic that isn't just figuring out the values for form fields. In addition, you have nasty cascade effects. What happens in this system when the price for an item changes at the same time the customer's changing the number of that item they want to order, the customer's credit limit is being updated and payments are being applied to his 26 orders each of which causes an update to his balance due? You end up with a massive cascade of updates propagating up the chain, with each change causing changes to values that depend on it which in turn cause yet more changes. Multiply that by thousands of customers and it can kill the systems dead. Meanwhile a non-RP system will handle it cleanly and without dying by simply not updating values until everything's finalized. So as the accounts payable system applies payments to his 26 invoices his balance doesn't change, then when payments are done it updates the balance once to reflect the payments.
I'd use RP to help with things like forms where I want live updates and RP will help automate what I need done anyway. Lots of other areas... not so much. It'd be just another tool in the toolbox, and it's not quite as useful in the real world as it appears in simple academic examples.
Part of it's that the people describing these things often haven't dealt with truly large, complex applications before. Simple example: we have 5 fields in a request, each data file has a match value for each of those fields, and we want to grab the file that matches any one of the fields in the current request. "The" file? Um, what happens when I have 3 files each of which matches a different field of the request? Which one should be grabbed? Nobody thought of that. They say "It doesn't matter, just grab one.". And then later "We wanted file 17 that matched field 3 being X, why aren't we getting that file?", answer "Because file 9 matches when field 1 is A, which it is in your request, and we find file 9 first when checking.". Simple request, just grab an arbitrary file. Translated into the real world, complications pile up that were always there but nobody wanted to talk about because they were... well, complicated.
I don't understand this new slashdot scheme where the link to the story in the summary, leads to another story under the slashdot domain. Slashdot Business Intelligence? It's still slashdot... The summary on slashdot.org proper and the story under Slashdot Business Intelligence both offer a comments section. Am I supposed to double post or what? I'm posting this here out of habit but in the midst of confusion. The road to hell is paved by websites that suffered unrecoverable identity crisis.
Brought to you by Carl's Junior.
I started using reactive extensions for .Net for over a year ago and am constantly learning new ways to use it. UI, message busses, async calls, business objects, the list keeps growing. It is by far the most useful programming concept I've learned in the last year. There is one for javascript if you are not into .Net
This article is just marketing bullshit double speak without a concrete implementation.
I like the idea of having variables auto-recalculate based on a dependency tree built automatically by a tool. But recalculating vars sounds kinda slow, esp when resolving the dependency calculations involves time-consuming operations like database fetches. What control does the tool provide in limiting the timing and scope of the recalcs?
which I think fits into this model. It was neat, but kind of a mess. As soon as I wanted to do something the creators of OL didn't think of I ended up writing a lot of JavaScript and stuffing it in weird places.
I like the HTML/Android approach: Use Declarations for the UI layout and let old fashion code control the logic.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
I guess the compiler checks if the reactions cause more reactions in a loop?
to /active/ programming.
Anyways, seriously, please, please please my fellow programmers. Don't jump
on this just because it looks or sounds esoteric.
This sounds very similar to (if not identical to) a (still non-existent) product called "R++" that Bell Labs STILL hasn't developed/released 18+ fricking years later:
http://cm.bell-labs.com/who/pfps/rpp
http://cm.bell-labs.com/who/pfps/rpp/UserManual.ps
ftp://cm.bell-labs.com/cm/cs/who/pfps/rpp
ftp://cm.bell-labs.com/cm/cs/who/pfps/rpp/oopsla95.ps
I originally downloaded their postscript User Manual and converted it to PDF years and YEARS ago, and have been waiting for them to get their shit together and release a commercial product ever since, like they promised to.
They STILL haven't, so screw 'em.
Someone should start an Open Source project to create an R++ preprocessor anyway since it's obvious they're never going to.
It's not like they'd be losing money either, since they're don't have such a product of their own to sell even though they've had 18+ fricking years to create one.
(I hate companies that sit on good technology to the detriment of the rest of society)
Tl;DR [yet], but initially this sounds a lot like HDL's like Verilog and VHDL. The two fundamental constructs there are things that happen continuously (collections of simple logic gates) and things that happen based on a clock (registers, built around flip-flops). At a high level, this sounds like A = B+C is a continuous adder, and A is changed instantaneously with B and C. In the HDL world, this gets "compiled" down to silicon, but in a software world this would be radically harder to do, because you have to notify anybody listening to A that B and/or C has changed. Yes, it's like a spreadsheet in that sense.
GStreamer - The only way to stream!
is in its usual place.
OK, B+C can be evaluated once when you assign A, or it can be evaluated every time you evaluate A. Sometimes you know that A will not change after you calculate it, or that a fresh value of A is required, and sometimes you don't.
Delaying the evaluation of A is usually safe; but it can sometimes be very slow. Like it or not, performance still matters. If you evaluated A unnecessarily inside a tight loop in a game, you'd be dead. Even less CPU intensive thinks like web apps could be killed by this kind of thing, if A is evaluated in JavaScript every time you move your mouse pointer.
Evaluating A every time is not only slow, it might not even be correct. Let's say you pull in some weather data and then render a map. The frontal zone should be a smooth line. If you pull in new frontal boundary data while rendering the map... you'll render a broken front.
Sorry. It looks like the programmer still has to think. Re-calculating A every time the symbol A occurs may or may not be required.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
That is "recent" as in "about half a century old"?
These "rethinking programming thinking" efforts almost never work out. I mean, I applaud the effort and encourage experimentation, that part is all to the good. However don't get too excited. What has been working in recent years is new programming languages. They are fairly conventional languages though, not entire new programming models.
The idea of embedding a calculation into the system that is automatically updated by underlying data changes -- is that not just a database view?
We use this sort of technique quite widely in a Ruby on Rails app I work on -- complex calculations such as for profitability and cash flow are defined as views in Postgres, and referenced by the app as read-only models. Thus we can: Profitability.where(product_id: 27).group(:month).sum(:value)
Performs monstrously fast, as is extremely flexible. It breaks the whole "for the love of gods don't put business logic in the database" separation of concerns idea, but we have a system to ship right now and we can't wait for RoR performance and flexibility to catch-up that much.
Yeah, and in client side systems they're known as call-backs or event driven, etc. etc.
I don't know about you, but if it's old let's call it new!
Event-driven programming is HELL, except for interfaces. Every once in a while a DB trigger is justified, but event-driven languages have failed time and time again because... it's impossible to predict what will happen when and avalanche of eventual complexity causes the system to implode under its own weight.
For a programming language to make the cut, it must be utterly predictable down to the last side-effect.
Sent from my ENIAC
Or if I need reverse queryable reaction to change, materialised views fit the bill... actual cascading data modification based on reverse trigger like syntax would seem to have the level of intelligent design choice on par with Intercal's comefrom statement.
I'm just seeing an observer pattern with publishers and subscribers. It's been done time and again in probably all the major languages. In the .NET space, using aspect-oriented decoration and an expression store, both of which are declarative, well understood, and promote reuse, this concept is already covered. Regardless of the platform, any implementation is going to be constructed in the context of the domain being addressed, and I believe any one-size-fits-all approach would lead to a confusing mess.
'In the RP paradigm, you can create complex applications from a series of simple declarative statements. Many of the details of implementation and work of gluing together various sorts of application constructs can be hidden because of the declarative approach.'
Sounds like another wishy-washy piece of shit language designed to be "easy", "fast", and "human readable" that will end up being kludgey, slow, and a pain in the ass.
This is what database indexes already do.
Are there any examples of real-world reactive databases in production or development?
No, I will not work for your startup
'In the RP paradigm, you can create complex applications from a series of simple declarative statements. ..."; or, even , visually:
http://hypercard.org/
It's too bad the Newton wasn't just a hypercard engine, preloaded with a couple of removable apps.
Rx isn't really the same as callbacks, otherwise you wouldn't need this. And if you object to the terminology, blame Microsoft.
In Coursera's Reactive Programming MOOC, the difference between reactive programming event-handling and traditional event-handling is described in two slides from the introductory lecture:
http://subbot.org/coursera/reactive/callbacks.png
http://subbot.org/coursera/reactive/howtodobetter.png
A traditional Java event-handler is first presented, and the problems enumerated: it relies on a side-effect (the variable "count" in the example), which involves shared mutable state; events and their handlers are not first class. Reactive programming tries to do better so that complex handlers can be composed from primitive ones.
Good god (or in this case bad god)
Every damn company I have ever worked in has some (albeit smart) idiot who creates a smart excel/access application which does wonderful things.
Who then distributes it to xyz people
Then the datasource (ip address) changes and the thing breaks.
Or the datasource changes (database table changes) and the thing breaks
Or his hardcoded status codes changes or is expanded and the thing breaks
Then they call you out of the blue and say, please fix it!
Access is an evil which should die. Advocates of it are welcome to come debate the matter with me. I need fresh meat.
Excel is extremely powerful and a handy tool, but as soon as anyone adds a datasource they should come and debate the matter with me.
There are three kinds of falsehood: the first is a 'fib,' the second is a downright lie, and the third is statistics.
The UI made by XAML is a bunch of rules, like TextBox monitoring user.name, or TabControl.TabIndex monitoring product.type to switch different views based on type of selected product, or GroupBox.IsEnabled monitoring your permissions. It supports complex paths and custom type converters (although M$ provides none themselves) which set it apart from old simple data-binding.
Most of time you need no programming if your data-model is RP-ready - with notifications of property changes and collection changes. But when you do, it's very difficult due to asynchronous programming everywhere and lack of support from framework and language (C#) - you can't do things as in XAML like { MyProperty = OtherObject1.A.B + OtherObject2.C.D, set when ready } or wait until all objects in the path become non-null and the path is ready to evaluate. I suppose it can be easily done by LINQ and custom evaluator, but again M$ is not providing that.
You won't find true examples of RP in M$'s own samples either. Their samples are all fucked up by mixing the simplest data-binding and manual data loading, hell of code-behind programming with added ViewModel layer to bridge the two (which they deem as incompatible but they're NOT) to accomplish very simple tasks. Basically as brainless as Sun's server-side java examples.
One problem with RP is that you need a higher-level infrastructure to track when the tables/data-model are changed, without writing extra lines of code (in XAML you do, and they introduce weak-reference to event handler). And in real world you also need to consider the frequency of updates because it a database with 10 tables but 1000 triggers and 10,000 event notifications to update client's data model or view is like to have some problem, at the same time you have to keep DB consistency.
> For example, the English definition of the customer balance field might be: the customers’ balance is equal to the sum of its order totals, for the orders that are not paid.
Been using Flowfields in MS Dynamics Nav for years now: http://msdn.microsoft.com/en-us/library/dd338766.aspx
The rules engine I've seen (BRMS) utilizes the Rete algoritm (http://en.wikipedia.org/wiki/Rete_algorithm)
actual cascading data modification based on reverse trigger like syntax would seem to have the level of intelligent design choice on par with Intercal's comefrom statement.
Right, I don't think current DB technology, based as it is on 1970s timesharing mainframes, is up to the task. I suspect to make RP work nicely with distributed updates you would need to abstract out the idea of 'database' a bit; make it more like the Git/Mercurial model, where there's a big tree of data that anyone can hook into and receive updates, but can't affect anyone else directly.
This is the direction I think the Net will eventually have to evolve; things like Content Centric Networking are a step in this direction, if we added computation on top of that, we'd be getting very close to something much like the original 1970s 'hypertext' concept. Ted Nelson's Xanadu -- confusing, patent-ridden and badly-specified as it was -- I think had a concept of 'applitudes', applications as spreadsheet-like cells that can import data from anywhere and use it to update themselves. Imagine what you could do with a Web of live data like that.
You are not a brain: http://books.google.com/books?id=2oV61CeDx-YC
Is this you?
The reactive programming course is about Functional RP, not ordinary RP. That is why it gets to a new level of composability.
Events and callbacks are reactive programming. The new wave just tries to them in a better manner.
About 18 years ago we did something that sounds exactly like this: a conceptual information processor spanning sveral papers. Including the syntactical framework and a design for the evaluation framework. The implementation went on to be called CRISP and was commercialised by IdeaLogic. The power of this is not as much simulating a program (you would need a very very massively parallel processor, each evaluating its tiny set of rules), but checking masses of data against rules/ lessons learned.
... I do hardware for a living for the past 3 decades or so
This is the first time I come across RP
No, I haven't tried my hand on it yet, so I am wondering out loud if the gotchas that we got using Verilog / VHDL could be reproduced in the software
Anyone ??
I remember experimenting with a C++ framework that would work somewhat similar. Lazy evaluation and such. http://sourceforge.net/projects/ditto
MUCHO KUDOS FOR THE LINK, my man !
It was also supposed to have lazy/dynamic/late-binding expressions (i.e. lazy_int a = b + c;), but it turns out you only very rarely have any use for that type of expressions. At the time, there were also far superiour C++ frameworks being developed which could do these things as a side effect of supporting more useful programming models.
This RP thing doesn't really seem to do anything more than that.
Looks like this weekend I'll be fully occupied
I need to know more about all this !
Thanks again, man !!
Muchas Gracias, Señor Edward Snowden !
Yeah but if we can put it in the Web 2.0 then its for sure something new.
In the RP paradigm, you can...
I'm sorry, but as soon as I read the word "paradigm" I knew you were management and therefor had no idea what you were talking about. I could be wrong, but I can' risk reading any more of the post for fear of running into that word again. I don't want to get contaminated.
Also I think the Observer pattern is all about this.
The Tao of math: The numbers you can count are not the real numbers.
If they use java as their example of traditional event-handling, they don't know shit about that. Sorry but java is far from the traditional model.
"Reactive Programming (RP) is a recent approach to creating applications..."
Reactive Programming is not a recent concept -- just take a look at Esterel, which was developed in the 80's.
http://en.wikipedia.org/wiki/Esterel
There is a nice book about composing complex systems from more primitive ones: Structure and interpretation of Computer Programs
shhh. don't mention this. it's a secret weapon. well, it has limitations, but it's very useful under the covers.
Access database is considered a front end! But the future is DB front end with server conponent attached to it to make it a high level DB!
And millions of DBAs collectively sigh "oh god not again".
It is believed that if only someone understood what they meant, the world would have been a much better place.
Just instinct, but I'm guessing you can buy something that does it from Espresso Logic, of which Val Huber is the CTO.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
I guess you're talking about http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-22.html#%25_sec_3.3.4 "A Simulator for Digital Circuits":
I suppose the modern version uses events more for user interaction, and also gets rid of lots of idiotic, stupid parentheses.
Yes...sorry I have no mod points for you Bugs...
Boy, they really hate grammar Nazis around here, don't they?
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Yes, there are products and libraries that do this for you. It is probably not something you'd want to implement yourself. Espresso Logic has a commercial offering that provides a combination of declarative logic, declarative security, and actions extensible with JavaScript to build complex SQL based REST systems.
It is a product. A service actually, so you can fiddle without download or install, at espressologic.com. Earlier generations were Wang Pace and Versata. We'd love to hear your reaction. On the client side, EL is a very close approximation, and Microsoft has done some work: http://msdn.microsoft.com/en-us/data/gg577611.aspx
I sincerely apologize for the harsh tone. That was uncalled for, and I shot off without reading what I wrote and how it comes across. Again, I apologize.
Like you I have some "triggers". One of my triggers is blanket dismissal. I think what I perceived as your blanket dismissal was what triggered. You are correct about Ruby-on-Rails. It turned out that it was not up to the job. But at the same time it *did* popularize the "web-MVC" pattern in a way that Java frameworks (e.g. Struts) never achieved.
When someone praises a new technology as the new "best thing since sliced bread", we should approach it with a healthy portion of skepticism. However, we must also keep our minds open to new advances.
Reactive Programming is not an alternative to OO or FP. It complements both, and at the same time it promises to make certain aspects much more manageable, like e.g. asynchronous and parallel processing.
Reading slashdot one-liner: (irm http://rss.slashdot.org/Slashdot/slashdot).rdf.item | fl title,desc*
http://deduced.sourceforge.net/
I liked the amount of functionality I could get in just a few lines of code. Even the user interface was generated by reactive rules.
The thing that puzzled me the most with this tool is the lukewarm reaction of most developers who looked at it even when I proved that it could do things in about 10 times less code than a typical OOP solution.