Do We Need Another OO RPC Mechanism?

Paul68 queries: "I am looking for an RPC mechanism for a project. Granted, there are many to choose from, yet there seem none that meet my requirements! When one has toyed with the requirements the solution generaly becomes obvious. So, yeah sure, I can set out and create the next RPC mechanism, but it is a lot of hassle. But does the world need yet another OORPC, or have I simply not looked in the right corners?"

"My requirements are:

1. object oriented
2. extensible
3. platform independent
4. supports signatures for integrity and sender checking
5. supports privacy of the message contents (i.e. encryption)
6. time sensitive: I should be able to detect a dead server and do failover while the user is waiting for the response
7. bandwidth efficient, as I am looking to deploy it in wireless environments

Now #1-3 are no problem, #4 and #5 can be found, and #7 rules out anything XML-based. #6 seems to be the killer, as this rules out anything over TCP, and at that point the list gets pretty short.

Does anyone have any options that I may have missed?"

Comments

It's time to stop

[Anonymous Coward]

And ask yourself, "What am I really trying to do?"

Re:It's time to stop

[mystran]

Well, I've done CORBA for work, play'ed around with all kinds of other RPC systems, programmed with a quite a lot of languages, and after some hobby research and all kinds of strange projects, I've come to the following conclusion personally:

There is one good RPC system, and it's called HTTP. It works wonderfully when you can generate stuff from a database into some text-representation.

There are a lot of bad ones, which mostly have the same thing in common: It is NOT a good idea to make a remote resource act like an object. It just doesn't work well. To get an impression that it works well, you need a huge bloated library, so you can pretend that everything works, until your network breaks and you have to deal with all the shit yourself anyway, but now add the complexity of the RPC system to the mix.

The thing is, when you design to work with a very simple system, either HTTP or something similar built on top of TCP directly, you end up with design, where the network is a natural part of the system. When you build something on top of a generic RPC system, you just add a layer of indirectness, which makes a good design much harder to get. In addition, when you restructure your data into text-representation (or other simple representation), you usually end up doing it in the right place, because it's usually the only place where such representation can be handled in a sane way.

That's my 1 cent about RPC. The other 1 cent deals with objects. Objects are nice, if you don't have 1) database or 2) proper functions. When you have a (relational) database, you end up with something like objects anyway, when you build a good database access layer. For many things this is what you would want to do anyway, with the added benefit that you never need to care about object references. You just tell the system how to find/alter what you need.

For the functions, as soon as you have first-class functions, objects no longer make much sense, as it takes about 15 seconds to build an object system on top of these functions. Most of the time you don't want object anyway, as you usually want functions and data instead. Learn LISP. It is more useful today than ever before. If there's something wrong with existing dialects, then help build a better one.

As for your requirements: You could use TCP directly, but you need encryption, so add ssl or something. Authenticate the connection or something, the encryption should handle rest of the integrity (this is pretty much a cipher question). Build a simple protocol with no tweaks and leave the extensibility there. If you want, you could send XML anyway, if you use zlib or something to compress it. Depending on your data, it might be the easiest thing to parse anyway. Separate the protocol layer, have it timeout and catch TCP failures, and automatically reconnect (same or other server), and you have the time-sensitivity. If you need more, send requests to multiple servers, and handle them with the one that answers first. Aim for stateless protocol, and it becomes easier. TCP and SSL are pretty platform independent, so unless your protocol is really bad that shouldn't be a problem. Now, you basicly want this all for every application in existence.

Forget the objects, they just make life a lot harder. Use cookies and numerical identifiers if you really need to. A lot of things look like they needed OO-RPC, but to this day I've not seen a single one where it really made sense. Doesn't even speedup development in the long run..

Re:It's time to stop

[duffbeer703]

The AC speaks the truth.

A million years ago, a caveman spent alot of time developing the wheel. Don't replicate his work. You're likely to make lots of mistakes when you reimplement RPC or yet another master framework product, particularly with the encryption and signing part.

I know of people working on a similar project using Java Jini. Check it out.

Also, base your decision on the actual architecture of your application, not the buzzword-compliance of the glossy brochure or website.

Does 7 really rule out XML?

[forsetti]

How much overhead do you expect XML to incur? Or more importantly, what do you expect your data/(data + XML) ratio to be? With large amounts of data, the relative amount of XML is relatively small.

Of course, if you are sending many packets, each with small amounts of data, perhaps strewn across many XML tag sets, then the XML "noise" ratio may be high. How about gzipped XML-RPC ?

Re:Does 7 really rule out XML?

[kaisyain]

And even if it does rule out XML per se, what's wrong with binary XML?

Re:Does 7 really rule out XML?

[RevAaron]

binary XML? Is that like "wireless DSL" I've heard advertised in my area?

Re:Does 7 really rule out XML?

[noselasd]

When I experimented with SOAP, it was around 60% metadata, and the bytes sent by the protocol were 600% more than the equivialent done with SunRPC.
(That was for one particular application, others will vary.)

Something XML-RPC (SOAP) doesn't have

[SteveX]

...that bugs me is that there's no way to make a call that will return multiple pieces of data over a long period. Like, I want to connect to a server and say "tell me every time the price of IBM stock changes". I can use SOAP for most of a trader type UI but for streaming the data back to the client, you have to use another protocol (unless I'm missing something).

The dude asking the quesiton here is, I think, a little too worried about overhead - the overhead of XML is fairly tiny, and the overhead of TCP/IP isn't that great either - if both of those really are too much then you're probably on your own as there isn't likely to be a "general purpose" solution that will work for you.

- Steve

Re:Something XML-RPC (SOAP) doesn't have

[ThenAgain]

I once implemented a web interface that used XML-RPC to communicate with a search engine back-end. Each request lived in a state machine on the server. When the server received a request it would start processing and if it took more than a few seconds it would return an object with a status report and a request ID. The client would send a browser meta-refresh that would check on the status of the request every few seconds until it was done or returned an error.

If you're looking for something closer to real-time (a way for the server to initiate contact with the client) check out the BEEP protocol.

From the BEEP web page:
BEEP is a turbocharger for Internet applications that offers advanced features such as:

• a standard application layer that supports dynamic, pluggable application "profiles" (protocols)
• peer-to-peer, client-server, or server-to-server capabilities
• multiple channels over a single authenticated session
• support for arbitrary MIME payloads, including XML

Re:Something XML-RPC (SOAP) doesn't have

[Doomdark]

As for overhead, I'm in agreement there: the overhead of TCP/IP is definitely minimal compared to the cost of marshalling RPC data to begin with. TCP/IP can be tuned anyway, and it'll almost certainly be faster and more robust than something that tries to reinvent it over UDP.

Perhaps I misread original points, but it seemed like he wasn't so much worried about overhead, but about time-sensitivity aspect. TCP tries really hard to get all the data get through (with re-sends etc), without worrying at all about delays. It's good for bulk transfers, and things that simply have to get through, but for real-time (or in general time sensitive) data it's pretty useless to get anything that is late. And worse, since there's no way to really know what's going on (from TCP POV, "app shouldn't care"), it's difficult for the app to try to failover or other sensible action to resolve the (likely transient) problem.

Trying to figure out information about why and how data just isn't getting back, via socket (or whatever) API is next to impossible... thus, protocols such as RTP were developed. It's been a while since I followed what was happening there, but there was definite need for something that would give better control over timing information; at the very least for apps to be able to find out if there are timing problems, and do the right thing whatever it is (mark server as likely being down, find alternatives etc).

As to overhead, I'd agree; esp. due to HTTP being so ubiquitous, all stacks are ridiculously optimized, and handle "non-optimal" usage of TCP (simple request-reply) rather fine, all things considered.

You don't mention...

[ThenAgain]

what you've already looked at. Whenever I've been tempted to implement my own RPC mechanism I've found that XML-RPC meets my needs perfectly.

It's easily capable of representing objects, platform independent, encryptable (via SSL), compressable (via gzip [and probably SSL as well]), and textual.

The advantages of being textual in your protocols is well laid out in Eric Raymond's book The Art of Unix Programming. He even treats it as a case study.

NML

[kinema]

You might want to take a look at the Neutral Message Language, NML. Developed at the Intelligent Systems Division of the National Instute of Standards and Technology is was intended from the start for use in real-time/time critical situations. I know that it currently has support cor C, C++ and Java.

Does 6 rule out TCP?

[jrstewart]

I think you'll find that in practice a UDP-based solution isn't going to make your life any easier than TCP. You may want to use TCP in conjunction with an external timer to fail the connection well before TCP gives up though. If you tend to do multiple RPCs to the same host the overhead of TCP should be minimal, and you get a lot of functionality (retries, backoff, etc.) for free. There's RFCs for transactional TCP which solves some of the overhead problems for small exchanges but I don't think they're widely implemented.

If you do end up writing your own UDP-based protocol, take some time to study TCP as well as other UDP based protocols and make sure you write something network friendly. There are a lot of naive protocols which fail spectacularly in less-than-perfect conditions.

RPC is rarely worth the trouble.

[tyrecius]

RPC causes untold security/authentication headaches and is often hard to program with besides.

See ESR

Message Queuing Server?

[psykocrime]

You could probably implement what you want using
some type of message queuing server, like IBM Websphere MQ, or equivelant.

As a mental exercise, and to make sure I'm not talking out my ass, let's run down your list of requirements:

equirements are:


1. object oriented


Well, in the case of Websphere MQ, I'd say "yes" to this one, at least partially. There is a C++ based client library, IIRC, a (IBM specific) Java client library, and a JMS client. And even if the client libraries weren't OO, you could write OO wrappers for procedural calls more easily than writing your own RPC mechanism from scratch, I think.


2. extensible


Check. If you use pub/sub messaging instead of point to point messaging, even more so. And if you use a message type (MapMessage in JMS) that is based on name / value pairs, it's easy to extend your messages without breaking backwards compatibility.


3. platform independent


Not completely, but with both Java and C++ clients available, you should be able to support most everything.


4. supports signatures for integrity and sender checking


Not sure if it has native support for signatures, but a signature can always be added to a message as a property.

5. supports privacy of the message contents (i.e. encryption)


Again, using Websphere MQ as an example, it does support the use of SSL for communications, if that helps meet your requirements.


6. time sensitive: I should be able to detect a dead server and do failover while the user is waiting for the response


You're right, that's the tricky one. I think there are ways to achieve this goal using message passing servers, but it might take some work.


7. bandwidth efficient, as I am looking to deploy it in wireless environments


I think this requirement is met as well. Most messaging products support a message type that is nothing more than a stream of bytes that you interpret as you will... and even if you use a slightly move involved message type (MapMessage or StreamMessage to use JMS types as examples) you're still not carrying a lot of overhead.


Now #1-3 are no problem, #4 and #5 can be found, and #7 rules out anything XML-based. #6 seems to be the killer, as this rules out anything over TCP, and at that point the list gets pretty short.


Depending on just how "time sensitive" #6 really is, I do think you could come up with a solution using a message passing server. If you're able to use Java for the clients, JMS makes doing an RPC style messaging very easy, using the QueueRequestor and TopicRequestor interfaces. And even if you can't use Java, it shouldn't be to hard to cook up your own request / reply mechanism, built on top of the messaging system.