Google Open Sources Its Data Interchange Format

A number of readers have noted Google's open sourcing of their internal data interchange format, called Protocol Buffers (here's the code and the doc). Google elevator statement for Protocol Buffers is "a language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols, data storage, and more." It's the way data is formatted to move around inside of Google. Betanews spotlights some of Protocol Buffers' contrasts with XML and IDL, with which it is most comparable. Google's blogger claims, "And, yes, it is very fast — at least an order of magnitude faster than XML."

Comments

As a former user of CORBA

[Anonymous Coward]

It looks like Google has taken some of the good elements of CORBA and IIOP into its own interchange format.
While CORBA certainly is bloated in a lot of ways, the IIOP wire protocol it uses is vastly faster and more efficient than any XML out there.. and yes it is just as "open" (publicly documented and Freely available for use in any open source application) as any XML schema out there. J2EE uses IIOP as well and its is technically possible to interoperate (although the problem with CORBA is that different implementations never really interoperated as they were supposed to).
    As a side note, I'd rather write IDL code than an XML schema any day of the week too, but that's another rant.

Fast

[JamesP]

"And, yes, it is very fast â" at least an order of magnitude faster than XML."

Just wait for the XML zealots to come crashing and not believing that XML is not the fastest, best, solution to all the world's problems (including cancer) and of course people at Google are amateurs and id10ts and WHY DO YOU HATE XML kind of stuff.

Or, as Joel Spolski once said: http://www.joelonsoftware.com/articles/fog0000000296.html

No, there is nothing wrong with XML per se, except for the fans...

Re:An order of magnitude over XML?

[dedazo]

Looks like Google just invented the IIOP wire protocol, which is also platform agnostic and an open standard.

I guess the main difference here is that their "compiler" can generate the actual language-domain classes off of the descriptor files, which is a definite advantage over "classic" IDL.

"Google protocol Buffers" is cooler than the OMG terminology, but this kind of thing has been around for 20 years.

JSON

[hey]

Looks kinda like JSON to me.

Re:JSON

[0xABADC0DA]

Modify JSON so unquoted attributes are 'type labels' and define the type of an attribute by giving a label or a default value. For instance:

phoneType: { MOBILE: 0, HOME: 1, WORK: 2 }

phoneNumber: { "number": "", "type": phoneType }

person: {
  "name": "",
  "id": 0,
  "email": "",
  "phone": [ phoneNumber ],
}

... now you have pretty much exactly the same message definition as protocol buffers, but in pure JSON. It could also use some convention like "@WORK" for labels/classes so that a normal JSON parser can parse the message definitions. You can write a code generator to make access classes for messages just by walking the json and looking at the types. I don't see that 'required' and 'optional' keywords help much... imo defaults are generally better (even if they are nil). But this could easily be expressed in a json message definition.

It's easy to make a binary JSON format that is fast and also small, so there is little advantage to protocol buffers there. It's also easy and ridiculously fast to compress JSON text using say character-based lzo (Oberhumer).

Maybe somebody can explain, but it doesn't seem like protocol buffers really have much advantages over JSON. It sounds like it is effectively just a binary format for JSON-like data (name-value pairs they say) along with a code generator to access it. The code generator is nice, but this is like a day's work max. Maybe I'm not understanding google's problems, but I'll stick with JSON since it actually is a cross-platform, language neutral data format... and you can always optimize it if actually needed.

Ok, I'll bite...

[Dutch Gun]

Obviously, those at Google felt XML didn't work well for them. They have the resources to invent a protocol and libraries to support it. And, they are big enough to be their own ecosystem, which means as long as everyone at Google is using their formats, interop is no biggie. Good for them, I don't begrudge that decision.

I'm actually a game developer, not a web developer, so I'll speak to XML's use as a file format in general. Here's a few points regarding our use of XML:

* We only use it as a source format for our tools. XML is far too inefficient and verbose to use in the final game - all our XML data is packed into our own proprietary binary data format.
* We also only use it as a meta-data format, not a primary container type. For instance, we store gameplay scripts, audio script, and cinematic meta-data in XML format. We're not foolish enough to store images, sounds, or maps in a highly-verbose, text-based format. XML's value to us is in how well it can glue large pieces of our game together.
* All our latest tools are written in C# and using the .NET platform (Windows is our development platform, of course). It's astoundingly easy to serialize data structures to XML using .NET libraries - just a few lines of code.
* Because it's a text-based format and human readable, if a file breaks in any way, we can just do a diff in source control to see what changed, and why it's breaking.

I'll make a concession that I've heard of some pretty awful uses of XML. But those who dismiss XML as a valuable tool in the toolchest are equally as foolish as those who believe it's the end-all and be-all of programming (I'm not saying that's true of you, just pointing out foolishness on both sides). Like any tool, it's most valuable when used in it's optimal role, not when shoehorned into projects as a solution to everything.

Re:Why another encoding scheme?

[joelwyland]

Just imagine how far we ahead we would be today if Google had put the same effort into creating tools the rest of the SQL-writing, open(2)-using world could use.

We wouldn't be ahead at all. We use different tools than they do because they are dealing with different volumes of traffic, data and demands. Let's take a moment and look at your specific complaints. You say Google suffers from NIH syndrome. Having previously worked at Google, I think you are half right. The difference is that Google both benefits _and_ suffers from NIH syndrome. Sometimes the company spends too much time reinventing the wheel, but sometimes the tools out there aren't (and shouldn't be) useful to Google. Apache shouldn't be changed to support the kind of traffic that Google handles because then it wouldn't nearly as good for all of the rest of the world. General software is great because it solves so many problems. However, general software isn't the right solution for all problems, especially extreme ones. Just about all of Google's needs are extreme ones due to the volume of traffic. You dislike the idea of BigTable. Why not use the right tool for the right job? BigTable is a ridiculously fast database system that works beautifully with petabyte sized databases. SQL isn't the right answer to all solutions. They DO use SQL... but when it is the appropriate solution. They have some really sexy internal tools for dealing with SQL and such and I'm hoping those are coming down the open source pipeline soon. :) You claim the Protocol Buffers are clunky. I've used them and developed with them extensively. They aren't clunky at all, they are actually quite elegant and easy to use. They streamline development, are incredibly reliable, and are incredibly fast. You obviously are confused by GFS as well. The system is transparent to the application by using standard i/o stream classes. It is inherently redundant to ensure data security. It is so fast in its response time that Google search is the fastest of any major player. The list goes on and on. I don't really see how you can be upset at Google for making awesome software and then giving us access to it.