W3C launches Binary XML Packaging

Spy der Mann writes "Remember the recent discussion on Binary XML? Well, there's news. The W3C just released the specs for XML-binary optimized packaging (XOP). In summary, they take binary data out of the XML, and put it in a separate section using MIME-Multipart. You can read the press release and the testimonials from MS, IBM and BEA."

Comments

Testimonial: Dancin Santa

[Dancin_Santa]

I was drownding in debt. There was no where to turn. My wife left me, my friends all left me. Even my dog, he left me too. I had to do something.

That's when I found Binary XML. They were able to help with the debt. They got the creditors off my back and got me back on my feet.

Thanks Binary XML!

(I thought this was going to be about a standardization of compressing XML files that got rid of the excess bloat in the markup.)

Re:Testimonial: Dancin Santa

[Anonymous Coward]

I was drowning in "maybe" or "perhaps" slashdot articles again. There was no real news in sight. My pickled horse left me. My limelight perl sherbet left me. My estranged bungee left me. Even my albatross cadaver, he left me, too. I had to do something.

That's when I saw Binary XML. They were able to give me a real, solid story. They got the Dancin' Santas on my back and a hedgehog on my porch.

Thanks Bob the Builder!

(I thought this was going to destroy my epsom salt epstein-barr files to get rid of the excess methcathinone lumberjacks.)

Re:Testimonial: Dancin Santa

[seanadams.com]

My wife left me, my friends all left me. Even my dog, he left me too. I had to do something.

Your life is a country song. For better results, try playing it backwards.

I got my wife back, my car back, my house back, and a full bottle of whiskey at the end!

Re:Testimonial: Dancin Santa

[__aaclcg7560]

I got my wife back, my car back, my house back, and a full bottle of whiskey at the end!

Lucky you. I played my record backwards and the Devil from Georgia shows up. Took a while to get rid of that son of a gun. :)

Re:Testimonial: Dancin Santa

[Anomalous Cowturd]

Hmm. I played that one backwards, and I wound up going down to Hell and challenging the devil to a fiddle contest. Didn't win.

nothing else to work on?

[seanadams.com]

The tech industry seems really starved for ideas lately.

Binary file formats are hard.
Let's use XML because it's easier.
No wait... let's represent that XML in a more efficeint binary format.
Ah yeah that's the ticket - the best of both worlds!

Now let me just fire up my code-morphing processor which, through emulation ahieves x86 compatibility with "low" power consumption. Never mind it's slower overall and has worse MIPS/mW than an underclocked x86 - look Ma, we *inveted* something!!!!

There are some real technical problems out there... why are people chasing non-problems like XML?

Re:nothing else to work on?

[tomhudson]

There are some real technical problems out there... why are people chasing non-problems like XML?

Because they're hacks more into buzzword bingo and "selling the next big thing"?

Whatever happened to the virtues of simplicity, like a file containing a header record detailing the field names, and rows containing the data in either fixed-length or delimited form? Damn fast to implement, debug, read from and write to. Parsing? What parsing? Read the first line, split it to get your headers, and read 1 line per record.

Ideal for data exchange. Easy to manipulate via javascript on the client. Simple to display and manipulate via the DOM (Document Object Model). Not resource-hungry. Handles both text and binary data. Dirt easy on the server.

I ran a test to compare, and I'm able to select, format, and serve 1000 records this way in less time than 100 records in simple HTML, never mind xml. By doing this, the client can page through, say, 25 records at a time without having to hit the server every few seconds to see the next/prev pages.

Re:nothing else to work on?

[lupin_sansei]

What you are talking about is CSV. CSV is great, but it's only any good for table structured data. You can't implement a tree or any arbitrary nested structure like you can in XML.

Re:nothing else to work on?

[Anonymous Coward]

id, parentid, tag, text
1, -, root, yes you can
2, 1, child, it's simple
3, 1, child, do it like you would in actual code
4, 3, grandchild, you don't really think memory has magical trees in it do you?
5, 4, answer, it doesn't
6, 1, child, you can create trees in CSV

==

<root>yes you can
<child>it's simple</child>
<child>do it like you would in actual code
<grandchild>you don't really think memory has magical trees in it do you?
<answer>it doesn't</answer>
</grandchild>
</child>
<child>you can create trees in CSV</child>
</root>

Which one is *really* easier to parse? (Not necessarily read.)

Re:nothing else to work on?

[SunFan]

I'm waiting for the fantastic MIPS/MW of the Pentium VIIV!

Re:nothing else to work on?

[SunFan]

I generally agree with you except:

1) Endianness will probably cause problems when you least want them.

2) Parsing wide-character string data can be a pain.

I mostly think XML is 95% overrated and 5% genuine usefulness, but, in a world of people who have never heard of a big-endian computer regardless of a degree CS/CE/EE, it's a tough call.

You know, I think colleges should start offering 4-year degrees in XML. That way we would be assured of having a few people in the world who actually know how to use it properly.

Re:nothing else to work on?

[lupin_sansei]

XML is easier to parse, since I can use the XML parser that comes with the browser's DOM implementation, or my languages API.

Re:nothing else to work on?

[tomhudson]

What you are talking about is CSV. CSV is great, but it's only any good for table structured data. You can't implement a tree or any arbitrary nested structure like you can in XML.

Well, I was actually talking about fixed-length records as well (even quicker to manipulate - no complicated parsing involved, random access for r/w, etc). Need some data?

fseek(HEADER_LENGTH + RECORD_SIZE * (desired_record_no - 1));
if (bytes_read = fread(FH_IN, ibuff, RECORD_SIZE) == RECORD_SIZE) {
// do processing
} else {
// error_handling code
}

As for implementing a tree or nested structure using tables ... look at the display options for the site you're posting on. It allows you to view the data as threaded, nested, or flat, but the original postings are stored as table data.

Data presentation to the end user != data storage method. The two are, and always should be, independent.

Re:nothing else to work on?

[renehollan]

Well, yeah, but that's cheating, using a parser you get for "free". The table format for structured data is a hack. I've seen it used to store tree-structured data in databases. With very bushy trees, XML is a more compact representation. With deep sparse trees, it's the opposite.

The big advantage to XML is its ability to represent tree-based structures. The anount of fiddling needed to represent self-referential structures is on a par with table-based encoding of tree structures, so that extention is easy.

Of course, there are lots of ways to represent structured data -- XML is but one and the grandparent's post illustrates another. Perhaps the right approach is to separate the structure of the data from the means used to encode it, with a few standard methods. (Anyone beside me remember XDR/RPC?)

I happen to love using XML for a config-file text-based representation of small amounts of tree-structured data. The hodge-podge of formats in the typical /etc directory is absurd. However, parsers that insist on parsing an entire XML document at once are useless for arbitrarily complex documents of which only part is of interest. XML is bloated and slow -- no argument. But then, so is interpreting source code. An XML to binary-XML compiler would be welcome.

Re:nothing else to work on?

[thogard]

yes you can
. it's simple
. do it like you would in actual code
. . you don't really think memory has magical trees in it do you?
. . . it doesn't
. you can create trees in CSV

dots optional...
I've been using computers which seem to have linear runs of memory for a long time and I don't have any problem telling them to keep stuff in all kinds of complex structures that don't fit how stuff is in memory. Why should a file be any different?
The above format is how my todo list program has been keeping things for over a decade.

Re:nothing else to work on?

[starm_]

" but, in a world of people who have never heard of a big-endian computer regardless of a degree CS/CE/EE, it's a tough call.
"

The solution to that is: Standardize on an endianness for binary serialisation and let the computer that doesn't follow the standard do the conversion.

See? that wasn't hard, No need for XML.

Re:nothing else to work on?

[asb]

You, and whoever modded you up as "interesting", are an idiot.

This standard is not about representing XML in binary format.

This standard is about representing binary content in an XML document in binary format.

See, previously, if one wanted to include binary data in an XML file one had to Base64 encode it. This takes space and processor time.

This standard moves the bloated Base64 content into a pure binary MIME object.

Maybe you should have RTFA first, eh?

Re:nothing else to work on?

[Jugalator]

See? that wasn't hard

Hehe, look at the standardization processes between companies and you'll see that it's more than hard. ;-) After all these years, we're still living with disagreements on how the return character should look like.

Re:nothing else to work on?

[Keeper]

No wait... let's represent that XML in a more efficeint binary format.

Except that's not what they're doing at all. They're encoding binary data IN an XML document. They're using a principle similar to how one would go about attaching a file to an email.

Re:nothing else to work on?

[DrSkwid]

do you even know what binary xml is ?

RTFA

Re:nothing else to work on?

[mshurpik]

You mean LF vs CR/LF? Well, CR/LF has a higher resolution than LF, and was introduced later (Windows), so I would have to say that Unix needs to catch up and standardize to CR/LF.

It's true, though, that an ASCII character standard would be far more useful than XML ;)

Re:nothing else to work on?

[EvilIdler]

The only good thing to come out of Electronic Arts was their IFF EA 85 data format standard. It is more or less to binary what XML is to text.

Re:nothing else to work on?

[master_p]

You are absolutely correct. What the industry needed was a standard format for data exchange, not the bloat XML is.

Re:nothing else to work on?

[Chris_Jefferson]

Whatever happened to the virtues of simplicity, like a file containing a header record detailing the field names, and rows containing the data in either fixed-length or delimited form? Damn fast to implement, debug, read from and write to. Parsing? What parsing? Read the first line, split it to get your headers, and read 1 line per record.

Then of course you have the problem that your data wants to be variable length. Then you want to have the deliminator actually in the data, so you have to invent escape codes. Then in some lines you want to allow multiple occurances of some of the parameters so you put in some basic markup. Then you want to be sure that any data users enter is of the correct format, so you write a verifier. Then you are basically back at XML again.

XML isn't that great. However take at face value, it saves time and programming errors, the same way I wouldn't expect to have to wite my own doubly-linked-list, or hash table. Neither are complicated, but my language should come with one pre-written which is safer and faster than one I could knock together.

Re:nothing else to work on?

[flink]

Whatever happened to the virtues of simplicity, like a file containing a header record detailing the field names, and rows containing the data in either fixed-length or delimited form?

Because things don't stay that simple. Have you ever looked at, say, an X12 spec? Looping, conditional sub elements, destination-specific mandated fields... After writing 3 or 4 validating X12 parsers, you really start to pine for XML: Here's the data, here's the DTD, give me the parse tree.

Re:nothing else to work on?

[tomhudson]

It didn't properly seperate content from presentation. So your script broke when you added a field.

No, the script won't break when adding or removing a field:

1. read header
2. create arrays for field name, field type, field length by split()
3. for each line in rest of file
   1. grab field_length[field_number] number of bytes
   2. display as appropriate to that field type

Heck, you can even use a style sheet to determine how it's displayed by having a style for each field type. Don't like how it's displayed - change the style sheet.

Re:nothing else to work on?

[tomhudson]

All the problems you mention were solved decades ago.

1. data wants to be variable length
   Not a big deal. You don't necessarily need embedded escape codes (though they work well) - you can also use overflow buckets like databases have used for, say, 30 years
2. Then you want to have the deliminator actually in the data, so you have to invent escape codes.
   regexes make this easy to implement.
3. Then in some lines you want to allow multiple occurances of some of the parameters so you put in some basic markup
   Not necessarily. Master/detail records handle this problem nicely.
4. Then you want to be sure that any data users enter is of the correct format, so you write a verifier. Then you are basically back at XML again.
   No - it's a lot easier to write a function to verify that a postal code is formatted correctly when passed a string, than it is to interface wit a general-purpose verifier. Besides, you still have to verify it independently on the server before saving it (I don't want to run an xml parser every time someone clicks submit).

Re:nothing else to work on?

[letdinosaursdie]

Definitely not very useful for data with any kind of structure. Why not s-expressions though?

Re:nothing else to work on?

[renehollan]

It is not a fair comparison. If you had an existing library to make loading CSV data easy, would you still come to the same conclusion?

If the issue is one of "Which external data format is better?", I don't think it's fair to answer it on the basis of which one is supported better by existing tools. While that is certainly a factor when implementation expediency is concerned, it is not the only characteristic worthy of optimizing.

The big issues with XML encoding are encoded-object size, and incremental parsing.

Re:nothing else to work on?

[starm_]

no that is not what endianness is. Look it up on google.

Re:nothing else to work on?

[tomhudson]

Definitely not very useful for data with any kind of structure

Sure it is ... most xml "data" is artificially over-structured to "conform" to xml. Real data (the stuff that lives in relational and/or hierarchal databases, and that generates real revenue) is certainly structured, yet works quite well with this. Stuffing it into xml is a waste of time and resources.

Re:nothing else to work on?

[tomhudson]

The length of the field is NOT measxured in a fixed amount of bytes:

var fields = "Lastname:Char:20;Firstname:Char:20;Notes:Char:50" .split(';');

... can be changed to ...

var fields = "Lastname:Char:20;Firstname:Char:10;Notes:Char:500 0".split(';');

... with o problem, or even to this ...

var fields = "Id:Number:10;Me:Char:100;School:100;Notes:500;Bir thdate:Date:8;Spouse:20;Work:20;".split(';');

... and it will still work. The end result is an array filled with strings containing field names, data types, and the actual data, which can all be split in a for loop.

The actual data does NOT have to be kept in lines - it can all be one string of size h+x*n length, where x=record size, h=reader size, n = number of records.

If you had tried it, you would know it works, and that it's not something new - its been done like this in other languages for decades.

So, stop with the all-caps shouting ... xbinary xml is an oxymoron.

Re:nothing else to work on?

[tomhudson]

Notice that in your example any program that wanted to parse out data from your message would have to know the format of your message! If that format changed...all existing code would break and would have to be changed.

Not really. You can add, remove, change, fields, field types, etc., without changing how the code that interacts with it behaves. You can easily add code to take in cases where the header only contains partial information, supplying defaults for the rest, and you also can add a meta-header, which would include information about the header seperator character sequence, field spec separator char sequence, and the header length. That would cover a LOT of modifications w/o crapping out the code.

It could even be made to recognize different headers embedded in the data, for other differently-formatted data, w/o getting bloaty or hard to code/debug.

Transforms/exchanges could also be handled by a simple script. Efficiency = programmer efficiency + code efficiency.

As for the rest, the presentation to the end user would be handled by css.

Re:nothing else to work on?

[Nailer]

The length of the field is NOT measxured in a fixed amount of bytes:

var fields = "Lastname:Char:20;Firstname:Char:20;Notes:Char:50" .split(';'); ... can be changed to ...

var fields = "Lastname:Char:20;Firstname:Char:10;Notes:Char:500 0".split(';');

Yes, I know. You don't understand what I'm saying. For one final time: you shouldn't need to change anything. The language should deal with it. Oddly enough ,like XML does.

You also fail to understand that I'm not arguing for or against binaries encoded within XML, I'm reponding to a parent post that's against XML per se.

Re:nothing else to work on?

[tomhudson]

The whole point is to get away from xml's flaws (lack of seekability, high resource usage). Binary XML isn't the way to do this ... not when much simpler solutions are possible.

Trying to be all things to all people, or fill all niches, means being neither fish nor fowl. Shoehorning XML into everything is short-sighted.

Besides, the term "binary xml" is a good candidate for oxymoron-of-the-month, IMO :-)

Re:Binary... XML... Nah!

[metalhed77]

except SQL isn't very useful when it comes to technologies like RSS say......

if you're using an XML file in a place where you need a high performance SQL database then you're doing something wrong. If you're using XML as datastorage for some small webapp who cares so long as it's fast enough for that particular application.

I for one...

[Vombatus]

welcome our XML wrapped binary overlords

or is that our XML-binary Optimised Packaged overlords?

Re:Binary... XML... Nah!

[FrankHaynes]

if you're using an XML file in a place where you need a high performance SQL database then you're doing something wrong. If you're using XML as datastorage for some small webapp who cares so long as it's fast enough for that particular application.

As you point out, it is the wrong tool for the job, much like using tables to layout HTML pages (as the CSS religionists like to point out).

My 64 million dollar question is why they put an acronym inside another acronym: XOP stands for XMLOP? WTF??!!

They REALLY have too much time on their hands!

Binary XML Lite

[noidentity]

Here's my binary XML-like file format which gives the best of both text and binary file formats. It's human readable and efficient at the same time! Finally, an end to the text-versus-binary wars. Here's an example file:

The following data is in binary.
UH)(&T^( @#t79nui**&tb x9#@ $Y*_@$ji[P{O@JIOHXIOU$HIIU#$hiuoHOP$UJ [etc.]

Re:Binary XML Lite

[g0at]

Something else I hate about this whole thing is the perpetuating of "binary" to mean "non-plaintext". I mean Christ, are we to infer that plaintext data is represented in ternary while everything else can only be expressed in binary? Yep, no hexadecimal for non-text, no sirree...

-b

Seems like a great way to package media...

[bryanrieger]

This seems like it would be an ideal fit for services such as Flickr as it would allow for image (or other binary media files) to be sent with xml data - in a compressed binary format.

Re:Seems like a great way to package media...

[psyclone]

Exactly. It seems like a way to have a "text" file that is easily parsed (all the XML info -- in this case possibly a description, comments, image meta-data, etc.), yet binary info (a jpeg compressed image) fits along-side for when you want it. One file with all the goodies.

How this is different than simply base64 encoding the image inside a tag is yet to be seen. Perhaps because it's a standard?

Re:Seems like a great way to package media...

[malfunct]

I would assume because base64 coding of binary data bloats its size (I think up to 40% additional size over the uncoded binary) and takes time to encode/decode. If you were to be able to put a marker in an element that says "binary blob 100 goes here" and include binary blob 100 in some other area that is pure binary then you would have the binary data without encoding overhead.

As a software developer

[Anonymous Coward]

As a software developer I find this particularly good.

While I myself would prefer to write a binary protocol and send the data through a TCP socket I can no longer do that.

When we land big contracts at work that deal in government and health the key thing they need now is interoperability with others. What does this mean? XML. Whether or not you like it, XML is here to stay. Its what everyone is pushing.

Therefore we had to adapt and start using it. Not just for B2B, our rich desktop clients now communicate with the server using XML web services.

The problem we've encountered is sending binary data. Right now we have to encode the data in base64 XML which uses lots of resources. I will give more look at this but it looks particularly good.

Re:As a software developer

[andalay]

So Binary XML solves the problem that XML creates?

Something to think about.

Re:As a software developer

[SunFan]

B2B ... rich desktop clients ... XML web services

Baarrrffff!

Re:As a software developer

[caseih]

Yes, but this is what ASN1 encoding is for. It's a structured, self-describing encoding scheme that works very well for structured data. What advantages does this binary XML have over ASN1? Both require external descriptions to attach meaning to the data.

In your case, ASN1 is what you should be using, not XML in the first place.

Re:As a software developer

[Keeper]

All this does is create a standardized process for what it is you're doing. You won't see any improvements with this, because you have to encode the binary data in MIME form.

Now I can embed my evil virus without the...

[Anonymous Coward]

.. virus scanner picking it up!

evil virus/xml?

Uhhh...

[Phexro]

Unless I'm horribly misreading the specification, it appears to be a way to package up XML documents and binary data that they reference into a neat package with MIME - not a way to convert a (text) XML document into a binary one.

Re:Uhhh...

[Anonymous Coward]

You'd be entirely correct.

Of course, this is Slashdot.

Re:Uhhh...

[Albinofrenchy]

Yeah, you are most definatly right on that. Looks like CowboyNeal needs to RTFA.

Re:Uhhh...

[Phexro]

I read the article... I'm already a bad slashdotter.

Re:Uhhh...

[Anonymous Coward]

XOP doesn't define a packaging format, MTOM does define that the packaging used for SOAP is MIME.
With XOP you can define that your packaging format is zip or tar or jar or whatever suits your application.

Re:Binary... XML... Nah!

[System.out.println()]

AIM = AOL IM
GNU = GNU's Not Unix (and many other recursive acronyms)

To name a few :)

Here it comes

[ICECommander]

This sounds like the era of small/efficient XML is gonna end: here comes the bloat! (Remember Netscape)

Re:Here it comes

[hanshotfirst]

I thought it already had.

More bloat!

[Hobart]

I thought this was going to be about a standardization of compressing XML files that got rid of the excess bloat in the markup.

So did I. Then I looked at that example and my heart sank. What the hell! 12 lines of bloated crap text turned into 46+ lines of worse bloated crap!

And they're going to do what, say "gzip it" ? The amount of bandwidth and CPU time this wastes is abysmal.

Someone needs to stop these people.

Re:More bloat!

[rohanl]

Then I looked at that example and my heart sank. What the hell! 12 lines of bloated crap text turned into 46+ lines of worse bloated crap!

You do realise that the original just includes file names, and the XOP version actually includes the serialised data, don't you!

Re:More bloat!

[Anonymous Coward]

So did I. Then I looked at that example [w3.org] and my heart sank. What the hell! 12 lines of bloated crap text turned into 46+ lines of worse bloated crap!

The examples given in the article haven't included the binary data for berevity. The problem that exists now is that binary data has to be encoded into a form compatible with the charset of the document, which usually means base64. This increases the size of binary documents enourmously (think twice), and also requires CPU cycles to encode it.

Being able to send the binary data in a seperate MIME payload means it doesn't need to be encoded in this manner which is a big help for any reasonable sized binary resources. It also means they become first class MIME objects and can have associated headers which provides additional benefits.

Re:More bloat!

[Laxitive]

What the hell is wrong with just gzipping it?
It's just another encoding that happens to be source-language agnostic and provide redundancy elimination.

You have no problem with the overhead of parsing binary XML, but dictionary lookups and tree rotations involved in decoding a compressed file.. that's out of the question?

Not to mention the added benefit that a standard compression layer shrinks not just the tags, but the content as well.

Look, stop thinking of gzip (or bzip, or whatever), as a "compression" scheme. Just think of it as a input-language agnostic encoding mechanism which has the added bonus of eliminating redundancy.

-Laxitive

Re:More bloat!

[Ignominious Cow Herd]

Just think of it as a input-language agnostic encoding mechanism which has the added bonus of eliminating redundancy.

Yeah, leave the redundancy to Slashdot^H^H^H^H^H^H^H^Hthe professionals!

Re:More bloat!

[starm_]

"You have no problem with the overhead of parsing binary XML, but dictionary lookups and tree rotations involved in decoding a compressed file.. that's out of the question?"

huh... last time I checked binary was the language the computer natively understood and it didn't need to be parsed or processed in anyway by software.

Also, it seems to me that he did have a problem with the parsing of the XML part.

Re:More bloat!

[Laxitive]

Uhm, you still need to parse the XML structure.

Technically, ASCII is binary, too. 'A' is 65, which is 01000001. Binary XML will not do away with parsing. The tags will still be there, the content will still be there. Only the restriction that the tags must be an alphanumeric string will be lifted.

Making things "binary" doesn't magically remove the burden of parsing. You know the binary executables you run? The system loader loads it.. and parses it, and arranges it in memory the way it needs to be arranged, and tells the cpu "ok, start executing the code located here". Anything with structure needs to be parsed if you want to manipulate and query that structure in any meaningful way.. and XML is all about structure.

-Laxitive

Re:More bloat!

[starm_]

Yes but usually when one refers to binary, one refers to the prefered language of the computer. Not some transliteration or translation of it that adds yet another conversion step to the data.

And as I suggested above he did not like the XML tags either calling them:"12 lines of bloated crap" and all.

I feel like Im talking to a two year old. I don't know what else to say. If you can't comprehend that binary is much faster to parse than XML theres nothing I can do. Oh I give up you're right. I propose to change the CPU and memory of the computer to work in XML.

A3D2

D22A ...

Re:More bloat!

[starm_]

What I meant to add was:

<ASM instruction="JMP"><PARAMETER type="32bitAdress">A3D2</ASM>
<ASM instruction="NOP"></ASM>
<ASM instruction="ADDA"><PARAMETER type="32bit Integer">D22A</ASM></ASM> ...

Re:More bloat!

[interiot]

You're talking about two different kinds of parsing. Breaking out opcodes is hugely different from counting '<' and '>' characters 30,000 times in a row, just to find one little bit of information burried in the middle of text, but you just don't know where.

Why are databases fast? Indexes. What do all XML databases do? Store XML internally in a way that machines read much faster, but makes it a pain for humans to update. Indexes. So if you have all these computer programs passing around data, each with their own different deserialized structure, why don't we just standard that side of things too? Indexes. Computers MUST have them to work efficiently, but no human updates them by hand. Ergo, one form easy for humans (plaintext XML) and one form quick for computers (indexes), with an easy way to convert back and forth.

Re:More bloat!

[Unordained]

Databases (typically relational) aren't just fast because of indices; they can also assume more about the structure of the data. A table (relation) is a set of tuples, each one with the same attributes, each one with a single value (however complex it might happen to be, in spite of what OODBMS people think.) When you've got that, you only need to store the meta-structure of the relation once, in the relation header. Then you can assume all sorts of stuff about what's to follow, and you can optimize the hell out of the data (CSV is so much more efficient than XML for transmitting relational data, it's amazing. And it's not even good.)

On the other hand, when you've got a document in which structure fluctuates wildly from item to item, and you have to look at the item to know what it might be (let alone
whether or not it's what you're looking for, what you can do with it, etc.) ... things take a little longer. XML just doesn't have shortcuts for "and now some relational data" that can speed that up.

Then again, when you're sending data, it seems redundant to send indices -- that's the sort of thing each receiving party should recreate as desired. But nobody said XML was about efficiency in reading/writing/searching/manipulating. It was about a standard, do-almost-anything, here's-a-prewritten-library-for-you, read-file-until-you-grok-it format. Efficiency's not in the list.

Re:More bloat!

[bit01]

If you can't comprehend that binary is much faster to parse than XML theres nothing I can do.

Where is your numerical proof that binary is much faster to parse than text? It is amateurish to just assume this is true. Good parsers are damn fast and can operate in O(n) time.

Of course binary may be faster. I doubt that it will be much faster when compared to a decent parser and when you realise that the binary format should be platform agnostic for word size, endianness and forward and backward compatibility.

For instance, gzip'ed text files can sometimes be much faster to access than uncompressed binary files because it reduces the amount of file IO. e.g. 64 bits of binary to encode the number 1 rather than 8 bits of text.

While compression increases the CPU usage because the disk is so much slower and because the CPU might otherwise be idle waiting for the disk it can lead to an overall win. The same may apply to a slow network link. Unless you measure it is difficult to know. I've lost count of the number of binary formats I've seen that in hex dump had vast numbers of zero bytes and were thus highly inefficient. The people who work at a "high level" designing such file formats without checking such simple things are poor programmers. Even when using indexes the saving of a single extra random disk/network access can sometimes justify a huge amount of CPU usage.

---

Don't be a programmer-bureaucrat; someone who substitutes marketing buzzwords and software bloat for verifiable improvements.

Re:More bloat!

[DrSkwid]

I use a trinary computer, you insensitive clod !

Re:More bloat!

[PostItNote]

Just CPU actually, and we have that to spare - and you don't even need to explicitly gzip it now that mod_gzip is widespread on both the client and server end.

Re:More bloat!

[ikkonoishi]

The point I believe is so you can serialize actual data in an xml format without having to uuencode it into a CDATA struct.

Re:More bloat!

[master_p]

Why XML isn't like the following? it certainly has much less characters, plus it is similar to the most popular programming languages:

envelope(soap="http://www.w3.org/2003/05/soap-enve lope", xmlmime="http://www.w3.org/2004/11/xmlmime") {
body {
data(m="http://example.org/stuff") {
photo(contentType="image/png") {
/aWKKapGGyQ=
}
}
sig(contentType="application/pkcs7-signature") {
Faa7vROi2VQ=
}
}
}

It's easy to parse, and editing tools need small modifications to handle it.

As for namespaces, they could do something like 'using namespaces' or 'import', in order to avoid repetition of the same namespace tags over and over.

Re:More bloat!

[ray-auch]

NOT in that example it doesn't - just a one line comment saying they didn't include the data.

The extra lines are the additional Mime and XOP packaging.

Re:More bloat!

[ray-auch]

There is no binary XML here. The XML is not compressed (go ahead and gzip it as well if that is what you want).

This is about packaging other binary data _within_ XML. RTFA

Re:More bloat!

[mabinogi]

Base64 is not twice - it's 3 bytes into 4 characters - with a little overhead, as technically you're supposed to put a newline after every 78 characters - but I suspect that's only really relevant in the context of email.

But it's still significant enough difference to easily make the XOP overhead irrelevant in all but the smallest cases.

Re:More bloat!

[Anonymous Coward]

Good parsers are damn fast and can operate in O(n) time.

And n is smaller for binary data; in a best-case situation (XML document consists largely of tags rather than text, tags are 10-20 characters but can be reduced to single bytes in a binary encoding), that would mean that switching to binary data would give you O(n/10) parsing, i.e. an order of magnitude faster.

Ergo, binary XML could theoretically give you a considerable performance enhancement.

I've lost count of the number of binary formats I've seen that in hex dump had vast numbers of zero bytes and were thus highly inefficient. The people who work at a "high level" designing such file formats without checking such simple things are poor programmers.

You demonstrate your ignorance once more.

• Those vast numbers of zero bytes will be being used to pad records to equal lengths. This can make lookups O(1).
• Those vast numbers of zero bytes (a) compress better than ASCII text in any decent compression program (so gzip would do much better on those formats than text-based XML) and (b) take up NO SPACE AT ALL on any modern file system that can handle sparse files.

Re:More bloat!

[pomakis]

Good parsers are damn fast and can operate in O(n) time.

Even a horrendously slow XML parser operates in O(n) time.

Re:More bloat!

[bit01]

Even a horrendously slow XML parser operates in O(n) time.

True. I just wanted to make the point that a well written/generated parser is not only fast but scales well.

---

Patents by definition restrict distribution and are incompatible with standards which by definition are supposed to promote distribution. Say no to patents in standards!

Re:More bloat!

[bit01]

What is it about binary versus text formats that brings out the religion in people? Almost as bad as vi versus emacs.

I am happy to use binary formats when they have clear, measurable advantages but, like programming languages, readable data formats have advantages for debugging and understanding that need to balanced against. YABFF (Yet Another Bloody File Format) syndrome also needs to be considered here. To the extent that XML is a general purpose file format I am happy to consider yet another binary file format but the onus is on the proponents to show it has measurable advantages compared to text compressed with an existing, general purpose compressor like gzip. Binary formats can be considered to be domain specific compression and if there's one thing the compression literature has taught us it is that it is usually hard to beat the general purpose compressors with special purpose compressors on real data by more than a few percent.

Good parsers are damn fast and can operate in O(n) time.

And n is smaller for binary data;

It can be. Not necessarily because numbers in text form are variable length (1/2 bytes) and typically will be shorter when compared to 64 bit integers.

in a best-case situation (XML document consists largely of tags rather than text, tags are 10-20 characters but can be reduced to single bytes in a binary encoding), that would mean that switching to binary data would give you O(n/10) parsing, i.e. an order of magnitude faster.

Not compared to text compressed with an existing general purpose compressor rather than YABFF. With a good parser that takes only a few microseconds per byte. Parsing time, for a good parser/compressor, will almost always be dominated by the IO time.

Ergo, binary XML could theoretically give you a considerable performance enhancement.

Probably some enhancement, yes. But enough to outweigh the support overhead of YABFF? The necessity of supporting convertors, documentation, education, opaque data formats, potential inconsistency, binary corruption/recovery, debuggers etc. etc.? That is the question here.

I've lost count of the number of binary formats I've seen that in hex dump had vast numbers of zero bytes and were thus highly inefficient. The people who work at a "high level" designing such file formats without checking such simple things are poor programmers.

You demonstrate your ignorance once more.

Not really. I'm well aware of the tradeoffs. It is you that appears unable to grasp that YABFF requires a significant and measureable advantage to outweigh support costs.

• Those vast numbers of zero bytes will be being used to pad records to equal lengths. This can make lookups O(1).

Pointless if a random access will take milliseconds. Usually faster to just slurp the whole thing in and interpret it there. Depends on the file system, the block size of the file system, the seek times of the disk, the read times of the disk, the size of the disk caches and the size and fragmentation of the file amongst many other factors.

• Those vast numbers of zero bytes (a) compress better than ASCII text in any decent compression program (so gzip would do much better on those formats than text-based XML)

This assumption is incorrect. My experience has been that gzip does a better job on text based file formats; not surprising given the extra redudancy. Depends on the file, file size and statistics of the file of course.

• and (b) take up NO SPACE AT ALL on any modern file system that can handle sparse files.

In a file system that supports sparse files only if the zero's are not written. I'm also not just referring to zero's but to constant values and patterns. On a compressing file system a text file will be compressed as well as a binary file. Compressing file systems may require extra disk accesses making their advantage minimal.

---

Don't be a programmer-bureaucrat; someone who substitutes marketing buzzwords and software bloat for verifiable improvements.

a bit confused

[newsdee]

I'm a bit confused... reading the document, it seems that the difference between XML and SOP is just where the data is:

XML:

<mylabel>(text)</mylabel>
<mydata>(stuff in binary)</mydata>

XOP:

<mylabel>(text)</mylabel>
<mydata>"hey, there's stuff in binary here, id 1!"</mydata>
---- MIME ---
Binary ID 1: (stuff in binary)

Is this right? So the benefit is just standardizing the binary representation using MIME? But that doesn't make the tags less verbose... so how is this faster than XML?

Re:a bit confused

[Svartalf]

Um, I believe that the data was encoded in the same manner as the "optimized" format, it's just that now you have to put it at the end in a MIME encapsulated formatting. Zero doesn't factor into this in the first place. Now, concerns of accidentally closing a tag might play into things, but the likelihood of this actually happening is slim to none as the coincidence of "" in a base 64 stream is going to be an astronomical feat. However, I buy into that as a reason (can't have accidental data loss that way...) but to call it "optimized" is really, really counter-intuitive as it's NOT optimal.

Re:a bit confused

[MassacrE]

Incorrect.

XML, being a text format, required proper text encoding. In particular, XML does not allow most of the codepoints (speaking in unicode terms) between 0 and 31 (tab and newline excluded). If you use UTF-8, you cannot use byte values beyond 126 as those are used for forming higher-value unicode characters. In addition, the five main XML markup characters (< > and &) can only be used in some places.

So, to make a long story short, you base64 everything. For every three bytes you have, you output 4, giving you a 33% increase in space.

outside of the XML document you do not have to require text. data can be considered 8-bit clean and sent in a big binary block.

So for example, an additional requirement of 200 bytes for specifying all the MIME information would be made up for within the first 600 bytes worth of binary data. Even without this space benefit, you get the benefits of a standard way of including binaries, and the ability to potentially access the binary data directly if the transport was indeed 8-bit clean.

Re:a bit confused

[brunogirin]

You're right. The point is to standardise the binary representation, not to make it faster. But it doesn't mean this hasn't got very useful applications in the real world. Here is a situation where this technology will be useful:

You design a web application. It all looks good, you can access all functions through a web browser. One of those functions involves uploading binary files (let's say images) so that they can be retrieved online later on. Fine, HTTP allows you to upload binary data using MIME.

Now, your PHB comes in and tells you your biggest customer would like to be able to integrate directly with their own system over the internet and they can do it through web services. So you need to add web service support to you application. This means using XML as data exchange format. No problem for all textual data but what do you do for your upload function that happens to be an essential part of the application? If you use a proprietary mechanism to encode that binary data in the XML stream, chances are you and your customer won't be compatible and someone will have to do extra development. Using XOP, you know both sides are compatible out of the box and you can re-use the same routines available in your HTML channel to decode the MIME encoded data.

All in all, using XOP would probably save you weeks of development and enables you to re-use existing MIME code to have a leaner and simpler system. It's not rocket science but it's that sort of simple but sensible standards that make our day to day work easier.

Re:Binary... XML... Nah!

[ingo23]

The benefit of XML over SQL is that XML is a format while SQL is a language. It's like comparing a verb to a noun.

Things like SOAP do not make it seem like a protocol, they just use the format.

Adding binary objects to XML specs is going to be quite useful. And the point is not that you could not do that before. The point is to make it a standard.

This is NOT binary XML

[IHateSlashDot]

I can't believe all of the replies making fun of this because they think it's a binary representation of XML. Didn't anyone read the RFC that was referenced in the summary?

This is simply a way to reference binary data from within an XML document and to have that binary data included in the same payload (using MIME).

Passing binary data in XML is a big problem. Everybody just invents their own method of doing it (although most are just variations on the theme presented here).

There is a need for this specicification but it is not ground breaking or even particularly /. newsworth.

Re:This is NOT binary XML

[seanadams.com]

This is simply a way to reference binary data from within an XML document and to have that binary data included in the same payload (using MIME).

And you find this less absurd?!?

Re:This is NOT binary XML

[lupin_sansei]

What was wrong with CDATA? I thought you could embed binary into XML that way.

Re:This is NOT binary XML

[Leo McGarry]

The "c" in "CDATA" stands for "character." It's technically supposed to include character data encoded in the document's character set.

Can you put arbitrary bytes into a "CDATA" section? Sure, prolly. But there are people out there who get paid by the hour to cook up new standards for stuff we'd already figured out. You don't want them to have to go out and get real jobs, do you?

Re:Binary... XML... Nah!

[tepples]

it is the wrong tool for the job, much like using tables to layout HTML pages (as the CSS religionists like to point out)

CSS religionists like to deny the existence of the 90% browser, whose CSS implementation has too many bugs and deficiencies to make it a complete replacement for some forms of table based layout. Personally, I prefer using a hybrid of tables and CSS on sites that I develop.

My 64 million dollar question is why they put an acronym inside another acronym

It gets worse: RISC = Reduced Instruction Set Complexity; POWER = Performance Optimization With Enhanced RISC; PPC = POWER for Personal Computers

Thank you!

[gammoth]

I thought I was losing my mind.

What's the deal?

[j.bellone]

I really don't understand. XML is great for information that might need to be easily parsed, readed, or changed by a number of applications. In fact; it's probably something that we've needed a lot longer than we've had it. But that's where it ends.

Usually (at least, from my limited experience) a serialized object from a program is normally only needed to be loaded up by that program (which usually results in their own .dat files). Still when I have been toying around with my own serializer; I haven't found a need to actually need to hand change any of this information.

Why reinvent the wheel though? It seems this would only be useful in web application for a database of sorts. The only usefulness (actual useful ness) would be for a document format (Word, Excel, Access, etc). This is the type of format that you would want to be portable; easily changeable (if need be) by hand; and thrown across several different platforms where it can be read.

I guess that would make there to be a use for it then; but do we really need a standard for the such? Anyone who plans on using this idea will create their own standard anyway (look what Apple has done with their new format for Pages).

Noscript

[tepples]

[CSV is] Easy to manipulate via javascript on the client. Simple to display and manipulate via the DOM (Document Object Model).

Problem: A lot of users will try to browse your site with DOM scripting turned off to avoid the more annoying advertisements and malware installers. What will you put in the <noscript> element?

Re:Noscript

[tomhudson]

I'd tell them to switch to firefox :-)

It's time to stop thinking of "web sites" and start thinking along the lines of "web apps" - not the old-style form-based "web app", but more along the lines of gmail - heavily client-side-scripted, nice presentation and data manipulation.

What I see is very few pages (or even just 1 page) as the UI, data exchanged between server and client w/o page refreshes (can be done just w. javascript by sticking the data in iframes with a width and height of 0px, and reading/writing to the iframe. no need for a separate "data window" hidden from view - happy coincidence - I wrote code to do this last night).

These work without a hassle even with popup blockers, etc. It's not necessary to turn off *all* scripting capabilities. Just get a competent browser :-)

Re:Noscript

[DrSkwid]

Where can I download Firefox for symbian ?

Where can I download Firefox for PocketPC [or whatever it's called this week] ?

Re:Noscript

[DrSkwid]

do you even know what Symbian is ?

it certainly isn't anything like "the primary focus of Minimo to date has been system with ~32-64 MB of RAM, running Linux"

I said where can I download Symbian not where can I look at a project that doesn't even have Symbian on the roadmap.

Could have been simpler

[Anonymous Coward]

.. if they just mimic'd the CDATA section with an equivalent BDATA section. Would be so easy to just jump to the end of the binary data given the BDATA offset and continue on.

I know it's easy, because that's exactly what I did, hacked the gnome libxml library and it worked nicely, was easy to code (yank/paste from CDATA) and best of all it was *fast* without consuming resources like base64 does (tried that too originally).

Critiques

[Effugas]

Ummm...it's "OK". This is probably the least ambitious Binary XML spec imaginable. That may actually be good, but I don't know. Lets see what's up here...

First of all, it's completely impossible to stream this format. All the binary chunks have to be read at some point in the future when the actual XML non-opaque content is complete. In a stream, that never happens. (Of course, XML isn't the most stream friendly protocol...you can't validate a stream.)

Secondly, this isn't wonderful for large files either; you're constantly seeking for binary data that can be many megabytes away. We solve this in web pages by having the images be completely separate (binary) files.

Thirdly, its telling that they used a PNG as a data type. Besides being yet another file format that needs its own custom binary parser (heh, I like PNG, I'm just complaining about it in the XML whinespace), it's big and simple and there's just one there. One of the things I really liked about the various Binary XML formats was the degree to which they expressly typed things like arrays of floating point values or little-endian integers. Converting values between binary and string format is an enormously painful process, one that frankly I'm astonished hasn't received CPU acceleration at this point. Every other Binary XML format has seriously thought about how to efficiently but correctly manage large arrays of such values. XOP just says...heh...you wanna dump alot of data efficiently? Check your typing at the door. Feel free to bring a buffer-overflow ridden parser in with you if you like, though.

Don't get me wrong, there's a fundamental simplicity to XOP that I can certainly understand how it's appealing. But it seems to go so massively against what XML represents that I'm not entirely sure XOP encoded content deserves to be compliant with the very regulations that forced XML adoption in the first place: Opaque formats are too expensive to maintain for any amount of time, therefore either self-describe or don't get deployed. A self-decribing document that says "All performance-critical content is opaque" seems rather counter to this spirit.

Re:Critiques

[thpr]

I agree with your post - and really wonder what people are doing with binary data in their XML files anyway. But I digress from what I wanted to ask about:

Converting values between binary and string format is an enormously painful process, one that frankly I'm astonished hasn't received CPU acceleration at this point.

This initially strikes me as a brilliant idea, but as I got to reply, I'm wondering how useful it would really be. Many if not all of things we accelerate today (e.g. FPU and matrix operations) have have significant uses in loops which do not require significant access to memory - which is the bottleneck in a typical PC.

Under the assumption that XML is being used for I/O of some sort, I would guess that the network, disk or memory is already the bottleneck. I can't envision a small loop that could swamp the CPU with conversion (like a scientific simulation could swamp a CPU without an FPU because the calculations don't necessarily drive heavy memory access)

I'm thinking out loud here, but can you think of a conversion case (XML or not) where the conversion speedup wouldn't just cause a greater bottleneck on the memory bus?

Re:Critiques

[Effugas]

Hmm. Threw together some code to play with this.

#include <stdio.h>

int main(int argc, char **argv)
{
char *mixed = "0.123124 12345";
double foo;
int bar;

int i;

for(i=0; i<1000000; i++){
sscanf(mixed, "%f %i", &foo, &bar);
}
}

Results:

$ time ./bench.exe

real 0m2.541s
user 0m2.393s
sys 0m0.010s

So we're looking at maybe 787K symbols per second on my machine, at 100% CPU. How does this translate to XML parsers? You're right, this is something I should look into.

Re:Critiques

[OldMiner]

This looks like a false benchmark. You're keeping mixed in cache the whole time. In parsing actual XML, you'd be loading new strings in and out of the cache the whole time. As stated by grandparent, I doubt you'd be CPU bound.

Re:Binary... XML... Nah!

[a_karbon_devel_005]

Really? You can't see the benefit?

You mention SVG, but then you fail to see the benefit of reducing the size of, say, a large SVG file in a standards compliant way so that it can be transferred and take up less bandwidth. A good binary standard will DEFINITELY be smaller than the verbosity that is XML. Sure you can compress it, but when you compress a whole bunch of unneeded crap, you still have a whole bunch of unneeded crap... just compressed. If this standard reduces the amount of space it takes to write:
<someLongTagWithANameLikeThis>1</someLongTagWithAN ameLikeThis>

... that will help a lot. People have been clamoring for a STANDARDIZED recommendation for XML for a LONG time.

Headline should read...

[WasterDave]

"Remember the recent discussion on Binary XML? Well, this has nothing to do with it, but we are proud to present a standard for larding out XML even more before attaching it to an email."

I, for one, welcome our new bandwidth eating plaintext overlords.

Dave

Re:Binary... XML... Nah!

[tha_mink]

it is the wrong tool for the job, much like using tables to layout HTML pages (as the CSS religionists like to point out) CSS religionists like to deny the existence of the 90% browser, whose CSS implementation has too many bugs and deficiencies to make it a complete replacement for some forms of table based layout. Personally, I prefer using a hybrid of tables and CSS on sites that I develop.

Be fair...(and I am a Firefox guy) NO [ right...NO] browser fully conforms to CSS standards. I am a CSS religionist and have found it impossible to test my work since NO [right...NO] browser fully conforms to CSS standards...(Yes folks...that means CSS1 AND CSS2) BTW...I thought XML was on its way out...

This business will get out of control.

[wizard_of_wor]

It will get out of control and we'll be lucky to live through it.

Seriously, this is strongly reminiscent of designing C++ APIs, called only in-process by C++ code, that use XML blobs for every single parameter type. I came across one of these and asked the "architect" why he chose to use XML for every parameted (at significant cost).

"Well, you know, it's XML," he said.

"And?" I asked.

"Well, it's... I mean, c'mon, it's extensible," he explained.

When all you have is a hammer, everything starts to look like a nail. And there are too many developers who are a few parts short of a toolbox.

Re:This business will get out of control.

[renehollan]

The scary thing is that there is a damn fine justification for this... but the "architect" in question likely doesn't know it.

Using an XML representation for method parametrization provides a common interchange format when linking bitween different implementation languages, and, obviously, across network boundaries. Furthermore, the natural XML specification of method signatures makes possible the compilation of said method signatures by different compilers.

Of course, the right thing to do is to compile to a more appropriate native format if the linked/networked methods are known before execution time -- not doing so if they are (i.e. for a monolanguage in process lingage) would be quite silly. One would resort to XML all the way to execution time only if the target of a call was implemented in an unknwown manner (which is why XML works well in public networked applications -- it's inefficient, but a reasonable lingua franca for structured data).

Then again the reverse is true: any modern compiler worth its salt should be able to produce XML representations of method signatures.

The real extensibility question here is whether the method call semantics in your application lend themselves naturally to in-process as well as networked process implementations: what is naturally a synchronous method call in process might be better as an asynchronous method call in a networked environment -- it can sometimes help permit the use of single-threaded multi-instance clients instead of multi-threaded synchronous clients.

Re:Why???

[danielrose]

Agreed on both counts!
Whats wrong with getting the regular xml and zipping/compressing it with your tool of choice? Does it really need a whole new spec?
In other news, I'm putting that link in my sig.

Accessibility?

[tepples]

This page is designed to be viewed with scripting enabled. Enable scripting and click here to refresh, or click here [goatse/tubgirl]

Which text-based web UA supports DOM scripting? Or would you deliberately shut out users with vision disabilities and run the risk of losing your U.S. government contracts under Section 508 and having to defend a lawsuit under the Americans with Disabilities Act or foreign counterparts?

Re:Accessibility?

[ytpete]

IE, as much as we all hate it, supports screenreaders pretty well. Look up a program called "JAWS" for example. Javascript and screenreaders *can* coexist in a non-text browser.

RTFA - Re:Binary... XML... Nah!

[Ignominious Cow Herd]

It is not binary XML. It is a method to extract binary data that is embeded in XML (e.g. CDATA) and store it outside the XML, but in the same document. It is NOT a method to reduce the text encoding (overhead) of XML to a binary format.

Re:Binary... XML... Nah!

[Leo McGarry]

While I'm sure there are some people out there who think the idea of an acronym that means itself is just the funniest thing ever, the gnu (the "g" is silent; it's pronounced "nu") is one of two species of African antelopes. They have big heads, long tails and horns. They smell fairly terrible, but they're really very beautiful ...as long as they're downwind.

Like I say, I'm sure there are some people who think the old "it's an acronym" joke is a real knee-slapper. But it's kind of a shame that the people who chose to name their organization after a beautiful animal turned around and dissed that animal by trying to define its name.

Screen readers

[tepples]

I'd tell them to switch to firefox :-)

How well do the major Gecko-based web UAs work with screen readers, compared to say Lynx/Links/w3m?

Re:Screen readers

[ytpete]

Check out the Sharkware project. I think this was actually mentioned on Slashdot recently.

Maybe I forgot to mention...

[Spy der Mann]

For those who didn't RTFA:

The main application of this XML-referencing-to-binary-attachments is SOAP, and that means web services.

In other words, you can simplify your God-help-me-XML-handling-and-parsing-code into something maybe 10% simpler. This means leaving the binary stuff OUT OF THE XML PARSER, putting it into the upper levels or processing. Cleaner, faster.

Also, it helps adaptive compression (gzip) by tightening up the textual data - remember web services are about information transfer, not storage.

Re:Binary... XML... Nah!

[Leo McGarry]

NO [ right...NO] browser fully conforms to CSS standards

Which kinda makes one question whether having such so-called "standards" is really worth all the trouble.

Amazingly, HTML compatibility was easier before it was "standards" this and "standards" that. There were certain constructs that only worked in certain browsers, sure, but we didn't have the god-awful mess of supposed-tos and should-nots that we have today.

It seems to me, from a distant perspective, that the problem with Web standards isn't that nobody adheres to them. It's that they're really shitty standards.

Script Data Structures in place of XML

[Camel Pilot]

I am currently writing a xul client/server application. I am using the xmlhttprequest function. however instead of processing xml data which is very slow, especially when you need to parse a data set several times a second, I started sending data stuctures in javascript code instead. This I believe is what Google Suggest does also.

In addition the server code is written in perl so for storing status and configuration information, I used serialized perl data strucures processing requirements fell dramatically. With serialized scipt you still have the clear text editing and inspection capabilities without the speed and space issues. for example instead of

<container>
<title name="title">
<item><name>Name1</name>
<item><name>Name2</name>
<description>Bla bla</description>
</container>

You have:

{
title=>"title",
item=>[ { name=>"Name1" }, { name=>"Name2" } ],
description=>"Bla bla"
}

It seems like serialized script code, in either perl, python, java provides the benefits of xml without the headaches.

Re:Script Data Structures in place of XML

[oliverthered]

I'm currently writing some C code and I'm using char* pointers and C structures instead of XML,
you should see how fast my app runs.

I believe this is the way the linux kernel does things.

XML is good for data interchange between anonymous sources any other use is more or less an abuse.

Re:Script Data Structures in place of XML

[zallus]

{
title=>"title",
item=>[ { name=>"Name1" }, { name=>"Name2" } ],
description=>"Bla bla"
}

Hmm... I think that can be a little more standard-compliant, without increasing length any:

title: title
item:
- name: Name1
- name: Name2
description: Bla bla

Mmm... YAML.

Re:Script Data Structures in place of XML

[lux55]

Returning a JavaScript data structure is sufficiently compliant (ie. it's part of the language itself), and has the benefit of skipping the deserialization step entirely.

Plus, on the server-side, you should be able to write a JavaScript serializer in 30 lines or less. For example, the one used in our project (sitellite.org) is exactly 30 lines of PHP code, and properly handles strings, numbers, booleans, arrays, and objects.

Re:Script Data Structures in place of XML

[Erwin-42]

This idea was on Slashdot just a few days ago -- look up JSON.

Re:Script Data Structures in place of XML

[spongman]

you've thrown away information:

• the name of the 'container' element.
• the fact that it's the 'name' attribute of the 'title' element whose value is "title"
• what would it look like if the 'foo' attribute of the second item's name element was "bar"?
• it's not obvious how you'd support namespaces or processing instructions...

Re:Binary... XML... Nah!

[dubious9]

My favorite: VHDL: VHSIC Hardware Description Language.

We have a Winner!

[Ignominious Cow Herd]

Only 120 posts before someone said something more intelligent than, 'Binary XML, blecchho!'

Only wish I had a prize for you. :)

Don't need an iframe

[michaelggreer]

Grab the data directly into javascript with xmlhttprequest. However, the web has many Ui conventions now which people don't want broken, even if its better. I still keep hitting the back button in Gmail, and I think it should work that way.

Re:Don't need an iframe

[tomhudson]

Thanks, but why would I want to parse out xml when I can grab my records really quickly with 12 lines of javascript, and don't have to bother xml-izing (is that even a word) my data on the server, either?

Re:Don't need an iframe

[michaelggreer]

Because it is just called "xml"httprequest: it can return anything. So don't be afraid. Iframe feels like a hack to me.

Re:Don't need an iframe

[tomhudson]

It is a hack (but I *like* hacks), but it's better than using an image object, converting it to raw data, and reading the individual bytes - or is it (maybe its time to play around with some more code this weekend :-)

Oh so we store it using binary

[dynamo]

Find me an XML file that is not already represented as binary data. Oh, not looking so revolutionary now, is it?

Wait, you say this allows xml to reference binary data? I say "href" attribute, bi-atch, look it up.

You say, but no, it allows you to send the binary data along in the same stream / document? Check out multipart/mime. It's been around a long time.

Here's a wild thought. Have the XML file reference it's binary resources by relative filenames. Tar the XML file together with the resources. Now pay me $100,000 in consulting fees.

Re:Binary... XML... Nah!

[metalhed77]

Well, prepare to get angry cuz they made these techs a long time ago:

XPath
XSL
XLink
XHTML
XForms
XPointer

It makes sense to put the X in front of tech that uses XML. CSS doesn't need an X because it works on SGML too for instance

Full Circle

[roman_mir]

The circle is complete. We started with binary format, moved to XML for readability purposes and then switched XML back to binary for speed.

Obviously someone needs a knock on the head - when you design your application, don't you think about such things as a balance between performance and maintainability first and then implement what is suited better for your specific case? Obviously not! Just a little while ago everyone and their grandmother switched to XML for whatever reason but then they realized: -OMFG, XML processing is processor intensive! I probably shouldn't have handled every single internal type as an XML string that needs parsing and typecasting for every operation. I probably should have used more suitable memory structures for those data structures that are used within the same application on the same computer and not used a gigantic XML just because I can! What to do what to do? OH, I KNOW! Let's change this char based XML into a binary XML, that will make it faster! (It won't make it more human readable, that's for sure.)

So what's next? A char based XML that wraps around a binary XML for readability? A binary wrapper for a char based XML wrapper to a binary wrapper around a char based XML wrapper for recursive processing?

This is awesome

[HenryKoren]

I was just involved in writing a routine that involved Base64-encoding something to be included in XML, then GZipping the whole XML file to recover the space the encoding tacks on. Now I can just do a XOP file, Awesome!

Re:This is awesome

[thogard]

The cool bit about that is all the junk at the start of the file will make sure that the most common base 64 encoded tokens aren't the smallest bit patterns so your end up with a larger file than if you had just gziped the binary in the 1st place.

Why not XBop?

[lux55]

Wouldn't it make more sense to include the B for Binary, which is the essential purpose of the new "standard"? Plus, XBop sounds more natural when spoken than XOP does, and it's way more fun too! :)

I'd have to agree with you on this... ;-(

[PaulBu]

I've been thinking about the shortcomings of HTML (and everything else that followed it!) from the position of a computer scientist for YEARS... Those standards ARE shitty, big time.

Conmtrast this to IEEE standards -- they get developed when a bunch of companies are ready to invest several mega$$ for a chip spin -- and they just want to choose the best course, arguing with each other about technical merit of this or that approach. And in the whole HT|X/ML world there can be (almost) no competition on technical merits, just a bunch of guys arguing if it should be or BAR .

I wish I'd have the time on my hands and their budgets to actually try something revolutionary. Leke the original WWW, which was NOT designed by a committee...

Paul B.

Re:I'd have to agree with you on this... ;-(

[PaulBu]

Of course, the filter filtered mine &lt and &gt and totally screwed up the meaning of "if it should be or BAR" ;-)

Paul B.

Re:Binary... XML... Nah!

[Anomalous Cowturd]

Hey-

Check out this great site that is giving away totally FREE Mini Macs!

I've joined and I think you should as well.

It's a completely legitimate offer, and this company has already given away $4 million in FREE stuff!

All you have to do is join, complete an online offer, and refer friends to do the same. That's it!

Here is my referral link. To help me get my Mini Mac, click this exact link to join, or copy and paste it into a browser:
http://www.FreeMiniMacs.com/?r=********


My god. It's the first Slashdot spam. Note that the original comment is all in bold with a live link.

I think this referral-Amway thing has gotten out of hand. Are we going to start seeing these things in every story now?

You sir, are a SnapperHead

[SirSnapperHead]

There's something fishy about developers who use XML for Rich Web Client Desktop-Like Application Applets.
You should be using a string and two cans instead.
Technology stinks when SnapperHeads get their hands on it.

I like it: XMLs strength abstract, not concrete

[nightcrawler77]

XML has become at least two things since its evolution:

1. an abstract structure consisting of (possibly-nested) elements and their corresponding attributes.
2. a human-readable representation of that structure

The interesting part of the story is that #2 came first. Since then, the W3C has recommended the Infoset abstract concept.

For the developers out there, think of how often you parse the "angle brackets" yourself. Most everyone these days (yes, I know there are exceptions) uses an API which presents elements and attributes in a wire-format-agnostic way.

As a developer, I would love to have the option to flip a switch in my code to permit Binary XML. If I can read and use the Infoset in exactly the same way, why would I object to the wire format being binary instead of text? My API is the same, but the transport is more compact and efficient.

Human-readable wire formats are great for debugging during development, but provide no real advantage in production systems (provided there are utilities available to produce human-readable XML from the binary wire format.)

Re:I like it: XMLs strength abstract, not concrete

[sgt101]

Remember maintenance! You are 100% correct for fire and forget projects, but introducting an opaque layer into the stack could mean trouble later on. People will start injecting data directly into the wire - for example from network devices or information appliances - and subtle issues and bugs could creep in. I thought this was one of the reasons we ditched CORBA?

The buzzword importance

[phunqe]

Reminds me of a meeting I had a couple of years ago with some representatives for one of the largest market making houses in the US.
Bascially we were promoting an automated trading system and the first question I get is...

"Does it use XML?"

There you have it.

Re:The buzzword importance

[sgt101]

Well - I understand your frustration, but perhaps he wasn't being as dumb as it seems on first blush.

CIOs are challenged by the need to find people who understand poorly documented proprietary data formats, and by finding techies who are techie enough to write parsers for them. XML, with standard ways to use the open source parsing libraries such as the Apache ones, support from every vendor you can think about and a course in every CS department, is very attractive to people in the position of dealing with systems ten years down the road!

Very exciting but ...

[Anonymous Coward]

It's too early yet. I'm waiting until MSBinary_XML comes out

I hear it's going to introduce 263 special MS tags and nodes and extra layers into the standard that only works on MSWord in Windows XP. It won't validate as XML anymore but who cares. You will use a special version of Front Page to do this.

The files will be a little bigger too, so with MSBinaryXML will add approx 257k thanks to the special proprietary MS extensions but will have superior functionality compared to other types.

It will be particularly good at carrying ,pif,.src,.bat and .exe files within the context of an XML binary and what's more MS will be writting low level OS support into future XP updates and Longhorn and a special API to execute the contents of said MSBinaryXML files. It will also communicate with hooks in IE and Active X Controls and MS's excellent Java.

this is gonna be sweet. I can't wait

Re:Binary... XML... Nah!

[DrSkwid]

Amazingly, HTML compatibility was easier before it was "standards" this and "standards" that.

Are you *sure* about that ?

<blink >
<marquee >
<object >
<bgsound >

No-one forces you to validate your html (unless you work for me =). Why I come from it's comformance first, compatibility second.

So, You're Against Innovation?

A common misconception is that folks who advocate HTML validation are retro-thinking, "backwater unix geeks" who stubbornly oppose innovation. It's true that many advocates of HTML validation are indeed seasoned computer professionals, who have learned the hard way that portability and compatibility are key elements to ensuring the longevity of any software product (including Web pages).

Re:Binary... XML... Nah!

[DrSkwid]

/. spam has been around for a long time

Surely you must remember the Free iPods guy, that one wasn't so long ago.

I can't remember any other specific ones because they fade into insignificance.

Attachments, Not Binary XML

[Kopretinka]

These specs (XOP and MTOM) were created becase Web Services people wanted to be able to add binary attachments to XML messages (in SOAP). Initially the attachment technologies (like SOAP with Attachments) worked by just slapping the binary data alongside the XML message, without a clearly defined processing model for the receiver. Now with XOP attachments are logically in the XML document, but physically transported outside without the bloat of base64 or other XML-safe encodings. It's important to notice that XOP is just an optimization of the situation where binary data is put inside an XML document.

Well, doh...

[Kjella]

..if you're going to transfer a small fixed header, XML is not for you. If you actually have no binary data of signifiance, XOP is not for you. Let's say you want to include a 300kB picture in your XML. Your choices are:

1. External link (unpractical)
2. XML/Base64 encoding (~450kB)
3. XOP/binary encoding (~300kB)

In that case, your 30+ lines of extra code are completely irrelevant. That being said, I was under the impression that you could do this already by sending your binary data in a "document fragment" element, where the only valid characters to end it would be ]]> or so (been a while since I looked at it). A simple remap of that one combo using e.g. a yEnc variety ahould let you pass binary data with minimal overhead. It'd be a hack though, the "document fragment" is intended to let you place XML fragments there, hence it ignores normal XML tags.

Kjella

Re:Binary... XML... Nah!

[Leo McGarry]

Haven't the foggiest idea what your post was intended to mean, or what it had to do with mine. Got any clarity, brother? Spare a spot of illumination?

base64?

[bsiggers]

Why not just base64 encode what you need to? Jeez, talk about a solution looking for a problem. Who in the world is encoding huge binary stuff in their XML anyway?

Re:base64?

[vidarh]

Duh. Read the spec. Most people who include binary in XML DO base64 encode it. But base64 wastes a lot of space. If you want to include larger amounts of binary data, this standard lets you save space by using a MIME wrapper and referencing a MIME part containing the raw binary data from the document instead of inlining it directly.

And you're right, most people don't want to include huge binary stuff in their XML. But sometimes you DO need to combine XML with huge amounts of binary data. So far, the alternatives have been non-standard wrappers (including people doing more or less what this standard does, by using MIME multipart documents) or base64 or some other space wasting encoding inside the XML document, or wrapping everything in an archival format (like OpenOffice does, for instance).

All this does is define a standard way of letting you keep a document and associated raw binary data together, while allowing you to treat it as if it is inlined in the XML if you so choose.

The principles are exactly the same as for sending an HTML e-mail containing images (or other data) as attachments and referring to them with url's of the format "cid:foo" (they refer to the MIME element with the matching "Content-ID: foo" header.

Re:Binary... XML... Nah!

[DrSkwid]

I'll summarize then :

Yes, it is worth the trouble.

Re:Binary... XML... Nah!

[Anomalous Cowturd]

I don't remember a specific iPod guy, just all the sig links - I guess I was lucky there. This is the first out-and-out spam I've seen, that looks like it could have been in my email. I fully expect to see Costa Rican land scams and Vioxx ads any day.

Just remember what Usenet used to be like. This could get a lot worse.

Re:Binary... XML... Nah!

[brushybill]

Except that SQL is nothing at ALL like XML... XML is a markup language, vs. SQL, which is a concrete expression language. I actually can barely understand your implied relationship between the two... SQL is a relatively static, industry defined syntax for performing database operations, while XML is a completely open-ended, schema based facility for defining basically ANY piece of knowledge. In fact, I really can't comprehend why you'd even compare it to SQL...

Mod Parent Up

[ray-auch]

Mod parent up (since I can't and I'm fed up of making the same point)

Obligatory ...

[quarkscat]

yes, but is it available in PDF format?

Re:I am enlightened

[vidarh]

You do realise that the examples where not intended to demonstrate the benefits, don't you? (as in, the fact that they're only showing it used for tiny binary sequences doesn't mean that will be the normal usage) The complexity increase is minimal, as there are tons of libraries available to handle MIME, and "cid:..." style URL's have been in use for e-mail for ages, and it'll be fairly trivial to scan a document for nodes in the right namespace and expand the include tags.

The main reason it is defined the way it is, is exactly so that the "complexity" of it can be easily hidden in the parsers if you prefer not to deal with it yourself - base64 is well established as the preferred way of inlining binary in XML. With this spec you now have a way of saving space by using a MIME document to wrap it and move any large binary sections out, while allowing software using parsers with support for this standard to treat it just as if the binary data is still inlined base64 if you don't want to change your app.

Re:Binary... XML... Nah!

[TeknoHog]

My 64 million dollar question is why they put an acronym inside another acronym: XOP stands for XMLOP? WTF??!!

Never mind the question why people say "acronym" when they mean "abbreviation"...

Re:Binary... XML... Nah!

[zootm]

Another point is that SQL is more limited in expressive power than XML - foreign keys can redress some of the difference, but not even nearly all. XML is Turing-complete, and SQL can only query on first-order predicates, if I remember correctly.

This is why XML query optimisation is so difficult, and why there's so much work still going into it.

Why oh why?

[hummassa]

Why would an external link be unpractical?

Congratulations!

[hummassa]

You just (re-?)invented TeX.

what about matroska ?

[dangil]

what about matroska ?

isn't it supposed to be binary XML ?

SXML

[Anonymous Coward]

Ever heard of SXML? It's XML represented as Lisp-family S-Expressions. This makes it a native data structure to e.g. the Scheme programming language. It's also lighter than XML, since it is less redundant and verbose. Your XML (which is not well-formed XML, you left out a bunch of closing tags) example would become:

(*TOP*
(container
(title (@ (name "title")))
(item (name "Name1"))
(item (name "Name2"))
(description "Bla bla")
))

XML For Buzzword Based Engineers

[Jagasian]

There are two types of engineers. Those who can get the job done, and those who will try to use the buzzwords of the day to get the job done. The former tends to be a better engineer, while the latter tends to look better at the beginning of the development process.

XML will come full circle when true binary XML is a w3c standard. People will be using high-level GUIs to generate text-based XML files, which will be converted into binary XML. On the other end, somebody will receive binary XML, convert it to text-based XML, open it in their application that presents the data in a high-level graphical format.

Only then and only maybe, will everyone begin to realize what a farce XML really is. Its selling point was that it was easily human readable, as well as machine processable unlike other more simple formats. Everyone will realize that XML was never very readable (unless you were blind to angled brackets) nor was it easy to efficiently parse compared to existing formats. Everyone will realize how inefficient XML is, and with the roundabout nature of using GUIs to generate XML and then another tool to convert it to binary and then another for binary to text again and then a final GUI... ...it is like running in circles. The buzzword engineers need to put in their place. Let the real guys lead again, please.

Re:Binary... XML... Nah!

[say]

Amazingly, HTML compatibility was easier before it was "standards" this and "standards" that.

If you're referring to the days of Netscape 3 and IE 1, you must have a very bad memory or never been engaged in making a webpage these days. The XHTML standard, for instance, is really easy to understand, and as long as you use simple CSS, you'll get the same result in a lot of browsers. Yes, IE has misunderstood some of the CSS specification, and the CSS2 layers model is far too advanced for today's browsers. But the differences are extremely small now compared to that of 1996.

goodbye to quirks mode

[Zirtix]

Hear hear. As soon as XMLHttpRequest is a proper standard I'm jumping on it like a shot. Like Gmail, it's time to raise the bar on which browsers we web developers are going to support. CSS2, DOM 2, and XHTML please. IE6 just about makes the cut ;)

Re:Binary... XML... Nah!

[NoMercy]

This is a way to put binaries into XML, eg a jpeg image without incuring a huge size penalty to encode the data in viweable characters only.

I doubt a binary version of the XML format would see any popular use, but a decent compression system for XML would be a nice bonus, just something to put alongside the other compression standards commonly seen in HTTP 1.1's Accept-Encoding header.

Re:Binary... XML... Nah!

[passionplay]

I think everyone that's posted to this thread to this point has missed the point here. This XOP optimization has nothing to do with making XML more compact or anything. It has to do with delaying latency for large payload transfers and allowing the client application to decide if it wants the large binary payload.

Seriously, you guys need to re-read the article again.

The problem with XML binary payloads now is that you find out that you have a large chunk of payload too late in the game and can't avoid it if you don't need it.

This method allows you to know everything there is to know about the payload before you get to the payload.

In theory, you should be able to skip all the payload data until such time as you really need it, thereby speeding up large transfers of XML data when only the metadata about the payload is required.

Make sense? I think so too.

P.S. Binary XML is entirely different animal.

XML 'is' useful, just not this binary XML spec

[Drog]

There's a lot of XML-bashing going on here from people talking about how XML is just a buzzword and how XML is not necessary. Sure it's a buzzword, and sure it's unnecessary in some situations. But that doesn't make it useless.

I create data-driven web apps for a living (i.e. data-driven graphics, UI and text via SVG and HTML), and I firmly believe that XML is the way to go for such creations. It offers a hierarchical structure that is excellent for temporarily storing data pulled from a database, which can then be converted to HTML or SVG or some UI markup (XUL, XForms, or your own thing) via XSLT.

I don't really care that XML is human-readable--I like the fact that because it is extremely well structured, it is therefore easy to create with authoring applications as well as being easy to manipulate real-time by with script (i.e. manipulating its DOM).

I have long wished for a true binary XML spec to make the transmission and parsing/decoding quicker, and this spec isn't it. But I think one day we'll have it, and that won't mean that we've "come full circle" and therefore XML is useless. It just means that we'll have the best of both worlds--speed plus standardized, hierarchical data structures.

the yin, the yang

[SparafucileMan]

And we have come full circle.

computers: 1
language designers: 0

MS sez.....

[heybo]

Microsoft is committed to MTOM as the definitive solution for including opaque data in XML and SOAP messages, and we plan to implement support for MTOM across our XML-aware product line. -- Don Box, Architect, Microsoft Corporation

Yea right.... It will be MSMTOM and won't work with anything BUT IE and M$ products. Look at what they did with their version of XML.

WBXML (this is really binary XML)

[rewound98]

Read this if you want to know about binary XML (http://www.w3.org/TR/wbxml/).

The XOP proposal is a mechanism to represent and refer to binary data in an XML document.

XOP is not a proposal to compress XML documents.

You might say, oh, I can use CDATA, right.

Unfortunately, no. CDATA cannot be reliably used because the character range for CDATA is loosely 0x9, 0xA, 0xD, and anything above 0x20. (http://www.w3.org/TR/REC-xml/#NT-Char)

Currently, you have to resort to your own scheme to reliably include binary data in an XML document.

Re:Binary... XML... Nah!

[Leo McGarry]

as long as you use simple CSS, you'll get the same result in a lot of browsers

So much for the "standard," huh?

Look, I guess I was unclear or something. We have these massive and baffling standards, standards that overlap, standards that contradict, standards that no mere mortal could possibly understand. Nobody conforms to them. Your solution? "Just use very simple CSS." Um. Bad solution. Bad! Go lay down!

Better solution: Take a moment to wonder if, maybe, it's the standards themselves that suck. If nobody but nobody complies with them, maybe it's not a compliance problem as much as it is a problem at the head end. Ya know?

And no, the differences most certainly are not "extremely small" compared to the old days. In the old days, it was a matter of whole features that either worked or didn't. And guess what? The HTML format was designed to fail gracefully. So if you went to a Web page that used some feature your browser didn't support, you just didn't see that part of the page. Everything else worked.

But now we're fucking with the box model. Now we're fucking with the way things are actually drawn on the page. Not which parts are drawn and which are ignored, but rather how things are drawn. When you give a browser a basic instruction, like "make this box 300 pixels wide," and the browser responds by making the box 302 pixels wide because somebody misinterpreted the baffling, bewildering spec, that's not a "very small problem." That's a massive fuck-up.

Re:Binary... XML... Nah!

[TeknoHog]

Most of the explanations in your link refer to "words". That means they have to be pronounceable, like RADAR or NATO.

Simply taking initials of words is an abbreviation. RADAR and NATO are abbreviations too. Acronyms are a subset of abbreviations, the kind that can be pronounced like words.

There is a reason for two different words, abbreviation and acronym, because they mean different things. If you think they mean the same thing, then let's just ditch one of them to avoid any redundant redundancy.

This in not binary XML

[tungwaiyip]

The poster has mixed up. This is what W3C are doing with binary XML

http://www.w3.org/XML/Binary/

I don't find them publish anything yet.

What the poster link to is a spec to include binary data in XML message, mostly for use in SOAP.