Slashdot Mirror
Does the World Need Binary XML?
sebFlyte writes "One of XML's founders says 'If I were world dictator, I'd put a kibosh on binary XML' in this interesting look at what can be done to make XML better, faster and stronger."
← Back to Stories (view on slashdot.org)
For starters, keep Microsoft out of it.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
Then what happens, do you base64 the binary xml and wrap it in an ascii xml document?
"Piter, too, is dead."
Use the Z-modem protocol between Information Superhighway routers to compress the plaintext.
Binary XML = zip file.xml > file.xml.zip
Thats all you need. XML compresses great.
On the face of it, compressing XML documents by using a different file format may seem like a reasonable way to address sluggish performance. But the very idea has many people -- including an XML pioneer within Sun -- worried that incompatible versions of XML will result.
I agree with his point.
What's wrong with just compressing the XML as it is with an open and easy-to-implement algorithm like gzip or bzip2?
I don't need no instructions to know how to rock!!!!
Binary XML is nothing new, as I wager that many people here are already using it, albeit unknowingly.
One of the earliest projects that has tried to make a binary XML (as far as I'm aware) was the EBML (Extensible Binary Meta-Language) which is used in the Matroska media container.
Somebody fill me in ...
... its called zipping, most webservers have it as an option to zip the data up as it streams to the client browser
i fail to see the need to have a "binary xml" file format when there are already facilities in place to compress text streams
Programs written in assembly can run faster than programs written in C, but it's easier for someone to open a .c file and figure out what's going on.
I'm sure when C came out, the argument was similar that the performance hit doesn't make up for the readability or cross compatibility. But as computers and network connections became faster, C becomes a more viable alternative.
Text compresses quite well, especially redundant text like the tags. So why not just leave XML alone and compress it at the transportation level with protocols like sending it as a zip, let v.92 modems do it automatically, or whatever. No need to touch XML itself at all.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
But secondly, no, you don't need Binary XML, all you need to do is Gzip it on the wire. It gets as small as Binary XML.
One of the easiest ways to shrink your XML by about 90% is use tags like:instead ofYou can use a transformation to use the short names or long names on the wire.
XML, as implemented today, is often little more than a thin wrapper for huge gobs of proprietary-format data. Thus, any given XML parser can identify the contents as "a huge gob of proprietary data", but can't do a damned thing with it.
Too many developers have "embraced" XML by simply dumping their data into a handful of CDATA blocks. Other programmers don't want to reveal their data structure, and abuse CDATA in the same way. Thus, a perfectly good data format has been bastardized by legions of lazy/overprotective coders.
The slew publications exist for the sole purpose of "clarifying" XML serves as testament to the abuse of XML.
Obliteracy: Words with explosions
Of course binary doesn't equal proprietary. Those are two completely different concepts.
PNG is a binary format. It isn't proprietary, though. And although I can't immediately find a text-based proprietary format, such formats are not impossible (although arguably easier to reverse-engineer than binary proprietary formats).
But if the XML is really such a problem, I suggest the simple solution. Compressing XML with a simple and open algorithm like gzip or bzip2, is the way to go. XML usually compresses very easily.
A huff transform will give you entropy +1 compression. Not suitable for larger data sets (dictionary based compression is even better for this). 7z compression (or is it z7?) will give you a neat storage format.
u itcake
Lets talk about where this verbose talk of verbosity is stemming from:
apple
orange
pineapple
this is a data set. Noone knows what it is.
Here it is again with some pseudo xml style tags
I am listing vegetables here
this is a list of vegetables
vegetables are listed on thier own without any children pr parent tags, there can be one or more of them, this is version 1 of the document
here now follows a vegetable
tomato
that was a vegetable
here now follows a vegetable
leek
that was a vegetable
here now follows a vegetable
potato
that was a vegetable
here now follows a vegetable
haddock
that was a vegetable
as you can see, this is (albeit slightly weird looking) list of items called 'vegetables'.
The beauty of XML is two fold, the description of the document format (DTD and schemas) and the abilty to verify a document is valid, for any specified format.
XML is a human readable file specification language, and file format, all in one, written in itself!
A binary format of XML would be nice, you can make it yourself though.
veg:http://slashdot.org/veg.xml
v:tomato
v:fr
v:lemongrass
v:cat
this is a minimal way to represent the same xml like structure, in a less verbose way.
This is undeniable complexity, a binary format is just like a way of saying introduce a standard loosless compression format for XML, without changing what XML is.
I say anything that gets the W3C stamp of 'this is official' gets my vote. After all, 1 bad standard is better than 11 good proprietary solutions in a world of millions of interconnected systems.
#hostfile 0.0.0.0 primidi.com 0.0.0.0 www.primidi.com 0.0.0.0 radio.weblogs.com
The XML guys are funny. First make a text version of binary protocols to make it easy to sell XML them to the mass of "31137 HTML PRogrammers" who feel comfortable "programming" in dreamweaver; and then make a binary version to make it work.
Of course not! That's not XML!
<file=xmlbinary> <baseencoding=64> <byte bits=8> <bit1>0 </bit><bit2>1 </bit><bit3>1 </bit><bit4>0 </bit><bit5>1 </bit><bit6>0 </bit><bit7>0 </bit><bit8>1 </bit> </byte>
<boredcomment>(Umm, I'm gonna skip a bit if y'all don't mind)</boredcomment>
</baseencoding> </file>
Now it's XML!
FTFA "The goal of the Fast Infoset project is to generate interest among developers and eventually create a standardized binary format." I'm not sure why they think that one has to come before the other.
Because standards written in a vacuum tend to suck. Why wouldn't you want input from developers with different backgrounds and needs, then cherry pick the best ideas (many of which you didn't think of), toss out universally reviled ones, and implement a broad, useable standard?
Here come da fudge!
That's the dumbest statement I've ever heard.
As long as it's standardized, the standard is freely available to anyone who wants it, it does not depend on an external library, and it is unencumbered by any sort of patent, it isn't proprietary.
I hate XML right now because of all the string processing and parsing. Text is a sloppy way of defining something, and it begets lots of big processing libraries. It's OK for big PC memory hog apps, but I can't build a small enough one that is still robust enough to want to integrate it into the work I do (small, compact stuff). I find myself doing other, backwards things, or worse, fracturing XML into useable subsets. It somewhat defeats its utility.
Binary XML sounds like a great idea to me, as long as we're clear on a few things. One, it has to be totally documented in a standard (see above for my definition). Two, the standard must define a tool that can read an XML file and say "Yes this is XML" or "No, this is some [microsoft] non-compliant crap". Three, keep it simple: no compression, no outside library dependencies, no cruft.
If those things cannot be achieved then it will not reach maximum utility and something proprietary will swoop down and take over (*cough* microsoft *cough*).
However, if anything, XML has shown us the power of well-structured information. XML has given the possibility of universal interoperability. Developments in XML-based technologies have led us to the point where we know enough now to create a standard for structured information that will last for several decades.
It's time that we had a new ASCII. That standard should be binary XML.
When I think of the time that has been wasted by every developer in the history of Computer Science, writing and rewriting basic parsing code, I shudder. Binary XML would produce a standard such that an efficient, universal data structure language would allow significant advances in what is technically possible with our data. For example: why is what we put on disk any different from what's in memory? Binary XML could erase this distinction.
A binary XML standard needs to become ubiquitous, so that just as Notepad can open any ASCII file today, SuperNotepad could open any file in existance, or look at any portion of your computer's memory, in an informative, structured manner. What's more, we have the technology to do this now.
However, you wanted to go to a binary encoding you could try for something relatively straight forward like:
original:patented XML encoding algorithm (hexideximal):
Waltz, nymph, for quick jigs vex Bud.
This is all about different companies trying to get THEIR binary format to be the "standard" with XML.
From the article Images are already binary data. They really don't compress much more (if you've chosen the right format). That means that they will take the same amount of time to download, binary XML format or not.
Yeah, right ! XML binary images... So needed...
of "I told you so!" coming over. Between all the people who jumped on the web services bandwagon without any clue how to handle distributed systems efficiently and the "OMG! It's human readable!" crowd, the architecture de jour has become a bloated PITA. Why this wasn't built into the spec in the first place alludes me. If we can use tools like ethereal to read those binary IP datagrams, why wouldn't the same concept be used for this standard? A standardized, compressed, data format with a standardized API for outputting plaintext (XML), would have allowed this system to be much more efficient.
Didn't anyone remember that text processing was bulky and expensive? Sometimes the tech community seems to share the same uncritical mind as people who order get-rich-quick schemes off late night infomercials. I doubt XML would have gotten out of the gate as is, had the community demanded these kinds of features from the get-go.
Arrogance is Confidence which lacks integrity. -- me
I think that's where the true problem lies. HTTP.
.gz files, .zip files etc. since that would be pointless).
We need to look towards http 2.0. What I would want:
- pipelining that works, so that it could be enabled for use on any server that supports http 2.0
- gzip and 7zip support.
- All data is compressed by default (a few excludes such as
- Option to initiate persistant connection (remove the stateless protocol concept), via a http header on connect. This would allow for a whole new level for web applications via SOAP/XML.
There are tons of other things that could be enhanced for today's uses.
HTTP is the problem. Not XML
The problem is that XML is being used for web services which are unlike HTML: the requesting machine will not like waiting 2-3 seconds for the response to the method call. These are interoperating applications, not people downloading text to read, so the response time is much more critical.
I agree that gzip compression is a simple solution to the network problem. It does not address the parsing time problem, and in fact exacerbates it, but in my opinion the network issue is the big one. Time works in favor of faster parsing (faster processors), but works against network issues (more congestion). I would go with compression, test the results, and only then look into a binary solution.
form Re: Lisp syntax, what about resynchronization?
Attributes in XML are inherited from SGML and they were thingking markup for textual documents. When you want to represent data it being attribute or not is completely irrelevant.
Deep explanation: From:The horror that is XML
Dyslexics have more fnu.
Aside from the mistakes pointed out by others, you also forgot to reference the xmlbinary namespace, the xmlbyte namespace, and the xmlboredcommentinparentheses namespace, and to qualify all attributes accordingly. You also didn't include anything in or any magic words like CDATA, and you didn't define any entities. You also failed to supply a DTD and an XSL schema.
This is therefore still not _true_ XML. It simply doesn't have enough inefficiency. Please add crap to it
Whence? Hence. Whither? Thither.
The real problem with XML is that it adds the extra verbosity of the metadata text tag for EACH INSTANCE of a pice of data even in cases where that metadata is identical for row after row of data. In the case of table data, that is really stupid. There should be some sort of XML means to handle a table of values better. A way to say "Column 1 has the following XML properties: name, etc", then "Column 2 has the following XML properties: name, etc".... and then after that section, a way to syntactically list just the values up until the end of the loop.
This is what made us balk at using XML for storing NMR spectroscopy data, even though it is already in a textual form to begin with. The current textual form is whitespace-separated, little short numbers less than 5 digits long, for hundreds of thousands of rows. That isn't really that big in ascii form. But turn it into XML, and a 1 meg ascii file turns into a 150 meg XML file because of the extra repetative tag stuff.
In another bit of irony, we can't find an in-memory representation of the data as a table which is more compact than the ascii file is. The original ascii file is even more compact than a 2-D array in RAM. (because it takes 4 bytes to store an int even when that int is typically just one digit and is only larger on rare occasions.)
Don't label something "offtopic" unless you know the topic well enough to tell what's on topic.
The problem is that not everything in a typical XML message is text, so there can be a lot of translation going on between XML text and the binary format that an application needs (e.g., double). In our tests we've found XML to be 100x - 250x SLOWER than other approaches (e.g., JMS MapMessage). (FWIW, the 100x is using the MS parser, the 250x is with Xerces/Xalan). For high-volume, high-performance apps that's simply intolerable. Note that this has nothing to do with size on the wire, which is another consideration entirely.
It doesn't tell us what the specific performance problems are with XML. Does it take too long to transmit? Does it take too long to validate? Does it take too long to parse? Does it take too long to format? What's the real problem here?
From experience, I can state that using XML in any high performance situation is easy to screw up. But once you get past the basic mistakes at that level, what other inherent problems are there?
Oh, and just stating "well, the format is obviously wasteful" just because it's human readable (one of its primary, most useful, features) is NOT an answer.
I get the feeling that this perception of XML is being perpetuated by vendors who do not really want to open up their data formats. Allowing them to successfully propagate this impression would be a very real step backwards for all IT professionals.
Please mod this post only if you think others should/n't read this. I have enough ego^H^H^Hkarma. Thanks!
Dude! Wake up! How often do you open an XML-RPC packet trace with your morning coffee and think 'Gosh, how cool it's in a readble bloated text format and I don't need to parse it with Ethereal !'
:-/
Seriously, the only time readability is needed is when you edit an XML web page with a notepad. Otherwise it's a brain-dead technology that first got popular among scripting developers, which are notoriously afraid of anything binary, and then it got pushed into the areas where it didn't belong.
Unfortunately, the majority of XML zealots are plain ignorant. Should they took time to learn what the byte ordering and TLV encoding mean, we would've not probably have this XML craze now
Don't get me wrong, XML has its place. But it is next to HTML, and not next to RPC or databases!
3.243F6A8885A308D313
Three ideas, in order of increasing significance and increasing difficulty:
Stop using bad DTDs. There seems to be a DTD style in which you avoid using attributes and instead add a whole lot of tags containing text. Any element with a content type of CDATA should be an attribute on its parent, which improves the readability of documents and lets you use ID/IDREF to automatically check stuff. Once you get rid of the complete cruft, it's not nearly so bad.
Now that everything other than HTML is generally valid XML, it's possible to get rid of a lot of the verbosity of XML, too. A new XML could make all close tags "</", since the name of the element you're closing is predetermined and there's nothing permitted after a slash other than a >. The > could be dropped from empty tags, too. If you know that your DTD will be available and not change during the life of the document, you could use numeric references in open tags to refer to the indexed child element type of the type of the element you're in, and numeric references for the indexed attribute of the element it's on. If you then drop the spaces after close quotes, you've basically removed all of the superfluous size of XML without using a binary format, as well as making string comparisons unnecessary in the parser.
Of course, you could document it as if it were binary. An open tag is indicated with an 0x3C, followed by the index of the element type plus 0x30 (for indices under 0xA). A close tag is (big-endian) 0x3C2F. A non-close tag is an open tag if it ends with an 0x3E and an empty tag if it ends with an 0x2F. Attribute indices are followed with an 0x3D. And so forth.
If you ask me, the transparency of a text stream far outways any cost in performance.
It far outweighs it huh? I guess you have never heard of a large segment of the computing world refered to as embeded systems.
If you can develop a good parser (not that hard), the cost difference is negligable, if any.
This is simply untrue, development of a good parser is easy, but it's added bloat that isn't negligable for many computing devices outside of the PC/Server realm. Not to mention the added network traffic that uncompressed text yeilds (embeded devices don't always have the fastest I/O). Some say that the solution to reducing the network overhead of XML is compression. Compression takes CPU power, another thing lacking in may embeded devices.
My point is that there are actually a lot of applications where XML is just not well suited.
Had data to be delivered to client, dumped from a database. As flat files they were ~20mb in size as flat files. That bloated ~120mb after conversion to XML.
Client attempted to open in a DOM based application which I suspect used recursion to parse the data (easy to code, recursion). Needless to say it brought their server to its knees.
We switched to flat files shortly there after.
In my problem domain, where 20MB is a small data set, XML is useless. XML seems does not scale well at all (though using a SAX parser helps at times).
YMMV.
putting the 'B' in LGBTQ+
It seems to me that the problem isn't with XML, it's with what people are using it for. I read some complaints here from people saying "I tried to use XML for BLAH and it was too slow." However, if they'd thought about it, BLAH would have been better served by some binary format in the first place. The article also discusses the fact that mobile devices need something less cumbersome for transferring pictures/media. Why are they using XML for that at all? One of the benefits of XML is that it's human readable, but in those applications you don't need that benefit, so don't use XML. Instead of coming up with a binary XML standard, come up with a generic binary standard that does exactly what you want. Too many people have been given the hammer of XML and now everything looks like a nail.
You had me until then; no self-respecting engineer would ever use those terms.
Dewey, what part of this looks like authorities should be involved?
A good binary XML specification could be an extremely good fit for us.
And, don't suggest that we just compress XML and send that. Here's why: first we have to expand all that digitized data into some sort ASCII encoding, which is then compressed. End result: no gain and a possible loss of precision in the data.
A real, live, useful binary XML spec could help us immensely. I say BRING IT ON!!!!
BTW, wasn't DIME supposed to address these problems? What happened to DIME, anyway?
In the course of every project, it will become necessary to shoot the scientists and begin production.
take an example on microsoft XML formats. Word, or the MSN messages format... they're _NOT_ xml. They're proprietary formats DISGUISED as XML.
If Microsoft doesn't respect text-only XML, what do you think will happen when^H^H^H^Hif binary XML is out?
to make inaccurate interpretations of the data and not using proper and accurate specifications.
:(.
Many people claim that XML is so great because you can "just read and understand it" without having to use cumbersome and hard to understand specifications. This exactly is what makes XML, indeed, nice for typesetting purposes like HTML, maybe as an alternative for simple configuration files etc, but indeed NOT for RPC and databases as you write. I couldn't agree more.
I have seen so much time and money lost due to intuitive but false interpretations of XML schema's. People think that because its human readable with "meaningful" tagnames that they don't need a proper spec no more. Well I guess it fits in nicely with todays "cut and paste" programmers who don't really know what they're doing
What is text? It's a binary code that a computer translates into graphical glyphs. Is it proprietary? Not any more. Your computer is what turns that binary code into something that means something to you. It doesn't even mean something to everyone (in fact iirc the first line in an xml file identifies a code page for using the intended symbols). So firstly, opaqueness even on "text" is not quite black and white. Second, what is transparent to YOU may be completely opaque to software, I'll elaborate on this later.
So what could binary XML be? It's a binary code that translates into XML syntax. Except it's easier to deal with for software, there's no processing. Let me present this example, which I will endeavor to use over and over. . I could write this in binary as 0001010203, obviously to do that i'd have to store the strings "mytag", "is", "simple" in a string table elsewhere, but this is just a simple example. I made "0x00" mean "a tag", the first "0x01" mean 1 attribute, and the rest are string references. Reading this tag would be very simple, fread(buffer,10,1,file) (i picked the two middle numbers out of the air since we have not really defined this format).
Saying that binary is proprietary makes absolutely no sense. Proprietary means property of an owner (usually a business). A file can't be proprietary. It's contents, the format of it's contents, certainly. But a binary file is atomic, it's like the sky. It just is what it is. Binary XML COULD become proprietary, but it will not NECESSARILY happen. Nothing is inherently proprietary about a binary file. If the binary XML format satisfies the constraints of a standard in my first post, it will absolutely not be proprietary by construction, or so I think. I don't work in standards groups, people more experienced with their goings on may point out additional refinements.
Your next point, recovering data. What do you use to read XML files? A text editor usually. What does that do? It reads a binary file (uh a text file!), applies some understanding of what the ASCII code (as an example) means, and displays it to you. Most of it is usually character data, but not always, there's a bunch of special characters that text editors often respond to for formatting or other things. Unix and PCs can't agree even on how to terminate a line. The point being even right now you can't totally say plain text XML is transparent, magic happens for you to just see it. Nothing about its presentation is defined (nor should be, imho). So what could binary XML be viewed with? Only slightly more overhead. You could "textify" as a preprocessing step to be viewed in a text editor. bin2txt myfile.bin.xml > myfile.txt.xml as an example. Or you could write your xml in plain text, and do the opposite. It's one to one, no loss. XML is just a syntax.
Now as for processing, I'll admit to waving my hands and skipping a few pieces. The XML syntax is defined clearly, there's no ambiguity (that i know of). However the step of choosing text-like strings to declare the syntax elements is where it gets hairy. Your first step in writing a parser is to grab the syntax elements out of their native text string. This is disgusting as compiler writers, language developers, etc. understand. You have to make lexx/yac scripts or workalikes to generate code, or worse, write your own (no one should do this but that's purely my opinion and not defendable). Theres a complicated state machine, some funny thing called LRM, and some other gotchas. All this just to take and break it into it's constituent elements. Usually then you have a tree structure or some hierarchy that a computer can understand.
Take a look at some common XML libraries: xerces, libxml, a few others I can't remember. They're pretty damn big. Mostly, I argue, due to the text nature of their data. A lot of work goes into making text files useable by a program. A lot (but not all) of cruft can be cut by adopting a format that is simpler for softare to understand.
Sure, people who write MS Word (i.e.
DNS is binary; does that make it proprietary? Not at all. It is a published open standard in RFC 883 and later documents. Other examples include ASN.1/BER as used in SNMP. It's not whether it is binary or text that matters; it's whether it is openly documented and unencumbered by intellectual property claims (a separate issue some of XML has).
The decision of binary vs. text for a format should be the result of specific needs. XML is verbose. XML can be compressed for transmission purposes, but it still has to be uncompressed to its verbose form for parsing. If speed in parsing is necessary (it might be as I have noticed quite many XML based progams are rather slow), a binary format can have things like length prefixes and continuation tags, instead of having to detect and verify collection of characters whose position is unknown. A parser that does not recognize a given tag, or does not need to process it, in a binary format can simply skip it by jumping the specified number of bytes. Binary format is very optimal for machine processing.
The usual argument for a text format spans the range of permitting humans to create the content for most things directly in an editor like vi or emacs (no wars here, I listed my favorite last), or reading that content directly, such as to diagnose the real cause of misunderstood errors. XML is too utterly complex for human creation or interpretation to be effective on a direct basis. There may be some argument that it can still be effective for diagnostic purposes (I have in fact needed to do so many times). Given that it is the powerful tools of XML that are used as the basis for the benefit of XML and promoting it, then what does it really matter what format is underneath as long as it is open and unencumbered?.
A binary format for XML will absolutely not kill XML. DNS is obviously not dead (and you'll love it even more when IPv6 rolls into your network). What a binary format might do is weed out some of the weaker programmers who are sticking their fingers a bit too deep into the inner workings of some applications and tools.
now we need to go OSS in diesel cars
The "Fast Infoset Project" for creating Binary XML as mentioned in the article is using ASN.1. See this blog entry by Rick Jelliffe for details.
Fast Infoset is to ASN.1 what XML is to SGML. At least if it becomes the standard anyway.
...but I thought that the strategic goal of XML is to sell more hardware.
We should rejoice, buy more CPUs, and move the problem from XML, to languages with poor concurrency support.
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear