Slashdot Mirror

There is a binary XML format called bxml. If OpenOffice were to use BXML for the internal representation of the data, there is reason to believe that parsing XML documents could be significantly faster. In fact, there are performance comparisons using XML from OpenOffice.org's Writer. It looks like a speed up of 4x to 6x would be possible if bxml were to replace xml. This is still an open format and it can be (trivially) converted back to XML. This would be a natural way to have the portablity of XML with an efficient binary data strructure. And it can be done with open source libraries, hopefully with only small changes to the existing code.

If you want to see open web mapping standards in action (and maps of the whole world) try here or look at some of NASA's data here

Ian

Check out CWXML/BXML. Especially significant though perhaps unintuitive is the savings in compression time from the source data being more compact.

The OGC has been developing ways of sharing and displaying geographic data for some time now. This allows (at its simplest) a user to request an image of a set of vector data (see demo ). If you need a bigger image change the size params. There are many useful WMS available for the US and some for the rest of the world where we don't give data away as freely. It should be well within any slashdotter's ability to write a modfied html form to grab a bigger map. There are also java clients available for free.

To answer questions about where places are check out the alexandria digital library gazeteer again there is a simple html interface or a java client is available.

Finally if you need more maps than are available its pretty easy to down load a server that can be used to publish date using the same protocols (for example the US statistical atlas files).

These ideas will obviously only help users with some sight. How to provide map data to the blind is a much harder problem which geographers are still working on.

Ian, (Computational Geographer)

A navigatable map of the locations of the Mars landers (current and past) can be found here.

Others have said it before, but I'll say it again. XML is heavy weight and isn't free.

XML needs to be updated to allow binary encoding. The open-source high-performance parser/generator library at the link demonstrates the performance gain.

Others have said it before, but I'll say it again. XML is heavy weight and isn't free.

XML needs to be updated to allow binary encoding. The open-source high-performance parser/generator library at the link demonstrates the performance gain.

Both of those are new to me, and seem pretty interesting. Thanks for the links!

Here are some additional rough ideas to make the encoding even more compact. Content & tokens are collapsed for compactness. The byte-oriented macro idea makes the encoding just as compact as schema-based approaches (factoring out fixed structuring) without the complexity of schema processing. It is also designed to facilitate byte-for-byte-identical round trips between XML and BXML encoding.

You could define a macro so that foolishly overstructured data such as the following can be represented with a raw byte array and still be processed generally or with a special reader that recognizes the macro signature and reads only the raw array:

<Scanline>
<Pixel>
<Red>111</Red>
<Green>222</Green>
<Blue>123</Blue>
</Pixel> ...

Define a macro for the fixed parts of <Pixel> with single-byte instance-data references for the color components and then a 1024-pixel scanline can be represented with a macro-reference token byte, macro id byte, count of 1024 (two bytes), and 3*1024 raw bytes of sample values. The macro is replayed 1024 times filling in the instance-data bytes where appropriate. General records work the same way, with fixed and variable-length fields of instance data.

I use XML quite a bit in my data-based programs. But I've seen it used WAY too often and in applications where XML just doesn't make any sense (like parsing 1GB files, for instance). XML is a nice tool, but isn't the fastest way to get to data.

Employ a binary-encoding approach. Use embedded indexes. Get binary performance and XML flexibility. W3C is pondering the subject of binary encoding.

I use XML quite a bit in my data-based programs. But I've seen it used WAY too often and in applications where XML just doesn't make any sense (like parsing 1GB files, for instance). XML is a nice tool, but isn't the fastest way to get to data.

Employ a binary-encoding approach. Use embedded indexes. Get binary performance and XML flexibility. W3C is pondering the subject of binary encoding.

I know that XML is "trendy" but it's the wrong tool here - a binary protocol should be used.

If you want a fast XML parser, check out the open-source cwxml library, plug, plug. It is 3x as fast as libxml2 (10x as fast as Xerces) and itt also supports a binary encoding format for XML that is 5-60x faster again and cures the various limitations of WBXML. The W3C is also investigating binary encoding for XML.

I've already roughed in a framework to support the XL 2003 xml format. However, we've heard the MS Office / XML chant before.

The day that Microsoft makes its file formats truly open will be the day that they apply encryption to all file content so that it can only be decoded or generated by registered software.

It did not see much common usage, the performance hit with non-trivial files was just too large.

They just need to pursue binary encoding of XML. The performance is as good as many binary formats, and much more flexible. W3C is holding a workshop on the subject in a month.

<element x="0" y="0">
<component name="red" value="10" /> ...

Yep, the standard XML-bloat joke. Here is the serious solution: binary encoding. I have done some testing by defining a simple demonstration XML image format as:

<XmlDemoImage version="1.1.0">
<Header>
<Width>x</Width>
<Height>y</Height>
<SampleType>byte</SampleType>
</Header>
<Scanline row="i"> <!-- optional attr: filter="diff" -->
<RgbSamples>r g b r g b ...</RgbSamples>
</Scanline> ...
</XmlDemoImage>

If we can avoid the bizarre and hugely self-defeating but all-too-common urge to way-overstructure the pixel representation and use raw binary encoding especially for the dense arrays of numbers, the representation and performance is essentially equivalent to that of PNG format itself (though for some images, BZIP2 compression is significantly better). Here is a study of the issue. On an Athlon-XP1800+ Linux box, I get a raw (Binary)XML reading speed of 188 MB/sec for an uncompressed image. W3C is holding a workshop on binary XML encoding in September, so it may finally be prepared to humour such radical efficiency with XML.

The company I work for is actually pretty good about this sort of thing. In fact, I've occasionally written code for the product that I wanted to be in the public domain (because I knew that I'd want to re-use it myself later), so I just said "I want this code to be in the public domain". They replied "fine, as long as it's not written on company time". An example of such code is my C HashTable. (Yes, I know this has been written a million times before, but all of the other public domain implementations I've seen suck!)

Of the six senior software developers in our company (CubeWerx, Inc.), five of us use Linux as our primary (and in most cases, only) operating system. The suits still use Microsoft Windows, but that's mostly because they don't know any better. :-)