Open Source XML Databases?

tarun asks: "I am creating the next version of my open source UDDI registry and decided to use an XML database backend - if I can find any good ones. The reason to make this choice was that I was impressed by oracle and db2's xml capabilities in my past lives. However, when I tried looking for an open-source alternative it seems there is nothing around except perhaps xindice -which clearly is less then perfect. I am looking for something that can work with more than one existing databases (I will ship my software with MySQL but if a large organization wants to deploy it, it should be able to do it using Oracle, DB2 or whatever they want to use) and xindice currently only works with Berkley-DB. Also, I am looking for something that can create database tables for me given an XML schema (I can tweak it later to create indexes, stored procedures etc) and given an XML document - write it to these tables. If it supports something standard like Xupdate or XQuery, that is even better."

"There are some other XML Database projects but either they have too few features or are not open-source. What is the XML-aware portion of the Slashdot community using? Have you ever run across such problems? Do you guys create your database schemas by pain-stackingly copying every element in every XML schema you have to handle to database tables and write huge amount of parsing/deparsing code both ways?"

Comments

Use an XMLJavaDB toolkit

[jefflinwood]

Try using an XML to Java toolkit with a Java object-relational mapper. XML Databases aren't really all they are cracked up to be. Really quickly, projects you should look at are:

• Castor - http://castor.exolab.org/
• Hibernate - http://sourceforge.net/projects/hibernate/
• Torque - http://jakarta.apache.org/turbine/torque/
• ObjectRelationalBridge - http://jakarta.apache.org/ojb/

Also you might want to check out Brett Mclaughlin's O'Reilly book on Java XML binding's, or the Wrox Professional Java XML book.

Why XML?

[jacoberrol]

Could someone explain to me the benefits of an XML database? I can't think of an XML document that can't be expressed as a relation. I've seen vague references to performance advantages and ease of development for certain applications, but I haven't heard a convincing argument. Am I missing something, or is this just XML hype?

Re:Why XML?

[DevilM]

Sure! To use XML with an RDBMS you have to do one of two things. First, map the XML to a relational schema. It is well understood that doing this has two main problems. The first problem is the resulting schema. To create a schema to support heirarchical data results in a complex and ugly schema. The second problem is a spatial one. To retrive a given XML document, the database must pull data from a variety of pages. This results in poor performance as the database has no context to store the different bits of an XML document together on the disk. FYI, this one of the main reasons for the creation of OO databases.

The second way of handling XML in an RDBMS is to store the document as a CLOB. Storing it as a CLOB has the advantage of solving the two above issues, but introduces one of its own; You can't query the data that is represented by the CLOB because it is all stored in a single column. This means you have to extract the document from the CLOB and parse it before being able to use any of the data. Some databases now have built in XML parsers so you can do this from stored procedures and combine the XML document with tabular data, but the performance sucks.

I do cover why you would want to use an XML database and how to use Xindice in an article I wrote for DevX that can be found here.

Re:Why XML?

[Ayende Rahien]

What about using an hiercial DB? They *are* the most efficent kind, in all terms.

Re:Why XML?

[pthisis]

Am I missing something, or is this just XML hype?

Basically, yes. "XML database" doesn't mean much about the database itself, unless you mean that the file format used to store the data is XML (which is pretty much uninteresting, except for being fairly braindead for many sorts of data sets). It tells you nothing about how, logically, the data is organized and what operations it supports (which is what "object database", "relational database", "hierarchical database", etc. attempt to convey), which is generally what a programmer using the database is _most_ interested in.

It may mean that data is presented in XML at query time and XML queries are accepted; if so, that's a moderately more interesting claim but really speaks to a database interface (a la JDBC or pydb) rather than anything interesting about the database itself. Which is not to be dismissed, but formatting results as XML is trivial compared to having to implement e.g. relations in code (for instance), and that sort of interface can be added to any kind of underlying database.

It doesn't really speak at all about the on-disk storage structures (even if data is "stored as XML"), which is often the most interesting thing from a performance standpoint and often interesting from a usability standpoint (e.g. "can efficiently store data in the existing native filesystem" is often mandatory for non-dedicated applications).

Sumner

Umm..no

[quinto2000]

Do you guys create your database schemas by pain-stackingly copying every element in every XML schema you have to handle to database tables and write huge amount of parsing/deparsing code both ways?"

I use Dia, Agata, and Dia2SQL. (There are several variants of Dia2SQL). What exactly are the benefits of XML? I'm new to databases, but this way seems to me more efficient. At what level of complexity does the XML schema become so useful?

What's wrong with Xindice?

[DevilM]

I am curious what the poster thinks is wrong with Xindice. He seems to indicate that he is looking for an XML to RDBMS mapping engine, not a native XML database. Further, he seems to think that Xindice is limited to one type of store. This is simply not true.

In Xindice, XML documents are stored in collections that have filers associated with them that do the actually storage. Xindice provides a Filer interface as well as several different implementations both in-memory and persistent. Additionally, it would be quite trival to implement a DBFiler that stored the data in an RDBMS.

So again I ask, what is wrong with Xindice?

Re:Right here, baby

[tarun]

From eXist homepage,

XPath support XPath support is still preliminary. Some functions and numerical operators are missing and only abbreviated XPath syntax is supported. The parser has also some problems to recognize the full range of unicode characters. I have started to write a new XPath parser (using JavaCC instead of ANTLR) to overcome these limitations.

XUpdate The basic model has been designed to provide efficient, index based retrieval. As a drawback, eXist does currently not support direct manipulations of the DOM tree like node insertions or removals. A document always has to be deleted or updated as a whole.

This is clearly a major restriction for applications which need to directly manipulate the DOM tree. Such applications have to create a new document (as XSLT does) and insert this into the DB after all transformations are done. Documents should be kept small to easily reinsert them whenever they change.

DOM manipulation methods and XUpdate are planned for one of the next releases

From the sound of it...

[broody]

Your looking in the wrong place.

It sounds like what your looking for is really an object-relational wrapper rather than an XML based RDBMS. Take a look at OJB and see if it is not really what you are seeking.