Slashdot Mirror
Do XML-based Databases Live Up to the Hype?
douthitb asks: "I have recently started work as a contractor with a company developing/improving an application for exchanging large amounts of data. The current solution exchanges data via XML, but the data itself is stored in a SQL Server database. There is a concern about the overhead involved with wrapping and unwrapping the XML to get the data in and out of a relational database. The proposed solution is to use Tamino, an XML-based database. Neither I nor any of the other developers have any experience with Tamino, but the desired result is to remove the bottleneck of converting the XML back and forth. Does anyone have experience using Tamino (or any other XML-based database)? What benefits and/or difficulties did you have in using an XML database, as opposed to its relational counterpart? How large of a learning curve should be expected with a product like this? Do XML databases really live up to the hype? A similar topic was discussed on Slashdot way back when, so I was hoping to get some more up-to-date feedback on the subject."
"Sales reps from Software AG, the makers of Tamino, were brought in to discuss the benefits of their product with us. They, of course, presented Tamino as the end all, cure all database system (it will even clear your acne and make you popular with the girls!). The management of the company I'm contracting with were basically eating out of the sales reps' hands, without asking any of the "tough" questions about what the product can do; I was less convinced. Doing some initial searching on the Internet, I have had trouble finding much information about Tamino outside of the Software AG website."
The thing the XML databases are nice for is if folks can't really lock down the schema. Often you have the case where you are mapping attributes to columns, which works fine in a relational database. Then things change over time.... Usually turning a nice relational design into a mess. Being able to use Xpath is great when you are searching for nodes too, once you get your arms around the syntax and assuming the stuff you are storing is XML. Some of the other bits in their toolkit were interesting.
If things are fixed, there are a lot of other options out there for faster manipulation. XMLBeans (now an Apache project, formally BEA) is good stuff. Hibernate is lovely kit for mapping objects to a relational DB.
+++ UGUCAUCGUAUUUCU
it runs in tomcat or similar. it's really crashy. we can't wait to get rid of it.
I haven't tried it, but the regular Berkeley DB is highly regarded, and both are open source and (depending on your situation) free, so it is definitely worth a look.
Berkeley DB XML 2.0
Oracle and XSQL/XSLT works fine for the database we use at work. The overhead of wrapping and unwrapping the data doesn't seem to be any problem.
The first question to answer is, why is this data in a relational database to begin with? More to the point, is this application the only one that accesses the data, or are there other, non-XML centric databases that make use of the same data? The relational model gives you flexibility that XML does not for dealing with the data in arbitrary and unforeen ways (XML can be quite flexible with XSLT, but a programmer must still intervene for each and every new way you want to use tha data, with a much bigger performance hit). The normalised relational database stores your data in a mathematically sound way that puts the priority on integrity of data independently from its past, present or future structure; XML preserves data structure based on its present use while leaving the door open to moving from that to any arbitrary future use... which of the two ideals is more attractive depends on the nature of the data and how many applications need to use it.
Relational databases with good XML support (my background is DB2 but most major databases should be able to do this) reach a good compromise by giving you acces to normalised relational data as XML (which you can compliment with XSLT it if that's what needs to be done), while preserving it internally reduced to its bare essence as data (according to relational calculus' idea of what constitutes the bare essence of data, anyway.)
On the other hand, for single-app applications, or data that is more file oriented than datum-oriented (databases of XML documents where the document rarely or never needs to be abstracted from the data it contains), XML databases offer simplicity and efficiency by removing the need to work out a relational data model. Why break up your structured documents into a DBA's hand-tuned data model when 99.9% of your queries will just build these data sets back into XML documents (even when DB2, Oracle, and I assume SQL Server can automate this last task)? An XML database can give you more flexibility in querying than an all-XSLT solution, while saving a lot of unnecessary work over an SQL-to-XML solution for what is really an XML-to-XML application.
As I see it, that's the big picture. The actual decision has to come down to your applications. An XML database will be less efficient for non-XML applictions, plain and simple. Querying XML cannot be made as fast as querying relational tables, meaning extra overhead for non-XML apps. But *your* application encurs overhead in turning relational tables into XML (probably via the RDBMS's internal facility), and in transforming it if necessary. The question is therefore: who makes more queries on the database, this application or other non-XML ones? Who will make more queries in 5 years?
If you answer 'others' to either question, use a relational database--their XML support is decent now and will only get better, and they're far more popular in business which is an important CYA factor. If you answer 'your app' or 'other XML-based apps' for both questions, it's time to check out what XML databases have to offer right now. I expect other posts to comment on the current state of the art right now, but you can expect things to only get better as industry support for XQuery et al. improves--but don't expect them to *ever* pass up the relational databases in terms of raw performance, it's impossible. But as the evolution from Assembler to C to Java has shown in programming languages, the day may come when raw performance takes a back seat to other concerns.
There is a concern about the overhead involved with wrapping and unwrapping the XML to get the data in and out of a relational database.
So, can you explain how an XML database will fix this?
Your database still needs to translate the verbose, human readable XML into an internal storage representation. If you're transfering the data between two SQL databases now, then I can't see why it should matter if you're parsing XML and putting into a "traditional" row-column RDBMS or parsing XML and putting into a datastructure more suited for storing XML data. The parsing is going to take exactly the same amount of time.
The XML database would help if you've mapped your data representation to XML, and are having a difficult time persisting it to SQL. For some data representations, going from XML to parsed binary RDBMS representation back XML may be difficult, and it may be easier to just go from XML to parsed binary representation of XML back to XML again. But either way, you're doing the parsing.
You're solving the wrong damned problem.
In an RDBMS XML is just treated as a big "blob" of data- Although translating the raw binary data to XML theoretically has some tiny overhead, in practice large data objects take time to transfer over a network and that far outweighs the tiny conversion time the client needs to do to convert the data into an XML document.
XML databases are possibly useful if you think about them as: an elaborate bucket for storing non-normalized data via an XML interface.
If your current relational database schema is either 1) small flat files or 2) a few big tables with most/all of the data stored in "blob" columns: i.e. blobs, clobs, byte arrays, or big varchars. You might be a candidate for an XML database. I'd get two experienced DBA's to agree there was no realistic way to normalize the data, first, but that's me.
If you actually need a database (as opposed to a few files, XML or flat) and your data can be normalized (it almost always can), then a relational database will tend to provide important advantages in three areas: unforseen query handling (OLAP, data mining, etc.), scalable performance, and availability of people with the skills to maintain it.
As for the tradeoff of converting to XML, a number of the commercial RDBMS's allow you to obtain query results as XML. Though I don't know for certain how they handle inserts and updates, I suspect that there are XML equivalents for those as well. However, even if you have to completely roll your own conversion from SQL to XML, that cost is minimal against the cost of accessing the disk to fulfill the query, which both RDBMS and XMLDBMS will have to do.
In general, after working with a commercial XML database and attempting to work with another XML database written in house, I'm categorically unimpressed. I think that a lot of engineers have discounted the relational programming model without first understanding it. In my opinion, people familiar with functional and object programming models would do well to learn about relational programming with an eye to determining the appropriate model for different kinds of problems.
Regards,
Ross
Ick. I suppose you could do it that way if you want to. Maybe its just me, but I like to keep data in relational DBs and keep the XML stuff for when I need to provide a way of sending information to outside people who will not have direct access to the DB. Most of the time the DB is being accessed, it is for internal applications which can access the tables via accessor methods. Now I suppose you could just write accessor methods against the XML DB..... Relational DBs for storage, XML as a transmission format. But the types of things I tend to build are quite small, so YMMV.
XML is kinda nice for some things, and really rotten for some things. Please do yourself a favor and sit down and try to decide what problem you are trying to solve. XML really stinks when it comes to sets: something that SQL based databses excel at.
I think that with the XML fetish we have these days, that we are reverting to the preSQL days of CODASYL or IMS (pre 1980s for those of you young'uns).
What benefits and/or difficulties did you have in using an XML database, as opposed to its relational counterpart?
.NET) but then you have to realise that you are gambling with your data that may be rendered inaccessible in few years (and that is if you are lucky and don't lose its consistency before) and those unfortunate enough to inherit the responsibility of maintenance of your system will curse you to no end wishing you were dead, and not without a reason. You can be fancy with your applications and front-ends, but RDBMSs are probably the most mature computer systems known to man. Ignoring it is foolish, to say the very least. You may say: but my application will always be the only front-end to that data and it will always be an optimal way to work with it! To which I say: Kids these days!
Benefits: XML is new and trendy.
Difficulties: Ignorance of the decades of scientific research and engineering experience in the field of relational database management systems, relational algebra, set theory and predicate calculus; lack of real atomicity of transactions, lack of guaranteed consistency of data, lack of isolated operations, lack of real durability in the ACID sense, and in short, the lack of relational model; scalability, portability, SQL standard, access to your data after two years and after twenty years; to name just a few.
How large of a learning curve should be expected with a product like this?
Certainly smaller than a real, relational database.
Do XML databases really live up to the hype?
No.
I believe that you are confusing an RDBMS with an object store. You should read this excellent comment posted almost three years ago by Frater 219. I understand that you may be inexperienced but you should not be ignorant. Literally decades of scientific research has been put into relational database management systems. Of course you are perfectly free to forget about computer science, jump on the bandwagon and choose whatever buzzword is trendy these days (yesterday it was OOP, today it is XML, tomorrow it will be
Sincerely,
Pan Tarhei Hosé, PhD.
"Homo sum et cogito ergo odi profanum vulgus et libido."
A few years back, I was brought in to a small company to build their new software on top of the Tamino DB. XML was "the way of the future" and we were asked to use it as much as we could. Software AG promised that everything would be easy to program and that their software functioned perfectly. Software AG's sales rep used the fact that Tamino was used in production by (insert major national company here) as a major selling argument. I later found out from a friend working there that they had only evaluated Tamino, found it useless, and never used it in production.
...) We couldn't even get a reliable unique key from the database. The id we did get "could change" if we were to backup and restore the database. Tamino also scaled very badly with simple queries taking up to a minute on the fastest PC we could buy.
Well, we did finish the software on time, but it was a complete nightmare. Software AG hardly gave us any straight answers (even though they charged big $ for customer support).
Tamino itself was missing a lot of features and seemed designed as a system for storing documents, totally lacking traditional database qualities (uniqueness, reliability, scalability,
Needless to say, the software was thrown away and rebuilt with a reliable SQL database.
I would strongly discourage anyone from bilding an application on top of an xml database, especially Tamino. If you really want to build your application on top of an xml database, I would seriously ask myself why and what difference it would make. Also, if you really need an xml interface, choose an ordinary sql db that has a xml plugin.
XML is a file format. Repeat after me. A *text file format*.
It is not a database, nor a data model, nor should it have anything to do with data storage and manipulation. You can store XML documents *in* a database (just like you can store dates, IP addresses, or JPG data). You can index and join on XPath components of an XML file. And you get XML documents *from* a database. But the database itself has little to do with XML. A well-designed XML database is just a well-designed relational database, and XML is just another data type.
People are now reverse-engineering a hierarchic data model from XML text files. But the hierarchic data model is less general than the relational model, and in fact was used and rejected *40 years ago* as not being general or powerful enough. Funny how history repeats itself.
Example: for simplicity, the relational model specifies that ALL data must be stored explicitly in the database. For instance if you have three rows of data, you can't assume any particular order unless the order can be calculated from the contents of each row. But XML nodes have implicit order, which means even the simplest XML document mixes data with metadata. Even a simple query requires dealing with both.
I recommend anyone who has ever uttered the term "XML database" with straight face to go back and learn some basic relational principles. I think you will agree that all data models are either 1) flawed and incomplete; or 2) reduce to the relational model.
In CS we don't have a lot of formal models to guide us, as in engineering or other science. Much of CS is entirely ad-hoc. However we do have a sound and complete model for data storage (relational model) and hardly anyone uses it. It boggles my mind. Do people not *want* their programs to work predictably?
... is that XML is only half of the solution.
For an XML database to really shine, it needs to be integrated with with a TCP/IP filesystem. Once the physical data is stored using TCP/IP (as opposed to FAT or NTFS), the XML database really begins to take off because the data is already in a network format.
I swear to god there was a Dilbert on this...
help me i've cloned myself and can't remember which one I am
How's about simply using a database dump? What's the point of introducing an XML parser when you already have an SQL parser at the ready?
The current solution exchanges data via XML, but the data itself is stored in a SQL Server database. There is a concern about the overhead involved with wrapping and unwrapping the XML to get the data in and out of a relational database.
Premature optimization is the root of all evil.
You say "you're concerned". That means you don't know.
Why don't you find out?
If you have a schema and some of your major transactions speced out, then do some performance testing and see where your bottleneck may be. For Gods sake, don't guess. Know. Find out.
I will bet that you will find that one solution isn't overly performant than the other, and the benefits of a good RDBMS will make up for any delta in performance from a specialized DB.
This is just me, but I tend to write my own database programs to suit my own purposes. It's really not that extraordinary; track entries with overhead, define column-separation string (or put column overhead at the start of the entry), a few other boring details and you can have a whole table in a simple text file.
Now I'm not an XML expert so this comes with a grain of salt, but I personally don't like the human-readable format because it's really not that hard to get, or code a program that'll read a normal database out for you.
XML just happened to annoy me because of the overhead. Normally, you can track a file up to 16,777,216 bytes with three bytes of overhead per entry, and store data types (char string int) with a single byte. XML writes out "2" just to store a single number, not to mention the preceding spaces. This certainly doesn't do much for speed and file sizes.
So this method will just use up a lot more CPU and bandwidth than necessary, even with the tricks servers may use to encode and decode entries. I tend to just stick with the other formats.
I am NOT a number! I am a - oh wait, I'm number 761710. Look! 761710!
In .NET you go to the Server explorer and choose your database and table from a tree control and drag the tables directly to a canvas to design a dataset. You can then define releations. Then, you drag a data adapter to a web page or windows form, type the sql and visually associate it with the data set. Its done all visually except for the sql and even that can be generated.
I currently work with Oracle, and all of communication between server to server and server to client is done in XML. The server XML and networking code is written in C and Tcl. Converting the requested data into XML is not really the slow part. Where we see the slow part is in pushing very verbose data across a network. Tamino will have this same problem.
If you are not happy with SQL Server for whatever reason, choose something else like Oracle, DB2, or friggin MySQL! They can all do XML. I know that Oracle and SQL server both have a lot of built-in XML support.
Great ideas often receive violent opposition from mediocre minds. - Albert Einstein
In case anybody is interested, here are some suggestions for making a more relational-friendly alternative to XML, Here is a wiki topic.
Another potential problem is that existing RDBMS tend to be strong-typed. However, "dynamic relational" is not out of the question. Just because current RDBMS are strong-typed and have "static schemas" does not mean that is the only way to do it. There is a distinction between limits of implementations and limits of relational theory.
Table-ized A.I.
There seem to be a lot of confused comments on this, but hey, it's slashdot :-)
:-)
:-)
If you mostly deal with the sort of data for which relational databases are generally optimised, you'll probably not be very interested in XML solutions, as they are solving problems you don't have.
If you routinely get questions like "how often is part 1976 mentioned in the same repair procedure as part 2001?" or "which of our 150,000 documents have chapters containing five or more subsections any of which does not yet have a summary?" then the XML approach becomes more interesting.
In my book on XML databases (1999 so I don't recommend going out and getting a copy today) I talked about using a hybrid system, with metadata picked out of XML whenever a changed version is stored (e.g. you might use a CVS commit script) and stored in a relational database.
With a relational database you have a lot of flexibility to change your queries but the data representation has to be static. Even changing the type of a column can be difficult in an RDBMS.
Queries may be a little harder with the XML system, but the data storage is more flexible and you have native knowledge of sequence and hierarchy that are traditionally absent using SQL.
More recent versions of SQL have added some XML support, understanding the different sorts of queries that people typically run against such very different sors of data. There has been a lot of research over the past 30 or 40 years (hierarchical databases predate the relational model) on hierarchy, sequence and thesort of irregularity that RDBMS people call semistructured data and the rest of us call XML
XML Query is a query language designed to run over both relational and XML-native data sources (and others, for that matter) and to be optimized very efficiently, so that people like IBM (makers of DB2), Oracle, BEA, Software AG and othes can have efficient implementations. There's also standards work on how to embed XML Query expressions in SQL.
The public XML Query Web page is at www.w3.org/XML/Query and lists quite a large number of implementations. Software AG have participated in the XML Query development.
You might like to look at the XML Query use case document and see how close the examples map to your own situation.
Disclaimer: I work for the W3C, participate in the XML Query WOrking Group, and maintain the XML Query Web page. But it sounded like it's the sort of information you were looking for.
I can't comment on the quality of Tamino, as I have not used it, but I will also note that if you stick to openly-defined standard query languages wherever you can, there's a good chance you could move to a different implementation if you needed to with relatively little cost. This is similar to SQL, of course.
There was lots of hype around XML, but that doesn't mean it's all false, nor that it was all true. XML is a good way to interchange structured, hierarchical imformation, but it probably won't cure acne
Liam
[slashdot::Ankh -- Liam Quin, W3CXML Activity Lead]
Live barefoot!
free engravings/woodcuts
dude, stay with sql.
I like relational, but as a relational language, SQL sucks. It is the COBOL of relational query languages. Let's move on and find a better relational language. Maybe if SQL didn't suck so much, then people would not be playing with so much XML. SQL gives people a legit reason to complain about existing relational systems. Let's remove that wart.
I sat in on one over a year ago. Thank heaven it was all technical people.
We ruled it out because of expense, IIRC. Looked like a really nice product, if you had big bucks to spend on a document management system (which is what we were after). I did not get the impression that it was any sort of replacement for a proper RDBMS--speed was acheived, from what I remember, by storing the data hierarchically. And not abstracting any relational features on top of that.
There was another product out of Canada called TextML that we looked at. Significantly less expensive, but at least as limited. And the company or product one had some sort of beef with certain portions of the XQuery rec that was in place at the time, so they refused to implement parts of it until it was done their way (ISTR it was full-text searching, but am not sure). I distinctly recall at least one very irritated reply from one of their folks about how they weren't going to implement it because it was a bad idea or something (time obscures details, sadly), which prompted some digging, from which it emerged that they had submitted an alternative proposal.
Anyway, they looked like nice solutions to storing and accessing XML documents, but a terrible solution for storing or retrieving data in XML, because they were both significantly geared toward text searching and CMS-type stuff. In the end, we used neither. Our need was not so pressing, and our budget was.
Canthros
I almost suggested adding
if ( 0 ) { printf("enough"); }
But then I realized I was on the wrong verse. Can't think of anything for "I set it up".
William of Ockham had no beard. The most likely explanation is that it was chewed off by squirrels every morning.
There are two possible reasons you're using XML to transport your data from one database to the other.
The first is that you just heard XML is a great way to transport data, and decided to use it.
The second is that you're using the XML for more than just transporting data from one database to another; you're using it at some point with your application.
In either case, the bottom line is that XML is not good for you. If your data fits in a relational database, you should USE RELATIONAL MEANS TO ACCESS YOUR DATA. Don't use that nifty new XML reader to access your data. It's not nearly as fast or flexible as basic SQL; it's actually much more trouble than it's worth.
If you're just transporting data from one relational database server to another, use a flat file, or better yet raw SQL dumps. If you're accessing the data with an application, use SQL or the underlying API.
The only reason you *ever* need to use an XML database is when your data doesn't fit into a standard relational schema. In fact, if you try to fit standard data into an XML database, you're much more likely to end up with a ton of overhead, both in storage and speed.
Fortunately, non-relational data is extremely rare. So rare, in fact, that I've yet to see a non-contrived-proof-of-concept "real life" example.
"Times have not become more violent. They have just become more televised."
-Marilyn Manson
I have used eXist XML:DB.
It supports XPath and XQuery, give it a try:
http://www.exist-db.org/
I have been playing for some time with the idea of "best of both worlds". With some applications, especially document oriented applications, there are benefits to working in XML directly. One of the largest drawbacks is search and query. I have played with the idea of storing most of the data in XML and using an RDBMS to index it. Often we can meet 90% (arbitrary) of searches with just a few well defined fields. I am exploring running each file through a SAX parser or even simpler string regular expression scripts on the way to the file system and extracting a few key fields to a RDBMS. Then execute my quries against the a combination of the RDBMS and the file system. Anyways, in direct answer to your question, no I have not used that software.
When dealing with XML, you need a DTD that defines the data contained in the XML expression in order to parse the string into meaningful data structures (right?)
When an "XML database" is changed, is the data prior to the change left in its old XML format pointing to the original DTD, or does it require conversion of all existing data? How can the data be accessed while that conversion is going on?
How would the method of implementing a schema change be communicated to other places which have already archived copies of an old XML data entity? DTD only defines current state information - it doesn't communicate "If XYZ = 1 in DTD.v1 then set XYZ2 to "A" and set new field ABC to "foo" for DTD.v2". Each iteration of change would become increasingly more complex unless the data is converted.
This is not to say that the same issues don't exist with SQL or relational databases - but just abstracting the organization of the data doesn't mean that your problems are solved.
Lately, I've been using mySQL - and the developers have some curious ideas about the "real world". Even the most trivial changes to the database schema require mySQL to copy and rebuild the entire table... like adding a new index or adding a new field at the end of the table. When tables start having millions of rows, that means this becomes a much less attractive product.
The rationale for doing things this way had two reasons - first, it was the easiest way to implement schema changes. Second, "People should never be changing data schemas in a production environment".
Oh, really? When did we regress to the idea that databases can go down overnight in order to back them up and to implement schema changes?
Final 2006 "Proof of Global Warming" US Hurricane Count -> 0
The SQL DB doesn't store its data internally as "SQL". It's stored internally in some proprietary binary format. Which is optimized for the peculiar performance profile of that RDBMS. Relational DBs use different algorithms for working with their data, and the data is stored with either redundancy or precomputed values, depending on the unique algorithms. From which they derive their higher performance. SQL is just a high-level (more "human") language interface between programmers and the DB engine. Which was specified in such a way that it's not interchangeable across different DBs, partially because it does not specify a schema description which can be packaged with the data to be decoded with the context of that schema.
XML is designed to package schema info with the data exchanged between DB instances. It's higher level, more verbose, and not optimized for data processing (except for the import/export). So you'd better be absolutely certain that your overall system performance is bottlenecked by your interchange processing performance, more than it will be bottlenecked by the "XML-native" DB processing XML data, which isn't optimized for performance.
--
make install -not war
When words like caching, views and snapshots don't show up in your speech.
I worked on the http://www.ananova.com/ website, which was originally built on Tamino. Tamino couldn't handle the load and was a nightmare to admin at the time. Doubtless SoftwareAG will have fixed the lack of backup and restore tools by now. Not soon enough for us to migrate the whole thing onto Oracle shortly after release though.
re-invent wheels
Actually, I'm using exist-db.org and it works fine but I have some performance problems when I want to sort data.
I have tested Ipedo, TextML, dbXML, XHive,
and TextML was the faster but it doesn't support XQuery.
Ipedo was the faster with an old XQuery version support. I think it's the best product because it provides an RDBMS bridge to query with SQL and XQuery and some other features like XViews.
My application use XQuery/XSLT to read data and JAXB to check and execute business method before storing.
I think the main problems with XML Native database is performance and no transaction support but document locking.
But, the advantages are:
- Powerful query language XQuery
- No code to modify data directy in the database.
- Database export as XML documents, modify with "notepad" and re-import.
- Easy and quick test data editing. Don't need to use SQL and Java to insert the test data.
- Easy database deployment without DBA !!!
- Power(full) Full Text searching
I think XML Native databases are to use when your application needs to manipulate a lot of text data.Like CRM, Groupware, Administrative application, fulltext and contextual searching,
You must try at least one XML Native Database in your life to compare it with RDBMS and Object databases and make your own opinions.
I've done some research for myself on xml databases. My conclusion is that it does not scale very well.
Once your xml files get to be Megabytes the whole xpath tree query 'concept'doesn't work anymore. My best bet would be the xmldb from sleepycat (berkeley db).
Because xml is tree structured it's very complex to efficiently work with. I would suggest using an RDF 'triplestore' DB. In RDF everything is a triple (a flat format). This means it's easier to manipulate, because your db is nothing more then a list of triples.
Also while RDF is not a pure XML technology, you can serialize it to XML. It's a W3C recommendation, and there's plenty of software/frameworks around that do work and perform and are stable (I use redland myself http://librdf.org).
You say you are concerned about the overhead of wrapping and unwrapping XML, so you are considering using a database that keeps everything in XML all the time. I think you are trying to solve the wrong problem.
Have you timed the job of wrapping/unwrapping XML? My guess is that on modern hardware, that task is trivial. Bandwidth is a more common bottleneck for XML data transfers, and that problem is usually mitigated by compressing the XML before transfer. But I never heard anyone complain about a CPU taking too much time to extract the data from XML.
If your application queries the data selectively, you will probably find that the difference in query-processing time, between a traditional SQL database and a native XML database, more then makes up for any difference in format-conversion time.
Let your database use its own, efficient, optimized internal data formats. XML is much more suitable for data transfer than for data manipulation.
Why don't you just use files and a http/ftp/webdav or whatever server?
Don't make things more complicated than you have to!
I think xml databases, or xml extensions to relational database, have their uses but they are (off course) not the ultimate answer to all our problems. In specific situations they certainly live up to the hype in others no, but that's also the case for relational databases.
I think it is vital to make a difference between the actual storage and the way to access it, the 'interface'. With Oracle (since 9iR2) one can store the xml as a clob, as an XMLType and object types that are based on your xml complex types but also (traditional) in a relational model. To access data one can use different strategies: the more or less traditional way using SQLXML (maybe as xml view) to present the relational data as xml or using XPath or XQuery. It is also possible to present your xml data in a relational manner. All the options have their pros and cons. But I want to stress it again: there is not a single answer, it depends on the situation. If only for storage it may be a very viable options to store the xml as one big clob. For extensive searching it may be better to use the xsd based XMLType. But for retrieving relational data the use of an xml view may be the better solution.
I my current project We use Oracle (9iR2) for centrally storing our xsd's and xslt's. Our applications can retrieve them via standard http calls and with a webdav client like XML Spy we can easily maintain them. Heck, in this case we are even not interested how they are actually stored.
Juat as background - I work in Application Integration and spend much of my time dealing with moving data to and from RDBMS systems (Oracle, SQL Server, etc) and various external formats (EDI, flat file, XML, etc.)
It sounds like you are in danger of changing the original data store (the relational database) in order to preserve a data transfer mechanism (XML). This is probably a bad idea.
Why is the data in the database to begin with? Is it the database for some other business application? Probably - be carefull not to adversly impact that other application by changing it's database.
Why is the data being moved as XML? Do you have control of both ends of the transfer or does some outside entity control the data format?
Be carefull not to solve the wrong problem. Is the _primary_ business requirement to move data between two business systems, or is it to store and transfer XML data? It almost always is the former, even with an externally driven need to use XML to transfer data. In which case: leave the database alone and deal with the problems integrating the data tranfer (XML or otherwise) with the database.
And the parent post is right - XML is a bit of a fetish right now. This leads to it getting mis-applied to a lot of problems.
sql server 2005 supports xml documents as an internal data type that can be selected, inserted, updated, deleted, and indexed. The indexes on xm include all the nodes / attributes of the document, in order. When you query attributes / nodes out of the xml document, they are returned in the order which they occured in the document.
Its really VERY fast. It handles xml documents as fast as any other standard data type such as varchar , int etc.
Whatever else XML is, it's not a good representation of data for queries and transformation. Different pieces of data don't have a fixed parent-child or one-to-many relationship in real life tasks and so shouldn't be stored as a tree.
Store your data in a well-researched normalized, relational form and format your query results as XML if you like. How is the progress on binary XML to avoid killing your network and CPU on your thin client?
For instance, eXist DB (http://exist-db.org) serializes XML data as binary DOM tree. So, it doesn't need to parse text XML data all the time. One more thing is Cocoon (cocoon.apache.org) which converts XML to a pipe of SAX event processors.
. The current solution exchanges data via XML, but the data itself is stored in a SQL Server database. There is a concern about the overhead involved with wrapping and unwrapping the XML to get the data in and out of a relational database.
/etc/fstab)
1: you can always put the XML in a text field if you want to.
2: I would be far more concerned about using raw XML, it's not RIFF so you can't goto a single point without parsing the whole file, it's not indexed, so you have to search the whole file for the data you need. etc....
3: If the data was in XML to start with you would probably need to use XSL to transform the data, which because of 2 is far slower than just wrapping the rational database data into XML.
Basically you want a back-end that's good for searching and retrieval (e.g. a rational database) that can export data in a portable format e.g. XML.
A friend of mine used to have to process tones of GIS data sent in XML format, the processing the XML format was the slow bit, everything else (like dishing up datasets based on the data) was nice and fast. The company held several TB's of data, so I would say it represents the scalability of XML.
BTW, I am all for XML and all against proprietary data formats that do the same thing but worse (e.g.
thank God the internet isn't a human right.
If your current relational database schema is either 1) small flat files or 2) a few big tables with most/all of the data stored in "blob" columns: i.e. blobs, clobs, byte arrays, or big varchars. You might be a candidate for an XML database. I'd get two experienced DBA's to agree there was no realistic way to normalize the data, first, but that's me.
We're doing "scientific" computing, and we're finding that classical "SQL/RDBMSs" just don't cut the mustard:
Classical SQL/RDBMSs are fine as long as your data is very ASCII-ish and very short [e.g. first name, last name, street address, zip code, phone number, SSN, etc]. But as soon as you start dealing with very large datasets containing strongly typed data, you're SOL with a classical RDBMS.Anyway, two systems we had been thinking about were Objectivity and Caché. Any thoughts?
Obviously it would be nice if we could get the standard functionality you'd expect from a mature RDBMS package: Seamless backup to a failsafe server, seamless integration with a tape backup system like BackupExec or ArcServe, seamless integration with an industry standard authentication service like ActiveDirectory or Novell Directory Services, etc.
Thanks for any advice you can offer!
Back to your problems: I've never looked at the two systems you mention, but I would suggest rethinking your data management approach at the same time you are thinking about vendors. If you want blobs over 4GB, what you want is to store metadata in the RDBMS and the data block on disk as a file. The reason most RDBMS's set the blob limit to 32kB (not 32MB) by default is to discourage mass storage of untyped data in the database.
Right - I don't have any problem with a classical RDBMS serving as not much more than a front end to a file system [although, at that point, you're essentially dealing with little more than what M$FT has been touting as "WinFS" lo these many years].
The problem is that, as far as I know, there is no standard [non-proprietary] way to store a "file pointer" in a classical RDBMS package. That gets back to the ASCII-ish nature of classical RDBMSs and their access languages [SQL and VB in particular]: They're great if all you have is very ASCII-ish [poorly typed] data [e.g. names, street addresses, zip codes, phone numbers, SSNs, etc.], but the minute you throw something at them that needs to be strongly typed [e.g. "what follows is a pointer to a file in the NTFS file system that lives on the D: drive" or "what follows is a pointer to a symbolic link to a file that was mounted from an NFS share on a server that resides across an FDDI line to a server farm in Lower Gondwanaland"], then you're just screwed.
With the data sizes you're talking about, I don't think that any common means of db optimization will allow for particularly fast queries and I suspect that that requirement isn't applicable to the raw data. When I did scientific computing at Fermilab in the early '90's, we left the raw data on tape and created a whole series of databases for analyzed results. We used a 64-bit physical/virtual addressing system to generate references back to the taped data blocks, but pretty much left the raw data alone after processing.
Nowadays, everybody is bumping up against the 2^32 barrier.
For instance, the new standard in audio recording is 24-bit [i.e. 3 byte] samples at 192K samples per second. But then a single channel comes in at
And that's just one channel for one hour; if you're doing something like 5+1 [surround sound] audio, you're looking at more than 12GB per hour [and God forbid that your studio session should go ten or twelve hours at a time].Now SQL blobs max out at 4GB [2^32 bytes], and Java is an equally 32-bit language. For instance, nothing like the following will work in even the most recent versions of Java:
So if you want to have any hope of getting at data that is inherently 64-bit in nature, then you need something like a very recent version of C++ [at least as recent as the 1998 standard], or C# itself.PS: You mention some time at Fermilab; I think they're using Objectivity at CERN, which is what caught my attention in the first place.
"XML what?"
You can hold down the "B" button for continuous firing.
1. You can download Tamino from this website to trial it: http://www.xmlstarterkit.com/
2. There is support on this website: http://forums.tamino.com/ I recommend that you take a look at these pages, then download the trial version and try it for your needs. Maybe it will be just what you need, maybe not. Spend a few hours to find out!