Slashdot Mirror
With XML, is the Time Right for Hierarchical DBs?
"There have been some pushes to create pure XML databases (info on XML in connection to databases is here and info on XML database products is here) with claims that as they support XML natively, they can offer many advantages over relation databases.
Some of these claims include speed, better handling of audio, graphic and other digital files, easier administration, and handling of unexpected elements. Software AG, a German firm, produce and sell a suite of XML products, including Tamino, a native XML database. They have lots of information on why they think there database is great, not surprisingly, but no benchmarks. So, do the Slashdot community think that with XML the time has come for hierarchical databases? Or is it better simply to use a relational database that can output in XML, or script your way to achieve the same goal?"
I found this SQL queried XML database in PHP. Seems very kewl.
Computer Help
Hierarchical databases won't take over because they're relational counterparts are already so well developed. A relational database can do everything a hierarchical one can, with few exceptions. Even if there is a slight gain to using a hierarchical system, there are much fewer solutions, and consequently the one's that do exist aren't as well developed, so implenting one is more difficult.
Tim ODonnell (trying to be the most
Rather than discard the advantages of relational and object databases, should we instead ask how XML can be used to represent those kinds of relationships?
What do you mean they cut the power? How can they cut the power, man? They're animals!
1337ness for sale.
Just because XML is a hierarchical markup language does not mean that it can only be used for hierarchical things. Perhaps you should look at RDF which can use many to many mappings through resources and groupings (sequences, bags, and alternates). (A resource in one grouping can refer to another grouping i.e. many to many.)
marotti.com
There is lots said on this over at Database Debunkings
Special Relativity: The person in the other queue thinks yours is moving faster.
An afterthought, databases are about storage and speed of insertion/extraction. I honestly don't believe that fitting the database to the data structure is worth the cost or the trouble, just yet.
I think these discussions come up part of the time because people want something new and sexy. In this case OO DB's, which 'XML DB's' are a variant of, may have benefits in specific and limited cases. But I have not been impressed.
Take your classic orders table. Part NO, Custoemr NO, etc. etc. The number of apps with only one parent is tiny, the flexibilty limited, and the whole metadata scanning business awkaward.
For anyone doing and serious larger scale database work some of this stuff is a joke. The idea these vendors have is that we'll be storing XML data in these DB's, ignoring that even for a simple phone directory, the XML data probably takes up a significantly greater amount of space than a simple relational DB would require
And this ignores the significant amount of time and energy invested in toolsets and models for the existing setup. Sure, someone might come out with a chip that runs 2x as fast as an intel at the same price, but unless it is intel compatible how many people would buy it or care?
Maybe its just me, but the goal today is integration and having a special database for XML and special database for this and that just because its faster for this particular problem creates such a level of complexity, which prevents accomplishing even of the most trivial tasks.
Still, XML is only a way how to describe data, that might be often in their structure relational. Why do not store data in their native form and create XML documents out of database on fly by filters?
This question of hierarchical databases is just plain trolling in my eyes.
If programs would be read like poetry, most programmers would be Vogons.
In my experience with XML and RDBMS systems, mapping one onto another is always a dicey task. The primary reason (IMHO) is that XML's ability to represent order as well as structure as data doesn't fit into an RDBMS database without some work. I've seen people try to map both XML and regular DB's onto each other, and my opinion is that the results don't "feel right" on one side or the other unless great pains are made to preserve the structure of the XML doc in the DB schema.
That said, I'm not sure a hierarchial DB will necessarialy be any better than something like an OODBMS with well-modeled objects.
Dojo: defanging browsers so you don't have to
I don't think that heirarchical db's have any real chance of taking over or replacing relational dbs in the future. There may start to be more of a place for them, but many application service providers that use XML still have a fair amount of relational data that needs to be maintained. XML is mainly being used for communication protocals and not so much for internal data structure storage. I think the more likely db trend in the future will be for many users to maintain both relational and heirarchical databases..
WeFunk
The relational model has no major shortcomings. The only thing XML offers that is not already very well done is easier data interchange. As a database administrator, I can tell you there is NO chance XML will dictate a change of how we store data. There are much higher priorities in database management than easier data interchange.
Relational databases didn't come to dominate the database market because they pushed aside equally valid alternatives, they dominate the market because relational databases implement relational calculus. Indeed, that's the very touchstone that distinguishes relational databases from something like DBM and its many descendants.
And *that* is important because it assures the desiger and user that every possible operation is well-defined and (hopefully) correctly implemented. The exact syntax for a "join" may differ, and a specific implementation may be flawed, but everyone agrees to a common baseline.
For hierarchial databases to really take off, they need to have an equally strong mathematical underpinning. For now, AFAIK, there is none other than that you get when you map a hierarchial database into relational tables and use exactly those relational properties. That's a good start, but if you're only using the properties in relational databases, why not stick with them?
As for XML, that's completely irrelevant. It's a good format for transferring data, but that's about it. You can store hierarchial data in an XML file, but you can also use it to store purely relational data or completely unstructured data (in some CDATA block).
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Hi,
I wrote a paper on native XML databases and SQL databases that support XML that appeared on Slashdot a little while ago. While doing research for that paper I asked myself the same question, whether instead of coming up with hybrid methods to store relational and hierarchical data we should store XML in already existing hierarchical databases. Unfortunately things are not so clear cut.
First of all, a lot of data out there is relational and people aren't ready or willing to transition all that data to XML based storage so mixing of relational and XML data will probably be with us for a while. The biggest problem with object oriented databases is that they didn't understand this fundamental issue but it seems that with XMKL databases the vendors understand that hybrid data will be with us for quite a while which is why Tamino supports importing data from relational sources and even ships with a SQL engine.
Secondly, XML documents have a lot of metadata beyond the hierarchical parent-child relationships such as processing instructions, comments and entities which are require more intelligence in the support from the database than just storing parent-child relationships.
Finally all the major [commercial] relational database vendors have included some sort of native suppport for XML including XML types and there is a an ANSI standard in the works for combining XML and SQL. From what I've seen, none of the hierarchical databases plan to support XML as much as the relational databases have or plan to.
Now if you were simply asking whether a native XML database can be built on top of a hierarchical database then I believe the answer is yes. Then again native XML databases can and have been built on object oriented databases and relational databses so it makes sense that they can be implemented in a database system that is more suited to handling hierarchical data.
It certainly seems like the same thing is happening with XML that happens with any new toy: "my friend told me XML was cool for stuff, so I'm going to convert everything to XML so I can be cool too."
I was pretty sure that XML was useful in that it was a human-readable data-encoding mechanism that "average" users could get a grip on and utilize in sharing information between heterogenous systems, but it seems like people are completely missing the point these days in how to use XML effectively.
A lot of the benefit of using XML is quickly becoming negated by everyone coming up with their own DTDs and the lack of standard formats for encoding data that is to be shared. As an example, here at the university I attend, there is a project for sharing information about biological species' population data amongst sister organizations. The goal is make the information possessed by all these organizations available to all the others. The trouble is that they have all come up with their own format for storing the data they collect and can not agree on what standard should be used, so each organization is encoding all their information with a different XML labeling scheme. My first questions was: "Why in the heck are you using XML to encode the data anyway?" Seems easier and saner to just store it in your relational database and make the database accessible to sister organization who can then encode the information however they want for their end-users through their client applications rather than the organization holding the information imposing order on people wanting access to the information.
To make a long story short, XML encoding doesn't help you store the information more efficiently at all and with the state of the "formatting standards" today doesn't even really provide an efficient way of sharing information between organization or an efficient way of encoding the information for transmittal to other organizations. It seems as if people are missing the forest for the trees in how XML can be useful in its relation to data encoding and we should stick with our trusty ole relational and object-oriented database models as they have shown their usefulness and efficiency.
Or is it better simply to use a relational database that can output in XML, or script your way to achieve the same goal?"
I believe that RDBMS's should add functionality to read/write XML, especially as the XML Schema recommendations is basically done.
The idea that XML should be the permanent storage format is a bad one. There is a lot of power in a normalized data model -- it enforces data integrity , while eliminating data fragmentation automatically and it minimizes transaction resources.
Consider XML representations for different entities that all share some kind of child entity. For example: people, businesses, and schools all share addresses. In XML, you want the addresses to appear in the description of the individual object. Does that mean you want to store the addresses separately that way? Absolutely not, because then when you enforce constraints or ask questions about addresses, your data is fragmented in three places. For that matter, how do you know all the entities that might use addresses? In an RDBMS, you can inspect all the foreign keys to the address entitity. What's the XML analog?
I work for a company that has been doing hierarchical DBMSs for years. The company is Applied Technical Systems. We make a database engine called CCM.
XML is a great way for exchanging data, but the term XML databases is very misleading. If the database engine actually stores data in native XML, it's going to be *very* slow. I think the point behind XML is that nobody should really have to care what your backend is as long as you can export reasonable XML. Note that I say reasonable XML. And XML export that simple encodes the rows and fields in a table to XML with <row> and <col> tags is NOT reasonable. It conveys no actual knowledge of the real structure of the data.
Storing XML data in a relation DB can either be a very hard problem or a very easy one. Let me explain.You could look at some XML and define a DB schema for it, not too hard to do. Problem? It's not generic; a human has to re do it each time the XML structure changes. The alternative is to store it all in one big table and index the hell out of it. Problem? It's slow. At that point you aren't using any structure of the XML or the power of relational DBs.
I'm a firm believer that efficient XML storage, querying and retrieval will require a hierarchical database. The problem is that there's several features (bugs IMHO) in XML (and XPath) that, in a way, are throwbacks to relational DBs. IDREFs and the notion of document order particularly bug me. I ran into these this summer when I was on a team trying to build a XPath and XQuery front end for CCM.
We're gradually seeing the XML world change. Early XML documents were similar to the type mentioned above. They were flat. When you start adding depth the information inherent in the structure of the data becomes apparent. Another thing I'm glad to see the industry move away from is the notion that XML resides in files. Many (if not all) of the early XML parsers made this assumption. It was a pain in the ass to parse from some other source, like a buffer in memory.
XML is not a magic bullet. Relational database won out over the Hierarchical model for a lot of reasons. For instance, there exists a number of integrity constraints with the Hierarchical model such as
1) No record occurrences except root records can exist without being related to a parent record occurrence. This means that
a) a child record cannot be inserted unless it is linked to a parent record.
b) a child record may be deleted independently of its parent however, deletion of the parent record automatically results in the deletion of all its child and descendent records.
c) the above rules do not apply to virtual child records and virtual parent records.
2) If a child record has 2 or more parent records from the SAME record type, the child record must be duplicated once under each parent record.
3) A child record having 2 or more parent records of DIFFERENT record types can do so only by having at most 1 real parent, with all the others represented as virtual parents. IMS limites the number of virtual parents to 1.
In addition to these flaws, relational databases have had over a decade to become mature, optimized, and enterprise scalable. Harddrive partitioning for such databases as oracle work out perfectly with the cylinder, sector, and tracks of a hard drive to allow for the fastest read/write times as can be possible.
Too often people see that XML "can" do so many things and decides that it should be the way things are done but XML is NOT a magic bullet and just because it has the potential to do something does not make it the best methodology for doing so.
I was surprised to see so many questions of the form "what's wrong with relational databases"? Relational databases have a well-known problem called "impedance mismatch" when mapping multi-linked object structures. Many links on the impedance mismatch issue can be found at this Google search.
Anyone who has tried to take a natural set of application-side objects and map them onto a relational database is already quite familiar with the problems created by the proliferation of tables needed to map simple application data structures, as well as the large amount of development effort needed to deal with simple relationships that would be trivial to specify in an object model such as Java's or XML's.
There is clearly a need to move on to object databases, but installed base and skill set inertia have blocked this transition, with the result that database-oriented applications have remained hamstrung in their friendliness and feature set.
Tim
Zope uses an object database known as the ZODB. Some forms of many-to-many relationsships and such can be handled via the use of selection and multi-selection properties, which are designed to distinguish between a selected element and the list of available elements. The list of elements can be derived from a property on the current object, a property on a parent object, or be created via a method call - allowing for non-traditional (for OODBMS) cross-linking of objects. Of course, since this sort of thing is a workaround, no true relational links are created... 'Soft Relations' may be ok for MySQL, but in big application development, relationships must be enforced! Thus, the big-boys in RDBMS all enforce foreign keys (mysql does not)...
Of course, I've found that by careful creation of object heirarcies, very complex applications can be created on top of a OODBMS that are in fact more robust, in some ways, then the relational couterparts. The Bigest hurdle (Short-term) I see to OODBMS (including ones based upon XML [the ZODB can export objects as XML but they are stored differently internally]) is the lack of a true query and data manipulation language - like SQL. Sure, OQL exists, and is even technically a standard, but it A) sucks and B) is geared towards large java applications with huge amounts of active objects, not general purpose OODB queries. Thus, without such language, OODBMS are all disimilar in how one queries and creates/updates data, and in many cases, the only interface is a truely procedural one! Thus OODBMS are forced to use proprietary tools, and are locked into one system - not to mention speed of development (something normally associated with OO development and OODBMS in general) is hindered by the excessive amount of procedural calls one needs to simply query thier data...
Recently, an add-on to Zope addressed some of these issues. Called 'ZOQL' - it uses a SQL like syntax and allows for very discrete querying of the ZODB (something one had to do programatically using the 'ZCatalog' before) with all of the familar aggregate and comparison operators SQL users love... Of course, this _still_ doesn't address the issue of soft-relationships:
I think the bigest hurdle to OODBMS in the long term (tools like ZOQL are interfaces to existing systems, thus can be mplemented easily) is the lack of handling relationships. It seems that most RDBMS force a developer to think in Relational terms about the data, and most OODBMS force you to think in terms of objects... Most problems can be mapped to either of these domains, but you are forcing the data-model-type onto the problem. What is needed is a hybrid system, an 'Object-Relational' DBMS. This is to say that OODBMS system makers desist with the traditional OO idea that relations are of the following types:
- Object A is a Object B
- Object A Has a/many Object B(s)
What RDBMS systems excelled in (and thus fell into pupular use for) was ease of management and allowing common data to be moved and grouped. A 'Look-up Table' - for instance, which simply holds a list of common data (an enumerated list) and can be centrally maintained is a Boon in the RDBMS world. For example, you have a lookup table of car manufactureres, and one of them changes its name... Instead of updating all N Cars that are made by the manufacturer, you simply update the single record in lookup table. Since each car would have somehting akin to a 'Manuafactuer_ID' column linking it to the lookup table, the Cars belonging to the manufacturer are all taken care of.How does one do this in a hierarchal system? Well, the easy answer would be that each manufacturer object contains all the cars that manufacturer makes. Simple, right? WRONG. Why?
Because each car also has a body-type (compact, sedan, SUV, truck, van, etc...) - which in a relational database would simple by another lookup table, but in an OODBMS poses data management issues. Do we put body-type higher then manufacturer? If so, then we have to maintain the list of manufacturers for each body type, causing headaches. Or do we put body-type below manufacturer, causing us to need to maintain a seperate list of body types for each manufacturer - these lists of course need to match exactly if we ever plan on being able to search or do reports based upon all cars of a specific body type.
Sadly enough, this sort of seperate-enumeration-relationship isn't implemented (well) in any OODBMS I've found.
Take the ZOBD for example, its selection and multiselection lists Try to handle this situation, but fail because relational integrety is not maintained! That is to say, behind the scenes it's not a true reference to a value in the enumerated list, but just a text entry representing a value in the list. If the value in the list changes, the selection-property does not update, leaving you with the equivilent of MySQL's bastard-children, the orphaned records.
This sort of soft-relationship handling is Ugly and BAD for maintainaility, but OODBMS users are faced with two ugly choices each time they map such a relationship: Do I store this as a plain-text property and just update N records each time this changes, or do I map it into the hierarchy and deal with the headaches incurred by doing so...?
I don't think I've answered the question, but hopefully I've at least shed some light on the subject for members of both the OODBMS camps and RDBMS camps... Now if only a useful ORDBMS were to come along...
(Note that PostgreSQL and some other RDBMS actualy can be used in a semi-OO manner, but this is usually reserved for inheritable structures of data to be used for specific extensions to the data model - thus the SUV table inherits from the Cars table and adds some columns - but all other relationships SUV has will still be relational)
man is machine
While a hierarchy is often used by humans to organize and structure things, that should in no way impact how the data/information/objects are treated as individuals. Look at the common file system hierarchy and it's easy to see that burying files under a hierarchy of directories actually makes access to that information harder. It wasn't so noticeable when we were all just managing a few MB of files, but now people are beginning to store large picture, movie, and sound libraries. File managers have mistakenly stuck with the hierarchy instead of using information associated with the file itself (ID3 tags, etc.) to organize it all. What is really needed is a better approach to representing metadata so that information can be accessed directly based on those metadata attributes and not have it hidden in the hierarchy. I have a short essay on this from the work I've been doing on a Meta Object Manager (MOM), but it needs to be cleaned up before it could be published.
The desire to impose a hierarchy on the data itself instead of considering a hierarchy as simply one view on the data is a step backwards. Nobody who manages large amounts of data is looking to jam it into a static hierarchy, and so XML is not an answer, nor is any hierarchical representation.
In design, the logical construction of the program and its data structures should be relatively independent of the physical implementation of said.
Basically, as I read your question, you are using a logical design that is hierarchical (an object structure experessed in XML) and wondering if it would not make more sense to store it in a hierarchical database. Maybe.
However, relational databases form the current state of the art and have been highly optimized such that any theoretical performance gains from better matching of logical structure to physical lay-out in the database are likely outweighed. More generally, by insisting on a match between logical and physical lay-out, you would potentially be limiting yourself to a specific physical implementation, one that may not provide good performance relative to others.
A better solution to your problem might be something referred to as a persistence layer. This adds another layer of abstraction to your application, in the form of a mapping, between your logical design and your actual physical mode of storage. There now exist publically available free (as in beer, and in some cases open-source) tools that will automate this mapping. Generally, any performance hit from the abstraction should be made up in the speed of the superior physical implementation, and the freedom to switch later is also important.
Two that exist for java are castor available from exolab and a pilot implementation for Sun's emerging Java Data Objects standard (see http://java.sun.com for that tool).
First, hierarchical databases weaknesses are not limited to many-to-many relationship modeling. The simple fact is that some data is better represented in a hierarchical fashion (a directory service IS a hierarchical database) and others in a relational fashion. XML is a tool for exposing that data to external sources regardless of its internal representation(s).
A hierarchical model can easily be done in a relational system by simply using a self-referrential 2d table. Each element has an unique ID and the ID of its parent.
The enXyme project attempts to map XML data onto a relational database schema. The goal is to allow complex, specific queries of XML data. It's not easy to capture ALL of XML, with all its possibilities, but you can do parts of it. The project already has a basic XML schema parser, a script that takes an XML schema and generates a series of sql CREATE statements that reflect the hierarchy described by the schema.
I guess a pure XML database like the ones mentioned in the article would be better at this, but the advantage is that relational dbs are already in wide use.
Structured data. Structured searching. The Enzyme Project
Anyone can explain to me what is suddenly so wrong about relational database with hierarchical indexing?
/.) there is a correct answer.
Maybe its just me, but the goal today is integration and having a special database for XML and special database for this and that just because its faster for this particular problem creates such a level of complexity, which prevents accomplishing even of the most trivial tasks.
Forgive me for tooting my own horn on this one, but I believe that (for once on
I summarize the answer in a paper written for VLDB 2001 (www.vldb.org). The paper presents joint work between Stanford, Berkeley, and RightOrder, Inc. It can be found online here (in PDF).
What we found is that relational systems, with appropriate indexes for XML data, give the advantages of both worlds. XML is a hierarchical representation in only the loosest sense. It's written linearly in a flat text document, just as a child learns to write things down on a piece of paper. However, you wouldn't convince anyone but that same child that something written on paper can only represent two-dimensional objects just because the paper itself is flat. XML in many variants is plainly richer in concept than its simple hierarchical representation and thus quite suited to ER. I believe a previous poster mention RDF... a perfect example.
Punchline: XML is neat, XML is tasty, but XML is not inherently more or less expressive than ER; it just requires a little critical thinking (and index tweaking) to tune ER engines to deal with it. (Once tuned, the ER engines dominate all others in performance.)
You can represent the structure, but you can't manipulate it using standard relational logic.
For example, take a table representing a parent-child relationship. Now try to sort the persons in the table by their number of descendants. SQL has only recently been extended to allow this query to be posed. Perhaps your relational database can handle this kind of query, where you have arbitrary-depth path walking, ybut ou can't expect it to handle them efficiently.
For more complex hierarchical relationships, an object database is more apt, or an XML translation kit for your relational DBs.
Horses for courses.
LDAP is an example of a really good and well developed implimentation of the hierarchial database idea. However, try keeping track of whay your customers bought from who with LDAP. So while LDAP (and other hierarchical dbs)do certain things better, don't try to run a CRM suite off one.
The basic problem is that the entire database is rarely hierarchical in nature even though some queries may be.
LedgerSMB: Open source Accounting/ERP
Relational databases are here to stay and will be with us for at least the next fifty years. It is better to think of ways of translating relational data than supplanting it.
Okay, I've worked for two different network model database companies -- the network database model is just an extension of the network model to allow graph schemas instead of a strict hierarchy. I've also worked with two companies that we mapping hierarchical structures onto relational databases.
You can think of data structures as (leaving ternary relationships and such aside) some sort of network of relationships. When you think of it this way, relational and network model databases have more similarities than they have differences, especially when you consider that using surrogate keys is the moral equivalent of a network model "pointer".
Okay so you have this network of relationships, mapping a hierarchical structure onto that is simply picking a starting point and traversing the structure from that "viewpoint" without visiting a node via the same relationship twice (simplified algorithm but...) One of these groups used to think about this like you had a multi-legged turkey. You grab one leg and hold it up. All the other legs hang down -- you grab another leg and a different set of legs hang down.
So, if you buy that, does it really make sense to represent any sort of network of information in a hierarchical form? Well, yes and no. It makes sense from a presentation and maybe interchange perspective but not from a native storage perspective. It's simply to constrictive and you and up representing relationships that don't fit into a neat hierarchy programmatically in the application code instead of explicitly in the database schema. 25 years from now, someone is trying to reverse engineer your code and figure out how all this data is related -- blech. Ever wonder why IMS application are generally left alone and newer applications are not usually written to IMS. This is part of the reason why. (yes there are some but they are the exception).
Throw in to this my experience working with a bank that had hierarchical data and the extent to which they went to circumvent that restriction, and I'd say that native hierarchical storage for XML is a bad idea. Granted it's tempting but it seems ill advised since it's very likely that your data will survive long beyond the lifecycle of the system used to originally store it.
<RANT>
The original question didn't provoke this but I've seen a couple of responses about using XML as a native data storage format. Let me say that, unless the data is very static, it's a monumentally stupid idea to do that. XML is not a replacement for a database.
I find that most of the people who really want to do this are ignorant of all the work that goes into real database systems. They don't understand lock management, transactions, rollback and recovery, free space management nor the scalability issue that real databases take care of under the covers. If you feel tempted read this
You throw this plus the representation of non-hierarchical relationship with IDs and sooner or later you will find yourself in a text editor tracking down ID/IDREF pairs to find out where your data is corrupted. Or writing scripts to validate your "entire data set" -- above a few megabytes it can be really painful.
For God's sake, expect to use XML to store data that you are going to update with any regularity.
</RANT>
You are not a beautiful or unique snowflake -- but you could be if you got off your ass.
LDAP/X.500 heirarchical databases are all well and good, until you want to run a query that asks which customers have what services, especially when everything is keyed on phone number. You know what we ended up doing? Pulling the entire database out of LDAP every night, and putting it into Oracle to run the reports. Nothing sucks more than a full table scan in an X.500 database.
I do agree that heirarchical databases are great where you are only going to access the data from a single key, like passwords and email addresses... But, they should probably be provisioned from an external RDBMS if you are looking to do reporting.
Jason PollockXML may be hierarchical but the data it is used to markup is not necessarily hierarchical. For instance, XML can be used to markup conventional fielded (flat file) data to serve as an interchange format.
More importantly, XML is used to impose some structure on inherently unstructured text. The structure it provides is based on some assumptions of how the data will be used or how it will be presented. If the data is used in some otherway, the markup can be useless.
An example is a book. For XML purposes, it can be described as structured by chapter, section, subsection, and paragraph. For information purposes, tags are assigned to represent the ideas, terminology, names and other index-like content. There is virtually no structure in these index type of tags but they convey the most important information in the book.
Or not. These tags are assigned based on assumptions about what readers are interested in. A different set of assumptions would produce a different set of tags even thought the structure of the document would stay the same. If the sentences and paragraphs are shuffled and exerpted for some other publication, even the structure becomes irrelevant.
How this inherently unstructured information is stored is relevant to how it is managed, that is, how it is backed up, how access is controled, how changes are tracked. However, when it comes to putting the information to some useful purpose, it is the retrieval mechanisms that are important. The issues here are how easily the user can specify the type of information he wants and how accurately the mechanism can find it. This process is usually independent of the underlying structure and uses some higher level concepts of relevance and context.
The question of whether to use a hierarchical, relational or object-oriented data structures misses the point for textual data, for which XML is commonly used, because none of these structures capture meaning.
Topic maps make a heroic stab at capturing meaning in XML markup but still only within a set of assumption. I suspect a true meaning markup language is theoretically impossible, or at least theoretically very far in the future.
Contrary to some of the comments I've read here, LDAP isn't an implementation of a database, it is a *protocol* for accessing directories. LDAP data could be stored in anything - a hierarchical database, a relational database, an object database or a flat file. Let's not confuse the issue under discussion.
Great Windows SFTP Server!
Indeed, that's the very touchstone that distinguishes relational databases from something like DBM and its many descendants.
The alternative to relational databases is not "DBM", it is object oriented, tree structured, logical, and other kinds of database models. Those are just as well defined as relational databases.
And *that* is important because it assures the desiger and user that every possible operation is well-defined and (hopefully) correctly implemented. The exact syntax for a "join" may differ, and a specific implementation may be flawed, but everyone agrees to a common baseline.
Relational databases provide a common baseline for a primitive set of relational operations. Real-world implementations of those models have been augmented by zillions of operations that weren't part of the original relational model and that often don't even fit into the relational model. And without those extra operations, relational databases would not be useful in practice.
For now, AFAIK, there is none other than that you get when you map a hierarchial database into relational tables and use exactly those relational properties.
Are you kidding? It is a major pain trying to express hierarchical data in a relational database model: the relations that describe hierarchical data and the operations that you might want to execute often require complex, multiple, inefficient queries and updates, and the relational model provides few tools to ensure that the corresponding relations remain consistent.
The semantics of tree structures are trivial to define. People do it in programming language classes all the time. And it is trivial to formulate a database model corresponding to it. In fact, if you have an object-oriented database that respects language semantics, you get hierarchical databases automatically when you define an abstract tree datatype.
Still, so-called "relational" databases will continue to dominate the market for a long time to come. That's not because the relational model is particularly well-suited to a lot of applications. In part, that's because "relational databases" are not purely relational anymore: they generally include numerous facilities for object-oriented and hierarchical databases, under a "relational veneer". They even include the old "navigational" database systems, combined with the widespread use of stored procedures that do whatever they want whenever they want it on the database server.
In different words, traditionally relational databases will provide increasingly better support for hierarchical and object-oriented data, but they will continue to also support the relational model, as well as relational access to these other data types. And newly developed databases with other kinds of data models will provide an SQL or other relational frontend to their content. And marketing will continue to include "something-relational" in all the advertising because otherwise the old database hands won't buy it.
Taking out attributes with multiple values and putting them into a linked table is core to the functionality of relational databases.
I'm no newbie to RDBMSes; I Know that this is a core concept in the system. What I was referring to (and rather poorly, might I add) is that to get the same functionality of a hierarchial system you need to use these reference tables for everything lest you run across something which breaks your mold. This reduction to absurdity is eliminated in hierarchial designs.
I'm not a hierarchial guru; I have more experience and can relate (ha!) better to RDBMSes than I can hierarchial databases. But to say that a relational (table-based) database solves all your problems just because you can organize everything with relations is absurd. You lose the entire concept of a database entry or object by doing this. Instead of having a table consisting of contact information, you have a table for names, a table for spouses, a table for phone numbers, a table for fax numbers, a table for email addresses, a table for postal addresses... Lord help you if your documentation is ever lax or worse yet, you lose it (or the table views or the driving software) altogether! Each methodology has its place.
99+% of all corporate data that isn't in a flat-file or (possibly three-dimensional) spreadsheat is in relational tables. The typical task that XML has been designed for is to standardize data exchanges between differently-structured relational systems, by providing sets of tags specific to the standards of specific industries. The whole point of XML is to enable companies to continue to use their current investment in relational databases, without the drag of having to do custom data conversions when dealing with suppliers or distant divisions in the company.
... just about everything industry consists of. So if there's an impedence mismatch between relational and XML that's enough to make trouble, it's XML that should be replaced by another model.
If you're going to throw out the installed investment in relational databases, you might as well just design a common database standard per industry (rather than an XML data exchange standard) and let them all exchange native data rather than translating in and out of any exchange format. Obviously that won't happen.
Now, if you're a new firm, you might decide it's easier to go OO or heirarchical or keep your data in slips of paper in a shoe box. But most of the available tools and solutions will continue to respect that relational works real, real well for inventory, manufacturing, accounts
What design changes would be required to produce XML's relational equivalent?
"with their freedom lost all virtue lose" - Milton
I don't think XML by itself carries enough metadata to understand much beyond whether a document is valid or not. I think RDF and RDFS have a big role to play in getting XML database ready.
Perhaps hopping on the XML database bandwagon before RDF technologies mature could be a mistake. Forget the semantic web, I want to see the sematic database.
W3 RDF
A Good RDF resource
It's easy to dismiss a new database technology as irrelevant because of the dominance of the RDBMS, but you should really learn more about it and when it is appropriate and when it's not. It's not going to replace relational, and isn't intended to. Here's a few links where you can learn more beyond what's available on Ronald Bourret's site mentioned in the original post.
The XML:DB Initiative
The dbXML Project (open source native XML database) Soon to become an Apache XML project named Xindice
eXist (another open source native XML database)
My blog on the subject.
Kimbro Staken
A bit surprised to hear that 'Hierarchical databases were blown away by relational versions' - since I'm pretty sure they've been paying my pay check for the last three years... :-)
There are a large number of heirarchical databases out there. The big fellas are the X500 directories (X509 certs came out of this work). More common are X500's demented kid sisters, the LDAP directories ( rfc2251). The DNS system also fits the description 'heirarchical database'.
As far as XML goes, there are people storing XML in directories - although they're still fussing about exactly how to do it. There are a bunch of people trying to come up with standards - check the directory services markup language people www.dsml.org.
There are people trying to sell XML enable directories - Novell sells an XML directory, but most directories can be used to store XML (including our 'eTrust Directory').
As a final quicky - when do you use a directory over an RDBMS? Directories are good for naturally heirarchical data with few cross connections. They are usually optimised for slow writes/fast reads. They are *very* good for distributed data (e.g. DNS, international organisations etc.). The X500 spec defines a very fine grained security model, which can also be useful. However, if your data is closely cross-linked with lots of relationships... well, use an RDBMS!
Wer mit Ungeheuern kämpft, mag zusehn, dass er nicht dabei zum Ungeheuer wird.
You mentioned configuration files: if all you're talking about is a linear config file, then XML might not give much benefit. But if your config file has a hierarchical structure, XML does provide a benefit, since it provides a well-defined and standard way to represent that hierarchical structure. In addition, XML-aware editors make it easier to work with these files, plus you don't have to write a specialized parser for it, plus you can display the file in an XML-aware browser, plus you can run automated transformations on the file, plus...
XML is one technology where the mindless adoption by people who don't really understand why they're adopting it, may in fact be of benefit to everyone in the long run.
XML is the first and closest thing we have to a universal standard data format. We're better off having such a thing than not having it. Since XML is the first such format, naturally it has its problems and limitations. But it's a step in the right direction, and we'll only find out how best to improve it if we use it heavily.
On the subject of the article, though, you're right. Whether a database's native representation is XML is irrelevant.
The problem with your scheme is that as the list of information grows the amount of time to do the access grows linearly. With systems where there are thousands of concurrent users all doing large numbers of data accessing of the information your solution would grind the system to a halt. While technically possible to do this with a relational database it is impractical. Taking into account that 90% of the databases use is doing reads and the access of such a system would be even worse. Also take into account that the system has to be able to handle some data which does not change and some data that has to be grouped based apon a visit date. Your system would have no efficient way to group certain elements by date.
Nice to see some solid and useful info being posted on /. every now and then...
The question of whether hierarchical or relational databases are better, more flexible, dependable and so on has already been answered decades ago. I fail to see how a popular hierarchical data transfer protocol re-opens the question. If XML is going to be pushed far beyond what it was designed for then it needs to become far less hierarchical rather than having the rest of the world, and especially mainstream DBMS, re-arranging itself according to XML's limitations.
Today you might want to store your history of employee-department assignments by nesting employees under departments, but at some point you may also want to nest work histories under employees.
pr0n - keeping monitor glass spotless since 1981.
Speaking of DBMSes, one of the electrical engineers at work wanted to learn how to use the Oracle DB one of our projects is built on. Somebody told him "No problem, it's dead easy. Almost everything can be done with only two commands, SELECT and UPDATE. Simply learn those two and you'll know everything there is to know about DBs." Apparently, the CS guru who witnessed this nearly imploded. Wish I'd seen it...
-- ;-)
Kuro5hin.org: where the good times never end.
I think you're wrong. How about: "Not all data is a tree" instead?
pr0n - keeping monitor glass spotless since 1981.
The relationship property of two nodes in a tree IS [a relation].
Indeed it is. However, something like the "parent(x,y)" relation satisfies particular properties that the relational model has no support for enforcing. Furthermore, algorithms over trees are intrinsically recursive and usually require a recursive exploration of such a relation; you cannot express that with a bounded number of relational queries--it requires iterating queries together with transactioning across those queries, procedural code that falls outside the relational model and that, incidentally, is also very slow when implemented on top of standard relational databases.
The complexity for any representation is of the same Big-O order, regardless of the database type.
In real-world applications, constants matter, a lot in fact, so even if the algorithms weren't just the same big-O, but the same big-Omega, there would still be an issue. Second, the issue is not a clear-cut as you seem to think: depending on the specific relational database model one adopts, you may end up paying extra logarithmic or even linear factors in the size of the database.
No, in any RDBMS, you just have the numbers in a table of phone numbers, and use a one-to-many relationship from the person to the numbers.
Sure, many database "designers" (and I use that term very loosely), will do something bone headed like define a fixed ten-decimal-digit number as the phone number field, but that's not the RDMS's fault.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
we shouldn't be talking about "XML data" is if it was somehow the core representation.
Exactly. The question of whether to use hierarchical databases is orthogonal to the question of whether to use XML. NetInfo, the DNS, LDAP, and smalltalk source code trees are all good examples of where you should use a hierarchical approach, and this was true long before anyone thought of XML.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
For examples of how use relational architecture for hierarchical data see Trees in SQL by the irrepressable Joe Celko.
Briefly summarized, his approach is: "tree structure can be kept in one table and all the information about a node can be put in a second table."
The OO way to answer this is that body-type is a class and compact, sedan, SUV, etc... are instances of it. Each car would have some instance of body-type as a member. I've implemented this sort of thing in a roll-your-own OODB (in Ruby) and in a OODB-on-SQL (in Delphi); in both cases it was painless. The only thing that is remotely tricky is to avoid infinite loops in your low-level serialization code, by doing lazy streaming or by having a serialization flag, or stack, etc. just in case some later person creates a body-type (e.g. batmobile) that somehow refers back to one or more instance of car.
-- MarkusQ
I wish I knew more about filesystem programming, because I've long wished to write a simple file system that uses a structure which is independent of the presentation of files.
This doesn't require you write a fs, but rather it suggests an abstraction layer above any particular file/object store, be it data stored in a hierarchy on the file system or in an XML file or data stored in a database.
It would be simply wonderful to create a file system view, per user, which exists not only to restrict what they can see (almost like being chroot'd with lots of mounts in that directory), but also to make certain things more accessible or differently organized based on properties you feel are important. Doing so currently requires a shitload of symbolic links and manual maintenance when adding or removing files. Instead, you should be able to mount a file set under a name and put a query in that file set, so that it appears to be a directory with files that match some given attributes. Then you build a hierarchy of those, since that's a natural way to think about things.
Dead on. I wanted to give an example from my paper here, but the Slashdot lameness filters aren't allowing it.
The lack of categorization, or meta data, for files has been a thorn in users' collective side for decades, and with the death of Mac metadata in OS X, there's no real proponents out there for improvement.
Actually, Mac OS X metadata handling is richer than in previous versions, getting away from a file-centric model and closer to a user-centric one. It still isn't up to snuff, though, which is why I'm writing Mary, my Meta Object Manager, using Cocoa. So I guess you could say there is at least one proponent. :-)
I have been toying for a while now with how to represent data efficiently in a manner in which it can be efficiently walked with an XML document-object-model, if not in XML itself.
;)
My core concern is actually building the indexing tools to handle the matching of insane numbers of actual Xlinks (or a mildly simplified form thereof), to make a dataset that is distributed and provided by multiple subsystems hold together.
My most recent iteration used a simplified form of Xlink to join data supplied by code that implemented a specialized DOM model (code that understood how to browse a given object model, for reflection purposes), with data taken from a flat text XML document (browsed with a different implementation of that model).
The real concern I have with this sort of system is the time it takes to register a document against all of the current X-links and deriving an efficient system for parent-child connection when multiple parents can exist (transparent links from other nodes). Registering a new node or worse, changing data in a node is an expensive operation in this model.
My need for extensive X-link support is to be able to provide a sort of on-the-fly XSL translation of one branch of data into another. The links would then connect to the translated data rather than the original.
A secured view of the data could be provided by handing a link to a DOM node that was walking the XSL view, rather than the original data. User security becomes a XSL-like document.
My filesystem becomes a branch of the tree. It is admittedly an awkward security mechanism to do XSL-style matching against the tree, but it does not have to be done in XSL itself, another utility which walked a base dataset and returned a filtered view in the same DOM model provides the same security mechanics.
There are some issues with the current mechanisms for conveying XML information (DTDs do not localize well, etc).
Method invocations are shaky at best and rely on sharing handles of some sort (SOAP, CORBA, etc). I transmit data either by copying a branch of the tree, or handing off a handle to a CORBA or SOAP object that can walk the DOM on the local system.
Either way looks the same to the client.
A hierarchal model has a certain level of appeal because of the simplification of the new-branch registration process, but severely limits the effectiveness of the tree processing tools you can build. OTOH, the relational model could be reconstructed with some stylesheet tricks.
My present use is in an operating system project as the mechanism for accessing system resources, the 'file system', user data, etc.
There is appeal when it comes to generating a backwards compatible view of the data, because you can provide a translator which takes the current data and translates it back to a view compatible with what a given application expects, etc. Method invocations through a function call-translator can allow for constrained arguments to methods, etc.
The transparent linking model also has appeal for simplified remote method invocation, a filename is just an Xlink, etc.
Active Directory, eat your heart out.
Sanity is a sandbox. I prefer the swings.
Java applets. They were a goot first-shot at a portable sandbox interface, but there is better.
Well, the Amiga SDK is a start. Also, note that just because there is currently little better doesn't mean that the status quo is the best thing possible.