Slashdot Mirror
With XML, is the Time Right for Hierarchical DBs?
"There have been some pushes to create pure XML databases (info on XML in connection to databases is here and info on XML database products is here) with claims that as they support XML natively, they can offer many advantages over relation databases.
Some of these claims include speed, better handling of audio, graphic and other digital files, easier administration, and handling of unexpected elements. Software AG, a German firm, produce and sell a suite of XML products, including Tamino, a native XML database. They have lots of information on why they think there database is great, not surprisingly, but no benchmarks. So, do the Slashdot community think that with XML the time has come for hierarchical databases? Or is it better simply to use a relational database that can output in XML, or script your way to achieve the same goal?"
I found this SQL queried XML database in PHP. Seems very kewl.
Computer Help
Hierarchical databases won't take over because they're relational counterparts are already so well developed. A relational database can do everything a hierarchical one can, with few exceptions. Even if there is a slight gain to using a hierarchical system, there are much fewer solutions, and consequently the one's that do exist aren't as well developed, so implenting one is more difficult.
Tim ODonnell (trying to be the most
Rather than discard the advantages of relational and object databases, should we instead ask how XML can be used to represent those kinds of relationships?
What do you mean they cut the power? How can they cut the power, man? They're animals!
excelon has a very full featured XML database.
We use it exclusively and it kicks ass.
Well, the current version does. Pre 3.0 sucked ass.
http://www.exceloncorp.com
1337ness for sale.
Just because XML is a hierarchical markup language does not mean that it can only be used for hierarchical things. Perhaps you should look at RDF which can use many to many mappings through resources and groupings (sequences, bags, and alternates). (A resource in one grouping can refer to another grouping i.e. many to many.)
marotti.com
There is lots said on this over at Database Debunkings
Special Relativity: The person in the other queue thinks yours is moving faster.
An afterthought, databases are about storage and speed of insertion/extraction. I honestly don't believe that fitting the database to the data structure is worth the cost or the trouble, just yet.
I think these discussions come up part of the time because people want something new and sexy. In this case OO DB's, which 'XML DB's' are a variant of, may have benefits in specific and limited cases. But I have not been impressed.
Take your classic orders table. Part NO, Custoemr NO, etc. etc. The number of apps with only one parent is tiny, the flexibilty limited, and the whole metadata scanning business awkaward.
For anyone doing and serious larger scale database work some of this stuff is a joke. The idea these vendors have is that we'll be storing XML data in these DB's, ignoring that even for a simple phone directory, the XML data probably takes up a significantly greater amount of space than a simple relational DB would require
And this ignores the significant amount of time and energy invested in toolsets and models for the existing setup. Sure, someone might come out with a chip that runs 2x as fast as an intel at the same price, but unless it is intel compatible how many people would buy it or care?
The relational model is very good for most situations, and has been very studied and optimized. Noone would transition back to pure hierarchical DBs.
If at first you don't succeed, skydiving is not for you
Yes, I think XML databases can be useful sometimes, as even though relational is faster and better developed in some cases native XML products have the capability to store any data, without prior setup. I know I'm using dbXML (http://www.dbxml.org) in a product of mine which allows 3rd parties to store arbitrary data associated with a user.
Also, you get the full advantage of the XML technologies developed by the W3C and others - the ability to do a simple query, transform that data and then send to a web browser with very little coding involved is a great bonus.
(i've forgotten my login, time to go create a new one i think)
Maybe its just me, but the goal today is integration and having a special database for XML and special database for this and that just because its faster for this particular problem creates such a level of complexity, which prevents accomplishing even of the most trivial tasks.
Still, XML is only a way how to describe data, that might be often in their structure relational. Why do not store data in their native form and create XML documents out of database on fly by filters?
This question of hierarchical databases is just plain trolling in my eyes.
If programs would be read like poetry, most programmers would be Vogons.
Extropia has a detailed tutorial on databases of all types.XML:DB discusses the differences between object-oriented databases, hierarchical databases, and relational databases in detail. You may be interested in DBX a DBMS that is written completely in PHP, and works using XML style text files as its native format.
Computer Help
In my experience with XML and RDBMS systems, mapping one onto another is always a dicey task. The primary reason (IMHO) is that XML's ability to represent order as well as structure as data doesn't fit into an RDBMS database without some work. I've seen people try to map both XML and regular DB's onto each other, and my opinion is that the results don't "feel right" on one side or the other unless great pains are made to preserve the structure of the XML doc in the DB schema.
That said, I'm not sure a hierarchial DB will necessarialy be any better than something like an OODBMS with well-modeled objects.
Dojo: defanging browsers so you don't have to
It's my choice for DBs for my website since my website hosting co doesnt provide MySql or any DBs. It has all the XML modules you could use to use XML as database..... it's conveninece. It's very portable. And it's easy to read.
I don't think that heirarchical db's have any real chance of taking over or replacing relational dbs in the future. There may start to be more of a place for them, but many application service providers that use XML still have a fair amount of relational data that needs to be maintained. XML is mainly being used for communication protocals and not so much for internal data structure storage. I think the more likely db trend in the future will be for many users to maintain both relational and heirarchical databases..
WeFunk
With all this hype about XML and ubiquiteness of SQL, LDAP directories do not get the attention they deserve. How many of you have installed SQL-based authentication at your site, just to find out how limited a solution it is (maintain more than one database for all kinds of authentications, do you?). Not only does LDAP allow for a flexible hierarchical directory, it's also a standardized Internet protocol whereas SQL isn't. With LDAP, many applications work out of the box because it's a standard. Oh yeah, there's also the OSS server available at openldap.org.
IMHO, XML documents should be handled in both hierarchical and relational ways: the first for an efficient long-term storage and the other for transactions.
The relational model has no major shortcomings. The only thing XML offers that is not already very well done is easier data interchange. As a database administrator, I can tell you there is NO chance XML will dictate a change of how we store data. There are much higher priorities in database management than easier data interchange.
Relational databases didn't come to dominate the database market because they pushed aside equally valid alternatives, they dominate the market because relational databases implement relational calculus. Indeed, that's the very touchstone that distinguishes relational databases from something like DBM and its many descendants.
And *that* is important because it assures the desiger and user that every possible operation is well-defined and (hopefully) correctly implemented. The exact syntax for a "join" may differ, and a specific implementation may be flawed, but everyone agrees to a common baseline.
For hierarchial databases to really take off, they need to have an equally strong mathematical underpinning. For now, AFAIK, there is none other than that you get when you map a hierarchial database into relational tables and use exactly those relational properties. That's a good start, but if you're only using the properties in relational databases, why not stick with them?
As for XML, that's completely irrelevant. It's a good format for transferring data, but that's about it. You can store hierarchial data in an XML file, but you can also use it to store purely relational data or completely unstructured data (in some CDATA block).
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Organisation of containers are not what organisation of data is. Think of it this way, you can store all objects derived from FooObject if you have a node structure FooObject *Item, yet nodes themselves could be stored in a linked list (compare with a simple db table), or in a graph (cw a relational database with many to many relationships) or in a tree (cw a hierarchical database.) The difference between relational and object oriented databases are basicaly what type of things they can store. OO ones can store whole objects, while relational ones store fields in tables. Those fields are usually simple data types. I can't think of a reason that an OO database can not also a be hierarchical database, yet that does not have to be the case either.
Gentlemen, you can't fight in here, this is the War Room!
Hi,
I wrote a paper on native XML databases and SQL databases that support XML that appeared on Slashdot a little while ago. While doing research for that paper I asked myself the same question, whether instead of coming up with hybrid methods to store relational and hierarchical data we should store XML in already existing hierarchical databases. Unfortunately things are not so clear cut.
First of all, a lot of data out there is relational and people aren't ready or willing to transition all that data to XML based storage so mixing of relational and XML data will probably be with us for a while. The biggest problem with object oriented databases is that they didn't understand this fundamental issue but it seems that with XMKL databases the vendors understand that hybrid data will be with us for quite a while which is why Tamino supports importing data from relational sources and even ships with a SQL engine.
Secondly, XML documents have a lot of metadata beyond the hierarchical parent-child relationships such as processing instructions, comments and entities which are require more intelligence in the support from the database than just storing parent-child relationships.
Finally all the major [commercial] relational database vendors have included some sort of native suppport for XML including XML types and there is a an ANSI standard in the works for combining XML and SQL. From what I've seen, none of the hierarchical databases plan to support XML as much as the relational databases have or plan to.
Now if you were simply asking whether a native XML database can be built on top of a hierarchical database then I believe the answer is yes. Then again native XML databases can and have been built on object oriented databases and relational databses so it makes sense that they can be implemented in a database system that is more suited to handling hierarchical data.
It certainly seems like the same thing is happening with XML that happens with any new toy: "my friend told me XML was cool for stuff, so I'm going to convert everything to XML so I can be cool too."
I was pretty sure that XML was useful in that it was a human-readable data-encoding mechanism that "average" users could get a grip on and utilize in sharing information between heterogenous systems, but it seems like people are completely missing the point these days in how to use XML effectively.
A lot of the benefit of using XML is quickly becoming negated by everyone coming up with their own DTDs and the lack of standard formats for encoding data that is to be shared. As an example, here at the university I attend, there is a project for sharing information about biological species' population data amongst sister organizations. The goal is make the information possessed by all these organizations available to all the others. The trouble is that they have all come up with their own format for storing the data they collect and can not agree on what standard should be used, so each organization is encoding all their information with a different XML labeling scheme. My first questions was: "Why in the heck are you using XML to encode the data anyway?" Seems easier and saner to just store it in your relational database and make the database accessible to sister organization who can then encode the information however they want for their end-users through their client applications rather than the organization holding the information imposing order on people wanting access to the information.
To make a long story short, XML encoding doesn't help you store the information more efficiently at all and with the state of the "formatting standards" today doesn't even really provide an efficient way of sharing information between organization or an efficient way of encoding the information for transmittal to other organizations. It seems as if people are missing the forest for the trees in how XML can be useful in its relation to data encoding and we should stick with our trusty ole relational and object-oriented database models as they have shown their usefulness and efficiency.
Or is it better simply to use a relational database that can output in XML, or script your way to achieve the same goal?"
I believe that RDBMS's should add functionality to read/write XML, especially as the XML Schema recommendations is basically done.
The idea that XML should be the permanent storage format is a bad one. There is a lot of power in a normalized data model -- it enforces data integrity , while eliminating data fragmentation automatically and it minimizes transaction resources.
Consider XML representations for different entities that all share some kind of child entity. For example: people, businesses, and schools all share addresses. In XML, you want the addresses to appear in the description of the individual object. Does that mean you want to store the addresses separately that way? Absolutely not, because then when you enforce constraints or ask questions about addresses, your data is fragmented in three places. For that matter, how do you know all the entities that might use addresses? In an RDBMS, you can inspect all the foreign keys to the address entitity. What's the XML analog?
I work for a company that has been doing hierarchical DBMSs for years. The company is Applied Technical Systems. We make a database engine called CCM.
XML is a great way for exchanging data, but the term XML databases is very misleading. If the database engine actually stores data in native XML, it's going to be *very* slow. I think the point behind XML is that nobody should really have to care what your backend is as long as you can export reasonable XML. Note that I say reasonable XML. And XML export that simple encodes the rows and fields in a table to XML with <row> and <col> tags is NOT reasonable. It conveys no actual knowledge of the real structure of the data.
Storing XML data in a relation DB can either be a very hard problem or a very easy one. Let me explain.You could look at some XML and define a DB schema for it, not too hard to do. Problem? It's not generic; a human has to re do it each time the XML structure changes. The alternative is to store it all in one big table and index the hell out of it. Problem? It's slow. At that point you aren't using any structure of the XML or the power of relational DBs.
I'm a firm believer that efficient XML storage, querying and retrieval will require a hierarchical database. The problem is that there's several features (bugs IMHO) in XML (and XPath) that, in a way, are throwbacks to relational DBs. IDREFs and the notion of document order particularly bug me. I ran into these this summer when I was on a team trying to build a XPath and XQuery front end for CCM.
We're gradually seeing the XML world change. Early XML documents were similar to the type mentioned above. They were flat. When you start adding depth the information inherent in the structure of the data becomes apparent. Another thing I'm glad to see the industry move away from is the notion that XML resides in files. Many (if not all) of the early XML parsers made this assumption. It was a pain in the ass to parse from some other source, like a buffer in memory.
Strictly speaking the relational model doesn't specify how data is stored, only how it is retrieved. XML is a storage format; there is no reason why a relational database couldn't use XML for storage.
XML is not a magic bullet. Relational database won out over the Hierarchical model for a lot of reasons. For instance, there exists a number of integrity constraints with the Hierarchical model such as
1) No record occurrences except root records can exist without being related to a parent record occurrence. This means that
a) a child record cannot be inserted unless it is linked to a parent record.
b) a child record may be deleted independently of its parent however, deletion of the parent record automatically results in the deletion of all its child and descendent records.
c) the above rules do not apply to virtual child records and virtual parent records.
2) If a child record has 2 or more parent records from the SAME record type, the child record must be duplicated once under each parent record.
3) A child record having 2 or more parent records of DIFFERENT record types can do so only by having at most 1 real parent, with all the others represented as virtual parents. IMS limites the number of virtual parents to 1.
In addition to these flaws, relational databases have had over a decade to become mature, optimized, and enterprise scalable. Harddrive partitioning for such databases as oracle work out perfectly with the cylinder, sector, and tracks of a hard drive to allow for the fastest read/write times as can be possible.
Too often people see that XML "can" do so many things and decides that it should be the way things are done but XML is NOT a magic bullet and just because it has the potential to do something does not make it the best methodology for doing so.
I really wish people would stop focusing on the INTERCHANGE format and focus on the abstract implementation details.
:P
Just about any heirachial store CAN be implemented in a relational database -- they are called "intersection entities".
Trivial and fast (when indexed) to Manage one-to-many and many-to-many relationships.
Complete with constraint checks if you so desire.
The greatly exaggerated demise of ODBMS should point out the problem of adoption: What problem does this solve that I cannot solve using what I already know?
or to parody Dr. Ian Malcolm in Jurrasic Park
"you were so busy using BLOBS in relational databases, you didn't stop to consider whether you SHOULD"
Old age and treachery almost always overcome youth and skill.
I would suspect that companies/people who run
Unix would like that faster chip, as Unix is quite
portable. I have 2 Alphas, 3 PCs, a NeXT, and my
laptop is an iBook, all of them running Unix.
At work, I manage various flavors of Unix, many on
non-x86 hardware. But I digress..
For every problem, there is at least one solution that is simple, neat, and wrong.
XML has several standard hyperlinking paradigms (ID/IDREF, XLink, HyTime, TEI) which allow for the creation of non-hierarchical relationships.
Also, I don't like hearing so many people talk about using DB technology to hold XML data. Unless you are talking about a document management system, you really ought to be thinking of XML as an interchange format only.
In terms of the ability to represent information, XML tries to solve much of the same problem as a database: providing a framework for arbitrary structure. A database does it in a way that is highly optimized for query and modification speed, XML tries to do it in a way that is optimized for interchange and platform-independent processing.
Storing XML fragments in database fields is an odd thing to do, but I see more and more people doing it. I guess in an ideal world, your database schema would go down to the exact level of granularity you might be using the XML to capture. It seems half-assed to me to use a DB for high-level structure, then inside your records you have some other completely different type of structure. I guess people like this, though, since the DB vendors have all added technology to enable this.
To me it just means that people spend less time thinking about and designing the information models, and it is yet another case of software features shaping requirements (when of course we all know it should be vice versa)
I was surprised to see so many questions of the form "what's wrong with relational databases"? Relational databases have a well-known problem called "impedance mismatch" when mapping multi-linked object structures. Many links on the impedance mismatch issue can be found at this Google search.
Anyone who has tried to take a natural set of application-side objects and map them onto a relational database is already quite familiar with the problems created by the proliferation of tables needed to map simple application data structures, as well as the large amount of development effort needed to deal with simple relationships that would be trivial to specify in an object model such as Java's or XML's.
There is clearly a need to move on to object databases, but installed base and skill set inertia have blocked this transition, with the result that database-oriented applications have remained hamstrung in their friendliness and feature set.
Tim
I'm rather ignorant on this subject, but surely XML data could be viewed as being object oriented? If this is the case, then surely an OODBMS or more practically an Object-Relational DBMS could be used.
In case you were wondering if there are any out there, check out PostgreSQL which is way cooler than MySQL (it's open-source for a start)
Nevrar
Having just quit a pure XML company (can't say the name of the company, but lets just say I can now spy on the company instead of working there) I have to say that pure XML databases will most likely not pick up.
The reason why they will not pick up is not because they are not good in their own domain, but simply because the legacy of SQL is simply too large. To make XML do what SQL does today is about eight years away.
In those eight years SQL data will become huge and the problem will be converting the data. For example if you have a multi-terrabyte database how can you ensure there are no errors in transferrring. Hard disks have an error rate that works one in a billion. Now put that on a multi-terrabyte database that means a megabyte of data may be faulty at the best. This means that somebody will have a screwed up account.
This means the best solution is status-quo since the status quo works and does the job correctly.
I even predict that in ten years the "programmer" will almost cease to exist. In ten years we will become data mining extractors. Sure there will still be a task of extracting data using programs, but the main concern will be managing the data and figuring out interesting things from it.
What all this means is that we will live for a long time to come with SQL. Ok there may be XML adaptors, but it will still be SQL...
"You can't make a race horse of a pig"
"No," said Samuel, "but you can make very fast pig"
Having just taken a database course a year ago in which we had to deal with a hierarchical database. It is absolutely awful.
The copper bosses killed you, Joe. 'I never died', said he.
Zope uses an object database known as the ZODB. Some forms of many-to-many relationsships and such can be handled via the use of selection and multi-selection properties, which are designed to distinguish between a selected element and the list of available elements. The list of elements can be derived from a property on the current object, a property on a parent object, or be created via a method call - allowing for non-traditional (for OODBMS) cross-linking of objects. Of course, since this sort of thing is a workaround, no true relational links are created... 'Soft Relations' may be ok for MySQL, but in big application development, relationships must be enforced! Thus, the big-boys in RDBMS all enforce foreign keys (mysql does not)...
Of course, I've found that by careful creation of object heirarcies, very complex applications can be created on top of a OODBMS that are in fact more robust, in some ways, then the relational couterparts. The Bigest hurdle (Short-term) I see to OODBMS (including ones based upon XML [the ZODB can export objects as XML but they are stored differently internally]) is the lack of a true query and data manipulation language - like SQL. Sure, OQL exists, and is even technically a standard, but it A) sucks and B) is geared towards large java applications with huge amounts of active objects, not general purpose OODB queries. Thus, without such language, OODBMS are all disimilar in how one queries and creates/updates data, and in many cases, the only interface is a truely procedural one! Thus OODBMS are forced to use proprietary tools, and are locked into one system - not to mention speed of development (something normally associated with OO development and OODBMS in general) is hindered by the excessive amount of procedural calls one needs to simply query thier data...
Recently, an add-on to Zope addressed some of these issues. Called 'ZOQL' - it uses a SQL like syntax and allows for very discrete querying of the ZODB (something one had to do programatically using the 'ZCatalog' before) with all of the familar aggregate and comparison operators SQL users love... Of course, this _still_ doesn't address the issue of soft-relationships:
I think the bigest hurdle to OODBMS in the long term (tools like ZOQL are interfaces to existing systems, thus can be mplemented easily) is the lack of handling relationships. It seems that most RDBMS force a developer to think in Relational terms about the data, and most OODBMS force you to think in terms of objects... Most problems can be mapped to either of these domains, but you are forcing the data-model-type onto the problem. What is needed is a hybrid system, an 'Object-Relational' DBMS. This is to say that OODBMS system makers desist with the traditional OO idea that relations are of the following types:
- Object A is a Object B
- Object A Has a/many Object B(s)
What RDBMS systems excelled in (and thus fell into pupular use for) was ease of management and allowing common data to be moved and grouped. A 'Look-up Table' - for instance, which simply holds a list of common data (an enumerated list) and can be centrally maintained is a Boon in the RDBMS world. For example, you have a lookup table of car manufactureres, and one of them changes its name... Instead of updating all N Cars that are made by the manufacturer, you simply update the single record in lookup table. Since each car would have somehting akin to a 'Manuafactuer_ID' column linking it to the lookup table, the Cars belonging to the manufacturer are all taken care of.How does one do this in a hierarchal system? Well, the easy answer would be that each manufacturer object contains all the cars that manufacturer makes. Simple, right? WRONG. Why?
Because each car also has a body-type (compact, sedan, SUV, truck, van, etc...) - which in a relational database would simple by another lookup table, but in an OODBMS poses data management issues. Do we put body-type higher then manufacturer? If so, then we have to maintain the list of manufacturers for each body type, causing headaches. Or do we put body-type below manufacturer, causing us to need to maintain a seperate list of body types for each manufacturer - these lists of course need to match exactly if we ever plan on being able to search or do reports based upon all cars of a specific body type.
Sadly enough, this sort of seperate-enumeration-relationship isn't implemented (well) in any OODBMS I've found.
Take the ZOBD for example, its selection and multiselection lists Try to handle this situation, but fail because relational integrety is not maintained! That is to say, behind the scenes it's not a true reference to a value in the enumerated list, but just a text entry representing a value in the list. If the value in the list changes, the selection-property does not update, leaving you with the equivilent of MySQL's bastard-children, the orphaned records.
This sort of soft-relationship handling is Ugly and BAD for maintainaility, but OODBMS users are faced with two ugly choices each time they map such a relationship: Do I store this as a plain-text property and just update N records each time this changes, or do I map it into the hierarchy and deal with the headaches incurred by doing so...?
I don't think I've answered the question, but hopefully I've at least shed some light on the subject for members of both the OODBMS camps and RDBMS camps... Now if only a useful ORDBMS were to come along...
(Note that PostgreSQL and some other RDBMS actualy can be used in a semi-OO manner, but this is usually reserved for inheritable structures of data to be used for specific extensions to the data model - thus the SUV table inherits from the Cars table and adds some columns - but all other relationships SUV has will still be relational)
man is machine
While a hierarchy is often used by humans to organize and structure things, that should in no way impact how the data/information/objects are treated as individuals. Look at the common file system hierarchy and it's easy to see that burying files under a hierarchy of directories actually makes access to that information harder. It wasn't so noticeable when we were all just managing a few MB of files, but now people are beginning to store large picture, movie, and sound libraries. File managers have mistakenly stuck with the hierarchy instead of using information associated with the file itself (ID3 tags, etc.) to organize it all. What is really needed is a better approach to representing metadata so that information can be accessed directly based on those metadata attributes and not have it hidden in the hierarchy. I have a short essay on this from the work I've been doing on a Meta Object Manager (MOM), but it needs to be cleaned up before it could be published.
The desire to impose a hierarchy on the data itself instead of considering a hierarchy as simply one view on the data is a step backwards. Nobody who manages large amounts of data is looking to jam it into a static hierarchy, and so XML is not an answer, nor is any hierarchical representation.
I use Intersystems' Cache at work - under the hood it's hierarchical, so it would seem to be a good fit for XML (I hear more XML stuff is coming for version 4.2), but it also projects everything as relational tables through ODBC, and simultaneously as objects, through ActiveX and Java. (They're dropping CORBA, not enough interest apparently.) I find it a little tough to program natively in, but it's gotten a lot better with version 4.x. And it runs quite well under Linux. That's the platform I use :-)
In design, the logical construction of the program and its data structures should be relatively independent of the physical implementation of said.
Basically, as I read your question, you are using a logical design that is hierarchical (an object structure experessed in XML) and wondering if it would not make more sense to store it in a hierarchical database. Maybe.
However, relational databases form the current state of the art and have been highly optimized such that any theoretical performance gains from better matching of logical structure to physical lay-out in the database are likely outweighed. More generally, by insisting on a match between logical and physical lay-out, you would potentially be limiting yourself to a specific physical implementation, one that may not provide good performance relative to others.
A better solution to your problem might be something referred to as a persistence layer. This adds another layer of abstraction to your application, in the form of a mapping, between your logical design and your actual physical mode of storage. There now exist publically available free (as in beer, and in some cases open-source) tools that will automate this mapping. Generally, any performance hit from the abstraction should be made up in the speed of the superior physical implementation, and the freedom to switch later is also important.
Two that exist for java are castor available from exolab and a pilot implementation for Sun's emerging Java Data Objects standard (see http://java.sun.com for that tool).
First, hierarchical databases weaknesses are not limited to many-to-many relationship modeling. The simple fact is that some data is better represented in a hierarchical fashion (a directory service IS a hierarchical database) and others in a relational fashion. XML is a tool for exposing that data to external sources regardless of its internal representation(s).
The enXyme project attempts to map XML data onto a relational database schema. The goal is to allow complex, specific queries of XML data. It's not easy to capture ALL of XML, with all its possibilities, but you can do parts of it. The project already has a basic XML schema parser, a script that takes an XML schema and generates a series of sql CREATE statements that reflect the hierarchy described by the schema.
I guess a pure XML database like the ones mentioned in the article would be better at this, but the advantage is that relational dbs are already in wide use.
Structured data. Structured searching. The Enzyme Project
XML is a format. Using it just because you can (which unfortunately represents most of its current usage) is stupid. Examples:
The list could go on and on and on. Just remember: use each tool when it's needed. Don't just put XML in there because it's a nice buzzword, mkay?
PS: on the subject of the article... Again, I fail to see where XML comes in. Hierarchical DBs have been around for a long while, and even if they've been shadowed by the RDBMS' it doesn't mean they're dead (I'm working on a project which uses a hierarchical database simulated on top of BerkeleyDB. Works wonderful, and there's no need for SQL. Or XML, for that matter).
Petru
I think what this really points to is that XML is not the end-all, be-all of formats for exchanging data. All that XML does is bring structured data formats to where databases were in the late 70s (when the Relational model began to take off). Just as databases took the path from flat files to indexed files, to heirarchical databses, to relational database; XML is just the next step in the path from fixed with columns, to delimited values, to key/value pairs, to XML. We need at least one more revolutionary change before the transmission structures (XML) catch up to the live datastores (Relational Databases).
Why? Because the relational component still allows set-at-a-time interaction for efficient querying, but path expressions can also be used to navigate through the nested structure.
IMS, IBM's hierarchical database system, was originally developed for the Apollo space program. It is far from dead as you can read here. IMS systems are currently storing around 15000 petabytes of data, and executing 50 billion transactions/day. All stored in a hierarchical database system.
The client is the one that needs the information in easier to read form, not the server.
Relational databases are fast, dependable, and are well proven. The data storage that they provide is very adequate for most situations, if not all.
What really needs to happen is SQL needs to evolve to include XML, and better support for some other situations that constantly arise when writing complex queries.
I wish that the top clause could be used on a join stament.
left join top 1 some_table st ( ot.pk = st.fk ) with order by st.some_column.
Or that there was some syntax for querying out trees of data. Or there was a way to embed code in the joins that could analyze what had been joined at each loop. The data is stored fine in a relational database, sometimes SQL doesn't have the power to extract in a simple manner. I'm sure you can think of many other things that SQL could do a lot better.
Looking at one to many or many to many relationships in the standard two dimensional data set is a pain. Especially when you have several joins that can duplicate data. Cold fusion ( I know it sucks ) has a feature that allows you to query a result set, which makes extracting the data you need a little eaiser, but XML would make this a mute point.
Microsoft is developing an XML extension that will return the data in XML for SQL server. The bigger issue is that ODBC needs to do it before they (MS) do so that we don't get stuck using Microsoft XML over OLE DB, which means no Linux support. You can read about MS solution at http://www.microsfot.com/sql.
Looking at MS solution, it is still too complicated, because the SQL language does not allow you to direct the XML output. As well if foreign keys are setup it shouldn't need them, as well the join statements should be more then adequate to describe how the XML should be formated.
Another aspect is update/insert/bulk inserts. This could make doing updates to linked tables easier. As well as inserting complex structures of data. Also, it could be used to check data before it got inserted into the db and failed.
Data transfers would be much nicer as well. Taking what is one to many relationships in you database, and trying to make them horizontal instead of vertical is a real pain in the ass. It is also much slower usually because of the extra joining needed.
OLAP applications could stand to benifits as well. OLAP produces data in with multiple dimensions. Usually this is accomplished by
having binary data formats that are specific to vendors, now multi-dimensional data can be treated the same as any other result set.
I think we are a good year or two away from there being good XML support for databases, but it will increase application performance as front end guys and gals can issue less queries to the database to get the data they need, a in more logical fashion as well.
Anyone can explain to me what is suddenly so wrong about relational database with hierarchical indexing?
/.) there is a correct answer.
Maybe its just me, but the goal today is integration and having a special database for XML and special database for this and that just because its faster for this particular problem creates such a level of complexity, which prevents accomplishing even of the most trivial tasks.
Forgive me for tooting my own horn on this one, but I believe that (for once on
I summarize the answer in a paper written for VLDB 2001 (www.vldb.org). The paper presents joint work between Stanford, Berkeley, and RightOrder, Inc. It can be found online here (in PDF).
What we found is that relational systems, with appropriate indexes for XML data, give the advantages of both worlds. XML is a hierarchical representation in only the loosest sense. It's written linearly in a flat text document, just as a child learns to write things down on a piece of paper. However, you wouldn't convince anyone but that same child that something written on paper can only represent two-dimensional objects just because the paper itself is flat. XML in many variants is plainly richer in concept than its simple hierarchical representation and thus quite suited to ER. I believe a previous poster mention RDF... a perfect example.
Punchline: XML is neat, XML is tasty, but XML is not inherently more or less expressive than ER; it just requires a little critical thinking (and index tweaking) to tune ER engines to deal with it. (Once tuned, the ER engines dominate all others in performance.)
Taking out attributes with multiple values and putting them into a linked table is core to the functionality of relational databases.
Customer--------
CustomerID
FirstName
LastName
PhoneNumber
-----------
PhoneNumberID
PhoneNumberTypeID
CustomerID
Text
This kind of relation is basic functionality in relational databases. This ain't no kludge.
Hierarchal databases have so many limitations. Even simple things, like employee lists, suffer under the restrictions of a hierarchal database. Employees that work in multiple departments, or have multiple supervisors. Employees with multiple spouses (think International). Projects with three leads. Employees working on multiple projects.
Relational databases were created for a reason. Abandoning all those improvements just to fit more cleanly into the XML hierarchal model is ludicrous to me.
Raven
"I will trust Google to 'do no evil' until the founders no longer run it." Hello Alphabet.
Some time ago I came across the following letter by Henry baker regarding relational databases. It made interesting reading.
Perhaps there is an element of 'when your only tool is a hammer, everything is a nail.' to relational databases. They are certainly so pervasive now that any idea of using something different would be seen as taking a HUGE risk.
I have pulled a great deal of hair out at work because a client insisted that we used an XML based object database (eXcelon).
While the concept is incredibly powerful, the technology is still to young. The moment you try and do anything important, like an aggregation for example, life becomes very difficult.
I've noticed that there's a lot of cool new stuff being proposed for XSLT2. Maybe that'll help the market along a bit. As far as modeling the data is concerned, it really does kick butt over relational for many (but not all) applications - but it's simply not ready to hit the masses yet.
most business tend to use the relational dbs because necessary speed. I think most business will stick to that to a degree.
Let us throw something really nasty into this.
For some reason this has been ignored in the obvious "popular" discussions.
Neither RDBMS's or HDBM's can understandably respresent all data structures efficiently.
The "best" I have found are Entity-Relationship Databases. No, I do not mean those ER diagrams that most DB packages and modeling tools deal with.
Imagine a world (DBMS) where every "Entity" is allowed to be Related to any other "Entity" without regard for its data contents (that may be the origin of the link but afterwards the linkage can remain, if wanted, even if data changes on either side).
Also imagine that the Relationship itself can have any arbitrary data associated with it (including the aforementioned original data that caused the relationship to be linked).
Now imagine that the Relationship is NOT 1-to-1. It can be allowed to be any Relationship between any N Entities of any (probably) predefined types.
Now imagine that a Relationship is allowed to be an Entity itself.
Actually pure Enitities that do not describe Relationships between other Entities probably do not exist in real life - A person can be related to another person, but in special cases the person is the relationship: A legal father and mother of a child may have not relationship to each other except as parents of that child.
The other forms of DB's (Network, Relational, Heirarchical) are just efficiency-motivated reductions or special cases of the above.
The overhead in maintaining the arbitrary links, extra tables, keys,and joins, is the real reason the E-R model is not a popular implementation. However it can be easily implemented with a Relational database. A big advantage is that the data in the DB can change without necessarily changing the logical data-interrelationships.
Once you analysed things this way you stop having to have database theorists deal with normal ever-day applications.
http://www.zope.org
ParsedXML
For more complex hierarchical relationships, an object database is more apt, or an XML translation kit for your relational DBs.
Horses for courses.
LDAP is an example of a really good and well developed implimentation of the hierarchial database idea. However, try keeping track of whay your customers bought from who with LDAP. So while LDAP (and other hierarchical dbs)do certain things better, don't try to run a CRM suite off one.
The basic problem is that the entire database is rarely hierarchical in nature even though some queries may be.
LedgerSMB: Open source Accounting/ERP
Relational databases are here to stay and will be with us for at least the next fifty years. It is better to think of ways of translating relational data than supplanting it.
With the single exception of potentially higher speed, which also requires all hierarchical access code to follow the structure carefully, relational still has facilities untouchable by other structures.
The commonest strength is not simply the way relational handles many-to-many, but the simple extension of recursive relationships in which relational is one of the few paradigms with this ability.
Relational also supports the network model, again another major weak area in the heirarchical model. I've yet to come across a potential database structure that I can't handle relationally; that is absolutely not true for every other paradigm.
Don't get it wrong
The whole point with XML is interoperability . When designing a distributed application, one could choose from:
IIOP/CORBA (or variants like Java/RMI, etc.): theoretically very good interoperability, but practically almost none when you use tools from different vendors
design a custom binary protocol. Most of these designs are short-sighted - making them extensible is not an easy task. Moreover, most of them adopt an ostrichish attitude towards both byte ordering and character encodings
use XML: built-in extensibility, extremely good interoperability & support for different encodings. Certainly more of a bandwidth hog than the previous two, however, as we all know, the battle between bandwidth and processing power was won by bandwidth. When bandwidth is the bottleneck, one could still use standard HTTP compression to alleviate the problem.
...
XML-RPC is nothing but interoperability taken to the extreme: XML is the packaging format, mostly because any architecture/os can properly decode it, and HTTP is the carrier, because it can pass all proxies, and, again, almost everything understands HTTP
As far as configuration files are concerned, one extra reason for having XML as format is that it's cheaper & faster to use a DOM parser than to write your own
The Raven.
The Raven
Not saying that I agree with turning databases into XML, but it's really fun to write these xml queries. You can check it out here
Okay, I've worked for two different network model database companies -- the network database model is just an extension of the network model to allow graph schemas instead of a strict hierarchy. I've also worked with two companies that we mapping hierarchical structures onto relational databases.
You can think of data structures as (leaving ternary relationships and such aside) some sort of network of relationships. When you think of it this way, relational and network model databases have more similarities than they have differences, especially when you consider that using surrogate keys is the moral equivalent of a network model "pointer".
Okay so you have this network of relationships, mapping a hierarchical structure onto that is simply picking a starting point and traversing the structure from that "viewpoint" without visiting a node via the same relationship twice (simplified algorithm but...) One of these groups used to think about this like you had a multi-legged turkey. You grab one leg and hold it up. All the other legs hang down -- you grab another leg and a different set of legs hang down.
So, if you buy that, does it really make sense to represent any sort of network of information in a hierarchical form? Well, yes and no. It makes sense from a presentation and maybe interchange perspective but not from a native storage perspective. It's simply to constrictive and you and up representing relationships that don't fit into a neat hierarchy programmatically in the application code instead of explicitly in the database schema. 25 years from now, someone is trying to reverse engineer your code and figure out how all this data is related -- blech. Ever wonder why IMS application are generally left alone and newer applications are not usually written to IMS. This is part of the reason why. (yes there are some but they are the exception).
Throw in to this my experience working with a bank that had hierarchical data and the extent to which they went to circumvent that restriction, and I'd say that native hierarchical storage for XML is a bad idea. Granted it's tempting but it seems ill advised since it's very likely that your data will survive long beyond the lifecycle of the system used to originally store it.
<RANT>
The original question didn't provoke this but I've seen a couple of responses about using XML as a native data storage format. Let me say that, unless the data is very static, it's a monumentally stupid idea to do that. XML is not a replacement for a database.
I find that most of the people who really want to do this are ignorant of all the work that goes into real database systems. They don't understand lock management, transactions, rollback and recovery, free space management nor the scalability issue that real databases take care of under the covers. If you feel tempted read this
You throw this plus the representation of non-hierarchical relationship with IDs and sooner or later you will find yourself in a text editor tracking down ID/IDREF pairs to find out where your data is corrupted. Or writing scripts to validate your "entire data set" -- above a few megabytes it can be really painful.
For God's sake, expect to use XML to store data that you are going to update with any regularity.
</RANT>
You are not a beautiful or unique snowflake -- but you could be if you got off your ass.
Just use NDS.
Its simple use NDS as a binding agent between various data storage mechanisms.
As well as been scalable, fast and bullet proof, it has the added the added advantage of built in security at every level.
----
Modern definition of an expert: Someone who comes from far away with a powerpoint presentation.
The reason relational databases are so pervasive is that they work so well. Having worked with both, the flexibility of the RDB is what is so powerful. There is a book out on the remainder piles that is a serious attempt to create the 'object oriented' relational database by CJ Date. One of the big failings of current computer development is the insistence of developers to use hierarchical data structures for everything from oo class systems to file directories. Hierarchical might be quick and easy for getting something going, the long term restrictions and complexities it introduces are never ending.
Microsoft - Where would you like to go today, Maybe Jail?
LDAP/X.500 heirarchical databases are all well and good, until you want to run a query that asks which customers have what services, especially when everything is keyed on phone number. You know what we ended up doing? Pulling the entire database out of LDAP every night, and putting it into Oracle to run the reports. Nothing sucks more than a full table scan in an X.500 database.
I do agree that heirarchical databases are great where you are only going to access the data from a single key, like passwords and email addresses... But, they should probably be provisioned from an external RDBMS if you are looking to do reporting.
Jason PollockXML may be hierarchical but the data it is used to markup is not necessarily hierarchical. For instance, XML can be used to markup conventional fielded (flat file) data to serve as an interchange format.
More importantly, XML is used to impose some structure on inherently unstructured text. The structure it provides is based on some assumptions of how the data will be used or how it will be presented. If the data is used in some otherway, the markup can be useless.
An example is a book. For XML purposes, it can be described as structured by chapter, section, subsection, and paragraph. For information purposes, tags are assigned to represent the ideas, terminology, names and other index-like content. There is virtually no structure in these index type of tags but they convey the most important information in the book.
Or not. These tags are assigned based on assumptions about what readers are interested in. A different set of assumptions would produce a different set of tags even thought the structure of the document would stay the same. If the sentences and paragraphs are shuffled and exerpted for some other publication, even the structure becomes irrelevant.
How this inherently unstructured information is stored is relevant to how it is managed, that is, how it is backed up, how access is controled, how changes are tracked. However, when it comes to putting the information to some useful purpose, it is the retrieval mechanisms that are important. The issues here are how easily the user can specify the type of information he wants and how accurately the mechanism can find it. This process is usually independent of the underlying structure and uses some higher level concepts of relevance and context.
The question of whether to use a hierarchical, relational or object-oriented data structures misses the point for textual data, for which XML is commonly used, because none of these structures capture meaning.
Topic maps make a heroic stab at capturing meaning in XML markup but still only within a set of assumption. I suspect a true meaning markup language is theoretically impossible, or at least theoretically very far in the future.
Contrary to some of the comments I've read here, LDAP isn't an implementation of a database, it is a *protocol* for accessing directories. LDAP data could be stored in anything - a hierarchical database, a relational database, an object database or a flat file. Let's not confuse the issue under discussion.
Great Windows SFTP Server!
PostgreSQL already has support for hierarchical data. I've messed with it a bit, and it's nice. Unfortunately, there's a few problems. Number one is that if you want to keep your project completely portable, you should probably store it in a relational format, just because more databases store that way.
Is there any work on mapping OO models to relational? Surely there exists some sort of mathematical relationship between the two.
I also had a problem with people stating that XML will never dictate the structure of the database backend. This is fairly naive. Object models often times are a more natural representation of the structure of a problem domain. So why try to squash that to a relational model if you lose information in the process?
Or if you're losing efficiency by converting back in forth? If you do it enough, it only makes sense to put it on a lower level. Either with a data binding framework like castor or in the database server itself. Fuck, if nothing else, it'd be a good reason to push/sell the newest version of a DB server .
Josh
Indeed, that's the very touchstone that distinguishes relational databases from something like DBM and its many descendants.
The alternative to relational databases is not "DBM", it is object oriented, tree structured, logical, and other kinds of database models. Those are just as well defined as relational databases.
And *that* is important because it assures the desiger and user that every possible operation is well-defined and (hopefully) correctly implemented. The exact syntax for a "join" may differ, and a specific implementation may be flawed, but everyone agrees to a common baseline.
Relational databases provide a common baseline for a primitive set of relational operations. Real-world implementations of those models have been augmented by zillions of operations that weren't part of the original relational model and that often don't even fit into the relational model. And without those extra operations, relational databases would not be useful in practice.
For now, AFAIK, there is none other than that you get when you map a hierarchial database into relational tables and use exactly those relational properties.
Are you kidding? It is a major pain trying to express hierarchical data in a relational database model: the relations that describe hierarchical data and the operations that you might want to execute often require complex, multiple, inefficient queries and updates, and the relational model provides few tools to ensure that the corresponding relations remain consistent.
The semantics of tree structures are trivial to define. People do it in programming language classes all the time. And it is trivial to formulate a database model corresponding to it. In fact, if you have an object-oriented database that respects language semantics, you get hierarchical databases automatically when you define an abstract tree datatype.
Still, so-called "relational" databases will continue to dominate the market for a long time to come. That's not because the relational model is particularly well-suited to a lot of applications. In part, that's because "relational databases" are not purely relational anymore: they generally include numerous facilities for object-oriented and hierarchical databases, under a "relational veneer". They even include the old "navigational" database systems, combined with the widespread use of stored procedures that do whatever they want whenever they want it on the database server.
In different words, traditionally relational databases will provide increasingly better support for hierarchical and object-oriented data, but they will continue to also support the relational model, as well as relational access to these other data types. And newly developed databases with other kinds of data models will provide an SQL or other relational frontend to their content. And marketing will continue to include "something-relational" in all the advertising because otherwise the old database hands won't buy it.
"Do you think a hierarchical database would really be a better answer for storing XML data over the existing relational counterparts?" This is like asking "would multi-gigabyte tape or sub-gigabyte cd be better for storing audio?" Depends on your needs - do you need random access or is brute capacity more important? In the case of XML, the answer comes from what you want to do after you've stored the data. XML can represent hierarchical data by nesting tags. It can represent relational data by matching ID and IDREF attributes (part of the XML 1.0 spec). Another issue is whether a schema or DTD is available. With schema information you can optimize you storage strategy whether relational or hierarchical. Of course you *can* store hierarchical data in an relational database with the hierarchy preserved as a parent/child relation. For the longest time (decades) relational DBs have had SQL as an effective powerful way to express a query. Now we have XQuery as a query language for XML. XQuery may come to rival SQL but the experts are still figuring out how to implement and optimize XQuery. Optimized implementations of SQL are commonplace and have had years of testing. So ... figure out what your goals are and go from there!
-rick
...most of the time. Unless something shows such a HUGE benefit from a technology standpoint that the business side cannot ignore it, commercial support will always win out. Just ask anyone trying to push Linux in corporate arenas right now (read:me). The relational database has IBM, Oracle, M$ and all manner of other flora and fauna behind it. It's not going anywhere. And, as someone further up noted, if you can't beat it, integrate it. Most of the major relational DBs have facilities which allow for use of models other than relational (OO, Hierarchical, even Navigational).
we speak the way we breathe --Fugazi
Yes, the "P" in LDAP stands for protocol. You use it to talk to an LDAP server. The LDAP server *is* a database so the points are still valid.
First off, one of the primary selling points of RDBMSes originally was a standard DML that was relatively easily learned. Anyone who has ever written any IMS DML knows what a pain going done multiple trees to get the correct result set can be in a "hierarchical" model. Plus the fun of possibly losing your place in the chain. Think multiple, interrelated linked lists and you start to get an idea of the pain involved. Of course, this doesn't mean that the average programmer writes good SQL.
Second, "hierarchal" vs. "relational" is a logical concept. People seem to get wrapped up in what is better way of storing things and not remembering that they are dealing with an abstraction. It is completely possibly to put a SQL front end on another style DBMS and force coders to think in sets (CA did this with IDMS 10 years ago). Conversely, you could store LDAP data in Oracle and you'd still think of it as a tree structure.
As far as XML goes, if I were only dealing with data that would be handled in XML documents, and there was a standard "XML database management system", then I would be inclined to use that XDMS. However, I suspect that the real world set of that intersection is rather small.
Relational databases have distinct advantages over other datbases that far outweigh any superficial affinty with XML
- you can design schemas where the data is always consistent (no insert, update, or delete anomolies)
- schema changes are easier than with other database types
- ad hoc querying is easier
better answer for storing XML data
So now my data has to bend to a buzzword? XML is a system for data representation, not storage, you idiot fuck
Okay, technical reasons are good, but they do not seem to rule the world all the time. I want to mention some purely political ideas.
1. To my point of view, it is relational model that dominates because of collective efforts of several big companies that managed to persuade everyone of its technical superiority, and then to deliver unparalleled implementations. (It was hard to develop a good RDBMS in those ancient times). So everybody with non-relational databases was claimed down and out and only permitted to silently die.
2. There exists a direct relationship between DTD and relational model (just map !elements into relational tables and impose constraints). The question is whether one cares to develop a query language which operates over XML primitives instead of relational ones so as it were more convenient for plain old hackers and newly brewed developers (and possibly housemaids). I personally do not believe in rapid creation of universally accepted such a language.
3. Everybody has become clever enough to develop different complicated ways of doing simple things. But not everybody is wise enough to make complicated things simple. And the XML is something overy bloated to me.
99+% of all corporate data that isn't in a flat-file or (possibly three-dimensional) spreadsheat is in relational tables. The typical task that XML has been designed for is to standardize data exchanges between differently-structured relational systems, by providing sets of tags specific to the standards of specific industries. The whole point of XML is to enable companies to continue to use their current investment in relational databases, without the drag of having to do custom data conversions when dealing with suppliers or distant divisions in the company.
... just about everything industry consists of. So if there's an impedence mismatch between relational and XML that's enough to make trouble, it's XML that should be replaced by another model.
If you're going to throw out the installed investment in relational databases, you might as well just design a common database standard per industry (rather than an XML data exchange standard) and let them all exchange native data rather than translating in and out of any exchange format. Obviously that won't happen.
Now, if you're a new firm, you might decide it's easier to go OO or heirarchical or keep your data in slips of paper in a shoe box. But most of the available tools and solutions will continue to respect that relational works real, real well for inventory, manufacturing, accounts
What design changes would be required to produce XML's relational equivalent?
"with their freedom lost all virtue lose" - Milton
Bottom line is that relational tables are good for data, but hierarchical is better for describing behaviors.
I use Delphi in my daily work, and we can use its TClientDataSet components family both to:
1- Create a cliente-side cached data-packet in client-server/multi-tier environments.
2- Use "briefcase model" functionality (copy data to your notebook in the morning, go do your business and merge them back with the database in the end of the day).
3- Create small, single-user applications that stores data in XML format.
I mean, altough its obvious hierarchicals capabilities, XML is well adapted to relational stuff too.
Hierarchical or multidimensional databases have been around for a while. There are several pros and cons for both relational and multidim, but one can never replace the other. The performance of one is better for certain types of analysis than the other and vice versa. Check out the OLAP marketplace to see what multidim databases look like.
I programmed for about 14 years for a electric utility that used DMSII on a Unisys A-Series. DMSII is a heirarchical database. One of the things that I miss using Oracle now is the ability to do a "SELECT LAST ..." which was used to return the last record of a select. In Oracle you must do a normal select an retrieve the last record first by using the appropriate ORDER BY. The select still builds a result set of all of the records instead of returning only one record. If you are doing alot of these (one for each account, for example) and there are many rows which potentially could be returned, doing a SELECT LAST can be a real time saver.
Heirarchical databases can be as complex as any relational db; you may need to do a couple of selects instead of one on a relational DB. I believe that the heirarchical model is still useful.
I don't think XML by itself carries enough metadata to understand much beyond whether a document is valid or not. I think RDF and RDFS have a big role to play in getting XML database ready.
Perhaps hopping on the XML database bandwagon before RDF technologies mature could be a mistake. Forget the semantic web, I want to see the sematic database.
W3 RDF
A Good RDF resource
It's easy to dismiss a new database technology as irrelevant because of the dominance of the RDBMS, but you should really learn more about it and when it is appropriate and when it's not. It's not going to replace relational, and isn't intended to. Here's a few links where you can learn more beyond what's available on Ronald Bourret's site mentioned in the original post.
The XML:DB Initiative
The dbXML Project (open source native XML database) Soon to become an Apache XML project named Xindice
eXist (another open source native XML database)
My blog on the subject.
Kimbro Staken
Relational databases are here to stay, I'm afraid, the relational databases that are available are incredibly well developed and used for just about everything, I doubt anything will change that anytime soon.
I'm not sure why anyone would want to go back to the old model, but just the same, they will never become more popular than their relational counterparts.
A bit surprised to hear that 'Hierarchical databases were blown away by relational versions' - since I'm pretty sure they've been paying my pay check for the last three years... :-)
There are a large number of heirarchical databases out there. The big fellas are the X500 directories (X509 certs came out of this work). More common are X500's demented kid sisters, the LDAP directories ( rfc2251). The DNS system also fits the description 'heirarchical database'.
As far as XML goes, there are people storing XML in directories - although they're still fussing about exactly how to do it. There are a bunch of people trying to come up with standards - check the directory services markup language people www.dsml.org.
There are people trying to sell XML enable directories - Novell sells an XML directory, but most directories can be used to store XML (including our 'eTrust Directory').
As a final quicky - when do you use a directory over an RDBMS? Directories are good for naturally heirarchical data with few cross connections. They are usually optimised for slow writes/fast reads. They are *very* good for distributed data (e.g. DNS, international organisations etc.). The X500 spec defines a very fine grained security model, which can also be useful. However, if your data is closely cross-linked with lots of relationships... well, use an RDBMS!
Wer mit Ungeheuern kämpft, mag zusehn, dass er nicht dabei zum Ungeheuer wird.
You mentioned configuration files: if all you're talking about is a linear config file, then XML might not give much benefit. But if your config file has a hierarchical structure, XML does provide a benefit, since it provides a well-defined and standard way to represent that hierarchical structure. In addition, XML-aware editors make it easier to work with these files, plus you don't have to write a specialized parser for it, plus you can display the file in an XML-aware browser, plus you can run automated transformations on the file, plus...
XML is one technology where the mindless adoption by people who don't really understand why they're adopting it, may in fact be of benefit to everyone in the long run.
XML is the first and closest thing we have to a universal standard data format. We're better off having such a thing than not having it. Since XML is the first such format, naturally it has its problems and limitations. But it's a step in the right direction, and we'll only find out how best to improve it if we use it heavily.
On the subject of the article, though, you're right. Whether a database's native representation is XML is irrelevant.
Where do you think any of the technologies mentioned have been misapplied? Do you have any examples?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
The problem with your scheme is that as the list of information grows the amount of time to do the access grows linearly. With systems where there are thousands of concurrent users all doing large numbers of data accessing of the information your solution would grind the system to a halt. While technically possible to do this with a relational database it is impractical. Taking into account that 90% of the databases use is doing reads and the access of such a system would be even worse. Also take into account that the system has to be able to handle some data which does not change and some data that has to be grouped based apon a visit date. Your system would have no efficient way to group certain elements by date.
The popularity of relational databases is that, with hierarchical, the hierarchy has to exist prior to the entry of data. Together with single parent inheritance, it imposes a structure upon a systems designer which, does not necessarily represent a real world process. With time, the process alters, and the model fractures.
The same occurs with that special breed of database, the directory. LDAP owes its single parent inheritance from X.500. This model is becoming less useful in representing the complexity of electronic commerce transaction events.
Multiple inheritance is implicit in many program structures such as Python, but not in LDAP. This is where the big change will be seen - in iPlanet, eDirectory and Active Directory.
But now my COBOL/IMS experience will pay off BIG!!! HAHAHAHAHAHAHA
Nice to see some solid and useful info being posted on /. every now and then...
The question of whether hierarchical or relational databases are better, more flexible, dependable and so on has already been answered decades ago. I fail to see how a popular hierarchical data transfer protocol re-opens the question. If XML is going to be pushed far beyond what it was designed for then it needs to become far less hierarchical rather than having the rest of the world, and especially mainstream DBMS, re-arranging itself according to XML's limitations.
Well why don't you idiots just use MS Access? Its file based (ISAM). Its a piece of crap so you guys would love it.
....while are these issues can be addressed with
time and widespread adoption...until then businesses
or individuals are unlikely to jump in.
It's a chicken and egg issue, i guess.
Just my two cents worth.
Reality is what we taste, smell, see, hear and touch yet we cannot comprehend it...only approximate it.
Like some other posters have pointed out, XML is no magic bullet. Sometimes I really don't get the hype. I really wonder whether CSV will become the next big thing -- comma separated values!
Why store a database in XML when you could store it in some high performance binary layout that maps to a disk's layout?
Sure, XML is good for human-readable data, just as HTML was in a more primitive way, but that doesn't make it very efficient for enterprise-level stuff, and especially the *storage* of enterprise-level data.
Don't we already have a huge XML database indexed hierarchically? The WWW!
Today you might want to store your history of employee-department assignments by nesting employees under departments, but at some point you may also want to nest work histories under employees.
pr0n - keeping monitor glass spotless since 1981.
The normalized form of many (if not most) kinds of data is not hierarchical.
Think about it: how do you represent complex "types" of things in a strict tree? Is that person a client? Or are they a contractor? Or are they an employee? How do you represent these things while keeping just agile enough for the future?
The stupidest thing about xml is that it is strictly hierarchical. XML is an extension of printing idioms, and it leads to the same dopish proliferation of records that is documented in Brazil.
Do yourself a favor: Don't force everything into the same hole. That's why we have squares and circles.
You can use id atributes, and references to those IDs in other elements, to imitate relational DBs y XMLs. Of course, it's not "natural", but it can be done
It's just a BloJJ
I agree that any hierarchical database can be replaced by a relational one. But XML is not a database format in general. The data in XML file are mixed with parts of free text and the order of elements matters. XML (SGML) was invented as a markup language - ie a method for inserting metadata into your plain text (for example Docbook).
why relational databases took off in the seventies and eighties was that they encapsulated the entire list of operations people wanted to do on data at the time within an extremely easy-to-understand concept. One data type, one set of operations.
Why object-orientated databases suddenly started to take off in the nineties, was that the list of operations people wanted to do on their data, suddenly expanded as computers got more powerful and memory got larger.
So data got larger, including things that weren't previously seen as data. Object-oriented databases went back to the old hierarchical database "model" to do so, while everybody forgot that the relational concept deals not primarily with allowable data/operations and data structures, but instead with the way data was manipulatable.
Now data has expanded yet again, and people thing XML is something new. surprise,surprise! It isn't. All that has happened is that the list of allowable operations on data has expanded, due to the pervasive influence of the Internet. The central concepts of the relational database model are still valid. All that needs to be done is to expand the allowable data/operations to cover XML. Simple.
Speaking of DBMSes, one of the electrical engineers at work wanted to learn how to use the Oracle DB one of our projects is built on. Somebody told him "No problem, it's dead easy. Almost everything can be done with only two commands, SELECT and UPDATE. Simply learn those two and you'll know everything there is to know about DBs." Apparently, the CS guru who witnessed this nearly imploded. Wish I'd seen it...
-- ;-)
Kuro5hin.org: where the good times never end.
Uhm.. Have you ever really used a database, or are you karma surfing?
The alternative to relational databases is not "DBM", it is object oriented, tree structured, logical, and other kinds of database models. Those are just as well defined as relational databases.
I can't defend the original poster's assertion that DBM is the only alternative. I do argue that object oriented models are different in nature, since they allow functional bindings to data. The original question regarded hierarchical structure, but said nothing of bound functionality. Your example of object-oriented databases being alternatives may be true, but they are not equivalent.
As for the rest of your examples which are relationally different, they are subsets of RDB. Anyone could implement an hierarchical database interface using a RDB back end fairly easily, with no more than a constant performance penalty. However, it is impossible to do the reverse. Please, educate me if you think otherwise.
Your arguments have the ring of an emotional belief that implicit tree structures have some direct benefit that relational ones do not. They don't. Relational systems can explicitly describe a tree structure, which among other things makes it possible to extract small portions and move parts of them around (network, disk, memory) without changing their relationship or representation. A tree structure cannot without creating a relational database equivalent (through use of pointers or indices into a table, etc). This should be more than adequate to prove there is value in relational systems where trees are insufficient and inefficient for equal operations. Contrariwise, you cannot produce an operation for which trees are inherently faster. More natural to think about as a human, perhaps, but not mathematically advantageous in any way.
Are you kidding? It is a major pain trying to express hierarchical data in a relational database model:
Sorry, that's simply wrong. The operations required to modify a data structure are related to the data structure, not its representation. That's why they have classes on teaching data structures as concepts, and not on implementing them. The same amount of data must be present in some form, even if you as a user are not aware of it. The relationship property of two nodes in a tree IS data. But in a tree, you often don't have direct access to it.
the relations that describe hierarchical data and the operations that you might want to execute often require complex, multiple, inefficient queries and updates, and the relational model provides few tools to ensure that the corresponding relations remain consistent.
The complexity for any representation is of the same Big-O order, regardless of the database type. I don't see where you can do more work in less statements with a non-RDB system which would be less complex or less inefficient, without throwing out some functionality. Also, relational databases usually support constraints to maintain valid relationships, as well. Please feel free to chime in here with some actual examples anytime, though.
[...]they generally include numerous facilities for object-oriented and hierarchical databases, under a "relational veneer".
Humans think hierarchically. Computers don't. It's natural to have explorer apps to help visualize data, but data does not want to be hierarchically organized.
For the record, object-oriented is not meaningful to associate with hierarchical order in exclusion of relational order. It is a common folly to think an object must be hierarchical in nature to be an object.
Any connection between your reality and mine is purely coincidental.
I see plenty of armchair database experts, out there talking about Relational Calculus'. Yeah, I'm sure there are plenty of DBA's with deep knowledge or RDBs.
How many of you have actually worked with a real Hierarchical Database like IMS? Perhaps we could get an intelligent post rather than the pseudo-intellectualy, fresh out of college, /. is known for?
I wish I knew more about filesystem programming, because I've long wished to write a simple file system that uses a structure which is independent of the presentation of files.
It would be simply wonderful to create a file system view, per user, which exists not only to restrict what they can see (almost like being chroot'd with lots of mounts in that directory), but also to make certain things more accessible or differently organized based on properties you feel are important. Doing so currently requires a shitload of symbolic links and manual maintenance when adding or removing files. Instead, you should be able to mount a file set under a name and put a query in that file set, so that it appears to be a directory with files that match some given attributes. Then you build a hierarchy of those, since that's a natural way to think about things.
The lack of categorization, or meta data, for files has been a thorn in users' collective side for decades, and with the death of Mac metadata in OS X, there's no real proponents out there for improvement.
Oh well... In my dreams...
Any connection between your reality and mine is purely coincidental.
The relationship property of two nodes in a tree IS [a relation].
Indeed it is. However, something like the "parent(x,y)" relation satisfies particular properties that the relational model has no support for enforcing. Furthermore, algorithms over trees are intrinsically recursive and usually require a recursive exploration of such a relation; you cannot express that with a bounded number of relational queries--it requires iterating queries together with transactioning across those queries, procedural code that falls outside the relational model and that, incidentally, is also very slow when implemented on top of standard relational databases.
The complexity for any representation is of the same Big-O order, regardless of the database type.
In real-world applications, constants matter, a lot in fact, so even if the algorithms weren't just the same big-O, but the same big-Omega, there would still be an issue. Second, the issue is not a clear-cut as you seem to think: depending on the specific relational database model one adopts, you may end up paying extra logarithmic or even linear factors in the size of the database.
I thought it would be interesting to quote Jim Starkey, original designer of InterBase and various other RDBMS on this subject:
It is very interesting to see ideas come and go then return again.
The earliest database management system I know of was of what might be called the amorphous data model -- a database was a collection of records, each consisting of totally arbitrary attribute/value pairs. It had some nice retrieval characteristics, but systematic
processing wasn't in it. Then came the hierarchical systems, IBM's IMS and the ARPA Datacomputer. The hierarchical systems
were regular and the tree architecture seemed intuitive, but updates were a nightmare and complex structures almost impossible
to model. Then came the CODASYL "network" data model of records and named "sets". It could model anything, was readily updatable,
but was utterly impossible to layer a query language on. Then came Codd's relational database (for two points, can anyone
explain the name?) which was highly regular, easily updated, expressive, extensive, and easily retrievable. So good, in fact, that once the performance problems were worked out it put absolutely everything out of business.
Then the object data model showed up --inheritance, polymorphism,
so very OO. The venture guys threw great big sacks of money at them. But scrape off the hype and you find a warmed over CODASYL database with all of the warts, glitches, and gotchas
that resulted in a mass extinction a decade earlier. Another mass extinction.
So now, whoppee, another hierarchical representation, complete with all the problems that made IMS such a mega-pig. XML as
an operational data representation is not just a terrible idea, it's the reincarnation of a very famous terrible idea.
I'm now waiting for organic medium database management system. Based by an all natural renewable resource (farmed trees fueled by our sun) using an offshoot of optical storage
technology -- hole/no hole -- arranged in a user friendly intuitive format of 12 rows and 80 columns.
When will they ever learn?
When will they ever learn?
Jim Starkey
Relational databases have their benefits when the data and the access modes fit neatly into the relational model. Over-normalisation of data is a
..."
sign though, that the relational model is breaking down in that instance.
Hierarchical is a much better fit for an object-based data model: "this IP address is a host, and it's running these services; that IP address is a router, and its connections are
I was telling an ex-cow-orker about IBM's IMS hierarchical database recently, how the access modes facilitate more correct programming ("get next object; do something with it; update and get following object") and easier access to related data ("get the next object contained within this object"). Although he grew up on PostgreSQL his response was "cooool!".
No, in any RDBMS, you just have the numbers in a table of phone numbers, and use a one-to-many relationship from the person to the numbers.
Sure, many database "designers" (and I use that term very loosely), will do something bone headed like define a fixed ten-decimal-digit number as the phone number field, but that's not the RDMS's fault.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
we shouldn't be talking about "XML data" is if it was somehow the core representation.
Exactly. The question of whether to use hierarchical databases is orthogonal to the question of whether to use XML. NetInfo, the DNS, LDAP, and smalltalk source code trees are all good examples of where you should use a hierarchical approach, and this was true long before anyone thought of XML.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
It's reasonable to point out obstacles in the way of a course of action when there is so little on the "benefit" side of the equation to make the cost worthwhile.
Maybe there is a mathematically elegant hierarchical representation for data, but XML isn't it. My vote would be for s-expressions.
-- What do you need?
-- Gnus. Lots of Gnus.
One of the advantages of being a old git is that you can see history repeating itself. First relational vs hierarchial, then objects appeared and we fought the relational vs object (usually hierarchial) wars, and now with XML we do the same dance again. The original rationale behind relational databases still hold true and I recommend Date as a good read. Most of the stuff in his books seems like common sense, but it wasn't so obvious at the time. It just goes to show how deeply relational thinking is embeded in the way we do things. Relational databases will win again, after all the data is the thing between the angle brackets. No doubt there will be some applications where XML specific atabases are the go but in general relational dbs will be modified until the XML functionality required is available.
The main reason for relational databases was the speed with which queries would be carried out as opposed to other mechanisms.
And the simple Discrete Mathematics that was inherently used to get the queries to work.
-- "To ask a question is to show ignorance; Not to ask a question means you'll remain ignorant."
I'm old enough to have had to actually WORK with hierarchical databases. The system was on the old Data General systems, and I can't recall the name... but I hated it almost as much as I hated "pure" Pascal.
/know/ how much more freedom of expression and action both provide. Not that they are better in the same way, they just both represent a better (in many ways) technnology.
True liberation came in the form of relational databases and C. I
Count me out in the wave de-evolution back to hierarchical databases. Let's not let XML be the cart that precedes a dead horse.
I made my self a little application for storing my thoughts, addresses, todolists, random snippets of commandline hacks, quotes, links etc.
I found no other application capable of doing it out there,.. so I made my own (hnb), it turns out other outliners, as I found out these apps actually are called, existed, but none of them did it the way I felt natural.
I think hierarchical db's are nice for things that humans are supposed to organize and reorganize, but I doubt it would be really sensical to move from rdbms to pure hdbms because of xml.
--The mind is it's own place and in itself, can make a heaven of hell, a hell of heaven.
There is no hierarchy that a relational scheme cannot describe. (Unless it's not a well-defined hierarchy to begin with (e.g. Parent(A)=B, Parent(B) = A etc))
Relational tables are hierarchical, but the tree structure is implicit, it's implied by the relation. It can be generated from the relations, (very efficiently with today's hardware and software).
SELECT students WHERE teacher='Miss Jones' and school = 'P.S. #1' certainly elaborates a school-teacher-student hierarchy, doesn't it?
And relations can go where no singly rooted tree dare tread: bi-directional graphs, dags, lattices etc.
I was wondering about exactly this a little while ago when I came across this very interesting site called Database Debunking by Fabian Pascal. Very interesting to see such a "contrarian" view. He's the guy who wrote Practical Issues In Database Management.
Go for http://www.firstsql.com/dbdebunk/... there Fabian Pascal, Chris J Date and occasionally Hugh Darwen, the greatest living experts on database management systems, constantely debunk the need for object, hierarchical and network databases, includin XML... all we need is a properly implemented relational database management systems, something better than SQL.
Leandro Guimarães Faria Corcete DUTRA
DA, DBA, SysAdmin, Data Modeller
GNU Project, Debian GNU/Lin
He also states that there are currently NO real RDBMS out there - just SQL databases, which offer limitted support for the full relational calculus. To me this means that a hierarchical or an object database environment could be used to build a proper RDBMS. The thing Fabian Pascal warns about is not confuse the underlying technology with the relationsl model, if something uses tables, it does not mean it is an RDBMS.
if you have already made any experiences with an xml database - you will never be talking about performance in such a context again.
The OO way to answer this is that body-type is a class and compact, sedan, SUV, etc... are instances of it. Each car would have some instance of body-type as a member. I've implemented this sort of thing in a roll-your-own OODB (in Ruby) and in a OODB-on-SQL (in Delphi); in both cases it was painless. The only thing that is remotely tricky is to avoid infinite loops in your low-level serialization code, by doing lazy streaming or by having a serialization flag, or stack, etc. just in case some later person creates a body-type (e.g. batmobile) that somehow refers back to one or more instance of car.
-- MarkusQ
relational databases are easy to query, but rather difficult to navigate, hierarchical are easy to navigate, but hard to query. take your pick..
Your post is errr.... strange.
:-), for one thing:
/* your manufactor data here */ }
.less. Manufactor, Car .greater. CarManufactors;
....
... Errr ... what do you guess?
.... BUG or am I just not able to post correctly?
:-(
I can not follow it, so I do not dare to point out what is wrong.
Except, of course
In an OODBMS there is no MAPPING from objects in the programming language to objects in the DB.
The objects are stored one to one.
A OODBMS has only three purposes:
1) Allow querries on OO structures(graphs of objects).
2) Allow transactional save changes on a lot of distinged objects.
3) Isolate point 2) if different users access the same objects.
You example of cars and manufactors is to complex to talk on/make surgery on here. However:
class Maufactor {
class Car { }
template
class One_Parent_to_many_Children {
Parent parent;
Children* children;
}
typedef One_Parent_to_many_Children
This are C++ classes. Most people would simply have a pointer to the Manufactor in the Car class, of course.
If you have that above, the objects in the database are
Ermm.... Cars, and Manufactors and relations, noting else. Nothing to map.
For selections you make OQL querries. Yes, you can not REMOVE COLUMNS, like you can do in SQL. And you do not like to do that, or do you like to get HALF cars out of the DB? You are moving in a C++ world above, so you like to get Cars and Manufactors out of the DB and not something which is returned by an:
select Manufactor, Cartype from Manufactors, Cars, where production_date is_less_than 2001-11-19;
Because that SQL query only returns a list of two attributes of Car or Manufactor.
The similar OQL query:
select Manufactor from Manufactors, Cars where Car.production_date is_less_than 2001-11-19;
yields
It yields your favourite container class configured as query result, in this case it might be an STL vector.
Each Manufactor object referes in its list of produced Cars to the Cars fitting the querry above.
If you stick to the relation class I sketched above, you only might get Manufactors and then you make a new query for each Manufactor in the vector to get the Cars. If you would have used a pointer, as most C++ programmes would have done, the OQL query above would be just fine.
The result would have been trees/graphs of Manufactors with Cars built by them. FULL FLEDGED C++ objects, ready to call virtual functions on.
Not some array of text data to be converted into objects.
Regads,
angel'o'sphere
P.S. I have configured "PLAIN OLD TEXT" for submitting. less and greater signs are ALLWAYS interpreted as HTML leads. BOLD etc. does not work as the closing tag is note recognized
ARGGGGGG after posting I saw even the less_than operator in the SQL fragments messed my post up, sorry for posting twice
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
I have been toying for a while now with how to represent data efficiently in a manner in which it can be efficiently walked with an XML document-object-model, if not in XML itself.
;)
My core concern is actually building the indexing tools to handle the matching of insane numbers of actual Xlinks (or a mildly simplified form thereof), to make a dataset that is distributed and provided by multiple subsystems hold together.
My most recent iteration used a simplified form of Xlink to join data supplied by code that implemented a specialized DOM model (code that understood how to browse a given object model, for reflection purposes), with data taken from a flat text XML document (browsed with a different implementation of that model).
The real concern I have with this sort of system is the time it takes to register a document against all of the current X-links and deriving an efficient system for parent-child connection when multiple parents can exist (transparent links from other nodes). Registering a new node or worse, changing data in a node is an expensive operation in this model.
My need for extensive X-link support is to be able to provide a sort of on-the-fly XSL translation of one branch of data into another. The links would then connect to the translated data rather than the original.
A secured view of the data could be provided by handing a link to a DOM node that was walking the XSL view, rather than the original data. User security becomes a XSL-like document.
My filesystem becomes a branch of the tree. It is admittedly an awkward security mechanism to do XSL-style matching against the tree, but it does not have to be done in XSL itself, another utility which walked a base dataset and returned a filtered view in the same DOM model provides the same security mechanics.
There are some issues with the current mechanisms for conveying XML information (DTDs do not localize well, etc).
Method invocations are shaky at best and rely on sharing handles of some sort (SOAP, CORBA, etc). I transmit data either by copying a branch of the tree, or handing off a handle to a CORBA or SOAP object that can walk the DOM on the local system.
Either way looks the same to the client.
A hierarchal model has a certain level of appeal because of the simplification of the new-branch registration process, but severely limits the effectiveness of the tree processing tools you can build. OTOH, the relational model could be reconstructed with some stylesheet tricks.
My present use is in an operating system project as the mechanism for accessing system resources, the 'file system', user data, etc.
There is appeal when it comes to generating a backwards compatible view of the data, because you can provide a translator which takes the current data and translates it back to a view compatible with what a given application expects, etc. Method invocations through a function call-translator can allow for constrained arguments to methods, etc.
The transparent linking model also has appeal for simplified remote method invocation, a filename is just an Xlink, etc.
Active Directory, eat your heart out.
Sanity is a sandbox. I prefer the swings.
So, what would you say worked out better than applets?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Do not click the link, MS loser site
I'm a database software engineer and I am trying to understand all of these replies by people attempting to be professional database developers or high-level DBAs. First off, there are products to transform Hierarchial/Network format databases into a relational structure. IMHO they all suck. A major one is by Centura/Raima called ExecSQL which works with their db.Star and previous Raima Data Manager systems.
/. readers preach these replies as fact. These poser posters are probably the ones saying that RDBMS' don't need transactions ala MySQL. Hmm, how can the database be reliable internally without transaction support? You all need to get some ACID information, I like reading intelligent articles but dislike 500 factually incorrect posts/replies by uninformed and so-called industry professionals.
The relational data model, even when normalized, will contain redundant data not found in hierarchial systems however, doing an outer-join or view on a hierarchial system is much harder as the internal structures are not representative of relations, domains, attributes, and tuples. I've written several relational and hierarchial/network systems and have found that while relational may be a minute bit slower, the advantages are greater.
XML vs. RDBMS?
Why is XML, a storage format, being compared with a software system? I and many other manufacturers can easily write a RDBMS engine around the XML data model. IMHO XML sucks as far as speed and reliability compared to traditional storage methods employed by all major databases. Transactions stored in XML are great however, why? Because XML is a storage system? Get a brain, don't spread false gospel in here. I like the article that said, "we want something new and sexy". How true, and how many uninformed
sucks salty donkey balls