Slashdot Mirror
Distributed Versus Centralized DB?
OSXCPA asks: "I am developing a framework for processing accounting information. What kinds of advice would you have for setting up an environment where several (eventually, many) remote data aggregation and processing centers would update a central server 'after hours' to avoid processing delays. For those with ERP experience, I am trying to process transactions locally, instead of generating a batch in the central server, then update and validate the result to the central server, so my remote users can process at full speed all day, and central DB can get caught up later. Remote locations are all self-contained, in that no one will be posting transactions to another remote location. No big iron - this is all networked PC/Mac hardware and a central server (eventually to be a server farm, as needed). Feedback or constructive criticism would be appreciated. Technical advice is great, but I'm also looking for stuff like 'Gee, if I had only known XXX when I set this up, I would have done YYY...'"
Whatever solution you choose, be sure to give each DB entry a unique identifier that is unique across systems, and make sure the identifier does not change (E.g. Don't use the customer's name, phone number or any other changeable information).
One such solution: Each customer transaction has an internal ID number, like "Order #1000". Each system prepends it's own identifier to the order number, so order #1000 made on System 03 will have a UID of 03.1000. 03.1000 won't conflict with order #1000 made on system 02, because the latter UID is 02.1000.
Using a unique internal UID also lets you keep all order history around. If th customer wants to cancel their order and retry, you simply mark 03.1000 as cancelled, and start a new order with 03.1001.
This may seem obvious to you, but I can't tell you how many times I've seen a product fail because the development team decided to use the customer's name/phone number/address as the UID, which meant the customer couldn't change their name/phone number/address without breaking the index or other features in the DB. Not sure why, but this seems to be a common problem in databases.
"Can of worms? The can is open... the worms are everywhere."
I am not a professional DBA, but I tend to wind up working with them. The one thing I remember from a big project where replication and whatnot came into play was that none of that stuff ever worked as well as the vendors said it would, and it all required a lot more work on the DBA's part (and typically lots more $$$ in general) than it was supposed to.
Village idiot in some extremely smart villages.
I did that awhile back for a data warehouse and reporting system with remote data collection points. (Half a TB of data with 50 million queries a day and a couple Gig to integrate every night). I did all the data validation and scrubbing and some preprocessing remotely to utilize the processing power of the collection machines as much as possible and kept the backend stored procedures strictly limited to import and reporting functions. The front end reporting also had some functions built into the client so I could harness the power of those machines as well. YMMV in that regard though depending on the architecture and intended purpose.
:)
:)
Tip 1 - don't use any clustering software on the back end. Every clustering model out there tries to be generic enough to handle a variety of apps. This adds unnecessary overhead and complexity. If you design the app for a multiplicity of servers in the first place your performance and scalability will be much better! Not to mention everything will be a lot simpler which makes troubleshooting much easier! In my app I just had a table with a list of server names. To add a server you just loaded a blank database on it, plugged the machine in and put the name in that table. A few key tables (such as that one) were automatically replicated every few minutes and if the schema was missing on a server it got built and populated on the fly using self-generating scripts. (Very handy if you're making changes to the structure!
Tip 2 - when importing drop your indexes! It's infinitely faster to bulk import into non-indexed tables and then rebuild the index once afterwards when you're ready for post-processing than it is to try to maintain the indexes for every little insert. (Literally about a thousand to one speed ratio if the data is any size at all).
Tip 3 - like tip 2 this is more general database design advice. And that is to base your normalization on the nature of the data and the queries, not necessarily on the technical granularity. In other words a little bit of de-normalization is okay if it reduces the complexity of your queries enough. In our case we had an insane number of cases where many tables could be cross related in different ways. Based on the reporting we needed to do we purposely duplicated a few fields (mostly indexes) here and there between tables. This cut the memory utiliation of some of the more complex queries by up to 50%. When some of the joins can result in gigabyte matrices that's a significant savings!
Tip 4 - try to spread the load. In the complex queries above we would of course try to break them down into smaller queries that could be run in sequence but that often resulted in the final report taking too long. The answer was to move the aggregation to the client. I.e. have the client pass the different parts of the report off to multiple servers to run in parallel and then pull the results together at the client side.
That's all I can think of off the top of my head. If you want me to consult further just email me, I'm not terribly expensive.
Comment removed based on user account deletion
You mention that this is for an accounting application. You suggest a scheme with local databases that periodically send updates to the central facility. If the books and records are in a strict hierarchy (each node is contained in exactly one "higher" node), this can work fine for some purposes, like financial reporting.
In the situations I worked on, people initially agreed that this sort of setup would be fine. However, as we dug a little deeper, we realized it wouldn't work, because financial reporting wasn't the only real need. In the context of an investment bank, the capacity to take risk is a (finite) asset, and the amount of capacity that's available at any time is a function of all the trades that have been done anywhere -- since everything is ultimately reflected back to the parent's accounts. Had we not realized that, we would have built a fine system that we would have started rebuilding as soon as we'd finished it (cf. There's never time to do it right, but always time to do it over).
I won't belabor things like redundancy, backups, and so on; I assume you have taken careful thought for those.
Finally, IMO, the trickiest thing to get right in a distributed DB system is maintaining the referential and logical integrity of the database. I personally know of one well-known DB vendor's initial replication solution (names omitted to protect the guilty) that had a subtle but potentially very serious logical flaw. Moral: do your own homework.
I can't stress this enough: the purpose of a database is to guarantee data integrity. Be sure you understand that: storing/retreving/manipulating data is part of it too, but without data integrity, you're lost.
Unfortunately the state of the database industry (both vendors and users) is so fucked up right now I'm not even sure people know what "data integrity" means. I'll give you one definition: it means that every possible combination of values in your database has a legal business meaning. Does it makes sense to have order items without corresponding orders? No? Then make sure it's not possible.
Another definition: don't ever allow the two facts "X" and "not X" to ever be derived from the same database! With my order item example above: if order #123 doesn't exist, but there are order items referencing #123, then you can prove both "order #123 exists" and "order #123 doesn't exist" from the same database. BAD!
As for replicated databases, one approach is usually to partition the data. For instance, use a compound key (REGION#, ID#), where REGION# is hardcoded on each database. I.e. you have a constraint that says REGION# must always be "100" on database #1, "101" on database #2, etc.
You can combine all the data regularly into one BIG databases, or you can just go out and query each individual database depending on what region it's in. I'm maintaining an in-house app currently that keeps customer data for each individual city in a cluster physically located in that city. Then an outside app polls the databases and combines the data for presentation. Works pretty well, but my queries are pretty simple.
We looked into doing something like this a year ago, and determined that with the cost of networking and clusters of cheap computers being the way they are, a centralized solution was far cheaper.
My scenario was a distributed environment with approximately 3,000 remote sites. If you had less remote sites, the distributed processing may be cost effective, but you need to use extreme caution as distributed solutions can escalate in cost quickly.
Conformity is the jailer of freedom and enemy of growth. -JFK
I'm using globally-unique-id's for all records. These are generated by the database system I'm using and are guaranteed to be different across all the offices. No conflict between new records created at different offices. And all I have to do is use an existing feature from the database system.
I don't delete records. Just mark dead records using a timestamp. This way records don't have to be deleted from the central database and other offices; instead just propagate the dead timestamp.
I've decided to write my own update program that pushes changes from one office to the central database. I can use this same program in reverse to update the offices from the central database. And I can use it to update laptop users who only connect once a week. And, and, and...
It's a chore to write the update program. But I see some advantages:
Since you mentioned this application is being rolled out to a bunch of PC's and Mac's, it sounds to me like you might be deploying to a large installation base where you do NOT have centralized management of the machines...
At some point in the future, you probably will be upgrading the system to support new features / new data.
You'll want to design the intefaces between the remote sites and "the mothership" so that as migrations are partially completed, the system continues to run for the older and the newer versions.
You might want to take a look at Erlang and the Mnesia database for it. It is designed for distribution, replication, and transactions. Mainly used in telephone systems.
Be very careful about trying to "fix everything" after hours. That's how we do some things (and did the same thing at my previous place), and it's a nightmare. If anything is wrong in the morning, then it's hard to go back and re-run the previous night's batch processing. Often it takes too long, often you would just corrupt data further, etc... Very often this results in us having to just have incorrect data for a day, and then get it right the next day. In addition, due to global expansion, we only have 2 hours each day in which to run this stuff. If a market were to stay open an extra little while, or we try to expand somewhere in the forbidden slot, I don't know what would become of us.
You are far better off trying to get everything to be realtime, as much as possible, and build in mechanisms for the system to recover from corruption and incrementally do batching throughout the day. Do NOT rely on the fact that the system goes offline for 8 hours each night, that's an easy way to produce a system that costs you your job.
You are entering an area of database technology where angels fear to tread. This is something that if it were easy, we wouldn't be paying top DBA's hundreds of thousands of dollars to do it and keep it working with acceptable uptime. There is a reason why companies that run the scale of Amazon have IT expenses in the millions and millions, and why they do their darnedest to hire and keep the best talent in the country. You're entering the big leagues now. You're going to need a professional team to handle the gameplay, because your minor-league team can't cut it.
/in depth/ the field as it exists today, and (b) understand the problem you are being asked to solve /in depth/, and anticipate future needs, and (c) mapping a solution that addresses the facts of life in (a) and (b). This means paperwork, specifications, and a lot of meetings and sales calls and demonstrations and typing.
Your first step is to look your boss square in the eye and say, "I think I am smart. But I know there are a lot of smarter guys than me out there, and they have a hard time implementing this right, and they have a budget in the millions. Sure, we can hack together a solution that may solve your short-term needs, but I know what is coming down the road and that solution will not be it." You can elaborate what you know as far as your manager is willing to try and understand it. But you need to wake your manager up and explain to him that this is a different game now, run with different rules.
Your next step is to convince the boss to give you time and resources to learn the black arts of this field. The resources will include a *team* of DBAs who have experience with this kind of thing (and they aren't cheap!), software developers who have seen this before (again, not cheap!), machines to run on of the proper caliber (and you may decide you will need a mainframe - don't write it off too soon.) The time will be whatever it takes to (a) understand
If your manager can't agree to this, then you need to tell him this next part: "I cannot implement what you need unless you give me the tools and resources to do it. You are asking me to build a skyscraper with spare parts I find in the kitchen. You may be able to find someone who is willing to tell you that they can do it, and do it cheaper and faster. But I submit to you that they are selling you snake oil. When you see his solution, when it comes under load, and when you start to learn about things you've never heard of, which they'll call "minor technical details that can be easily fixed", but deep in your gut you'll get the sense that they aren't minor and that they are certainly not "details", you will remember what I told you about skyscrapers and kitchen tools. You will begin to grasp the enormity of the problem you have asked me to solve. You will begin to see that throwing the money after the snake oil salesman was a worse investment than throwing a lot of money at the real solution."
If I were you, I'd start filling out the resume and start looking for a new job at a smaller company that doesn't have these problems. That is, unless you absolutely believe you are going to be up to the task and management is going to take it as seriously as you will. I've seen too many people try to deliver to management what they could not because they wouldn't pay attention to those annoying "facts" and this thing called "logic". Don't fall into this group. Be up-front, bold, and clear as heck with your manager, and don't try to hide anything from them.
The radical sect of Islam would either see you dead or "reverted" to Islam.
Your sig is a pathetic twist of truth, Muslims never forced non-Muslims to convert because we are tolerant and forgiving unlike your macho butcher crusaders grandfathers who prosecuted non-Christians and wiped the red indians for the fun of it.
Muslims never forced non-Muslims to convert because we are tolerant and forgiving
Riiight. Tell that to Spain. Or Byzantium
Hi,
in case you are working with an OO language (Java or C# ?), you may consider looking into exisiting frameworks that provide object-oriented replication.
That would relieve you from building features like the following yourself:
- Unique Universal Identifiers (UUIDs)
- Cascaded replication with object member traversal
- Conflict resolution at object level providing callbacks
The following open source object database engine provides such functionality and you may even consider it as an alternative to relational databases for caching your data on the clients:
http://www.db4o.com/
db4o - open source object database for Java and
I constantly have to remind myself as well 1. Find the simplest solution that could possibly work. 2. Design for testing. If you can't easily test and debug the system, then your design is bad. 3. Avoid distribution and threading if possible (see #2) That being said...there aren't simple solutions to all problems. Try to find a base to build from with source-code and a license that you can live with.
Yes. Most of my job is data integrity on a distributed infrastructure. Issues are hard to spot and difficult to automate fixes for (a script to fix 1000 records here, another for a few hundred somewhere else). They hurt customers because of the slowness in fixing (requires a developer) and spotting (mmm, customer care are getting a lot of calls about this type of issue today).
I'm not an architect but if I ever had the chance I'd certainly investigate one database on stupidly powerful hardware. Single databases were designed to have solid data integrity and no distributed system cuts it in my experience.
this is just fear-mongering, with no primary data to substantiate the claim that the task is difficult, only hearsay that there are people who spend a lot to do something that sounds, in a 2 sentence description, vaguely similar.
from the original description, it sounded to me like the central DB was purely for mining purposes, to monitor what was going on, perhaps to learn from patterns in account activity. as such, the decoupled, do-it-yourself approach IS the right thing, and spending big bucks will do nothing but waste money.
sure, do your homework: study whether the central DB has any data path back to the subsidiary ones. whether there's really no interaction between subsidiaries. whether there really is "slack" time when data can be streamed back to central, and whether central can handle it.
observe that the goal is actually just locality: to provide sub data ot central without having to make remote access. as such, all you really have to do is snapshot/mirror sub DB's onto machines at central. doing it incremental is a useful efficiency hack. putting all sub data into one central DB is also just a convenience hack.
I've not been trained in building distributed systems but have learned on my own from the school of hard knocks.
:-)
As metioned a good Record ID strategy is a must; I used a packed timestamp (base 60, using upper/lowr case numbers) with a node/user identifier, besides keeping the records unique you can also use the ID to track down where/when problems mikght have occurred.
Also time stamps and delete logs!!! You need to know what data is new or old or should have been deleted.
Work on a way to automate updates, you are going to be coding not for one computer but for all your nodes on the network, set something out there that can keep your data/apps in check and updated. A big stumbling block is getting the data transfer structure and protocol worked out.
Look at the importance and frequency of data and make sure your system can handle it, if nightly is OK, then it's fine, though there might be times when your satte;lites might be down, don't paint yourself into a corner by expecing every transfer to work right every time.
Choose your apps wisely, since you mentioned Macs that narrows your choice of what you can use. We had used FoxBase +/Mac, but 4th dimension is cross-platform and looks real flexible. It could be done using LAMP at the nodes, but you will have to work out your own distributed data import/export scripts. (DBF xfer was sooo eay)
I find you have to implement a lot more remote validation, as you expect your data to be clean when it comes in from the other offfices.
Also things happen to different sites, and they may have to restore from backups etc. Work out how to re-sync the sites when such events occur.
You are looking at a very complex system to manage. (those thousand or so lines of spaghetti code will keep you employed)
Normalize and clean up your database as much as possible now include planned data element (even if you currently hide it for now), making structure changes when distributed can be a real pain.
With that said, after years of being distributed; technology here in our rural community now makes it possible to centralize, and I'm relieved that I can make the switch. (though I may keep bits distibuted, field laptops, and such.) It's a good skill to learn, adds a whole new level of paranoia when you're coding.
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
It all comes down to exactly what the original poster meant by Remote locations are all self-contained, in that no one will be posting transactions to another remote location.
Is the data self contained or the users self contained? If just the users I agree with you he's underestimating the difficulty. If the data then its pretty easy to do and I've done it before. He's not having to solve the remote data locking problem or consistency problem or, he just has to get his partition perfect.
Here is my short list based on real life experiences:
*Read Been There, Done that above.
*If your system is geographically dispersed there is no night time. If you can handle the transactions at night, then handle them "near realtime" by sending the transactions into a queue to be processed when bandwidth, CPU, etc. are available.
*Assume the Central database will not respond for three days (a tornado, etc. at the site) and see what your model will do versus what the business wants.
*Unplug the network during a transfer to the Central site. What do you expect it to do, when an admin pulls the cable for a second to clean behind the server.
*Identify units of work. Those transactions the business believes must occur together. These are often different than DB units of work.
*Expect bad data. People will tell you that it does not happen, but if you send transactions via a secondary process, it will. A record that was good on the remote box will be unusable on the Central or vise versa. Now what should happen?
*Compress all transactions being sent to/fro. While some will be tiny and hardly seem worth it, a consistent approach makes life at 3:00 a.m. a lot easier.
* XML is your best friend and worst enemy. If you choose to use it, know why it is good for your application.
*Avoid any product, template, approach or design built for the future. If you are not going to use it in the initial production roll-out. Stay away. If you are that good at reading the future, by lottery tickets. Unless this is a very small/fast project your options twelve months from now will be very different and you will have spent the time/money/energy building a solution without a use.
*Consider what areas are most likely to change, based on past business experience, and make them easy to change. (No, this does not conflict with the previous bullet
*New tools and additional bodies will not help when you are late. Add more people early and your life will be much easier. Rapid Development by S. McConnell is still the bible for PM.
*Create a lab where you connect the Centeral DB to a set of remote servers (also in the lab). They should be blazing fast as they are connected locally, and give you the ability to create "what if" cases in-house.
*Create a step by step process for rebuilding a server from scratch. From the time the hardware is turned on, to the time the server can be released. This is a painful experience and will take a week or better full time, but will make every additional site easy to implement. Have a standard for what is dynamic based on the server. (i.e. Step 26: Change ##siteName## to local host name) so people type "www.abcCompany.com" instead of siteName. Then have someone not involved in the documentation try it.
Hope this helps.
Cheers.
I'm in a situation where we have a few thousand retail locations that send data up at night. A few points of pain:
1. Use an extensible format for your batch uploads.
What you think you want to upload today will be different (probably less than) what you want to upload tomorrow. Use an extensible format for your batch uploads (XML is awfully nice in this regard, it just eats up bandwidth). That way when you make upgrades to part of the chain, your server software can gracefully handle differences in the data being sent up from different locations. If you try to "bleed the turnip" in terms of bandwidth and go with something like a fixed width or comma separated file, you're going to pay for it in maintenance down the road.
2. Keep a backup of the data at the remote location until the remote location receives a well-formed acknowledgement form the central location. Just because the remote location sends up the data doesn't mean it's captured; a lot of things can go wrong, and it makes life much easier if you can recover the data at the remote location later. Furthermore, PLAN to miss some uploads; it's just going to happen sometimes, and if your system is designed to handle that fact, you'll be much better off.
3. Document your data formats - the format at the remote location and the central location (presumably those are relational databases and can be documented with detailed ER diagrams) and the format used for transmission. Troubleshooting is a pain in the ass if you are unsure about how to map a transmission field to a remote or central DB field. Seems like a no-brainer, but if you don't do it from the beginning you're not ever going to do it.
4. You might want to give your remote locations random upload times within the overall upload window so as not to "slashdot" yourself. If you want the remote locations to upload between 1 AM and 4 AM, write a script and schedule it in cron that will trigger the upload at some random time during that period. If all your remote locations start uploading at 01:00:00.00, you're going to have trouble scaling.
This kind of thing isn't rocket science, but once you start down the wrong path, it's easy to get entrenched to the point where it's easier to live with your mistakes than to take corrective action.
If your manager can't agree to this, then you need to tell him this next part: "I cannot implement what you need unless you give me the tools and resources to do it...
/etc. Delineate the ways in which it is at risk of failing, explain how doing it right can mitigate those risks, and document all of this. Then let your manager decide. Whatever you do, don't be stand-offish and negative about what the manager wants, no matter how unreasonable; rather try to be informative, helpful, and can-do, yet wisely cautious based on experience.
Be up-front, bold, and clear as heck with your manager, and don't try to hide anything from them.
As an IT professional I certainly agree with your take on this, however I would take a different approach to informing my manager. Instead of saying "I cannot", I would couch the dilemna in terms of risk. Explain the difficulty of the problem in a way a reasonably intelligent but non-techicnal manager can understand or at least appreciate. Explain what it will take to do it right the first try, and explain how doing it right will mitigate the risk of failure/incorrect data/etc. Then explain that we can also hack something together quickly and *relatively* inexpensively that may address his immediate needs, but which will have a greater risk of failure / inaccurate data
I've also had success in explaining to my manager (a smart numbers guy, but no systems/development experience) that some things in IT can slapped together quickly and incremented/evolved over time (a la Paul Graham's Incremental Development), while others must be done right/correctly from the start. As a rule, databases fall into the latter category. Many applications fall into the former.
Approaching your manager like this educates him in a respectful way to which he will likely be more receptive, ensures that you are doing your job to the best of your ability and meeting your professional responsibilities, covers your ass in case the manager knowlingly decides on the riskier course and fails, and prevents you from being seen, as managers tend to do, as a complaining, "can't do", negative IT guy.
Flying is easy, just throw yourself at the ground and miss. -Douglas Adams
and you may decide you will need a mainframe - don't write it off too soon.
Yes lets see what companies are surplusing there old 70's area equipment so we can implement a brand new system on outdated hardware. You do realize the reason mainframes are still in use today is because the software sitting on them was so expensive to devleop that is cheaper to maintain there then to write new code on a modern platform. There is virtualy no new Mainframe development just upkeep programming. If you need something truely powerful look to Sun, Unisys, IBM , or HP to give you a 8, 16, 32, 64+ processor machine or somesort of cluster, but Mainframe no way.
I work with a retail POS system that pushes around updates at night. I've noticed that there is a problem in knowing whether or not the remote servers are up to date or not. So, for a new development, I would recommend that part of your update scripts send some sort of table statistics like
count(*) where date > yesterday
so that you have some idea if everything is in sync. Along a similar vein, mark records as "dead" somehow rather than try and push around record deletions -- it's a lot easier to troubleshoot.
Actually there's a third solution! Kill all Muslims!
Assuming you're building a custom system (as opposed to finding something off-the-shelf):
Tip 5ish - Communication lines fail. Power lines go down at your remote sites. If something fails at the wrong time, your nightly process might not get the data in time to finish. I recommend a "trickle" approach, where the transaction processing software on your remote nodes hands off the resulting data to a separate "central update" program. (This can be as simple as setting a "complete for transfer" flag in your local database, or it can be as complex as establishing a separate queuing mechanism with message passing.) The "central update" process (running on your remote site) continuously handles sending data to the central server (as a low-priority process) and receiving confirmations.
Tip 6ish - use date/timestamps for all your flags (such as "ready to send")! A null value indicates "false," a date/time indicates "true". This kind of information is invaluable when troubleshooting communication breakdowns, and will be of incalculable aid in getting your databases synchronized once again. Timestamps are just A Good Idea.
Tip 7ish - no update is complete until a confirmation has been received from the central server. A "closed loop" transaction, which receives a confirmation back from the central server, will save you more headaches than you can comfortably imagine. TCP/IP confirmations are not enough; those will tell you the message was received, but not that it made it into the database, or that it was successfully validated. You need a separate audit process on the central server(s), scanning your main database(s) for successful updates which have not been confirmed. That process then sends a confirmation back to the originator of the data. The beauty of this approach is that the remote system can be designed to automatically re-send transactions which have not been confirmed in a "reasonable" amount of time. No intervention on the part of live humans is necessary.
If you use an automatic re-send process, make sure the central update server is ready to handle duplicate transactions when a confirmation is lost.
In theory, there's no difference between theory and practice. In practice, there's a big difference.
I say that 'standard' databases are 'deathtraps' for the simple reason that the data is stored in a (usually) proprietary way that cannot be easily reversed engineered in case of a database corruption by way of hardware failure (a hard disk going bad) or some sort of software failure (a virus infected the affected software, causing it to introduce corrupt data records to the database).
:(
Why not use structured, totally readable flat ASCII files (meaning all binary numbers are stored as bulky, readable ASCII strings) in conjunction with additional binary files to index and create the 'table relations' among 2 or more files. With this setup, if the flat file gets corrupted and is detected in time, all you've lost is a handfull of records. With a proper database getting corrupted, cross your fingers and hope the DB repair tool does its job correctly. If it fails you appear to be thoroughly SOL!
Any rebuttals?
PS: I know 'handmade' databases like this are a pain to develop and use, my aim was ultimately for the preservation and re-use of data in a readable and understandable format. The drawbacks are obviously size, speed, and complexity but what is more important to you -- your data or lining the pockets of somebody selling you a RDBMS that may fall flat on it's face when you need to get your data back after a system crash. And before it gets mentioned, I'm lumping the freeware RDBMSes out there in the same group as the commercial, gotta-pay-for-them versions -- their only difference is the price (the free ones are $0.00 in the USA).
The only pluses RDBMSes have that I can see is convenience, flexibility, and 'compactness' -- a 'standard' late-model DB created by a RDBMS usually has a handful of files with it -- perhaps as few as 1 or 2.
If a 'real' RDBMS can be implemented as a bunch of flat, 100% readable ASCII files and supported by additional binary files as mentioned above, I'd like to be one of the ones to use one if it is available (now). The one that came closest to that goal that I know of was Ashton-Tate's DBASE II from the 1980s (but I imagine it stored binary numbers in databases in binary and not readable ASCII strings)....
PPS: Those extra 'binary files' could be done as straight readable ASCII files and you have a fairly corrupt-proof system I'd say. The person or company that can make such a RDBMS stands to make a fortune with new customers and customers of competing RDBMSes 'jumping ship' and (paying for) using the 'new' RDBMS.