Slashdot Mirror
Brian Aker On the Future of Databases
blackbearnh recommends
an interview with MySQL Director of Technology Brian Aker that O'Reilly Media is running. Aker talks about the merger of MySQL with Sun, the challenges of designing databases for a SOA world, and what the next decade will bring as far as changes to traditional database architecture. Audio is also available. From the interview: "I think there's two things right now that are pushing the changes... The first thing that's going to push the basic old OLCP transactional database world, which... really hasn't [changed] in some time now — is really a change in the number of cores and the move to solid state disks because a lot of the... concept around database is the idea that you don't have access to enough memory. Your disk is slow, can't do random reads very well, and you maybe have one, maybe eight processors but... you look at some of the upper-end hardware and the mini-core stuff,... and you're almost looking at kind of an array of processing that you're doing; you've got access to so many processors. And well the whole story of trying to optimize... around the problem of random I/O being expensive, well that's not that big of a deal when you actually have solid state disks. So that's one whole area I think that will... cause a rethinking in... the standard Jim Gray relational database design."
I couldn't... agree... more... I'd say that some... very valid... points... have been... raised.
Can we please have another loud, circular debate over which database is best? It's the only way your favorite database will ever win.
Thank you.
Gotta love that link between the hardware limitations and the software concepts that may seem fancy but are essentially only built to get around them. I believe someone once called it "the law of leaky abstractions" - would be interesting to see what the new limitations would be if you start combining solid-state storage with pervasive multiprocessing, i.e. what can you do with a multi-processor multi-sdd server that you can not do with a single-processor single-hard drive server?
I think TFA is pretty right on the money that parallellization and massive use of SSD could cause some pretty fundamental changes in how we approach database optimization - if I were to imagine that rack that I'm staring at being filled with SSD drives and processors instead of with nothing but hard drives... locality of data takes on a whole new meaning if you don't require data to be on the same sector of the HD, but rather want certain sets of data to be stored on storage chips located around the same processor chips to avoid having to overload your busses.
Then again, I haven't been in this game for so long, so maybe I'm overestimating the impact. Oldtimer opinion would be very welcome.
MySQL has people who are responsible for *designing* it? I'm shocked, Shocked.
If moderation could change anything, it would be illegal.
Solid state disks will never match with hard disks.
In 2025, this will be grouped with "640K RAM will always be more than enough." and you'll feel like such a Bill G.
Table-ized A.I.
It's more accurate to say that there will probably always be a tradeoff between slow and fast storage, there will probably always be a tradeoff between permanent and temporary storage, and there will probably always be a tradeoff between expensive and cheap storage.
In 20 years, I do not know what form slow, or cheap, or permanent storage may take. It may not be spinning magnetized platters. But I do know that in 20 years, every well-written database will have algorithms and data structures to deal with slow storage, permanent storage, and cheap storage.
I recently blogged on this, but essentially, as long as your average PHP developer thinks of MySQL as a glorified flat file system to place their serialized PHP objects, an always-available, pay-as-you-go distributed database is going to revolutionize application development in the coming years. For those that want to keep control of their data, HBase is coming along quite nicely.
I'm actually reading the article now, and as he is talking about design for a database taking multiple cores into consideration, etc, I'm wondering if the traditional lock approach used in MySQL (and most SQL databases as far as I know) somewhat kills parallel operations. Wouldn't the interbase approach work better in a parallel environment?
Again, I'm sure this is a stupid question, but perhaps someone could clue me in.
http://blindscribblings.com - Tasty pop-culture in conceptual fashion.
Worse than that. Part of that is the fact that RAID vendors tell people to go with drives from the same manufacturer to optimize performance, when in reality, they should be doing with the most diverse pool of drives possible, dramatically reducing the probability of multiple catastrophic failures at the same time (except through human error, some sort of physical impact, electrical surge, etc.). If a drive has a design defect, it isn't at all uncommon for them all to fail at n+/-k hours for some relatively small value of k. If all your drives are the same model from the same batch, your probability of losing the entire array is suddenly remarkably close to 100%. RAID with the same model of drive dramatically reduces reliability in the name of performance.
Even mirroring is basically useless for reliability if the drives are from the same vendor unless you swap out the mirrored drive daily so that no single backup has been operating more than half as long as the main drive. RAID is a nice idea in theory, but the reality is that with what seems to be a rapid decline in hard drive reliability over the past few years, in practice, performance notwithstanding, RAID just raises your power bill and guarantees you have two drives to ship back to the manufacturer for replacement instead of one.
As an aside, I had two personal hard drives die in the past week (and four within the last year). When you consider that I only have about 5 drives in regular use, that's alarming. Both of these drives were under a year old. One was a Seagate 500 GB drive in continuous operation in a heavily cooled tower (sides off the tower, four fans blowing outside air straight across the drive). Came into the house and it sounded like someone was using a radial arm saw. Cloned off enough data before the drive stopped reading any blocks at all, so my MythTV box is up and running again. That lasted about 9 months. The other was a Seagate 160 GB 5400 RPM laptop drive. Lasted 11 months and suddenly went into click-of-death mode where no data was accessible from the drive. It's going to be a long time and a lot of therapy before I'll ever trust anything important to a Winchester drive again... and Seagate went in a single week from my "high reliability, buy over all other vendors" list to my "not in a million years" list.
Check out my sci-fi/humor trilogy at PatriotsBooks.
Except that Bill Gates never said that. Bluefoxlucid did.
I'm sure he'll feel lots worse. While Gates gets hounded for something he never said, at least he has mountains and mountains of cash to console him.
Just as a reference Re: big applications and databases, the company I work for maintains a transport management system for the largest rail freight provider in Australia. The database for this application (Oracle 10g) currently stands at 1100+ tables (some containing a few million rows) 2400+ packages, and 450+ triggers. The client application is comprised of 2100+ java source files and 500+ interactive Swing forms. ;)
Your statement of "Databases need to scale to disgusting large numbers." is spot on.
Homonyms are fun!
You're driving your car, but they're riding their bikes there.
If I was migrating away from Oracle, MS SQL Server wouldn't be my first choice. Postgresql would. Given the choice between a free version that is similar to the original vs a product that is very different that I need to pay for it's a no brainer. Also take into consideration that for some database applications you're going to need some serious horsepower. You're limited in the number of procs you can have in a Windows system. Last time I checked, once you get past 8 processors Windows doesn't scale as well. Even linux doesn't do as well as Solaris, AIX or HPUX past a certain number of procs.
Oracle's RAC seems to be a better solution than MSSQL's approach. PostgreSQL (and EnterpriseDB) are working on a more RAC-like approach.
This is a good story about a company that successfully moved from Oracle to Postgresql. Basically, they had 2 database systems running Oracle, a data warehouse and an OLTP system. They moved their data warehouse over to Postgresql running on Solaris 10, then they used the licenses they no longer need for the data warehouse to boost the computing power of the OLTP system.
Open Source Java DAO Generator
I used to work for a casino company (the largest in the world at the time, and largely is).
Our slots database didn't just track totals for rewards credits and such, we kept every transaction ever. We can see how long you played at each machine, for how long, when you put money it, when you hit, when you took money out, etc. We know when you want to come in and play, which machines, etc. Consider that for every customer in a casino. Now consider than on a scale of 60 major casinos.
I'm not a DBA, so I was only responsible for resource managing on the servers.
http://blindscribblings.com - Tasty pop-culture in conceptual fashion.
I'm going along with the other two guys. I can't see what application would need more than 1000 columns in a single table. What really gets my is the MS SQL Server 2000 maximum of 8 KB ( SQL Server 7 was 2 KB) in a single row. Now there's a limitation that's badly designed. Oh, and you can define a table with 15 Varchar(8000) fields, just don't try filling every field. 1000 columns I could do just fine with (SQL Server supports 2048?) but the big killer is that you can't even use 2000 columns, because if you did, you would run out of space in the row, unless the average field size was under 4 bytes.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
"sudo" is that command which grants one user authorization to act as another user.
"pseudo-" is that verbal prefix which means "false".
I'm seeing language devolve in front of my eyes...
I wonder if I've been reading Slashdot too long - I can't tell whether this is a troll, a joke, a newbie, or an actual legitimate issue...
In my mind as a database engineer for a wall street bank, the biggest change in the near term that we forsee is data locality.
Given the amount of computing power on hand today, it may surprise many how difficult it is to engineer a system capable of executing more than a few thousand transactions per second per thread.
Why? Latency. Consider your average SOA application which reaches out to 4-5 remote services or dataserver calls to execute its task. Each network/rpc/soap/whatever call has a latency cost of anything between one and at worst several hundred milliseconds. Lets say for example that the total latency for all the calls necessary is 10 milliseconds. 1000/10=100 transactions per thread per second. Oh dear.
The amount of memory an "average" server ships with today is in the 32-64GB range. Next year it will be in the 64-128GB range. The average size of an OLTP database is 60-80GB.
So, the amount of memory available to the application tier will very soon be greater than the size of the database, warehouses excluded. Moore's law is quickly going to give the application tier far more memory than it needs to solve the average business state, exceptions noted.
The final fact in the puzzle is that for transaction processing, read operations outnumber write operations by roughly 20 to 1. (This will of course vary on the system, but that *is* the average.)
This situation is strongly in favor in migrating read only data caches back into the application tier, and only paying for the network hop when writes are done in the interests of safety. (There is a lot of research into how writes can be done safely asynchronously at the moment, but its not ready yet IMHO.)
Challenges exist in terms of efficient data access and manipulation when caches are large, performant garbage collection and upset recovery - but they are all solvable with care.
Its my opinion that in the near future large data caches in the application tier will become the norm. What has to be worked out is the most effective way of accessing, manipulating and administering that data tier and dealing with all the inevitable caveats of asynchronous data flow.
Some (not complete) examples of implementing this:
Relational Caches (there are many more):
http://www.oracle.com/technology/products/coherence/coherencedatagrid/coherence_for_java.html
http://www.alachisoft.com/ncache/index.html
Object Caches:
http://www.ogf.org/OGF21/materials/970/GigaSpaces_DataGrid_OGF_Oct07.ppt
http://jakarta.apache.org/jcs/
If you find you have to create thousands of columns response_0001, response_0002, ... response_4096, then you should probably realize that there's something wrong with your schema. It's just basic database normalization. (Though I suppose you might have a reason for doing it this way. But it sounds incredibly horrible.)
Yeah, like the other responders have mentioned. Sounds like you're doing it horribly wrong.
I don't know much about SPSS, but if you need to present the data in such a flat format, you can always store the data the proper way and create a view when you need to import it into SPSS.
I'm not sure if there is some sort of limit on the number of columns in Oracle views or not.
Open Source Java DAO Generator
Until recently, solid state storage devices have been treated as "disks". But they're not disks. They have orders of magnitude less latency.
For files, this doesn't matter all that much. For databases, it changes everything. Solid state devices need new access mechanisms; I/O based seek/read/write is oriented towards big blocks and long latencies. The optimal access size for solid state storage devices is much smaller, more like the size of a cache line. With smaller, lower latency accesses, you can do more of them, instead of wasting channel bandwidth reading big blocks to get some small index item. It's not RAM, though; these devices usually aren't truly random access.
It starts to make sense to put more lookup-type functions out in the disk, since getting the data into the CPU has become the bottleneck. Search functions in the disk controller went out of fashion decades ago, but it may be time to bring them back. It may make sense to put some of the lower-level database functions in the disk controller, at the level of "query with key, get back record". Cacheing at the disk controller definitely makes sense, and it will be more effective if it's for units smaller than traditional "disk blocks"
This could be the beginning of the end of the UNIX "everything is a stream of bytes" model of data storage. We may see the "database is below the file system" model, which has appeared a few times in mainframes, make it to ordinary servers.
UPDATE SlotMachines SET ChanceToWinPercent = 100;
COMMIT;
try {
machine.collectMoney();
player.run();
} catch(SecurityException e) {
player.resist();
if(player.isFast())
casino.exit(0);
else
player.yield();
}
Homonyms are fun!
You're driving your car, but they're riding their bikes there.
A database table is not an Excel sheet with fewer limits! Have some local wizard help you in the design. And if you use postgresql or any other full featured database, you can use views to retrieve your data in a format you need for analysis.
Had to comment, the reference to Jim Gray was a little weird? I was lucky to work with Jim and we were often talking about technology changes and enhancements. Now - see what for example Tandem did call "massively parallel" database! The system was already built to allow several cpus and several nodes to interconnect transparently, Jim did see how that could be used and how the database optimizer really could work. Of course making direct access to any disc faster will help, especially now when the SDD's are getting bigger but the theory is nothing new. Even SQLite can show you that and think systems where you have 32, 128 or even 256 bit flat, memory speed but storage backed world - will change the picture, or? But be careful, we have already gone through many iterations making part of the system faster, as fixed head disks and even indexing in solid state, and found that it may (will) create other problems, not always seen upfront (except by JG!)
I bought the same batch, but before setting up the RAID, I used each drive differently (ran benchmarks, copied files etc.). Some heavily, some light usage. The probability of them crashing at the same time should be smaller then if all of them get the same wear and tear.
I'm not insane! My mother had me tested.
Not trying to start a war here, but seriously Databases != RDBMS. It seems like no one knows that object databases have been around a long time too. In the context of the article, many of the points can be applied to all types of databases, but it's so focused on the RDBMS (no shock considering the author).
There were a multitude of issues in the past with object databases from agendas, performance, complexity, etc that put relational databases at the forefront. Hardware and the quality of object databases has more than caught up, so why are object databases so rarely used still?
One answer why object databases are ignored to a large degree is that people don't like to stray from the norm and tend to implement what they know. Another possibility is many people simply have never even heard of the concept of object databases. Further, in academia we almost exclusively focus on relational databases in most courses. Finally, legacy data is perhaps the biggest hurdle.
A corollary to the issues above is that there is an entire industry of DBAs and developers that fight learning something new. There's also mega corps with billions invested in the concept of the relational databases. I don't blame MySQL and some of the things said in the article because they're just trying to improve, but on the user level, it's amazing how much effort goes into adapting the RDBMS into the online world and resulting crazy architecture/technologies/code.
Object relational mappers are a great example of our unwillingness to leave the RDBMS world(unless you're working with legacy/existing data of course, but even then, investigate the possibility to migrate). Why do we need ORMs in the first place? They are a product of using relational databases. When I'm programming, I want to work in objects and not bizarre mapping layers, complicated DALs, etc. We spend so much time on mapping and layers to build bridges between a relational and object world at the cost of productivity and performance simply to continue to hang on to our old RDBMSs.
I've found that in most cases, object databases are faster for my projects. I've also tried related databases like grid/network databases. There are definitely cases where relational databases are also better, but I would use one over the other on a case-by-case basis. I find for the average case I've seen, hardware and architecture tips the balance in favor of object databases because of the way how we want to model things using objects anyway. If we look at a popular type of app right now, a social network... why use a relational database? Typically the associations and structures we make are objects and hierarchies or networks. Relational databases are ill suited at both. Instead, we start to develop hack and wtf schemas, rely to heavily on the app to sort out the data, or introduce object database-like concepts like table inheritance. This also forces us to introduce and learn yet another language.
SQL is a huge discussion in itself. I find SQL brilliant and easy to use, but nonetheless ill-suited for many tasks. Once cursors, user defined functions, etc. were introduced, the nightmare got worse. I find procedural and object constructs instead of set based constructs in SQL created by clients all the time as a result. This ends up crippling performance and instead of fixing the issues, decision makers will just throw more hardware at the problem or ignore it all together. There's also this myth that somehow SQL creates a way for the layman to query data in the database. This is true to a small degree, but has mutated into something not unlike "human readable" for XML.
I'm certified in SQL Server and Oracle, and Postgres is my home RDBMS of choice, so certainly I have a lot invested, but if I'm offered something that is better I will gladly abandon all my intellectual and time investments in these systems. I use whatever works the best for the task. After building several apps using Gemstone over the years, I have to cringe every time I return to Oracle or even w