Slashdot Mirror
Oracle Promises 100x Faster DB Queries With New In-Memory Option
Hugh Pickens DOT Com writes "ZDNet reports that Oracle's Larry Elison kicked off Oracle OpenWorld 2013 promising a 100x speed-up querying OTLP database or data warehouse batches by means of a 'dual format' for both row and column in-memory formats for the same data and table. Using Oracle's 'dual-format in-memory database' option, every transaction is recorded in row format simultaneously with writing the same data into a columnar database. 'This is pure in-memory columnar technology,' said Ellison, explaining that means no logging and very little overhead on data changes while the CPU core scans local in-memory columns. Ellison followed up with the introduction of Oracle's new M6-32 'Big Memory Machine,' touted to be the fastest in-memory machine in the world, hosting 32 terabytes of DRAM memory and up to 384 processor cores with 8-threads per core."
Especially upwind, but not 100x
still Emirates Team NZ only need to win one more race..to take back the Americas cup
This article:
Will Facebook, Twitter, LinkedIn Stay With MySQL?
Ellison is doing some public customer relationship management, and I guess this event comes along very timely.
" hosting 32 terabytes of DRAM memory and up to 384 processor cores with 8-threads per core. "
Let me be the first to point out the Beowulf possibilities with a few hundred of these clustered together. :)
With increasing surveillance on American citizens such database will provide security forces with instant profile of each person. Let's combine that with license plate scanning, cell phone tracking, sexual preferences and health records.
Now we can sleep well at night, our children are safe.
"Big Memory Machine"... So, they finally built Deepthought?
In-memory IO is grand, when that's your're bottleneck. Mine tends to be in the network level, so I use a local daemon for query result caching at the application level as "in-memory" speedup. The speedups are nice, but pricey. Color me unimpressed -- that's pink, BTW; I'm a Caucasoid your colors may vary, but only up to VARCHAR(20);
Uhg. Is "in memory" now just another buz-word? I guess we've come full circle back to Mainframe? Big memory banks are faster and better for a while, but then the bandwidth goes up and the price, reliability and scalability will favor distributed systems (as currently). I wonder which phase of the cycle quantum computing will favor: distributed / localized? You have to take into consideration your user distribution too...
So, eventually you'll want a hybrid system where the memory is distributed and cloned at each query-able interface, but still maintaining the entire dataset "in memory"...
...
SELECT * FROM earth WHERE answer LIKE "everything";
42 rows returned
As long as they keep with the extorsion techniques Oracle is famous for they can keep their hardware.
Wow, just catching up to Microosft. like SQL Server In-Memory OLTP http://technet.microsoft.com/en-us/library/dn133186(v=sql.120).aspx
And Column Store Indexes released 2 years ago. http://blog.sqlauthority.com/2011/10/29/sql-server-fundamentals-of-columnstore-index/
First, let me say that I would love to have a table option to keep a particularly heavily-hit table always in memory.
This ain't it.
From TFA, "Maintaining those indexes is expensive and slows down transaction processing. Let's get rid of them," Ellison remarked. "Let's throw all of those analytic indexes away and replace the indexes with in-memory column sort."
This merely minimizes the penalties of poor indexing and RBAR by making complete table scans on arbitrary columns faster. Apparently Mr. Ellison has forgotten his algoithmics and combinatorics - Oh, wait, no he didn't, he dropped out as a sophmore. Pity, because had he stayed, he would have learned that even with a 1000x slower storage medium, an O(log N) algorithm (index seek) will eventually beat an O(N log N) algorithm (column sort).
Thanks, Larry, but you want to make Oracle faster? Remove cursors from the core language, and although that alone won't "fix" it, you'll see all the hacks who can't think in set-based logic drop out overnight.
Some of that memory corrupts, Shhhh !! Who is going to know !! Back in the day INT2 would save our asses but that is no more !! Today it is better to let the errors creep than to report !! And no, ECC does not save itself !! ECC lets vendors sell defective RAM !! Bet your life on that !!??
32 TB is 256 T bits which is 2 ^
Finally, the reason Oracle requires joining to dual is revealed.
whats OTLP?
https://en.wikipedia.org/wiki/SAP_HANA
14 hours ago, itnews.com.au runs a story (promptly picked up by /.) about how the social networks are staying with MySQL. In the article, it is suggested that the switch to MariaDB by some Linux distros is a "political move", and that Google's switch might be a retaliation against an unrelated lawsuit from Oracle. Also, it's mentioned (twice, with the same wording) that the Mozilla foundation is "upgrading from MariaDB to MySQL 5.6" (emphasis added).
7 hours ago, itnews.com.au runs a story (promptly picked up by /.) about how Oracle's 12C database will be 100x faster, despite the fact that we only have Oracle's CEO word for it.
Now that's what I call an Oracle-friendly site (or two?)
Knowledge is power; knowledge shared is power lost.
Just you hope Oracle maintains the batteries properly, especially since an emergency save-to-disk is going to take more than a few minutes...
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
Now if we just put a few of those in a beowolf cluster...
Oracle is closed source and developed by professionals. Take a hint, MySQL.
Is it 4 or 8 i7s?
Columnstore databases such as MonetDB and their commercial spinoff Vectorwise (now Actian) already showed this can be achieved with open source and proprietary code.
Support Eachother, Copy Dutch Property!
For EEC Systems/SuperSpeed.com & it placed as a finalist 2 yrs. in a row @ MS TechEd 2000-2002 in its hardest category: SQLServer Performance Enhancement (by placing the db devices & tables (temp too)) into RAM via the SuperDisk II program (which I also increased SuperCache II's effectiveness by up to 40% for also) & it also reviewed VERY POSITIVELY in Windows IT Pro magazine that same 1st year 1996 (then Windows NT Magazine) - & yes, for performance of course, it works!
* Pretty much "goes without saying" that IF/WHEN you avoid diskbound latencies, you're going to speed up!
(Is THIS precisely the same? Not quite - they're distributing the data across diff. systems from what I read/iirc, & eliminating indexing - which is the ONLY part I question here...)
APK
P.S.=> "What has been will be again, what has been done will be done again; there is nothing new under the sun." - Ecclesiastes 1:9 ...
... apk
I state that since Mr. John Enck & I BOTH came up with the idea independently, specifically for DB work, & posted it to their "FAQ" section of the EEC Systems/SuperSpeed.com website in fact - I was doing that type of thing with DBase III temp/scratch tables way, Way, WAY before it also albeit native ramdisk softwares (DOS usually) -> http://ep.yimg.com/ty/cdn/superspeed/ScReadMe41.txt
(Hence the biblical chapter/verse quote I closed off the post you just replied to, since imo, it too "holds true" here)
I suspect not ONLY for Oracle doing pretty much what I did, but very possibly OTHERS DOING IT BEFORE MYSELF doing it too!
HOW/WHY?
Heck: Since I am FAIRLY certain I was "no 1st" here because it's SO DAMNED OBVIOUS to do for performance purposes per less latencies in RAM vs. being diskbound... (but, you never know!).
---
* I do, however, also see a LOT of usage of in-memory DB tech, Virtual Machine, Terminal Server applications the past 1/2 decade now in "industry" via SSD usage, which amounts to damn near the SAME thing too - & it seems the "youth of today" have stumbled upon it, finally, lol!
(Which is yet later another way I applied this, not with software ramdisks in software this time, but later, in hardware via CENATEK's "RocketDrive" PC-133 SDRAM 4gb/16gb PCI 2.2 bus striped-spanned hardware ramdrive cards, & later still doing the same with Gigabyte IRAM 4gb/16gb striped-spanned DDR-2 RAM SATA II bus PCI Express based ramdrive boards - this idea/technique's "taken off" most recently though on the areas noted, using FLASH based SSD's... the only REAL difference, but the principals are the same - albeit FASTER in system memory though using ramdrive softwares!).
---
Thus - I really don't *THINK* Oracle CAN patent this technique to be quite honest... why? Well, what the "artsy/fartsy" people are wont to state, in "It's been done..."
---
Heh - This ALL reminds me of the "Outer Limits" episode "The FINAL exam" - when Seth the main character says:
"When a Science is ready? It can't help but make the next discovery ..."
APK
P.S.=> Bottom-Line: Yes - I'd be VERY surprised if my idea came before anyone else on this note on the entire planet & mainly, since BOTH myself + Mr. Enck (technical editor for Windows IT Pro/Windows NT mag) both put it up on their website as a technique for superior performance purposes (albeit I had been applying it & told the folks @ EEC/SuperSpeed to do so too, on paid contract, while increasing SuperCache I/II's efficacy by up to 40% in code too)... apk
Ellison followed up with the introduction of Oracle's new M6-32 'Big Memory Machine,' touted to be the fastest in-memory machine in the world, hosting 32 terabytes of DRAM memory and up to 384 processor cores with 8-threads per core."
This should be on my desk in about 4 years.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
...how do you replicate this to an offsite hot standby, so that you don't look stupid when the power goes out? Or whatever other disaster occurs, because you just KNOW a single basket full of this many eggs is going to attract tornadoes, Cat 5 hurricanes, earthquakes, and meteors. Also coffee spills, hungry rodents, and burst pipes.
Also I believe OTLP == Oracle Typical Ludicrous Pricing
It's supposed to be completely automatic, but actually you have to press this button.
Comment removed based on user account deletion
for a university database class. Oracle automatically started when windows XP loaded. 10g used about 200 MB of RAM. back then i only had like 512 MB of RAM. wow. Hey, this was in 2007. lol it will be interesting to see how the new in-memory option runs on fast computers with lots of RAM and a 64 bit operating system
How is this going to be different from MemSQL? http://www.memsql.com/
So a 32GB RAM stick costs ~$1000. 32TB of RAM would cost $1,024,000. Given that RAM is about 1/4 of typical system build costs. This system costs about $4M raw. Oracle's markup is easily 1000% so $40M for this beast. And then the yearly licensing cost for Oracle at $2500/core: ~$1M.
At that cost you could probably develop your own MemCache/BDB cluster.
Custom electronics and digital signage for your business: www.evcircuits.com
Oracle will soon release the M6-96 server. That is three such M6-32 servers connected to a huge 96-socket server, with 96TB RAM and 9.216 threads. It will use the "Bixby" interconnect to do this. Then you can run huge databases from RAM. 96TB is a lot. Solaris 11 had to be rewritten to be able to scale to 100s of TB of RAM. No other OS can handle such amounts of RAM.
The Linux servers such as SGI Altix or SGI UV1000, might have 1000s of cores, even 10.000s of cores, but they are clusters. For instance, the ScaleMP Linux server has several 10.000s of cores and 64TB RAM, running a single Linux kernel image. But it uses a software hypervisor to trick Linux kernel into believing it is running on a single server. But it is a cluster, google for more information. This Oracle M6-32 is a single server. Not many small nodes.
So...in memory database = a fancy new term for a database that's (essentially) loaded onto a giant ram-disc?
As an added bonus I'm pretty sure that option would save you a mint in licensing fees.
Half the system cost is batteries to prevent total data loss t thge first power glitch.
Per my other reply (& biblical chapter/verse + "the Outer Limits" quotes there too) -> http://developers.slashdot.org/comments.pl?sid=4250479&cid=44923331 - I was doing stuff LIKE this (very like it) even on DOS + ramdrives in software even using DBase III (for temp/scratch space ops & indexing).
* "WoW", now specifically on what YOU said - "That's mgt. for ya!" or, rather, mgt. that has NO CLASSICAL CS TRAINING!
(Or could be that when many, especially EARLY ON in the art & science of computing (moreso than now, but it's STILL a PROBLEM imo) didn't possess formal CS training & when MOST mgt. never even DID THE WORK of their subordinates in computing ("hands-on in the trenches" to give them perspective + experience...))
---
Doesn't surprise me, but... then again, it does - I mean, honestly: How could one NOT see/realize it?
(Seriously - Even in theory alone!)
Imo - the performance boosts are SELF-EXPLANATORY + SELF-EVIDENT, For Pete's sake, when driven from a faster media (pure electrical vs. mechanical latencies improvements) & common-sense... same reasons (almost) why a std. Mustang V-8 4.6L isn't faster than it's "big bro" GT500 (supercharged).
---
In fact - What you state's ALMOST like XEROX not realizing their GUI interfaces were going to be BIG (which Apple & MS "stole" & made fortunes from is much the same here...)
---
Currently - Here's how I use & apply "True SSD's" (not FLASH based) @ home even:
I move the following off my wd Velociraptor SATA II 10,000 rpm 16mb buffered harddisks that are driven off a Promise Ex-8350 128mb ECC ram caching raid sata 1/2 controller (which defers/delays writes via said cache, & also lessens physical head movement on disks & this is where I am going to make it even faster via lessening its workloads, read on & reduces fragmentation as well in the same stroke - "bonus") onto my 4gb DDR2 Gigabyte IRAM PCIExpress ramdisk card 2006-present (& before it, a CENATEK "RocketDrive" 4gb PC-133 SDRAM based one on PCI 2.2 bus circa 2002-2006):
---
A.) Pagefile.sys (partition #1 1gb size, rest is on 3gb partition next - this I didn't do on software ramdrives though)
B.) OS & App level logging (EventLogs + App Logging)
C.) WebBrowser caches, histories, sessions & browsers too
D.) Print Spooling
E.) %Temp% ops (OS & user level temp ops environmental variable values alterations)
F.) %Tmp% ops (OS & user level temp ops environmental variable values alterations)
G.) %Comspec% (command interpreter location, cmd.exe in this case, & in DOS/Win9x years before, command.com also)
H.) Lastly - I also place my custom hosts file onto it, via redirecting where it's referenced by the OS, here in the registry (for performance AND security):
HKLM\system\CurrentControlSet\services\Tcpip\Parameters
(Specifically altering the "DataBasePath" parameter there which also acts more-or-less, like a *NIX shadow password system also!)
---
* All of which lessen the amount of work my "main" OS & programs slower mechanical hard disks have to do, "speeding them up" by lessening their workload, fragmentation, and speeding up access/seek latency for the things in the list above too.
(Thus - HDD's concentrate on program &/or data fetches that are still hdd bound (& not kernelmode diskcaching subsystem cached in 4gb of DDR3 system ram here either yet) done on a media that has no heads to move, & thus, more mechanical latency + slower seek/access as you get on hard disks + reduced filesystem fragmentations due to that all, also & it works, for BETTER/SUPERIOR overall performance even for HOME stuff!).
---
MS' "ReadyBoost" isn't THAT much different either when you come right down to it, vs. that technique of mine, on the software
It's still a product living in the 1970s. No decent debugger (don't even mention SQL developer or Toad as I am using both), and whenever my huge Oracle stored procedure erred, I still couldn't correctly pin-point the line number where the error occurred due to the comment lines screwing the output.