MemSQL Makers Say They've Created the Fastest Database On the Planet

mikejuk writes "Two former Facebook developers have created a new database that they say is the world's fastest and it is MySQL compatible. According to Eric Frenkiel and Nikita Shamgunov, MemSQL, the database they have developed over the past year, is thirty times faster than conventional disk-based databases. MemSQL has put together a video showing MySQL versus MemSQL carrying out a sequence of queries, in which MySQL performs at around 3,500 queries per second, while MemSQL achieves around 80,000 queries per second. The documentation says that MemSQL writes back to disk/SSD as soon as the transaction is acknowledged in memory, and that using a combination of write-ahead logging and snapshotting ensures your data is secure. There is a free version but so far how much a full version will cost isn't given." (See also this article at SlashBI.)

Comments

A nice approach perhaps...

[alphatel]

It sounds cool, but we can get 200k iops on Raid10 SSD without degradation.

Re:A nice approach perhaps...

[tomhath]

Price/performance is a better question. If it's fast enough that you don't need the Raid 10 SSD then it could be a good choice. Throw hardware at any DBMS and you'll get good throughput.

Re:A nice approach perhaps...

[hcs_$reboot]

So, two guys meet and churn out some code that cache most of the DB work in memory. Great. But MySQL has a MEMORY engine and is pretty well optimized (eg keep indexes in memory, does some caching as well)... the hardest part is probably its setup: setting the right options in MySQL to achieve top performance is not easy.
Besides, the "caching" or equivalent work is not the most difficult part of a DBMS, by far: What about the algorithms to "compile" queries in order to use indexes and perform the JOINS optimally. A good algorithm beats the architectural benefits of a "cache" oriented DBMS.

Re:A nice approach perhaps...

[guruevi]

You can get more than 10M IOPS on certain RAM-based SSD, they're just mighty expensive.

Re:A nice approach perhaps...

[hvm2hvm]

I wonder how fast MySQL, postgresql or any other disk based DB server would work if put on a ram disk? Of course it would make it riskier but it could work if you would use a memory mapped file s.t. it will dump new data as fast as the HDD can write.

Re:A nice approach perhaps...

[Anonymous Coward]

AFAIK, that's how any decent DB system works.

Yes, but he said MySQL...

Re:A nice approach perhaps...

[godefroi]

To be ACID the database must flush new data to disk (all the way to the platters, through the RDBMS cache, the OS cache, the controller cache if there is any, and the on-disk cache) at every transaction commit. If the RDBMS just puts the data in memory, and then the power goes out, then your data is lost, and the system isn't ACID.

Re:A nice approach perhaps...

[Tough Love]

All of computer science is just an exercise in caching, don't you know.

Re:A nice approach perhaps...

[rgviza]

Ask CCP (icelandic game developer). They put Eve's entire database on RAMSAN and it's a massive db with a lot of performance demands. There's actually a case study published about it.
http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onlines-server-model/
http://www.microsoft.com/casestudies/Windows-HPC-Server-2008/CCP-Games/Leading-Game-Publisher-Selects-Windows-HPC-Server-to-Scale-to-One-Million-Players/4000002990

There are various whitepapers about the performance increase of using ramdisk. Most dbs improve by going around 74% faster with no other optimization.

RAMSAN is not volatile so if you lose power, it doesn't lose data, which makes flushing every transaction to disk unnecessary. The RAMSAN _is_ the disk.

Ya Don't Say!

[Rary]

Really? Accessing RAM is faster than accessing a disk? What a novel discovery!

It seems to me that MySQL can also be run in memory. Apparently that's how the clustered database works (or used to work). I've never tried it, but let's see some benchmarks between MemSQL and an entirely memory-based MySQL.

Re:Ya Don't Say!

[JimboFBX]

I was going to say, how does this perform on large queries in a large database.

Re:Ya Don't Say!

[lucifuge31337]

It seems to me that MySQL can also be run in memory. Apparently that's how the clustered database works (or used to work).

Absolutely correct. NDB Cluster. It's quite fast, even on older hardware providing you have enough RAM to hold your database.

Re:Ya Don't Say!

[errandum]

That and memcached (I think that's the name).

This comparison is far from fair... Is it ACID? Or eventually synchs up? How does it compare with other memory based DB's?

Comparing it with a slow relational DB will not give you any kind of credibility.

Re:Ya Don't Say!

[MobileTatsu-NJG]

Isn't the implocation that in this case there's a lot less time between the transaction getting made and that data being committed to non-volitile memory?

Re:Ya Don't Say!

[bill_mcgonigle]

Not just that - you can get a FusionIO ramdisk device for really big databases and get performance that's somewhere between SSD and memory. Those are all battery backed and such, so no monkeying around with whether the ACID was done right or not.

Re:Ya Don't Say!

[gman003]

It's a bit more complex. There's four main ways to do MySQL storage in RAM (which I know of because my current work project is a MySQL application).

First, the NDB Cluster system is there, which is what you've mentioned. That's basically just a MySQL frontend to a distributed, memory-based NoSQL database, though. Convenient, but not truly "MySQL".

The second is using the "Memory" storage engine, where it actually stores a normal MyISAM table in memory. However, this is a surprisingly crappy option, because it uses table-level locks for writing, so parallel write performance is only marginally faster than disk.

The third is to store regular InnoDB tables on a ramdisk. This can be crazy fast, but it also means that if your server crashes or loses power, you're *fucked*

The fourth is to use Memcached, which isn't really a MySQL thing at all. You're basically just caching data in a memory-only NoSQL database, at the application level. This is actually what we ended up doing, because all the others are pretty crappy options - Cluster is the best one, but the hardware requirements are higher than we could justify spending given our performance requirements. Shoving memcached onto the web server (which has RAM to spare) and setting certain queries to cache their results there sped things up significantly, at minimal cost.

As far as I can tell from the summary (I refuse to read the articles for such a blantant slashvertisement), this "MemSQL" doesn't do anything you can't do by configuring MySQL properly, although they likely optimized some rarely-used modules to make them faster.

Re:Ya Don't Say!

[lucifuge31337]

No foreign keys with NDB

Also correct. Your point? Other than that specific tools were made for specific jobs.

Re:Ya Don't Say!

[arth1]

The third is to store regular InnoDB tables on a ramdisk. This can be crazy fast, but it also means that if your server crashes or loses power, you're *fucked*

Not necessarily. There are battery-backed volatile RAM devices that can last for days, and also non-volatile RAM devices like F-RAM and MRAM.
Battery backed volatile RAM can even be considered "cheap" - if the bottleneck are in tables small enough to fit on these, or the amount of overall writes is so high that placing the innodb logs there makes sense, it can be cheaper than a RAID10 or 50 of high-speed SAS drives.

The HyperCard / ACARD drives, for example, are only $300 plus RAM. And if the worst happens, you can even dump the RAM to a CF card before the battery runs out.

Re:Ya Don't Say!

[gman003]

I was referring to software-based ramdisks, not RAM-based SSDs. Although I suppose there's not much of a performance difference - the only difference is durability.

Re:Ya Don't Say!

[Paradise Pete]

I am a ramdisc fan since Mac+.

I'm gonna call BS on this one. Why would a ram disc need a fan?

Re:Ya Don't Say!

[arth1]

The biggest issue with RAM drives are their cost.

Yes and no. If you can fit the Innodb writeahead-logs and a few of the worst bottleneck tables on, say, an 8 GB ram drive, it's a bargain.

HyperDrive: $300
2 * 4GB 240-Pin DDR2-800 SDRAM ECC: $234
16 GB CF card for backup: $30
Total: $564

That's downright cheap compared to what a RAID 10 or 50 of SSDs or short-stroked 10k/15k rpm drives would cost.
If it solves a bottleneck, it could be a big money saver.

Re:Ya Don't Say!

[Alex Belits]

Didn't you get the memo? There are no large databases anymore, all database servers are supposed to have more RAM than the size of their database.

*** BARF!!! ***

Re:Ya Don't Say!

[Alex Belits]

You probably wouldn't want to use software RAM drives for anything serious, if your RAM has no error correction. Most hardware RAM drives use ECC (or at least buffered) RAM.

All server-class hardware always uses ECC RAM.

Re:Ya Don't Say!

[Hognoxious]

MySQL is not webscale because it uses joins.

Re:Ya Don't Say!

[hvm2hvm]

Well I suppose the point was you cannot say NDB is a viable alternative to MemSQL as a memory based DB if NDB has limitations over MemSQL. That's like when you're going to buy something and there is an asshole of a salesman that tries to sell you something you don't need/want because "it's just as good".

Re:Ya Don't Say!

[Anonymous Coward]

For those who don't get the reference:

http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale

Re:Ya Don't Say!

[drsmithy]

Given how trivial and relatively cheap it is to put 192GB+ RAM into a server these days, there's a lot of truth in that statement, whether you like it or not.

Re:Ya Don't Say!

[mwvdlee]

TFS states that transactions are writen to disk after being "acknowledged" in memory.
I assume that means transactions are written to disk only after the database reports back a succesful commit.
So failing to meet the D of ACID compliancy.

Re:Ya Don't Say!

[catmistake]

I am a ramdisc fan since Mac+.

I'm gonna call BS on this one. Why would a ram disc need a fan?

MacOS System 6 had RAMdisk applications available for it (one was called AppDisk), and by System 7 this functionality was built into the OS via a system control panel. The software allowed you to donate whatever RAM you had available to a disk that mounted on the desktop. You could use it for cache files, or copy entire applications to it to run at blazing speeds off the RAMdisk instead off the comparatively slow SCSI-1 HDD, or the mind-numbingly slow floppy disk drive. In 1989, when RAM was prohibitively expensive, if you had a Mac IIfx (not publicly available until 1990) or Mac SE/30 and were very wealthy, you could have a desktop with 128MB of RAM (along with the MODE32 control panel that allowed the SE/30 to see all that memory, or the ROM from a Mac IIfx or Mac IIsi which accomplished the same thing, unless you ran A/UX which was natively 32-bit clean), and with the RAMdisk software you could designate amounts up to whatever the system didn't need to be used as a RAMdisk... say... about 120MB, which at the time would have been about as large as the biggest HDDs available to consumers. You can still find this software on Gamba's site.

To answer your question, the Mac Plus was an AIO or all-in-one computer. I'm not sure if you are old enough to know or remember what a cathode ray tube is, but the Mac Plus used one as a display, and it generated a substantial amount of heat, requiring a fan to cool the machine.

Also, RAM itself will generate some heat, usually not enough to need its own fan, and RAM in a Mac Plus didn't have a dedicated fan. A RAMdisk is not the same as an SSD, which runs off the disk bus. Like RAM, a RAMdisk runs off the system bus... generally much faster than the disk bus.

Re:Ya Don't Say!

[dave420]

TFV says it's fully acid compliant.

Re:Ya Don't Say!

[mwvdlee]

TFS implies it doesn't.
TFV is an advertisement.

Re:Ya Don't Say!

[SteveAyre]

All recent versions of NDB can store data on disk too. RAM-only is a very old (5.0) requirement.

NDB's real advantage doesn't come from being in memory (if you have enough RAM you can get a massive speedup on standard MySQL by setting large enough buffers to keep a cache of most of or the entire database in memory).

It comes instead from auto-sharding, spreading data out over multiple nodes and having multiple servers transparently searching data for you at the same time so that your query runs much faster than just one server could manage. And you can easily add more nodes as your load increases so the system nicely scales up, even on writes.

Re:Ya Don't Say!

[DarkOx]

Wake me up when that is 800GB to a few TB. Then you can say. This might shock you to learn but some business uses their databases to drive more than just web forums.

Re:Ya Don't Say!

[xelah]

I don't think that's something which can be changed, except by changing the hardware. The starting point is this: When a COMMIT is made all changes have to be written to the write-ahead-log before a success response can be returned to the client. The WAL is written sequentially, and so if you're using ordinary disks and are sensible you give it its own set of spindles (RAID1, say). That means that between each write you have to wait for one disk rotation - you append to the log, you process the next transaction, then you have to wait for the disk to rotate to just after where you finished writing before you can write the next one. So you can do 1/15k transactions per minute with this basic setup.

You can do things to make this faster. You can write several transactions at once, and you can put slight delays in to transaction commits to wait for others to bundle them with (PostgreSQL I believe will do the first and can be configured to do the second). You can use battery backed caches in your RAID system, which will have much the same effect (and leave you limited by disk bandwidth and cache size). You can use SSDs that don't need to seek.

I can't see anything in TFA that MemSQL is supposed be doing differently here, or anything it CAN do differently. From TFA: 'The key ideas are that SQL code is translated into C++, so avoiding the need to use a slow SQL interpreter, and that the data is kept in memory, with disk read/writes taking place in the background.'. The first I'm not too sure I understand (presumably they're not turning it in to C++ and then passing it through a C++ compiler....) but maybe we can blame the journalist for that. Or maybe they've just reinvented prepared statements. The second is what databases do anyway - except, of course, for the WAL and when you're reading data which isn't in memory. Perhaps what they're doing is flushing the WAL after the commit has returned to the client - which makes the database very much not ACID, and is also something that other databases can be configured to do if you don't care about your data.

Potentially what they could do, though, is to have designed all of their data structures, algorithms, locking and so on around the assumption that everything is in memory. There are big differences in the best query plan to use when data is in memory vs on disk, and traditional databases don't necessarily make the right choices. They try, but may for instance use table scans for queries which return a large proportion of the rows in a table because sequential IO is faster, when they should be using indexes if the data is in memory. And BTrees and the way data everywhere is split in to pages is something traditional DBs do because that works well even when most of your data is on disk. So maybe that's what they've done differently that other DBs haven't already been doing.

Re:Ya Don't Say!

[drsmithy]

Wake me up when that is 800GB to a few TB. Then you can say. This might shock you to learn but some business uses their databases to drive more than just web forums.

You can already put a TB of RAM into a server if you want. If you really need to have that amount of data with next to zero latency, then the cost (which is still relatively low) is unlikely to be much of a stumbling block.

It clearly will shock you to learn that most databases are well under a couple of hundred GB in size.

Re:Ya Don't Say!

[sincewhen]

False. The Macintosh Plus was convection cooled.

A compact Mac, the Plus had a 9-inch 512×342 pixel monochrome display with a resolution of 72 PPI, identical to that of previous Macintosh models. Unlike earlier Macs, the Mac Plus's keyboard included a numeric keypad and directional arrow keys, and, as with previous Macs, it had a one-button mouse and no fan, making it extremely quiet in operation.

Source: Wikipedia

Re:Ya Don't Say!

[SQL Error]

Our database is 300TB. So.... Yeah.

Re:Ya Don't Say!

[cheekyboy]

if 90% of your DB is just strings, like VAR CHAR 255 for all names/addresses/streets etc..

Then yeah, your 900TB database is mostly fluff, and indexable in a much smaller index.

So wtf are storing that needs >5tb? All mobile phone calls? which is just blobs with references

Re:Ya Don't Say!

[rjstanford]

And for systems developed with only one access application to the DB (actually fairly common these days), the sacrifice of having to keep referential integrity rules in the application layer may be outweighed by the appeal of not having to put all sorts of sharding rules into the application layer because the core DB just got faster. There are always trade-offs when you start getting close to performance boundaries of an existing solution. And if you're not - then why change?

Re:Ya Don't Say!

[fatphil]

Were only that the case. If you can't have ECC, then at least have a reality distortion field instead:
http://everythingapple.blogspot.com/2004/07/virginia-tech-calls-original-power-mac.html

Re:Ya Don't Say!

[QuantumRiff]

I find it funny how easy it is to order an AMD system with 256GB of ram (or even 512GB, just much more expensive) yet the Intel ones all seem to max out at 192 or really, really expensive 384GB.. I know it has to do with the memory controllers, but our loads are very, very memory dependent..

Re:Ya Don't Say!

[catmistake]

Nice catch... I knew the first two Macs were fanless, and that Apple introduced a fan into the compact Mac, but wasn't sure if it was the Plus or the SE that was the first with a fan. Now I know.

Re:Ya Don't Say!

[Zak3056]

I find it funny how easy it is to order an AMD system with 256GB of ram (or even 512GB, just much more expensive) yet the Intel ones all seem to max out at 192 or really, really expensive 384GB.. I know it has to do with the memory controllers, but our loads are very, very memory dependent..

The Dell PE820 (a 4-socket intel server) supports up to 1.5TB of RAM. With 2-CPUs, though, it's only 768GB...

Re:Ya Don't Say!

[Junta]

IBM x3950 x5 can do 3TB of ram with Intel processors.

Re:Ya Don't Say!

[Xiaran]

Many years ago I used to work on a high transaction real time system running on QNX. We had a transaction type log that was in a Watcom SQL database... used to run that on a RAM disk with clever flushing code that made sure it got to a disk when the system had non-hard realtime deadlines. It was one of my favourite bits of code to tinker with and watch working.

Re:Ya Don't Say!

[Bigby]

In most DBs, VARCHAR prevents too much fluff and do you know how many transactions a bank can do every day? What about a stock exchange?

Re:Ya Don't Say!

[Fnord666]

MacOS System 6 had RAMdisk applications . . . generally much faster than the disk bus.

Whoosh!

Re:Ya Don't Say!

[Fnord666]

To answer your question, the Mac Plus was an AIO or all-in-one computer. I'm not sure if you are old enough to know or remember what a cathode ray tube [wikipedia.org] is, but the Mac Plus used one as a display, and it generated a substantial amount of heat, requiring a fan to cool the machine.

Yeah, these were a real joy to work on. Remember all the warnings about not opening your TV due to the dangerous voltages, etc? All of that was right next to the floppy drive/memory/motherboard you were replacing. The first couple of steps in the service manual after opening the case were to discharge various parts of the display so that you didn't accidentally electrocute yourself.

Re:Ya Don't Say!

[adisakp]

Just one point... HyperDrive is slower (even though it's RAM) than many SSD's simply because it's not evolving as quickly: 175MB/s Read rate + 145MB/s Write rate + 65,000 IOPS

The price point puts it above Vertex 4 with much more storage capacity: 550 MB/s Read Rate + 465-475 MB/s Write Rate + 120,000 MAX IOPS

The Vertex 4 is just a lot faster. There's no reason we can't have a fast hardware RAM drive but unfortunately, the Hyperdrive doesn't seem to provide much if anything over a smaller and more rugged SSD. Also, a number of the higher end and nearly all the enterprise SATA SSD's now come with a super-capacitor on-board to allow completion of all writes in the cache if power is lost.

Re:Ya Don't Say!

[catmistake]

To answer your question, the Mac Plus was an AIO or all-in-one computer. I'm not sure if you are old enough to know or remember what a cathode ray tube [wikipedia.org] is, but the Mac Plus used one as a display, and it generated a substantial amount of heat, requiring a fan to cool the machine.

Yeah, these were a real joy to work on. Remember all the warnings about not opening your TV due to the dangerous voltages, etc? All of that was right next to the floppy drive/memory/motherboard you were replacing. The first couple of steps in the service manual after opening the case were to discharge various parts of the display so that you didn't accidentally electrocute yourself.

Right... and couple that with the way the motherboard was tucked in and connected... discharge the CRT, remove the disk cage and disconnect the scsi connector, remove any PDS cards, disconnect mobo from the power/analog board and removing the board wasn't fun or easy and putting it back and reconnecting it all wasn't any easier. Doing this a few times in a row to find a bad RAM stick was, to say the least, a frustrating exercise. But to this day I still think the compact Mac was one of the most attractive machines to ever to sit on a desk.

Re:Ya Don't Say!

[DarkOx]

I can only imagine the incredible data volumes banks and sock exchanges would need to keep online. I have had some experience working in data warehouse architecture for a nation wide retail chain and the transaction processing systems we were pulling from typically needed to keep 800GB+ online between accounting close events. We obviously grew pretty fast in total size as we swallowed all that data in the DWH.

Re:Ya Don't Say!

[arth1]

Just one point... HyperDrive is slower (even though it's RAM) than many SSD's simply because it's not evolving as quickly: 175MB/s Read rate + 145MB/s Write rate + 65,000 IOPS

The price point puts it above Vertex 4 with much more storage capacity: 550 MB/s Read Rate + 465-475 MB/s Write Rate + 120,000 MAX IOPS

Keep in mind that databases tend to be bottlenecked by random writes. The sustained sequential speeds are less relevant.

Also, you cannot RAID an SSD without losing TRIM, which for a busy database means you're going to feel write amplification pains pretty quickly. In a bad case, a single insert can mean a dozen small writes, each of which potentially needs a a full sector erase and copy - you could look at several-seconds long hiccups.
RAM drives don't have this problem.

Re:Ya Don't Say!

[prionic6]

I'd say durability is relative... If a commit is written to a commit log on disk before it is reported successful, it may be _more_ durable than holding it in memory, but it can still get lost (drive failure, catastrophic event at the site of the server). So while there may even be a few orders of magnitude between the "durability values" of the two approaches, I'm not sure it is correct to say one is durable, while the other is not.

Re:Ya Don't Say!

[Alex Belits]

Well, Power Mac is not a server-class hardware.

Re:Ya Don't Say!

[metaforest]

Classic macs prior to the SE did not have fans.
They used convection cooling which worked very well unless you did something dumb.

Examples of dumb things(tm):
a) covering the top vents with a ring binder (saw that start a fire in an enginnering lab. bonus points: it was an apple engineer that did this dumb thing)
b) stack books/diskdrives on the left side of the machine thus blocking the feed vents that cooled the video control board.
c) installing an aftermarket HDD kit in a classic mac without installing a fan kit. The HDD kit messed up the airflow inside the case.

SE, SE/30, and all Mac II derivatives of desktop machines had fans. The 20th anniversary Cube was the one exception.

Re:Ya Don't Say!

[fatphil]

I never believed for one minute that the reality distortion field was so limited as to only be able to cover RAM choice.

I've always been a POWER fan (I've even worked for Freescale), and I found VT's "run it for one benchmark then dismantle it" to be an insult to the supercomputing field. Thank Knuth I'm not a US tax payer.

Re:Ya Don't Say!

[Paradise Pete]

I'm not sure if you are old enough to know or remember what a cathode ray tube is

I owned an original 128K Mac. I also remember black and white TV, rotary telephones, phone numbers like "Greenleaf 5-9709", and the Beatles appearing on Ed Sullivan. So yeah, I remember what a cathode ray tube is. :-)

Re:Ya Don't Say!

[rbrausse]

[completely off-topic]

I saw your comment while meta-modding and ROFLed. There was a time Apple had no fans? And they had to introduce the fan(bois)?

The marketing effort was successful :)

Looks good for testing

[All_One_Mind]

MemSQL is definitely good news, and hopefully it will encourage the MySQL team to play catch up with it's performance. Maybe it will provide an improved web experience if it gets wide adoption and deployment. As a long time SysAd/webmaster/developer, I'm certainly interested, but for obvious reasons I'm not putting any business critical servers onto something this fresh and new, regardless of performance benefits. I think I'll download a copy and use it locally for testing, but like any software, there are going to be bugs, maybe even data loss or security issues that may emerge on certain server setups. I'll see how the changelog looks in 6 months or a year before considering it for my mission critical servers. Regardless, kudos to the developers. Grabbing my download before/if it gets /.'ed

Re:Looks good for testing

[realityimpaired]

As a long time SysAd/webmaster/developer, I'm certainly interested

At the risk of sounding incredibly condescending....

If you were really a sysadmin who could benefit from that kind of speed improvement, you'd know that it's possible to achieve that level of performance with MySQL already, by either running it from memory or by using a fast hard drive array. The simplest/cheapest option to drastically improve MySQL performance is to throw a large amount of RAM at a system and point MySQL at the memory. MySQL can be configured to keep the database in active memory and sync to the disk on a regular basis, which is almost exactly the kind of behaviour described for MemSQL... for an exceptionally large database that can't be stored in system memory, I imagine that the advantage that MemSQL is boasting would evapourate. There are other ways to go about doing it, such as running a fast disk array or a cluster, in order to get around the limitations of using RAM, but ultimately the prime determining factor for speed in MySQL is speed of access to the database file itself.

Re:Looks good for testing

[errandum]

I'd love to see their tests when this DB needs to go into swap / pagefile. It's double the slowdown, needs to write into the swap (disk I/O) and then sync the DB (disk I/O again).

I can't, for the life of me, understand where this will be better than the already available options.

Re:Looks good for testing

[hawguy]

I'd love to see their tests when this DB needs to go into swap / pagefile. It's double the slowdown, needs to write into the swap (disk I/O) and then sync the DB (disk I/O again).

I can't, for the life of me, understand where this will be better than the already available options.

I think the point of an in-memory database is that you size your machine so it does *not* need to swap in normal use. Otherwise, since as you said, you lose all of the speed - worse because the operating system decides what to swap out, and may not make the most efficient choice. (though they probably just mlock() the memory buffers into RAM and prevent any of the database RAM from being swapped out at all.)

But if the architect did expect the machine to swap at times, he probably wouldn't put the swap and data on the same physical disk, so concurrent I/O's to both devices wouldn't take twice the time.

Re:Looks good for testing

[errandum]

I meant 2 disk access, some or another. From what I read they would never be simultaneous anyways.

Either way, this would be useful (actually IS, some solutions do this) in the Business Intelligence field. But the whole point of keeping everything in memory is moot when you have petabytes of information that you need to process during your ETL. What matters in this database is, how well does it behave in a cluster and how would it handle concurrency (ACID? Eventually synchronized?).

I doubt this is all that useful for common DB applications like websites and the like. Relational DB's have been proving to be enough for everything (ex: Youtube uses mysql shards - or used to) purely web related for a while now, I doubt this is a gamechanger at all.

Re:Looks good for testing

[errandum]

I see your point, but I disagree. I consider it better to have the webserver running slowly than having it crash because it ran out of memory. To do that might be a choice, but you could just go here ( http://unixfoo.blogspot.pt/2007/11/linux-performance-tuning.html ).

Re:Looks good for testing

[Anonymous Coward]

If you were really a sysadmin who could benefit from that kind of speed improvement, you'd know that it's possible to achieve that level of performance with MySQL already, by either running it from memory or by using a fast hard drive array.

The guys that wrote it are former Facebook employees. So I have to assume they know how to get the best performance out of MySQL, and that itdoesn't suit their needs for whatever reason.

The article doesn't really go into much detail about why, but my point is really about not jumping to conclusions and admonishing someone because you think you know more than they do. Maybe this whole product is useless, and maybe it's brilliant and useful, but you can't determine that soley from this article.

Re:Looks good for testing

[hawguy]

Just say no to swap. It's pointless, except as a crutch for broken software. And it's dangerous on a server. If an application wants disk-backed VM, it can use mmap.

Swap isn't just a crutch for broken software (though it can be), sufficient RAM is not always available. In a perfect world, all servers would have more RAM than their applications ever need, more cores than the processes can take advantage of, and all disks would be RAID-10 arrays of SSD's.

But back in the real world where most of us have to live, swap does come into use at times to let a server accommodate loads that it otherwise couldn't handle due the memory footprint of the software it's running. Swap doesn't have to be a death knell for the server - some light or even moderate swapping can mean that you're using the server more efficiently - especially when you're running on VMWare and it wants to reclaim some RAM of its own via the memory balloon driver. When VMWare is under memory pressure, it's better to let Linux decide what to swap out than to let VMware swap memory out from under the virtual machine.

Re:Looks good for testing

[hawguy]

I meant 2 disk access, some or another. From what I read they would never be simultaneous anyways.

Either way, this would be useful (actually IS, some solutions do this) in the Business Intelligence field. But the whole point of keeping everything in memory is moot when you have petabytes of information that you need to process during your ETL. What matters in this database is, how well does it behave in a cluster and how would it handle concurrency (ACID? Eventually synchronized?).

I doubt this is all that useful for common DB applications like websites and the like. Relational DB's have been proving to be enough for everything (ex: Youtube uses mysql shards - or used to) purely web related for a while now, I doubt this is a gamechanger at all.

Actually, I thought this would be less useful with large databases (like a large data warehouse), and more useful with webservers. If you have a busy website and your core database is measured in gigabytes and not terabytes, it's probably cheaper and easier to run it in-memory than to build out a distributed cluster of SQL nodes to handle the transaction load. $15K buys you a server with 16 cores of CPU and 384GB of RAM.

Re:Looks good for testing

[Gr8Apes]

if your webserver needs to go into swap... you have bigger problems than whether it stays up or not.

Re:Looks good for testing

[frosty_tsm]

Sysadmins benefit from tools that offer significant speed improvements while not sacrificing reliability or ease of recovery. There's some serious questions about the data loss from a system crash (which is more common these days due to cloud stuff) if a transaction is not committed to disk. It comes down to how valuable is that data lost when a crash occurs.

Getting a speed boost but setting a time bomb to being screwed isn't really a smart decision.

Re:Looks good for testing

[bzipitidoo]

Yes. The GP is crazy not to use some swap. The only reason I would not have swap is if the server has only SSDs for storage, and preserving that is more important than keeping the server running.

I consider swap essential for making out of memory failures more graceful. The memory leak is still a very common software flaw. Swap gives you time to notice you have a memory shortage, and a chance to do something about it. Then, there are all kinds of badly designed scripts and programs that leave shells, terminals, and what not hanging around in memory. For instance, startx and xinit run XWindows from a shell in such a way that the shell is left resident until XWindows terminates. This is not a true zombie process or orphan or leak, but it is a small waste of memory. Or someone has logged in several days ago and left some terminals running. Swap is great for stuff like that.

Re:Looks good for testing

[lgw]

With a good overall architechure, you'd prefer a webserver to fail rather than be slow. If the problem is some problem with the webserver hardware, the faster it's in the electronics recycling bin the better: you don't want any traffic hitting a faulty server. If the problem's some software thing, you often get some hours or days of good performance then crap until reboot, so better to reboot ASAP. Limping along is only useful if you only have 1 server or somehting.

Re:Looks good for testing

[lgw]

True that you don't always have a choice, but RAM's usually the cheapest way to boost performance - with any choice at all, enough memory to avoid swap is the first thing to fix.

When VMWare is under memory pressure, it's better to let Linux decide what to swap out than to let VMware swap memory out from under the virtual machine.

Not always - each memory manager knows things the other doesn't. If VMware is de-duping memory pages, and the guest swaps out de-duped memory, it's just a waste. But for a DB you never want to swap DB memory, as the DB knows better than either the OS or the host (and most DBs pin the memory anyhow).

Re:Looks good for testing

[rjstanford]

I'd guess I'm in the other camp too. Mainly because if you're using enough resources on a dedicated machine to need to move into swap then you either have a memory leak (let's assume not, because that's a whole different problem) or you're being hammered. When your machine starts swapping while being hammered, not only does your webserver generally become completely unresponsive (swap cycles taking too long to respond properly to requests, which then get cancelled and swapped out, which then...), but now your may not even be able to SSH in without waiting several seconds (or minutes) between commands.

More often a better approach is to run several smaller memory-bound instances with something like haproxy swapping between them (and your HW LBs swapping between boxen, of course). That way if there's an actual defect, one instance goes down cleanly (and can be restarted cleanly) and the box continues to operate as expected. Between them your appserver instances can allocate all available RAM, and you only have to release it 1/n portion at a time, keeping capacity mostly up to spec.

okay...?

[bhcompy]

When I think of fast databases to compare to, the first thing I think of is MySQL.

/Actually, I'd rather see a comparison to Pick or other lightning fast MV dbs

Re:okay...?

[Kergan]

MySQL is the last thing I think of, personally. It sucks as soon as you make it ACID compliant and start hitting it with thousands of concurrent requests. You're much better off with PostgreSQL.

Re:okay...?

[LurkerXXX]

But with MySQL you can get a wrong answer REAL FAST!!!

Re:okay...?

[Ziekheid]

He was being sarcastic..

Re:okay...?

[evilviper]

When I think of fast databases to compare to, the first thing I think of is MySQL.

MySQL is actually very fast under light loads / one-off queries, and if you choose to leave it at the non-ACID compliant default settings, and similar. eg. "innodb_flush_log_at_trx_commit"

That's probably the only reason why it got popular... There weren't any open source NoSQL DBs at the time, and MySQL seems fast when tested with a basic, shallow benchmark. Of course others like PostgreSQL completely leave it in the dust once there's some real load, or complex queries, or you WANT to be absolutely sure transactions were committed to disk before returning.

As a single point of evidence, I give you Zabbix... It supports the use of all the major databases (Postgresql, DB2, Oracle, SQLite, etc.) as backends, yet MySQL is recommended as it performs the fastest.
http://www.zabbix.com/documentation/1.8/manual/performance_tuning

/Actually, I'd rather see a comparison to Pick or other lightning fast MV dbs

Level-2 overflow! Resize analysis! Change the modulo! Ahhhh!

I've done the PICK-OS thing for a few years, and I'm not a big fan. I'm infinitely happier administering PostgreSQL DBs.

Besides, you don't have to go to something as exotic as PICK to get away from SQL. Try ages-old Berkley DB (db4), or any of the newer NoSQL options.

Re:okay...?

[julesh]

That's probably the only reason why it got popular... There weren't any open source NoSQL DBs at the time

Zope? BDB? Both of these were available at the time MySQL became popular.

Re:okay...?

[sortadan]

But MySQL can only return NULL 3,500 times in one second, this MemSQL thing can return NULL 80,000 times per second. That's 2285.71% faster!

Re:okay...?

[Areyoukiddingme]

Here on Slashdot, we have a convention for saying that with less typing. It's spelled like this:

*woosh*

Re:okay...?

[homb]

That's right. KDB is one of the little-known DB systems used in very high performance environments of financial system.
It is so ridiculously small and fast that I just couldn't believe it when I looked at it over 10 years ago.
I just looked now, and the 3.0 version of kdb+ is all of 258k in size (zipped, OS X).
It's got the database engine, q language interpreter and http server among others. And a small sample dataset.
Try it out, it's mind-blowing.

PS: I never did use it in production because (at the time) of the steep learning curve

Re:okay...?

[CadentOrange]

As a single point of evidence, I give you Zabbix... It supports the use of all the major databases (Postgresql, DB2, Oracle, SQLite, etc.) as backends, yet MySQL is recommended as it performs the fastest. http://www.zabbix.com/documentation/1.8/manual/performance_tuning

From the linked document:

rebuild MySQL or PostgreSQL from sources to get maximum performance

2003 just called. They want their Gentoo Ricers back.

Re:okay...?

[evilviper]

2003 just called. They want their Gentoo Ricers back.

When dealing with something that's performance-critical, recompiling from source is a perfectly valid suggestion. Packages like MPlayer recomend the same thing, for instance. And with Zabbix, we're talking about monitoring hundreds of data points on hundreds of thousands of servers, so the load is right at the edge of what the very fastest servers can handle, and a few percent increase is an important one. Hell, anyone who deals with massive, transaction-intensive databases would tell you this... The database is the bottleneck, and every bit more performance you can pull out of it is a huge win.

Show me vs a real DB engine

[Kergan]

Show me benchmarks vs Oracle, PostgreSQL or SQLServer. Spare me the comparison with MySQL or some other toy.

Re:Show me vs a real DB engine

[BitterOak]

Show me benchmarks vs Oracle, PostgreSQL or SQLServer. Spare me the comparison with MySQL or some other toy.

I think the reason the comparison to MySQL is appropriate is that this database is supposed to be MySQL compatible.

Re:Show me vs a real DB engine

[symbolset]

This i supposed to be funny. Oracle prohibits private benchmark publication in their license.

Re:Show me vs a real DB engine

[guruevi]

MySQL is not a toy (anymore). It's been very good for at least half a decade and has been ACID compliant if you have a half-way competent DBA. Also, MySQL is the fastest of the set you just mentioned in the most basic SELECT/INSERT/UPDATE benchmarks (although each of the rest do excel in solving some really specific problems).

Re:Show me vs a real DB engine

[Anonymous Coward]

PostgreSQL is definitely the best database software out there.

Re:Show me vs a real DB engine

[gman003]

Ah, but it's an "enterprise-grade" toy.

Re:Show me vs a real DB engine

[gweihir]

MySQL is not a toy. Oracle is a bloated monster that only survives by locking-in their customers. I know a lot of high-end customers that would ditch Oracle immediately if that would not mean rewriting a lot of software.

Re:Show me vs a real DB engine

[jmactacular]

Just wish we could easily re-order columns with FKs. =)

Re:Show me vs a real DB engine

[Gr8Apes]

Ah, but it's an "enterprise-grade" toy.

Only by wishful thinking...

Re:Show me vs a real DB engine

[rubycodez]

but anyone can download and run Oracle & benchmark it and publish it. This is the internet, information wants to be free.

Re:Show me vs a real DB engine

[nzac]

From the wiki article:

By SQL Server 2005 the legacy Sybase code had been completely rewritten.

So MS coders have rewritten the failed Unix DB this is hardly credentials.

Re:Show me vs a real DB engine

[mwvdlee]

I haven't yet had a need to re-order columns with FK's, despite having build, maintained and used hundreds of different tables in a variety of database products.
Is there any good reason to do so, besides a desire to make old database tables look slightly prettier?

Re:Show me vs a real DB engine

[pLnCrZy]

Ah, but it's an "enterprise-grade" toy.

Only by wishful thinking...

And price tag.

Re:Show me vs a real DB engine

[rjstanford]

I used to wonder about these until I got to spend some time (actually with Informix and IBM HW) working on some real DB performance testing on one of our products at an IBM capacity center. The reason for this limitation is simply so that people don't publish "official Oracle benchmarks from the official Oracle benchmark tool" when they had an incorrectly tuned DB. That's all. Like many things, its a lot less significant than it appears when you see it. Its not as if they restrict access to it for your own use.

Re:Show me vs a real DB engine

[Tablizer]

Oracle prohibits private benchmark publication in their license.

This should be a violation of free speech. The definition of "free speech" has been expanded to include bribing politicians, so why not for publishing benchmarks? Why are rights only being expanded that favor the fat cats? Oh right, because they are now free to bribe politicians. End Rant.
   

Re:Show me vs a real DB engine

[marcosdumay]

If that is the reason, why do they prohibit any benchmark? Just prohibiting you to claim the benchmark is official sufices.

The reason is that benchmarks can show that any of the four competitive DBMS are the fastest one, depending on how you tune things. It also can show a gap between products as huge or small as you want. Oracle just don't want people to discover that their hardware and their settings are orders of magnitude more important for performance than their choice of DBMS, so they may continue several times more on Oracle's DBMS than on hardware.

Re:Show me vs a real DB engine

[marcosdumay]

You know, in most of the democratic countries you can't sign a right away. Oracle can prohibit free speech all it wants, that can only have an impact where people didn't have that right at the begining.

Re:Show me vs a real DB engine

[Peter Harris]

Yeah, 'cause then you could compare transactions-per-second against dollars-per-month for that and other serious databases (such as PostgreSQL) and make an informed decision. Clearly they believe that decision would go against them for a significant proportion of use cases.

Actually, even if their benchmarks came out really good, the fact that they want to tell me what I can and cannot benchmark and what information I can and cannot share with my peers rules them out.

Oh, and Fuck Oracle.

Re:Show me vs a real DB engine

[Foddrick]

I've been an oracle dev/dba for coming up for 20 years and I find what "locks" people into oracle is the features in the db. I've used most of the popular dbs and oracle still out-features them all. Less code for me to write and that's a good thing. It's horses for courses and for some jobs oracle is the way to go. I may no comment about Oracle's non-db products as a lot of them are rubbish.

Err... what?

[Splab]

Ok, so both article and video is extremely thin on details, the explanation for the massive performance is pretty much gibberish and their argumentation for ACID compliance is bullshit.

Just leaves me with the question, what are they trying to get out of this BS?

Re:Err... what?

[viperidaenz]

Just leaves me with the question, what are they trying to get out of this BS?

Your money, its not a free piece of software.

Re:Err... what?

[gweihir]

Self-aggrandizement and money. When somebody claims they are better than everybody else, they are usually lying and knowing it.

Re:Err... what?

[gweihir]

Quite true. Of course this is something the "enterprise" DB vendors are desperate to hide and there are still enough people that do not have enough of a clue about database-theory. The problem is not SQL though, but the relational database model in general.

Meh.

[hey!]

Give me fast enough, robust, easy to administer and standards compliant. Maybe a little less fast means you throw more hardware at a problem, but it doesn't matter if overall the overall cost and risk is inflated. A platform decision boils down to three things: (1) is it good enough; (2) is it economical; (3) if we decide later this doesn't work for us, are we totally screwed.

In any case, there's no meaningful way you can make a claim that a database management system is the fastest on the planet. All you have is benchmarks, and different benchmarks apply to different use-cases.

Pedant alert!

[PPH]

What you have there is (or may be) the fastest database management system.

I have the worlds fastest database. One table, one record, and one field (NULL).

Re:Pedant alert!

[KingMotley]

That's the second fastest database. My database has no tables, no records, and no fields. Every request responds back with NULL. It doesn't even have to parse the request, so it doesn't even attempt to read it.

Facebook engineers? Gah!

[TheMiddleRoad]

I wouldn't run my toaster on software engineered by someone from Facebook, let alone a database. I'd have to spend ten minutes searching for my toast, and it would show up the following week.

Re:Bah - I had the SAME basic idea in 1996 onwards

[Anonymous Coward]

The hosts file thing can make networking hugely efficient, given how much time is eaten up on slow DNS servers.

I can easily see a DNS solution using P2P Bittorrent transfers and cryptographic signing that both allows distributed DNS, eliminates spoofing, and creates massive performance gains by mirroring hosts files to clients.

Re:Facebook engineers? Gah!

[duk242]

And then the next week, your toast would have changed from white bread to wholegrain and you're just going to have to get used to it.

Faster then MUMPS?

[stanlyb]

Or its nowadays name: CACHE? The best, the fastest, and the most reliable commercial database on the planet? Common, guys, get real.

Re:Bah - I had the SAME basic idea in 1996 onwards

[Anonymous Coward]

Windows NT Magazine (now Windows IT Pro) April 1997 "BACK OFFICE PERFORMANCE" issue, page 61

(&, for work done for EEC Systems/SuperSpeed.com on PAID CONTRACT (writing portions of their SuperCache program increasing its performance by up to 40% via my work) albeit, for their SuperDisk & HOW TO APPLY IT, took them to a finalist position @ MS Tech Ed, two years in a row 2000-2002, in its HARDEST CATEGORY: SQLServer Performance Enhancement).

This episode of "Tech Tips With Timecube Guy" brought to you by the letters W, T, and F, and by CapsCORP: If Bizarro Caps Are Your Thing, cALl uS!

Re:Bah - I had the SAME basic idea in 1996 onwards

[icebraining]

History

The ARPANET, the predecessor of the Internet, had no distributed host name database. Each network node maintained its own map of the network nodes as needed and assigned them names that were memorable to the users of the system. There was no method for ensuring that all references to a given node in a network were using the same name, nor was there a way to read the hosts file of another computer to automatically obtain a copy.

The small size of the ARPANET kept the administrative overhead small to maintain an accurate hosts file. Network nodes typically had one address and could have many names. As local area TCP/IP computer networks gained popularity, however, the maintenance of hosts files became a larger burden on system administrators as networks and network nodes were being added to the system with increasing frequency.

http://en.wikipedia.org/wiki/Hosts_(file)

vs DB2

[Hyperhaplo]

I would like to see the compare againsr DB2. Midrange DB2 if you really want like for like, mainframe if you have guts :)

Nothing to see here, move along, folks.

[140Mandak262Jamuna]

Some clever tricks and cache management. All the speed improvement seems to be coming via read/write speeds rather than any fundamental breakthrough or parallel implementation or massively parallel database of any such thing. And the test was the standard test but some hand picked data base and their own queries. Probably the original funders are planning to sell it down to the next set of chumps.

How do they write to disk faster?

[mounthood]

They're durable and synchronously log all changes to disk, so what makes them faster? They do say this, from: http://developers.memsql.com/docs/1b/durability.html

Reconfigure the server to use a faster disk. MemSQL exclusively relies on sequential (not random) disk writes, so using an SSD will dramatically improve durability write performance.

Are SSDs better at sequential writes? I thought their advantage was random reads, and they weren't any faster at writes then HDDs. Also, the data would become hopelessly out of order by only doing sequential writes, unless they're periodically re-writing all the data in order, which would mean lots more I/O then a typical DB.

Re:How do they write to disk faster?

[hawguy]

Are SSDs better at sequential writes? I thought their advantage was random reads, and they weren't any faster at writes then HDDs. Also, the data would become hopelessly out of order by only doing sequential writes, unless they're periodically re-writing all the data in order, which would mean lots more I/O then a typical DB.

They say they rely on snapshots and logging. I'm assuming that it periodically writes a snapshot of RAM to disk, then logs transactions in the log for recovery. Hopefully it snapshots different portions of RAM at different times so there's not one huge snapshot being written to disk every time.

Though if I had a database where I needed 80,000 query/second performance, I'd probably want a cluster of these so if one machine goes down, the other machine can take over so I don't have to wait for the service to restart, then a 100GB snapshot to be read and gigabytes of logs to be applied before the database is back online.

Re:How do they write to disk faster?

[Surt]

SSD is significantly faster than HDD at both sequential and random writes. Top 15K SAS drives write ~250MB/s sequential. Top SSD write 550MB/s sequential. Write random and it gets much worse for the SAS drive. Try to even find an enterprise HDD benchmark done in the last year. No one bothers because enterprise buys SSD if they care about performance.

This seems to be applicable

[QA]

http://www.youtube.com/watch?v=b2F-DItXtZs

Facebook you say?

[outsider007]

Is it written in php by any chance??

Re:Facebook you say?

[Surt]

Redundant?

Re:Facebook engineers? Gah!

[kiore]

Not only would it tell all your "friends" and relatives what you are eating and when but the control for turning notifications off would be deeply buried next to the mains power wire and mysteriously switch itself back on at random intervals.

Ahhhh, Pick!

[hedronist]

The most over-the-top DB God I know started in Pick-land (ca 1972?). Although he does (is forced to?) use SQL nowadays, he thinks in ways that do not come out of any SQL DBA handbook. As a result he gets DBMSs to do things that are ... unnatural.

He is currently doing some data-cubing stuff for us that I didn't think could be done with something less than a DOD budget. He says his touchstone is thinking in Pick and then 'translating' to SQL.

I still think that the 2 missing courses from any CS degree program are 1) how to debug, and 2) history of computing.

Re:Ahhhh, Pick!

[Samantha Wright]

I am intrigued by your ideas. Let us start a newsletter.

Re:Ahhhh, Pick!

[Zenin]

I still think that the 2 missing courses from any CS degree program are 1) how to debug, and 2) history of computing.

Practical software engineering is mostly about debugging. An actual course in debugging would imply that Computer Science curriculum had something to do with practical software engineering, which we're all painfully away it hasn't in the slightest.

Re:Ahhhh, Pick!

[SixDimensionalArray]

As someone who programs in Pick/D3 still every day (a skill I picked up working for a company with a legacy product), as well as having had worked in pretty much every SQL product that exists, I am both startled and amazed to see it mentioned on Slashdot. I think this is the first time I've ever seen anyone mention it!!!

And I am in agreement - Pick was something truly different which could have been as big as SQL - multi-value, "NoSQL"-ish which still had a query engine, fast, little to no maintenance, loosely structured among other things. Unfortunately, due to the marketing practices of the company that made the product (now Tigerlogic), the former leader (Dick Pick... what a guy..) and various other corporate cultural issues, it has languished as an artifact of history.... but it still has important lessons to learn and teach, and it still runs a lot of systems!

For that matter, while some today don't realize it, many NoSQL databases today bear a startling resemblance to systems like Pick, MUMPS, Cache, Universe, etc.

Re:Ahhhh, Pick!

[dkf]

I still think that the 2 missing courses from any CS degree program are 1) how to debug, and 2) history of computing.

Change "how to debug" to the somewhat-more-general "how to analyze programs" and I'd agree. Over and over, I see people who don't understand what code is doing and who reinvent things from the 1970s because they couldn't be bothered to know what happened and why.

Before anyone asks, SE courses are pretty much as bad. Too many people are "marooned in the now" and lack the mental tools to go truly forward.

Re:Ahhhh, Pick!

[pak9rabid]

and 3.) using version control software

Speed vs. speed

[Todd Knarr]

Speed's fine, but what kind? Or more specifically, over what timeframe? High transaction rates are fine, but they don't do any good if you can only sustain them for a few seconds or minutes before the whole thing collapses. I want to know the transaction rate the thing can sustain over 24 hours of continuous operation. In the real world you have to be able to keep processing transactions continuously.

That long-time-period test also shows up another potential problem area: disk bottleneck. In-memory's fine, but few serious databases are small enough to fit completely in memory. And even if it will fit, you can't lose your database when you shut down to upgrade the software so eventually the data has to be written to disk. And that becomes a bottleneck. If your system can't flush to disk at least as rapidly as you're handling transactions, your disk writes start to lag behind. Sooner or later that'll cause a collapse as the buffers needed to hold data waiting to be written to disk compete for memory with the actual data. You can play algorithmic games to minimize the competition, but sooner or later you run up against the hard wall of disk throughput. And the higher your transactions rates are, the harder you're going to hit that wall.

Re:Speed vs. speed

[symbolset]

FusionIO

Re:Speed vs. speed

[ByronHope]

+1, IO doesn't have to be a bottleneck.

Re:Speed vs. speed

[hawguy]

I can buy servers with over a Terabyte of ram, mutiple power supplies and 4 x 10G interfaces for FCOE.
What is a disk again other than to boot from.

The disk is something to hold your data when a backhoe cuts your datacenter power, and cuts the network connections that you use to replicate data to your remote site.... then your UPS runs out of battery after an hour of transactions have been applied to the database with no replication to the remote site.

Sometimes sh*t happens in ways you haven't planned for... when you have N degrees of redundancy, you'll get bit by the rare N+1 event. It's better to have your data stored somewhere that doesn't disappear after the power goes away (or the machine reboots).

(if you're using your FCoE network to connect to the SAN to store your data, you're still using disks but there's no reason to use a local disk to boot from)

Re:Speed vs. speed

[codepunk]

You obviously have way more cash than I do.

Re:Speed vs. speed

[Todd Knarr]

A terabyte of RAM costs quite a lot of money, far more than a terabyte of hard drive does. And it's not as big as it sounds, I've dealt with databases bigger. Usually the ones that demand the highest performance are also the ones that eat the most space once you start taking indexes and such into account.

And multiple power supplies? Won't help you when the data center rack loses all power. I recall at least 2, maybe more, reports of total loss at data centers in the last 12 months, so it's not like it's that rare an event. That's not counting partial losses, or cases where someone simply fumble-fingered and powered down or rebooted the wrong server. And it certainly doesn't count maintenance outages when the server or the database software had to be restarted to upgrade software. Redundant power supplies won't help against that, and while it's no big deal normally it's a really big deal when it means losing 100% of the contents of the database when memory gets cleared. Sooner or later you need the data on persistent storage, disk or an equivalent. You can handwave that need over the short term, minutes to maybe hours, but when you start talking about maintaining the database for months to years it's a different story. And if you want to say you don't need that kind of up-time, well, the business people where I work would probably boot you out the door so hard you'd bounce twice for suggesting they could just live with losing all our data a couple of times a year. Having it happen even once would probably be the end of the company.

Re:Speed vs. speed

[dkf]

The disk is something to hold your data when a backhoe cuts your datacenter power, and cuts the network connections that you use to replicate data to your remote site.... then your UPS runs out of battery after an hour of transactions have been applied to the database with no replication to the remote site.

We once had a "backhoe event" that cut the power cables between the point where the grid power and the UPS cables came together (our main UPS at the time was a 10MW diesel generator) and the point where they entered the datacenter building. There was about only 2 feet of cabling where they could have done this, but that's where someone put a jackhammer through. Aside from shutting us down in a great hurry, they also put themselves in the hospital, and at the same time blew the breakers on the grid substation that fed both us and that hospital. Fortunately for us, we'd just finished replicating key services to our other datacenter (on a different part of the grid, but without the monster UPS). After a quick mad scramble to reconfigure the failover system to not be a single point of failure (hah!) we were at least mostly operational, but that was definitely dodging a bullet.

The moral of this is that shit happens and you can't stop it, but you can be at least somewhat prepare for it. Committing transactions to disk to make them durable is a part of that.

"MySQL compatible"

[Ignacio]

Uh huh. MySQL is missing huge chunks of functionality by default, so this is not all that impressive. Wake me up when it's PostgreSQL-compatible.

Top coder

[Taco Cowboy]

They did have an ad to lure in "Top Coders" at http://developers.memsql.com/blog/

Apart from their ad, what they said about Top Coders was interesting - with the exception of top coders memorizing who books filled with algorithms, because top coders do not memorize nothing - top coders do not get to be top coders by memorizing.

Instead, top coders have that instinct to _know_ which algorithm to adapt and apply, and top coders know where (and how to) look for the algorithm (either from their own archive, from books, from old magazines, or from some strange corners on the Web)
 

Re:Top coder

[gweihir]

Quite true. That is also what competent computer scientists do: Learn the rough border conditions of a problem and its solutions and look up details when needed. Be able to construct something reasonable when no solution can be found in the literature. Committing details to memory is only for those weak of mind. Of which there are many.

Re:Top coder

[Surt]

All of the best developers I've met had phenomenal memories. I think both a natural reasoning ability and great memory are assets. If you are missing one, you aren't going to be as strong as someone who has both.

Re:Top coder

[gweihir]

I have met quite few people that could fake being good coders using really good memory. They were in fact at best mediocre coders and sometimes really bad ones. While these people can code solutions to simpler things really fast, they usually do not notice when they are out of their depth and would need to look up things or think about them for a while. Then they screw up royally. That most people mistake them for really good coders (and no, memory does not help reasoning ability, it hinders it) makes things worse. One of the hallmarks of a great coder is a very keen sense for when he/she needs to be careful because something is more difficult than it appears to be. Those with really good memories regularly fail that test. Bad memory is an asset here.

Re:Top coder

[Surt]

I definitely disagree. People with great memories can bring context to a problem that a lousy memory just can't. If you can't hold all 20 factors to consider in your mind at once, meandering from one to another will leave you with a solution that effectively considers only a couple.

I have reasonably solid anecdotal evidence on this. I've seen top coders with great memories produce software that dominates their industry in three different industries now, and some of that software is now in the mid third decade of proven maintainability. It may not always be a winning asset, but it clearly can be.

Re:Top coder

[mwvdlee]

Juggling 20 factors in your brain (short term memory) is not the same as having a good memory (long term memory).
In fact they literally use different parts of the brain.

Re:Top coder

[Anonymous Coward]

Not to mention that the ability to juggle 20 distinct factors in the brain at once is unheard of. See: http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two . Since the number of "chunks" available for processing are so limited, the much stronger determining factor is being able to internally contextualize and link chunks into a workable hierarchy. Having a "great short term memory" is highly intertwined with having a keen reasoning ability.

Re:Top coder

[dkf]

One of the hallmarks of a great coder is a very keen sense for when he/she needs to be careful because something is more difficult than it appears to be. Those with really good memories regularly fail that test.

They're distinct; a good memory doesn't really help when pattern matching against "trouble", but it does make remembering types of trouble (and the relevant solution areas) easier. Yet just remembering things isn't enough if you can't apply that memory correctly, and knowing to be careful without remembering what kind of solution to look for is horribly inefficient. The very best coders are those who are good in all the component parts of "programmer thought" and who can integrate it into one whole.

Re:Top coder

[zig007]

Except that so very little of programming these days is about algorithms.
Rather, it is about elegantly solving businesses problems and to know one's way around huge frameworks.
Being a "top coder" is in it self a very good thing of course, but there are very few companies that actually work with technical details like implementing a better hash algorithm and so forth.

Rather, in most developers jobs, it is very valuable to;
* be good at being able to understand, handle and especially change large systems.
* be good at producing solutions that at a reasonable rate balances cost and customer demands versus simplicity, performance, structure and other technical values.
* being able to foresee the usages of the solutions in different time frames, and through this make systems cheaper and easier to evolve. Sometime, a super quick and butt-ugly solution is a really good thing to get the customer going while it figures out what it really wants. As long as all parties are aware of the situation and knows that a complete rewrite will have to be paid for next.
* not act like a stubborn child when ones pet solution or technology gets scrapped or unaccepted or that the rest of the company think it is risky to invest time in going down that road. But to just keep pushing.
* to be professional and keep on working even though the current thing is really boring.

Re:Top coder

[Taco Cowboy]

Except that so very little of programming these days is about algorithms

True, very true !!

Being a "top coder" is in it self a very good thing of course, but there are very few companies that actually work with technical details like implementing a better hash algorithm and so forth.

Actually, not quite

Top coders in this age are a bit different from the top coders 20 years ago

Top coders today needs to realize what you've already outlined, plus, he or she needs to understand the shift of the industry

Or to put it another way - top coders of today must be a step ahead of his or her manager, and are well prepared for any changes that might happen 1, 2 or even 5 years from now

The end result is, the top geek is no longer a geek
 

Re:Top coder

[ciderbrew]

There is the case that when your memory is crap you have to keep working and re-working out problems, so the reasoning part of the mind is used more. So when you get to new problem you think up the solution "better".
I'd not argue against that a decent amount of both would be the best.
In terms of programming and admin I always know just enough to be dangerous, By realizing that makes me safer.

Re:Top coder

[zig007]

I agree, and that is, as evidenced by my post, near the definition of "top coder" for me as well.
However, the "top coders" as described above are the not that way, the "top"-iness is quite obviously measured by algorithmic ability and knowledge alone.

Re:Top coder

[Johann Lau]

Well getting daily radiation doses by indian developers that do not want you to succede kind of makes it hard for a talented individual to retain much in terms of long and short term memory.

I agree.

Trying to quanitfy an individual is difficult so only way to do so is by the resuls solutions that are obtained.

I disagree, because you can usually calculate two or at most three numbers, which you then just plug into the aptitude equation to give you the expected performance bounds for that particular task. Barnard and Meyers wrote a great paper on this, just google for "what your aptitude is good at".

Re:Top coder

[Joce640k]

All of the best developers I've met had phenomenal memories. I think both a natural reasoning ability and great memory are assets.

Learning from the successes/failures of the past is usually better than repeating them.

Re:Top coder

[gweihir]

Short-term memory important for reasoning and connecting things is quite different from long-term memory that allows you to remember things. People tend to mistake one for the other but long-term memory does not help with understanding at all. Obviously I was not talking about "bad short-term memory".

Re:Top coder

[gweihir]

Indeed. Of course "memory" in the unspecified form typically refers to long-term memory. Short term memory is something else and highly needed for really good coders.

Re:Top coder

[1s44c]

memory does not help reasoning ability, it hinders it

[citation needed]

No idea if the above if true or false but I'd also love a citation too.

The only study I found was published in 1968.

Re:Top coder

[mwvdlee]

I think that (or something like that) is what most people mean when they use the word "juggle".
With actual juggling, you're holding 2 items in your hands at any single time, but many more items are available very quickly.
Also, a good juggler can carry around more stuff than fits in his hands. :)

Re:Top coder

[Hillgiant]

A sufficiently abstracted algorithm is indistinguishable from a huge framework.

Re:Top coder

[Surt]

I'm referring to long term memory. The subset that you can easily or immediately juggle is constrained to those you can hold in your long term memory.

Re:Top coder

[marcosdumay]

I disagree that good memory hinders reasoning ability (I put a big [citation needed] there). As far as I know, there is nothing stopping great coders from having good memory, and even memorizing some algos or problems literaly.

That is just not what makes them great. But does not make them "not great" either.

Re:Top coder

[Tough Love]

Except that so very little of programming these days is about algorithms. Rather, it is about elegantly solving businesses problems and to know one's way around huge frameworks.

And that attitude illustrates exactly what leads to big, crap and unreliable systems. Coders who do not understand that even huge frameworks... especially big frameworks.... are just big algorithms. Which does not need to be explained to a true wizard. Foo, you are no wizard ;)

Re:Top coder

[bongey]

Amazon,Microsoft, Google and pretty much every computer science program disagree . Algorithms are just tool in a programmers toolbox. When you don't know them , it would be the same as trying to use a hammer to build and fix everything on a car. Using a hammer to take off a wheel , yes the wheel will come off eventually but it might not be the best tool when you can use a impact wrench.

Re:Top coder

[bongey]

Find that most that say algorithms aren't important don't understand them nor how to apply them. Since they aren't good at it , saying it isn't important makes them feel better.

Re:Top coder

[bongey]

Fool, you are no wizard. I pity the fool !

Had to fix that one.

Re:Top coder

[zig007]

This is true. Saying they are not important made me feel better.
What a minute, I didn't say that!
I probably only felt better because I looked a bit beyond the algorithms...

The thing is, I know quite a lot about algorithms, and still I say these..these heretic things.

I also know many programmers that many times have come to me, bursting with pride over a fantastic algorithm that solved a problem with never before seen elegance, only to be brutally let down when I show them that this problem has already been solved, O(n), somewhere in the framework, or some utility library.

Rather, it is my experience that most people that say that algorithms are very important don't understand frameworks or how to apply them. Since they aren't good at it. Those of us that has been around since algorithms basically was the only thing there was and sorting meant implementing it yourself have actually had to learn this the hard way.
There are extremely few difficult algorithmical problems in day-to-day programming that haven't already been generically solved by the frameworks.
Of course, it depends on what you do for a living. It isn't black or white. Or Black or Scholes, as shown. :-)

After the outright slackers, the worst thing there is are the overthinkers and do-it-yourselfers. The waste vast amount of resources that could have been put to better use elsewhere than on stuff that is just another way to enable them to show off to themselves. They tend to create unnecessarily generalized and deep object structures that, while they can be tested completely and are academical perfection are simply too smart, meaning that they are very expensive to use.

When in doubt #1:
Keep it simple.

When in doubt #2:
Do you really have to do it your own way, are you f#"ing SURE the solution in the framework doesn't work for you? Remember, you have to maintain your crappy solution forever for the likely reward of later realising that you just didn't understand the framework.

Re:Top coder

[zig007]

They are most certainly not just big algorithms since they are pretty far off from meeting the requirements of an algorithm.
So read up, Fool : http://en.wikipedia.org/wiki/Algorithm

They are frameworks(of algorithms). Or libraries. Or whatever. Just not algorithms fer Gods sakes.

Rather, my attitude is one of realism. And it is, which might surprise you, based on actual knowledge.

Unreliability is choice, just as crappines and hugeness is. I have found that people very seldom are unaware of the fact that they are making crap. Actually, some of the most ugly and sickening solutions I have seen have come from brilliant programmers that perfectly knowingly wrote them. Just because the had very little time and knew this super-fast way to deliver something that worked. Ugliness is not always bad, it can also be a creative way of thinking out-of-the box to use a strange behavior to save a situation from imminent disaster.

Anyway, reliability, performance and elegance costs. And elegant, reliable and high-performing system is usually *FAR* too expensive to make from scratch.
This is why we have frameworks, because in them, stuff has been sufficiently elegantly solved, creating a platform from where the customers demands can be met.

But yes, inherently frameworks, as they are not one big algoritm, but masses of likely unused algorithms, makes systems bigger.

But they sure as hell don't make systems using them unreliable, quite the opposite. That makes me suspect that you haven't really thought that position through.
I'd say that rather, unreliability usually comes when programmers do not know how to use the framework or go outside its beaten path.

Re:Top coder

[zig007]

Yep, in those workplaces there are many positions where algorithm design and optimization takes up much time.

But you are still wrong. Most programming jobs even in those places are exactly as I described. We should not kid ourselves here. Even if you work at Wolfram, you will find that most work is made within existing frameworks and that a large number of its employees work as I described. Mathematica has a huge amount of non-algorithmic stuff.

So rather it is the framework that is the programmers toolbox. Because therein lies the algorithms. If you don't know it, you will use the welder for everything, cut the wheel nuts and eventually not be able to put the wheel back on.

One should know algorithms. I am not saying one shouldn't. But on the job making a common business application, it may actually take YEARS between the algorithms you have to make because you haven't found a solution in the libraries and the framework.
Most of the common algorithm writing, which usually concerned data juggling, have been replaced by SQL and collection member functions.
There are just very few places where any advanced knowledge of algorithms are needed nowadays.
Advanced structuring, however, is a different story.

Re:Top coder

[zig007]

Wow.
You can really imply lots of stuff about me, impressive.
But I am really glad I got someone like you to allow to respond, there are so many of you anti-framework-types out there:

* performance issues:
No, most performance issues, to my experience, has absolutely nothing to do with bad algorithms. Most, even crappy, algorithms finish very quickly, so users will never notice. And yes, that is what is important.
Rather, it has almost always to do with databases unmanaged indexes, bad SQL queries, loading unnecessary data.
Also, badly configured servers are no fund.
Strange proprietary wrappings of old functionality is also a dear old friend that needs mentioning here.

* security issues
- Is usually when people DON'T use frameworks, and make their own crappy solution that no one else understand and forgets to sanitize data.
- When in frameworks, is not usually about about bad algorithms, but unchecked buffers and stuff like that.

* Mixing OO with....hibernate?
Actually, I have before, but do not use Java at all. Or C#. And I actually don't work on building a dog ass slow monstrosity.
I mix whatever I think mixes well. There is nothing wrong in combining different tools into a tool chain.

If no one realize there is a problem. Is there a problem?

And...relax, dude. You are making all of us feel like we all are doing something wrong, and that's not cool, man...here, take a puff.

Re:Top coder

[zig007]

And...relax, dude. You are making all of us feel like we all are doing something wrong, and that's not cool, man...here, take a puff.

By the way, it might taste a bit funny, it is wrapped in a $100 bill since it was all that I had laying around. I have earned so many of them from my evil work.

Re:Top coder

[bongey]

You are arguing one needs would need to know the words in the English language to be good writer, without actually knowing the definitions. Yes, you can create sentences , even sound intelligent maybe but you would end up creating crap , right along with your thesis.

Re:Top coder

[zig007]

You are arguing one needs would need to know the words in the English language to be good writer, without actually knowing the definitions. Yes, you can create sentences , even sound intelligent maybe but you would end up creating crap , right along with your thesis.

Eh..no I am not arguing that one should not know the definitions. I am not sure, but I think your example is at the wrong abstraction level.
If I didn't know the grammar, the application would not compile and you wouldn't understand me.
But I suppose that you mean that knowledge of algorithms supply the definition of how one should write programs?

Actually I do not agree there either, as that is more in the realms of coding convention rather than algorithm design.
Now that is if I have understood that sentence correctly. Anyway, I really have no idea of how you came to that conclusion.

And I don't understand what thesis you are referring to? I am just observing the current state of affairs, not much thesis there.

And to quote myself: "One should know algorithms. I am not saying one shouldn't."

Re:Top coder

[Tough Love]

Sorry, I have just too much experience watching knuckle dragging framework monkeys dig holes they can't climb out of.

Whatever you might wish to think, the computer science universe started with algorithms + data structures and that has not changed, nor will it ever. And certainly not to make a gaggle of marginally competent self styled software engineers feel good about the degree they got in button clicking.

Re:Top coder

[zig007]

Whatever you might wish to think, the computer science universe started with algorithms + data structures and that has not changed, nor will it ever. And certainly not to make a gaggle of marginally competent self styled software engineers feel good about the degree they got in button clicking.

Well, I have been around a while too, when I did my CS-stint nobody spoke about frameworks, and I started with assembly language demoing on the C 64 before that.
But you know, things has changed, and as things has gotten more abstract, the bottom-up knowledge I have of how a computer works has become less and less usable.
Also, software development has become easier, and so people with less technical profiles become developers which means more of them monkeys.
On the other hand, I am not sure how great I'd be if I suddenly got thrown into a large framework at age 19, it is a different set of problems.

I also think that many of the problems out there is because employers think that they don't have to educate the new arrivals.

Re:What's the magic?

[symbolset]

Newsflash: servers come with up to 2 TB RAM now.

Re:Facebook engineers? Gah!

[Rary]

Oh but come on. Their engineers are super leet! To work at Facebook, you have to win a drunken speed-hacking contest just to be a PHP coder!

TimesTen Database

[Anonymous Coward]

So what is the difference between MemSQL and TimesTen?

Other than the 16 years TimesTen has been out longer, the fact that Oracle now owns TimesTen, that it runs on both 32bit and 64bit Linux and Windows, that it can run in front of another database engine to give it a boost, and that it has customer installations up to the Terabyte range.

Just another lame attempt to reinvent the wheel.

Yeah, but is it web-scale?

[stigmerger]

http://www.youtube.com/watch?v=b2F-DItXtZs

Filesystem anyone?

[Anonymous Coward]

Remember the good old days, when XYZ-db wasn't always available (or even disirable)? we used to use files.

Yea, files. Novel concept, these days, mention ISAM to someone and they don't know what you're talking about!

If you really need speed, maybe a database isn't your best bet. Maybe, just maybe, you should consider structuring the data in a way that makes sense for your application using files.

Re:Filesystem anyone?

[Anonymous Coward]

I work on a system like that right now in a really big company. Let me tell you something- it's shit. If you need concurrent access to the files/directories by several processes, you'll have a heap of issues. Consumers pick up files before they are completely written by the producers (now fixed by file renaming, but required work). Sime directories now hold 300k files, and any file operations are extremely slow- filesystems aren't designed for this (in process of being fixed by splitting directories squid style into many subdirectories). On top of that, because we need to access the repository from multiple machines and we want reliability, it's now on NAS. This means any file open/close operation takes ~6 ms, at least 2 ping trips to the NAS server and back. This means 166 IO operations per second and no more. You want notification when a file arrives- you get to use polling. FS file modification notifications don't work on NAS. It is terrible. I wasn't there when this thing was designed, but I'd like to find the architector who thought this is a good idea and punch him in the face.

But not dislike the toast.

[TheMiddleRoad]

But not dislike the toast.

Re:But not dislike the toast.

[Vanders]

Ah! So you're a waffle man!

Re:VoltDB

[hlavac]

In memory database and ACID? More like ACIohshitpowerwentoff

Looks like that old Prevayler "database"

[Lisias]

"No more porridge". Right.

This thing is ACID at least?

Re:Facebook engineers? Gah!

[Gr8Apes]

Oh but come on. Their engineers are super leet! To work at Facebook, you have to win a drunken speed-hacking contest just to be a PHP coder!

You just insulted coders everywhere.

Re:nice slashvertisement, guys

[pipatron]

I'm not so sure that this database would fly that fast if it was running on a beowulf cluster of Raspberry Pi with OSX.

memSQL fully hubris acid trip compliant

[WaffleMonster]

MySQL the worlds most popular open source database

memSQL the worlds fastest database

PostgresSQL the worlds most advance open source database

SQLite most widely deployed SQL database engine in the world

I just wish people would dispense with their childish marketing bullshit already.

Re:memSQL fully hubris acid trip compliant

[Xtifr]

Maybe you should look into Firebird. The most slogan-free database engine in the world. :)

Re:Wondering about GPL violation

[unixisc]

Actually, which license does this use in the first place? CDDL? GPL2? GPL3? Any other?

Re:i remember this one top coder...

[jimmydevice]

Are you thinking of Barry White? or Barry Smith? White left the Metheus workstation group and went to work at Mt Xinu, Smith was at OMSI and later worked at Oregon Software.on OMSI Pascal
That's the only Barrys I know.
This was intended to be humorous.

created by two former FB developers

[die standing]

it very well may be the fastest and the bestest, my dad can beat up their dad to the googolplex infinitieth power!

The Devil Is In The Detail

[Anonymous Coward]

I've had a love-hate relationship with MySQL for over ten years now, and have as much cause to hate it as anyone, but I have to point this out. Read the MemSQL docs carefully, and here's the killer - they only support single-query transactions, and only at isolation level READ COMMITTED.

Until those two facts change, then its hardly a fair comparison.

VoltDB / H-Base

[Anonymous Coward]

I guess they forgot about VoltDB and H-Base.
On my laptop I can do 43.000 transactions/sec or about 90.000 SQL queries/sec.
http://www.sgi.com/pdfs/4238.pdf

Similar to changing Oracle commit behavior?

[chopsuei3]

It seems to me that MemSQL is just an implementation of MySQL with the commit behavior changed to something like "BATCH, NO WAIT." This would normally introduce a period of time when the transaction could be lost before it is written to disk, if there were a power outage or something, but with battery backups on enterprise RAID cards, the transaction should still be saved in the RAM on the RAID card. I think it would still be possible to lose the transaction if the server crashed mid-transaction, so perhaps this is a safe implementation of BATCH, NO WAIT on MySQL?

the crucial question:

[cas2000]

Is it web scale?

Citation

[shiftless]

I read it on Wikipedia, so it must be true.

Re:Citation

[Taco Cowboy]

I read it on Slashdot, so it must be true.

TFTFY

Excellent question!

[Anonymous Coward]

Actually, which license does this use in the first place? CDDL? GPL2? GPL3? Any other?

MemSQL uses the standard, Pay Us And STFU About Wanting Access To Source Code license.

Based on that it should be safe to assume the PUASTFUAWATSC licenese is an OSI-approved/GPL compatible license. Expect MemSQL to be bundled in the next major release of Trisquel!

Re:Excellent question!

[unixisc]

Doesn't that make it proprietary? How is that a GPL compatible license?

Re:Ahhhh, Pick! [dBASE]

[Tablizer]

dBASE/xBase also teaches one to think in ways that SQL may otherwise squelch. Rather than get into yet another argument about which is "better" or more "mathematically pure", let's just say that it stretches the mind to see how other database languages go about things. Some things that are difficult in SQL are easier in other database languages, and the reasons will expand your mind to other possibilities. (Database LSD? Oh my!)

While there is a lot of diversity in the app programming world, the database world has been pretty much monopolized by SQL such that people stopped thinking outside the SQL box.

I hope the memory of Pick, dBASE, APL, etc. stay alive to avoid a monoculture in database-land, and spark the database future to improve in part by assimilating the best ideas of the past. (BorgBase? Oh my!)

Uh, Oracle?

[bytesex]

Seriously, why do people, and then I mean slashdot nerds, think 'fast database' and then think 'mysql' ? 'MySQL-compatible' equals 'bad' in my world and, in comparison with Oracle, 'not so fast at all'.

Re:Uh, Oracle?

[marcosdumay]

MySQL was very fast up to version 3. But then they started adding most that functionality you need on a real DBMS, and now it is one of the slowest ones available.

Re:i remember this one top coder...

[ciderbrew]

That's wasn't funny until I read "This was intended to be humorous." It made me smile. I didn't know Barry White could code.

oh yeah

[decora]

first, we get a trunk full of pudding

then we get in the trunk full of pudding

then we start coding.

ohhh yeah

SSD is so yesterday, use DDR5

[cheekyboy]

Why not put the whole memsql inside a GPU with DDR5 memory, 4gig per card, X 3 cards would eat any DB cluster.

Re:SSD is so yesterday, use DDR5

[gman003]

1) There is no DDR5. There's GDDR5, which is essentially DDR3 optimized for parallel access (graphics cards manufacturers got into a bit of an "our numbers are bigger than their numbers" war a while back).
2) GPU stream processors are *terrible* for database work, and the latency of using GPU memory from the CPU would ruin any advantage GDDR5 memory has.
3) The 4GB cards are, coincidentally, the most expensive ones. I'm talking bare minimum $200-$300.

Yeah. You're better off just stuffing a box full of high-performance DDR3.

Re:VoltDB

[rjstanford]

Data center power problems. They can happen, and WAN replication is slow.

Having said that, 99.999% of the DB backed applications out there can quite happily survive a 0.0001% chance of losing a transaction, and should spec their DB solutions accordingly. Honestly, even most financial systems below the banking level can - you reach a point where its cheaper to just budget $10K per record to fix a problem should the once-in-a-lifetime perfect-storm error occur rather than to try to keep getting better at making sure they don't.

Kinda like the Ford Pinto, but with much better numbers (and less death when things go wrong).

or run out of FDs like apache

[cheekyboy]

Apache, whoops, no more Fds, pipe cannot open, whoops, ok, go super slow until linux, ssh, everything is unresponsive.

Myths about how the mind works.

[GrantRobertson]

Research has shown over and over again that we CANNOT hold 20 factors in our minds at once, regardless of the context. The most gifted can only hold 5-7 things. I believe that the people who are truly top-anything get there by giving up such egotistical nonsense and writing things out on paper or a whiteboard. Everything else is bluster.

Re:Myths about how the mind works.

[Surt]

Depends on your definition. I'm certainly 'holding' more than that in my mind right now.

Qualifications?

[Fnord666]

Shamgunov has excellent credentials in the database world, in spite of having worked at Microsoft on SQL Server for six years.

FTFY

Parsing

[hey]

Does it really take a long time to parse " SELECT * FROM customer" or whatever?
Doesn't prepare() usually precompile selects.

Their methodology

[izelpii]

Their methodology is like assembling a car, making a drag race against a civic, and claiming you have the fastest car because it beat it very badly.

So what?

[rabenja]

Looking at the resources consumed on our production database (SQL Server) over the last few minutes... Highest I/O usage: 1 MBps, average is near zero. Nearly all requests are happening in cached memory, and any I/O is to an SSD.

How does the disk/buffer cache affect all this

[Marrow]

People here are talking about configuring MySql to run out of ram, update to ram, etc. But can't you accomplish much the same thing by not flushing the buffer cache? Or even tuning the machine so that the buffer cache gets lots of the ram? Doesnt Linux support lazy writes to disk? Why isnt MySql running mostly out of disk buffer cache?

Re:"Run, Forrest - RUN!"

[icebraining]

Sure, man, whatever you say.

Nothing new here. Seems like a PR attempt to me...

[matthollingsworth]

Databases didn't use dynamic compilation of queries originally, so that's nothing new. Of course that performs better than dynamically generating a query plan. Even IBM DB2 on the mainframe still works this way. They state they use Write Ahead Logging. So do all modern RDBMS's including SQL Server, Oracle, MySQL, Informix, Postgress, etc. And the last claim is that they use SSD. Again, how is that specific to something they innovated? If you take any database and run it in memory (SSD really just being slower memory), it will run really fast. So what's new here? Maybe I'm missing something...

Performance Issue

[Taco Cowboy]

I can't speak for others, I can only speak for myself

I've been in this field for decades, and the one thing that I've found is this ---

If you're talking about Performance - for most apps, it's almost the 10%-25% of the code that takes up 80%-90% of the data-crunching time

Or, put it another way, when there is a need to really tune up the performance of an app, you need to understand how the code runs

And to do that, you need to profile the whole darn thing, fine where it takes most of the time, and then concentrate your effort on fine-tuning those segments

But of course, if you are a perfectionist - such as Steve Gibson of the GRC fame, - you can opt to go all the way to assembly language (I love asm, btw) and do the tweaking

Disclaimer: I am far from being great programmer, but I did have opportunities to learn from several great programmers that I had the fortune to work with, throughout my career
 

Re:What's the magic?

[symbolset]

I generally hate it when people say things like this, but... if you have this kind of problem, you likely have this kind of money too. They sort of go together.

In other news...

[Finite9]

In other words, two guys spent just a year (wtf?) developing a database engine that manages to run queries in memory very fast, just like other database engines that can run in memory (that probably took teams of engineers more than a year to build). I wonder if they had time to write some transaction consistency code, because im pretty sure a 6yr old can make a program that inserts data into memory pointers.

Do they commit, or is "memory" a new form of "the cloud ... maan!"? Maybe they commit to the cloud? Fire and forget?

Unkkown point of failure

[hesaigo999ca]

I am not sure about how an ssd fails, and what happens with the data on that disk...can you raid an ssd?
Having said this, if a failure on a hdd is eminent, you usually have it raided, so even if one fails, the transaction logs can be recreated.
This sounds like they managed to find a way to incorporate the writing to db log files unto the ssd itself, then transfer over to a hdd, but I wonder if during failure, you would lose 100% of the transaction logs due to that ssd not working. Anyone out there could answer this question?

Nowhere near the fastest

[CoolBru]

MySQLs handlersocket (included since 5.5) does NoSQL-style read and write operations bypassing the SQL engine. While it has some limitations, it will do >200,000 queries/sec on a low-spec server and there are benchmarks of it doing >750,000 on a 8-core Nehalem (faster than Memcached!), and it's not restricted to in-memory operations. The nice thing is that you can use that for the simpler parts of your app, then use transactional SQL on the same database for more complex operations.

Another one to look at is TokuTek's TokuDB, another InnoDB drop-in replacement, which is particularly good for inserts, low disk use and low-latency replication. They ran a demo doing 1 billion indexed inserts in 7 hours when InnoDB took a week.

For distributed 'cloudy' apps, one of the better choices is Drizzle, which retains the nice bits of MySQL (and MySQL client compatibility) and rewrites all the rest.

I don't think I'll believe MemSQL until Percona have benchmarked it...

Re:2 questions that will FLOOR you for good, lol..

[Sarten-X]

Of the two options presented, I most believe that the MemSQL team has beaten you to producing anything functional. You seem too incoherent to actually produce a useful program. The extent of your contributions to the world appears to be a list of anonymous posts to Slashdot (the vast majority of which are either downmodded to oblivion or only marginally appreciated) and a few old programs in Delphi, including an alarm clock with "extensive hand optimizations". I find no evidence that you have created anything of professional quality in the software realm. You have, however, produced a good many laughs with your seemingly genuine incompetence.

I never claimed I was a professor. I have a research job at a university, used to teach (as an adjunct faculty, actually), and wrote a few papers describing a new model that hasn't been explored before. You're the one twisting words now.

-Mr. H