Slashdot Mirror

I don't understand why they called Eve's launch bad. It would have been a catastrophe if it had been as big as some of the fantasy ones listed, because of the way they run everything on one server. Can you imagine them trying to do 50k simultaneous users back before they had RamSans, with their 2003 budget? No way.

But they have steadily grown and improved, just as intended; running a profitable independent game company for over 5 years is no small feat.

Granted it's very custom stuff, but putting tech like this in a box with a SATA interface is really just evolutionary... Cool none the less though. :)

Well, it's already happened. Take a look at the RAMSAN. It kicks ass.

Ok, got my facts wrong. 3.2 Million IOPS, 24GB/s sustainable random data access.

And a linky:
http://www.superssd.com/products/tera-ramsan/

It seems others disagree with you about 2TB SSD systems.

http://www.superssd.com/products/ramsan-500/

Ironically, these devices are less than an order of magnitude more than the Sun storage.

There are commercially available products that do exactly what you want: provide a large, very fast drive backed by RAM or Flash.

I seem to remember that one of those is at the core of the MMO game "EVE Online" - hosting the database for 35,000 concurrent players.

The only drawback is the price - I think you're looking at $25,000 for an entry-level solution.

Texas Memory Systems RAMSAN

A big bunch of ram for high speed and flash for permanent storage. Just use a super cap for a power backup and have it copy the ram to flash on power down.

They already have this device (although it uses a battery rather than a supercap) in the form of the HyperDrive 4 and the RamSan. Supercaps are pretty darn huge (and expensive) compared to batteries for the same amount of energy storage and since the drain is relatively stable during the backup operation, there's no reason to use one over a battery. Both of the devices are rediculously expensive though (between $250 to $2000 per GB).

400GB ramdisk?

Sounds a lot like a ramsan: http://www.superssd.com/products/tera-ramsan/

Can you really afford it?

Your non-MRAM scenario (battery-backed fast DDR volatile ram caches with flash drives behind them) is exactly what Texas Memory Systems has been doing for a while. If you really want to throw hardware at certain performance problems, their solutions are quite useful.

http://www.superssd.com/products/ramsan-500

You mean like this?

Will I lose data if I lose power to a RamSan?

No. The RamSan-300 and RamSan-400 systems are equipped with redundant batteries and four internal hard disk drives. When external power is lost, the batteries will power the system under full operation for five minutes (this is just in case the power outage is temporary). After five minutes, the system will turn off Input/Output for the system and complete backing up all data that is in memory to the internal hard disk drives. Even if the system is fully loaded with memory, backing up to the internal disks will take no more than 12 minutes to back up. The batteries in the system are N+1 redundant, which means that there is enough battery capacity with just two of the three batteries to power the unit and complete backup after external power fails. The hard disk drives in the system are RAID protected which means that even if a hard disk drive fails, the other disk drives will be able to backup the system in the event of power outage.

No. The RamSan-500 systems are equipped with redundant batteries to power the DDR cache long enough to flush the DDR cache to the Flash memory RAID. Flash memory is inherently non-volatile and does not require power to save data.

As has been mentioned already, TMS sells a solution that fills a rack. The article is about something to fill a drive bay.

We've had a few EVE-Online stories lately, so I thought it might be interesting to some to point out that one of the users of the TMS setup is CCP Games, the makers of EVE Online. In fact if you click on 'success stories' in right sidebar of the first link in the summary you'll see a short article about CCP's first install of the TMS RamSan a while back.

You said: "Texas Memory Systems http://www.superssd.com/benefits.htm says can saturate Fibre Channel (GBs/sec) and this one caps out at 100s of MB/s. Perhaps not quite so unequivocally outperforms as this statement makes it out to be."

FibreChannel speeds are measured in Gigabits, not GigaBytes, of data per second. () Due to the encoding used for 1 through 8 Gb FibreChannel (8b/10b), there is a 20% reduction in useful data transported at a given FibreChannel speed. 1 Gigabit/Second FibreChannel transfers data at 100 Million Bytes per Second (not 100 Megabytes/Second). 2 Gigabit/Second FibreChannel transfers data at 200 Million Bytes per Second. And so on. Not until you get to 10Gb FibreChannel do you see a billion bytes flying by every second (and it uses a different encoding).

A drive 'capping out' at data transfer rates of 100s of Megabytes per Second would, in fact, be operating at FibreChannel speeds. Depending on the exact FibreChannel speed at which the TI drive is said to achieve "saturation," the new drive could, in fact, be outperforming it if it can achieve 100s of Megabytes per Second.

Texas Memory Systems http://www.superssd.com/benefits.htm says can saturate Fibre Channel (GBs/sec) and this one caps out at 100s of MB/s. Perhaps not quite so unequivocally outperforms as this statement makes it out to be.

How about outperforms other flash based SATA SSDs now on the market???? What is surprising is that more of the SSDs don't max out the SATA pipe.

yeah they are in different price classes but it isn't like SSDs haven't been around for long time now. Inexpensive ones that you can put into your sub $1,000 computer... perhaps that is new. Yet another sensationalized copy in a Slashdot story abstract. Oh so surprising.

PCI-X? REALLY? That guy is holding up as a PCIe x1, thats easy 500MB a second. Why the hell go to PCI-X when you can go to just 4x and get 2x the speed of a PCI-X 133mhz.

Hell, do you even SEE servers with PCI-X slots? I know Dell doesn't sell them new unless the customer asks for it. Even then its using a converter chip from PCIe to PCI-X.

But to be honest, I really don't see anyone using this thing on a database. The useal answers are read/writes, price and size. What people don't know is that many storage systems are self monitoring beasts. How good, for the enterprise, is this software going to be? Will it use a kernel driver? Does it offer a bios emulation mode to even see it under dos? I am all for database performance, but I don't see it any better than a RamSan. Heh, I just noticed they offer an all flash product. Funny that.

Here's some benchmarks including db for the gigabyte iram (dram over sata). It would be hard to believe that db access is not i/o bound. Raid'ed atlas' are fast drives, and slugs in comparison. AFA ram duration goes, dram does not suffer the decay issues of flash, but is more expensive. The iram is the cheapest of the ram based sshd's, but not the fastest... something like the Texas Memory Systems Tera-RAMSAN is much faster.

RamSan is an alternative - very fast, no moving parts - as used by the database servers for the Eve Online MMORPG.

The only drawback is cost... And they're not totally solid state - if you get a power cut, the batteries last long enough to write the data to internal hard disks. I suppose potentially these hard disks could be replaced by flash memory since it won't be written to that often (compared with a normal PC's hard disk)

For now, hard disks far beat anything else for cost per MB, reliability and data density.

http://www.superssd.com/products/tera-ramsan/

While you mention those cute SANdisk drives.
This thing will beat the shit out of these SANdisk-toys.
I'd take one of these, too.

Another thing to mention is that Eve doesn't use "shards". There's none of this "Oh, you play WoW? What server? Oh, too bad, I'm on Mediveh". It's ONE SERVER, but at times, we've hit 25,000 simultaneous connections. They accomplish this with big hardware (IBM dual core dual xeon blades, at the moment, i think) and a RAMSAN from a company in Texas.

The game is... it's really hard to explain it to someone who hasn't seen it. It's almost entirely player controlled. All of the low-security space is permanantly up for grabs, and Might makes Right, period. The economy is by far the most complex I've ever seen in a game. Anything you can think of to make money is fair game. There is no "end game" - i.e. there is no lvl 60. If you get bored, join an alliance. Start a war. Train your character in a different direction. I mean... just go check it out. That's all there is to it.

(and I've only been playing since Feb.)

~Xiaodown

Piloting a Ferox with more tech 2 gear than you can shake a stick at.

Solid state disks and other servers

I'm pretty sure the Eve guys bought one of these: http://www.superssd.com/products/ramsan-400/

Like the RAMSAN - http://www.superssd.com/

We were using battery-backed RAM-based disk drives for database indexes 10 years ago. The news here is that flash RAM is finally catching up in terms of cost and performance, not that memory-based disks speed up database performance. That is old news.

EVE servers are using DRAM-based storage, not flash-based. They specifically mentioned the product they are using, too, when crowing about the recent upgrade: http://www.superssd.com/products/ramsan-400/

Perhaps not this products, but the fine folks at texas memory systems will hook you up with that (be it for a price, but I couldn't be bothered to find the "#2 we try harder", TMS is the market leader for DB oriented solid state drives).

If money was no object, I'd get one of these puppies:

http://www.superssd.com/products/ramsan-400/

That should make compiling nice and fast :)

SSDs are actually pretty expensive piece of equipment. And for this reason, they're usually out of reach of "normal" sysadmins, i.e. "normal" datacenters, websites, high-volume and high-latency datawarehouses, offices, tech labs, etc.

I'm actually lucky to be testing one of such toys, a 16 GB, 4x2GBit Fibre Channel RamSan 320 (the one linked is a 325), made by Texas Memory Systems. (Disclaimer: neither me nor my company are related to them in any way. We're testing their equipment, evaluating a purchase.)

The price tag of this 2U rackmounted doorstopper is $40'000 (forty thousands) and change. For sixteen gigs!

128- and 256-GB units cost like small and not-too-small houses. Stackable racks up to 64 TB are sold only to governments. (Kidding, but not too much! ;) )

People is crazy to buy storage at $2000+ per GB? Well, mainly yes, but these units aren't "storage" ones; it's unwise to use them as storage area - you'd put indexes here, views at most, not tables. The real point of SSDs isn't speed, which is at times even better than RAM, but it's latency. You can access ANY byte on the grid at the same speed, more or less regardless of what the app is doing in the meanwhile.

And support for concurrent access. It's perfect when the data to be accessed must be accessed from many "data users" at once (and with "at once" I mean dozens or hundreds of concurrent apps needing dynamica data, either for reading or writing!). Having four FC boards, each one with up to two ports, each one up to 4Gbit/sec in speed, each one linkable directly to a server OR in a fabric, means you aren't in presence of an 1:1 run of the mill SAN, but an N:M beast.

When you're a cell phone provider, and must know NOW if one of your 30 million users has credit on his phone to start the call he's making, you can't wait for a drive head to find the record on disk. Also because, most likely, another million users want to know the same information at the same time.

Consider these things as "database accelerators", for whenever your latency must be really, really low and your database must be shared between lots of consumers!

The market for SSD is a very limited one. But an humongously rich one, too.

Snatching 4GB for $500 is theft, if that's a real SSD SAN-able unit.

How about a raid of solid state disks that can sustain 1,500MB/sec?

Ever heard of solid state storage? Check this out: http://www.superssd.com/products/tera-ramsan/

I saw these people at SuperComputing 2004.
RAM-SAN

why not use one of these

Some other links with pictures:

Sam-650

Sam-650 User Manual

TM-44 ASIC

Solid State Disks

They come in up to 1Terabyte flavors and from the tech specs blow most other systems out of the water on IOPS. They also have partners in the form of SGI/SUN/IBM/Dell that I could find. Worth a looksie if your into seriously geeky, and seriously fast storage systems.

I amend my previous post by pointing to this. Hot damn!

http://www.superssd.com/default.asp

==>Lazn

For one thing, it would take A LOT of space (the tera-ramsan takes up 2 racks), and for another, then that fastness is only available to one machine. Make it look like a nice SAN and it can be used by X number of machines.

Read the fine article, follow some of the links, google is your friend...

Link

Solid state disks solve the problem of physical constraints by replacing hard disk drives with high speed circuitry. Instead of a rotating disk, a solid state disk uses memory chips (typically SDRAM)

So the data is volatile.

The immediate concern voiced about solid state disks regards data persistence and volatility. Unlike magnetic disk drives, SDRAM-based disks require power to maintain their data. The solution to this is surprisingly simple: solid state disks includes backup batteries and backup hard disk drives so that any data written to the SDRAM can be mirrored to or backed-up onto these drives.

Texas Memory Systems solid state disks have a 20 microsecond access time (250 times faster than hard disk drives).

and 400 times slower than 50 nanosecond DRAM (2000 times slower than 10 nanosecond static ram!..)

I wonder why the access time for the RamSan is so high - 50-150 nanoseconds access for SDRAM and 20,000 nanoseconds for the RamSan 340?

Actually, the article is likely getting it's facts right. Check out the specs of the system.

Each Tera RamSan system can have up to 128 ports and 24 Gbit/s. It also can fill up to 2 full racks. Since the government system takes 3 full racks, I imagine it's a slightly different configuration, so reaching those numbers is not out of the question.

Note that the "aggregate I/O rate" number they are talking about is not the same as the aggregate bandwidth of the Fibre Channel ports. It's probably limited more by the memory subsystems than anything else.

Sorry to take so long to get back, but I was tring to find the info for *both* the kinds of SSD we've tried here. But I could only find the one. Funny, if you Google SSD, it'll be the first thing you'll find, so I'm sure you'll claim that's where I got it. It was the TMS RAM-SAN.

Can't seem to find the info on the other...it was like Armadillo Systems or something like that. Ultimately doesn't matter much. Didn't perform very well.

RAM for our SQL servers, all it would take is one fool to trip over a power chord (or just a power-outtage) to lose one heck of a lot of data

UPS is old technology, the battery needs constant replacement, and very few have multiple redundant batteries and/or transistors to deal with wear and tear. Yes even a simple MOSFET transistor is not 100% reliable. Usually the only way to tell a battery is dead is your UPS fails when you need it (this happened to us when my MD was demonstrating our service live to customers, afterwards was the only time he's taken less than 9 months to sign off a purchase order on new equipment). A UPS also has a power cord to pull out when you recoil after burning your fingers on a Seagate Cheetah 15000RPM HD in the server room. A UPS also trips if you overload it, which again means the UPS fails when you most need it.

Other posts mention cosmic radiation at high altitude makes RAM fail. Last time I checked there were no Quad-Xeon Oracle databases on Concorde, although if the International Space Station were to use one this might pose a problem for non-ECC RAM. Anyway, somebody could always write a driver to do software-ECC with Reed-Solomon for RAM if it becomes necessary.

Huge databases (>500 Gigabyes) would benefit most from this as running a simple OUTER JOIN query on the biggest tables will require most of the database to be called into RAM.

• Small databases become slow due to HD latency problems if they do a lot of WRITE operations (the database is stored in RAM, the transaction log is appended to, COMMIT TRANS). This would benefit least FROM RAMdisk because a HD append operation is cheap, however it would benefit database speed in mid-backup
• Mid-size databases become HD-intensive due to aggregate queries/triggered operations over large '>RAM' datasets. For instance enforced cascading deletes where millions of tuples are being deleted cascaded to hundreds of other unindexed tables (in my job I go to the toilet whenever I run a query like this).
• Huge databases where 'Index size' > 'RAM size' - the simplest query would benefit hugely from more RAM or faster storage or RAM-storage. With current databases this would be a 10Gig Eth connection to a Terabyte RAMSan solid-state disk.

RAM for our SQL servers, all it would take is one fool to trip over a power chord (or just a power-outtage) to lose one heck of a lot of data

UPS is old technology, the battery needs constant replacement, and very few have multiple redundant batteries and/or transistors to deal with wear and tear. Yes even a simple MOSFET transistor is not 100% reliable. Usually the only way to tell a battery is dead is your UPS fails when you need it (this happened to us when my MD was demonstrating our service live to customers, afterwards was the only time he's taken less than 9 months to sign off a purchase order on new equipment). A UPS also has a power cord to pull out when you recoil after burning your fingers on a Seagate Cheetah 15000RPM HD in the server room. A UPS also trips if you overload it, which again means the UPS fails when you most need it.

Other posts mention cosmic radiation at high altitude makes RAM fail. Last time I checked there were no Quad-Xeon Oracle databases on Concorde, although if the International Space Station were to use one this might pose a problem for non-ECC RAM. Anyway, somebody could always write a driver to do software-ECC with Reed-Solomon for RAM if it becomes necessary.

Huge databases (>500 Gigabyes) would benefit most from this as running a simple OUTER JOIN query on the biggest tables will require most of the database to be called into RAM.

• Small databases become slow due to HD latency problems if they do a lot of WRITE operations (the database is stored in RAM, the transaction log is appended to, COMMIT TRANS). This would benefit least FROM RAMdisk because a HD append operation is cheap, however it would benefit database speed in mid-backup
• Mid-size databases become HD-intensive due to aggregate queries/triggered operations over large '>RAM' datasets. For instance enforced cascading deletes where millions of tuples are being deleted cascaded to hundreds of other unindexed tables (in my job I go to the toilet whenever I run a query like this).
• Huge databases where 'Index size' > 'RAM size' - the simplest query would benefit hugely from more RAM or faster storage or RAM-storage. With current databases this would be a 10Gig Eth connection to a Terabyte RAMSan solid-state disk.

You can fake it with a file server and a UPS, and set the server up to cache writes very aggresively, but to throw everything on disk when the UPS signals that shit is happenning. Of course then you're limited by LAN bandwith, and you have to keep that server simple to avoid any other program crashing it.

I recall some things from some years ago where there were even transparent colored cubes that looked like things straight out of Star Trek, but they had problems with the registration. It was next to impossible to reseat the cube exactly correctly so that you could retain access to your data. but obviously, other solutions have worked well.

I would love for the cost of these things to come down to something reasonable for the consumer. Recalling the old Tandy laptops that some folks still use, one of advantadge of them is their virtual indestructability, all because of the solid state memory drives inside. (admitting they are small, but they work very very well indeed)