Linux Support for Hybrid Hard Drives?

christoofar asks: "HHDD (Hybrid Hard Disk Drive) technology has been receiving some buzz lately. The concept is not new, but Samsung has been working on a consumer version of HHDD that everyone can use. HHDDs are disk drives that carry onboard RAM (in this case, NAND flash) which is non-volatile and offers to speed boot times and writes to the disk. This carries enormous benefit to laptop users who need to keep their disk activity to a minimum in order to preserve battery life. Given that Microsoft is adding support for Hybrid Hard Drives in their upcoming Windows Vista release, what efforts are being undertaken in the Linux realm to use this new storage technology?"

Comments

It should just work

[saskboy]

Why aren't the drives designed in a way that the drive technology worries about the details of retrieving and writing files, and sends them the way USB, IDE, SATA, or SCSI drives already do? Why would these drives need special drivers?

Re:It should just work

[mrchaotica]

I would guess that it's because having a software-based queuing and caching system is more flexible. For example, if you want to optimize operating system startup, then you need some way to specify that the operating system is what you want in the flash memory, instead of some other data.

Or if you've got a laptop, you maybe want to cache writes to the flash if you're using the battery, but write directly to the disk if you're hooked up to the mains. Since the operating system knows about this but the drive doesn't, the operating system needs to control the drive.

Re:It should just work

[AKAImBatman]

I'm guessing that it's not so much that it can't work that way, as it is that it would kill performance.

Today's operating systems try to do exactly what this drive is doing by pumping all reads and writes through the paging system. The problem is that the OS can't take a full performance boost from this or data will be lost in a crash or power loss. You may remember that EXT2FS could easily lose all the data you'd recently written if it wasn't unmounted properly. (It ticked me off when I tried to move RedHat 5.2 packages to a special partition, then found out that they had all disappeared on reboot because I hadn't explicitly unmounted the disk. *grumble*)

Since this technology has little to no chance of losing data in a power failure, the OS can be modified to write the blocks immediately. This could easily result in a performance increase of 2 to 3 times what you normally see today. The improvements in writing meta-data alone could easily provide this increase.

Re:It should just work

[EvanED]

There are a couple good replies to you already, but I'll add one more place the OS might be able to improve its performance if it knew about the hybridness of the disk.

I have a hunch (this is unsupported by data; you'd want to do some profiling before making a decision to do this) that you would want swap reads and writes to go right to the platter rather than flash. One reason is that flash does "wear out" after a while, and moving swap so that it doesn't use the flash would likely increase its lifetime substantially. From a performance point of view, I suspect it'd hurt a little compared to writing to the cache, but I doubt much.

Re:It should just work

[Shag]

For example, if you want to optimize operating system startup, then you need some way to specify that the operating system is what you want in the flash memory, instead of some other data.

Still sounds pretty much like disk caching. During bootup, the system's going to be asking for the OS anyway*, so if you've got a big enough amount of RAM to load it into, you can just do that, can't you?

*I realize that this does not explain current boot times. Windows, Mac OS X et cetera are obviously spending some amount of time subjecting our hard drives to "bible code"-style analysis to see if they can come up with anything amusing, or something.

Re:It should just work

[Simon80]

This would only have an effect if your swap gets used. During regular usage patterns ( so most of the time), my swap doesn't get used, with 512M of RAM.

Re:It should just work

[EvanED]

I regularily see the amount of memory allocated to Firefox alone over 500 MB...

Currently Windows reports 920 MB pagefile usage. I think that's probably total memory use, not the amount that's swapped out, but I can almost guarantee that there's something on disk. Windows seems to be very bad with swapping; before I added another GB of RAM (for a total of 1.5) it paged fairly frequently. After coming back from work and unlocking it, it would take probably 10 seconds to page everything in from disk.

A laptop (with 512MB) I used at work was much worse; it could barely keep the applications I used regularily (Eclipse for development + Javac for compilation + Tomcat for serving + Firefox for testing) in memory at once. If I opened pretty much anything else, it would go page-happy, and would frequently take 30 seconds to do something I hadn't done in a while in Eclipse as it paged it in.

(That said, I've only seen Linux use my swap a couple times, one of which was when I was doing image editing in the Gimp on a couple multi-hundred meg files.)

Re:It should just work

[Simon80]

just one reason why I stick to Linux :) I guess it could be argued that the parent poster was thinking about the performance differences and what not if Windows (and not Linux) moved the swap to the platter, but that's irrelevant anyway, because we have no say in what MS does with the technology, and the article's about Linux. Thing is, Linux will probably have to be developed on the technology only after it is released. Unless Samsung is collaborating with someone, I assume they wouldn't have access to it at all.

Re:It should just work

[Skapare]

It would not kill performance at all. It might fail to improve it if the OS is paying no attention to what is going on with the drive. If the OS queues the writes to the RAM-ified sectors, then you get the old performanc as if this was not a hybrid drive. Reads can't get better if the page is already in system RAM anyway. But I could argue that an OS that queues writes for a reason other than the drive being busy or powered down is doing queueing badly, anyway. If the drive is idle and a process writes a page of data, there should be a very small time wait before physically writing that data. That time wait should be about twice the time it takes for that drive to seek from start to end, linearly scaled to the distance between the most recent physical write (e.g. where the head is now) and the new writing just queued. We're talking milliseconds.

The OS can improve, of course, by knowing that the RAM-ified portion of the drive does not need to be taken into account as a time delay. But in the worst case, the OS ignores that and queues/times for a standard drive and you get no worse than standard performance.

As for the EXT2 data loss problem, that's just a design flaw in the filesystem itself by not doing a combination of correctly ordered writes and quick syncronization (e.g. once writing starts, keep the drive busy until everything that needs to be written is written, while writing the meta sectors that support the filesystem structure in the correct relational order). Journaling isn't even needed to get almost all the benefit of journaling. But you can approximate this by mounting the EXT2 filesystem with the sync option, at somewhat of a performance hit.

Re:It should just work

[wfberg]

I would guess that it's because having a software-based queuing and caching system is more flexible. For example, if you want to optimize operating system startup, then you need some way to specify that the operating system is what you want in the flash memory, instead of some other data.

You could easily have a drive that appears to the OS as two drives; one "drive" would always write to NAND, while the other would be the real disk, with any free space on the NAND part being used as the persistent cache. It would be trivial to use the NAND "drive" for OS startup files, or a swap file, or a database log - just type in the right drive letter to use for those files when configuring your OS or database!

Still, it wouldn't really matter except for startup files, since they don't get used often but impact the startup time heavily. For all other uses, a simple cacheing algorithm that makes sure the most-read blocks stay in NAND (and uses NAND for writes) would be effective. It would figure out that if you're writing a lot to a certain file, then that's what it needs to put in NAND. If your database log file isn't the most heavily used file on your filesystem, your problems aren't with the database.

Or if you've got a laptop, you maybe want to cache writes to the flash if you're using the battery, but write directly to the disk if you're hooked up to the mains. Since the operating system knows about this but the drive doesn't, the operating system needs to control the drive.

You mean; write to NAND when on battery, use main-memory RAM cacheing when on mains. It would still be better to use NAND on mains; mains isn't 100% reliable. In fact, when on battery, at least the OS has an idea as to when the inevitable power failure will occur, so it can choose to flush the cache in time! Not cacheing at all slows things down intolerably, so NAND to the rescue! In fact, on-disk-NAND-for-write-behinds is an even better idea for mains-connected computers than for laptops. Laptops would benefit equally from read-cache on NAND, so the disk doesn't have to spin up too much during normal use (though, for read-cache you could just plug in some more RAM - which is cheaper).

(Hybrid) NAND drives are a real boon to high throughput/volume, highly-transactional systems, like database(log)s, persisten message-queues, etc. That's why NAND drives are being sold right now to places to (massively) speed up their databases.

Non-transparent configuration (other than the fact that you'd be configuring the database logs, or the queues, to be written to a non-default drive letter) really don't add much.

The OS doesn't need to know. It just sees another disk, that maybe a bit faster.

There is an exception. The OS needs to know in case the user is wont to misconfigure the machine (for example, by enabling write-behind cacheing, even though the drive has a nice big fat NAND write cache of its own!). That's why windows needs to add support. Those users don't need to be confused by having another drive letter, or by having to set an arcane, hidden option that would make non-hybrid drives slower.

Re:It should just work

[Nethemas the Great]

If access to drive contents are bound to runtime drivers then that content can be made bound to DRM. Am I the only one noticing a nice little trend happening...

--Neth

Re:It should just work

[mgblst]

It is not disk caching - the OS requests that certain files be kept in that memory at all times. This memory would not be changed until it is told to change. This is so that files that have to be read, either to boot up or continue running the os are easily accessed, and quickly accessed. The drive would do disk caching as well, seperately.

Is it me or does this seem pointless?

[Kasracer]

Hard drives go at a certain speed. Adding NAND or more cache boosts speeds for transfering data because it's held in the Cache or NAND.

Am I the only one who thinks this is pointless? When writing this is delaying when data is committed to the hard drive but it gives the feeling that it's faster. I understand the point for reading but when I save a file to my hard drive, I want it committed immediately. I don't want it to "feel" faster because it's still in the cache and hasn't finished actually writting to the disk.

Am I the only one who thinks we should be focusing more on complete data storage solutions using solid state memory? It would give huge advantages over mechanical hard drives especially for laptop users.

Re:Is it me or does this seem pointless?

[mrchaotica]

But this is non-volatile memory we're talking about, so when it writes to the cache it is done writing the data. Subsequently transferring the data from the flash to the disk is more akin to defragmenting than "completing the write."

Re:Is it me or does this seem pointless?

[msbsod]

Well, they only produce the isolinear chips for Star Fleet.
Seriously, the NAND flash memory stores the data immediately. It is a perfect write cache, because it does not forget the data. Almost perfect, because its write cycles are limited. But a more reliable file system would do the trick, too. Journaling is one way of making things more reliable. But, you have a good point. It is important that some data get written ASAP onto the medium. Unfortunately Linux file systems are not exactly the right choice if this is of importance for you. VMS would be a much better alternative. In order to utilize these hybrid disks it is also important to ensure that the important data gets written immediately, and that the hybrid disk knows which data needs to be cached in the flash memory. The disk allocation tables and directories are obviously important. But some databases might be important too. Bottom line is, the disks simply cannot be smart enough to handle all file systems and all applications. Therefore I agree with you, the whole thing is pointless. It would be much easier to back up some of the computers power lines with a battery, as part of the power supply, not as external UPS (stupid HV-LV-HV-LV conversion - HV=High voltage 110/230V, LV=3.3-12V). There are a few power supplies available which do exactly that (internal LV backup).

Re:Is it me or does this seem pointless?

[Anonymous Coward]

Right like, perhaps in addition to the kernel, or other essential boot files, it would also store the journal for journaled filesystems - it will cache the writes and commit them the next time that power supply and demand warrant. Consider a 1 Gbyte flash, in addition to your 100+ Gbyte drive, that's alot of non-volatile write cache.

Re:Is it me or does this seem pointless?

[EvanED]

This could IMPROVE the situation you're talking about.

First of all, you know that right now your writes are cached in your computer's memory for up to 30 seconds before they are flushed to disk. This is done so that synchronous read calls are less disrupted by writes.

It's possible that this new technology would allow the OS to commit writes to disk immediately (or at least much sooner). The disk writes to NVRAM or flash, at which point it should be stable through power loss or crash. The disk doesn't need to commit it to the actual platter right away for it to be safely stored.

Re:Is it me or does this seem pointless?

[xouumalperxe]

If you'd read the topic properly, you'd have seen we're talking about non-volatile NAND RAM of some sort. Hence, the extra speed does NOT come at a cost to data integrity!

Re:Is it me or does this seem pointless?

[NetRAVEN5000]

No, the new "Hybrid" hard drives don't just add more cache, they actually have built-in flash memory similar to what you find in USB Flash drives.

Flash memory lasts quite a while - even longer than a hard drive, from what I've heard.

The Mybrid drives load commonly-used files into their flash memory - not only for faster access, but also to prevent wear-and-tear on the hard drive's cylinders.

Whatever is in this flash memory is already committed to the flash memory, and it'll stay there until it's overwritten with new data.

Solid state memory is still a little too expensive to be able to buy a 250GB drive. These Hybrids kind-of alleviate the problem by being half-solid state and half-mechanical. They provide the benefits of solid state - high speed and durability - with the benefits of a mechanical drive - low price and high amount of storage.

How is This an Ask Slashdot?

[oirtemed]

I'm sure they will be support for it by the time it is really *needed*. If it is an open and available spec, support could be coding before these things start becoming popular. Otherwise, as the linux community doesn't enjoy the advantages of MS, it will be reverse engineered after release as someone's need dictates. Honestly, how is this a useful question? These drives seem to have limited use outside of laptops and there shouldn't *Need* to be special support, as other posters have said. It should adhere to already existing SATA standards or what not and handle everything on the inside.

Re:How is This an Ask Slashdot?

[Anonymous Coward]

Yeah, just like the SMART support on SATAs? Couple years after every enterprise environment has already used it on other platforms with SATA discs the OSS peeps started working on it. "Because only recently the official specifications for SMART on SATA were released." Too often some sort of fundamentalism wins and the users lose. Thanks a lot guys for that sort of thinking.

Random writes needed?

[Loconut1389]

If they don't even the writes across the media, bits will wear out faster at say the beginning of the memory. Go through a few million writes in a hurry. No thanks for me! Or am I wrong about that?

Re:Random writes needed?

[Loconut1389]

i should add, if a couple million writes is true, even if you were to randomize or even out the bits algorithmicly, I still would be afraid of failure.

What to put in there?

[aspoon]

The question is not whether Linux will support the drive -- someone will eventually write one. The question is what files to put in there to make it boot faster. Perhaps should be done is to put the entire swap partition in there, and put those "hibernation" files in the swap. So it would resume fast, and would also make normal operations faster with the faster swap.

Re:What to put in there?

[dozer]

would also make normal operations faster with the faster swap.

If you're swapping while running normal applications, something is wrong with your system. If an increase in swap speed is immediately noticeable as better system performance, you need more memory. I do like the idea of suspending to flash though.

For the pendantic: sure, OCCASINALLY you might have to dip into swap. And sure, OCCASIONALLY you'll have to pull in some really stale pages. But these are supposed to be rare events. But if these are not rare occurrences, adding some flashy feature (sorry) to your hard drive isn't the best way to solve it.

Re:What to put in there?

[dugenou]

At the time we loose our cycles on slashdot, the MTD project probably already has support for these devices.

Oh, and it's a very bad idea to put swap in a Flash device, unless you want to wear out the sectors pretty quickly and render the chip useless.

Re:What to put in there?

[tepples]

If you're swapping while running normal applications, something is wrong with your system.

Either that, or you're just dirt poor and trying to surf the net on 5-year-old equipment.

Re:What to put in there?

[AnyoneEB]

It sounds like they want to mostly cache files that get read every boot-up, which would tend to not get changed much. This would avoid the however-many-writes limitation of flash.

Re:What to put in there?

[YU Nicks NE Way]

That is exactly how this work. The reason it's so useful for a Windows box arises from one of the ways in which Windows makes interesting use of files on disk.

When a DLL is loaded into memory from disk, it's only relocated if there's already another DLL loaded in the same slot in virtual memory. If there's no conflict with the DLL, it's not relocated; code pages are read directly from disk and never written back. Instead, when they're swapped out (it the machine doesn't have enough RAM), the page is simply dropped into the big bucket; when it's needed again - say, because the component is loaded into another process space - it's reread from disk. The same situation obtains with "true" EXE's, except that it's slightly easier: since there's only one EXE in a given process space, no EXE's ever conflict with one another, and so they are always read, unaltered, from disk.

(If you write code for Windows, by the way, you should know that. One of the easiest ways to speed up the loading of a DLL is to rebase it on disk. There's a Windows executable called rebase.exe which rebases a DLL automatically.)

You can see why this makes a Flash-based code store compelling on a Windows system -- rebased DLLs and EXEs only change very rarely; system code, in particular, only changes when the component is updated. If this persistent store uses an MRU updating algorithm, then the code pages which are frequently used wind up in this cache, and whenever they are first loaded into process memory, they are available instantly.

Re:What to put in there?

[zCyl]

If you're swapping while running normal applications, something is wrong with your system. If an increase in swap speed is immediately noticeable as better system performance, you need more memory.

And of course, memory is free, so we'll just all run out and solve that problem right away. :)

Why wait? Hack one up yourself!

[Myself]

You shouldn't have to wait for the drives to come out. Laptop drive controllers can address a master and a slave, they just don't have a slave drive connected in normal usage. I was just looking for an adapter to let me put a pair of CF cards in place of a 2.5" hard drive. (I can only find the single-card version in a 2.5" form factor, all the dual-card ones are for 3.5" mounting.) I figure, put a solid-state card in one slot and a microdrive in the other, and I've got a hybrid-drive laptop, right?

I just got rid of an old toughbook cf-25 that would've been perfect for this, as the drive mounting is gel and would easily accomodate an oddly shaped adapter instead of a regular drive. Or for the truly insane, a CF card piggybacked on a regular 2.5" drive! All I need is the ability to home-brew those little flex cables, and I'd be in business.

Re:Why wait? Hack one up yourself!

[couchslug]

It eats a PCMCIA slot, but I use a CF card in a PCMCIA adapter (cheap at about 8 dollars) in my CF-47 whose hard disk died. I boot from live CD.
If you hard disk is working, then you have two hds.
Partitioning CF cards is also useful. On my Portege I run a Damn Small Linux frugal install on the first partition and storage in the second.

Nice to see

[Telvin_3d]

With all of the improvements in RAM, on-die cache and processor speed, the bottleneck in performence is HD speed. Anything that helps boot that is a welcome improvement.

Re:Nice to see

[gnud]

Use a bigger block size. Writing will speed up, theoretically.

No need for anything special really

[erroneus]

Just HD partitioning should be enough really.

There are two ways that RAM can speed up a HD: as cache and as mirror. According to what I've read, it serves both purposes but mostly as mirror. And since mirror will represent a specific, unmoving window of the drive, a partition scheme that fits the situation is all that's really needed to best take advantage of it. Now deciding WHAT works best in that area of the drive is a matter for the really smart people. I'm not one of them...

But that said, I don't know why the memory needs to be flash at all? For me, I can see adding a couple of memory cards from a laptop into slots on the drive unit and at power-up, the drive just copies that window of the drive into its memory. It would take just slightly more time over the amount of time to spin up. Since an OS wouldn't be controlling that action, it would be quick, simple and sequential (fast). After that, so long as OS files in that area aren't changed you're gold.

I can't speak for Linux, but I know that when Windows opens files for execution, it keeps them open for write access because there's always that possibility for a .EXE to be changed by the program that uses it. (STUPID but I've seen it done.) This is evidenced in my mind by the common fact that you can't copy a file that is currently being used as a running program. I'd be pretty surprised if that was the case in Linux. That might be the reason it would take some special tricks to make it work well under VISTA since the drive itself might think it's writing to the OS files when it's really not... or perhaps it really is when it shouldn't need to. (Linux is more ROMable than Windows for sure.)

But anyway, I think conventional RAM would be just fine and wouldn't need to boost the price much over conventional hard drives either since dynamic RAM itself doesn't cost that much in comparison to non-volatile RAM devices. So beyond the additional hardware to support dynamic RAM, all that's needed is firmware code to support the new configuration. Having 1GB for just OS files might even be a little excessive for many systems and 2GB would be enough to hold all of the most commonly used software too. Both RAM amounts are conceivable today.

It would rock for laptops. It could conserve a LOT of power to be sure.

Re:No need for anything special really

[we3]

Why not just do that now by adding the memory to your system(if it can handle it)? It shouldn't be to hard to setup a shell script that will load a portion of your hard drive into memory. I wonder if you could setup unionfs to write the changes to the hard drive? You could probably setup proccess acounting or something to track program usage and load only the most commonly used programs. But, to get to the point, the value of the flash ram is that you can accelerate boot time by having the data already in the cache waiting when you boot. You save power by not having to spin the disk all the time. And you eliminate the risk of doing so with volatile memory. Your Idea about partitioning is good though. It'd be just like when we all had to make sure the boot partition was within the first 1024 cylinders. Nice.

Re:No need for anything special really

[Anonymous Coward]

"(Linux is more ROMable than Windows for sure.)" ??

You're never supposed to run anything on priviledged accounts. User accounts running software should always have only read access to practically everywhere. It's Windows administration 101. Actually, it also thwarts most of the security problems and keeps your system a lot cleaner. Most of the properly designed applications have no problems with running with low priviledges. There are guidelines from Microsoft for creating software properly in such way.

There are Windowses running from CDs, such as BartPE. Actually Windows requires surprisingly little amount of places where it really needs to be able to write to. I don't have a list here ready but it's very short. Pretty much similar than what Linux has.. (Temp locations and such.)

Yes, you see lots of badly designed software on Windows. But if you take a look at freshmeat.net you notice most of the "Linux applications" are trash as well. The bottom line however is that Windows is just as "ROMable" as Linux.

Re:No need for anything special really

[Anonymous Coward]

I've tried to run as a nonprivileged Windows user, but there were so many roadblocks I gave up. None of the devs I've worked with at Microsoft even try anymore. Vista's "user account protection" actually modifies the kernel's security token setup in an effort to make this practical, and even that is by no means certain to succeed.

Write caching in flash...

[rew]

I'm not sure if write caching in the flash chips is a good idea...

Flash chips write at around 1Mb per second. Tops. Modern harddrives write around 50Mb per second. You'd need quite a lot of flash chips in parallel to top that.

Of course, you can cache acouple of writes on the laptop drive to prevent having to spin up the disk. But it is not going to be a speed issue.

Re:Write caching in flash...

[artakka]

For many application, like the windows swap file, that read and write small random block, flash disks offer huge performance improvment, because their seek times are much better then regular hard drives.

The sustained write speed does not really matter for the small block operations.

Re:Write caching in flash...

[Anonymous Coward]

More important than write cacheing is frequently accessed data cache, given that NAND reads at 20Mb/s, and you can fairly easily parallelize it (unlike hard disks)

At that point, using 128Mb (bit, not byte) flash chips times sixteen of them (256MBytes) you'd easily saturate the fastest SCSI or SATA bus, and unload your entire 256MB chunk in seven seconds.

If you need to write fast, you can always direct-write the disk, then reprogram the flash slowly (and transparent to the OS, as well)

Re:Write caching in flash...

[rew]

If your OS is WRITING to swap in a "small random blocks" it should reevaluate its swap allocation strategy.

Re:Write caching in flash...

[greg1104]

Samsung's OneNAND design uses some interesting implementation details to run faster than is normally expected with flash. Their specs at http://www.samsung.com/Products/Semiconductor/OneN AND/index.htm suggest writes at 9.3MB/s and reads at 108MB/s; plenty fast for many applications even without running multiples in parallel. It's certainly much slower than writing directly to the drive if the drive is active. However, if the drive isn't spun up at the moment and the amount of data to be written fits in the NAND flash cache, I could see this being a net performance boost over the spinup/seek time combination of the hard drive.

They also spec 100,000 erase cycles before it's worn out. As was noted by an underrated poster the last time this came up, intelligent flash controller designs like this can cope with bad bits and assuring level usage of the memory much better than what you normally see in random hunk of flash.
An analysis at http://www.sudhian.com/printdocs.cfm?aid=686 suggests 33 years of usage for a typical worker. When you run the numbers it doesn't sound that difficult to create a design that would likely outlast the mechanical parts of a standard hard drive.

Re:Write caching in flash...

[Biogenesis]

50Mb/s yes, but only sequentially. As soon as you're seeking you get a few ms of delay on a HDD, but a few us or ns on anything solid state. So basically if you're encoding video a HDD or RAID array would work great, but if you're reading/writing lots of little files (say, in swap, or in a DB server) flash would beat it hands down.

Re:Write caching in flash...

[JacobO]

Also consider that flash uses considerably less power to operate than a hard drive, meaning that your laptop could survive a lot longer without having to spin up the mechanical drive.

Other good uses for this

[nial-in-a-box]

I think the real demand for flash-cache hard drives should be in the area of data loss prevention. Anyone who has any doubts about the fact that laptop hard drives are still way too fragile for the job needs to work at a help desk for a week. Any way to keep those drives spun down as much as possible should be seen as a good idea, since speed does not really matter much if you lose your data and/or need a new drive every three months simply because you like to move your laptop while it is running.

I know this just adds another point of failure to the mix with the addition of flash memory. However, with the apparent improvements in the quality of flash memory, I would expect one of these drives to outlast a current laptop drive by at least 50%. (Note: this is just idle speculation, but I don't know of any solid real-world statistics on laptop hard drive lifespan. I'm guessing laptop makers don't want us to know, either.)

I think that the point of drivers is, therefore, valid. There needs to be some sort of intelligence behind this system to allow frequently needed files to be held in cache in order for this to work effectively. Sure, you could build a drive that could try this on its own, but odds are you would totally throw out any performance or power advantage by doing so.

Another use?

[agentcdog]

So I see a completely different picture here. There are a lot of files on the HDD that are never rewritten. System files, etc. This is where having hybrid drives really helps. Put the main boot files, executables and libraries which are accessed when staring things up in there. So no fast saves, but no load time.

Re:Another use?

[emj]

This has nothing to do about speed, you will not gain speed by using the flash, it's just a way to write data to the harddrive without spinning up. This will earn you
1. power consumption (alot)
2. less wear on the HD

nothing else.

Re:Another use?

[agentcdog]

ok... whatever, smart guy.
*eats cookie*

sync

[Stephen Samuel]

...then found out that they had all disappeared on reboot because I hadn't explicitly unmounted the disk. *grumble*

The SYNC command forces the system to flush all buffers to disk. Once it returns, you know that all currently cached data has been at least scheduled to be written to disk. (current documentation claims that it doesn't return until the data has actually been written). Once data goes to the HD, it may still take a second or more for it to clear out of the internal drive cache.

for older kernels, you can also run the 'update' progrem which will periodically flush the cache. (RH5.2 probabably was old enough to need this). On the most recent kernels, this capability is now supposed to be built in.

Re:sync

[Bloater]

On W2K3 there is an option for enabling proper syncing (if your hard disk is enterprise suitable). It is enabled by default and the performance drop from W2K to W2K3 is shocking, until you disable it. Which means: don't use W2K for anything important - upgrade your servers to W2K3, take the performance hit, but keep your data safe. Microsoft ought to give you a refund on your W2K Server purchase as it is not fit for the purpose for which it is sold.

Having flash in front of the platters will allow this same robustness but without the poor performance.

Re:sync

[Stephen Samuel]

If you do it properly, you shouldn't have that much of a performance hit. Linux also has subroutines for explicitly flushing data on a specifid File Descriptor to disk (for situations where the order is important, like databases).

Between the horrific speed differences between RAM and disk and the importance (nay -- criticality) of permanent storage, disk caching is an especially touchy space to work the tradeoffs.

If you don't cache at all, then you get horrific speed hits. If you cache too eagerly, then risk losing gobs of data if the device is disconnected, the OS horks or you suddenly lose power. Microsoft seems to be controlled by the marketing people, so I can easily see engineers being forced to make bad tradeoffs by marketing people responding to customer complaints.

Re:sync

[Bloater]

I saw it under email loads, so it could well only be syncing for important data (ie, ensuring the email is in the queue folder an on disk before telling the SMTP client that it has queued it).

Re:sync

[Stephen Samuel]

That barely makes sense. You should be able to write to disk far faster than you read network data -- unless you're reading from a RAIDed mail server over a gigabit LAN.

Re:sync

[itwerx]

unless you're reading from a RAIDed mail server over a gigabit LAN.

Which is how most businesses are set up these days...

Re:sync

[after fallout]

RAIDed mail server on a quad processor system, over 10gb lan

Re:sync

[itwerx]

RAIDed mail server on a quad processor system, over 10gb lan

Very nice! :)
      Is that the new 10Gig-E or fiber? (Or 10x1G NICs trunked together? :)

Should be easy enough to use with Linux.

[Anonymous Coward]

I haven't seen the details yet- has anyone? One way to do it would be by LBA- a certain group of blocks could be the FLASH RAM, and the rest mechanical disk. By careful fdisk-ing/ partition administration, one could be sure files like /bin, /etc, /usr/bin, or documents, or whatever you want for the application, were in FLASH area. (Some FLASH is presumably used by the drive for lazy write cache.)

Another way, which I'd LOVE to see someone make, would be a dual drive- the primary ATA could be a flash drive, and secondary could be the hard disk (or vice versa.) Two ATA devices in one. Again, intelligent use of fdisk/ partition allocation, directory placement, etc. Maybe keep documents on the FLASH drive, or whatever.

Windows doesn't open executables for writing

[Myria]

Windows doesn't open executables for writing when you run them. In fact, when you run a program, it prevents writing entirely. Windows loads executables through the memory-mapped file interface. It does this so it can share code pages between different processes, and get away with paging out part of a program without actually writing to the page file first.

Windows programs that modify their own EXE have to do some crazy tricks. While you cannot modify or delete a running program, you *can* rename or move one. You rename yourself then you make a new copy of yourself with the desired changes. When you're done, you execute the new copy and terminate yourself so that the new copy can delete you.

Melissa

What I'd rather see anyway.

[IBitOBear]

I'd like to see linux have support for (and have hardware makers create) what SUN used to call NFS Accelerators. Basically add-on NV (or battery backed) memory sufficent to transparently cache write information. On my laptop I'd put it in my ExpressCard(tm) slot.

Then again I have been considering using an SD type card to contain the journal for my ext3 file system.

Actually, that combination; an ext3 file system in full data write-back journal mode, a solid-state flash device for that journal, and a "large" flush interval (more than the default five seconds) could accomplish essentially the same thing.

I wonder what the flash wear rate would be...? Do they make NAND SD cards?

(etc. od nausium. ahmen.)

Re:What I'd rather see anyway.

[igb]

Is flash memory actually faster than a decent hard disk? I assume the seek time is better, but my recollection is that the transfer rates aren't anything to write home about.

ian

Re:What I'd rather see anyway.

[IBitOBear]

I was thinking more about avoiding drive spin-up. I am not so impatient for "my OS to boot" and more concerned about lengthening the drive lifespan by having it be (e.g. STAY) spun-down for longer. If you had a two-gig SD card and could get the ext3 journal flush time to be "forever if the drive is stopped and there is at least 10% of the log space left" [and if you have enough main memory that you don't need to page/swap] then you could run the laptop virtually all day with the drive spun down if you weren't doing huge downloads.

That may seem counter-intuitive, but really, if you started your word processor and your browser, and your email (or whatever) you would end up with a fairly stable working set. Save that important document several times that day and do moderate browsing (or use only memory cache etc), you end up with a _very_ stable working set. Your modules are loaded and your shared libraries are mapped and paged in.

The memory management might need to be tweaked a bit (which is why I haven't started the project "for real") but it should be really doable.

Hype?

[tacocat]

Seeing as I can't buy one of these drives from my usual shopping websites, why would I care? Obviously the technology is very new. I would much rather let Samsung and Windows sort out all the defects at their cost and embarrassment than risk buying a kludgey piece of hardware for my linux computer.

Simple solution ... still no special driver needed

[Skapare]

The simple solution is to just make a specific part of the hybrid drive be RAM based. It would default to being sectors starting at sector 0. Then the method to change it would be to partition the drive with a special partition code for a partition that will change the sector locations to be in RAM to what they are for that partition, up to the end of the partition or the RAM capacity, whichever is first. Then RE-partition it back to what the OS needs, which will not make changes on the hybrid drive (it will just update the RAM location when a partition table is written that includes the special partition type code). Alternatively, bypass the partition program's overlap inhibit and let the specially coded partition overlap whatever is being RAM-ified to avoid ever being in a state where you have an unusable partition table. BSD and Linux users can just use ordinary tools after reading the directions on how to do this. Windows users might have to get a special tool or a modified partitioning program (I'm sure all the commercial ones would pick up on this quickly).

The hybrid drive controller would simply monitor writes to the partition table and check for the special code in a partition entry. If no code, do nothing special (but of course, write the data if partition table writing is not otherwise inhibited). If the code is present, check that partition table entry for sanity, and if valid, perform the steps needed to make the change (lock RAM, store RAM data to platter in old location, fetch new location data to RAM, save new location sector numbers, unlock RAM).

The driver won't need to know because it will still just behave as usual, issuing standard commands for its drive interface type (SCSI, IDE/PATA, SATA) for any sector. So the driver won't need to be changed.

This method won't have to rule out also including other methods that can do the deed by other means. I would suggest a jumper spot on the drive hardware to inhibit changes so you can be sure your changes stay permanent regardless of software upgrades, installs, viruses, trojans, etc.

Re:Simple solution ... still no special driver nee

[Anonymous Coward]

An even simpler solution pretty much on the same lines as your thinking but slightly different: Just have a certain range of cylinders right at the beginning of the drive set aside as the flash memory. When you partition your hard drive, partition that range of cylinders to install your operating system on, and it will all be in flash and load super-fast for boot. Then, install the rest of your junk to the other partition that is regular hard drive. It will separate the data perfectly.

solid state hard drives

[Spazmania]

Its not a hybrid drive, but the clever one I saw was a hard disk-backed 2 gig "solid state" drive. It was a board in a 1U rackmount box with 2 gigs of regular ram, a battery and a 2 gig hard disk.

All reads and writes are to the ram. The hard disk isn't even running. When power is lost for more than 15 seconds, the hard disk spins up and the contents of the ram are written out to disk. When power comes back on, the drive spins up, the contents of the disk are loaded back in to ram and operation resumes.

Re:Simple solution ... still no special driver nee

[Skapare]

Maybe someone doesn't want their operating system on the RAM-ified part. Maybe they want swap space there instead. Maybe they want the reserved journal space of a journaling filesystem that is frequently updated to be there. Of course all of these needs could be addressed in some way (at least I know some journaling filesystems let you put the journal in a different partition). You'd have to choose which sectors (don't think in cylinders anymore, it's all fake anyway) would be RAM-ified in a way that would fit various needs in various operating systems, including that one with the largest market.

loss prevention

[Elwood P Dowd]

Most common source of data loss.

Laptop hard drives: the new floppy

Re:loss prevention

[nial-in-a-box]

Hahaha. That is so right-on. And to think I actually saw someone using a floppy just last week. I wanted to warn them, but I figured they would have to learn the hard way.

It looks like the answer is "yes"

[NetRAVEN5000]

I've got the 2.6.15.4 kernel and there's a spot to enable support for "Memory Technology Devices".

Here's the description:
"Memory Technology Devices are flash, RAM and similar chips, often used for solid state file systems on embedded devices. This option will provide the generic support for MTD drivers to register themselves with the kernel and for potential users of MTD devices to enumerate the devices which are present and obtain a handle on them. It will also allow you to select individual drivers for particular hardware and users of MTD devices."

It's got options for a whole bunch of things, such as "FTL (Flash Translation Layer)", "NFTL (NAND Flash Translation Layer)", and "INFTL (Inverse NAND Flash Translation Layer)".

Given that it says the hard drives will use "a one-gigabit OneNAND flash chip" (according to the article), it sounds like it will work.

Re:It looks like the answer is "yes"

[daneturner]

Please mod this up. It's nice to have facts in this discussion.

Re:It looks like the answer is "yes"

[booch]

It's nice to have facts in this discussion.

You must be new here!

use something else

[geekqwerty]

i use a 12 hp dot matrix and stone tablets [modified] ive had some of my bakups for 15+ years and no deterioratin equivalent to 100000rpm hdd read 35000 write

Re:use something else

[Anonymous Coward]

Congratulations on the successful chisel-matrix printhead refit, but stone? Oh man, you're begging for serious erosion and seismic problems only a few millennia down the road. You need to step up to micro-etched nickel for data worth keeping.

Re:use something else

[geekqwerty]

i also use solid steel microencoding but it only runs at eqivalent of 35000rpm read 20000rpm write and itcan work as ram also 100 gig per inch

suspending to flash

[way2trivial]

how about 'boot state' recorder to flash.. as in, unless you install software or add hardware, your system memory at bootstate can be written to a flashchip the same size as your ram, and you can boot, or reboot, as fast as you can re-dump the chip into ram.

add new hardware? fine, as soon as yer system is booted for real, save it to the bootchip.

Re:Simple solution ... still no special driver nee

[Anonymous Coward]

Pretty much nobody wilfully uses a flash medium for swap space or transaction logs. Remember, each sector is only good for a limited number of write cycles, so they're only well suited for mostly read-only data (like system binaries or audio/video playback).

Cameras resort to using flash as a space and power tradeoff, and they can only get away with it because users don't fill and erase the same card ten thousand times.

Re: hd faster than network unless...

[aaron_pet]

The HD is ofcourse used for other things as well... it has to get done writing something and then seek.. yada yada...

I'd think that most email systems are multi-threaded so that a mail message waiting to be put on disk wouldn't block another one from comming... /. had an article on why vista won't suck.. and the ability to use a usb thumb drive for system purposes was mentioned.

I would think that the flash would be used for moderately frequently read items... (because superfrequently read stuff would be in RAM already, and stuff that isn't read often.. we could wait for... unless it's a critical low latency item...

So, for those things infrequently read, but needing low latency, the first part of the file ( or most critical to be read) could be stored in the flash section... and the hd could fill it's cache with the rest of the file between seeks, or just go and read the file...

I though flash was inappropriate for swap.. because of it's limited writability.. however, there filesystems for flash take this into account.

In linux, I might make /boot be flash.. I might make some databases be part flash...

I'd have to say that the whole hybrid drive could be done without anymodification of the drivers...

The hard drive controler would do analysis of what files get changed little... er sectors or what-not.. and read frequently... it would then copy the files, or initial parts of the files to flash. I would think that it would be more prudent to "copy" rather than "move" the part of a file... just because it makes error recovery a bit easier.

I'd just like to get a hard drive that takes sticks of sdram and has a battery backup... so that when the power goes off, it sits there still spinning writing the ram to disk and safely parking it's heads before shutting down.. it wouldn't take that big of a battery!

Remember that our 3 1/2 inch drives are half height!!! I had a 300mb scsi disk that was full height.. and I asked my dad... why is this double height?! and I was told... back in the day son...

anyway... we could have hd's with ram sticks, battery, and cooling solution if we just let the dogs be a bit bigger.

-AP

Write cache isnt bad

[nurb432]

As long as your ram has a battery, no worries.

Some ( all? ) SAN devices have write cache, as well as read.

Can I get this for my HDTV?

[Anonymous Coward]

I just want to know if I can get one of these to put in my HDTV. Then I could have an HHDDHDTV.

Do we really need Hyrbid Drives?

[ZephyrXero]

I would think the easiest way to accomplish the goal of HHDDs is to simply have 2 drives, one made of Flash RAM and one traditional.... You use the flash drive for your OS and programs, and then store all your data on the traditional drive.

Watch out...DRM'd hard drives ahead

[Anonymous Coward]

Am I the only one who smells a large DRM rat in the very idea of a lot of RAM memory in a hard drive, and the need for special software in a future 'operating system' to 'access it'. My guess is that it is for use of the DRM artists and pigopolists and not at all by we users who will be its victims. I will buy all the hard drives that I will need for the next thirty years in the next year and never buy nor allow into any of my systems any of these pieces of DRM garbage.

Re: hd faster than network unless...

[Stephen Samuel]

Remember that our 3 1/2 inch drives are half height!!!

Most people are no longer aware that a regular CD-ROM is half-height for a 5.25" disk bay. About the last thing that I've seen that took a full-height 5" slot was 8mm tape drives.

Re: hd faster than network unless...

[Anonymous Coward]

I did have a full height 9GB SCSI disk. I was like a very large and heavy brick.

When it failed it lived under my desk for a while, and saw occasional usage as a door-stop.