Breaking the ATA Addressing Barrier
BitMan writes: "If you haven't heard, there has been a new disk geometry limitation looming for some time at 128GB (gigabytes of 2^30 bytes), which is 137GB (gigabytes of 10^9 bytes). As many will note, there have been various BIOS and OS limitations in disk geometry before -- e.g., 512/528MB, 2GB, 8GB, 32/33GB, etc... But what makes the latest 128/137GB "limit" different is that it revolves around the "hard, physical addressing" limitations of the ATA (AT Attachment) interface. 28-bits are used for addressing, which results in the 2^28 sector * 512 bytes/sector = 128/137GB limitation. As such, hardware fiends like myself were wondering when the industry would get around to addressing this "hard" limitation in the ATA interface.
Fortunately, the solution is already in the works. The ANSI ATA committee has accepted a proposal from Maxtor that extends the ATA bus addressing to 48-bits. This allows for up to 128 pB (petabytes of 2^50), which is 144pB (petabytes of 10^15), sizes. This should tide the PC world over until the 2TB (terabytes of 2^40) limit is reached, which is the maximum number of sectors a 32-bit OS can address -- i.e. 2^32 sector * 512 bytes/sector = 2TB.
In addition to breaking the addressing limitation, another addressing limitation was overcome for performance considerations. The maximum number of sectors transferrable in any command was boosted from 8-bit = 256 sectors/command (~128KB max. transfer/command) to 16-bit = 65,536 sectors/command (~32MB max. transfer/command). This should increase ATA/IDE performance in burst transfers and many other operations.
A whitepaper on the new proposal can be found here from Maxtor. Small correction in the article: Maxtor says 144 pB (petabytes) = 144,000 GB (gigabytes), which is quite incorrect. 144pb (petabytes of 10^15) = 144,000 TB (terabytes of 10^12) = 144,000,000 GB (gigabytes of 10^9).
Thanx goes to the most excellent StorageReview site where I first heard of this."
A. C. Clarke says you can store an entire human consciousness in a petabyte. This should be a big enough address space, then, for a while anyway.
Why can't the hardware and bios just use a 64-bit sector number. There, problem solved, and no more stupid limititions because someone is trying to save a buck or two since they only used a 12-bit address, er, now 16, er, now 28-bit address.
ENOUGH already. Design it RIGHT the 1st time.
I'm sure it has to do with physical limitations and performance. Anyway, this is a different problem then the BIOS one, ATA has been able to handle 137gig drives since it's creation. And now were talking about increasing that capability by A Million Even if we apply more's law to hard drives, that's still over 30 years away. 64bit on the other hand would be useful for about 54 years. Either solution would will work for quite a while, and I doubt being able to use the same hard drive standard between 2030 and 2056 is really that much more then performance considerations today.
ReadThe ReflectionEngine, a cyberpunk style n
ATA already has features which don't "bog the CPU"
and "cripple the user interface". Your criticism is 5 years old; a few OSes (like Linux) have this fully
implemented.
Of course, the OS can also cripple the user interface during heavy disk access, SCSI or ATA.
Is a gigabyte 10^9 bytes, or is it 2^30 bytes? It depends what you're talking about.
For computer memory, the SI prefixes are certainly used to refer to powers of 2: 640 kB of RAM means 655360 bytes, not 640000 bytes.
For networks or clock speeds, the SI prefixes are certainly used to refer to powers of 10: 10Mbps ethernet can carry 10^7 bits per second, not 10*2^20 bps; similarly, a 1GHz processor runs at 10^9 Hz, not at 2^30 Hz.
And disk space? The manufacturers all specify their sizes in terms of decimal powers. And why not? Everything else, with the exception of computer memory, is expressed in terms of decimal powers.
Let's put this silly argument to rest; I'm sure people have much more important things to argue about (vi vs emacs, BSD vs linux, bash vs ksh...)
Tarsnap: Online backups for the truly paranoid
Maybe just once they should make the painful switch to a simple flat 128-bit address space and be done with it.
Revamping the ATA spec is like dusting off the plans for the pyramids for revision. ATA, and even SCSI are showing their age in more ways than one. We should be investing in Infiniband, RapidIO, etc.
Someone you trust is one of us.
Well, actually, I was thinking about VM policy
issues, but yeah, using the BIOS is a sure way
to shoot yoursel fin the foot.
To achieve this, I propose the creation of the "GATE-A29" to allow older software that wrap around the addressing space to access the first sector of the hard disk to continue to function properly. This gate could be controlled with one of the keyboard controller's free lines (to save one cent), and could be turned on and off from the BIOS. Also, there should be a new INT21h function to control the "GATE-A29".
What a perfect way to extend the PC-AT architecture in the totally unencumbered and elegant fashion it has evolved so far !
[Seriously people, just buy SCSI drives, they already do the work properly]
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
I'm not trying to flame here, I just want to learn more about this.
I have not seen an ATA-based system perform this way. I find my 333MHz 20MBit (read: old!) SCSI system at work is much more pleasant to use than the 750MHz boxes with ATA66 setups (setup by Dell). All of these systems are running RH7.1.
Examples of the situation that will "bog" or rather "cripple" the faster ATA boxes but not the SCSI one include:
- Starting VMware and booting up the windows virtual machine
- Running updatedb (not really that bad though)
- Installing a couple hundred megs of RPMs.
- Installing Oracle 8i. (~500MB, uncompressing to 1.4GB)
Is there anything I can do to make the newer ATA boxes perform as well as the SCSI one? I'm getting a new computer at work soon to replace the 333, and it probably won't have SCSI, so I'm very interested to learn more.
I do know that Linux is recognizing the drives as ATA and not "reverting to PIO mode" which I've heard is the obvious solution to this type of complaint.
--
Ahem, ah, yeah, that should be "20MByte" not "20MBit" SCSI system.
I don't think a "20MBit" SCSI system would stand much of a chance at competing with anything except my TI99/4A.
--
Well, you didn't say if you had DMA enabled for the ATA drives or not. It doesn't come that way out of the box, and I have no idea if Dell does the right thing or not. The program you use to investigate this is "hdparm", and you should be able to find a HOWTO which discusses it.
However, DMA mode is only the half of it. The amount of memory you have and the ability of your OS to properly manage it is the other half of it.
SerialATA was supposed to be hitting in the same time frame as this. Why are they dicking with 48bit just to make the hardware implimentation cheaper?
Also, has anyone checked to see if CPRM is being "stealthed" into the spec?
This sig left unintentionally blank.
Are they going to try to bundle the copy protection stuff with this upgrade? Or are the two issues completely disconnected?
It's easy to say that we'll stick with the uncrippled technology we already have. But as it ages and becomes obsolete (ie., can't handle normal sized disks), we'll be pushed into the next generation whether we like it or not. If that next generation includes copy protection, we'll have to live with it.
The drive interface should last longer than the current technology in storage- should holographic storage hit the market en mass, you can bet there'd be drives that will use this new standard, forinstance.
-bugg
I propose a new unit of measure:
How about the Bogo-Gig
For instance, 137 BG Bogo-Gig (base-10 or marketing-style), which translates to 128 GB (base-2, fair and actual non-marketing gigabytes)
Any ideas on better units for monitor sizes?
Maybe ideas for MTBF (Marketing Time Between Failures?) I was told that hard-drive manufacturers actually count on several returns per drive. It's definitely not 100,000 hours like they like to say.
-d1: Use DMA
-c3: 32-bit IO
-m16: Transfer 16 sectors at a time
-u1: Unmask interrupts
According to the hdparm man page, the -u1 option will greatly increase system responsiveness. The other options mainly improve throughput. Many people also use the "-X66" flag to select UltraDMA mode2 transfers, although my BIOS seems to do that automatically.
To test your IDE transfer rate (do this before and after tuning), use "hdparm -tT /dev/hda". It is recommended you test your IDE settings on a read-only filesystem before you start using them regularly - usually the commands don't cause problems, but they can cause major data corruption with a few buggy chipsets (having a recent kernel might help).
You may also be able to recompile your kernel, having it use IDE drivers specific to your chipset, rather than the generic IDE drivers. Kernel 2.4 is probably needed, and you should know what kind of IDE chipset you have (check your motherboard manual). Go to "ATA/IDE/MFM/RLL support" in the kernel config menu, and make sure it is set to "Y". Then go to "IDE, ATA, and ATAPI block devices", make sure the following options are set to "Y":
Then save the config, compile, and install the kernel. You may still need to use the hdparm commands after doing this, just put them in a startup script.
128GB (gigabytes of 2^30 bytes), which is 137GB (gigabytes of 10^9 bytes)
Calling both of these "gigabytes" is confusing. The second figure should be referred to as "metric gigabytes"!
Drives spinning at 7200 rpm will still kick data out at the same speed, and will still be as prone to failure, regardless of capacity. For performance and redundancy (I can't imagine that nothing in your 128GB dataset is not valuable), you need to go multi-spindle, i.e. RAID. Case in point:
Local screwdriver shop has 40GB deskstars for $120. They have a 75 GB for $240. The obvious solution = buy 2x40 and stripe them. Same $$, more capacity, and 2x the speed.
I want to delete my account but Slashdot doesn't allow it.
Actually, that's only 8.0 zebibytes, which is the SI binary prefix for 2**70.
9.4 Zettabytes is correct. Zetta is the decimal prefix.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
SCSI-3 FCP has a 10Km cable length. Is that long enough for you?
SCSI-3 divorced the transport and access protocols. Yes, parallel SCSI has limitations, but you can run SCSI over Fibre Channel (SCSI-FCP) and even IP (iSCSI).
Nice try.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
Wouldn't anything that divides a byte by powers of ten be an imaginary concept anyway? You can divide it into 8 bits, but smaller than that isn't it like discussing the "integer-ness" of pi?
I see even classic Slashdot is now pretty much unusable on dial up anymore.
So you have no idea if it's enabling ATA DMA or not... especially the bit that allows faster interrupts? You know, the Linux hdparm -u1 flag?
The MPAA and the RIAA will surely kill this technology!!!!
--
Knowledge is, in every country, the surest basis of public happiness.
So what if SerialATA won't be on an Intel chipset until 2003? FireWire will never be on an Intel chipset (as they want USB2.0 - to which they hold the patents - to become the standard) but you can buy a 4-port FireWire PCI card for £25.
Remember ATA66? Intel was the LAST vendor to adopt that standard into their chipsets - Via, ALi et al all had solutions in the marketplace while Intels BX was their champion and CaminoGate was giving us all a jolly good laugh.
Remember too PC133 memory. Other chipset vendors have been supporting this for ages, but Intel have only just "gotten off the dime".
You should also give the drive makers more credit. They will realise that SerialATA is a change of maybe 15% to the drives controller board - just a change to the physical interconnect and the silicon that drives it. They're already doing this to produce both SCSI and ATA drives, so rolling out another is not that big a deal.
This sig left unintentionally blank.
For performance and REDUNDANCY (I can't imagine that nothing in your 128GB dataset is not valuable), you need to go multi-spindle, i.e. RAID. Case in point: Local screwdriver shop has 40GB deskstars for $120. They have a 75 GB for $240. The obvious solution = buy 2x40 and stripe them. Same $$, more capacity, and 2x the speed.(emphasis added)
To save people needless grief from following this advice, I feel that something should be explained.
There is a very good reason why striping is called RAID Level 0 (that's a zero), which is because it's not redundant! By utilizing RAID-0, you double the probability that one of the two hard drives will fail, and since 50% of all the data is on the other drive, even the drive that doesn't fail is basically unrecoverable! Sure, it doubles read speed, but even hardware implementations will have slightly slower write speed unless both drives are absolutely identical in geometry, access speed and spindle speed. RAID-0 is essentially useless for anything where reliability is concerned. There are numerous RAID tutorials that explain the differences. Just check Google, or see this quick explanation of RAID levels.
(Moderators, I'm expecting at least one mod point for this short but informative post which includes a link to an informative web page. If I do not get at least one mod point, I will go postal and kill my boss. Urm, I'm self-employed, nevermind.)
--RedBear
=============================================
There have been a lot of improvements to ATA since it first appeared. But when I see so many changes and improvements year after year just to approach the same level of performance (and not really even try for the same flexibility) as SCSI has had since day 1, I can only conclude that ATA was, and presumably still is, misdesigned from the start. That's why I don't use it, and why you shouldn't either. Use SCSI, or 1394, or FC, or one of the hot new technologies that you can't buy yet. It's long past time for ATA to die. They blew it from the start and have never really recovered.
Of course, in that regard, ATA and the peecee are a match made in hell. Neither one has ever let the cost of competent design work raise the price by even 1 dollar. You might say you get what you pay for, but if you're buying this stuff you must be paying for rape.
Realizing that G=(10^3)^3=1,000,000,000 bytes, the NIST (a governmental body that sets standards) decided to formalize binary multiple prefixes, which of course are based on powers of 2 rather than powers of ten. They defined Gi (gibi) to be (2^10)^3=1,073,741,824, which is quite different from 1,000,000,000. In this article the reader confuses gibi with giga. Please use GiB for multiples of 1,073,741,824 bytes, and GB for multiples of 1,000,000,000 - this will help avoid confusion, as well as keep drive manufactures honest.
But this won't be the end of "storage barriers". As a quick example, Linux 2.4.0 had a maximum size for any volume (even LVM & RAID) of only 1 TB last time I checked. Maxtor's white paper mentioned 2 TB limits in nearly all OS's due to use a 32 bit integer to store the sector address.
Microsoft undoubtedly has many similar implementation limits, even if the raw format of FAT32 and NTFS can handle more. If past history is any indication, PC bios code will also consistantly have problems with "next years" drives, simply due to bugs and short sighted coding, even though there's no "real reason" for it.
PJRC: Electronic Projects, 8051 Microcontroller Tools
Actually, the metric definition of Giga = 10^9 has been around a lot longer than the binary variation. Somewhere in the dim mists of computer history, someone noticed that 2^10 ~ 10^3, and started talking about "kilobytes". If anything, we should call these "binary kilobytes", "binary gigabytes", etc.
[100% ISO 646 Compliant]
SVM, ERGO MONSTRO.
Not if everyone switches to it and it becomes the mainstream. The cost difference is almost completely due to economy of scale.
Probably the real reason that SCSI isn't mainstream, is that if it were, it would kill the differentiation betweenn the low-end and high-end markets, and then manufacturers wouldn't be able to get away with charging extra for SCSI. There is strong incentive for there to be more than one standard, so some form of IDE will always have its niche.
---
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
The keyboard controller is not involved in memory access; it's only used to enable or disable one of the address lines. If this is enabled while the processor runs in real mode, another 64 KB of memory beyond the 1 MB mark is accessible. For total compatibility with real-mode software written for the 8086 it must be disabled. If it wasn't for the continued use of DOS while memory got much cheaper, no-one would have had to care about it, but some bright sparks decided that it would be useful to get that extra 64 KB while running DOS.
I agree with you, accept that you are incorrect in saying "ATA only allows for two buses" which is incorrect. There are only two well-known sets of I/O addresses for ATA interfaces, but it's possible to have many more. There are 2 on my motherboard and another on my sound card (yes it really is an ATA interface, not a proprietary CD-ROM interface). I think Linux allows for up to 8 or 10 ATA interfaces.