Slashdot Mirror
Never Underestimate the Bandwidth of a Suburban Filled With MicroSD Cards
toygeek writes "If you've been in IT long enough, you're bound to have heard the phrase 'Never underestimate the bandwidth of a station wagon filled with backup tapes.' These days moving data has become so much easier; We've surpassed baud rates and are into Gbps fiber on the backbones, and even in some homes. So, what's the modern equivalent to this, and what does it take to make the OC fiber connections cringe? Follow along as we theoretically stuff MicroSD cards into a Chevy Suburban and see what happens, and take sneakernet to a whole new level."
In my high-school days, we talked about a 747 full of CDs...
I think it may have something to do with growing up on an isolated island nation... there's not many useful places a station-wagon will go.
My book about LSD and Self-Discovery
Also on facebook as: DroppingAcidDaleBewan
The original saying was coined in a time where reading from a tape *was considerably faster* than reading over a network. Hence, transferring data via sneakernet was quicker, inclusive of the read-write times.
Now, with multi-gigabit pipes making up the networks, data can be written, pushed, and read again, all at much higher bitrates than reading any storage medium. It's the read-write to physical medium that are the bottleneck with the sneakernet now.
From the article: "A MicroSD card is only .1 cubic inches, so if all things were equal you could stuff 100 64gb cards into a cubic inch of space! But, that does not seem realistic. In fact it doesn't even seem remotely possible."
Perhaps that's because 1 cubic inch = 10 * 0.1 cubic inches and not 100 * 0.1 cubic inches.
http://what-if.xkcd.com/31/
That is all
Swallowing capsules containing a terabyte (about 12mm in diameter, and 15mm long of microSDs) is quite plausible.
You can easily swallow a hundred of these, and it'll come out over the next 2 days.
100TB/2 days = 600 megabytes a second.
kids these days don't know what the fuck a station wagon is
I wonder where the idea came from? http://what-if.xkcd.com/31/
As they point out in the article, the tremendous bandwidth achieved does not include the logistics or time required to initially copy the data onto SD cards, and then back off of the cards upon reaching the destination. Still, beats a flock of parrots trained in Morse code.
I did something similar after a conversation at work using a 53' Semi Trailer and 4TB HDDs.
Semi = 630" x 94" x 102"
HDD = 4" x 1" x 5.75"
Total HDDs = 262628
Total Storage = 1050512 TB
Bandwidth from NYC to SF = 55.58 Tbps (42 hours according to google maps)
-SaNo
Great, you've managed to transfer X terrabytes of data across state lines. I salute you.
Now... go find this list of files that I need. Also that collection of data that our servers need to process. What, that's going to take you HOW long?
Yes might be quicker to send the data from point A to point B by just shipping disks... but only for archiving purposes. If you actually need to access the data, then you still have that last mile (or 10') of having to load that data into a system or network.
Now, if you were shipping a truck load of servers, or maybe a car full of NAS devices that you could just plug into a network and be done with it... then it's not too bad. Then you can start electronically searching the data within minutes, index it, use it for your data-store, whatever.
But just a car load of disks for non-archival purposes... you're asking for a headache.
This is neither news nor does it matter. You should consider changing the site's tagline to "We'll post anything!".
they forgot the time it takes to transfer data onto the SD cards, remove them, and fill the truck. add another 24 hours
For real-time gaming, this would be awful. Well, except compared to the average American ISP.
I have left slashdot and am now on Soylent News. FUCK YOU DICE.
Since bandwidth is generally measured per second, we'll measure the trip that way too. A 40 hour trip is 144,000 seconds. Now lets measure the bandwidth:
1,225,029,888 GB in 144,000 seconds = 68057Gbps
That's the bandwidth if you are using a non-latency aware protocol. There's no reason to wait for the preceding car to arrive before sending the next one, you can pipeline them as the traffic allows. Assuming we can dedicate one lane of highway traffic to this link and the drivers follow the 3 second rule for safety (wouldn't want any ahem..."packet loss") the the bandwidth of the link is
1,225,029,888 GB / 3s = 408 PetaBytes/s
That's more porn than the whole Slashdot can consume.
and stuff that matters" how exactly??
FTA:
These days you can't really get a good sturdy station wagon, but the modern equivalent seems like it would be the SUV. Since Chevrolet Suburbans have been around for so long, I'm going to pick that.
Subaru still makes the Legacy station wagon which is sold in the US as the gussied up Outback. That would be a fairer vehicle for comparison. It has 71.3 cu. ft. of space with the seats down vs. the stated 137 cu. ft. of a Suburban so the end result needs to be scaled by 0.52. The stated 68057Gbps bit rate should really be 35390Gbps in a fair evaluation.
I am becoming gerund, destroyer of verbs.
In 3001: The Final Odyssey, Arthur C. Clarke wrote about a petabyte tablet as though it were the ultimate storage medium, something humanity finally arrived at after a millennium. In the book it was enough to store the contents of a human brain!
16,000 or so microSD cards could store a petabyte in roughly a 1-foot by 1-foot by 2-inch space, probably leaving enough room to wire them up as well. Of course, it would cost nearly a million bucks, not counting the hardware necessary to wire it up to be accessed. But, still, I find that very impressive.
(T>t && O(n)--) == sqrt(666)
Assuming 2500kg for the car, 0.0005kg per cards, $50000 for the car and $50 per card:
Not only a bandwidth of 68Tbps, but a mass of 12 metric tons (4.8 empty cars).
Also a market value of $0.96B, or the equivalent of 19,141 new cars, plus one car with a broken suspension.
That in the old days, IBM had many facilities North of NYC
And the hotshots at IBM would discuss how to move data around; various hitech ideas like compression over telex lines
And what always won was Joe: every friday, he would make the circuit of the facilities, and drive up to the loading dock with his station wagon, and load and unload tapes (reel to reel)
???
What about round trip latency?? How long does it take me to get my 10GB Full HD video after I've clicked??
John_Chalisque
[quote: "First we need to find out how big these little guys are, exactly. Wikipedia says that they are 0.59×0.43×0.039 inches. [..] .1 cubic inches..."]
A MicroSD card is only
err, i get 0.0098943 cubic inches?
["so if all things were equal you could stuff 100 64gb cards into a cubic inch of space!"]
no. if it were 0.1 cubic inches, it would be 10 per full cubic inch, but if they are 0.001 cubic inches, then it's more like 1000.
either way it's wrong.
didn't take the time to check the rest of the article after this, since the author clearly didn't care either..
btw, it's much easier to calculate these things in metrics ;)
... but if they actually DID IT that would be way cool... and a waste of money.
But it WOULD BE way cool to get a 1970s station wagon and fill it with 1970s-technology backup tapes and drive it across the country.
Bonus points if the "trek" was widely publicized and made part of a "follow that station wagon online" educational event for kids.
Don't have enough 1970s-era backup tapes? Simulate it with something of the same dimensions with a micro-SD card taped to it, holding one tape-full of data.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
I first heard it as "truckload of CD-ROMs".
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
One thing this does not consider is the time it takes to load the data as well as the time to transfer. With the internet the internet the data you use is consumed almost as you use it.
The transfer speed of MicroSD is around 800 MB/s. For the 1,225,029,888 GB in the article that would be 435566 Hours to load the data and unload the data at max speed of the MicroSD, not to mention time to load the boxes and swap out the drives themselves.
Wouldn't that be 10? You know, with the decimal and all that.
Seriously, off by a factor of 10 is still a lot, no matter what scale you're talking about. Shouldn't someone check this stuff?
Beyond that, the time to read those things has got to be enormous, MicroSD can only be read at 104 MB/s, how many cards do you need to read in parallel to match a decent backbone link? These are not the days where you can have scores of malnourished workers clambering to sort and connect things for you, like the manual switchboards of old, you cannot assume these 19,141,092 MicroSDs can possibly loaded and read apart through the efforts of hundreds of workers, each taking union mandated work breaks and working no more than 8 hours a day without overtime pay.
Take 1.9 million SD cards, after reaching your destination, being loaded in parallel into thousands of slots by an army of whatever minimum wage yokels you can find, each card transferring at 104 MB/s and waiting for checksums to determine which cards must be reshipped in the same Chevy Suburban. Compare this to a single DWDM fibre optic link transferring 100 gigabits per second, day and night, unceasingly, unerringly, unquestioningly until every last bit is transmitted, in order and has been acknowledged.
Sneakernet has gone the way of mercury delay line memory, thermionic valves and punch cards. Storage has not gotten smaller and cars have not gotten slower, so this is the time to celebrate the leap that computer networks have taken.
When Argumentum ad Hominem falls short, try Argumentum ad Matrem
Calling something unknown as fact is not science. The theory may be science, but to call the theory fact is religion. I have no problem with macro-evolution, but the beginning of life is a major problem for the theory. Rock + water does not make life. You all suppose that intelligent design is equal to God. That is a jump that defies logic. What is being said is that there is obvious design in the universe. We know that nature does not have the power to design. Therefor something beyond our realm of reality/knowledge must be involved. It could be that there is another force such as gravity that has natural design properties, but then again where did that come from. Our scientific laws state that nothing is eternal in nature. Where was the start? The same issue exists for God, gods etc. So we are talking about an unknown. One last thought here, have you ever looked into the probability of carbon based life existing at all? Have you looked into the intricate ratios that exist in the universe that allow for life to exist on earth at all?
However many cubic inches it is, it's the fundamental problem with this method. The one with the really crucial data on it is bound to find its way under a seat, into an ashtray etc.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
http://www.dansdata.com/gz105.htm
We can overnight a padded envelope full of 32GB drives anywhere in the country. That's hard to beat when you need to send a few dozen gigabytes in 12 hours.
As I read this, I'm on a trip to Houston to retrieve several terabytes of data for Clonebox. At 90 minutes each way, if I bring back 8TB, that's what, 60 Gbps. There's no way I could transfer that data over my cable modem.
Geneticists, geographers, astronomers and anyone else familiar with working with incredibly large (multi-terabyte) sized data sets use sneakernet over the internet these days. It's faster, cheaper (you're not leasing a ISP-tier net connection for the times you need the dataset transferred) and offers nominally better error control than TCP/IP.
Try doing a DC Promo of a very, very large Active Directory (think Air Force large) to an island with a satellite link of 512K then tell me the SneakerNet is dead.
Yeah we did the DC Promo on a machine here then sent the disks via the next plane. I think that qualifies as a SneakerNet for today's kids.
Short of an actual man in the middle attack, with rifles and warrants, your information doesn't get mirrored to the NSA's data center. Being able to see your packet in transit means security.
A few years ago, I was involved in the conversion of the Stanford AI lab tape archive to modern media. This involved reading thousands of reels of 1/2" magnetic tape. It was a slow process. Volunteers were loading a tape onto a tape drive every 15 minutes for weeks. After each tape was loaded, its contents were sent over an Internet connection in under a minute. It took much longer to wind through the tapes than to transmit the data.
The data went to a server farm at IBM Almaden Research, where the file systems were reassembled (these were incremental dump tapes) and text files were converted from the Stanford AI lab's unique character set to Unicode.
The result was the SAIL DART archive. See the source code for EMACS, the early years.
Yes, but how many libraries of Congress is that?
There is a good idea hiding in the rather silly premise of this story (If you didn't read the comments under the original article, someone calculated that it would take 1500 years to fill the sd cards with data, which puts this slightly in the "impractical" category).
But why not have plenty of sd storage built into jackets, bags, cars, whatever? A few hundred gigs doesn't cost so much, and if it could be accessed with micro usb cables from any portable device we'd need far less storage on the devices themselves, and have easy access to our stuff wherever we went.
I think sneaker-net is funny phrase from the days of 10baseT networking being slower than just copying to another media and walking it over. Also used when the stupid networking just wouldn't work...
This article clearly shows why you should go metric.
Tape drives like T10kD and LTO6 have sustained transfer rates that are individually larger than single (non SSD) hard drives. Combined into tape libraries, they are capable of outrunning the fastest RAID arrays on the market.
That of course assumes you are using them as a backup/archive medium and streaming data from a RAID to the tape drive rather than trying to use them in random access mode.
Otherwise you don't have a chance reading them in the right order when restoring the backup.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
http://www.dansdata.com/gz105.htm
The capacity of MicroSD cards has improved a bit since then, resulting in a moderate increase in achievable bandwidth; but other than that the analysis is still essentially the same.
For posting a story like this
Did no one else immediately think of the weight as soon as the author started talking about filling an SUV with microSD cards? I'm reminded of the saying '100lbs of pillows/feathers is still 100lbs', in reference to how people seem to overlook that very light objects are still heavy if you carry enough of them.
While the exact weight of each of the 19 million microSD card would vary a nice starting point is about 0.4 grams plus or minus 0.1 based on general specs. That's well over 16,000 lbs or 8 tons of microSD cards in the back of that SUV, which according to the page linked in the article is rated for a payload of only 1580 lbs. To get an idea of how much 8 tons is, that's the weight of a medium sized Caterpillar backhoe.
With 2.5TB native/6.25TB compressed on an LTO-6, bandwidth has gone up quite a bit since the days of 1.3GB native/2.6GB compressed of a DDS-1 tape.
All of you people who maintain that hard drives or flash media are "the way" to store data don't have any idea how the real world works. You simply can't beat tape for media cost per TB or reliability. And it's pretty hard to beat it for density (TB/cubic inch). Of course, tape drives aren't particularly cheap, so home users are probably stuck with inferior options.
To quote wikipedia:
Baud is related to but should not be confused with gross bit rate expressed in bit/s.
Even digital fiber optic systems have a baud rate.
-mto
You cannot eavesdrop on a station wagon full of SD or even micro-SD cards or sneaker net in general.
Undetectable Steganography? Yep, there's an app fo
You'd get a very large amount of out-of-sequence deliveries and packet loss. Better get a Toyota Hi-Lux or something like that if you want your data to actually arrive in a predictable amount of time.
I was promised a flying car. Where is my flying car?
Please give some credit to Andrew Tanenbaum. "Computer Networks", 2nd chapter.
My favorite one was figuring out the data transfer rate of a snail pulling a chariot made of 2 DVDs
A micro SD card weighs about 0.25 grams, so their calculated 19,141,092 micro-SDs weigh in at 4785 kg. The maximum load of a Chevy Suburban is 2561 pounds (1161kg). Assuming you have a 75kg driver, that lonly leaves 1086 kg for micro-SD cards. So you can only carry about 4,346,596 micro SD cards, less than a quarter of what the authors estimated. The bit capacity is 278 petabytes.
http://what-if.xkcd.com/31/
...and a play ticket to Hong Kong (with transfer to Moscow).
"Love is a familiar; Love is a devil: there is no evil angel but Love." --William Shakespeare ('Love's Labors Lost')
Which is still true between many locations on the planet. The big change is that instead of tapes being used for large volumes you see hard drives being posted around instead. Uploading 2TB via ADSL2 with an upchannel of 256kbps or so would suck immensely, and 10Mbps is still making sending the thing by courier a much more attractive idea.
Comparing LTO5 at 140MB/s to ADSL upload speeds shows a considerable difference (close to four orders of magnitude so *considerably faster*). Beyond a certain amount of data you can get a tape delivered and read in long before someone on a suburban network connection could have uploaded it. Increase the volume and eventually everything apart from really good fibre all the way loses.
Even 10Mbps of HSDSL is roughly two orders of magnitude slower than that 140MB/s tape.
Love it!
Now the tunnels running from So Cal, Arizo and TX into Mex breaths new life in the an old concept.
Brillant! :D
Well, I thought f it as well. ctrl-f "weight" brought me to your comment:)
The other factor is cost. Backup tapes aren't used just because they're small, but because they're cheap. Amazon charges $50 for that SD card (For comparison, you can buy a tape that stores 40 times the amount for $78). That's retail, so bulk buy might be cheaper, but even at half that price, you're pushing a billion US dollars.
Laying fibre optic cable costs about $200,000 per mile according to the highest estimate. So that's about $550 million for the same journey. Although you'll need a lot of parallel cables, and I have no idea how this breaks down between the cost of cable and the cost of digging trenches. I do think the high bandwidth cable would be a much better investment if you did need to shift all that data.
you lost me at converting cubic inches to cubic feet. Do I need to use a calculator to verify that the math is correct?
When you transfer data from A to B, the needed data is directly accessible in B after the transfer.
In case of the SD cards, you have to count in also the time you need to plug them in and probably copy them to another media in B.
Anybody added this to the bandwith calculation?
A site I was working on did not have enough computer time free to run an important program so hired the use of some computer time in the Bracknell Weather Center forty miles away. I drove an operator and 13 tapes from the site to Bracknell and back again after the run. The vehicle was a Reliant Supervan II as made famous in "only Fools and Horses" http://www.motorstown.com/images/reliant-regal-supervan-07.jpg
At $45/per 64GB card, that's almost a billion dollars!
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
I have worked in IT long enough and I have never once heard anyone say that...
19,141,092 MicroSD cards will fit in a 2014 Chevy Suburban. [...] 1.14 EiB
What about seek time? If I need file #4455256, How do I go through all those 19M cards? You need a lot more infrastructure and SPACE to find something in that huge stack of cards.
Fill a car full of WiFi enabled SD cards and drive around the city... Sure the latency will really suck but you could have a "packet-size" of up to ~32Gb depending on how much you could upload during the short time the car drove by......
Fill a 18-weeler with SD cards then... They should be able to handle quite a few tons... And don't cost that much per mile :)
Back at the end of the cold war I thought about buying up a bunch of used ICBMs and setting up a one hour New York to London parcel service. "When it absolutely, positively has to be there right now," for a slogan. The idea would be a 15 minute chopper from Manhattan to an offshore launch, 30 minute sub orbital hop to near London with the payload ejected and parachuted where it could be grabbed by a harrier while the missile splashed into the sea. The the payload could be zipped to downtown to the financial district. Was a nice dream.