Slashdot Mirror


Never Underestimate the Bandwidth of a Suburban Filled With MicroSD Cards

toygeek writes "If you've been in IT long enough, you're bound to have heard the phrase 'Never underestimate the bandwidth of a station wagon filled with backup tapes.' These days moving data has become so much easier; We've surpassed baud rates and are into Gbps fiber on the backbones, and even in some homes. So, what's the modern equivalent to this, and what does it take to make the OC fiber connections cringe? Follow along as we theoretically stuff MicroSD cards into a Chevy Suburban and see what happens, and take sneakernet to a whole new level."

41 of 208 comments (clear)

  1. Jet full of CDs by YttriumOxide · · Score: 5, Interesting

    In my high-school days, we talked about a 747 full of CDs...

    I think it may have something to do with growing up on an isolated island nation... there's not many useful places a station-wagon will go.

    --
    My book about LSD and Self-Discovery
    Also on facebook as: DroppingAcidDaleBewan
    1. Re:Jet full of CDs by JavaBear · · Score: 4, Interesting

      I do recall a story about Maersk moving their data centre from the US to Denmark, using one of their planes as a carrier, filled with harddisks.

    2. Re:Jet full of CDs by YttriumOxide · · Score: 2

      Iceland? New Zealand? Do tell.

      Got it on the 2nd guess - New Zealand.

      --
      My book about LSD and Self-Discovery
      Also on facebook as: DroppingAcidDaleBewan
    3. Re:Jet full of CDs by rossdee · · Score: 2

      How about a rollon-rolloff cargo ship? Or a supertanker? The bulk of the worlds commodities are still transported by sea (even to and from the most isolated of nations like NZ.
      The limitation of a 747 would be the weight it could carry, rather than volume (microSD cards sink so they are heavier than water.

      Do you think Dean Barker can finnally win a race this afternoon (I guess its morning over there in Godzone)

    4. Re:Jet full of CDs by nurb432 · · Score: 2

      .... CDs....

      Kids these days.

      --
      ---- Booth was a patriot ----
  2. This is pointless by Anonymous Coward · · Score: 5, Informative

    The original saying was coined in a time where reading from a tape *was considerably faster* than reading over a network. Hence, transferring data via sneakernet was quicker, inclusive of the read-write times.

    Now, with multi-gigabit pipes making up the networks, data can be written, pushed, and read again, all at much higher bitrates than reading any storage medium. It's the read-write to physical medium that are the bottleneck with the sneakernet now.

    1. Re:This is pointless by jc42 · · Score: 3, Insightful

      It's the read-write to physical medium that are the bottleneck with the sneakernet now.

      Yeah, but it's competing with high-speed networks that are crippled by the ISPs at both ends using a single fibre to feed an entire neighborhood, and intentionally slowing the speed at the customer's site to a crawl unless you pay an exorbitant rate for a higher speed (which is then unused 99% of the time, and doesn't deliver if 2 or 3 others in your neighborhood are using high speed at the same time).

      It's not surprising that vehicle+SD card could outperform such a network. The ping times can be rather long, though.

      --
      Those who do study history are doomed to stand helplessly by while everyone else repeats it.
    2. Re:This is pointless by Waffle+Iron · · Score: 3, Interesting

      Now, with multi-gigabit pipes making up the networks, data can be written, pushed, and read again, all at much higher bitrates than reading any storage medium. It's the read-write to physical medium that are the bottleneck with the sneakernet now.

      TFA says that they have 19 million SD cards. If each one is a mid-range 6 megabyte per second speed and we access them all in parallel, that gives 912 terabit per second potential max bandwidth, which almost certainly exceeds any network you're thinking about.

    3. Re:This is pointless by mooingyak · · Score: 4, Informative

      Now, with multi-gigabit pipes making up the networks, data can be written, pushed, and read again, all at much higher bitrates than reading any storage medium. It's the read-write to physical medium that are the bottleneck with the sneakernet now.

      TFA says that they have 19 million SD cards. If each one is a mid-range 6 megabyte per second speed and we access them all in parallel, that gives 912 terabit per second potential max bandwidth, which almost certainly exceeds any network you're thinking about.

      19M card readers (or slots or whatever) probably isn't even necessary.. I'm sure working in rotation you'd have a steady stream of cards whose data was fully read before other cards are even unpacked. I believe the optimal number of card readers would be (time to read) / (time to unpack).

      --
      William of Ockham had no beard. The most likely explanation is that it was chewed off by squirrels every morning.
    4. Re:This is pointless by jedidiah · · Score: 2

      > The original saying was coined in a time where reading from a tape *was considerably faster* than reading over a network.

      Reading from tape is still faster than reading from a network.

      Locally attached storage is always going to be faster. For large amounts of data, it will likely always be the case that a courier with a hard drive will move the data faster than a network.

      --
      A Pirate and a Puritan look the same on a balance sheet.
    5. Re:This is pointless by HockeyPuck · · Score: 3, Funny

      Yeah, but it's competing with high-speed networks that are crippled by the ISPs at both ends using a single fibre to feed an entire neighborhood, and intentionally slowing the speed at the customer's site to a crawl unless you pay an exorbitant rate for a higher speed (which is then unused 99% of the time, and doesn't deliver if 2 or 3 others in your neighborhood are using high speed at the same time).

      Last I checked an OC-768, as referenced in the article, isn't going to be crippled by the ISP.

      Maybe it's time for you to realize that your uber-ultimate-epic-extreme bandwidth package from your ISP isn't really that fast compared to say what's in the article...

    6. Re:This is pointless by Zero__Kelvin · · Score: 2, Informative

      "That would take far more time than the actual trip itself, but we're not counting it because the original saying "Never Underestimate the Bandwidth of a Station Wagon Filled with Backup Tapes" didn't count it either."

      The AC actually had something to say. That is rare and should be encouraged, not discouraged. He specifically pointed out why it wasn't considered ("The original saying was coined in a time where reading from a tape *was considerably faster* than reading over a network. Hence, transferring data via sneakernet was quicker, inclusive of the read-write times.", assuming it wasn't considered, which is a poor assumption backed by zero evidence, clearly made to try to lend credibility to a worthless idea and point.

      --
      Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
    7. Re:This is pointless by icebike · · Score: 5, Informative

      Well, actually not a poor assumption, considering that TFA also indicates that much data is 398,772 3TB hard drives and, moving that much data onto or off of that many drives isn't something that even Google can do in the time period measured. They discounted read-and write times on both media types.

      (Although they make no allowance for handling). Its a fun mind game, but as usual a pointless exercise.
      The Microsd cards will cost you $855 million dollars, and probably consume the entire production of 64gb cards for a year.

      --
      Sig Battery depleted. Reverting to safe mode.
    8. Re:This is pointless by bored · · Score: 4, Informative

      That is true for something like SD, but modern tape drives can do well over 500MB/sec in compressed streaming mode, and have native uncompressed capacities of 8.5TB.

      That means that even with a midrange tape library with 56 drives, your talking a read/write bandwidth of 27GB/sec (aka 1/3 Tbit/sec). Tape bandwidth scales linearly with the number of drives in the library, and things like the the SL8500 from STK can support up to 640 drives.

      It still faster if you have a PB of data you need shipped from NY to LA to write it to tape, put it on a plane and read it back in LA. Plus, it all fits nicely into a big suitcase. Furthermore, even for smaller amounts of data (say 10-20TB a day) the cost of a tape drive and an next day delivery is going to be significantly less than the Gbit/sec or so of bandwidth required to ship a similar amount of data in most places in the US.

    9. Re:This is pointless by sjames · · Score: 2

      Why in the world should I do all that for a simple thought exercise? Just to make you feel better?

      If I had an actual need, I would actually do the work. But if you like, you can go ahead and do a formal estimation of how long it might take me to do that, then rent yourself an OC-768 and see how much data you can actually cram down it. Don't forget to count the time it takes to get the contract negotiated and to get the people actually onsite to terminate the connection.

      We'll be waiting for your report.

    10. Re:This is pointless by sjames · · Score: 2

      You seem to want me to actually build one (since you demanded testing), so I want you to actually build one. Now MUSH!

    11. Re:This is pointless by turbidostato · · Score: 2

      "You can hook an eternal storage medium up to USB 3 or SATA"

      Buh! don't drink the cool-aid from storage vendors... their MTBF are not *that* high.

  3. Common sense almost prevails by jlf278 · · Score: 5, Informative

    From the article: "A MicroSD card is only .1 cubic inches, so if all things were equal you could stuff 100 64gb cards into a cubic inch of space! But, that does not seem realistic. In fact it doesn't even seem remotely possible." Perhaps that's because 1 cubic inch = 10 * 0.1 cubic inches and not 100 * 0.1 cubic inches.

    1. Re:Common sense almost prevails by jlf278 · · Score: 3, Informative

      ah, their mistake was the MicroSD card is only 0.01 cubic inches.

    2. Re:Common sense almost prevails by SJHillman · · Score: 4, Funny

      They got confused by microSD cards not being one millionth the size of an SD card. Damned misleading names.

  4. XKCD by Anonymous Coward · · Score: 5, Informative

    http://what-if.xkcd.com/31/
    That is all

  5. The bandwidth of a human. by queazocotal · · Score: 5, Funny

    Swallowing capsules containing a terabyte (about 12mm in diameter, and 15mm long of microSDs) is quite plausible.
    You can easily swallow a hundred of these, and it'll come out over the next 2 days.
    100TB/2 days = 600 megabytes a second.

    1. Re:The bandwidth of a human. by Anonymous Coward · · Score: 4, Funny

      Something something memory dump.

    2. Re:The bandwidth of a human. by EmagGeek · · Score: 2, Insightful

      If you're trying to smuggle your own data across the border without it being copied by the NSA (as is routine for entry into the US for CBP to confiscate and make an image of your laptop HDD for NSA), swallowing a MicroSD is not so implausible or impractical.

    3. Re: The bandwidth of a human. by davidbrit2 · · Score: 5, Funny

      I've got Crohn's disease, so my ping times are faster.

    4. Re:The bandwidth of a human. by phayes · · Score: 3, Insightful

      Yeah, right. Border authorities couldn't possibly detain someone on suspicion of smuggling data internally & wait for it to come out like they already do for drug mules. No, no, it could never happen...

      It'd be nice if you wouldn't demean words like "routine" into meaninglessness and stop confusing the NSA with the US customs authorities. Yeah we've all heard of incidents where laptops & drives have been confiscated but this is not a routine occurrence -- If you want to claim otherwise deliver a reliable reference giving the total number of incidents per year (you're the one claiming it's "routine" so it's on you to justify your claim). I'll divide that by the number of border crossings & we'll all have a benchmark on how often it happens per border crossing & just how "routine" it is.

      --
      Democracy is a sheep and two wolves deciding what to have for lunch. Freedom is a well armed sheep contesting the issue
    5. Re:The bandwidth of a human. by bill_mcgonigle · · Score: 2

      If you want to claim otherwise deliver a reliable reference giving the total number of incidents per year

      You can argue frequency all you want, but so long as the unconstitutional searches are > 1 *and* well-publicized, the panopticon effect becomes real.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  6. Calculations omit one tiny detail by Ygorl · · Score: 4, Insightful

    As they point out in the article, the tremendous bandwidth achieved does not include the logistics or time required to initially copy the data onto SD cards, and then back off of the cards upon reaching the destination. Still, beats a flock of parrots trained in Morse code.

  7. Semi Truck Full of HDDs by sanosuke001 · · Score: 2

    I did something similar after a conversation at work using a 53' Semi Trailer and 4TB HDDs.

    Semi = 630" x 94" x 102"
    HDD = 4" x 1" x 5.75"
    Total HDDs = 262628
    Total Storage = 1050512 TB
    Bandwidth from NYC to SF = 55.58 Tbps (42 hours according to google maps)

    --
    -SaNo
    1. Re:Semi Truck Full of HDDs by Bengie · · Score: 3, Funny

      Imagine the TCP transmit window! A single lost packet would wreak havoc on that stream.

  8. This old chest nut again by kannibal_klown · · Score: 4, Informative

    Great, you've managed to transfer X terrabytes of data across state lines. I salute you.

    Now... go find this list of files that I need. Also that collection of data that our servers need to process. What, that's going to take you HOW long?

    Yes might be quicker to send the data from point A to point B by just shipping disks... but only for archiving purposes. If you actually need to access the data, then you still have that last mile (or 10') of having to load that data into a system or network.

    Now, if you were shipping a truck load of servers, or maybe a car full of NAS devices that you could just plug into a network and be done with it... then it's not too bad. Then you can start electronically searching the data within minutes, index it, use it for your data-store, whatever.

    But just a car load of disks for non-archival purposes... you're asking for a headache.

    1. Re:This old chest nut again by jedidiah · · Score: 2

      Use an array enclosure and plug it directly into the system that needs to use it. 8-bay arrays are cheap and plentiful already. You don't have to get into anything "expensive and fancy".

      Even a "car full of disks" is doable if you already have them set up to be plugged in at the destination. Many array enclosures have been built with this in mind for decades now.

      --
      A Pirate and a Puritan look the same on a balance sheet.
  9. Re:Bandwidth by EmagGeek · · Score: 3, Funny

    "That's more porn than the whole Slashdot can consume."

    Challenge: ACCEPTED!

  10. Re:They forgot something by jones_supa · · Score: 2

    Exactly. For most scenarios a high-bandwidth link is not that useful if the latency goes to hell.

    Even for my normal internet usage, I'd rather take a 5Mb/s connection with 2ms latency (to ISP's default GW) than a 100Mb/s connection with 20ms latency.

  11. Petabyte tablet by Ken_g6 · · Score: 3, Informative

    In 3001: The Final Odyssey, Arthur C. Clarke wrote about a petabyte tablet as though it were the ultimate storage medium, something humanity finally arrived at after a millennium. In the book it was enough to store the contents of a human brain!

    16,000 or so microSD cards could store a petabyte in roughly a 1-foot by 1-foot by 2-inch space, probably leaving enough room to wire them up as well. Of course, it would cost nearly a million bucks, not counting the hardware necessary to wire it up to be accessed. But, still, I find that very impressive.

    --
    (T>t && O(n)--) == sqrt(666)
    1. Re:Petabyte tablet by Anonymous Coward · · Score: 2, Interesting

      If that's the scene I'm thinking about, they were trying to pull incredibly valuable data off an ancient computer on a dead planet. Considering its condition, that kind of file recovery speed would make me happy, too.

  12. A 2014 suburban filled with SD cards by Arancaytar · · Score: 4, Funny

    Assuming 2500kg for the car, 0.0005kg per cards, $50000 for the car and $50 per card:

    Not only a bandwidth of 68Tbps, but a mass of 12 metric tons (4.8 empty cars).

    Also a market value of $0.96B, or the equivalent of 19,141 new cars, plus one car with a broken suspension.

  13. Re:station wagon? by istartedi · · Score: 5, Interesting

    A station wagon is what moms wanted to drive. Whenever somebody says "Americans want SUVs". They're wrong. The SUV is a legal hack to get around fuel economy standards. See? If we build a station wagon with a center of gravity so high that it tips over in the parking lot, it's legally defined as a truck and we don't have to meet the same standards.

    Sorry to get into this, but it's one of my pet peeves. Whenever I hear, "Americans want SUVs" it just grates on my nerves. No we didn't. We wanted station wagons. Mom didn't want to tip the kids over and throw them, her, and the groceries into a ditch. Shortsighted regulators left a loophole in CAFE, and they literally drove a truck through it.

    Now all these kids don't even know what a station wagon is. Sounds about right. It's the vehicle that the mom down the block had. I distinctly remember us piling in there with the neighbor kids on more than one occasion, and she smoked like a chimney. Shotgun! I get to ride up front with Mrs. Potter and yeah, it smells up here but we didn't know nothin'. We didn't wear seatbelts and... well... I know this is survivorship bias talking but... we survived!

    In other words, get your damned SUV off my lawn.

    --
    For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
  14. Stupid fiddly little things by Hognoxious · · Score: 3, Funny

    However many cubic inches it is, it's the fundamental problem with this method. The one with the really crucial data on it is bound to find its way under a seat, into an ashtray etc.

    --
    Confucius say, "Find worm in apple - bad. Find half a worm - worse."
  15. Re:station wagon? by Jah-Wren+Ryel · · Score: 2

    Whenever somebody says "Americans want SUVs". They're wrong.

    Men don't like to drive station wagons and women like to feel invulnerable. Tipping over is a lot less likely than hitting or getting hit and women as a group are paranoid that if they don't have the biggest monster on the road, they and their children will be killed by someone else driving the biggest monster on the road who hits them.

    You might argue that if nobody had SUVs then nobody would want them. But that ain't the world we live in. So it is entirely true to say that "American want SUVs" no matter how we arrived at the current state.

    --
    When information is power, privacy is freedom.
  16. Re:station wagon? by Sorny · · Score: 3, Interesting

    "A station wagon is what moms wanted to drive. Whenever somebody says "Americans want SUVs". They're wrong. The SUV is a legal hack to get around fuel economy standards. See? If we build a station wagon with a center of gravity so high that it tips over in the parking lot, it's legally defined as a truck and we don't have to meet the same standards.

    Sorry to get into this, but it's one of my pet peeves. Whenever I hear, "Americans want SUVs" it just grates on my nerves. No we didn't. We wanted station wagons. Mom didn't want to tip the kids over and throw them, her, and the groceries into a ditch. Shortsighted regulators left a loophole in CAFE, and they literally drove a truck through it.

    Now all these kids don't even know what a station wagon is. Sounds about right. It's the vehicle that the mom down the block had. I distinctly remember us piling in there with the neighbor kids on more than one occasion, and she smoked like a chimney. Shotgun! I get to ride up front with Mrs. Potter and yeah, it smells up here but we didn't know nothin'. We didn't wear seatbelts and... well... I know this is survivorship bias talking but... we survived!

    In other words, get your damned SUV off my lawn."

    This American wants his SUV, not a station wagon.
    I have an SUV instead of a station wagon. Not because I'm a mom (I'm not), and not for grocery getting, but because I live in a place with 4 real seasons and big 4x4 SUVs are handy for getting around for the 2 days it takes the DOT to clear roads after a winter storm. Besides, a car won't pull my snowmobile trailer or the boat.

    That said, I don't use the SUV for daily driving. It's 17 years old and unlike the vast majority of SUVs, is muddy as often as not since it also does go off road. As a 2nd vehicle, it is great. Mine is nowhere near as big as a Suburban (2 door full size Yukon GT), and I sure as hell wouldn't want to load it up with micro SD cards; each one may be tiny, but enough to fill the thing up would have it so grossly overloaded it wouldn't be funny.

    I'll try to keep my SUV off your lawn.

    --
    OSX pwns.