I love Dropbox, but hope that in the future they will integrate with other storage systems besides S3. After reading Backblaze's blog the other day about how they designed their own cheap storage since no one else offered it for their backup startup, I'd love for them to put the S3 API onto their storage platform and watch the price of clustered (people who use "cloud" should be beaten") storage drop like a rock.
I mean what's next? Engineer ourselves to not feel pain? Then is it OK to murder?
Poor example. I'd say, "Is it ok to murder someone if you know their consciousness will be immediately downloaded into an exact duplicate of their body?" Murder isn't about pain, as you can die in non-painful ways. It's about not existing anymore.
Good article, I coundn't find anything to argue with in it. I never did understand why the concept of "cloud computing" was attractive to anyone. I wish someone would explain it to me.
No upfront investment. Example: Amazon invests huge amounts of cash in infrastructure so they can handle transactions at peak times (Christmas). The rest of the year that gear sits idle. You get to use it for your app at a per hour rate, and it will scale quickly if your app/site/whatever are a hit. Have an idea but not the gear to demo it? You use the cloud, and your only cost is the rental time fee. Have a hugely popular site already? You use dedicated equipment in your own space.
Not all the fiber is where it needs to be. In rural areas it may run next to rail lines that travel through the country. To pull fiber to homes = expensive. To pull fiber to cell towers = expensive (most towers currently use a T1/T3 or microwave backhaul currently).
Most carriers are moving to software driven radios. For them, moving from GSM to WCDMA will be as simple as changing config options and restarting each cell overnight.
Wouldn't your reception be different because of different frequencies but not your throughput? It might all equal out, as AT&T may have more swaths of frequency but it's brought to it's knees by all the users thereby T-mobile being actually faster in the long run. Disclaimer: I'm a T-Mobile customer, and have had no problems with their data speeds. Rather pay less for slower speeds than get raped by AT&T for shitty service.
Thanks for reminding me! I'm taking a trip to Japan in Oct. and everyone has told me I need to take a 3G phone with (I have a Blackberry Curve). Now I remember I can turn my Wifi on and use T-Mobile's Hotsport@Home to make calls/texts for free back home as long as there is wifi. Sweet!
For the solid state gear, you can use heat exchangers running along the exterior of the aircraft. It's mighty chilly at FL350. You just need efficient heat pipes to get the heat outside (like you mentioned).
HDFS transparently checksums all data written to it and by default verifies checksums when reading data. A separate checksum is created for every io.bytes.per.checksum bytes of data. The default is 512 bytes, and since a CRC-32 checksum is 4 bytes long, the storage overhead is less than 1%.
Datanodes are responsible for verifying the data they receive before storing the data and its checksum. This applies to data that they receive from clients and from other datanodes during replication. A client writing data sends it to a pipeline of datanodes (as explained in Chapter 3), and the last datanode in the pipeline verifies the checksum. If it detects an error, the client receives a ChecksumException, a subclass of IOException.
When clients read data from datanodes, they verify checksums as well, comparing them with the ones stored at the datanode. Each datanode keeps a persistent log of checksum verifications, so it knows the last time each of its blocks was verified. When a client successfully verifies a block, it tells the datanode, which updates its log. Keeping statistics such as these is valuable in detecting bad disks.
Aside from block verification on client reads, each datanode runs a DataBlockScanner in a background thread that periodically verifies all the blocks stored on the datanode. This is to guard against corruption due to "bit rot" in the physical storage media. See Section 10.1.4.3 for details on how to access the scanner reports.
I don't think RAID6 is going to be all that slow across 15 SATA drives. Bonus if the controller has acceleration to speed up parity calculations for faster writes.
Your bad sectors should be automatically reassigned via the drive's firmware anyway. And the RAID cards should be doing surface scans on a fairly regular basis. So no, you don't need to worry about individual errors on storage nodes when you're storing a chunk of data across several servers. Google for Hadoop and read how it handles storing chunks of data (two copies in one rack on different servers, one copy on a node in a different rack).
Really? Fiber channel tops out at what? 4Gb/sec? 8Gb/sec? Distribute your data chunks across enough chunk servers, and you can easily compete against that much cheaper.
Disclaimer: I'm currently doing HPC work at a US accelerator lab as part of one of the LHC experiments. I know how to move data around *fast*.
Show me an admin job where you *can't* do damage when you leave. You can always do damage when you leave if you're intelligent enough. What kind of damage is determined by the situation.
After all that, he was just being a dick, and it's stupid to be a dick to people who can have you locked up in jail. This may be United States, but it's not the United States you were (probably) taught about in grade school.
True. But it's also against the law to lock someone up just for being an arrogant asshole. When he gets out, he's going to be able to sue the fuck out of them. Enjoy the windfall sir!
Funny story. I'm in the same boat. Netflix and Hulu streaming to everything with PlayOn and a PS3 and Popcorn Hour box. Comcast begged me to sign up for television. They're giving me a year free (I was already paying $60/month for high speed internet). I consider it a nice thing to have to watch stuff like the Food Network until more stuff moves to Hulu-like broadcast methods. Once my free year is up, back to internet only and getting all my content for free online (and my $16/month Netflix subscription).
Slashdot Mirror
I love Dropbox, but hope that in the future they will integrate with other storage systems besides S3. After reading Backblaze's blog the other day about how they designed their own cheap storage since no one else offered it for their backup startup, I'd love for them to put the S3 API onto their storage platform and watch the price of clustered (people who use "cloud" should be beaten") storage drop like a rock.
p.s. I tried to use the N with a squiggle over it. The slashdot editor choked on it =(
I mean what's next? Engineer ourselves to not feel pain? Then is it OK to murder?
Poor example. I'd say, "Is it ok to murder someone if you know their consciousness will be immediately downloaded into an exact duplicate of their body?" Murder isn't about pain, as you can die in non-painful ways. It's about not existing anymore.
Good article, I coundn't find anything to argue with in it. I never did understand why the concept of "cloud computing" was attractive to anyone. I wish someone would explain it to me.
No upfront investment. Example: Amazon invests huge amounts of cash in infrastructure so they can handle transactions at peak times (Christmas). The rest of the year that gear sits idle. You get to use it for your app at a per hour rate, and it will scale quickly if your app/site/whatever are a hit. Have an idea but not the gear to demo it? You use the cloud, and your only cost is the rental time fee. Have a hugely popular site already? You use dedicated equipment in your own space.
The "because you can" argument is getting old. I like the "because you can AND you can benefit from it".
Feel free to start "Applied Linux Journal"
Not all the fiber is where it needs to be. In rural areas it may run next to rail lines that travel through the country. To pull fiber to homes = expensive. To pull fiber to cell towers = expensive (most towers currently use a T1/T3 or microwave backhaul currently).
Most carriers are moving to software driven radios. For them, moving from GSM to WCDMA will be as simple as changing config options and restarting each cell overnight.
Seconded. I heart T-Mobile. Have been with them since they were Voicestream (before VS got bought by TMO).
Wouldn't your reception be different because of different frequencies but not your throughput? It might all equal out, as AT&T may have more swaths of frequency but it's brought to it's knees by all the users thereby T-mobile being actually faster in the long run. Disclaimer: I'm a T-Mobile customer, and have had no problems with their data speeds. Rather pay less for slower speeds than get raped by AT&T for shitty service.
Thanks for reminding me! I'm taking a trip to Japan in Oct. and everyone has told me I need to take a 3G phone with (I have a Blackberry Curve). Now I remember I can turn my Wifi on and use T-Mobile's Hotsport@Home to make calls/texts for free back home as long as there is wifi. Sweet!
For the solid state gear, you can use heat exchangers running along the exterior of the aircraft. It's mighty chilly at FL350. You just need efficient heat pipes to get the heat outside (like you mentioned).
My mistake. In that case you need to ensure checksums are being done, and you have the data stored on more than 1 box.
Couldn't use HDFS due to lack of HSM, so I do HDFS on my own time.
4.1.1. Data Integrity in HDFS
HDFS transparently checksums all data written to it and by default verifies checksums when reading data. A separate checksum is created for every io.bytes.per.checksum bytes of data. The default is 512 bytes, and since a CRC-32 checksum is 4 bytes long, the storage overhead is less than 1%.
Datanodes are responsible for verifying the data they receive before storing the data and its checksum. This applies to data that they receive from clients and from other datanodes during replication. A client writing data sends it to a pipeline of datanodes (as explained in Chapter 3), and the last datanode in the pipeline verifies the checksum. If it detects an error, the client receives a ChecksumException, a subclass of IOException.
When clients read data from datanodes, they verify checksums as well, comparing them with the ones stored at the datanode. Each datanode keeps a persistent log of checksum verifications, so it knows the last time each of its blocks was verified. When a client successfully verifies a block, it tells the datanode, which updates its log. Keeping statistics such as these is valuable in detecting bad disks.
Aside from block verification on client reads, each datanode runs a DataBlockScanner in a background thread that periodically verifies all the blocks stored on the datanode. This is to guard against corruption due to "bit rot" in the physical storage media. See Section 10.1.4.3 for details on how to access the scanner reports.
I don't think RAID6 is going to be all that slow across 15 SATA drives. Bonus if the controller has acceleration to speed up parity calculations for faster writes.
We're about 20-30% write 70-80% reads. That could have something to do with it.
What kind of drives are you using? We've got 4800+ spinning drives, and we only have 1-2 failures a month.
Yes. You get a beer. Cheap resiliency is better than expensive redundancy.
Your bad sectors should be automatically reassigned via the drive's firmware anyway. And the RAID cards should be doing surface scans on a fairly regular basis. So no, you don't need to worry about individual errors on storage nodes when you're storing a chunk of data across several servers. Google for Hadoop and read how it handles storing chunks of data (two copies in one rack on different servers, one copy on a node in a different rack).
Disclaimer: I'm currently doing HPC work at a US accelerator lab as part of one of the LHC experiments. I know how to move data around *fast*.
If you need the support, go pay the premium. Those of us with the appropriate technical background welcome the cheaper implementations.
Show me an admin job where you *can't* do damage when you leave. You can always do damage when you leave if you're intelligent enough. What kind of damage is determined by the situation.
After all that, he was just being a dick, and it's stupid to be a dick to people who can have you locked up in jail. This may be United States, but it's not the United States you were (probably) taught about in grade school.
True. But it's also against the law to lock someone up just for being an arrogant asshole. When he gets out, he's going to be able to sue the fuck out of them. Enjoy the windfall sir!
Doesn't uTorrent support uPnP so even the port forwarding is handled automatically? I don't know as I'm a Vuze person myself.
Funny story. I'm in the same boat. Netflix and Hulu streaming to everything with PlayOn and a PS3 and Popcorn Hour box. Comcast begged me to sign up for television. They're giving me a year free (I was already paying $60/month for high speed internet). I consider it a nice thing to have to watch stuff like the Food Network until more stuff moves to Hulu-like broadcast methods. Once my free year is up, back to internet only and getting all my content for free online (and my $16/month Netflix subscription).