I actually built a similar system, but you lost me at
"run a backup utility... Norton Ghost to wipe the drive"
ImageX is built-in to Windows (in the AIK) and it does a fantastic job of backups, it even does compression and single-instance storage to save time and space. To wipe the disks, you can run any number of free/cheap utilities (Active KillDisk?) or you can just run 'diskpart' with 'clean all' to write zeros (good enough for 99.5% of cases).
"Linux dd doesn't have a problem doing the backups of anything as long as it is mounted"
Linux DD will also save all your deleted data as gobbledygook and lead to GIANT image files. If anything, you want these backups at the file-level, not block-level. Bonus points if you can backup to something with deduplication or single-instance storage.
You'll need a Windows Server 2008 R2 with Windows Deployment Services role. You basically want to set up an isolated network with PXE booting, load a Windows PE disk into the PXE server. Modify the PE image to mount a drive off the server (to store your backups), then run a wipe script. As soon as the backup is done, you can actually fire up the next machine, you don't need to be 'connected' to wipe the disk.
For our purposes, we use Active Killdisk to wipe, and ImageX to backup.
You'll need PXE-enabled NICs or a gPXE boot disk. You can also do the exact same thing with a Samba server, a PXE daemon, and a basic Linux boot drive being served-up over PXE, but the learning curve is steeper. Also, ImageX is pretty awesome because it allows single-instance storage. You can append ALL you computer backups to one image file and only the different files will be stored more than once, cutting a massive amount of time and space from your backups.
> I sure wouldn't like some guy touching my food before I eat it
I love this idea that people seem to have that food is somehow at all 'clean'. Everything about food except for the part when the server brings it to you or the stockboy puts it on the shelf is hands-on and 'unsanitary' by most people's understanding.
I've fished commercially before. Do you know what the back-end of that market is like? Imagine a football-field sized warehouse with two or three inches of fish-gut water flowing by while strange-looking dudes poke and prod your fish and offer prices.
Those bags of chickpeas at the falafel place are sitting on the sidewalk for a good fifteen minutes, where thousands of people walk, hundreds spit, and several piss or vomit every day.
It's not a big deal, the world is awash with benign germs, they're not going to kill you. The stuff that will make you sick is the real nasty germs found in spoiled meats and such, not the harmless stuff on all of our hands.
In reality, someone picking through a bag of M&Ms isn't at all unsanitary or unsettling.
In that case, it's time top open up some performance counters and figure out what's causing the bottleneck. It could be the VM or host choking on interrupts, latency in the network adapters (virtual or physical), or the TCP window being too small for the latency.
Well yeah, I can saturate a gigabit link from a Windows 2000 box running a Pentium III, or on my embedded Linux-based consumer router. My guess is that the issue here isn't actual network performance, it's the underlying disk performance. 600-700mbits is remarkably close to the single-spindle speed of the average spinning media.
"And any 100+ connection is materially more useful today than the existing 10-20Mbps connections."
Do you really think so?
Nobody at my house (four users) even noticed when I changed from 5mbits to 20mbits. Hell, from the back yard the wireless only gets 3-5mbits anyways.
I noticed, but that's because I do a fair amount of large file transfers and remote backups. Most people don't. Most people just want to watch Netflix while Johnny browses. Netflix connections only stream at a few megabits anyways. Even raw blu-ray is only 36mbits, and any distribution medium is going to use a codec that squeezes a bit more, just for sanity.
I have a 20mbit uplink at home and a milti-GB uplink at work.
Honestly, I see almost no difference when it comes to 'regular stuff' like watching movies, browsing, email, and gaming. Home users generally don't need to upload or download multi-gigabyte archives to the 'net. When they do need to download huge files, it's usually in the context of streaming, where the downlink just needs to be fast enough to play the media without hosing the connection.
"...in a virtual environment where there's no possibility of crappy hardware or drivers."
Um, there's definitely going to be a performance hit running in a VM. Every packet between VMs generates an interrupt on each VM involved and at least one on the host. There also very much is the chance for crappy drivers to be involved, your CPU is likely emulating an ancient generic NIC. Try using the Paravirtual drivers if you can, they'll offload as much as possible to the host, which knows more about how to schedule and buffer things.
Well yes, sort of, but the 'receive window' is the size of the data to be transmitted. TCP is designed to give reliability in near real-time, this kind of application needs nothing like that.
Well there's no point is using IP at all, IP is designed to allow for four billion addresses. What you need in this case is a point-to-point, like a serial cable, not a network stack. Having a network stack would seriously add to the overhead.
And you certainly don't need UDP, or FTP running inside it. Dropped data is a no-no, I believe, but there's no reason you can't just send commands and return data in the raw, then ask for corrupted blocks from the data to be sent again.
It would look like this:
E: Send me file 102. M: Sends file.... E: Thanks, now send me sections x, y, and z of that file again because I missed it. M: X, Y, Z... E: Thanks, now delete the file, point the camera ten degrees left, and take another picture.
How do you know if the data is corrupt? Well you do CRC checking on the wire, and you split all file transfers into chunks and include hashes with them. The chunks need to be right-sized so there aren't too many retransmissions, but big enough not to incur massive latency and overhead.
"If it does not benefit me, then it does not need my support, and will cease to exist when it does not maintain my support."
Um, you aren't familiar with how government projects really work then, are you?
I won't engage much more, just pointing out that while you're spinning a tale of some magical utopia where everything is optimized through the government, the VAST majority of us in the USA feel very differently about it. In other terms: "Your ideas do not benefit us, so you don't have our support."
Not just the desert. I live in the northeast US, temperature fluctuations in the spring and fall can be 30+ degrees in a matter of hours. If I packed my walls with this stuff set to phase-change at 65F, I could probably go until December before turning on the heating system. In the summer, it would probably totally negate the need for air conditioning (not that I have it anyway), since nights tend to be below 65 even when days are in the 90s.
The big question I have in terms of practical use in a retrofit revolves around mold mitigation and the dew point. If my walls are 65F when it hits 90 out, they'll start condensing water, which isn't good for homes.
The Prius sells well because it's a Prius and it's associated psychologically with eco-friendliness, not because it gets good mileage or is a good value. The sales of other lines of cars that get higher mileage but look exactly like their gas-only counterparts are dismal.
Hybrid technology isn't even needed to get to 55+ MPG, Ford sells a version of my Focus in Europe that gets almost 70MPG (yes, on diesel), which would be an economic/environmental win/win here in the USA, but they say that there's 'no market for it.'
Americans are -very- strange consumers. On the whole, they don't look at long-term value, they look at appearances and price. Just look at the kind of consumer goods we buy now, we'd rather buy cheaply constructed goods and replace them annually than buy durable high-quality stuff. They'll buy a sports car or SUV and vote for the guy who promises cheaper gas.
Or based on Apple's success with iOS, which derives directly from OS X/Darwin.
There are really big advantages to keeping one solid, portable 'core' OS and building libraries that can be 'scaled down' for mobile devices. Just look at how Apple's small team of OS developers have things set up:
Darwin becomes OS X and iOS... Webkit is the browser and JavaScript app runtime on both platforms. The CoreAnimation API that accelerates OS X eye candy is used in iOS to do the screen drawing (ala DirectX on Xbox and WIndows).
A mobile developer can easily graduate to application development on the main platform, code can be re-used, and security and performance improvements correlate on both platforms. I'd say that there are 'monoculture' security issues, and I'm sure there are, but I don't think that most 'buffer overflows' can be cross-platform by their very nature, and these portable devices are running on ARM, while the desktops are amd64.
Well, as an electric customer, I'd rather save a few bucks a year than have you guys spend it building redundancy for once-a-century events. Really. Now if this stuff was happening often and I was cold and shivering for too long, I might change my mind, but so far, so good.
I deal with this shit all the time in I.T., do we want to spend $500,000 on a storage system that 'never' goes down and can handle fifteen times the load we could possibly generate, or $50,000 on a system that has a few hours of downtime a year and saturates the pipe it's connected to?
Say that this kind of upgrade you're talking about adds $1 to my $100 monthly bill and these events happen every three decades... That's $360 for a night without electricity every three decades. I'll take it. If it happened every year, I'd gladly pay $12 to keep the lights on.
It's my job to look at why your comany is still using Y when it costs several servers and a whole person, then find a way to replace Y with Z, which I will operate for a decade before someone does the same to me.
I like some of this idea, but frankly, it doesn't go far enough. Take a look at Windows PowerShell. Instead of the UNIX 'everything is a file' philosophy, it says 'everything is an object', and it's pretty cool.
I would pay good money for a PowerShell implementation on Linux, and even more if Linux internals were exposed in the same way that WMI objects are on Windows.
Slashdot Mirror
I actually built a similar system, but you lost me at
"run a backup utility... Norton Ghost to wipe the drive"
ImageX is built-in to Windows (in the AIK) and it does a fantastic job of backups, it even does compression and single-instance storage to save time and space. To wipe the disks, you can run any number of free/cheap utilities (Active KillDisk?) or you can just run 'diskpart' with 'clean all' to write zeros (good enough for 99.5% of cases).
"Linux dd doesn't have a problem doing the backups of anything as long as it is mounted"
Linux DD will also save all your deleted data as gobbledygook and lead to GIANT image files. If anything, you want these backups at the file-level, not block-level. Bonus points if you can backup to something with deduplication or single-instance storage.
Here's what we do where I work:
You'll need a Windows Server 2008 R2 with Windows Deployment Services role. You basically want to set up an isolated network with PXE booting, load a Windows PE disk into the PXE server. Modify the PE image to mount a drive off the server (to store your backups), then run a wipe script. As soon as the backup is done, you can actually fire up the next machine, you don't need to be 'connected' to wipe the disk.
For our purposes, we use Active Killdisk to wipe, and ImageX to backup.
You'll need PXE-enabled NICs or a gPXE boot disk. You can also do the exact same thing with a Samba server, a PXE daemon, and a basic Linux boot drive being served-up over PXE, but the learning curve is steeper. Also, ImageX is pretty awesome because it allows single-instance storage. You can append ALL you computer backups to one image file and only the different files will be stored more than once, cutting a massive amount of time and space from your backups.
This. If you can MITM and decrypt 500MB of HDMI data on the fly, you may as well do some basic video compression before dumping out to disk.
Also, for high-data linear loads like video dumps, HDDs are still superior to SSDs.
> I sure wouldn't like some guy touching my food before I eat it
I love this idea that people seem to have that food is somehow at all 'clean'. Everything about food except for the part when the server brings it to you or the stockboy puts it on the shelf is hands-on and 'unsanitary' by most people's understanding.
I've fished commercially before. Do you know what the back-end of that market is like? Imagine a football-field sized warehouse with two or three inches of fish-gut water flowing by while strange-looking dudes poke and prod your fish and offer prices.
Those bags of chickpeas at the falafel place are sitting on the sidewalk for a good fifteen minutes, where thousands of people walk, hundreds spit, and several piss or vomit every day.
It's not a big deal, the world is awash with benign germs, they're not going to kill you. The stuff that will make you sick is the real nasty germs found in spoiled meats and such, not the harmless stuff on all of our hands.
In reality, someone picking through a bag of M&Ms isn't at all unsanitary or unsettling.
In that case, it's time top open up some performance counters and figure out what's causing the bottleneck. It could be the VM or host choking on interrupts, latency in the network adapters (virtual or physical), or the TCP window being too small for the latency.
FireWire was also supposed to 'go optical' at some point, but market forces kept it copper.
Well yeah, I can saturate a gigabit link from a Windows 2000 box running a Pentium III, or on my embedded Linux-based consumer router. My guess is that the issue here isn't actual network performance, it's the underlying disk performance. 600-700mbits is remarkably close to the single-spindle speed of the average spinning media.
"And any 100+ connection is materially more useful today than the existing 10-20Mbps connections."
Do you really think so?
Nobody at my house (four users) even noticed when I changed from 5mbits to 20mbits. Hell, from the back yard the wireless only gets 3-5mbits anyways.
I noticed, but that's because I do a fair amount of large file transfers and remote backups. Most people don't. Most people just want to watch Netflix while Johnny browses. Netflix connections only stream at a few megabits anyways. Even raw blu-ray is only 36mbits, and any distribution medium is going to use a codec that squeezes a bit more, just for sanity.
I have a 20mbit uplink at home and a milti-GB uplink at work.
Honestly, I see almost no difference when it comes to 'regular stuff' like watching movies, browsing, email, and gaming. Home users generally don't need to upload or download multi-gigabyte archives to the 'net. When they do need to download huge files, it's usually in the context of streaming, where the downlink just needs to be fast enough to play the media without hosing the connection.
"...in a virtual environment where there's no possibility of crappy hardware or drivers."
Um, there's definitely going to be a performance hit running in a VM. Every packet between VMs generates an interrupt on each VM involved and at least one on the host. There also very much is the chance for crappy drivers to be involved, your CPU is likely emulating an ancient generic NIC. Try using the Paravirtual drivers if you can, they'll offload as much as possible to the host, which knows more about how to schedule and buffer things.
Cool, I've never heard of that before. Thanks!
Well yes, sort of, but the 'receive window' is the size of the data to be transmitted. TCP is designed to give reliability in near real-time, this kind of application needs nothing like that.
Well there's no point is using IP at all, IP is designed to allow for four billion addresses. What you need in this case is a point-to-point, like a serial cable, not a network stack. Having a network stack would seriously add to the overhead.
And you certainly don't need UDP, or FTP running inside it. Dropped data is a no-no, I believe, but there's no reason you can't just send commands and return data in the raw, then ask for corrupted blocks from the data to be sent again.
It would look like this:
E: Send me file 102.
M: Sends file....
E: Thanks, now send me sections x, y, and z of that file again because I missed it.
M: X, Y, Z...
E: Thanks, now delete the file, point the camera ten degrees left, and take another picture.
How do you know if the data is corrupt? Well you do CRC checking on the wire, and you split all file transfers into chunks and include hashes with them. The chunks need to be right-sized so there aren't too many retransmissions, but big enough not to incur massive latency and overhead.
"If it does not benefit me, then it does not need my support, and will cease to exist when it does not maintain my support."
Um, you aren't familiar with how government projects really work then, are you?
I won't engage much more, just pointing out that while you're spinning a tale of some magical utopia where everything is optimized through the government, the VAST majority of us in the USA feel very differently about it. In other terms: "Your ideas do not benefit us, so you don't have our support."
Not just the desert. I live in the northeast US, temperature fluctuations in the spring and fall can be 30+ degrees in a matter of hours. If I packed my walls with this stuff set to phase-change at 65F, I could probably go until December before turning on the heating system. In the summer, it would probably totally negate the need for air conditioning (not that I have it anyway), since nights tend to be below 65 even when days are in the 90s.
The big question I have in terms of practical use in a retrofit revolves around mold mitigation and the dew point. If my walls are 65F when it hits 90 out, they'll start condensing water, which isn't good for homes.
I see your *whooosh* and I reverse it upon you:
http://en.wikipedia.org/wiki/Glendale,_California#Armenian_population
What's a Bothnian? Is that like an Bothan from Glendale, CA?
The Prius sells well because it's a Prius and it's associated psychologically with eco-friendliness, not because it gets good mileage or is a good value. The sales of other lines of cars that get higher mileage but look exactly like their gas-only counterparts are dismal.
Hybrid technology isn't even needed to get to 55+ MPG, Ford sells a version of my Focus in Europe that gets almost 70MPG (yes, on diesel), which would be an economic/environmental win/win here in the USA, but they say that there's 'no market for it.'
Americans are -very- strange consumers. On the whole, they don't look at long-term value, they look at appearances and price. Just look at the kind of consumer goods we buy now, we'd rather buy cheaply constructed goods and replace them annually than buy durable high-quality stuff. They'll buy a sports car or SUV and vote for the guy who promises cheaper gas.
Agreed. I'm glad the users here all see it, because I'm losing faith quickly.
Or based on Apple's success with iOS, which derives directly from OS X/Darwin.
There are really big advantages to keeping one solid, portable 'core' OS and building libraries that can be 'scaled down' for mobile devices. Just look at how Apple's small team of OS developers have things set up:
Darwin becomes OS X and iOS...
Webkit is the browser and JavaScript app runtime on both platforms.
The CoreAnimation API that accelerates OS X eye candy is used in iOS to do the screen drawing (ala DirectX on Xbox and WIndows).
A mobile developer can easily graduate to application development on the main platform, code can be re-used, and security and performance improvements correlate on both platforms. I'd say that there are 'monoculture' security issues, and I'm sure there are, but I don't think that most 'buffer overflows' can be cross-platform by their very nature, and these portable devices are running on ARM, while the desktops are amd64.
10 miles of rail in one day
This makes me sad. In my state we're going to take several years and $25 million dollars to add a 1.5 mile section alongside an existing track.
http://www.projo.com/news/content/KINGSTON_STATION_05-28-11_KDOB7US_v9.3040bfb.html
free market be damned
Well, as an electric customer, I'd rather save a few bucks a year than have you guys spend it building redundancy for once-a-century events. Really. Now if this stuff was happening often and I was cold and shivering for too long, I might change my mind, but so far, so good.
I deal with this shit all the time in I.T., do we want to spend $500,000 on a storage system that 'never' goes down and can handle fifteen times the load we could possibly generate, or $50,000 on a system that has a few hours of downtime a year and saturates the pipe it's connected to?
Say that this kind of upgrade you're talking about adds $1 to my $100 monthly bill and these events happen every three decades... That's $360 for a night without electricity every three decades. I'll take it. If it happened every year, I'd gladly pay $12 to keep the lights on.
It's my job to look at why your comany is still using Y when it costs several servers and a whole person, then find a way to replace Y with Z, which I will operate for a decade before someone does the same to me.
(cue 'Circle of Life')
I like some of this idea, but frankly, it doesn't go far enough. Take a look at Windows PowerShell. Instead of the UNIX 'everything is a file' philosophy, it says 'everything is an object', and it's pretty cool.
I would pay good money for a PowerShell implementation on Linux, and even more if Linux internals were exposed in the same way that WMI objects are on Windows.
And this is from a thirteen-year Linux veteran.