Plane crash tests - in so far as they are done - are thus more about making borderline situations more survivable. Not flying a perfectly good Airbus into the water at speed in a sustained stall, but ditching an airliner in the Hudson after ingesting geese and losing all power. Bringing one down on an abandoned runway turned speedway in a successful unpowered glide. Crash landing without main and/or nose gear without the aircraft breaking up or tumbling. Getting passengers off alive and quickly when an aircraft catches fire on the ground. Those sorts of situations DO matter; surviving a mid-air collision, break-up or controlled flight into terrain not so much.
Unfortunately they're so expensive to do that we tend to rely on investigation reports from the real thing to inform future design choices and suggest revisions. Important and great, but a lot could be learned especially about making aircraft water-ditchable and field-crash-landable by doing remote control crash tests. I suspect it isn't considered worth the cost.
Crashes give you one important thing - things you never thought to test. This is true of both staged and real crashes. Controlled tests find out how much a plane can take; crashes help you find things to check for in controlled tests and new things to add to design guidelines.
Pity it is so expensive to crash airliners for testing.
OK, I've gone and looked into the current situation with tape. All prices are in AUD from vendors in Australia; 1 AUD = 1.06 USD at the moment, so they're close enough to the the same.
LTO-5 drives are now AU$2500 to AU$3000 + SAS HBA, a good year or two after I built that backup server. Most of our data set is data in formats that're already efficiently compressed with JPEG, LZO, PNG, deflate/zlib, etc, there's no significant compression gain; tapes can be presumed to be 1.5TB. The weekly hot set is over 1TB and the archives are over 5TB. We can probably pare the weekly down enough to fit it on a single LTO-5 along with the differentials, but we don't have tons of headroom. Library/loader units are AU$5k (sans tape drive) and up. Tapes are AU$80 or so, and shipping renders direct import no cheaper except in vast quantities.
Because of the cost of the LTO-5 drive it's just not worth it when you're only running a few tapes. We'd be better off with a fire-protected power-protected LTO-4 autoloader system. I've found one on end-of-line special at AU$4k down from AU$10k for 19TB of capacity (24 tapes), which is almost attractive. With LTO-4 tapes at about AU$55, that's a total of a bit over AU$5000 for library+tapes, or $263/TB, plus the controlling server. Still not exactly cheap.
Given that 1TB HDDs were down to below $100 ea, I could build a 10TB backup array (11 disks; 8TB usable; RAID6 + 1 hot spare) for about $1100 + enclosure. 4xSATA HBAs are less than $100 ea and putting three in a regular full-ATX or mini-ATX board is no big deal. Linux's software RAID (md) does a great job for this sort of thing, where write-through caching is acceptable because writes are mostly linear.
Given that we need the same kind of fire-resistant enclosures etc whether we're using tape or HDD, HDD wins by a mile.
If we were removing and rotating tapes daily then we might be able to get away without the fire protection. Maybe. The backups tend to run overnight with tapes exchanged in the morning, and that means there's a big window for loss before the tape gets taken away. Then there's the issue of finding someone trustworthy and reliable to exchange the tapes and, more importantly, not lose them.
As far as I can see, tape still loses unless you're doing LOTS of off-site archiving and have a large rotating media library. Even then it's tempting to just use HDDs in hot-swap caddies.
I used to use DDS4 back in the dark old days, and I quickly became all too familiar with their tendency to be write-only media even with regular drive cleaning, tape replacement and occasional tape re-tensioning. I know LTO is better, but when I was building this backup server high-capacity LTO was also eye-bleedingly expensive.
LTO has been forced down dramatically by effective competition from rotating magnetic media. Tapes and drives sure weren't those kind of prices when I built that system! I priced alternatives and disk won by a *lot*. Of course, I'm in Australia, the land of $OMGWTF pricing for IT equipment.
Given our data set size we'd be up for a pair of drives or a changer - because as you know, if human attendance is required for a backup to complete, the backup will fail on the day it's most important that it succeeds. With a pair of drives we'd need a lot more human intervention to swap tapes than we currently require. Physical access to the backup server is a pain, as is to be expected if it's separate enough to be any good, so tape changes would be pretty annoying.
Nonetheless, thanks for the tip. I'm increasingly tempted to move to tape for my long-term archives, because HDDs are bulky, delicate, and not as stable as tape over long idle periods. I should really consider using tape for archival and keeping the HDDs as a staging area before copying to tape, reducing the frequency of tape changes and making sure backups are in more than one place.
While I like the *idea* of cloud hosting making the office portable, it only works for some kinds of work, and has its own downsides.
Australia has only a couple of big international data links, and they've been known to go down when cables get cut by idiot trawlers. This usually causes severe congestion on the other links and I would NOT want to be running my business off those links when this happens. If you're not in.AU/.NZ, this probably isn't an issue for you, and in.AU or.NZ one can always host within the country - if you one find a provider.
The bigger issue is volume of data. I work at a small local newspaper, and even here we're dealing with over a terabyte of data in our "hot" set that we absolutely must have to produce each edition. This is not going to work over Internet links to cloud hosting. Keeping the data set on the other end but simply isn't practical when colour-correcting and touching up images, producing artwork, etc.
Yes, this! Car+train, bike+train, etc are key ways to make public transport more usable and time efficient, but Maps doesn't understand them.
Maps needs to not only understand mixed journeys, but which services you can take bikes on. In Perth, Western Australia, for example you can bring a bike on the train (but not bus), except between 7am-9am and 4:30pm-6:30pm weekdays. There are also secure keycard-controlled bike lockups at stations if you want to just ride to the station. If you make use of those facilities it makes getting around a lot easier.
I usually ride my bike to the local train station, where I chuck it in a keycard-controlled lockup and hop on the train using the same keycard. Maps can't understand that I can get to the train stn in 5 mins not 20 mins, so it makes poor planning choices. This is no big deal when following regular routes or planning well ahead, but it's a real PITA for the ad-hoc "I'm here, get me there" journeys Maps is so great for.
This is a pretty minor limitation in a generally amazing application, though. I'm truly impressed it works as well as it does.
Give me an easily crawled listing of individual files, an rsync-able directory, or an RSS feed. Something where I don't have to constantly re-download data that *hasn't* changed along with changes.
I must second that - I rely heavily on Google Maps in Perth, and in fact it's helped me avoid needing a car for the last five years. I only recently got one to make it easier to get out of the city, go windsurfing, and get around on sundays.
The only issue I've ever had with Google Maps transit in WA has been the odd special-occasion public holiday or special event where Transperth appears to have failed to inform Google of the schedule changes. That can be annoying. On the other hand, Google Maps had perfect data about all the New Years' Eve special and adjusted services, so they're clearly getting it pushed most of the time.
I cannot possibly praise Transperth and Google enough for Google Maps Transit. It's fantastic, and it's a real shame that so few people seem to know about and use it. It was a real lifesaver when I last visited Auckland, too, as I could just use Maps instead of having to fart about with a different city's transit systems and timetables. Fantastic!
Now try being in Australia, where in addition to those downsides we have tightly metered traffic on Internet links, not just for international traffic over the undersea cables but for ALL traffic not to/from our local ISP.
People trying to sell cloud storage in this environment are off their nut.
I have the same issue. I work for a small suburban newspaper, and even our hot data set is over 1TB, plus append-only archival data of more than 4TB.
When I tell these "cloud backup" providers this they do a double-take and then start talking laughably high prices or they just back off and say they can't really handle our archival data set. It's quite pathetic when my 10TB backup storage server in a fire-resistant, water-resistant enclosure in the shed cost under $5k when built - and that was when 10x1TB disks was a lot so the disks cost over $2500 by themselves.
Because I'm in Australia I also have the issue of bandwidth. I'd need a backup provider to peer with my ISP via a local peering point that offers unmetered traffic; with 100GB/month limits considered very big here I couldn't possibly back up over a metered link. Even then, my redundant two ADSL2+ links achive about 6Mbit/750kbit and 4Mbit/500kbit per second each, so I'd probably need to pay to run fibre from the nearest line along the train line (est $50,000) and pay over $1000/month for a fibre service just to talk to the backup storage host.
I'm negotiating to move our backup server to a business down the street and run an 802.11n point-to-point directional link between us instead. We each get to fail over to each others' Internet services if necessary, we exchange backup storage, and neither of us gets to pay through the nose for it. It's not as good as a fast link to a DC somewhere, but it's a hell of a lot more practical.
The other issue with cloud backups arises when you need that 5TB (mine) or 38TB (yours) in a hurry, for disaster recovery. You can't exactly run down the street and grab the server with its disk array then restore over 1Gbit ethernet or direct to locally attached SAS/eSATA/whatever. Nope, you have to download all that data over whatever Internet link you have access to. If that's not the dedicated fast link your premises has (say, if they've burned down) then you are screwed.
I'll keep my primary backups within driving distance, thanks.
For me the one attractive use case for cloud storage is for backups - and it's one that's catered to particularly poorly by current offerings.
For backups, you want (a) fast, unmetered links to the host and (b) moderately reliable, cheap, and not-that-fast storage you can access in a variety of different ways depending on what's most convenient, with or without running your own VPS to mediate between storage and storage clients.
One user will want to rsync to their cloud storage. One will want to remote-mount a file system on it via iSCSI. Another will want to run a Bacula storage daemon on it. Yet another will want to use it as a co-ordinator for a full network backup system. All these use cases should really be supported, and the first two shouldn't need the customer to maintain their own VPS to control the storage.
As things stand, almost everyone wants to sell SAN-based high performance storage that's *expensive* and *fast*, not cheap and slow. Most backup services seem to want you to use their tools or a local appliance to talk to their storage. Half of them act very confused when you mention "Linux" or "UNIX" and ask if that's a new kind of Mac or something. At least in Australia I've found the market miserably unsatisfying so far.
What I'd really like is for ISPs to begin offering, or partnering with others to offer via peering, bulk near-line storage at moderately affordable rates. That way you can talk to it over your business's main ADSL/SHDSL/fibre/whatever link(s) without dealing with quotas, it's fast, there are multiple routes to it, and it's unlikely to go down if an international link has a hiccup.
iiNet's cloud offering looked like it might have potential for this, but it turns out to be just another EC2-wannabe crossed with Linode-done-badly-and-expensively. The storage offerings are miserable and they don't even mention whether traffic between iiNet internet services and their cloud is metered
I'm releasing tools from my work that I developed for our operations.
We don't want to sell the tools - for the kind of money we could get for them in a market full of existing commercial options, it wouldn't be worth the trouble, let alone the sales and support overheads.
We could keep them closed in-house. There's nothing wrong with that and it's a viable option, but it means we give up the chance of sharing maintenance costs with others and benefiting from others' improvements to the tools.
Consequently, we've decided to open them up. This will permit competitors to use them - but most of our local competitors have already licensed expensive commercial equivalents they're committed to, so the only way they're likely to benefit is if we push pricing down across the industry, which isn't likely at this stage given that our tools are significantly less polished and more limited than the existing commercial offerings. It'd also permit new start-ups who wanted to compete with us to use them - but we're the dominant player in a mature and saturated local market with significant community loyalty. Startups have consistently failed despite having vast amounts of cash pumped into them by outfits who want to knock us out of the way and don't mind taking epic short-term losses to do it.
The upside of opening our tools up is that we're hoping to see participation from other companies and non-commercial publications, reducing the cost of ownership of our in-house tools, making them easier to maintain and less dependent on just one person in one company. That should help future-proof them for us if they're successful, and hopefully get us the use of contributed enhancements we wouldn't have developed ourselves.
IMO this is one area where OSS is really key in commercial use: when you need to build tools that help your business but aren't viable as a product.
All this is is a list of blacklisted IMEIs that's shared between most (not all) carriers. The phones are still perfectly functional when used in other countries with compatible UMTS/GSM frequencies, and on carriers that don't use the IMEI blacklist.
Some carriers do subscribe to the IMEI blacklist but take so long to update it that they might as well not. I'm looking at you, Vodafone.
Not only can stolen phones be sold overseas, but it's pretty trivial to rewrite the IMEI on many phones. This is a disincentive to casual theft, but not much more.
I *love* ISPs that block port 25 outbound... by default. It's a great spam control measure for Judy and Joe's unpatched Windows XP SP1 machine connected directly to the Internet via a USB DSL modem. Most ISPs, however, let you turn it off via a control panel offered for your service - if you know you want to and know enough to do so. Those that don't let you turn it off at all because they're trying to force you to pay them to unblock ports, they piss me off.
I relay all my oubound mail via a VPS at a reputable host - in my case, Linode, but many others would do. The VPS has a static IP allocated from space the VPS host has registered as used for hosting static customer services. Reverse DNS is configured to match the hostname it reports on EHLO and the hostname listed in the MX records.
That way I'm freed from all those annoying DSL/cable modem filter rules, and I get a secondary MX as part of the deal.
Out of interest: how do you find them? Scanning tools? manual inspection?
I don't run AV here, but I have AV software installed so I can use it for the odd system scan. I never turn anything up. I never see any evidence of virus activity, either, and I'm pretty used to spotting the signs thanks to experience fixing the home computers of users at work.
OTOH, it probably helps that my Windows installs have tended to last a year at the most before I reimage them for some reason or another (HDD failure/upgrade; OS upgrade; boredom; etc) so I'm rarely running a crufty old install.
Alas, right now I'm moving many services from RHEL to Windows 2008 Server.
Why? Group Policy, and Volume Shadow Copy Service.
I cannot overstate the importance of Group Policy in simplifying the management of a client network. Especially when combined with Windows Software Update Services, it's been wonderful. I've been a Linux guy since forever, but I'm really being swayed against my will toward the Windows server stuff for managing Windows clients.
As for the Volume Shadow Copy Service - it's all well and good to have 10-minutely Bacula incrementals thoughout the day, but nothing beats near-zero-cost snapshots that automatically age out when space is exhausted and that are very space efficient because they're done at the file system level. No, LVM cannot do this, it's block level and thus wastes a lot of space snapshotting changes to "free" space etc. Additionally, snapshots must be mounted, and old snapshots age out rather ungracefully. I've had a server fail to boot because of a broken LVM snapshot multiple times, and it's a major piss-off. It can't touch versioned files in NTFS. Maybe BTRFS will be there in 5 years.
Truly, though, it's Samba's quirks/limitations, and the lack of Group Policy, that's driven me to drop Linux for managing Windows clients. This isn't surprising, as Microsoft doesn't want to make it easy to manage Windows clients with anything other than Windows servers, and while the Samba folks are doing a heroic job there's only so much they can do.
Alas, Nokia kind of missed the boat with Maemo/MeeGo. They let Maemo idle for years on the no-cellular tablets that were interesting, but never really went anywhere. Then they tossed the N900 out the door - just as they decided to massively rework the OS, effectively EOLing the N900's OS before it was released. Unsurprisingly, app developer interest has been... limited. *I* know you can upgrade an N900 to MeeGo (when it's properly ready, hopefully) but Nokia hasn't been too clear on this and it's unlikely app devs will want to target a platform where users have to reflash to a new OS to run their apps.
I love coding with Qt and have wanted it in phones for ages, so I was really excited to see Maemo move over - but the timing, amid a product launch, was horrifying.
MeeGo would've been great if it (instead of Maemo half-way through an API breaking transition to Qt) was released in finished form at about the time the N900 hit market. Now, by the time it sees real-world products, I think Android will be pretty much unstoppable, especially as it's now allowing native apps, the main advantage MeeGo had. I don't rate it's chances.
Personally I like MeeGo a lot more as a concept of how a phone OS works. It's my phone, not the carrier's / handset manufacturer's phone that I happened to pay for. Unfortunately, carriers (especially in the US) don't like that, and given the likely higher prices and limited app coverage of MeeGo, I don't see it going far.
Were I Intel and Nokia, I'd be thinking very hard about offering Dalvik and.apk support for apps without native code, at least for a subset of Android API features. Get some app coverage from the start, but encourage targeting of Qt by offering Qt Jambi from Java and offering better API access via the native interfaces. Be a better Android than Android.
It's pretty similar here in Australia. A distant friend of mine is, in fact, a convicted child sex criminal - because of photos of HIS CHILDREN taken WITH HIS WIFE THERE, in the BATH. A photo developer called the police, who seized his computers and media. They were unable to prosecute based on the original images, but the search turned up a collection of manga/hentai on a machine used by him and friends. Some of which "could be considered" to depict children, especially by a suitably encouraged jury. Bang, you're on the sex offender register and your life is fucked.
W.T.F.
Under Australian law, I could be considered a sex criminal if I accidentally include photos of partly clothed children in a wide-angle photo of a beach. Unsurprisingly, I'm now incredibly paranoid about taking photos - in fact, I flatly refuse to photograph anybody's children even with their permission or by their request. I'll let them do it, but only if I'm somewhere I can download the images, burn a CD to give to them, and destroy any copies I may have.
Sad, isn't it, that it's come to this. And all the fuss is further sexualizing children in people's minds, while doing NOTHING to even slow down the real perverts, who won't notice or care.
External access is - thankfully - locked down fairly tight. Otherwise I wouldn't mention it in any public forum, it'd be a big flashing "hack me, I'm stupid".
Users have IMAP and SMTP (both TLS, both requiring client certificates) and key-only sftp. That's it. There's no password-only auth for external access, it's all based on crypto.
Because users who "leave" the company tend to and doing patches of work for the company every now and then - and often do so remotely - I can't even assume that a user who's quit won't be working from home as a short-term casual six months down the track.
I find it's a remarkable incentive for keeping really good backups.
Personally, I've given up removing employee logins here. I rarely even lock them. We seem to have a revolving door, where employees leave and return all the time. Full timers leave, then return as casuals years later. Casuals leave, and years later re-appear as full time staff. The people responsible don't think this is in any way noteworthy, and don't give any warning when someone who's ostensibly left the company will be re-appearing. They just expect the user's login to still work. In that sort of environment, there's not much you can do exept sigh, sent a "not my problem" written warning to the boss, and forge ahead as best you can.
Dell: Numerous examples. I have one of them, the otherwise excellent XPS M1330 that has a defective nVidia 8400M GPU.
Apple: Numerous examples (including right now - iphone). The various iBook motherboard defects also come to mind.
nVidia: The afore-mentioned 8400M remained in production long after nVidia discovered the defect. They kept the defect secret for as long as possible, then when forced to admit it continued selling the faulty part without any warning for users and refused to talk about any arrangements they might've made with individual OEMs for RMA/warranty.
Acer: Frequently sells shoddy hardware and yells "la la la la" loudly when told about it. I have one of those, too, an Acer laptop with a fairly powerful GPU and a cooling subsystem for a basic one, so if you actually use it the GPU overheats and the machine crashes.
Hell, the list is basically endless. Everyone does it, because the consequences are small compared to the profits. Unless that changes, it'll keep on happening, too.
They require you to pre-allocate space, so you have to guess how much copy-on-write difference will accumulate between the original and snapshot over the lifetime of the snapshot.
If the snapshot runs out of space, it *should* cleanly disable its self. Pity about the file system mounted from it that has no idea its backing block device just vanished. It gets messy, fast, when an LVM snapshot runs out of storage.
LVM snapshots are really inefficient, because they track all block changes, not just user-level file data and metadata. This massively bloats the snapshot, reducing how long it lives until it runs out of backing store and disables its self.
LVM snapshots don't share backing store. If you have three of them, snapshot t+3 has to store all the data snapshot t+2 and snapshot t+1 do, and so on. The differences between the real fs (t) and snapshot t+1 land up being stored three times in three separately allocated backing stores. You waste a HUGE amount of space this way, and it's hard to reliably predict how much you need so your snapshots often vanish out from under you are you're trying to use them.
LVM is useful, but for someone used to the Volume Snapshot Service (VSS) on Windows servers, to ZFS, or to any of the "enterprise-y" file systems often seen deployed with big SANs, it's just going to make them cry.
"In particular, SQL databases are completely unsuitable for this kind of backup (this is why they have their own backup and transaction log handling procedures)"
While snapshots aren't ideal for SQL DBs, any real snapshot is equivalent to a point-in-time copy of the state of the file system. Restoring it and starting the database should seem to the database just like it's recovering after unexpected power failure or a process crash.
Any database that doesn't recover properly after a snapshot restore will also fail to recover properly after powerfail or a sudden hardware reset, because it's not ordering its writes properly.
Of course, proper snapshot implementations (ie not LVM) notify apps that a snapshot is about to happen so they can pause their work and enter a stable, easy to recover from state for the moment it takes to make the snapshot. So it's even easier on the database.
Now, I'll grant that for databases it's usually *better* to do incremental block-level copies, SQL-level dumps, etc using the databases own tools because doing so is usually much more _efficient_ than taking a snapshot then archiving it somewhere. But sometimes you just want the snapshot around as insurance before doing a major config change or upgrade, and for that they're just unbeatable.
While I don't much like Windows servers in general, I have a major case of VSS envy (Volume Snapshot Service, not Visual Source Safe - blech!), because it's worlds ahead of anything seen on any open *nix and has been for nearly ten years. Hell, my one and only Windows server maintains in incremental backup of its self on a remote iSCSI volume, including many point-in-time snapshots, that I can just unplug from the iSCSI storage host and boot if I need to for disaster recovery. It's impressive, and VSS is the core of what makes it possible.
Slashdot Mirror
Plane crash tests - in so far as they are done - are thus more about making borderline situations more survivable. Not flying a perfectly good Airbus into the water at speed in a sustained stall, but ditching an airliner in the Hudson after ingesting geese and losing all power. Bringing one down on an abandoned runway turned speedway in a successful unpowered glide. Crash landing without main and/or nose gear without the aircraft breaking up or tumbling. Getting passengers off alive and quickly when an aircraft catches fire on the ground. Those sorts of situations DO matter; surviving a mid-air collision, break-up or controlled flight into terrain not so much.
Unfortunately they're so expensive to do that we tend to rely on investigation reports from the real thing to inform future design choices and suggest revisions. Important and great, but a lot could be learned especially about making aircraft water-ditchable and field-crash-landable by doing remote control crash tests. I suspect it isn't considered worth the cost.
Crashes give you one important thing - things you never thought to test. This is true of both staged and real crashes. Controlled tests find out how much a plane can take; crashes help you find things to check for in controlled tests and new things to add to design guidelines.
Pity it is so expensive to crash airliners for testing.
OK, I've gone and looked into the current situation with tape. All prices are in AUD from vendors in Australia; 1 AUD = 1.06 USD at the moment, so they're close enough to the the same.
LTO-5 drives are now AU$2500 to AU$3000 + SAS HBA, a good year or two after I built that backup server. Most of our data set is data in formats that're already efficiently compressed with JPEG, LZO, PNG, deflate/zlib, etc, there's no significant compression gain; tapes can be presumed to be 1.5TB. The weekly hot set is over 1TB and the archives are over 5TB. We can probably pare the weekly down enough to fit it on a single LTO-5 along with the differentials, but we don't have tons of headroom. Library/loader units are AU$5k (sans tape drive) and up. Tapes are AU$80 or so, and shipping renders direct import no cheaper except in vast quantities.
Because of the cost of the LTO-5 drive it's just not worth it when you're only running a few tapes. We'd be better off with a fire-protected power-protected LTO-4 autoloader system. I've found one on end-of-line special at AU$4k down from AU$10k for 19TB of capacity (24 tapes), which is almost attractive. With LTO-4 tapes at about AU$55, that's a total of a bit over AU$5000 for library+tapes, or $263/TB, plus the controlling server. Still not exactly cheap.
Given that 1TB HDDs were down to below $100 ea, I could build a 10TB backup array (11 disks; 8TB usable; RAID6 + 1 hot spare) for about $1100 + enclosure. 4xSATA HBAs are less than $100 ea and putting three in a regular full-ATX or mini-ATX board is no big deal. Linux's software RAID (md) does a great job for this sort of thing, where write-through caching is acceptable because writes are mostly linear.
Given that we need the same kind of fire-resistant enclosures etc whether we're using tape or HDD, HDD wins by a mile.
If we were removing and rotating tapes daily then we might be able to get away without the fire protection. Maybe. The backups tend to run overnight with tapes exchanged in the morning, and that means there's a big window for loss before the tape gets taken away. Then there's the issue of finding someone trustworthy and reliable to exchange the tapes and, more importantly, not lose them.
As far as I can see, tape still loses unless you're doing LOTS of off-site archiving and have a large rotating media library. Even then it's tempting to just use HDDs in hot-swap caddies.
I don't think the economics support tape anymore.
I used to use DDS4 back in the dark old days, and I quickly became all too familiar with their tendency to be write-only media even with regular drive cleaning, tape replacement and occasional tape re-tensioning. I know LTO is better, but when I was building this backup server high-capacity LTO was also eye-bleedingly expensive.
LTO has been forced down dramatically by effective competition from rotating magnetic media. Tapes and drives sure weren't those kind of prices when I built that system! I priced alternatives and disk won by a *lot*. Of course, I'm in Australia, the land of $OMGWTF pricing for IT equipment.
Given our data set size we'd be up for a pair of drives or a changer - because as you know, if human attendance is required for a backup to complete, the backup will fail on the day it's most important that it succeeds. With a pair of drives we'd need a lot more human intervention to swap tapes than we currently require. Physical access to the backup server is a pain, as is to be expected if it's separate enough to be any good, so tape changes would be pretty annoying.
Nonetheless, thanks for the tip. I'm increasingly tempted to move to tape for my long-term archives, because HDDs are bulky, delicate, and not as stable as tape over long idle periods. I should really consider using tape for archival and keeping the HDDs as a staging area before copying to tape, reducing the frequency of tape changes and making sure backups are in more than one place.
While I like the *idea* of cloud hosting making the office portable, it only works for some kinds of work, and has its own downsides.
Australia has only a couple of big international data links, and they've been known to go down when cables get cut by idiot trawlers. This usually causes severe congestion on the other links and I would NOT want to be running my business off those links when this happens. If you're not in .AU/.NZ, this probably isn't an issue for you, and in .AU or .NZ one can always host within the country - if you one find a provider.
The bigger issue is volume of data. I work at a small local newspaper, and even here we're dealing with over a terabyte of data in our "hot" set that we absolutely must have to produce each edition. This is not going to work over Internet links to cloud hosting. Keeping the data set on the other end but simply isn't practical when colour-correcting and touching up images, producing artwork, etc.
Yes, this! Car+train, bike+train, etc are key ways to make public transport more usable and time efficient, but Maps doesn't understand them.
Maps needs to not only understand mixed journeys, but which services you can take bikes on. In Perth, Western Australia, for example you can bring a bike on the train (but not bus), except between 7am-9am and 4:30pm-6:30pm weekdays. There are also secure keycard-controlled bike lockups at stations if you want to just ride to the station. If you make use of those facilities it makes getting around a lot easier.
I usually ride my bike to the local train station, where I chuck it in a keycard-controlled lockup and hop on the train using the same keycard. Maps can't understand that I can get to the train stn in 5 mins not 20 mins, so it makes poor planning choices. This is no big deal when following regular routes or planning well ahead, but it's a real PITA for the ad-hoc "I'm here, get me there" journeys Maps is so great for.
This is a pretty minor limitation in a generally amazing application, though. I'm truly impressed it works as well as it does.
Zip file?
Give me an easily crawled listing of individual files, an rsync-able directory, or an RSS feed. Something where I don't have to constantly re-download data that *hasn't* changed along with changes.
I must second that - I rely heavily on Google Maps in Perth, and in fact it's helped me avoid needing a car for the last five years. I only recently got one to make it easier to get out of the city, go windsurfing, and get around on sundays.
The only issue I've ever had with Google Maps transit in WA has been the odd special-occasion public holiday or special event where Transperth appears to have failed to inform Google of the schedule changes. That can be annoying. On the other hand, Google Maps had perfect data about all the New Years' Eve special and adjusted services, so they're clearly getting it pushed most of the time.
I cannot possibly praise Transperth and Google enough for Google Maps Transit. It's fantastic, and it's a real shame that so few people seem to know about and use it. It was a real lifesaver when I last visited Auckland, too, as I could just use Maps instead of having to fart about with a different city's transit systems and timetables. Fantastic!
Now try being in Australia, where in addition to those downsides we have tightly metered traffic on Internet links, not just for international traffic over the undersea cables but for ALL traffic not to/from our local ISP.
People trying to sell cloud storage in this environment are off their nut.
I have the same issue. I work for a small suburban newspaper, and even our hot data set is over 1TB, plus append-only archival data of more than 4TB.
When I tell these "cloud backup" providers this they do a double-take and then start talking laughably high prices or they just back off and say they can't really handle our archival data set. It's quite pathetic when my 10TB backup storage server in a fire-resistant, water-resistant enclosure in the shed cost under $5k when built - and that was when 10x1TB disks was a lot so the disks cost over $2500 by themselves.
Because I'm in Australia I also have the issue of bandwidth. I'd need a backup provider to peer with my ISP via a local peering point that offers unmetered traffic; with 100GB/month limits considered very big here I couldn't possibly back up over a metered link. Even then, my redundant two ADSL2+ links achive about 6Mbit/750kbit and 4Mbit/500kbit per second each, so I'd probably need to pay to run fibre from the nearest line along the train line (est $50,000) and pay over $1000/month for a fibre service just to talk to the backup storage host.
I'm negotiating to move our backup server to a business down the street and run an 802.11n point-to-point directional link between us instead. We each get to fail over to each others' Internet services if necessary, we exchange backup storage, and neither of us gets to pay through the nose for it. It's not as good as a fast link to a DC somewhere, but it's a hell of a lot more practical.
The other issue with cloud backups arises when you need that 5TB (mine) or 38TB (yours) in a hurry, for disaster recovery. You can't exactly run down the street and grab the server with its disk array then restore over 1Gbit ethernet or direct to locally attached SAS/eSATA/whatever. Nope, you have to download all that data over whatever Internet link you have access to. If that's not the dedicated fast link your premises has (say, if they've burned down) then you are screwed.
I'll keep my primary backups within driving distance, thanks.
For me the one attractive use case for cloud storage is for backups - and it's one that's catered to particularly poorly by current offerings.
For backups, you want (a) fast, unmetered links to the host and (b) moderately reliable, cheap, and not-that-fast storage you can access in a variety of different ways depending on what's most convenient, with or without running your own VPS to mediate between storage and storage clients.
One user will want to rsync to their cloud storage. One will want to remote-mount a file system on it via iSCSI. Another will want to run a Bacula storage daemon on it. Yet another will want to use it as a co-ordinator for a full network backup system. All these use cases should really be supported, and the first two shouldn't need the customer to maintain their own VPS to control the storage.
As things stand, almost everyone wants to sell SAN-based high performance storage that's *expensive* and *fast*, not cheap and slow. Most backup services seem to want you to use their tools or a local appliance to talk to their storage. Half of them act very confused when you mention "Linux" or "UNIX" and ask if that's a new kind of Mac or something. At least in Australia I've found the market miserably unsatisfying so far.
What I'd really like is for ISPs to begin offering, or partnering with others to offer via peering, bulk near-line storage at moderately affordable rates. That way you can talk to it over your business's main ADSL/SHDSL/fibre/whatever link(s) without dealing with quotas, it's fast, there are multiple routes to it, and it's unlikely to go down if an international link has a hiccup.
iiNet's cloud offering looked like it might have potential for this, but it turns out to be just another EC2-wannabe crossed with Linode-done-badly-and-expensively. The storage offerings are miserable and they don't even mention whether traffic between iiNet internet services and their cloud is metered
I'm releasing tools from my work that I developed for our operations.
We don't want to sell the tools - for the kind of money we could get for them in a market full of existing commercial options, it wouldn't be worth the trouble, let alone the sales and support overheads.
We could keep them closed in-house. There's nothing wrong with that and it's a viable option, but it means we give up the chance of sharing maintenance costs with others and benefiting from others' improvements to the tools.
Consequently, we've decided to open them up. This will permit competitors to use them - but most of our local competitors have already licensed expensive commercial equivalents they're committed to, so the only way they're likely to benefit is if we push pricing down across the industry, which isn't likely at this stage given that our tools are significantly less polished and more limited than the existing commercial offerings. It'd also permit new start-ups who wanted to compete with us to use them - but we're the dominant player in a mature and saturated local market with significant community loyalty. Startups have consistently failed despite having vast amounts of cash pumped into them by outfits who want to knock us out of the way and don't mind taking epic short-term losses to do it.
The upside of opening our tools up is that we're hoping to see participation from other companies and non-commercial publications, reducing the cost of ownership of our in-house tools, making them easier to maintain and less dependent on just one person in one company. That should help future-proof them for us if they're successful, and hopefully get us the use of contributed enhancements we wouldn't have developed ourselves.
IMO this is one area where OSS is really key in commercial use: when you need to build tools that help your business but aren't viable as a product.
All this is is a list of blacklisted IMEIs that's shared between most (not all) carriers. The phones are still perfectly functional when used in other countries with compatible UMTS/GSM frequencies, and on carriers that don't use the IMEI blacklist.
Some carriers do subscribe to the IMEI blacklist but take so long to update it that they might as well not. I'm looking at you, Vodafone.
Not only can stolen phones be sold overseas, but it's pretty trivial to rewrite the IMEI on many phones. This is a disincentive to casual theft, but not much more.
I *love* ISPs that block port 25 outbound... by default. It's a great spam control measure for Judy and Joe's unpatched Windows XP SP1 machine connected directly to the Internet via a USB DSL modem. Most ISPs, however, let you turn it off via a control panel offered for your service - if you know you want to and know enough to do so. Those that don't let you turn it off at all because they're trying to force you to pay them to unblock ports, they piss me off.
I relay all my oubound mail via a VPS at a reputable host - in my case, Linode, but many others would do. The VPS has a static IP allocated from space the VPS host has registered as used for hosting static customer services. Reverse DNS is configured to match the hostname it reports on EHLO and the hostname listed in the MX records.
That way I'm freed from all those annoying DSL/cable modem filter rules, and I get a secondary MX as part of the deal.
Out of interest: how do you find them? Scanning tools? manual inspection?
I don't run AV here, but I have AV software installed so I can use it for the odd system scan. I never turn anything up. I never see any evidence of virus activity, either, and I'm pretty used to spotting the signs thanks to experience fixing the home computers of users at work.
OTOH, it probably helps that my Windows installs have tended to last a year at the most before I reimage them for some reason or another (HDD failure/upgrade; OS upgrade; boredom; etc) so I'm rarely running a crufty old install.
Alas, right now I'm moving many services from RHEL to Windows 2008 Server.
Why? Group Policy, and Volume Shadow Copy Service.
I cannot overstate the importance of Group Policy in simplifying the management of a client network. Especially when combined with Windows Software Update Services, it's been wonderful. I've been a Linux guy since forever, but I'm really being swayed against my will toward the Windows server stuff for managing Windows clients.
As for the Volume Shadow Copy Service - it's all well and good to have 10-minutely Bacula incrementals thoughout the day, but nothing beats near-zero-cost snapshots that automatically age out when space is exhausted and that are very space efficient because they're done at the file system level. No, LVM cannot do this, it's block level and thus wastes a lot of space snapshotting changes to "free" space etc. Additionally, snapshots must be mounted, and old snapshots age out rather ungracefully. I've had a server fail to boot because of a broken LVM snapshot multiple times, and it's a major piss-off. It can't touch versioned files in NTFS. Maybe BTRFS will be there in 5 years.
Truly, though, it's Samba's quirks/limitations, and the lack of Group Policy, that's driven me to drop Linux for managing Windows clients. This isn't surprising, as Microsoft doesn't want to make it easy to manage Windows clients with anything other than Windows servers, and while the Samba folks are doing a heroic job there's only so much they can do.
Alas, Nokia kind of missed the boat with Maemo/MeeGo. They let Maemo idle for years on the no-cellular tablets that were interesting, but never really went anywhere. Then they tossed the N900 out the door - just as they decided to massively rework the OS, effectively EOLing the N900's OS before it was released. Unsurprisingly, app developer interest has been ... limited. *I* know you can upgrade an N900 to MeeGo (when it's properly ready, hopefully) but Nokia hasn't been too clear on this and it's unlikely app devs will want to target a platform where users have to reflash to a new OS to run their apps.
I love coding with Qt and have wanted it in phones for ages, so I was really excited to see Maemo move over - but the timing, amid a product launch, was horrifying.
MeeGo would've been great if it (instead of Maemo half-way through an API breaking transition to Qt) was released in finished form at about the time the N900 hit market. Now, by the time it sees real-world products, I think Android will be pretty much unstoppable, especially as it's now allowing native apps, the main advantage MeeGo had. I don't rate it's chances.
Personally I like MeeGo a lot more as a concept of how a phone OS works. It's my phone, not the carrier's / handset manufacturer's phone that I happened to pay for. Unfortunately, carriers (especially in the US) don't like that, and given the likely higher prices and limited app coverage of MeeGo, I don't see it going far.
Were I Intel and Nokia, I'd be thinking very hard about offering Dalvik and .apk support for apps without native code, at least for a subset of Android API features. Get some app coverage from the start, but encourage targeting of Qt by offering Qt Jambi from Java and offering better API access via the native interfaces. Be a better Android than Android.
It's pretty similar here in Australia. A distant friend of mine is, in fact, a convicted child sex criminal - because of photos of HIS CHILDREN taken WITH HIS WIFE THERE, in the BATH. A photo developer called the police, who seized his computers and media. They were unable to prosecute based on the original images, but the search turned up a collection of manga/hentai on a machine used by him and friends. Some of which "could be considered" to depict children, especially by a suitably encouraged jury. Bang, you're on the sex offender register and your life is fucked.
W.T.F.
Under Australian law, I could be considered a sex criminal if I accidentally include photos of partly clothed children in a wide-angle photo of a beach. Unsurprisingly, I'm now incredibly paranoid about taking photos - in fact, I flatly refuse to photograph anybody's children even with their permission or by their request. I'll let them do it, but only if I'm somewhere I can download the images, burn a CD to give to them, and destroy any copies I may have.
Sad, isn't it, that it's come to this. And all the fuss is further sexualizing children in people's minds, while doing NOTHING to even slow down the real perverts, who won't notice or care.
External access is - thankfully - locked down fairly tight. Otherwise I wouldn't mention it in any public forum, it'd be a big flashing "hack me, I'm stupid".
Users have IMAP and SMTP (both TLS, both requiring client certificates) and key-only sftp. That's it. There's no password-only auth for external access, it's all based on crypto.
Because users who "leave" the company tend to and doing patches of work for the company every now and then - and often do so remotely - I can't even assume that a user who's quit won't be working from home as a short-term casual six months down the track.
I find it's a remarkable incentive for keeping really good backups.
Personally, I've given up removing employee logins here. I rarely even lock them. We seem to have a revolving door, where employees leave and return all the time. Full timers leave, then return as casuals years later. Casuals leave, and years later re-appear as full time staff. The people responsible don't think this is in any way noteworthy, and don't give any warning when someone who's ostensibly left the company will be re-appearing. They just expect the user's login to still work. In that sort of environment, there's not much you can do exept sigh, sent a "not my problem" written warning to the boss, and forge ahead as best you can.
Then I guess they're all rotten.
Dell: Numerous examples. I have one of them, the otherwise excellent XPS M1330 that has a defective nVidia 8400M GPU.
Apple: Numerous examples (including right now - iphone). The various iBook motherboard defects also come to mind.
nVidia: The afore-mentioned 8400M remained in production long after nVidia discovered the defect. They kept the defect secret for as long as possible, then when forced to admit it continued selling the faulty part without any warning for users and refused to talk about any arrangements they might've made with individual OEMs for RMA/warranty.
Acer: Frequently sells shoddy hardware and yells "la la la la" loudly when told about it. I have one of those, too, an Acer laptop with a fairly powerful GPU and a cooling subsystem for a basic one, so if you actually use it the GPU overheats and the machine crashes.
Hell, the list is basically endless. Everyone does it, because the consequences are small compared to the profits. Unless that changes, it'll keep on happening, too.
LVM snapshots also just aren't all that good.
They require you to pre-allocate space, so you have to guess how much copy-on-write difference will accumulate between the original and snapshot over the lifetime of the snapshot.
If the snapshot runs out of space, it *should* cleanly disable its self. Pity about the file system mounted from it that has no idea its backing block device just vanished. It gets messy, fast, when an LVM snapshot runs out of storage.
LVM snapshots are really inefficient, because they track all block changes, not just user-level file data and metadata. This massively bloats the snapshot, reducing how long it lives until it runs out of backing store and disables its self.
LVM snapshots don't share backing store. If you have three of them, snapshot t+3 has to store all the data snapshot t+2 and snapshot t+1 do, and so on. The differences between the real fs (t) and snapshot t+1 land up being stored three times in three separately allocated backing stores. You waste a HUGE amount of space this way, and it's hard to reliably predict how much you need so your snapshots often vanish out from under you are you're trying to use them.
LVM is useful, but for someone used to the Volume Snapshot Service (VSS) on Windows servers, to ZFS, or to any of the "enterprise-y" file systems often seen deployed with big SANs, it's just going to make them cry.
I think the "shut down the database" bit is what's wrong with dump.
"In particular, SQL databases are completely unsuitable for this kind of backup (this is why they have their own backup and transaction log handling procedures)"
While snapshots aren't ideal for SQL DBs, any real snapshot is equivalent to a point-in-time copy of the state of the file system. Restoring it and starting the database should seem to the database just like it's recovering after unexpected power failure or a process crash.
Any database that doesn't recover properly after a snapshot restore will also fail to recover properly after powerfail or a sudden hardware reset, because it's not ordering its writes properly.
Of course, proper snapshot implementations (ie not LVM) notify apps that a snapshot is about to happen so they can pause their work and enter a stable, easy to recover from state for the moment it takes to make the snapshot. So it's even easier on the database.
Now, I'll grant that for databases it's usually *better* to do incremental block-level copies, SQL-level dumps, etc using the databases own tools because doing so is usually much more _efficient_ than taking a snapshot then archiving it somewhere. But sometimes you just want the snapshot around as insurance before doing a major config change or upgrade, and for that they're just unbeatable.
While I don't much like Windows servers in general, I have a major case of VSS envy (Volume Snapshot Service, not Visual Source Safe - blech!), because it's worlds ahead of anything seen on any open *nix and has been for nearly ten years. Hell, my one and only Windows server maintains in incremental backup of its self on a remote iSCSI volume, including many point-in-time snapshots, that I can just unplug from the iSCSI storage host and boot if I need to for disaster recovery. It's impressive, and VSS is the core of what makes it possible.