SCSI is the protocol, FC or ethernet is the transport. Those chunky big cables you may be thinking of, might be correctly referred to as 'Parallel SCSI', which might be the source of the confusion.
What's that disk load counter actually measuring though? I had a look, but I can't actually tell.
Disks are a bit more complicated that processors or memory in terms of measuring how much of their 'performance' is in use.
Factors to look at, include.... well most of the ones you'll see under 'PhysicalDisk' in Windows perfmon.
I/Os per sec and bytes transferred per sec are of interest, but the one that's _really_ in indicator in terms of performance is disk queue length. A long queue, means that for 'whatever reasons' requests are not being serviced fast enough to keep up with the system. Usually there will be some items queued, when there's a burst load. It's when your queue is high, and remains high that you have a problem indicator
You might also find that you can tweak the queue depth on your HBAs. Check your manufacturer what your HBA settings _should_ be for their particular array. I seem to recall that Windows has a 'design feature' that can lead to the queue depth being set to 0 in some situations on a SAN. Which leads to every IO being confirmed before the next IO will happen, which vastly reduces performance.
I'm afraid I can't recall any more, and a quick google doesn't bring anything to light, but... well, chances are if you're using your 'default' HBA settings, they will be wrong. At the very least most SANs can handle a _much_ larger SCSI queue depth. So yes, check your HBA parameters, and compare them to what your vendor recommends. Confirm you have the latest driver, and check if there's a more specific driver/firmware for your particular array. (Some array vendors 'brand' their own, with the right defaults, settings and that have been tested)
Actually, looking that the responses, there's quite a few people replying who would appear to have 'real world' SAN experience.
Getting someone in to fix it _may_ end up being the right choice, but it does help to check first, where the problem lies - there's no point in getting a 'SAN expert' in if your problem is merely filesystem fragmentation.
The fact you're using a SAN is likely to be fairly irrelevant here. SANs are a way to move data between server and disks. They're not really much more complicated than that.
First question, is what's the symptoms of the problem - how do you know you're 'pegging your disks'? If you're seeing IO load to your HBAs being really high, then yes, you might find that you need to upgrade these. From experience though, HBAs are rarely your limiting factor.
Much more likely is that you're experiencing local disk fragmentation, as you correctly point out. I can't offer specific advise for your array, but in my experience, SANs are 'blind' to filesystems. They work on disks and LUNs. LUNs are the devices a host sees. This can be safely and easily be defragmented, in all the normal ways that you would do normally.
Are you accessing your SAN over fiber channel or iSCSI? IF it's fiber, then again, you _may_ have network contention, but it's unusual in my experience (especially on a 17 servre SAN). If it's network, then you have contention to worry about. Is it possible that your 'gimme profile' requests across your network are also contending with your iSCSI traffic?
You may find that your SAN has 'performance tools' built in. That's worth a look, to see how busy your spindles are. Because of the nature of a SAN, you may find that the LUNs are being shared on the same physical disks. This can be a real problem if you've done something scary like using windows dynamic disk to grow your filesystem - Imagine having two LUNS striped, when in acutality on the back end, they're on two different 'bits' of a RAID 5 set. This is bad, and is worth having a look at.
One place where SANs do sometimes have issues is in page files. Which is possibly a problem if you're SAN booting. SANs have latency, and windows doesn't like high latency on page files. If you really push it, it'll start bluescreening.
This is fixed by local disks for OS, or just moving swap file to local disk.
HBA expansion _might_ improve performance, assuming this is your bottleneck. However you'll need to ensure you are multipathing your HBAs. (Think of them like network cards, and you won't go far wrong - you need to 'cheat' a bit in order to share network bandwidth on multiple cards). But like I say, you probably want to check this is actually a problem. If they're not very old, then it's unlikely, although it might be worth checking which internal bus the HBAs are on. (Resilience and contention).
It's possible your SAN is fragmented, but it's unlikely this is your problem - SANs don't have the same problem with adding and deleting files (LUNs) so all your backend storage will be in contiguous lumps anyway.
And I apologise if I use terminology that you're not familiar with. Each SAN vendor seems to have their own nomenclature when it comes to the 'bits', but they all work in roughly the same way. You have disks, which are... well disks. RAID groups, which are disks bundled together, with a RAID 1, RAID 1+0, RAID 5 (with variable numbers of parity ratio) and very occasionally RAID 0. You have LUNs. Logical Units. These are... well, chunks of your bundles of disks. The first 100Mb of a 5 disk RAID 5 group, might be a LUN. The LUN is what the host 'sees', as a single atomic volume. Most disk groups can have multiple LUNs on them, which is why you do need to watch out for how volume management is operating. I have seen a case where a Windows 2000 server added a second LUN, and used dynamic disk to stripe. Not realising that on the back end, both those LUNs were on the same RAID 5 (4+1). Which cause the disks to seek back and forth continually, and really hurt performance.
Oh, and this is also probably a good excuse to be booking SAN training. IMO SANs are fun and interesting, not to mention in demand and well paid:)
Really? I must admit, I'm surprised. SAN is just a way to attach disks over scsi. I've yet to see an array that allows you to defragment a volume (a host device). EMC Clariions will let you defrag raid groups, but that doesn't do anyhting more tham move the LUNs around on they physical devices.
Or are you perhaps thinking of NAS (Network attached storage) devices?
The record industries haven't realised that their model is obselete. When the only option for getting your music is buying 15 tracks on a CD, for whatever the going rate is, then this model works fine. (For CD, substitute LP, Tape, or whatever).
You pay your money for an item, and the right to use is kinda handwaved away.
The problem is, that in this digital age, it's entirely possible to replicate 'content' for negligable cost. If I have a CD, a rip and MP3 takes relatively short amounts of time, and then copying it to friends is similarly, pretty trivial.
And it's inevitable. I mean, when even when it took some effort to re-burn a CD it happened, but now it's a trivial matter.
It's not that people object to paying for music exactly - there will always be people who prefer free, but they were also the ones who were copying CDs, recording off radio etc. before.
No, the real problem is, as you say, people don't want to be ripped off for chaff. Chances are, they're buying that album for a couple of tracks, and might find a couple more they really like on it. When the cost of distribution in electronic format is demonstrably negligable (at least in comparison) then it's a rip off.
The thing is, Apple and iTunes have demonstrated that the model can and does work. You just have to make it cheap and convenient. You no longer have a finite quantity of CDs to sell, you have an infinite amount of 'bytes' - ok, so it's not quite infinite, but the unit cost goes through the floor, where the initial cost (of actually producing the track) remains about the same.
Simple business logic, is to drop your price, sell more units, and absorb your initial costs in more sales. Sell to the consumer not based on 'we have it, and no one else can give it to you' but instead do so on the basis of 'effective and simple' music distribution. I've bought a lot of stuff on amazon based on their 'recommendations' engine. Give me a 'music site' that will let me do this with _any_ music, download it, recommend things I might also like, and make it feel like a good deal, then I will buy more. As will a whole long list of other people, who are latching on to the MP3 player bandwagon.
All these MAFIAA lawsuits are just shoring up the sandcastles whilst the tide's coming in.
Working from home, technically, no problems at all.
Practically though, it requires a notable 'culture shift' which may be hard to accomplish.
You see, an office area, you have several people in the same area. They're talking, and interacting. And if you don't know something, then you ask, and it's ok within seconds. Simple things like 'who should I send this request to', or 'how do I get more tapes ordered for the backup servre'.
This doesn't happen any more if your employees are isolated. Now this isn't always a disaster, but there's some areas of business where I'd be reluctant to let them 'work from home'. The ones where 'deliverables' are hard to measure.
At the end of the day, if you have a programmer, programming, then as long as the necessary work gets done, who cares what hours he does?
But you'll still have the limitation in the social dynamics, where you can just bounce ideas off other people.
IF you're set on going this route though, I'd suggest the following:
Instant messaging is vital. It's a stand in for the "Hey, bert, where do I send my expenses" in the office, and as such is invaluable.
Email, well, yes, definitely.
Proper 'collaboration'. Documents, code, whatever, you don't want a 5way email chain going on.
Voice comms. Doesn't matter if it's phone or voip, but it does need to exist. Actually, I'd argue that something maybe like ventrillo or teamspeak is a fairly reasonable option - gives you the ability to chat and 'bond' with your co-workers.
Find a mail account that bounces (fails) your mails when your inbox is full. Ideally 2.
Then:
dd/dev/sda | split -b 1048576 | mail you@account1,you@account2
Add in a bit of procmail to 'forward' any incoming mails too.
Send off your disk dd twice for resilience, and rely on the latency of rather a lot of 1Mb emails as your storage.
I know quite a few few mail systems that are sufficiently slow, that you'll get a day of latency before your mails start to return, especially if you do something horrific like mailbomb a slow link:)
First, get a server. You can just go buy one in a disaster, but many companies have 'some arrangement'. This might be an agreement to supply, or it might be a complete replicated environment. There's a tradeoff of price vs. speed, but... the essence is the same - you have a deal, which means you get new kit in a disaster.
Second, you're doing a bare metal recovery. First one to get up and running, is of course, you backup server. Typically, this is done by - building the OS. Loading the backup software. Importing the tapes. Most backup software has a 'disaster recovery file' which is pretty short, but... well basically points you at the location of the 'last good index'. So you get that file. Your DR plan should include some way to get it back. Usually that's done by emailing it to a separate environment, but... well printing out a hardcopy and sending them 'away' with your tapes is also on the list.
Got your database, import it into your backup server, then you get to start your recoveries
If you're not doing anything more clever, then you're probably looking at build OS, install backup software, then recover the information from the backup e.g. system state, contents of databases and installed applications.
This is all quite intensive, and virtually guaranteed to not work seamlessly. Which is why companys run DR sites.
Automatic verify only goes so far, and is no subsitute for occasionally picking on a random server, and getting _everyone_ familiar with how to resurrect it an in emergency.
How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?
Yeah, it's far too hard to read the HEADLINE and/or the SECOND SENTENCE of the article, which both explain EXACTLY where the figure comes from...
I'm sure it was much faster to post a comment to/. and check every few hours until someone to posts an answer.
Software works to a point. Pro data recovery firms are a little more expensive, but absolutely terrifying in terms of what they can recover data from.
There's a reason that classified stuff on disks, there's all manner of hoops to jump through before the disk can even be removed from the 'secure' areas.
I agree. There's many places where disks are way better than tapes. At the moment, we backup to disk first, and destage onto tape for our retention period. Works nice, and gives us the best of both worlds.
Although to be fair, we've not had much need for tape recovery for quite some time - filesystem snapshots are really really good for 'user has been a muppet and deleted his excel file'.
Get the person with the purse strings to go through the 'cost of downtime' calculation.
Lead them throught it, point out all the lovely parts like contractual obligations (engineering companies tend to need to keep designs for long long periods of time) or 'regulations' (Sarbanes-Oxley has a lot to answer for).
Add in the cost of x many people not working for a week.
Include the 'well, can our business still function if we lose our customer database'.
And if that really doesn't work, then clearly your last resort is artificially induced panic, where you raise the possibility of 'something important' being gone, and unrecoverable. Payroll records are a good example, as that's a personal terrror as well as a 'problem for the company'.
In a backup system where you're taking full backups once a week, even with no data churn at all, you end up backing 1/7th of your estate every night. Starts getting a real challenge at about the 15Tb mark, and becomes a whole new adventure in pain when you're talking 100-200Tb.
And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.
But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.
Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.
Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and... well, no one's going to spend that kind of sum of money (millions) trivially.
Re:Everybody IT needs these skills, not just bosse
on
IT Manager's Handbook
·
· Score: 1
"Buy in" happens at all the levels. Almost anyone will react negatively if they get told 'here is the new way. You do it this way now.' That goes for users, developers, managers or... well almost anyone really.
Now, if you'd got them involved in the actual process of speccing and choosing the new toys, you'd probably have a very different reaction, to the same end result. Or you'd have a different, and better 'end result' that _might_ mean you don't have to spend as much money on bells and whistles in the first place. Most people aren't completely stupid, and appreciate the virtues of budget constraints and 'cost effectivity'. OK, they're biasd when it comes to 'their stuff' and everyone things they derserve the best stuff in the company, but there's only a relative minority who live in la-la land.
My driving instructor, in the UK, called it 'gas'.
He apologised for doing so, but pointed out that the reason he did was because when giving a newbie driver instructions 'more gas' was much simpler/more obvious than any of the alternatives.
You already know the answer to this question, otherwise you wouldn't have asked it.
There is _NO_ amount of compensation that makes it worth doing a job you don't like doing.
Find something you like and want to do first and foremost. If you do like doing it, then it's pretty likely that you'll end up progressing rapidly and easily - as an enthusiast, you'll learn and grow quickly into the scope of role you're looking for.
Yes, it's true that some areas aren't as well paid as others, but at the end of the day, you're working for 'quality of life'. The trade off of doing a job where you bounce out of bed in the morning, looking forward to going into work is impossible to truly appreciate until you've done it.
Absolutely correct. The mistake a hardcore techy is making, by keeping things smooth, is that the 'other guys' don't really understand/appreciate what's involved. They wouldn't sacrifice the weekend with their kids, or the night with the SO, so... well clearly it can't have been all that big a deal for you.
Also, when a system's 'still running' then they don't care when you say you want more money. If you're running flat out holding together a stressed system, screaming for a network upgrade, you won't get it.
Let it drop though, and crash and burn _horrifically_, and then when asked, refer them to your previous 18 emails, and report on the problems.
I can virtually guarantee the 'funding' you've been screaming for for the last 5 years will materialise.
And yes, this has happened to me. Each year, budget a network upgrade, because the switches were flakey and horrible. Keep on 'nursing' them, replacing them, repairing them, updating them, because it's 'short term'. And each year, seeing the 'upgrade budget' vanish again, because 'it's not urgent'.
And then one year, it finally did break. Actually, it was more the case that the one guy who was putting in the effort to keep things going, went on holiday for 2 weeks, and one of the other guys realised this was the only way it would get fixed. It broke, and 'limped' for a week and a half until he came back, because the _rest_ of the department wouldn't let anyone call him.
The network upgrade got approved, and 3 months later, was in place.
Unprofessional to let things break? Yes, probably. I don't like the idea of letting something 'fall over' because no one can be bothered to actually replace it. It smells a bit too much like sabotage to me. But I can't deny that it worked. We weren't lying when we were pointing out, each year, that our network hardware was overdue for replacement, upgrade and recabling, and a lot of our 'intermittent issues' that we were working on fighting fires with, were simply due to that. But it took an 'incident' to get someone important enough to approve the paperwork, to actually do so.
The real irony is if you do 'work like a hero' you will not get any thanks or recognition for it. It'll just become 'the standard'. There's no point working twice as hard as your cow-orkers because... well, that'll just be assumed as 'de-facto'.
But once in a while, if you make every effort to work miracles, for a day or two maybe a week, and do a 'long hard slog', and then blow your own trumpet, you'll do very nicely. Although the guy who's been working 75 hours a week for the last 6 years will hate you for it.
There's more truth in that than you might think. Lets face it, the _last_ people who should be made 'leaders' are the ones who actually want it. That goes for politicians, Managers and a whole host of other things - the people who want the 'power' are ABSOLUTELY not the people who should have it - your 'average techy' wants to do tech. And sadly, as a result, tends to neglect management skills, and also has a complete lack of interest in doing it.
So who ends up managing? The people who in an ideal world, wouldn't be.
Actually, there's another reason to disallow web mail.
Compliance.
Ugly word, but one that means that a _lot_ of regulated industries need to be keeping records of email and the like. Which makes messageing and external emails a problem.
A few T2 BPOs are hardly 'totally imballancing'. The ones that got spawned, yeah, they were useful, but... well, I know how much revenue a refinery outpost makes in a good location. I know how much alliances tend to charge for tenancy. The profit from a couple of 'not ultra uber' T2 BPOs is a drop in the ocean. You can make 3-4 billion a week from an outpost, no problems. BoB and other large alliances charge 'around' 300mil/week/corp. That's why BoB are doing phenomenally well - they're _actually_ doing business, rather than (just) chest beating and whining about 'zomg cheating'.
Slashdot Mirror
SCSI is the protocol, FC or ethernet is the transport. Those chunky big cables you may be thinking of, might be correctly referred to as 'Parallel SCSI', which might be the source of the confusion.
Disks are a bit more complicated that processors or memory in terms of measuring how much of their 'performance' is in use.
Factors to look at, include.... well most of the ones you'll see under 'PhysicalDisk' in Windows perfmon.
I/Os per sec and bytes transferred per sec are of interest, but the one that's _really_ in indicator in terms of performance is disk queue length. A long queue, means that for 'whatever reasons' requests are not being serviced fast enough to keep up with the system. Usually there will be some items queued, when there's a burst load. It's when your queue is high, and remains high that you have a problem indicator
You might also find that you can tweak the queue depth on your HBAs. Check your manufacturer what your HBA settings _should_ be for their particular array. I seem to recall that Windows has a 'design feature' that can lead to the queue depth being set to 0 in some situations on a SAN. Which leads to every IO being confirmed before the next IO will happen, which vastly reduces performance.
I'm afraid I can't recall any more, and a quick google doesn't bring anything to light, but ... well, chances are if you're using your 'default' HBA settings, they will be wrong. At the very least most SANs can handle a _much_ larger SCSI queue depth. So yes, check your HBA parameters, and compare them to what your vendor recommends. Confirm you have the latest driver, and check if there's a more specific driver/firmware for your particular array. (Some array vendors 'brand' their own, with the right defaults, settings and that have been tested)
Getting someone in to fix it _may_ end up being the right choice, but it does help to check first, where the problem lies - there's no point in getting a 'SAN expert' in if your problem is merely filesystem fragmentation.
First question, is what's the symptoms of the problem - how do you know you're 'pegging your disks'? If you're seeing IO load to your HBAs being really high, then yes, you might find that you need to upgrade these. From experience though, HBAs are rarely your limiting factor.
Much more likely is that you're experiencing local disk fragmentation, as you correctly point out. I can't offer specific advise for your array, but in my experience, SANs are 'blind' to filesystems. They work on disks and LUNs. LUNs are the devices a host sees. This can be safely and easily be defragmented, in all the normal ways that you would do normally.
Are you accessing your SAN over fiber channel or iSCSI? IF it's fiber, then again, you _may_ have network contention, but it's unusual in my experience (especially on a 17 servre SAN). If it's network, then you have contention to worry about. Is it possible that your 'gimme profile' requests across your network are also contending with your iSCSI traffic?
You may find that your SAN has 'performance tools' built in. That's worth a look, to see how busy your spindles are. Because of the nature of a SAN, you may find that the LUNs are being shared on the same physical disks. This can be a real problem if you've done something scary like using windows dynamic disk to grow your filesystem - Imagine having two LUNS striped, when in acutality on the back end, they're on two different 'bits' of a RAID 5 set. This is bad, and is worth having a look at.
One place where SANs do sometimes have issues is in page files. Which is possibly a problem if you're SAN booting. SANs have latency, and windows doesn't like high latency on page files. If you really push it, it'll start bluescreening.
This is fixed by local disks for OS, or just moving swap file to local disk.
HBA expansion _might_ improve performance, assuming this is your bottleneck. However you'll need to ensure you are multipathing your HBAs. (Think of them like network cards, and you won't go far wrong - you need to 'cheat' a bit in order to share network bandwidth on multiple cards). But like I say, you probably want to check this is actually a problem. If they're not very old, then it's unlikely, although it might be worth checking which internal bus the HBAs are on. (Resilience and contention).
It's possible your SAN is fragmented, but it's unlikely this is your problem - SANs don't have the same problem with adding and deleting files (LUNs) so all your backend storage will be in contiguous lumps anyway.
And I apologise if I use terminology that you're not familiar with. Each SAN vendor seems to have their own nomenclature when it comes to the 'bits', but they all work in roughly the same way. You have disks, which are ... well disks. RAID groups, which are disks bundled together, with a RAID 1, RAID 1+0, RAID 5 (with variable numbers of parity ratio) and very occasionally RAID 0. You have LUNs. Logical Units. These are ... well, chunks of your bundles of disks. The first 100Mb of a 5 disk RAID 5 group, might be a LUN. The LUN is what the host 'sees', as a single atomic volume. Most disk groups can have multiple LUNs on them, which is why you do need to watch out for how volume management is operating. I have seen a case where a Windows 2000 server added a second LUN, and used dynamic disk to stripe. Not realising that on the back end, both those LUNs were on the same RAID 5 (4+1). Which cause the disks to seek back and forth continually, and really hurt performance.
Oh, and this is also probably a good excuse to be booking SAN training. IMO SANs are fun and interesting, not to mention in demand and well paid :)
Or are you perhaps thinking of NAS (Network attached storage) devices?
Latch on to that model (and some bands have) and yes, you get less profit per track, but you sell a hell of a lot more than you would.
You pay your money for an item, and the right to use is kinda handwaved away.
The problem is, that in this digital age, it's entirely possible to replicate 'content' for negligable cost. If I have a CD, a rip and MP3 takes relatively short amounts of time, and then copying it to friends is similarly, pretty trivial.
And it's inevitable. I mean, when even when it took some effort to re-burn a CD it happened, but now it's a trivial matter.
It's not that people object to paying for music exactly - there will always be people who prefer free, but they were also the ones who were copying CDs, recording off radio etc. before.
No, the real problem is, as you say, people don't want to be ripped off for chaff. Chances are, they're buying that album for a couple of tracks, and might find a couple more they really like on it. When the cost of distribution in electronic format is demonstrably negligable (at least in comparison) then it's a rip off.
The thing is, Apple and iTunes have demonstrated that the model can and does work. You just have to make it cheap and convenient. You no longer have a finite quantity of CDs to sell, you have an infinite amount of 'bytes' - ok, so it's not quite infinite, but the unit cost goes through the floor, where the initial cost (of actually producing the track) remains about the same.
Simple business logic, is to drop your price, sell more units, and absorb your initial costs in more sales. Sell to the consumer not based on 'we have it, and no one else can give it to you' but instead do so on the basis of 'effective and simple' music distribution. I've bought a lot of stuff on amazon based on their 'recommendations' engine. Give me a 'music site' that will let me do this with _any_ music, download it, recommend things I might also like, and make it feel like a good deal, then I will buy more. As will a whole long list of other people, who are latching on to the MP3 player bandwagon.
All these MAFIAA lawsuits are just shoring up the sandcastles whilst the tide's coming in.
Practically though, it requires a notable 'culture shift' which may be hard to accomplish.
You see, an office area, you have several people in the same area. They're talking, and interacting. And if you don't know something, then you ask, and it's ok within seconds. Simple things like 'who should I send this request to', or 'how do I get more tapes ordered for the backup servre'.
This doesn't happen any more if your employees are isolated. Now this isn't always a disaster, but there's some areas of business where I'd be reluctant to let them 'work from home'. The ones where 'deliverables' are hard to measure.
At the end of the day, if you have a programmer, programming, then as long as the necessary work gets done, who cares what hours he does?
But you'll still have the limitation in the social dynamics, where you can just bounce ideas off other people.
IF you're set on going this route though, I'd suggest the following: Instant messaging is vital. It's a stand in for the "Hey, bert, where do I send my expenses" in the office, and as such is invaluable. Email, well, yes, definitely. Proper 'collaboration'. Documents, code, whatever, you don't want a 5way email chain going on. Voice comms. Doesn't matter if it's phone or voip, but it does need to exist. Actually, I'd argue that something maybe like ventrillo or teamspeak is a fairly reasonable option - gives you the ability to chat and 'bond' with your co-workers.
Find a mail account that bounces (fails) your mails when your inbox is full. Ideally 2. Then: dd /dev/sda | split -b 1048576 | mail you@account1,you@account2
Add in a bit of procmail to 'forward' any incoming mails too.
Send off your disk dd twice for resilience, and rely on the latency of rather a lot of 1Mb emails as your storage.
I know quite a few few mail systems that are sufficiently slow, that you'll get a day of latency before your mails start to return, especially if you do something horrific like mailbomb a slow link :)
Second, you're doing a bare metal recovery. First one to get up and running, is of course, you backup server. Typically, this is done by - building the OS. Loading the backup software. Importing the tapes. Most backup software has a 'disaster recovery file' which is pretty short, but ... well basically points you at the location of the 'last good index'. So you get that file. Your DR plan should include some way to get it back. Usually that's done by emailing it to a separate environment, but ... well printing out a hardcopy and sending them 'away' with your tapes is also on the list.
Got your database, import it into your backup server, then you get to start your recoveries
If you're not doing anything more clever, then you're probably looking at build OS, install backup software, then recover the information from the backup e.g. system state, contents of databases and installed applications.
This is all quite intensive, and virtually guaranteed to not work seamlessly. Which is why companys run DR sites.
The major one is the operator.
Automatic verify only goes so far, and is no subsitute for occasionally picking on a random server, and getting _everyone_ familiar with how to resurrect it an in emergency.
There's a reason that classified stuff on disks, there's all manner of hoops to jump through before the disk can even be removed from the 'secure' areas.
Although to be fair, we've not had much need for tape recovery for quite some time - filesystem snapshots are really really good for 'user has been a muppet and deleted his excel file'.
Get the person with the purse strings to go through the 'cost of downtime' calculation.
Lead them throught it, point out all the lovely parts like contractual obligations (engineering companies tend to need to keep designs for long long periods of time) or 'regulations' (Sarbanes-Oxley has a lot to answer for).
Add in the cost of x many people not working for a week.
Include the 'well, can our business still function if we lose our customer database'.
And if that really doesn't work, then clearly your last resort is artificially induced panic, where you raise the possibility of 'something important' being gone, and unrecoverable. Payroll records are a good example, as that's a personal terrror as well as a 'problem for the company'.
And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.
But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.
Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.
Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.
Now, if you'd got them involved in the actual process of speccing and choosing the new toys, you'd probably have a very different reaction, to the same end result. Or you'd have a different, and better 'end result' that _might_ mean you don't have to spend as much money on bells and whistles in the first place. Most people aren't completely stupid, and appreciate the virtues of budget constraints and 'cost effectivity'. OK, they're biasd when it comes to 'their stuff' and everyone things they derserve the best stuff in the company, but there's only a relative minority who live in la-la land.
He apologised for doing so, but pointed out that the reason he did was because when giving a newbie driver instructions 'more gas' was much simpler/more obvious than any of the alternatives.
But he still didn't like the idea.
There is _NO_ amount of compensation that makes it worth doing a job you don't like doing.
Find something you like and want to do first and foremost. If you do like doing it, then it's pretty likely that you'll end up progressing rapidly and easily - as an enthusiast, you'll learn and grow quickly into the scope of role you're looking for.
Yes, it's true that some areas aren't as well paid as others, but at the end of the day, you're working for 'quality of life'. The trade off of doing a job where you bounce out of bed in the morning, looking forward to going into work is impossible to truly appreciate until you've done it.
Also, when a system's 'still running' then they don't care when you say you want more money. If you're running flat out holding together a stressed system, screaming for a network upgrade, you won't get it.
Let it drop though, and crash and burn _horrifically_, and then when asked, refer them to your previous 18 emails, and report on the problems.
I can virtually guarantee the 'funding' you've been screaming for for the last 5 years will materialise.
And yes, this has happened to me. Each year, budget a network upgrade, because the switches were flakey and horrible. Keep on 'nursing' them, replacing them, repairing them, updating them, because it's 'short term'. And each year, seeing the 'upgrade budget' vanish again, because 'it's not urgent'.
And then one year, it finally did break. Actually, it was more the case that the one guy who was putting in the effort to keep things going, went on holiday for 2 weeks, and one of the other guys realised this was the only way it would get fixed. It broke, and 'limped' for a week and a half until he came back, because the _rest_ of the department wouldn't let anyone call him.
The network upgrade got approved, and 3 months later, was in place.
Unprofessional to let things break? Yes, probably. I don't like the idea of letting something 'fall over' because no one can be bothered to actually replace it. It smells a bit too much like sabotage to me. But I can't deny that it worked. We weren't lying when we were pointing out, each year, that our network hardware was overdue for replacement, upgrade and recabling, and a lot of our 'intermittent issues' that we were working on fighting fires with, were simply due to that. But it took an 'incident' to get someone important enough to approve the paperwork, to actually do so.
But once in a while, if you make every effort to work miracles, for a day or two maybe a week, and do a 'long hard slog', and then blow your own trumpet, you'll do very nicely. Although the guy who's been working 75 hours a week for the last 6 years will hate you for it.
So who ends up managing? The people who in an ideal world, wouldn't be.
Compliance.
Ugly word, but one that means that a _lot_ of regulated industries need to be keeping records of email and the like. Which makes messageing and external emails a problem.
A few T2 BPOs are hardly 'totally imballancing'. The ones that got spawned, yeah, they were useful, but ... well, I know how much revenue a refinery outpost makes in a good location. I know how much alliances tend to charge for tenancy. The profit from a couple of 'not ultra uber' T2 BPOs is a drop in the ocean. You can make 3-4 billion a week from an outpost, no problems. BoB and other large alliances charge 'around' 300mil/week/corp. That's why BoB are doing phenomenally well - they're _actually_ doing business, rather than (just) chest beating and whining about 'zomg cheating'.