Lux Interior asks:
"Help! I am the ad-hoc computer guy in a small satellite office of a larger company. We have no CIO, no IT department, and no policies whatsoever as regards data retention or backup. Therefore, a lot of company property exists one place-- on individual hard drives. The office is made of almost entirely of rudimentary users, on WIN98 and 2000 machines, who never, ever, back up any company information. Has anyone out there had experiences in a small-office setting with: changing users' behavior in regards to managing their data; setting up best practices for backing up information properly; and making sure that the most computer-apathetic users comply with what you've put in place?" Sometimes the best way to make users conform to policy is to not give them a choice in the first place. Automated backup systems on each workstation can go a long way in helping this. Which software packages have such functionality (the more unobtrusive, the better)?
"Several weeks ago we lost six years' worth of extremely important data on current and continuing projects that not even a data recovery service could get back. As a consequence, it is now my job to make sure this doesn't happen again. I have an offsite data storage service retained, but now, how do I get people to back up their files to our file server so I can back up our data from one location? (Also, having the data backed up on our file server of course means that most inadvertent deletions can quickly be fixed in-house).
This is all taking place in a Windows environment, with an NT 4.0 file server, and I am far from an experienced Sysadmin.
Fun, Fun, Fun.
Any input from slashdot readers would be great, and save me much dyspepsia, insomnia, and general hassle."
Slashdot Mirror
that'll teach them to backup! BOFH!
I wouldn't call this "easy" but there's plenty of sites that should have instructions on re-usering people to log on to the fileserver as a domain server and resetting their my documents folders to be on the file server... then it's just going around and mopping up. Then you back up the file server and tell people if it's not in 'my documents' then it's gone gone gone and its their fault.
Automating backups on workstations, very bad, very difficult. For example, getting people to not turn them off, or even turn off a power bar (wake on lan doesn't work so good in such a situation.) What you do, if you're on NT, is set your system policies so that my documents, all that stuff, is on the server, in their home share. Tell them to put everything on home share. Tell them that anything that's on their hard drive, and lost, will be their responsibility. Explain why they need a central data repository. MAKE DAMN SURE YOUR BACKUPS WORK, OR YOU'LL LOOK LIKE AN IDIOT. Then, if need be, pick a sacrifical lamb. Back up their harddrive, then engineer a 'hard disk failure.' Make sure something important was there, that wasn't properly placed onto the fileserver. This'll drive the point home.
Vintage computer games and RPG books available. Email me if you're interested.
If the user doesn't want to be bothered backing up their data, make them sign a waiver absolving you of responsibility when (not IF) disaster strikes and s/he loses vital data. At least then, when they're angry and upset and looking for a chump to take the fall for their stupidity, you've got a convenient ass-cover with their autograph on it, and it won't cost you your job. :-)
~Philly
Create a network drive that everyone can use, H:, the home directory. Usually already set up on networks, but whatever. Tell people that any work related stuff that isn't saved to the H: drive will be deleted.
Warn them a week in advance, warn them a day in advance.
Then, in the middle of the night, format everyone's machines and stick fresh OS installs on all of them. If possible, ghost one machine's fresh install and use it everywhere. Then, the only backup you have to worry about is the H: drive.
If anyone ever has a computer problem, just reghost their drive, removing whatever pointless software (screen savers, comet cursor, kazaa, etc) that got installed and caused the problem.
Minimal hassle for you, easy backups, and everyone will fear you.
[o]_O
Build yourself a samba server on your favourite brand of UNIX (I prefer Freebsd, many don't). Attach a tape drive and use Amanda as your backup program. Get them all accounts on this machine and get them used to using their "Z:" drive for everything. Then everyone has a daily backup and you are in control - which helps a lot when dealing with people less technically competent than you.
Policy doesn't work until something really bad happens and someone with real power in the company says "Do it or you're fired".
I've been in this situation with dozens of companies, and policy only takes root when error rears its ugly head.
Sometimes the errors cost headaches, sometimes they cost you a lawsuit.
My $0.02 will always be worth more than your â0.02, so
If enough users are Windows 2000, format the drives as NTFS and lock them down. Don't allow them to write files to their own directories. Obviously, some concessions will have to be made for application use, but they probably won't find their way to the few directories where they could theoritically save files. Give all users mapped network drives, both personal and shared, such as H for their home drive and S as a departmental drive, K as a common drive, etc. Again with 2000, point their My Documents folder to their H drive, making them save stuff to the network without realizing it.
Now, get a good back-up scheme on your file server, which I assume you already have, and you won't have anymore data loss problems. It also removes accountability from you. You manage the computer systems. Tell users that IT is simply not responsible for data lost off their local HDs. If they ignore you and then lose data, shrug your shoulders and point to IS policy. That'll learn 'em and learn 'em fast.
Linus once said "Real men don't use backups. They upload everything to an FTP site and everyone mirrors them." (According to Lars...)
Seriously, SMBtar is a wonderful tool, just require them to share out important directories. You can do the same thing using a simple shell script. That is what I have done in the past. This can then be stored on tape, hdd on a major server, etc.
LedgerSMB: Open source Accounting/ERP
Changing people's philosphy on where to store data goes a long way.
We have a server with a large-enough tape drive to back up users' data. We then encourage people EXTREMELY to save important data to their personal LAN Drive, and eeinforce the idea that the Hard Drive should be considered expendible. No excuses, no tears.
We then back up the data nightly and rotate tapes daily during the work week (M-T-W-TH, and Weekly Friday Tapes, with the Last Friday of the Month going on rotating Monthly Tapes.
It a bit of habit-breaking for people used to saving everything to C-Drive, but a little Pavlovian experience of "Ohmigosh, my file is gone!/Oh wow, you got my file back!" will reinforce people that Hard Drives Are Bad/LAN Drive is Good.
You can even reinforce the idea with encouraging people backing up files on floppies/CD R/W Drives.
Mod Karma -1: I sed bad wurds. If I cep my mouf shut, I wud be at riyses.
So, you have two options: (1) If you have reasonablu fast network connections, take the choice away and install automated workstation-to-server software that runs every night. This won't work for roaming laptop users though. (2) Hold a series of "computer training classes". Say 4 or 5 half-hour classes where you teach e-mail ettiqute, tips and tricks, Internet searching, that sort of thing. Make them mandatory (you can usually finagle this through the HR or Training group of the parent org). At one of the classes, discuss backup, then pull out a form stating "I have attended the backup class and understand the consequences of failing to back up my work. My department and I accept full responsibility for failing to use backup tools provided". Require them to sign and turn it in (again, HR and Training will usually help with this). Send copies to the department heads.
That won't prevent data loss (or even the loss of your job if something goes wrong!), but it will help somewhat and also get at least some people thinking.
sPh
We used to have the same thing in my office, and I was in the same position. We setup a network file server with lots of hard drive space, and forced users to logon to the domain. Then we secretly replaced their hard drive with 4 gig versons, so there wasn't much room for them to be saving on. And then we setup a nightly backup of that entire server. Withing 2 weeks, pretty much everyone had their stuff on the network, and as an added side effect, we came up with a naming convention for saving client data, and all client data is saved in the same place now, making finding info much easier and much more efficient. Every user also has a don't ask, don't tell, personal folder that only they (except me, of course) have access to. For personal stuff, like docs, pictures of family, mp3's, pr0n, you name it, they got it...
Hope this helps
Th
Then become the data gestapo and slowly, kindly, patiently drill it into their head to always save things into "My documents." If they ask you to help them with their computer, and you see files that should be in the "My docs" folder, move them there, after they get used to always loading stuff from there, they will get used to saving there too.
The key is patience and Persistence. Practice your waitress smile! ;)
What are we going to do tonight Brain?
Check out Retrospect by Dantz software. We use it for a mixed network of Macs & PC's. Backup occurs automatically from all workstations at administrator defined times. This way, you don't have to "teach" everone to store to a network drive or anything like that.
You buy a TAPE DRIVE. Do not buy a cd-rw. Buy TAPE. Get DDS-3 dat, or VXA, or AIT, or DLT. Make your life easy, buy an autoloader. Make sure retrospect supports it (they have a list). Buy enough tape capacity to back up all the files on all the hard drives without you having to sit and change tapes for hours on end. Retrospect will automate the entire deal so you just need to be there to pick up the pay check.
Someone will give you grief about the cost of tape drive. Tell them to FUCK OFF. Tape drives are CHEAP compared to how much it'll cost your company to LOSE DATA. Buy LOTS of tapes.
With this, you back up EVERYTHING. The first part of a new backup is a pain, but from then on in Retrospect will just back up the changed files, making life very easy. Use multiple tape sets and rotate so you always keep a couple of good backups around.
I can't stress this enough, back up EVERYTHING. Do not say "I will only back up 'my documents'". People save their files all over the damned place and never know where they are. They delete stupid system files they didn't know what they were for. Once you're over the pain of the first full backup, which can take a couple of days depending on the size of your place it's easy and the incrementals are fast. And you can do DISASTER RECOVERY. As in "my hard drive crashed and I lost everything, please restore my computer to the way it was". You can point, click, blast everything onto a new drive in the machine and the machine runs exactly as it used to.
Do not wait for disaster to happen to try this, after you get a backup under your belt. Go through the restore procedure. Get another hard drive and practise doing complete system restores so that you can do it in your sleep when your CEO calls you at 2am to find his deleted girlie pictures.
If you can do this, your cow-orkers will love you. Women will love you. Men will want to BE you.
Dramatic maybe, but I'm a damned happy user of Retrospect for years and it has saved my ass more times than I can count. There are other products (Backup Exec) but I have not used them, and so I cannot vouch for them. I use Retrospect every day.
SuiteSisterMary:
MAKE DAMN SURE YOUR BACKUPS WORK, OR YOU'LL LOOK LIKE AN IDIOT.
Speaking as a former idiot *koff*, that goes without saying.
Any backup system should be tested and shaken-down to verify that data is recoverable. (Then again, any good autobackup system should have a Verify Mode, and a log of the backup to review the morning after.)
The amount of blood, sweat, toil, tears, and non-comped off hours will be worth it.
Mod Karma -1: I sed bad wurds. If I cep my mouf shut, I wud be at riyses.
I would advise against a waiver. Its likely the users would complain to this guy's boss and get him trouble. Users=lusers and they want to have someone to point there fingers at when things fail. They want there problems solved by someone else and if shit happens THEN ITS THE HELP DESK GUYS FAULT AND IF HE/SHE CAN"T GET IT BACK THEN FIRE THEM! I use to do helpdesk and it was the shortest job I have ever had. The network goes down due to a miss configured router=my fault, user unplugs lan hub underneath desk for a lamp causes her boss to unaccess her mission critical project due in 1 hour=my fault, user's hard drive fails and she never backs it up and co-worker screws up backups= my fault and she calls my bosses boss and gets me fired even though I didn't fuck with the server tapes! This happened 2 years ago and I am still very angry over this because I couldn't fin another IT job before the .bomb .
:-)
Anyway here is a more non confrontational method. Just setup a user policy and have it downloaded automatically when each user logs in. In that policy map the users default drive on the server. Make sure the users name is on each subfolder on the servers main backup folder. This will make a big difference to clueless lusers when they see there name on a folder that ms-word tries to automatically save in. You may want to put a greeting pop up message in their profile when each user logs in and telling them to save there files with the directory with there name on it. THen send an email out to everyone and warn them that hard drives tend to fail and just tell them to save all their work with the folder with their name on it. Sounds simple. right?
The program to do this is called poledit and its on the windows cd. To have the profile downloaded automatically you need to create the profiles and then go to user manager and setup it up to download automatically when each user logs in. This is what I would do. If shit hits the fan you can tell management what you did with the pop-up messages, the email's and the profiles and that it was the users fault. With all of these things combined, the blame factor will move away from you and towards the user. Unless of course the server dies.
But I advise not to have users sign anything. It makes them angry and uncomfortable and they could get you in trouble. Remember that IT is customer service just like any entry level job. The customer is always first and its your job and not there's to make sure the data is backed up.
http://saveie6.com/
First, write a policy for users to follow (not more than a page), get your boss to sign it, and then distribute copies. This tells everyone that the boss is behind this. If your boss does not have line authority over the people in question then get someone who does.
Second, get your boss's approval for a half hour tutorial for all staff on the subject, attendance mandatory for all users including him/herself. Get the boss to start with a brief repeat of the disaster story, then hand over to you (write the boss a script if necessary). Tell people why they need to do it and what it is they need to do, but obviously don't go into techie detail. Also emphasise that unsafe behaviour is letting the team down: its not just your work at risk, its everyones.
Its your responsibility to determine policy, configure machines, tell people what to do, monitor progress, and report to your boss. This can and should include saying that certain users are refusing to following departmental policy. Its then his/her job to take things further, upt to and including disciplinary action if necessary. Its not likely to be necessary: few people are that boneheaded.
Good luck. Culture change is hard, but its one of the most valuable things you can do.
Paul.
You are lost in a twisty maze of little standards, all different.
This story reminds me of my earlier years as a indy IT contractor...
One of my first customers was a NAACP trial lawer. Every day one of his jeloppy win95 machines would take a shit, and every day I would fix it. I would constantly remind him how he needed NT workstation instead of 95.
When it came time for billing we would always go around in the same circle, "Now Wobbert (he had the elmer fudd ebonic accent) Why am I gettin chwarged 5 times for the same fix?"
To which I would respond, "Because you broke it 5 times!"
Then his killer statement would come, "Well then shouldn't you have fixed it right the first time so it wouldn't break again?"
To which I would go into why 95 was a POS and NT4.0 was alot better. Then we would go into costs and I would show him how his long term maintenence cost would drop if he made the switch. It never really registered with him though.
I also made the pitch to him about having a centralized server for his employee's to store data on. He just could not understand that this computer wasn't for the employee's to use nilly willy as they pleased.
I feel for you man, just run while you still have some sanity left.
One idea is to tie this into Novell's iFolder idea. The concept is you install a small client onto someone's computer, then "tag" which directories you want synched up with the users home director on the server (such as My Documents, etc).
Every time a file on either the server side or the workstation side updates, the client makes the same changes (note: changes, not the entire file, so if you change 1 sentance on a 30 MB document, you only change the few bytes of code).
We're going to try this out at my Day job for our Laptop user types, but so far, it's looking cool. Novell has mentioned future support for OS X (which I don't believe, but I'm an eternally hoping idiot.)
52 Weeks, 52 Religions with John Hummel
After many years of painfull experience, I have only one suggestion:
**** NEVER EVER UNDER ANY CIRCUMSTANCES BACKUP WORKSTATIONS ****
Got my point?
Instead, go out and buy a cheap server. You can get a AtlonXP 1800 w/ 512MB of RAM and 100 GB of disk for around 200-400 dollors if you put it together yourself. From there map all drives. If you ever get stuck in a situation where you back up the PC's
A) It will get difficult to wean users off of later.
B) Builds bad saving practice. To comply with document policies, you really must centralize where your documents are.
C) Backup software tends to fail/hide/be to verbose when too many boxes are used.
If you absolutly have to backup workstations, look at network backup products like Veritas or Seagate software (err. they may have sent the product to some other company). Ask a user explicitly for a single directory to backup.
For servers, a image level backup is always a good idea. It tends to be the different between 3-4 hours recovery time and 12-24 hours.
Anyways, that's my advice.
Nice idea, but 4 gig drives suck.
I'd suggest a variation; reformat their drives with a fairly small primary partition. That way you'll get the speed of a modern drive with the same effect.
Then, once people are used to saving their data on the network drive, you can create a secondary partition with an odd drive assignment for "personal use".
It is actually the way a lot of large (and I mean 100,000 desktops large) companies essentially do it. Create one 'Build' and roll that out to users. All data (Lotus Notes data, word docs, etc) is stored on servers.
Then when Level 1 support can't fix the problem by doing a 3 finger salute (Ctrl-Alt-Del) then they simple redeploy the build.
/b
[Please type your sig here.]
Additional clarification: Buy a good tape drive. Do not let cost be your guide; buy the best-rated drive you can find. Again, if they give you grief about the drive's cost, the parent poster's advice still applies.
I used to work at $(MUMBLE_SALTPILE_MUMBLE), whose IT department was staffed by people of diminished capacity. One day, due to a re-org, they physically moved a server containing critical data. Somehow, the move killed the drives. So they went to restore from backups.
During the restore, the drive ate itself and the tape. Backup destroyed.
Now, because the IT department had the aforementioned staff of diminished capacity, the next available backup was a week old (it turns out they were doing daily backups serially onto the same tape, because it was "faster").
So the lessons here are:
Schwab
Editor, A1-AAA AmeriCaptions
My experience is that the less your users have to do with backups, the better - they click through error messages, they don't read EULAs, they will not take the care you will because it is not a concern they have internalized. In general, it needs to be as automatic, hands-free and brainless as possible. REMEMBER, SIMPLE == GOOD, COMPLEX == BAD! However, you also need to realize that the backups procedures are just one part of the larger picture. Simply backing up isn't good enough, you need to create a situation where they don't have to make decisions about backing up. From their perspective, they have to just remember to keep their files in X location and the backups will just happen.
Here's (roughly) what I did:
Create a folder to hold all user files on the server, say "Files". Make sure this folder gets backed up. :)
Share the folder, and create a logon script that every user runs at logon to map this share to a COMMONLY AVAILABLE drive letter, say W:.
On W: Create several top-level folders: Private (create a folder in here for each user accessible only to them) Shared (create folders here that need to be shared by groups) Apps (create folders in here to hold for application files and data used by programs) Software (for program install files) etc...
Make sure security is set up to PREVENT users from saving files where they do not belong (like the root of W:). You may want to create a user group for each folder and use membership in that group to control access to the files. They should have to come to you to create a new folder for them in most cases, that gives you the opportunity to review the request with your superiors to ensure proper Policy & Procedures are being followed.
Teach your users this mantra "PUT ALL FILES ON W:" Put it in your email sig, your memos and on your voicemail! Use it in casual conversation - I'm not kidding, REPETITION!)
Go to each PC and move their files into their W:\Private\username folder.
Delete the moved files from their hard drive.
Reset all apps (word, excel, etc.) to default open/save on the W: drive.
Your goal should be to reprogram them to think of the network drive as the only place there is to save files.
"Lawyers are for sucks."
- Doug McKenzie
Where I work, all Linux users NFS mount their /home directories (via automount and NIS) from a central server. They can log into any of our workstations, and have all of their desktop preferences and files.
:-)
All of the Windows machines are on a domain that's handled by the same central server running Samba. They can log into any Windows machine, and all of their preferences and files are there waiting for them.
Backups are done nightly to a DLT 8000 drive that hangs off of the central server. This has saved people's asses many times when a machine blows up (sometimes literally).
With 120gig drives as cheap as they are, and entry level robotic tape changers on the market for less than $10,000, there's no reason anyone should have to suffer through a loss of critical amounts of data.
I sleep very well at night with this arrangement.
Then, in the middle of the night, format everyone's machines and stick fresh OS installs on all of them. If possible, ghost one machine's fresh install and use it everywhere. Then, the only backup you have to worry about is the H: drive.
You're offloading system administration tasks on the users, and giving them an drop-dead ultimatim. Not cool. No fallback. You'll cause much harm.
Instead try billing it as an "upgrade". That way they'll take any inconvenience as a side-effect of something useful to them, rather than as you deliberately screwing up their data and lives to make your job easier.
Also:
Do it by departments, workgroups, or segments of the cube farm, in stages.
Start with a very small group. You get to work the kinks out with a minimum of trouble if something went wrong, and the group will spread the word to other users on how to ease the transistion. That will let you do larger groups later.
Don't just format their disks. Swap 'em out for fresh ones and keep the old disks handy. Help the users recover any data from the swapped out disks for a few days, check that they've got all they need, maybe back the disks up just in case. THEN format them and swap them IN on the next group of victims.
Make a point of how much extra work you're doing to be SURE they don't lose any important data during the transition (even though you're not doing all THAT much extra). And of course harp on how the main point of the upgrade is to protect their data in the future (which IS true).
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Any regular processing that requires human intervention is setting
yourself up for failure, not setting yourself up for success. People
are people, and we all forget things. Working late, bleary eyed, the
best of us can forget to "copy files to the network server for back."
Suggestions as to have users sign papers to say I know I need to back
up, and if I don't it's my fault, yada yada are bad too. That's not
solving the problem, that CYA. Lame.
Workstation backup isn't that hard. If you totally lose a drive, all
you need to restore it are the app and OS binaries, and the user's
data. The app and OS are on CD's, so those serve as backups for that,
so it's the user's data we need to focus on.
Designate a few folders to back up. E.g. where-ever email is stored,
the desktop, a user folder (if it's in windows, Documents and Settings
is a good one, as a lot of programs default things to save in there.
if it's unix, just make it the user's home directory). You don't need
to backup the entire drive. In fact, that's more than likely a waste,
except in a few cases.
Users can easily understand the should work in a specified folder or
folders underneath that folder. This doesn't require an additional
step (it's still just saving), it's a matter of where they're saving.
Are there hypothetical holes still? Sure. Do they happen in practice
that often (if ever)? Not really. If you're super paranoid (or super
diligent, depending on how you look at it), you can write a process
that looks for modified files outside the targeted back up region. If
it's a file common on a lot of machines, it's probably a standard file
(e.g. a config, preference, etc.), and you can most likely whitelist
those. Others could notify yourself (or the user) via email, and
politely ask that they move it if it is to be backed up.
Lastly, *test* *your* *setup*. This cannot be emphasized enough. You
don't need to delete a user file or anything dumb like that. Just
ask someone to name a random important file, and confirm you can
restore it to a different drive. Or pretend your production server
just crashed and you need to bring it back.
This has two key advantages: 1) obviously confirms your setup works,
2) lets you get comfortable with the restore process in a non-stressed
manner, which things going wrong is okay. You don't want to be trying
to figure it out when you're already stressed out because things have
gone horribly wrong.
Anyway, HTH.
-Bill
SlashSig Karma: Excellent (mostly affected by moderatio
Now, I know that an all-winders shop was part of the spec, but I really haven't seen anything that works as well as BackupPC, including some expensive proprietary packages.
BackupPC is a set of very nice Perl scripts and modules that uses Samba to connect to your Winders machines and back up their data to a 'data pool' on the BackupPC server's hard drive. It can be configured to run the backups at night, and will run the backup during the day if it missed the nightly due to the user shutting their machine off or taking their laptop home. It also uses MD5 hashes to check for duplicate files and will make hard links instead of duplicates in order to save space on the server's drive. You can set it up to access client machines via the hidden shares, i.e. C$; there's no software to load on the clients. User data can also be compressed, or just stored on the server's drive as plain files. Makes restoring a snap, and you don't have to worry about aging tapes or corrupted backup databases.
BackupPC as far as I know only works well when the server is running on Linux, as it depends on samba, tar, Apache, etc. Setting it up is easy for someone with a Linux/Unix background, but it can be a learning process for someone new to Linux and Perl.
Did I mention it was free?
include $sig;
1;
I had a similar problem at the school I work for. We have a central file server running Vertias backup exec. Gave everyone a "home" directory on the server and mapped the drive via login script. I told everyone to put all their documents there....and of course some people didn't do that.
So I forced the issue by installing Fortres on every machine and locked down the user's ability to save files locally....so if they want to save the document, they need to save it on the server. I am also running the central control module to easily push out changes to the workstations.
I thought about using roaming profiles (Windows 2000/NT/9x required), but they are a pain in the ass. Some local apps don't like the roaming profile thing, and the file synchronization on login and logoff was killing our network bandwidth (as well as annoying the users with really long login/logout times.)
Hope this helps.
-ted
One of the big problems with Windows is that there's no standard home directory. Microsoft apps save everything to My_Documents, other apps to wherever they feel like. Even worse, sometimes it's really hard to find where an app has saved its files becuase the directories are cryptically named. Then you have the problem of people moving their data to places that make more sense, and other people moving them to somewhere else entirely.
More than once I've had to rescue a small business who moved their Quickbooks data into My_Documents, then their accountant worked on it and saved it back to the original location. Anyone who's worked with Quickbooks knows what a mess this is- you can't just merge the two files. It's back to square one- sometimes with weeks' worth of data!
If Microsoft and Windows developers would just standardize one one home directory, it would make everyone's lives a lot easier.