This may not scale well to your 300 servers, but I saw it used effectively for 20-30 servers at one company I worked for.
The basic idea is to use a locked box to store the passwords in. The box is secured with a simple padlock, to which every knows the combination. For each machine you want to manage the password for, you have an envelope in the box with that machine's name on it in the box, along with a bunch of empty envelopes and some blank password sheets which I'll describe later.
So, you're setting up a new server, say, "fred". You take a blank password sheet and fresh envelope out of the lockbox. The password sheet is basically a simple table with password, name, date, and reason columns on it. Write "fred" at the top of the password form, and then fill in the first line with the root password you've just chosen, your name, today's date, and the reason for the password change, in this case "new machine setup". Fold up the password sheet, put it in the envelope and write "fred" on the front of the envelope. Seal the envelope, and then write your name and the date across the seal. Put the now-sealed envelope in the lockbox and lock the padlock back in place.
Ok, that was quite a bit of work. So, what's it do for you? Suppose you need to do some administration on fred, and you've either forgotten or never knew its root password. You go to the lockbox, open it up (remember, everybody knows the padlock combination), and find the envelope labelled "fred". Now, look at the back of the envelope. Is it unsealed, or does it look like the signature/date over the seal has been disturbed in any way? If so, you might want to change the root password before you're done.
Assuming the date and signature on the seal looks good, open the envelope and pull out the password form. The current password is the last line. Fill out the next line with the same password, you name, the date, and some reason like "forgot password". Throw away the old envelope, take a new one from the box, refold the password form and seal it in the new envelope. Write "fred" on the front of the envelope, and sign and date the seal on the back. Now you know the password.
Now, what do you do if you actually want to change the root password for some reason? It's pretty much the same as above. Find the envelope for the system, and open it to find out the current password (if you don't already know it). Make a new password table entry, and seal a new envelope as described above, and put it back in the lockbox. Note that you can do this at any time, and your fellow admins can still find out the new password without any help from you.
This approach assumes a relatively low rate of password change, and that administrators have physical access to the lockbox most of the time. It also assumes that you want per-machine root passwords. In addition, it allows admins to memorize passwords for machines they use relatively frequently without having to write down passwords for machines they use rarely.
Slashdot Mirror
This may not scale well to your 300 servers, but I saw it used effectively for 20-30 servers at one company I worked for.
The basic idea is to use a locked box to store the passwords in. The box is secured with a simple padlock, to which every knows the combination. For each machine you want to manage the password for, you have an envelope in the box with that machine's name on it in the box, along with a bunch of empty envelopes and some blank password sheets which I'll describe later.
So, you're setting up a new server, say, "fred". You take a blank password sheet and fresh envelope out of the lockbox. The password sheet is basically a simple table with password, name, date, and reason columns on it. Write "fred" at the top of the password form, and then fill in the first line with the root password you've just chosen, your name, today's date, and the reason for the password change, in this case "new machine setup". Fold up the password sheet, put it in the envelope and write "fred" on the front of the envelope. Seal the envelope, and then write your name and the date across the seal. Put the now-sealed envelope in the lockbox and lock the padlock back in place.
Ok, that was quite a bit of work. So, what's it do for you? Suppose you need to do some administration on fred, and you've either forgotten or never knew its root password. You go to the lockbox, open it up (remember, everybody knows the padlock combination), and find the envelope labelled "fred". Now, look at the back of the envelope. Is it unsealed, or does it look like the signature/date over the seal has been disturbed in any way? If so, you might want to change the root password before you're done.
Assuming the date and signature on the seal looks good, open the envelope and pull out the password form. The current password is the last line. Fill out the next line with the same password, you name, the date, and some reason like "forgot password". Throw away the old envelope, take a new one from the box, refold the password form and seal it in the new envelope. Write "fred" on the front of the envelope, and sign and date the seal on the back. Now you know the password.
Now, what do you do if you actually want to change the root password for some reason? It's pretty much the same as above. Find the envelope for the system, and open it to find out the current password (if you don't already know it). Make a new password table entry, and seal a new envelope as described above, and put it back in the lockbox. Note that you can do this at any time, and your fellow admins can still find out the new password without any help from you.
This approach assumes a relatively low rate of password change, and that administrators have physical access to the lockbox most of the time. It also assumes that you want per-machine root passwords. In addition, it allows admins to memorize passwords for machines they use relatively frequently without having to write down passwords for machines they use rarely.