Recovering From apt-get Failures?

Ross Vandegrift asks: "Once upon a time long ago, apt-get totally trashed the company webserver while we were trying to upgrade it. Since then I have been very suspicious of apt-get. Recently, a friend from school talked me into trying it again. I slowly eeked back into it and was keeping our machines up to date, and automatically applying security patches. It was cool, I was starting to trust it again. Well, today that changes..." I'm sure everyone who has ever used RPM or apt can understand the frustration one goes thru when running into this problem. For those who have, what did you do to get your system functioning again?

"I ran 'apt-get update; apt-get upgrade' last night to upgrade one of our machines. Nothing seemed out of the ordinary, I recieved no error messages indicating something terrible happened. But now when I try to use 'su', it returns a libpam error that I've been unable to find reference to, either on the web or in the Debian mailing list archive.

So this poses the question: what do I do if apt-get fails and screws up the system? I've tried reinstalling/reconfiguring the affected packages to no end. If we used my distro of choice (Slackware), I'd have an intimate understanding of my system and would know right where to look when I get an error. But with apt, most of the packages on the machine are black-boxes; I don't know much about them outside their package name and function. What would cause apt-get on a debian-stable machine to actually cause problems? And what can I do to back out of its changes?"