Slashdot Mirror
Preventing Shutdown on Active NFS Servers?
Ed Almos asks: "Like many Slashdot Readers, I run a small network at home with a server and a number of desktops. The server holds all our files as NFS shares and doubles as a desktop machine should the need arise. Problems however occur if the server is shut down whilst there are NFS shares in use, the minimum disruption is a crashed desktop and a couple of times I have had to deal with corrupted files. Does anyone know of a way to prevent shutdown of a machine if someone else has drives mounted to its NFS shares ? I have already explored use of the /etc/shutdown.allow file but all this does is determine who can kill the machine. The minimal solution would be something similar to a Microsoft Windows system, where a request to shutdown brings up a warning window that there are users connected to the system, but I am not sure how to achieve this on a Linux system. Ideally I would like to prevent shutdown of a system with active NFS shares altogether, or at least until the user has unmounted and logged off the network."
NFS is stateless from the server's perspective. This is done so that the server doesn't have to track the state of a whole fleet of clients (and so that the server can pick up where it left off when it crashes and restarts).
So the server, by design, has no notion of the number / names of users connected to it.
The best you could do would probably be to monitor NFS traffic, and present a dialog on shutdown if there has been any traffic in the last 5 minutes or so.
Not quite an answer to the article's question, but...
Theoretically, once the NFS server has crashed, shouldn't all clients simply freeze until the server is back? On all systems I used, this was the observed behavior, and it is quite useful actually: it seems to avoid all data loss problems (under conditions). When the NFS gets reachable, all running program go on executing as if nothing had happened.
A solution to the original problem, though, would be: tell all user that the NFS machine is to be powered on constantly.
1) Put a big piece of tape/wood/whatever protecting the power switch on the NFS server, and disable software shutdown by non-root.
2) Put your NFS server on the *best* machine, not the worst one, so that users want to use it first. If worst comes to worst, put signs on the other machines advertising the other's superiority. (and without the NFS overhead, it will really outperform its clients!)
3) Put your NFS server on a g3 laptop or other ultra low power system, and hide the system in your closet so others can't find it to turn it off. (hardware suggestions, anyone?)
4) switch to samba (heh heh heh)
If your NFS server rebooting, shutting down, or crashing causes any problem but temporarilly 'hung' clients, you have something wrong.
:)
NFS is explicitly designed to be stateless, precisely to allow it to function across server reboots, crashes, and other fun. If your clients are crashing, or getting back corrupted data, something is screwed up somewhere.
And, by the way, if you're getting corrupted data on a server crash, and the server is linux, you just had an object lesson on why it's bad that linux NFS defaults to async writes.
I know when I am shutting down my computer from a command prompt, it pops up saying the computer is being shut down in KDE. Obviously I know this because it's my computer, but don't you get a similar message when you shutdown your server? I am sure you can delay the shutdown for 30 seconds or so to allow users to unmount. I remember at our university, the 'server' when down on our solaris machines, and everything froze, but then re-started again in a couple of minutes, without problems. People booted into Windows got blue screens....
Bored? http://www.dodgybloke.co.uk
If that isn't good enough for you, there are a couple other possibilities. You could probably cobble together an utterly trivial Python (or Perl or whatever) script on your client machines, then have the server invoke it via ssh when a shutdown starts. If you aren't a programmer at all, you could try firing off an email to the client machines. As long as you have a periodic mail-checker going, it would alert you to the arrival of a new message. (Since you'd be able to use the local spool, you could have it check every 15 seconds.)
If I understand you correctly, it sounds like you just want to replace shutdown with a wrapper script.
/sbin/shutdown to something like /sbin/shutdown.real.
/sbin/shutdown that checks for your NFS mounts before invoking shutdown.real.
/etc/fstab...or just compare /etc/mtab to /etc/fstab. Whatever....just look to see that your have NFS mounts. If you do, don't shutdown; if you don't, go ahead and shutdown.
a) Move
b) Write a shell script called
If you want to make it really fancy, you can do something like calling and parsing the output of 'df', comparing that to the contents of
Is that what you want? If that *is* what you want, make sure you backup your wrapper and be aware that the next time you upgrade your base system, the wrapper will get blown away. Also make sure your wrapper script is passing all command options to the real shutdown.
Tyler's words coming out of my mouth.
If you use the correct mount options you should not have to worry about corruption when the nfs server goes away.
The options you want (for filesystems mounted rw) are:
rw,hard,nointr...
A lot of people don't like these options because it means that the clients will hang until the server returns, but it is THE RIGHT THING TO DO if you are mounting important data rw. If you can't stand for your clients to hang, maybe replace 'nointr' with 'intr', but you've been warned.
Steve
use lsof to monitor tcp/udp/rcp sockets that are open on the host and pointing at the file space that nfs is serving.
/.?
Then write a wrapper around each of halt, shutdown, and reboot to check the open ports and fail if they are active.
Seems fairly hackish, but... whaddya expect from
-- The morphemes of your disquisition are ascertainable, but they have eschewed an ambit of transpicuous exposition.
I can't remember the details on this, but would the NCF Locking Services work for you?
NFS input/output is stateless, but I believe the locking mechanism is stateful.
When clients are accessing a file, a lock is established. When the client is done with the file, the lock is removed. You can see who has what resource locked with a utility (I forget which, but fcntl() and lockf() come to mind).
In a shutdown script, look for locks, and refuse to procede until the locks are cleared.
"Can of worms? The can is open... the worms are everywhere."
but wouldn't one key things to consider when building such a system be that a) it's down as little as possible and b) when it goes down it's well known beforehand(and the users can be told in advance that it will go down at time x and they're fsced if they don't get out before it).
look, what point would there be in initiating the shutdown if you didn't know when the users will get out anyways? it could take hours/days before it would actually boot, and if that doesn't matter(waiting _hours_) then why would you be booting it in the first place? just out of habit?
anyways.. it sounds a lot like you should be fixing on why you have to be booting it instead of how the booting occurs.. so that you wouldn't need to be booting it.
world was created 5 seconds before this post as it is.
hide the system in your closet so others can't find it to turn it off
NEVER, EVER PUT A COMPUTER IN A CLOSET.
As a pretty general rule of thumb, closets are made of wood, which is a polite way of saying kindling.
NEVER, EVER RUN ELECTRONIC GEAR IN A WOOD CLOSET. EVER.
I work for a major NAS storage company. Using the mount option "hard" is the right advice. It sounds like the submission author is using soft NFS mounts, which is a big no-no with rw mounts where you want any kind of data integrity.
As a hack, you could replace /sbin/shutdown with a shell script that pops up a dialog (If $DISPLAY is set) or asks on the console.
tasks(723) drafts(105) languages(484) examples(29106)
put a program on each client machine, call it nfsmounts and it would go a little something like this:
mount | grep $1 | wc -l
then write a wrapper on the server that does
foreach client (client list)
mounted = ssh client nfsounts `hostname`
ok = false if (mounted)
end
you can hook that into your shutdown script, and abort if there are any clients who think they have a mounted drive.
of course, read the other suggestions about mount options. Noone's mentioned sync yet, but don't mount your shares async even though the performance is so much better or you'll loose data.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
I know that it's not quite what you wanted, but using Coda, which is designed to support disconnected operation (i.e. servers goes away for a while and then comes back) may be an appealing option for you.
May we never see th
First off, I would recommend that you never run applications on your file server. That just seems like tempting fate to me. Get a cheap old system from reseller/friend/garbage and use that.
Second, use FTP instead of NFS. Allow it to support resume, only let it talk locally. I believe there are utils which will allow you to mount an FTP as a drive in windows as well...
couple of ideas anyways.
> 4. Log back in and try to moderate your AC comment...
> you can't!!!
I just did.
> Big brother is watching. So while I might be a "troll" a lot
> of the AC things I said were to protect myself from Slash-
> bot groupthink. They punished me for voicing my opinion
> freestyle.
It's an online forum. Go get some sun.
Where I work they had this problem and someone implemented a automounter program that mounts the shares when they are needed and releases them as soon as they are not needed. Not a perfect solution but it works really well.
No, you haven't got it wrong, this is EXACTLY what I am looking for. Some sort of wrapper around the shutdown command which looks for active NFS shares before shutting down the system.
Ed
The more corrupt the state, the more numerous the laws. - Tacitus, 56-120 A.D.
The file /var/lib/nfs/rmtab on the server keeps a list of what systems have mounted NFS drives. What you then do is this:
1. When shutting down, first go through rmtab and send an rwall message to those machines, saying 'get the heck off because the server is going down shortly'
2. Two minutes later try again and send a more forceful message.
3. Two minutes later tell them they are about to get it in the neck. And shutdown.
If you really dont want to shut the machine down with NFS mounted stuff still there, then modify to taste - dont shutdown unless rmtab doesn't have any shares from machines you dont want to annoy.
This is untested of course.... Until you try it! Read the warnings about rmtab in man mountd. It may not be trustworthy.
Baz
rm -f /usr/sbin/shutdown
10b||~10b -- aah, what a question!
... a long time ago. Basically, every user gets a simple switch, all switches are connected in parallel, so you get a "wired-or". All users are told: If you need the server, turn on that switch. If you don't need the server any more, switch it off again. A little piece of electronic connected to a RS232 port and a tiny C programm control the power supply for the server. Details (german only, sorry) are here.
The trick is: The server now knows best when to start and to shutdown, there is no more need for manual boot or shutdown. If you replace the switches with a relay per workstation that closes the contact whenever the workstation is powered on, you don't even have to tell the users about the switch. You don't even need extra cables if you use 10/100 MBit/s ethernet, there are two unused wire pairs in every ethernet connection.
OK, this is a hardware solution, not a nice popup window. But don't we all hate those nasty popups? ;-)
Tux2000
Denken hilft.
Couldnt you use lsof to get a list of open files and check if they are in mounted nfs shares ? If there are any when shutdown is called use a bit of scripting to pop up a dialog ? Just my 0.02
--- Did I say that ?
I use samba to share my files so i can't comment on NFS too much.
Similar setup:
one Linux file server, 3 desktops (mixed nix and win2k), when the server shuts down, any cients active go nuts (winamp freezes, explorer complains about the sudden disconnect)
best theing to do, since you're on a home network is to make sure no one's using the damn thing before you shut it down.
This is easy, even with 6 systems, (assuming samba), just make sure no programs are activly using the shares and reboot the box, the workstations will yelp in agony, until the server returns, then all's well.
In windows, when a mounted (read: drive mapped) smb share goes off line, windows complains, programs too, then windows marks the drive as disconnected, when the server's back up, reconnectying by just clicking the drive in My Computer restores everything to normal. I haven't messed with client side SMB under nix enough to know what happens when a samba server reboots (gracefully or otherwise) while files are in use client side.
--
Logistical Chaos Officer http://www.slagg.org - LAN Gaming in Sarasota FL,USA
showmount -a is your friend ... that will show you all clients that currently have a filesystem mounted... just move your shutdown, reboot or whatever out of the way, replace them by a wrapper script that checks if showmount -a returns any clients, and only executes the real shutdown when noone has the filesystems mounted... if they do you can always print out a list of workstations that have mounted filesystems...
Does anyone know of a way to prevent shutdown of a machine if someone else has drives mounted to its NFS shares ?
Put a sign on the server that reads "Do not shutdown this server". If you want a technical solution - use a post-it.
The parent posting is well worth reading.
A Shadeless room is a brighter room.
Define all of your enviros at the start of your script and you won't have that problem. {BTW Not everyone uses csh... it just reeks of being a Sunny)
I use nfs locking for everything. It's just as good as regular file locking.
/nfsmount/.lockfile on the nfsdrive readable by everyone. /nfsmount/.lockfile & Put it in the background. /nfsmount/.lockfile & ) /nfsmount/.lockfile || exit; ...)
:). If you have problems contact me: slashdotNFSlockdec03.12.cudgle@spamgourmet.com
Try this:
Create
After the clients mount the nfs drive, have them perform a read (non-exclusive) lock on
(clientnfslock
In the server shutdown script put in a routine that fails if it can not perform a write (exclusive) lock on that same file.
(... servernfslock
Note that flock will not work across NFS, so you have to use fcntl. Of course you can do this in perl:
servernfslock:
#!/usr/bin/perl
use Fcntl;
$cinfile=$ARGV[0];
open(TEMP,"+ $cinfile") || die "Couldn't open the file $cinfile\n";
&lockfile(TEMP) || die " Could not lock file $cinfile\n";
sub lockfile {
$bub=$_[0];
$lockz = pack('s s l l s', &F_WRLCK, 0, 0, 0, 0);
$gg=fcntl($bub, &F_SETLK,$lockz ) || -1; # do a write lock
return $gg;
}
and clientnfslock is the same except you use a F_RDLCK instead of a F_WRLCK, and you put a
sleep
at the end of the script so the script never dies and the file never closes and never becomes unlocked.
The beauty is that this utilizes the very service of interest (nfs) and none other.
Of course, don't run clientnfslock on your server, or it won't shut down
That is what "showmount" does.
Simply write a wrapper shutdown that runs
showmount and lets you know who has it
mounted.
People like you are a great example of why there should not be truely anonymous posting.
Oh THE POWERS THAT BE! You fascist Waffen Shutzstauffel Waffen SS Hitler's secret police. You just thought yourself judge jury and executioner. You said, you don't like someone for revealing a truth in you fascist machinations and you would rid EVERYONE'S RIGHTS TO SUPRESS YOUR POLITICAL ENEMIES. This is a legitimate justification for you death. I hope freedom fighters around the world target you and mark you for death.