Slashdot Mirror
Ask Slashdot: NFS on Free OSes Substandard?
Yet another fearless member of Clan
Anonymous Coward wrote in with this intriguing
issue: "I am trying to convince
my company to move off of Digital Unix and
Sun OS to either FreeBSD or Linux as our
primary server platform. The main argument
I am getting is the NFS client performance
on these free OSes is much worse than that
of Solaris or DU. Can anyone give any
recent data on relative NFS performance on
these platforms?"
NFSv3 support was finished for Linux 2.2 last week. Check for the patches in the kernel-list archives.
http://www.linuxhq.com/
NFS preformance has increased /greatly/ with the new kernel based NFS implementation.
It is unfortunately not NFSv3 yet, though a fair amount of the features have been implemented. And there are still a few lingering bugs from what I've heard, though I personally have yet to run into a problem with it.
The raw preformance I've seen is around 40% faster on a single client, which is corroborated by RedHat's experience. They also claim over 400% improvement in multi-client environment.
Probably the most important considerations is WHAT ARE YOUR NEEDS?
If you have extremely large traffic requirements (read large number of clients or large files) or if you absolutely need NFSv3 compliance (for 64-bit file handles, etc), then don't use Linux or *BSD.
If, on the other hand, you are handling a couple dozen clients with low-to-middling NFS requirements, save yourself a boatload of money and use a Redhat server.
But even if the free Unices don't make sense for your NFS servers, by all means recommend them for other tasks - mail server, web server, database server, router, etc, etc.
Some weeks ago there was a huge thread on freebsd-hackers about NFS and the implementation in FreeBSD which has "slight" problems currently. JKH estimated the time necessary to fix these problems in months. It was even suggested to fund one or two developers to take care of the NFS "thingies."
Then, there is Linux. The 2.0 kernel suffers from the userland-only nfsd implementation which has a real impact on the speed on especially fat pipes (>100MBit/sec). The interaction between userland/kernel demands many context changes and data copying between the two areas decreasing the overall speed and increasing the server load.
Linux 2.1/2.2 uses a kernel nfs implementation which is currently under heavy development and as such overall reliability cannot be foreseen. It still suffers from problems with using bigger read/write blocksizes. But HJLu (I think he is working on that) and the other contributors are doing a great job, so this area will improve over time.
If you want to run a big network with many clients (300+), you should currently go with a commercial OS such as Solaris (I don't know anything about HP-UX' NFS performance/reliability) and run it on vendor hardware (yes, I'm conservative). At the current stage of open source implementation of NFS, it would only discredit open source and yourself as a open source advocate, if you would suggest to use open source software for running a huge network. You can easily go with Linux or FreeBSD, if you want to build a rather small network (I have a client with ~70 networked stations depending on a FreeBSD 3.0 server) and don't need a really scalable solution.
We have a small development network of about 8 client machines (linux & sparc boxes) and a linux box for an nfs server. These are some of the times I have collected while I was following the optimization section of the linux NFS howto:
/etc/vfstab file
/opt/stuff nfs - yes rsize=1024,wsize=1024,rw
/etc/fstab file:
/opt/stuff nfs rsize=4096,wsize=4096,hard,intr,suid 0 0
I use the following commands to write and read a file, respectively (see the NFS howto):
(1) time dd if=/dev/zero of=/opt/stuff/testfile bs=16k count=4096
(2) time dd if=/opt/stuff/testfile of=/dev/null bs=16k
One of the sparc clients has a line like this in its
linuxServer:/opt/stuff -
A linux client has the following line in it's
linuxServer:/opt/stuff
This is a typical (I say typical because I'm substituting the average times)
out put from (1) on the sparc client:
4096+0 records in
4096+0 records out
real 0m20.90s
user 0m0.24s
sys 0m2.49s
And for (2):
4096+0 records in
4096+0 records out
real 0m0.69s
user 0m0.04s
sys 0m0.62s
For the linux client, (1):
4096+0 records in
4096+0 records out
0.01user 2.05system 0:21.00elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (89major+15minor)pagefaults 0swaps
and (2):
4096+0 records in
4096+0 records out
0.00user 1.49system 0:36.13elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (96major+15minor)pagefaults 0swaps
For (2), the sparc is significantly faster.
If I change the rsize and wsize of the sparc client to 4096 each, the sparc
client will crash nfsd on the linux server.
We are just a small group of developers who happen to be linux enthusiasts, so we configured this setup ourselves. In short, we don't claim to be masters at configuring unix networks.
The sparc client is an ultra 5, the linux client is an 450MHZ HP Vectra (p.o.s.), and the linux server is a 333MHZ Dell Dimension.
Doesn't the Hardware itself play a large role in the NFS server? I can't see an NFS server needing massive CPU power, but I can draw some lines to Memory I/O bandwidth, SCSI systems, and Network Interface devices. When any hardware componant is "weak" it could potentially effect the preformance some percentage, right?
So, comparing a Sparc w/ Solaris to a x86 w/ Linux/FreeBSD just makes me think your actually comparing a lot more than just OS's, and I would want to know the detailed specs on the systems being compared.
Or can someone somehow prove to me that the software is the over-riding influencing factor, and the hardware doesn't matter?
I have to agree with others who say the NFS implmentation in Solaris is the one that others should be measured against. Rock solid, having adminned Solaris and SunOS for many years.
Recent versions of HP-UX seem to have borrowed a lot of Sun technology (an update to 10.20 gave NFSv3, Sun's autofs, and ONC+ in one fell swoop, and 11.00 incorporates all these also). I work at an all HP shop right now, and I've have to say it works OK, though we don't make heavy use of NFS (no shared home dirs, SW builds on NFS filesystems, etc.), just some light data sharing.
As for FreeBSD, I only have a 3.0-CURRENT box current as of Jan. or so, and a 2.2.8 box, so I don't have firsthand experience, but reading the mailing lists, significant progress has been made on general NFS stability and functionality (e.g. NFS over TCP). I don't have a Linux box, so I can't comment there.
Mike.
Of course, people are working on fixing this...
Host your own websites, anywhere!
i have a P5-100 and an ultra 10 sitting next to each other on my desk at work. i use NFS to cart crap back and forth between them and they're on their own ports on a 100base-T switch. with 2.0.x and 2.2.x and the latest userland nfsd (whatever the latest RH 5.2 update was) i got 2 MB/sec pretty consistently going both ways. i didn't tweak read and write block sizes; they're whatever the out-of-the-box default was. now the P5-100 is running RH 6.0, kernel 2.2.5, and knfsd 1.2.2 (also not tweaked) and i get 3 MB/sec going both ways which is faster than i usually get via ftp. if i had a faster cpu and disk on the linux end it would probably be even better. it also seems like knfsd is a lot more responsive for automounting and grabbing lots of small files, but i don't have any numbers to back that up. as a comparison, i get 5-5.5 MB/sec when moving files between two ultra 10's running solaris 2.6 (seagate cheetah drives on both ends).
linux's forte really is as a desktop unix (my friggin' P5-100 is _so_ much more responsive under X on the console than the ultra it isn't funny) and i think even the performance hit of the userland nfsd is outweighed by the performance gains in other respects (mostly X and file caching). it's also true that linux comes with a lot more software prepackaged whereas with most commericial unices you have to spend a week digging up and compiling such basic stuff as perl, python, or even bash.
until knfsd shakes out a bit more i probably wouldn't want to use linux as a really hardcore NFS server (multiple hundreds of clients, heavy load, etc.), but in my experience it's fine in more modest environments and as an NFS client with HP or Sun NFS servers.
tim
hiding in shadows / i hear you coming closer / you will explode soon -- a quake haiku
(damned enter key, excuse me...)
It seems like most of the responses lean
toward "not using NFS" or "using something
else (CODA, AFS)" but apparently we are still
lacking in the NFS department.
Unfortunately for linux in at least one place
I know, this is terrible. What if (your shop) wants
to use Network Appliance for your storage solution? All of a sudden you have an argument
against linux based strictly on a technical merit -- NFS Performace. Not good. (When you get a
Netapp filer talking coda, call me!) Even dyed-in-the-wool linux advocates in my company are
forced to bite the bullet and use other platforms
because linux isn't suitable to task for NFS with lots of writes.
NFS itself is not to blame, after all Digital Unix
performs well in this context. Even a commercial
NFS implementation would be okay for a solution.
Poor NFS performance has been a problem with linux
for too many years now. I keep waiting for the
problems to magically go away, but I guess they
aren't going to. I've studied filesystems but still don't think I can fix this...
-fb Everything not expressly forbidden is now mandatory.
Linux NFS is just not in good shape, IMHO. I would recommend against using it in any critical situation... but then again, I'd recommend against using NFS in general... =)
Since I don't yet have access to Transarcs AFS client for Linux 2.2, I am using NFS to mount AFS off another machine that is capable of AFS. Working out authentication was a pain (luckily somebody did most the work for me) and it still isn't trustworthy. I login remotely to machines that are really on AFS if I have to do anything more than just a quick edit...
but among the odd things i've noticed: files I don't have permission to just don't show up. This is really annoyingish, when say, I accidently open a file with a stupid mode and it suddenly disapears (spent a good while debugging programs tonight before i thought to log into a remote machine and voila... there was my file, mode 000) I don't know for sure, though, if this is Linux's fault or NFS in general...
anyway, to sum up, I'd say stick with something else for NFS for now (I like Slowlaris... it was my first UNIX)... Linux still has a ways to go, methinks...
This is documented right at the top of the nfs man page, and makes a world of difference. My group at work has a very similar situation to yours (most shares served by Digital Unix but adding more Linux boxes every day), and NFS was definitely a problem until we fixed this.
Div.
--
But my grandest creation,
As history will tell,
Was Firefrorefiddle,
But my grandest creation, as history will tell,
Was Firefrorefiddle, the Fiend of the Fell.
While there are some obscure bugs in FreeBSD's implementation of NFS client under very high loads (many of which are fixed in FreeBSD-current or will be fixed by pending changes to FreeBSD-current), I believe it to have extremely good performance. The attribute cache which was introduced in FreeBSD-3.x reduces network and server load significantly.
I am biased since I work on the FreeBSD kernel and in the past have been involved with fixing and optimising the NFS client but I also use NFSv3 on a regular basis and see excellent performance with 100baseTX (I haven't measured performance for about a year but I seem to remember multiple Mb/sec write performance).
I can't comment on Linux performance since I have only tested RedHat 5.2 (which has terrible NFS performance, IMHO) and I believe that many improvements have been made in the 2.2.x kernel series.