Linux Kernel Benchmarking: 2.4 vs. 2.6-test
frooyo pastes from kerneltrap: "Cliff White recently posted some re-AIM multiuser benchmark results comparing the stable 2.4.23-pre5 kernel against the 2.6.0-test5 and 2.6.0-test5-mm4 development kernels. In his conclusion he makes reference to earlier scheduler tests posted by Mark Wong saying, "Short summary: we mostly rock.""
Comment removed based on user account deletion
The SMP code (written by Linux developers by the way) is supposed to be kicked up a notch in the new kernel. That's what I've heard anyway. I'd love to see Linux being the best OS for multiple CPU scaling.
That will help everyone from the server market, to me when I save up enough for a two processor motherboard.
Slashdot Syndrome: the sudden, extreme urge to correct someone in order to validate one's self.
About two weeks ago I decided to try and install Linux on my old K6-2 450mhz machine gathering dust in the basement.
A friend of mine gave me a few cd's that had something called 'Mandrake' on it.
He said "This is supposed to be the most user-friendly 'distro' out there. Give it a try."
So with trepidation about wiping out my beloved win98se install on the old machine, I jumped right in.
On firing up the install disk, the Man-drake installer asked me if I wanted to remove the win98se partition that already existed. After pondering this for several minutes I though, 'what the hell, I can always reinstall it!' So I let it fly.
After what seemed like 45 minutes of swapping cd's in-and-out of the drive, the man-drake (isn't that some sort of bird?) installer ask me what I wanted to use this linux machine for. So many choices! games, office, mail server, web server, about 2 dozen choices flooded my screen. This is madness! So after carefully considerating my options
I decided to choose them all! I would be a Linux power-user to end all linux power-users!
So after this decision was made I waited. And waited. And waited. During this I started to wonder. My Windows XP Home intallation on my other Peecee didn't ask me thse kind of questions, and it easily has the all the abilities that man-drake advertised to have. After all, I paid for WinXP Home. Sigh, I guess this it the price one pays
for being part of the linux elite.
Approximately 50 mintues later I get another prompt from the man-drake installer asking me what kind of GUI I wanted
to use, KDE or GNOME. Fool me once, shame on you, fool me twice shame on me! I selected both and let it fly.
After only about 20 mintues this time it appeared the install was completed. The mandrake installer told me it was going to reboot and then I would revel in Linux goodness. I waited with baited breath while the reboot churned away, eagerly waiting the opportuntity to use the KDE/GNOME interface. Page after page of command line
stuff flew by my screen, seeming to get faster and faster as the time of my linux deliverance approached. Then, the screen flashed black (kinda like those scenes from the movie Wargames). I gasped and was presented with something like this:
bsh: blah/blah/blah/ ____
What the hell was this? Wasn't this man-drake linux supposed to be user friendly? Instead of the friendly confines of a WinXP like GUI instead I was given an ugly DOS like prompt, which looked supiciously like the TRS-80 system I first learned BASIC on in high school. Is this all the farther the great open-source movement has progressed?
After serveral minutes of sobbing and knashing of teeth, I came to a decision. All the linux fags out there were not going to defeat me! They were not going to cry "Bend over WinXP boy, you're going to take linux OUR WAY and like it!".
I quickly found my old musty copy of 'Unix in a Nutshell' from my college days and got to work. In a few hours I found out how to start the KDE GUI. This made life so much easier. After several days I was able to get the machine's 14.4 internal modem working with man-drake and connected to the internet, using a browser called Mozilla. Where oh where were the glorious pop-ups that appeared as I was surfing porn sites? Those bastards!
After several more days I was starting to feel somewhat comfortable. Using something called Gimp to manipulate my growing collection of adult images was becoming a habit. And because I was ashamed to let my friends and neighbors know I was using a gasp! free operating system like mandrake, I kept the pee-cee in the basement. Nowmy girlfriend thinks the sounds emanating from below are me just woodworking or lifting weights. I guess linux has freed me after all!
CMDRTACO CHECK YOUR EMAIL!
Is it really worth posting this kind of stuff on its own on Slashdot, surely it would make more sense to include it in a roundup of news.
And I didn't call you Shirley.
now i need another CPU to increase performance!
Go figure. An OS that gets faster with each version.
---
"how can the same street intersect with itself? i must be at the nexus of the universe!" - cosmo kramer
If you thought SCO was mad over 2.4, just wait until they make up evidence for the 2.6 kernel!
Trolling is a art,
What the hell all those numbers mean ?
Well, I use Gaim constantly. Come to think of it, I haven't bothered to figure out how to connect to AIM through a Windows machine. I will say that MSN Messenger is not something to compare it to; lots of people have AIM accounts and not MSN accounts. Though, perhaps I have an old model, but I have some problems like links not working properly, not accepting file transfers, and in chats "Ctrl+C" brings up the colour menu...so no copy and paste.
OT, but I'm pretty sure I've never seen "real world" and "instant messenging" in the same sentence. Except maybe with the accompanying phrase "no relation to".
Not to be a n00b, but I can't make too much sense of the benchmark the story linked to. Could anyone give a short simple little explanation of what it means? Thanks so much!
I am a viral sig. Please help me spread.
Seriously, its great and all, but when will it be ready for the masses? I.e. the holy 2.6 release? For us, loading a beta (or even alpha) kernel is something that we can do in our sleep, but look at it from this perspective: all of these improvements will only really make an impact once developers can write applications specific for this environment, which requires, at a minimum, an official release.
Children in the backseats don't cause accidents. Accidents in the back seats cause children.
It's only significantly faster if you have 8 processors.
Whereas it is 7% slower if you have one processor.
I suppose they'll have uniprocessor version which runs faster? Lots of people have uniprocessor pcs.
Hyperthreading doesn't really count.
I run 2.4.22 at work and 2.6.0-testX at home. The 2.6.0test(vanilla) series feel much more responsive, especially in X. I have not done any real benchmarks of my systems, but after working with 2.4 all day 2.6 seems to fly.
Just my observation
-the_crowbar
Have you read the Moderator Guidelines
AIM (now at version 7) is not an instant messanger client. It's a benchmarking tool. Click on the link in the story to see what it is/does/etc.
A better comparision would have been against Solaris x86. Solaris scales very linearly with every added processor.
I assume that when they say the 2.4 Kernel outperforms the 2.6 on a uniprocessor computer, but not on a multi processor computer, that they have recompiled the kernel for each hardware environment.
This struck me as strange, because when the kernel is compiled without SMP support, all that code is left out. So it doesn't seem like the 2.4 should outperform the 2.6 on one cpu.
Does anyone know why this might be?
Slashdot Syndrome: the sudden, extreme urge to correct someone in order to validate one's self.
Looks like that 1970's UNIX code really increases performance for SMP P-III's.
Now we can appriciate the forsite that our Unix fathers had when developing Xeon SMP code in the late 1970's.
"the general trend in the metric indicates everything has been improving, so I think we rock."
:o)
For some reason, the scheduling seems to get more and more choppy (in that i've noticed) with every iteration of 2.4.x kernel. Currently i'm on 2.4.22, and while i don't have any specific tests, numbers or statistics i'm noticing some issues.
Easiest way to reproduce it is to have the machine do something cpu intensive, such as mkisofs, cdrecord, bzip2 some huge file, cp anything large, installing (via aptitude) or even the "Reading Package Lists...." stage of apt-get update.
Oftentimes, the machine will become unresponsive for about 3 seconds at a time, then jolt back up to speed, then pause for 3, on and on. Even after the command line returns the prompt, or gkrellm's cpu and proc krells show that everything is all done, i will still see lag in responses from the kb, mouse, or whatnot off and on for about 10-15 seconds.
I've gone over my kernel config and tweaked a few things here and there but with no change. I can back down to a 2.4.18 kernel and it's not as bad. Going down to a 2.2.x kernel completely solves the problem, but of course will bring its own issues with some of my newer packages (such as gcc) and a few pieces of newer hardware.
A friend of mine and I have gone over this (on my machine and his) and he experiences a lot of the same issues i do.
Mind you, i'm not complaining. I'm very grateful to all the developers of the world that i even *have* a linux system to run. But this is something that makes me more excited about the kernel 2.6.x series. I haven't tried one out yet, but from what i've heard and read, it should be awesoe.
do() || do_not();
Want to get me excitied about Linux ...
1) Integrate seemless plug-in support into Mozilla.
2) Make Open Office slicker/ handle Mickeysoft documents better.
3) Make a spreadsheet that doesn't suck.
4) Make is to that I can actually cut and paste weird stuff between applications.
5) Make is so that my PC can get updated just by clicking on items and not chasing down library incompatiblites or typing "rpm --force" or "make install" or whatever.
I don't care about 1% improvement in 8 SMP mode on some kooky benchmark.
I didn't see anything in the articles to support this, but I'm assuming this is based on x86 architecture. Has 2.6 been ported to other architectures? And if so, have these AIM tests been run ?
Apperently no body here cought the joke of the parent post. But then, it's not very funny, so...
Yeah, you can get AIM the benchmark tool (not the instant messager) from here
New kernel is faster and better than current ones :) And I didn't even RTFA :)
Space for rent, inquire within
and even though Microsoft is financing SCO, the two of them are backed by DMCA legal angles. And there's no way they can never achieve those kind of AIM benchmark numbers like Linux can! I know a certain McBride scalliwag that will be floundering on the end of a penguin's giant meat lance.
After all these years since I first tried to dial in to a Microsoft network I still can't do it without first compiling my own kernel and pppd! I'm just a bit annoyed as I'm sitting here watching my Debian Unstable kernel recompile. For one change: added CONFIG_PPP_MPPE=m. This is a frustrating waste of time! Will this be built in the 2.6 kernels, or do I have to hope that somebody comes up with a better implementation (in Debian non-free perhaps) for this?
Gnumeric (which I have on KDE at least) is a non-sucky spreadsheet. In fact, in the course I was TAing last spring the prof had to switch to it from Excel because it could handle the operations better. The only complaint I have about it is that I can't (or at least I haven't figured out) how to cut and paste into a text document (and vice versa). ...But that was point #4 as opposed to #3, so you can strike one off.
I'd love to see Linux being the best OS for multiple CPU scaling.
You do need a scalable OS to suport lots of processors, of course, but you also need hardware that scales too (clustering doesn't count). Example - SGI is using Linux with NUMAflex on the Altixes to cluster 64-processor system images, but that kind of hardware isn't commodity in any way, and isn't going to be anytime soon.
Anyway, Linux doesn't scale THAT well...as of 9/2000, SGI was using IRIX for a 1024-processor single-system-image supercomputer; I've heard they can go to 2048 now, but I don't have anything to back that up. Dunno about Solaris, but I imagine it's pretty scalable as well.
Facts do not cease to exist because they are ignored. - Aldous Huxley
Are you implying that developers are not designing for the environment untill its out of beta?
I can only think of 2 justifiable reasons for this:
1) Developers can't figure out how to install a beta or alpha kernel..
2) Developers dont trust it enough to belive that code written for a beta will work on an official release.
Can you please tell me Mark's email again, so I can put him on ... err ... the opt-out list? Not that I would send him ghey pr0n or something ... god behold!
something more cpu intensive but still using network resources: SETI anyone?
anyway, I haven't had the time to crunch and digest all these wacky numbers listed in the benchmark, but when I read that "the latest stable 2.4 kernel still out-performs the 2.6.0-test development kernel on a uniprocessor server, but not on a multiprocessor server" it also means that they perform averagely equal on a multiprocessor system, or the latter outperforms the former?
Hack your mind out of its sandbox.
The "We mostly rock" statement was referring to a different benchmark (the one in the story's second link), in which the scheduler performance on single processor machines more than tripled (and performance on 8-way machines went up ~50%) between 2.5.30 and 2.6.0-test5. The first link's benchmark isn't very impressive, like you point out, but it's also not the same program.
They made the single processor version slower, so that it seems to scale alot better when adding processors. Good job.
Hmmm...I don't see the slashdotting benchmarks anywhere in this report.
Just becaues you can't see its use outside of a toy, doesn't mean everybody can't.
Linux IT Consulting and Domino Development in Michigan
Devphaeton, you hit the nail on the head about 2.6.0. Its main advantage over 2.4.x (for this luser anyway) is the smoother multitasking even on a uniprocessor system. I'm running a tweaked 2.6.0-test5 on my laptop, and jobs that would make 2.4.x unusable are barely detectable (from the standpoint of moving the mouse around, typing up slashdot articles, and the like).
:-)
Of course, the ACPI support and swsusp doesn't hurt either
Recently Novell acquired Ximian. They plan on developing a desktop for the company. Since Novell supports Open Source, the whole community will benefit from this. Think about all the improvements they could make to Gnome...
I know companies have tried to make desktop linux a success, but they failed. In my opinion they failed because their effort was not backed by the open-source community. Novell is trying (and succeeding) in engaging the OSS-community in their plans, and that is why they might succeed where others failed.
How 'bout benchmarks with new versions of X?
I keep hoping for faster and smaller....
mark "silly me"
Coupla weeks ago I told one about cold fusion (the nuclear reaction) vs. Cold Fusion (the web application wonkulator from Macromedia). It got modded offtopic.
N4st0r, trixx0r h0bb1tz0rz! Th3y st0l3 0ur pr3c10uzz!
Yeah, look at all the new intellectual property they can claim there!
Seriously, they'll have a field day prosecuting people on any version of Linux, 2.4 or 2.6. It seems the entire goal of the SCO lawsuit is to defame Linux. The companies on the recieving end should get an injunction to silence SCO until the thing gets resolved.
In the end, the only entity that clearly benefits from all this is Microsoft, cause it's just weakining faith in Linux.
Peter M. Dodge,
Chief Executive Officer,
LiquidFire Studios
Platinum Linux - www.
actually they were the 'whigs'
That was the Whig party.
The Wigs are white kids who wish they were niggers.
That would be Whigs.
The benchmark pretty much proves that some change is needed for it to cater to the standard user, as the standard home Linux user, myself included, only have one processer in their computer.
Peter M. Dodge,
Chief Executive Officer,
LiquidFire Studios
Platinum Linux - www.
The benchmarker could _at least_ made some graphs for FAST recognition of the numbers; this is a whole mess and I refuse to RTFA as whole.
It is not too complicated copy'n'pasting the numbers into OpenOffice and click on some symbol for diagrams (I don't have any office-stuff, sorry, else I'd done that in a second).
That is the kind of stuff open-source-stuff is really lacking (speaking in general).
DARL: So, um, hey. It looks like there's this new "too-pointe-six colonel" out on the market from those Lenn-ucks people. We own all that too, right?
SUIT: Well, sir, it's like this. Do you remember how the 2.4 kernel had all of those lines of code in them that are ours, even though they showed up in textbooks before most of our stuff existed?
DARL: Sure, but how does that help us with this new thing?
SUIT: Think about it. Most operating systems, according to my extensive research during years of never having looked at a computer before, contain the same code that they always did, plus a couple of lines of new comments and an extra variable or two that shows how much you're able to charge users for the new features. Just think about the Windows 95 and 98 thing. Perfect example there.
DARL: But...my mansion only has 93 windows. Where is this heading?
SUIT: *blinks* Errr...yeah. Well, it's all the same code, and even those sneaky Linux commies try to pull a fast one on us and put one of those different codes in there, we can always assert our ownership of these "opened sources" files that I just printed out. I asked this guy, you know, and he said that all of these sources are what's in Linux, and since I printed it on paper and stuff, I figure it must be a textbook. Since we own all the words that show up in textbooks, and this has a lot of words, I think we've found ourselves a new angle here.
DARL: Smithers, cry havoc and let slip the Lenn-ucks colonel lawsuit monkeys once more!
I do so hate having to correct you people. *sigh*
"Linux doesn't exist. Everyone knows Linux is an unlicensed version of Unix"- Kieren O'Shaughnessy
linux-2.6.0-test5 - 992.06 - Uni
linux-2.6.0-test5 - 1017.43 - Dual
linux-2.6.0-test5 - 5406.68 - Quad
Does this mean that you only gain 3.49% when adding a 2nd processor? Obviously I don't expect things to scale linear but 3%!? Am I missing something here? And then 81.65% for quad? I'm not trolling, I'm looking for someone to explain what I'm missing.
awww poor you
{{{BitwizeGHC}}}
When will we see a stable version of it?
Sometimes I feel like my whole life is a rhyme, full of punchlines and jokes. I got some punchin', it's like I just can't get it right the first time, or something. When no one knows what your name is and your vinyl's is still in the stores. Once you get a little life, they'll argue over who feels it more. We've got 16 year old netheads buying garbage, wanting to keep you for their personal private artist. I can't fully become my mothers guiding light before my dad returns and tell me what the other side is like. I keep the things you taught trapped in mind. I know you cared even though you weren't here half the time. Who am I to blame, I would probably do the same in your shoes. I never held that against you, complained or assumed you never went through what I'm living. Hell, who am I kidding. Depression is practically part of family tradition. So I keep the time we shared close. It sucks to loose. It also sucks we had to share the month of June. I would've shared eternal time before you left. Each month I celebrate my birth, I'm reminded of your death.
How small a thought it takes to fill a whole life
C'mon /.ers, don't be shy. OK, here's the lowdown:
;D )
STP id: Surgical Temperament Plane identifier
PLM #: Process Longitudinal Multiplexer number
MaxJPM: Maximum Joules Per Month (usage)
MaxU: Maximum Undulations
Change: Change in MaxJPM from previous version
Host: Kazaa user name
Options: Optional meaningless data; always named "profile" and always set = 2.
Hope this clears things up. As you can see, there have been significant improvements in performance.
------
(Note: If you've read this far, you are taking this way too seriously!
Sure, it's markw@osdl.org
Say it with me now, markw@osdl.org
Oh, just like how the No Child Left Behind Act cuts the funding of schools who need more! EXCELLENT!
you can.
The latest 2.4.23-pre* has had substantial amount VM updates from Andrea Arcangeli's branch integrated.
You should give that a shot as it has been reported to substantially improve kernel behavior when under VM pressure like when you're doing intensive IO as described in your post.
BTW, out of the tasks you listed most of them are IO bound, not CPU bound. Notable exceptions are bzip2 unless you have a REALLY fast CPU.
I've tried the new kernel, and I got more responsiveness issues than improvements. But besides that (I might very well have misconfigured something), I'd like to point out that the kernel itself isn't all that matters: the new drivers that accompany it are just as much important. I noticed a significant increase in X's launch time as well as a whopping 250 FPS with glxgears to be compared to the 150 FPS I got with my 2.4.22 setup. This is probably due to major improvements that were brought to the drivers for my i830M chipset.
Sorry, these workloads have nothing to do with instant messaging. They are simulations of real-world database systems. The AIM company not longer exists, but the name lives on.
This is a simulation of a database load. Basically, larger numbers are better. The numbers are tasks per minute and peak user count. The load adds users each iteration until a max is reached. See http://developer.osdl.org/cliffw/reaim/index.html for more
Ok, I think I can keep that.
markw@osdl.org
markw@osdl.org
markw@osdl.org
markw@osdl.org
Oh, perhaps we should confuse the spambots a bit:
markw@osdl.org markw@osdl.org markw@osdl.org markw@osdl.org markw@osdl.org markw@osdl.org
If you had RTFA it says
Short summary: we mostly rock.
- latest 2.4 is better at UP, worse at SMP vs 2.5
- mm3/4 and stock -test5 now very close for most loads,
Which actually means 2.6 is slower for the average joe with only one CPU.
Second, unicasting looks to be slower. Ugh. I don't like that. That suggests to me that there are segments of code which are optimized for multi-processor use - which is great - but either there aren't uniprocessor versions, or the uniprocessor versions are highly non-optimal.
Third, scaling needs to be improved. I don't know if Linux staggers bus use, to optimize usage, but if it doesn't, it should. Perhaps make use of the existing QoS code, or inject wait states if you know an internal bus is going to be heavily loaded.
Fourth, the scaling between 2-CPU and 4-CPU suggests that something is kicking in at the 4-CPU level that's seriously good. My guess would be NUMA, which IIRC, was tied to the 4+ CPU level. NUMA developers might want to look and see if there's anything which stipulates more CPUs than it actually requires.
Fifth, we need to sort out this damn RTOS issue. Linux 2.6 is supposed to go RT when the priority reaches a certain level, but it seems to be more of an rtsched-type scheduler trick than actual RT code. There are lots of approaches out there (RTAI springs to mind) and it might be good if someone added some good hooks into the OS for real-time operation.
Sixth, and this goes along with the above, HP have a scheduler plugin system. Ok, it seems that pluggable schedulers aren't in vogue, but I do like the concept of a scheduling tree with tunable branches.
Seventh, either use UGASI or don't. Adding in IPSEC is cool, and all that, but the old IPv6 stack is beginning to get stale. (In fact, there's also a lot in the IPv4 stack that's stale, and needs to be replaced.)
Last, we've got the filesystems. I've not seen much serious CODA or Intermezzo work, recently, and I've never known the kernel-provided Intermezzo to work without problems. On the other hand, we've got Lustre, which seems to be a whole lot faster, and seems to be under active development. Yes, it's "late in the day" to go adding whole new components, but the impact would be minimal, and it would make a lot of networked users very happy.
(And happy network users give karma to slashdot freaks!
Good questions. But I've got a few more...
Using the numbers you quote:
Single-to-Dual comparison: 1017.43 - 992.06 is a difference of 25.26, which is 2.5% (you rounded up to 3%) of 992.06. I'm with you so far...
But where did you come up with 81.65%???
Now for Single-to-Quad: 5406.68 - 992.06 = 4414.62 which is 445% of uniprocessor performance. Somehow with quad, you get even MORE porformance per CPU than you get with a single???
Dual-to-Quad: 5406.68 - 1017.43 = 4389.25, or 431.4% increase. That's VERY odd.
Why knows if Debian lurks in the hearts of men.... The Penguin knows! Debian redefined Linux for me.
AIM (now at version 7) is not an instant messanger client. It's a benchmarking tool.
Hmm, you wouldn't want to be on the receiving end of the AIM-7 benchmark.
It packs quite a punch
Never ascribe to malice that which is adequately explained by incompetence.
I'm osrry, but he's right.
:p )
Linux's SMP support is in its infancy- Solaris, HPUX, AIX etc have been hammering it for YEARS.
You can't just expect Linux to instantly be equal to those OSes in that regard.
(INsert obligatory "without illegal insertion os SCO-owned SMP code in to Linux by IBM blah blah blah
I browse at +5 Flamebait- moderation for all or moderation for none.
How about some charts/graphs... Reading raw numbers in long strains is rather dull. It's easy to get them mixed up as well.
;)
Are they trying to hide something this way?
-]Phreak Out[-
Suck it down, biatch whores!!!
I for one, welcome our new multiprocessor-prefering overlords.
Amen. 99% of x86 users would have a single processor. Servers or die-hard hardware magnates might have more than one processor. Isn't there an ongoing debate on whether GNU/Linux is ready for the desktop? It seems that the 2.6 kernel is aiming away from that market now. Perhaps more development is needed to be aimed at the desktop user if it is to compete with XP Home and not just 2003 Server.
I don't post charts when sending to a text-only mailing list such as linux-kernel. Not much point to that. If you'd like charts, see the full reports here: http://developer.osdl.org/cliffw/reaim/index.html
HAL: "I honestly think you ought to calm down; take a stress pill and think things over."
Most humor on Slashdot is lame. You can correct for that by putting a negative modifier on "Funny" in your Slashdot settings. I find a value of -2 works well.
Single proccessor systems are the most common, and always will be. There are millions of millions of them out there.
Dual proc and quad proc systems are also semi-common, but usually only in server scenarios. The best ratio you'll ever see on them is probably around 50 to 1 -- 1 quad CPU server, 50 uni CPU clients.
Each time you move up, you get diminished returns. At what point do you go, "hey, making Linux scale more is causing a detriment to our single-CPU millions, and we're spending man years going from 512 CPUs to 1024 CPUs -- is it worth it?"
I'm sure Irix is great for 1024 CPU installs, because SGI sells the hardware and has no problem charging an arm and a leg for the software needed to run it. After all, if you're on the market for a 1024 CPU nuclear fission simulation computer, you can't really go and grab just anything off the shelf to do it on.
The Linux community won't gain anything if we go to the point where 16 or more CPUs is something the kernel can do easily. How many of us own these systems? How many of us can develop and bugtest on these systems? How complex will the kernel locking on all the subsystems become in that setup? When is Linux scaled too much?
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
How does affects the increasing of clock speed, in order to get a smoother multitasking, specially in older machines?
Currently, the clock ticks at 1KHz, vs the older 100Hz frequency... This means, a lot more time spent in context switching... As the clock freq is independent of CPU speed, is has greater impact in such cases.
I still have to work with 3+ y.o. computers (My own is 5 y.o). How much gross degradation that means?
Got Pike?
But then you miss the ones that are genuinely funny (rare, but worth it. sort of like browsing at -1 when you moderate)