Slashdot Mirror
SGI announces Linux Kernel Crash Dumps (LKCD)
Alphix writes "SGI has announced their Linux Kernel Crash Dumps project - and it's gone to release. It's intended to simplify the examination of system crashes thru saving the kernel memory image when the system dies due to a software failure, recovering the kernel memory image when the system is rebooted and then examining the memory image to determine what happened when the failure occurred."
> And also, it's one of the things I really, really, really HATE about NT. No debugger comes with the OS, and there's no free, distributable one out there, so from a tech support standpoint, if your customer's server barfs, you kind of have to guess at what went wrong, or establish a pattern from multiple calls, or try to reproduce it in-house.
Yeah it sucks. The only solution that I'm aware of is to get your customer(s) to install MS Dev Studio, or even NuMega's SoftIce. Not very practical, but its better then nothing.
Cheers
I think you misunderstood him. Solaris (I assume) has the ability to dump core *for the kernel*. Obviously, not into the filesystem - thus the swap/savecore dance.
And no, it's not only for applications. And it's *very* useful.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
Crucify me for saying this, but I have identical results with NT as both a workstation and a server... Your mileage may vary.
--
E2 IN2 IE?
As you note, this is far more than a mere Kernel core dumper. I know this site attracts many professional developers and sysadmins, but there are far more who have never had the pleasure of driving IRIX. Linux is really good, given it's maturity level, and I use it at work and home - I develop for it at work for our products : see Ariel Corporation ISP products, but IRIX has a some real jewels, and SGI has chosen to give the technology to the open source world. The first was XFS ( imo the best filesystem ever invented )plus some other assorted stuff that SGI is paying it's programming staff to give to us, and now the technology to pinpoint the exact cause of a kernel crash. The IRIX kernel crash postmortem technology is far beyond a mere core dump and pointer - it tries it's best to identify the offending system call, and pid if it can. This release appears to be a port of that technology to Linux.
Stop whining folks - we have just been given one of the best debugging tools ( especially for kernel hackers and device driver writers!! ) in existence as a gift. Try using it, and be sure to thank SGI. After all, even though they have market reasons to do this, they still *did* it.
Yes, every reasonable operating system can be configured to save the core files resultant from a kernel panic to swap, and yes, many provide excellent tools for conducting a post-mortem analysis of the image to diagnose what caused it to croak. But in the past, with the notable exception of IRIX, this process required a fairly intimate knowledge of the operating system and even the underlying hardware, and was considered something of a black art. An excellent book on core dump analysis issues/procedures is 'PANIC!' Unix System Crash Dump Analysis, published by Sunsoft. IRIX, and now Linux when properly configured, automatically conducts the crash dump analysis upon re-entering multi-user, saving a legible and comprehensible report detailing what was going on at the time of the crash and providing a suggestion as to the cause.
This facility can be an excellent way of quickly tracking down the cause of the panic, or at least determining if the problem lay in hardware or software. Below are three examples of some recent reports generated at our site:
Sample 1
Sample 2
Sample 3
While this utility is no replacement for an experienced sysadmin and a debugger when it comes to deciphering the cause of failure in complex systems (especially SMP), it will likely be a boon to the hundreds of thousands of Linux admins supporting small workgroup servers and workstations. And yes, Linux is stable.. but c'mon: kernels panic.
That guy has no clue, its obvious.
He runs a default RedHat install, with everything enabled and still running the same kernel that came with it, he has no SCSI devices, or a clue to even know what SCSI is, his largest partition is probably a 15GB Windows98SE partition, and he boots linux to winnuke his non-elite irc friends.
Come on, "Stopping md devices..."? Does he actually use md features? I seriously doubt it. Its just that default RootHat install pushed it down his throat. And apmd? It's kind of pointless on a AC powered system. And it's really pointless to run RedHat on a laptop since even the "Laptop" installation still installs updated whichi will spin your hdd every 5 seconds and make your battery last less than it does in Win95.
I fucking hate ignorant people that Redhat and similar idiotic distributions bring to the world.
You should have considered placing it under a BSD-style license. Microsoft's feelings are going to be hurt over the fact that they can't incorporate it into Windows 2000.
One of the things I remember back from my days when I was tinkering with SunOS 3.4 and Ultrix and IBM's AOS (not AIX) was that many BSDish Unixes would write what were basically kernel "core dumps" to the swap partition when they died (I may be getting details wrong -- might not have been swap file, might not have been all those OSes, etc.). Sophisticated gurus could then fix things. (Back in those days I was not enough of a guru to do this myself, but I lived and worked with people who were.)
I think it's *wonderful* that a facility like this is coming to Linux. It makes me much more enthusiastic about taking on kernel hacking myself.
But out of fairness I do have to ask... don't the BSDoid operating systems already have this?
And it's a little embarassing to point out that NT has something like this as well.
What happens when the LKCDA crashes during a system crash? Who recovers from that??
Nobody. A crash dumper is going to be a minimal, always-resident program designed to simply copy physical memory to disk. If that can't be done, the system is either fried at the hardware level, or is so far corrupted that a core dump wouldn't mean much anyway.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
I have been thinking about a solution to kernel panics and no-reboot kernel upgrades for a while, and here is the only thing I have come up with that seems viable:
We have redundant power supplies, hard drives, and many other pieces of hardware. I am thinking it may be good for developpers, at any rate, to use redundant kernels. Kernel 1 dies, kernel 2 realises this and kills kernel 1 and takes over the system. Interrupt in service: a few clock cycles. Perhaps a new runlevel should be implemented into the linux kernel...runlevel 7, which would be against the POSIX standard I think, not sure, but would allow a condition in which the kernel is replacing itself in memory, by having a redundant kernel take over while one is being replaced in memory, and the second kernel handing off resources to the new primary kernel when it is ready, returning to the previous runlevel.
The long and the short of what I am saying is that there should be a second kernel in memory at all times ready to take over at any time, but programmed to not run until the first kernel dies or is being upgraded.
The disadvantage: it starts to consume extra memory resources, and process table entries, and will take a long time to perfect.
What do you think?
OFTC: By the community, for the community
This is not a core dump of a running application, but rather, a core dump of the entire running system. If a kernel failure occurs, this patch will dump the contents of system to memory to disk, allowing you to analyize system state from just before when the crash occured.
This would be very useful, for example, when debugging a device driver. It is not something the end-user, or even system administrator, is likely to use. It is for the kernel developer.
Other OSes (Sun Solaris, SGI IRIX, Novell Netware, to name a few) have had this capability, but Linux has not. Linux has traditionally dumped a summary of the kernel state to the screen, but that is (1) tedious to copy down by hand (which you have to, since the system is dead), and (2) not as complete as an entire system image is.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
WinDbg, while probably not redistributable, is a free download from MS. It can read NT kernel dumps. Try here or here. Unfortunately, they're already orienting these places to Win2k.
IRIX isn't Sun's UNIX, it's SGI's UNIX. They're unlikely to sue themselves for stealing an idea from IRIX....
BSD, as others have noted, has had it for ages; many other flavors of UNIX probably got the idea (and, in some if not all cases, the code) from BSD.
What jafac was saying is that Microsoft does not give you or offer any low-cost, distributable tools for making sense out of this massive pile of arcane charachters.
Got 128 MB of system memory on a NT workstation? 512 MB on a server? Hope you've got Einstein and a couple years to sort through the thing by hand to find your problem!!
And UnknownSoldier has another very good point: The analysis tools are not cheap, and you can't share!
C'mon Microsoft, didn't you learn anything in kindergarden?
"...America's great minds of today, teaching America's great minds of tomorrow. Poor bastards." -- A Beautiful Min
On the NCR tower this was done by toggling a switch to get into the boot ROM, then choosing appropriate options, then the memory would be written to the dump device.
This helped us diagnose many strange crashes where the system wasn't functioning correctly, but it hadn't actualy paniced, for example, one system had init die, which made logging in a bit hard, but the kernel was still running.
I've missed this feature on more recent PCish hardware, as they don't really have a boot ROM.
Perhaps someone would like to make a more Linux biased BIOS, which could include these sort of nice features.
I don't think so.
As far as I know, OS/2 has had this for years now.
While it will tell you to write down the information which it dumps to screen (stupid!), it actually also saves a copy to disk.
The only time I've had the pleasure of this experience was when I fried my mobo...
I hope SGI would be smart enough to do this "clean room." If not, SCO (not Sun, not HP, not IBM, not Compaq, not GNU, not AT&T, not Novell) could sue them. SCO owns the UNIX source code.
Let me know (of course you will) if I'm wrong on this.
--Al
Kernel Panic: Linux Kernel Crash Dump Subsystem received signal 11. giving up.
--- Sueños del Sur - a webcomic about four young siblings
Sounds like a good idea... although windows could use this more, lol. But seriously, i have noticed some problems when it freezes (yes it has happened to me, but rarely) i have no idea why. [coughnetscape].
If we fail, we will lose the war.
Had to do it lol
Restating the obvious since nineteen aught five.
I talked to Stephen at the Expo in London and
is not his intention to push this into 2.3. So
unless he (and Linus) changes his mind, it won't
be going into 2.3.
The description says that it saves the dump to a SCSI partition. What happens if you're running IDE?
;-)
I think the idea is pretty cool -- no more trying to figure out why ksymoops didn't grok what you hastily scribbled down. I suppose all the hardcore kernel hackers will cry "Sacrilege!" though.
P.S. Sun won't sue for stealing their crash dump idea, right?
Glückwünsche, haben Sie Slashdot ermordet, indem Sie zum korporativen Druck beugten und Subskriptionen einlei
Another comment - I know that reiserfs is being submitted for 2.3, but nobody knows whether Linus decides that it goes in at this time. It would be very nice to have for 2.4, though.
And 2.3 is not as simple as you think - although it is "just" ext2 with a journal, you have to consider stuff write ordering, for instance.
Forgive my ignorance, but I'm not to educated on the "under the hood" stuff in *nix environments, but is this a new thing? I've noticed that under Solaris that I've got core files after a crash. Are these the same type of thing or do they not apply to the kernal? If not what are core files for? Do they have any use beyond cluttering up directories?
This is good... lets you see just what went wrong when the server went down for the first time in 2 years. Should make for finding the bad programs that do bring linux down.
Well, chances are this article does has nothing to do with your every-day user. I tried booting 2.3.x once, and it was a no-go. (scsi driver failed).
;-)
as for redundant.. well, at the time I posted, it wasn't redundant.
and.. NO, I don't sit there waiting for first post.. It just happened to be that way when I checked. I saw an article about kernel panics.. and I thought.. "Well, my kernel has _never_ panicked, this is pretty useless!"
As I was typing, though.. I figured it would be a great help to kernel hackers.
Having worked in a Solaris shop, you can see the value of having crash dumps to send to your vendor.
Actually, we were the vendor and got crash dumps from customers that was able to pinpoint very quickly what the problem was. Once that was found, it was easy to fix. Without the crash dumps, it could take weeks to find the cause of a nasty bug. Especially intermittent ones.
With Linux having this feature, it'll be easier for driver authors to debug their code, and most likely boost the confidence of customers who want 99.999% uptime.
-- Ever notice that fast-burning fuse looks exactly the same as slow-burning fuse? I didn't... (Edgar Montrose)
Then you get a "double panic", a very cryptic message, and no crash dump. Very rare, but it can happen.
At least, that's how *BSD handles it. "double panic" is engineereese for "fix your broken hardware".
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
Has anyone said it was a new thing. Linux just lacked a kernel debugger and now it has one. Linux still lacks a journaled file system, and will eventually have one. Nobody's saying it's new, but it's still a reason to be happy for.
Opus: the Swiss army knife of audio codec
I don't normally have problems with crashes either. Currently, however, I am working on kernel modules for solaris. Until I learned how to use adb on the kernel crash dump, debugging was impossible. Now it is relatively easy, just use adb -k unix.0 vmcore.0 and $c will show you the call stack. This works great for debugging kernel level drivers and modules. I can't wait to try this under linux!
--
Mike Mangino Consultant, Analysts International
Mike Mangino
mmangino@acm.org
Hey AC, why don't you get your facts right ?
1 - the SGi hardware is amazing. They make some of the finest machines available ( Octane 2000, O2 ), and they achieve a level of parallelism that Linux still dreams about ( in an allways undervalued Irix machine )
2 - their contributions to Linux have been very good and well received. KDBG is a very useful tool, and coupled with LKCD will make kernel and driver development a lot easier. GLX iwill be used in XFree86 4.0, they're working in the Linux for Merced port, etc.
3 - OS scalability: have you ever heard of IRIX and its support of more than 64 CPUs ?
4 - PROVEN fact that Linux never crashes is bullshit. It's just an OS like any other that can also crash, in partiuclar during development releases. I'm doing multiprocessor research on it and today I made it crash twice. I like Linux very much, but I try to keep my eyes open.
Finally, what have you done for Linux lately ? SGI has been supporting Linux constantly during this last year, and they don't deserve to be treated this way ( remember, there're people hard working there that contributes their code under the GPL )
Enough, you don't even deserve my time answering your stupid post. Go back to your perl scripts ( or was it VB ? ).
The resource kit includes tools to interpret the core dump and regurgitate the BSOD contents (which, BTW, almost always points to a video driver file). If that isn't good enough go over to www.sysinternals.com where there is a utility that saves the screen contents specifically.
I've got this one computer at home that crashes EVERY time it hits runlevel 0. I think it's got somehting to do with apmd, but anyways, it's one of those computers that can turn themselves off through software. Normally, the last thing you see is:
Stopping all md devices.
System is halted.
Power down.
And at that point, you either have to turn it off yourself, or the software (apmd?) does it for you. Well this one box I have (a crappy HP I got for free) gets right to the words "power down" and then it dumps all sorts of crap onto the screen, including the values in the CPU's registers, and what I assume to be some crap from memory. What I'm thinking here though, is that since all the filesystems are already unmounted, LKCD wouldn't make a lick of difference for me. Am I right in assuming this?
-- My neighbors dog has a four inch clit.
At least two journaled file systems will be in 2.4. Reiserfs and ext3 should both make it in. Posibly XFS as well (not heard any news about that one). Hey, and 64GB max memory in 2.4 as well (still 4GB max per process though).
Plato seems wrong to me today
" Memory dump files are created when a STOP error occurs, and the system is set to save debug information in the 'Startup/Shutdown' tab of the 'System' Control Panel."
source: support.microsoft.com
This is a real easy one to setup. The feature's not usually used on small workgroup servers because there's usually no one around who can do anything with a 256MB binary. I was going to say a lot of nasty things about dumb NT admins, but I thought I'd be nice as I was one (and will be again if the money's right).
It's better to be uninformed than misinformed.
_damnit_
_damnit_
It's my job to freeze you. -- Logan's Run
" Memory dump files are created when a STOP error occurs, and the system is set to save debug information in the 'Startup/Shutdown' tab of the 'System' Control Panel."
source: support.microsoft.com
_damnit_
_damnit_
It's my job to freeze you. -- Logan's Run
That wouldn't really solve the problem -- obviously, someone has a cronjob that runs a script to scan for new /. stories.
Naturally, there are people and things we'd rather not deal with, but just like IRL, it's unavoidable. If that thought is too traumatic to deal with, you have two options:
Any kind of censorship (including an IP ban) is bad, bad, bad -- but what do I know? I'm just as much a part of the problem as anyone else.
--
E2 IN2 IE?
A good hacker should be able to do with just a register dump, stack trace and some program text surrounding the instruction pointer where things went belly up.
Hacking the kernel is supposed to be hard and tracing crashes given minimal information is a big part of the fun and attraction of ``iron man'' programming.
Then again, having a full dump doesn't necessarily make debugging that much easier. It's an incremental improvement over oops text.
Here is the real advantage: a dump is good from the point of view of users who need to report crashes to developers. I think that even a hack to get oops text (rather than a full dump) written to a partition would be better than asking the poor user to copy the oops text appearing on the frozen console down on a piece of paper! Forget it!
You need TWO PeeCees hooked together by serial
port. Then you put one computer in "debug boot
mode", and control the debugger using the other.
Feh.
On Solaris, you just grab the core and symbol
files, and use adb. On just one computer, with
no special boot modes, with the machine running
whatever.
Having this ability on linux will be very very
convenient.
-----------
"You can't shake the Devil's hand and say you're only kidding."
yup once, accessing a floppy drive, dont know why, dont want to know why cuz it's never happened since :)
I've personally never seen a userland program crash the Linux kernel. The closest I've come is having bugs in the X server lock up the keyboard and display, but the machine was still running fine in all other regards, and I was able to telnet in and initiate a clean reboot.
Couldn't agree more, we have a farm of O2s and they crash like hell, by Unix standard that is; almost one crash per month. We had some code that would systematically crash machines with panic error. I just hope that what SGI has to contribute to Linux is not instability!!!!
strace -p pid
Hope this helps...
--- polarbear
Dunno about you, but my FreeBSD boxes all have truss...if Linux doesn't have a version of truss written for it yet, somebody definitely needs to work on it.
"That's Tron. He fights for the Users."
As someone who has been doing Linux device driver development for about a year and gotten annoyed at the lack of kernel development tools, it's really nice to see this. Now if only Linus would make Andrea Arcangeli's Intergrated Kernel Debugger a part of the standard tree, it would make day.
--
This comment is (©) Copyright Deepak Saxena.
Deepak Saxena
"Computers are useless, they can only give you answers" - Picasso
This is a good thing, but it is part of a more general problem.
And that problem is that we accept tools for Linux development that are distinctly sub par. There is a lot that could, and should, be done.
I would say more, but I cannot possibly say it better than this rant does.
Cheers,
Ben
PS The Microsoft program works right and has a bad interface, the Linux program has a nice interface but sucks! Whodathunkit? (Read the link.)
My usual seat in the cluetrain is at A HREF="http://pub4.ezboard.com/biwethey.ht
Having done just a very quick glance over the specs I may be wrong, but I believe they are doing what they have been doing on the SGI for awhile. When a SGI running a newer flavor of IRIX does a system panic (SCSI, memory, whatever) it dumps a core out. Dumping this file is not for the drivespace week, if you have half a gig or ram you have a half a gig core file, but the beauty of this is it then automatically examines the core file and tries to figure out what killed it, you don't have to go in and run the debugger yourself.
Having the machine tell you what memory page you were at when it took a dive makes life much nicer for the harried admin; of course if you want to dig through a core at a later time with your debugger you can but it gives you a good starting point, and tends to make tracking things down much quicker since you have a guess as to where the problem resides. Having your box tell you that you had a memory error in SIM 3 bringing the box down, having analysed the core file before you even have a chance to fire up your debugger, is a pretty nice thing.
Of course this is dependant upon my assumption that it works in the same kind of fashion as Irix (which it seems to).
Actually in the support directory on the CD's are both the kernal symbols and i386 kd. Now there's not a lot of documentation on this on the Cd, but if you buy the Book Inside Windows NT you will get an introduction to the kernal debugger. (Also some of this information is in the Device Driver kit). If you are supporting NT, you need at the Platform SDK and the DDK kit.
But a fat lot of good a kernel debugger does you on a closed-source OS.
NT had the future almost in its grasp, but let it slip away by being impossibly unreliable and horribly admin-unfriendly compared to any Unix product. [We worked with it for a year but eventually had to discard it as a worthless toy.)
But that was then. Now it's just plain obsolete. Face it.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
-- Abigail
Maybe you're an unusually strong source of gamma radiation :-)
Choice of masters is not freedom.
NT saves it in swap as long as the swap partition is on the same drive as the boot partition. Again, RTFM. The serial port option is only an option and is NOT the default or usual manner for memory dumps.
As someone else in this thread commented, the dump is of the entire contents of memory. This changes in Win2000, but I have not personally seen this.
_damnit_
_damnit_
It's my job to freeze you. -- Logan's Run
Look into the KDE screensaver.
the BSOD is alive and well on my FreeBSD desktop.
-bugg
hm. If I was granted moderator points more often than once every 9 months, I'd moderate you up.
This is JUST what I'm looking for, well, it answers most of my complaints. Unfortunately, it does seem to be W2K only, not NT 4, which is likely to represent 99% of my install base for well past 12 months. (I seriously doubt that there will be any significant migration to W2K until this time next year. Oh, there will be a few early adopters, but among MY customers - almost nobody plans on putting it into production).
I wish I had a nickel for every time someone said "Information wants to be free".
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
>NT has had this for years, where have you been?
... yes.
running mission critical apps on unix with fantastic uptime.
>Hiding under the Unix rock?
if that means remotely administering and monitoring hundreds of multiuser servers from your office, then
>if more Unix people would actually try NT...
sorry, we have,we did, and finally our managers have stopped making us try to deploy on it... I developed a deathly serious allregic reaction to BSOD and rebooting, and I'm not going to try windows again until it includes a decent shell like bash, ksh, tcsh and an xserver.
We actually ran into very few crashes, so we added code to the kernel to crash it for us with user level commands. The code for crashing the kernel is listed on the LKCD FAQ page: http://oss.sgi.com/projects/lkcd/faq.html
http://softick.8m.com/
Possibly others would work as well. Check out http://www.suddendischarge.com/debugg ers.html for just about every (free/shareware) debugger ever made.
Well, ext3 is fairly simply, on disk it is ext2 with a log file (its self a regular file), reiserfs is coming along. What goes in may not be feature complete, but from the word on the reiserfs mailing list is that they both will make it into 2.4, and should be in 2.3 fairly soon. Perhaps they will be flagged experimental in the early 2.4 kernels.
Plato seems wrong to me today
> What exactly are you supposed to do with a kernel core dump under a closed source OS?
Figure out what application was running when your system hung, tell your support provider, and get them to fix it.
This is great.
I think SGI is going to more for linux than most people expect. They are helping us move into the Enterprise so much faster than I ever thought possible. You should look at their web page and see all the code they have contributed, it is very nice. SGI may be strugling, but they have a large cash reserve, and are staking their existance on Linux.
I hope they succed and will personally see that I get as many SGI servers around here as possible
geach
Sorry to disappoint you, but not only are you not first, but the previous _six_ posts (a new record for Slashdot!) are not lame "First Post" posts.
Glückwünsche, haben Sie Slashdot ermordet, indem Sie zum korporativen Druck beugten und Subskriptionen einlei
I think that this was one of the greatest features of Novell - the fact that if your server was barfing, you could go into the debugger, and neuter an offending process; or if the server was really in trouble, it would drop into the debugger, so you could at least figure out what went wrong, or dump the memory image and send it to someone who could.
And also, it's one of the things I really, really, really HATE about NT. No debugger comes with the OS, and there's no free, distributable one out there, so from a tech support standpoint, if your customer's server barfs, you kind of have to guess at what went wrong, or establish a pattern from multiple calls, or try to reproduce it in-house. Switching from supporting Netware products to NT products has been hell, and this is 90% of the reason. This kind of thing in Linux can only help "the cause". (and because my company is working on some fairly significant Linux products, and I may end up supporting them, this makes me more optimistic about the future.)
I wish I had a nickel for every time someone said "Information wants to be free".
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
this guy probably running a script to post 'FIRST POST!!!' every time. or just plain _cOUgH_eGGhEad_cOUgH_.
--
You're a cartoon of rebel! You're all like exaggerated version of yourself! - Gerard Jones
Being able to view a dump of the memory at the state it was in when the crash occured is an invaluable piece of data for any developer, and/or support-type person.
It makes the PTF/patch process go so much more easier. Of course, that's when the stack in the dump hasn't been corrupted by "unexpected" behaviour. Then, all bets are off.
BTW, does anyone know if they have any tools tailored to viewing these dumps and being able to quickly navigate through the stack, popping/pushing when needed? That would be nice too, but I never noticed anything from the announcement, and I don't have a Linux system handy at the moment to check the package.
> After all, ROM wasn't built in a day. :-)
Wich ROM? 8)
I have the strong feeling that some ROMs of my hardware were coded in less than a single day...
You are mistaken. NetApp boxes do *N*O*T* run any flavor of BSD as their OS.
The underlying kernel is one written at NetApp; it doesn't support multiple address spaces, any notion of userland, or demand paging (heck, until recently, it didn't even change the page tables; it now uses the paging hardware, but only to make virtually contiguous physically-discontiguous pages, to make allocation of large chunks of memory a bit less painful).
A significant part of the of the code did from BSD - the networking stack came from BSD (4.4-Lite, with some bits of the FreeBSD and NetBSD stacks thrown in), as did many of the commands (although those had to be chainsawed a bit to run in kernel mode in a shared address space), as did the dump and restore code (although the dump code was significantly changed to work with our WAFL file system). Various support routines also came from various BSDs as well, and the NFS server code is somewhat remotely derived from the BSD code (although it was also significantly changed to fit into our environment as well).
However, that doesn't mean NetApp boxes run anything you'd recognize as "BSD" (and, in particular, the crash-dumping code isn't BSD-derived, although the savecore command is based on the BSD command, although, again, significantly modified to run in our environment, and to extract the core dump information from the core dump areas on the disks).
(Yes, I know this first hand. I'm one of the developers there, and have been since early 1994.)
You must be joking.
:)
Hey, GUYS!
THIS IS A SYSTEM BASED ON GNU TOOLS!
YOU HAVE A DEBUGGER WHICH HAS SPECIAL HOOKS FOR DEBUGGING KERNEL CORE DUMPS!
This is *crazy*! That's like, uhm, like sort of a hack perpetrated by someone who was in a hurry and didn't know about prior art.
Which, I guess, is the allegation that Linux always faces from people. *sigh*. Oh, well, it'll get better.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
send flames > /dev/null
Only 'flamers' flame!
The most recent Linux 2.3.25 kernel does not have ext3. ext3 is still way alpha. Linux 2.3 is already under feature freeze. If Linus plans to release Linux 2.4 by 2000 Q1, I doubt ext3 will be part of it.
cpeterso
People here are saying that yes, even NT has the ability to dump kernel core when it BSODs, but:
What exactly are you supposed to do with a kernel core dump under a closed source OS? Throw a printout of it into a bonfire to propitiate the Windows Demons? Send it to Microsoft and wait for their rigorous QA process to leap into action and send you a fixed kernel? I can't imagine trying to debug it yourself without being able to get a backtrace and look at the problem source code. Does Microsoft even leave a symbol table of internal function names in the NT kernel? What exactly do you do with a Kernel Debugger in Solaris if you can't see anything more than what a disassembler will tell you about the kernel being debugged?
One other point to note: there are a number of features still to add to reduce the size of the memory image and to speed up the dumping process. We're also looking to make NMIs work properly under Linux. Stay tuned.
--Matt
I was just thinking the other day that running Linux in a VM in Linux would be handy for, among other things, somewhat more secure network services (Mail, web services, etc.) Run your server in a virtual machine and who cares if it gets cracked? Just wipe it clean and reinstall (Once you get it the way you like it, you could write the image to CD and just restore from CD every time you reboot.)
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
The core files you're seeing save the segment of memory in which the program was running. They can be used in conjunction with a debugger and image with debugging information to recreate the state of the application when it crashed, enabling the programmer to glean information about which instruction caused the crash.
Dumping the kernel on a crash is not new but it is useful, in much the same way.
Under HP-UX, as far as I remember, when the kernel crashes it is dumped into the swap device starting backwards from the end of swap. One of the first actions of the boot sequence (and boy can that take a long time) is to check whether there is a kernel image written in swap. If so, it's copied out and can be sent back to the kernel team for investigation.
Of course, if your boot sequence doesn't copy out the kernel, you've got a finite time to get it out yourself before it's overwritten by the ever-advancing swap data.
-John
http://oss.sgi.com/projects/kgdb/
This makes very little sense. The purpose of the kernel is to be the thing which allocates resources. If it screwed up, how do you "recover"?
Do both kernels have access to the serial ports? Does only one? If it's only one, how do you guess what state the port is in when it dies and the other takes over? If it's both, how do you keep them from conflicting?
It turns out that the program which keeps the kernels from conflicting is, in fact, the kernel.
This mind is not buddha.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
I think that this is excellent news. And more so for Linux than any other system.
It is crucially important that a community project like Linux have good debugging tools, both from the perspective of quality control, and to encourage others to get involved in the community.
Other systems that are open but don't actively encourage contributions, or worse yet are closed - well, these debuggers are usefull in the sense that it helps pin point a problem. But in many cases you don't have control of the source code, so there isn't much you can do except mail it to the developers. If they even have a place to mail it to.
=======================
LCRASH CORE FILE REPORT
=======================
GENERATED ON:
Thu Nov 4 19:15:19 1999
TIME OF CRASH:
Fri Nov 5 03:12:27 1999
PANIC STRING:
User created crash dump
MAP:
map.5
VMDUMP:
vmdump.5
================
COREFILE SUMMARY
================
The system died due to a software failure.
===================
UTSNAME INFORMATION
===================
sysname : Linux
nodename : peak-pc.engr.sgi.com
release : 2.2.13
version : #1 SMP Fri Nov 5 02:59:34 PST 1999
machine : i686
domainname : engr.sgi.com
===============
LOG BUFFER DUMP
===============
Linux version 2.2.13 (root@peak-pc.engr.sgi.com) (gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #1 SMP Fri Nov 5 02:59:34 PST 1999
mapped APIC to ffffe000 (0026f000)
mapped IOAPIC to ffffd000 (00270000)
Detected 348940216 Hz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 348.16 BogoMIPS
Memory: 95448k/98304k available (1100k kernel code, 424k reserved, 1268k data, 64k init)
Checking 386/387 coupling... OK, FPU using exception 16 error reporting.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
per-CPU timeslice cutoff: 100.26 usecs.
CPU0: Intel Pentium II (Deschutes) stepping 02
SMP motherboard not detected. Using dummy APIC emulation.
PCI: PCI BIOS revision 2.10 entry at 0xfcaee
PCI: Using configuration type 1
PCI: Probing PCI hardware
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Unix domain sockets 1.0 for Linux NET4.0.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
Starting kswapd v 1.5
Detected PS/2 Mouse Port.
Serial driver version 4.27 with no serial options enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
pty: 256 Unix98 ptys configured
PIIX4: IDE controller on PCI bus 00 dev 39
PIIX4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
hda: WDC AC24300L, ATA DISK drive
hdc: NEC CD-ROM DRIVE:28C, ATAPI CDROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: WDC AC24300L, 4112MB w/256kB Cache, CHS=524/255/63, UDMA
hdc: ATAPI 32X CD-ROM drive, 128kB Cache
Uniform CDROM driver Revision: 2.56
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
(scsi0) found at PCI 14/0
(scsi0) Narrow Channel, SCSI ID=7, 3/255 SCBs
(scsi0) Warning - detected auto-termination
(scsi0) Please verify driver detected settings are correct.
(scsi0) If not, then please properly set the device termination
(scsi0) in the Adaptec SCSI BIOS by hitting CTRL-A when prompted
(scsi0) during machine bootup.
(scsi0) Cables present (Int-50 YES, Ext-50 NO)
(scsi0) Downloading sequencer code... 413 instructions downloaded
scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.20/3.2.4
scsi : 1 host.
(scsi0:0:6:0) Synchronous at 20.0 Mbyte/sec, offset 15.
Vendor: IBM Model: DDRS-34560 Rev: S97B
Type: Direct-Access ANSI SCSI revision: 02
Detected scsi disk sda at scsi0, channel 0, id 6, lun 0
scsi : detected 1 SCSI disk total.
SCSI device sda: hdwr sector= 512 bytes. Sectors= 8925000 [4357 MB] [4.4 GB]
3c59x.c:v0.99H 11/17/98 Donald Becker http://cesdis.gsfc.nasa.gov/linux/drivers/vortex.
eth0: 3Com 3c905B Cyclone 100baseTx at 0xdc00, 00:c0:4f:90:6e:54, IRQ 11
8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
MII transceiver found at address 24, status 786d.
MII transceiver found at address 0, status 786d.
Enabling bus-master transmits and whole-frame receives.
Partition check:
sda: sda1 sda2 sda3
hda: hda1 hda2
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 64k freed
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
dump_open(): dump device opened: 0x803 [sd(8,3)]
Adding Swap: 130748k swap-space (priority -1)
Adding Swap: 130748k swap-space (priority -2)
Kernel panic: User created crash dump
Dumping to device 0x803 [sd(8,3)]
Writing dump header
Writing dump pages
====================
CURRENT SYSTEM TASKS
====================
ADDR UID PID PPID STATE PRI FLAGS MM NAME
===============================================
c0234000 0 0 0 0 0 0 c0215320 swapper
c5ffa000 0 1 0 1 20 100 c5fb4060 init
c5fe8000 0 2 1 1 20 40 c0215320 kflushd
c5fe6000 0 3 1 1 20 40 c0215320 kupdate
c5fe4000 0 4 1 1 20 840 c0215320 kpiod
c5fe2000 0 5 1 1 20 840 c0215320 kswapd
c59ec000 1 248 1 1 20 140 c5fb4260 portmap
c5686000 0 263 1 1 20 140 c5fb4460 ypbind
c578c000 0 270 263 1 20 140 c5fb44e0 ypbind
c5644000 0 324 1 1 20 140 c5fb42e0 syslogd
c5602000 0 335 1 1 20 140 c5fb43e0 klogd
c55c4000 0 349 1 1 20 40 c5fb4560 atd
c5c3c000 0 363 1 1 20 40 c5fb41e0 crond
c55a2000 0 381 1 1 20 140 c5fb45e0 inetd
c5518000 0 395 1 1 20 140 c5fb4660 snmpd
c5348000 0 409 1 1 20 40 c5fb46e0 named
c52fe000 0 423 1 1 20 140 c5fb4760 routed
c5272000 0 437 1 1 20 140 c5fb47e0 xntpd
c523e000 0 451 1 1 20 140 c5fb4860 lpd
c51e4000 0 469 1 1 20 140 c5fb48e0 rpc.statd
c5194000 0 480 1 1 20 40 c5fb4960 rpc.rquotad
c5174000 0 491 1 1 20 40 c5fb49e0 rpc.mountd
c5158000 0 515 1 1 20 140 c5fb4ae0 rpc.rstatd
c513e000 0 529 1 1 20 140 c5fb4a60 rpc.rusersd
c511e000 99 543 1 1 20 40 c5fb4b60 rpc.rwalld
c50f6000 0 557 1 1 20 140 c5fb4be0 rwhod
c513c000 0 577 1 1 20 140 c5fb4360 rpc.yppasswdd
c5078000 0 589 1 1 20 140 c5fb4ce0 amd
c5086000 0 591 1 1 20 40 c0215320 rpciod
c504a000 0 592 1 1 20 40 c0215320 lockd
c4f54000 0 626 1 1 20 140 c5fb4de0 sendmail
c4f22000 0 641 1 1 20 140 c5fb4d60 gpm
c4e12000 0 655 1 1 20 140 c5fb4e60 httpd
c4e0a000 99 658 655 1 20 140 c5fb4ee0 httpd
c4e06000 99 659 655 1 20 140 c5fb4f60 httpd
c4dfc000 99 660 655 1 20 140 c4dfe040 httpd
c4df2000 99 661 655 1 20 140 c4dfe0c0 httpd
c4de8000 99 662 655 1 20 140 c4dfe140 httpd
c4dde000 99 663 655 1 20 140 c4dfe1c0 httpd
c4dd4000 99 664 655 1 20 140 c4dfe240 httpd
c4dcc000 99 665 655 1 20 140 c4dfe2c0 httpd
c4dc0000 99 666 655 1 20 140 c4dfe340 httpd
c4db6000 99 667 655 1 20 140 c4dfe3c0 httpd
c4a28000 0 699 1 1 20 140 c4dfe540 smbd
c499e000 0 710 1 1 20 140 c4dfe4c0 nmbd
c4658000 9 767 1 1 20 40 c4dfe840 actived
c48a2000 0 806 1 1 20 100 c5fb4c60 mingetty
c4928000 0 807 1 1 20 100 c5fb4160 mingetty
c4904000 0 808 1 1 20 100 c4dfe7c0 mingetty
c498a000 0 809 1 1 20 100 c4dfe440 mingetty
c4766000 0 810 1 1 20 100 c4dfe5c0 mingetty
c4c6e000 0 811 1 1 20 100 c4dfe640 mingetty
c479e000 0 812 1 1 20 100 c4dfe8c0 getty
c4798000 0 817 381 1 20 100 c5fb40e0 in.rlogind
c4976000 0 818 817 1 20 100 c4dfe740 login
c45b8000 0 819 818 1 20 100 c4dfe6c0 tcsh
c5204000 0 838 819 0 20 0 c4dfe940 crashdump
===========================
STACK TRACE OF FAILING TASK
===========================
===============================================
STACK TRACE FOR TASK: 0xc5204000 (crashdump)
0 __dump_execute+153 [0xc010da21]
1 dump_execute+149 [0xc011b925]
2 panic+167 [0xc0114b6f]
3 sys_setpriority+25 [0xc0115689]
4 system_call+45 [0xc0107a61]
===============================================
In many Commercial UNIX systems, the proc filesystem isn't as broad as that of linux and the low-level tuning parameters are configured using the Kernel debugger. In some cases, the kernel debugger is the general program debugger with the ability to traverse /dev/kmem - and some functions to manipulate the appropriate data structures.
Debuggers aren't only used for debugging.
But as it is, what's the point. Has anyone ever actually SEEN a kernel panic?
So if you play with a x.(2y+1).z kernel while rubbing your feet on the carpet and a lightning rod attached to an ISA slot, then this is for you. If you only use a x.(2y).z kernel with z>2, then this'll probably do nothing more than occupy disk space.
Christopher A. Bohn
cb
Oooh! What does this button do!?
Just about every other OS I know of (except for NT) includes this. Having a Kernel Debugger, Kernel Core Dump, and a few other tools available over the past few *YEARS* has saved me a lot of hassle. If Linux hasn't had this till now, I'm sooooooooooo sorry. Thats really dissapointing.
*BSD, Solaris, Dynix, and bazillions of other OS'es have had this ever since they were created.
This sounds like just about the coolest software utility that I will almost never have to use!
A Windows version would be orders of magnatude more useful!
What happens when the LKCDA crashes during a system crash? Who recovers from that??
Can your IM do this?
Forget knowing why my system crashed, I just want my BSOD. They better have someone working on bringing that to Linux...
This sounds like just about the coolest software utility that I will almost never have to use! [Grin]
A Windows version would be orders of magnatude more useful! [Bigger grin!]
OK, I could see as how it might help the developers... ;-)
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
This will be tremendously usefull for us device driver writers, and all other breeds of kernel-hackers. True, Linux rarely crashes under normal use, but when your code is running with the kernel and you make a mistake... OOPS!
For some examples, see my PIC page.
There's a kernel debugger, right? Hit a special key sequence and the whole system stops, so you can look around at data structures? Is there? Someone please say yes...
I'm sorry, but this made me laugh out-loud.
I too am curious if there is some sort of kdb under linux. This sort of thing was avaliable for SCO when I was doing driver maintenance for it at my last job (but I barely knew how to use a kernel debugger at that time). I've been sysadmining IRIX at my current job, and I've been looking for a kernel debugger and the key sequence to get into it, but haven't found it yet. (Admittedly, I haven't been looking too hard, but I'm curious nonetheless).
Catcha' later,
Paul.
Take a closer look at that icrash report, dude. The test system was running IRIX, not Linux. Sorry to disappoint.
Catcha' later,
Paul.
Well, as a data point, BSD/OS (www.bsdi.com) is arguably "closed source", and gives out crash dumps.
I'm in the support group, and we find crash dumps *very* valuable. It is not necessary for the customer to necessarily have all the source, just a kernel with known characteristics...
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
Then again, I pulled the IDE cable out of the drive when it was running. I bet that dident help the situation. Other then that, a 2.1.132 kernel mysteriously locked up on me, but that was a develoment kernel, and the machine had a 70+ day uptime. Also, I tried one of the latest 2.3.x kernels, and it crashed on me detecting the USB controler, but then I expected that :)
How is this "resource crunching"? I agree Linus may have a problem with having this in the kernel, but certainly not because it consumes resources.
...
There's only one resource it currently "consumes", and that's about 64K of decicated memory. We're looking to eliminate that requirement in the next release.
I guess the real test will be asking Linus
--Matt
Uh, yes at least 4 times in the past 6 months.
"Oh, you BSODed? Did you install Service Pack 5?"
"You did. Well, what applications were you running?"
"Just a Q3Test server? Okay, send me the core dump and I'll check it out." (yeah right!)
Few minutes go by
"Okay, we've thoroughly analysed the data. It seems that NT crashed."
"Oh, you know it crashed? Yeah, right! You told me it crashed. Forgot about that. Hmmmmm... Have you tried rebooting? That usually works."
"It works now? Great. That'll be $3500 payable to Microsoft Inc."
"You want to know why NT crashed? I'm sorry, we cannot do that. That is proprietary information. We seem to have solved your problem, however. Your computer seems to be working fine. We expect payment within two weeks. Have a nice day."
"Evil will always triumph over good, because good is dumb." - Dark Helmet (Spaceballs)
>So did you take a cut in pay..
no the managers got re-orged or got the boot.
>http://www.interix.com
fist off, interix or exceed aren't free, and by the time I get exceed approved for 20 users, I could have upgraded them to linux.
all the interix stuff makes it painfully clear that Micros~1 never intended to be interoperable with unix, and didn't want file or app sharing. Jeez. look at how ultimately feeble the standard windows telnet is. Look at the lack of scripting in command.com and cmd.exe...
I'm pretty underwhelmed with interix and most PC xservers to be honest.. I'd be happy if the windows desktop supported x apps transparently and I could "xhost +backup1 server2 mail4 dev344" from a dos shell, telnet to a unix server from that dos shell without spawning a "telnet" windows app, set my display back and launch an app. -- oh, but that's exactly the level of interoperability that Micros~1 never wanted to achieve. Actually, I want to be able to use a nis/nis+ server for my NT logins without a hack.
When Micros~1 find someone who can implement all that functionality in a shipping version of windows as standard features, I'll consider revising my opinion.
xservers on windows: like putting lipstick on a pig.
--
--
This sounds *EXACTLY* like the way BSD kernels have, since the dawn of time, handled panics. If you have enough swap space, the kernel dumps a complete core image (in a special format) to the swap device. Then, on boot, it extracts it before enabling swap, and copies a kernel over. (Goes in /var/crash, if such a place exists.)
I've used this to debug (or have someone else debug) kernel panics on BSD/OS and NetBSD systems. It's a *very* nice feature, because, in the real world, you often have a crash that can't be encouraged to happen right when the engineer is handy.
Common feature, been available for years. I just *assumed* Linux had it.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
Wow - Karma = -30?? Not exactly adding intelligent insight to stories... On a related note, would Rob or whoever be willing to block his IP address from commenting or something? I mean, with 22 comments and nothing but "First Post"...would people be willing to support this? Just curious...
Windows NT 4.0 does a very similar thing when the appropriate options are checked in the Startup and Recovery options under the System object in Control Panel. Problem is...it's the ENTIRE memory space. If you have 128 MB of RAM, you better have 128 MB of swapfile space on the system partition. Not a smart thing to do when the boot partition (where the \WINNT directory is) resides in the same partition...the hard disk has to constantly shuffle btw the swap file and the \WINNT directory. If you place the swap file on a different partition (i.e. optimize for I/O speed), the crash dump file (memory.dmp) is not created when NT bluescreens. This particular thing that SGI's doing is a MUCH smarter way of going about it. Though one of the coolest things about Win2K is the fact that you can choose btw a full mem dump, a kernel mem dump, and a 64K minidump. That's a Good Thing for those of us who like to optimize our swap file and move it to a different partition or split it up a bit.
However...sifting through that crap with the dumpchk.exe and dumpexam.exe utilities is akin to getting your teeth pulled...:)
Another nifty thing NT has is the ability to t-shoot a box by hooking up another NT box to it thru the serial port (or remotely, with a modem) and, by using the symbol files, find out EXACTLY where in the OS code a particular process is failing, because when NT bluescreens, it's not really crashed...the kernel is still spinning happily away churning out that dump file. That ain't too bad, but it's a bitch to set up.
I prefer just to decipher the bluescreen and find out which piece of shit hardware (or driver) is causing the failure...:)
-Kevin Bunn, MCSE/MCT - MCP ID # 1198191
PS: Yes, you heard it correctly. The way NT does it, the BOOT partition is where the system files (i.e. \WINNT) are and the SYSTEM partition is where the boot files (i.e. boot.ini, ntldr, and ntdetect.com) are. Another weird MS-ism...:P
My posts don't reflect the opinion of my employer, and my employer's opinion doesn't influence the content of my posts.