Linux Developers Consider On-Screen QR Codes For Kernel Panics

An anonymous reader writes "Linux kernel developers are currently evaluating the possibility of using QR codes to display kernel oops/panic messages. Right now a lot of text is dumped to the screen when a kernel oops occurs, most of which isn't easily archivable by normal Linux end-users. With QR codes as Linux oops messages, a smart-phone could capture the display and either report the error string or redirect them to an error page on Kernel.org. The idea of using QR codes within the Linux kernel is still being discussed by upstream developers."

Comments

Good idea

[Primate Pete]

I'm not sure how hard it would be to pull this off in practice, but kudos to the team for improving (or at least thinking about) better usability from the kernel out.

Re:Good idea

[Kjella]

Very unlikely.. the information in a QR code is probably just enough to say "I run kernel X (build Y) and it crashed with error code Z at instruction 12345 in module 123", if it was a kernel dump that's different but I have seen these without the QR codes and there's nothing sensitive there.

Re:Good idea

[Zocalo]

It might actually be more than that. Worst case, the screen in in 80x25 text mode (assuming a PC), which gives 2,000 binary bits, but if you start playing around with extended ASCII graphics characters you could probably encode a KB of data quite easily. Hardly a crash dump, but easily enough to get across the essentials.

Re:Good idea

[Levex]

We are encoding the full Oops, i.e. from the "cut here" to the "end trace" marker. Classic won't ever go away, and we had already created a configuration option called CONFIG_QR_OOPS that can disable this at all. In case your distro or you had compiled it in and you don't want to have QR codes on your screen, I just added a new kernel parameter currently called 'qr_oops', which can as well disable it.

Re:Good idea

[Pinhedd]

Kernel crashes occur when the kernel enters an inconsistent or invalid state from which it cannot recover.

When a user program fails, the kernel maintains consistency, can cleanly terminate the process, and can accurately report the cause of the failure if need be (illegal instruction, deadlock, access violation, etc...).

When a kernel fails the very systems that it relies on to report failures may very well be compromised by whatever caused the kernel to fail in the first place. As such, any kernel fault reporting needs to be incredibly robust and as independent of other kernel mechanisms as possible. Dumping text to a serial terminal is the preferred method because it's incredibly simple and relies on nothing else, meaning that barring a failure of the system memory it should always act as a reliable fallback.

Dumping kernel memory to a disk might fail if the state of the file system is compromised, if the storage controller is compromised, or if any number of intermediary systems are compromised by the inconsistent state of the kernel. Many operating systems do attempt to dump crash memory to the swap file / swap partition as this is less likely to cause data corruption than writing to a particular file in the file system.

It "can" be done, but that does not necessarily make it a good idea.

Re:Good idea

[AmiMoJo]

You usually don't want to write to the filesystem in the event of a kernel panic. It could make things worse and corrupt it. Once you kernel panic you are basically screwed and can't rely on any services beyond really low level BIOS stuff to work. Poking some text to the screen buffer is about it.

Windows does core dumps using a specially reserved area of the boot drive and using low level boot driver calls. It can still fail but at least has a fairly low probability of damaging the filesystem further. I suppose Linux could maybe dump to the swap partition or something.

Re:Good idea

[Lemming Mark]

As AmiMoJo also noted, when you have a kernel panic all bets are off regarding which parts of the kernel are OK. If the behaviour of the disk driver or filesystem have been affected, it could damage your filesystem to try to write a kernel dump into a normal disk partition. It might work but it does seem a good idea to be properly paranoid. I didn't know that Windows uses a special reserved area of the boot drive - that does make sense as a solution!

There have been various systems for crash dumping under Linux, though. I think the de-facto solution (the one that was accepted by the kernel devs) ended up being kdump, which is based on kexec (kexec is "boot directly to a new kernel from an old kernel, without a reboot"). This allows full crash dumps with (hopefully) decent safety, so it is possible to do this if configured.

In kdump, you have a "spare" kernel loaded in some reserved memory and waiting to execute. When the primary kernel panics it will (if possible) begin executing the dump kernel, which is (hopefully) able to reinitialise the hardware and filesystem drivers, then write out the rest of memory to disk. I'm not sure how protected kdump's kernel is from whatever trashed the "main" kernel but there are things that would help - for instance, if they map its memory read only (or even keep it unmapped) so that somebody's buffer overflow can't just scribble on it during the crash.

Obviously, having a full kernel available to do the crashdump makes it easier to do other clever tricks, in principle - such as writing the dump out to a server on the network. That's not new, in that there used to be a kernel patch allowing a panicked kernel directly to write out a dump to network, it just seems easier to do it the kdump way, with a whole fresh kernel. Having a fully-working kernel, rather than one which is trying to restrict its behaviour, means you can rely on more kernel services - and probably just write your dumper code as a userspace program! Having just installed system-config-kdump on Fedora 20, I see that there's an option to dump to NFS, or to an SSH-able server - the latter would never be sanely doable from within the kernel but pretty easy from userspace.

Various distros do support kdump. I think it's often not enabled by default and does require a (comparatively small) amount of reserved RAM. So that's some motivation for basic QR code tracebacks. I suppose another reason is if they expect they can mostly decipher what happened from a traceback, without the dump being necessary - plus, with a bug report you can easily C&P a traceback.

This discussion has just inspired me to install the tools, so maybe I'll find out what it's like...

Re:Huh?

[ledow]

You lose nothing.

Anything that could have been logged to disk will have been.

Anything that couldn't is probably FAR TOO LONG to even start taking down any other way and almost certainly will cut through the screen buffer limit anyway (every kernel panic I've had - which is about a dozen I think - was like that).

Let's compare and contrast to, say, Windows. Bluescreen with minidump and error code that has 7 million potential causes.

At least with a QR code, for those totally undumpable errors, you stand half a chance of snapping it and providing several kiloybytes of useful information for someone to work from - that they know hasn't been transcribed wrongly. And can be taken from even a completely hung machine.

It's a good idea. Someone needs to make a patch for it. The biggest problem - as always - will be making sure you can get to the point that you can write to the video memory and do so with enough processing / storage to be able to write something useful into the QR code.

Re:Dump kernel to serial printer

[smittyoneeach]

You gonna need a big bowl to catch all them corn flakes, mister.

Re:Wish other OSs did this...

[Anonymous Coward]

And with QR codes, the conversation becomes this:

"My computer froze."
"What happened?"
"It put some white and black crap on the screen."
"What did it say?"
"How the fuck should I know? It was random white and black dots! Like a fucking Rorschach test!"
"It probably was a kernel panic. What was the error?"
"I dunno, because like I said, ALL IT HAD WAS SOME DOTS AND SHIT. Then it rebooted! So it's gone! FUCK!"

How is that an improvement? Yes it's a change, but it's not an improvement.

Re:Huh?

[Guspaz]

QR codes use Reed-Solomon error correction, so you don't get missing or corrupt data (in that the QR reader knows if it reconstructed all the data correctly or not). Readers will typically only "read" the code if they manage to reconstruct the entire thing. The error correction helps compensate for poor image quality, and the fact that the image is monochrome makes things like exposure less critical. There are four levels of error correction, which allow for the reconstruction of 7%, 15%,25%, or 30% of codewords respectively.

QR codes can store up to a bit under 3KB of data (the largest size with the lowest error correction), but I couldn't get my phone to read any v40 QR codes (the largest ones), and v25 took some effort. The plan for QR codes of kernel oopses will probably fail for that reason, if nothing else (that they need v40 codes to store an entire oops, and few phones will read v40 codes).

No way!

[msobkow]

I am NOT buying a fucking cell phone to read a core dump.

Just fuck right off already. Not everyone wants a digital leash.

The matrix

[BlazingATrail]

I prefer all my BSOD, crashes and core dumps to use the Matrix dripping green characters and pixel crap method of reporting errors. It's easier to see the patterns. Guru meditation # 42

Re:Dump kernel to serial printer

[flyingfsck]

Bah. Punch cards are so much better. You young, know nothing, whipper snappers with your newfangled hoosammawhatsits...

Just show smileys

[Anonymous Coward]

Linux must be ubuntufied. We need to hide everything because it's way to complicated for the common user or his dog. We need more splash-screens to hide all the stuff that makes no sense anyway. Who want's to know if a module didn't get loaded? As a matter of fact, we should remove unnecessary logs (like message, dmesg, audit), because nobody gives a rats ass. Also: Why have a console? Or init-mode 3 ? People want the graphical stuff, let's get rid of all the ballast like command-line. Those few people still using ancient tools like 'make', 'vi' or (o my god) 'ifconfig' should go and find themselves something else to brag with. Linux MUST go mainstream.

Re:Wish other OSs did this...

[Anonymous Coward]

I doubt the kernel developer that implements this would forget to put the message

"Make a photo of this black-and-white dots and send it to crash@kernel.org so we can try to figure out what happened. Thanks for making the Linux kernel better!"

at the top of the black and white dots.