Slashdot Mirror
NetBSD To Support Kernel Development In Lua Scripting
An anonymous reader writes "NetBSD 7.0 will support the Lua scripting language within its kernel for developing drivers and new sub-systems. A Lua scripting interpreter is being added to the NetBSD kernel along with a kernel API so developers can use this scripting language rather than C for developing new BSD kernel components. Expressed reasons for supporting a scripting language in a kernel were rapid application development, better configuration, and "modifying software written in C is hard for users." In a presentation it was said that Lua in the kernel will let users explore their system in an easy way."
This is, quite possibly, the stupidest kernel feature that has ever been unleashed upon this poor planet.
Surely they are joking, right?
"modifying software written in C is hard for users."
i thought *users* had no business writing device drivers?
of course if they had said "modifying software written in C is hard for programmers." it would gather a fierce shitstorm.
Looking for people to chat about multicopters, coding, music. skype: gtsiros
Won't that kill performance?
I mean that's the reason we still write kernels in C. Every cycle counts..
This is a good thing, and it's long overdue. Linux should do the same.
It's actually pretty easy to get Lua running in the kernel and call kernel APIs; but to make it really useful, it needs additional kernel hooks and callbacks, and that requires cooperation from the core kernel developers.
Apparently the groundhog is really off his game :D
And it gives me great confidence to install kernel components from developers who aren't able to master C.
On the one hand, well, wanting to reach out to "users" is purdy nice and all that, yanno. On the other hand, the consequences of running lua with kernel privileges, written by people not good enough to grok C pointers? I think they've gone off their rocker.
Then again, weird stuff's been done before in this realm. I recall something about SCSI drivers written in perl, until they got around to redo it in C. So hey, innovation and all that, let's give them a chance. We'll see if this stands. Though I do suspect it'll go down in flames before not all that long.
Why LUA? Why not implement BASIC while you're at it?
This is one of the most retarded ideas I have ever heard. I'm sorry, but kernel level stuff should be kept as close to the machine hardware as possible, and that means using C or assembly. Deal with it. If you can't be bothered to learn C like a real goddam programmer, then you have no fucking business sticking your head in something as complicated as an operating system kernel.
I swear to god, this mentality of "let's appease everyone, including the idiots who aren't smart enough to figure it out as it is right now" needs to stop. If you can't grasp C, then kernels are off limits to you. Either you lack the ambition to learn C, or you're too stupid to comprehend it- in which case you're going to be useless drifting around the kernel source code anyways.
So what the hell is wrong with saying "no, sorry, you're too stupid to understand this"?
I honestly think that this is a great idea. It's a novel approach to extending a kernel, especially considering NetBSD doesn't have that big of a market share and expressly focusses on supporting as many platforms as possible. Prototyping new features or certain drivers in a portable scripting language may give them a leg up in the functionality aspect of the race.
Also, LUA is super fast, small, easy to learn, concise and the C API/embedding it is straight forward.
You may knock it all you want, "because kernel land is C land", but after all bias being said and flamed, this may turn out to be a really fun, interesting and possibly very useful idea.
I have an old P4 with 512MB and FreeBSD with no desktop: I do everything from a terminal - old school. It runs wonderfully.
I would really hate to install a package that uses this Lua to extend the kernal (and god knows what security issues that will bring up!) and have that system slow down to the point where I can't use it.
See folks, I keep my hardware until the wheels fall of and then some. There's very little of MY old hardware polluting third world countries and I'm cheap - this buying the latest and greatest thing is just a waste of money.
If *BSD turns into something that I can't use on old hardware anymore, I'll be really sad.
Adobe had a lot more to do with the human rights abuses of the last decade than most people realize
I agree, celebrities are dressing like tramps these days for the paparazzi sites.
...there will finally be something Perl CAN'T do?
Okay, sure. But...why Lua? :/
For I cannot see any actual purpose for doing this. Just because you can run Lua script in the kernel does not mean that you SHOULD. And also, I could really not give a rat's ass that some script kiddie finds kernels too hard to mess with. (Awwww, poor widdle script kiddie...) There's a reason even I don't muck around in the kernel, and I do software dev in C and C++ amongst other compiled and scripted languages...
Please do NOT bring this type of feature into the Linux kernel.
This looks like a symptom of Zawinkski's Law: http://catb.org/jargon/html/Z/Zawinskis-Law.html
This is awesome. What is old is new again. Lisp machines were some of the most beloved equipment for os and ai hackers because of the ability to get to the underbelly easily. This is no different. Making the operating system more approachable makes doing operating systems research easier. This is a great idea.
Besides, it's a natural extension of the RUMP kernel paradigm. If you can run a netbsd kernel, as part of a userspace process on linux, then it only makes sense to be able to talk to these 'kernels' under the hood easily. THIS IS WAY FRIGGING COOL.
After playing Natural Selection 2 on the Spark engine, I'm not at all surprised more projects are taking up LUA.
NS2 is 95% LUA and runs incredibly fast for what its doing.
I dont know what packages are going to be bound with this, but in many cases your right back in C adding bindings for lua. Thats one of the good points about lua, is that it can be tied to anything, but the bare bones basic lua is not too much more complicated than a batch file
From TFA:
Rather than Lua, other language possibilities were Python and Java. Python is not difficult to integrate with C but has problems with being a huge library, memory consumption, and difficult object mapping. Java is easy to write but also poses problems with object mapping and memory might be an issue, but it's worked out before for driver development.
Surely this article deserves the whatcouldpossiblygowrong tag more than any other this year. And possibly last year too...
This is excellent imho because I will now be able to code NETBSD kernal extensions so I can run a Beowulf cluster of naked and petrified hot grits on my Natalie Portmen! All the while you people are complaining about the inefficiencies of using LUA in a kernal, you are forgetting about the thousands of people who have died from a terrorist attack! GET SOME PRIORITIES!
MediaWiki developers are almost ready for Lua scripting to be enabled for all Wikipedia and related sites, and It has already been deployed to http://mediawiki.org./ Lua was chosen mostly because of how easy it is to sandbox and limit memory consumption.
http://www.mediawiki.org/wiki/Lua_scripting/Tutorial -- Introduction
I like to think I'm okay at C. I certainly don't generally find it particularly intimidating. I've written kernel code, though not much. I maintain a program that emulates root privileges by intercepting syscalls -- and which works on Linux and OS X. I'm basically clear on how this works.
But when I do iOS/Android stuff, I have a pretty strong preference for Lua, because I don't always want to be thinking about pointers. And yes, LuaJIT is... plenty fast. The mere fact that it's a JIT implementation means that it can beat precompiled code for real use cases, and even when it doesn't, it's not as though it's particularly slow.
Don't assume this is only of interest to people who think C is hard. C's by far the language I'm best in, and I still see plenty of appeal here.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
I don't know how much Unix/BSD/Linus code you have read or written, but I have been developing core code, Scheduler, MM and Drivers since 1980 and this is a great idea, long overdue.
The Abstractions are now well understood and most code dos't need to be quick, but safe, putting PRE in kernels will clean up acres of cockamamie code now written in C, most of which just runs at boot time and is then freed. ... there are exceptions, you cant't have the garbage collector running whenever it likes, eg in the middle of an interrupt routine, but managed memory, not C, C++ scoping is the way to stop slow memory leaks.
Put simply you are an idiot and dont know what you are talking about.
MFG, omb
I mean, seriously, WTFF? Come back and see me when your kernel also allows LISP and OO-COBOL. I could use a good laugh.
The B5000 produce in the 1970s had I/D spaces, and ran pseudo Algol machine code, emulated by hardware, they had zero-days too.
Intel has had an execute permit bit in the page tables for 10 years, but both GCC and LLVM use thunks on the stack which is the prime space for ... PUT stuff on the stack and then EXECUTE it. Compilers, linkers and debuggers need to be able to alias code as data, think and reflect.
MFG, omb
With the BSD's, there no such thing as a tainted kernel. But with Linux, non-GPL drivers are actively discouraged. If they were to add scripting to the Linux kernel, it would actually make it easier to write proprietary drivers, so they'd never go for that. Forget the fact that it would make it a hell of a lot easier to sandbox drivers so they don't clobber other kernel memory. Forget the fact that code in a higher-level language is smaller; countless drivers would suffer no appreciable performance hit by being written in an interpreted language (especially Lua), so you'd have one small interpreter and many shrunken drivers, which would make the kernel memory footprint smaller. Forget the fact that mutually exclusive access to shared variables could be made implicit in a majority of cases, eliminating a whole class of bugs in one shot. No. We've got our precious GPL to protect. (Oh, and forget the fact that nVidia basically gets a free pass on this.)
There are multiple non-"C" DSL languages and virtual machines in the BSD kernels already and they have been there for over 20 years.
Here is a prime example http://en.wikipedia.org/wiki/Berkeley_Packet_Filter
Lua is a very clean, small, and relatively fast object oriented language. It is probably the best possible choice for a general purpose dynamic language to embed in a kernel.
I have one question. If the Japanese Ministry of Agriculture is not in charge of Gundam, then who is?
How about
peek(xx)
poke(xx)
The best aprt of Commodore BASIC
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
Why not just expose a driver API to user-space? Why should an entire interpreter run on ring zero?
We need less kernel code...anything that makes writing kernel code harder is good as people will think twice about doing it, punt to user space and the result will be a more stable platform.
Otherwise what I think kernels really need are DSLs to handle hardware and resource access patterns not new general purpose languages. With more or less everything hanging off a serial bus of one type or another these days there is huge opportunity to innovate.
Have spent enough years thumbing thru the change history of kernels to see the same types of problem crop up again and again in system after system, driver after driver. Its a crying shame. What we need is better architecture and less compartmentalization to the extent duplication of effort is minimized. Syntatical musical chairs does shit if your goal is real progress.
I'm in no way a kernel expert, I done some C and C++ programming.
But if you want to expose kernel API to scripting language, why you don't just expose those kernel API to userspace?
Anything performance or memory critical should be still written in C, so just expose those API to user space and not only get the benefit that you can now use not only LUA but any other language, you get the benefits of separation between kernel and user space: security.
Just expose those API to userspace and offer a libnetbsd.so so that for example, LUA, Python, Ruby, etc. can use the kernel API. Why would the Netbsd developers limit it to just LUA? Or are they planning to add more script engines in the kernel?
http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
why waste cpu cycles on interpreters? they don't make things 'safer'.. if anything they make the entire code base more complex than it was before, with a massive performance hit as a 'bonus.'
the last place I'd want one of these things is in a kernel, and I'd never consider a lua scripted device 'driver' as legitimate support for any hardware.
But if it doesn't need to be quick, why is it in the kernel in the first place? I would have thought that if you are tempted to put a scripting language in the kernel, it's a pretty good indication that the division between kernel and non kernel is somehow wrong.
That line of reasoning sounds like "All applications will be written in Python, therefor all applications will be written in Python. And since Python is not native code, it's a waste of CPU cycles. Therefor Python is a bad idea".
"possibility exists" != "everyone will do it all the time".
The lua features are for prototyping, playing around and just testing crazy ideas. Don't expect NetBSD to ship with any production drivers written in lua.
I use Python to prototype, but I always release applications and libraries written in plain C. I don't see why the existence of a feature must imply that it's used everywhere?
http://benchmarksgame.alioth.debian.org/u32/benchmark.php?test=all&lang=perl&lang2=lua
In some tests, one is faster than the other, but by no more than 4x to 1/9th each way.
At least they didnt choose ruby.
Liberty freedom are no1, not dicks in suits.
*BSD and MIT-licensed packages like Lua can be relicensed under GPL* if necessary. It causes some friction between FOSS teams when that is done (dear Theo is especially prone to blowing his top over it), but it's perfectly legal under the terms of these permissive licenses. It's what makes them permissive.
After all, you can even incorporate BSD/MIT code into proprietary software and it's perfectly OK, so copying it into GPL code is relatively benign. That's the whole idea: no restrictions on how BSD/MIT code is used.
Another approach to get rid of traditional OSes and their overhead to switch process contexts is to use a memory safe language for the OS and the applications. An "OS call" is then just an ordinary method call. Servers can run tightly integrated with the network stack to ensure minimal delay to process a request.
You don't need Java or C# (and all their inefficiencies such as the GC and "allocate everything on heap") for that. Here is a (still somewhat prototypical) language of mine which attempts to marry memory-safety with C++ efficiency and soft-realtime-ness:
http://sourceforge.net/projects/sappeurcompiler/
Here is how to build "external" apps and run them safely (at the end of the document):
http://sourceforge.net/p/sappeurcompiler/code-0/2/tree/trunk/doc/SAPPEUR.pdf?format=raw
I'll use this opportunity to do the piper for my own language:
http://sourceforge.net/p/sappeurcompiler/code-0/2/tree/trunk/
http://sourceforge.net/p/sappeurcompiler/code-0/2/tree/trunk/doc/SAPPEUR.pdf?format=raw
It is actually very unfortunate that people confuse "memory safe" with "needs GC", "must be inefficient, sluggish" and "will randomly freeze". It appears the Algol guys already had memory safety but much of it was forgotten because "Unix is free and it comes with a portable assembler called "C""
Looks like a draw. Perl is excellent; something you can love.
With a tiny bit of googleing you will find lots of hashtable implementations for C.
E.g. http://freecode.com/projects/libghthash
I guess the main selling point of Lua is some sort of lazy-ass type system and memory safety. The latter greatly contributes to security, especially when novice or under Management Duress.
Here's a language which does not force inefficiency on you, but is still memory safe:
http://sourceforge.net/p/sappeurcompiler/code-0/2/tree/trunk/
..they need a Memory Safe Language. They only know of crappy* ones (Java to Lua) so they use the least crappy one. They make no effort to build something like "memory safe C" ? Quite shoddy decision-making.
Here's a memory safe C++ variant (conceptually, not literally !):
http://sourceforge.net/p/sappeurcompiler/code-0/2/tree/trunk/
* I consider it crappy if a C program launches in 10ms and less while the ordinary Java program needs 1000ms. And yeah, it matters dramatically in shell programming (that's what the really professional sysadmins do) where you like to launch programs by means of xargs probably 50 times per second. I consider it crappy if you cannot control memory/resource deallocation. I consider it crappy if ergonomics quality is killed by random freezes due to GC.
their Lego port.
none
Won't somebody think of the precious CPU cycles!
"Ignorance more frequently begets confidence than does knowledge"
- Charles Darwin
The Portable Assembler is actually counter-productive when it comes to numerical code, because the compiler can only do lowest-level optimizations. Two pointers could point to the same or to overlapping memory, for example.
FORTRAN is much older than C, much older than almost any programming language, but it still leads in scientific code ! The people in Illinois have devoted their lifes to doing thinks like reordering for-loops to get better memory/cache access patterns. Google for the work of a Mr Kuck and read his papers.
So, I guess your claim is at least not entirely based on facts.
Don't get me started on the horrible security issues in typical C code. Let just say "ping of death".
The world would be a better place if we used much more Ada and Fortran. Or Sappeur (search it on sourceforge), my invention. Ok, take that with a grain of salt.
If we were to head in the direction of removing non-essential stuff from the kernel, soon enough we'd end up with a microkernel architecture, and dear Linus would burst a vein. That wouldn't do.
512MB is a massive amout of memory. I have personally run GUIs on 4MB. I ran an excellent HPUX/PA RISC workstation with just 128MB RAM. It had a very functional GUI (based on X11 and Motif) and it was indestructible without water.
There was a time national police mainframes ran on 512 KILObytes. Boy, you are sitting in front of a supercomputer !
I mean it very serious when I say that these days we are pissing away memory for essentially no tangible benefit.
the entire point of the free software movement and the PC movement is that users should be able to do whatever the hell they want with their machine. they bought it.
Process A calls kernel. That's okay, kernel-userspace transitions can be reasonably fast. Function called is actually implemented in userspace process B. Whoa, context switch. You can easily have a 100x slowdown in your kernelspace interpreter and still come out ahead compared to incurring that context switch. This all assuming that the driver will not do a lot of complicated processing. Rather, it will probably just need to check some basic data structures to identify whether the operation is currently allowed and maintain some shared state between processes. (Not) surprisingly enough, that's all there is to many drivers. I would even think that some file systems currently implemented in FUSE would be faster using a reasonably well-behaving "safe" kernel language.
I just got a new colleague, fresh from uni, addicted to Perl. I am a long time user and dealer. Do you want to try it ? The first three questions are answered for free, you know. After that, my rate of 100$/hour applies, of course.
..even better, you can have memory safe C++ generated for you. No GC crapola needed whatsoever:
http://sourceforge.net/p/sappeurcompiler/code-0/2/tree/
This single feature might allow in near future for custom appliance shops to use non-kernel-level developers to build purpose built systems/devices.
It is important to note that NetBSD is an OS built to support large number of various CPU architectures
Having a combination of flexible kernel, easier access to developer resources and tooling to support various CPUs -- might be the right mix of things to take purpose bulit OS to the next level
Lua interpreter instances are small. In a kernel environment, you'd like run lots of tiny Lua interpreters, one for every "task" you want to perform. You can impose limits on memory and CPU usage on each instance. Under an out-of-memory condition, you can also ask each of them to reclaim a maximum amount of memory, via a hook and via the GC interface. And if that fails, you can start killing them off one by one in order of least importance, just like you might with other processes or functions. The fact that Lua has better defined semantics and interfaces for resource management than a chunk of C code means that it's actually easier to deal with out of memory conditions with Lua interpreters than for general C code.
Regardless of how much this would simplify kernel hacking, it means that a lot of controversial or legally restricted code can be added to the kernel with just some tiny text files being passed around. Users could likely add exFAT support, cryptography, DVD decryption, etc. with nothing more than a command like "cat exfat-support > /proc/lua".
And this would also be available at boot time, making it much easier to work around all the weird problems that keep Linux systems from starting; instead of ever increasingly complex configuration files, blacklists, and C-based workarounds, most of those boot problems could be handled by loading a little bit of Lua code at boot time.
So Linux is old-fashioned in that it is rock-solid ? They should do more "experiments" for the sake of experiments ? I assume something "functional" or "monads" or similar crapola. Surely it would be as stable and efficient as Windows at some point, if they did that.
Guess what ? Lots of people run the world's business on zOS, which dates back into the 60s and they are very happy. They usually do NOT use dynamic memory allocation. Their stuff simply works while the kids play with a new mad idea every week and think they are somehow "innovative".
If they have a JIT compiler for it, then it actually won't hurt too much, except at the initial use of a feature. Unless they have the incredible stability of Windows 9x... This shouldn't be a problem.
Self proclaimed typo king, and inventor of the bear destroying coffee table (patent not pending).