It's actually more than that. It means that the drivers should be both GPL *and* in the mainline kernel. The second part is often glossed over, but when a driver is included in the kernel the maintenance load on the original author drops significantly. If a vendor upstreams the driver, they don't need to track every single release anymore. One of the major rules of Linux kernel development is that if you change a global kernel API, then you have to change all of its consumers as well. Vendors would get that upkeep for free.
Yes, BKL is Big Kernel Lock. If you look at the code, you'll see reiserfs_write_lock(sb) everywhere. This a macro for lock_kernel(), which is the Big Kernel Lock. It's a regular spinlock[1] with two interesting side effects. It's dropped when the thread sleeps and is reacquired when the process reawakens, and you're allowed recursive locks[2]. Since the kernel will schedule when it's waiting on a slow disk, it drops the lock, so that's not a source of contention. The real issue is when it's doing things like balancing the tree or other CPU-intensive task, it's holding the BKL - and that means completely unrelated file systems can't do anything.
To get back to your other question: When you're looking at your top(1) output, since a spinlock is just a tight loop waiting for a value to change, it would appear as sy. When the kernel is waiting on a slow disk, it appears as wa.
1: There is a config option that uses a semaphore instead of a spin lock, but I'm not sure how widely used it is. It would appear as sleeping rather than system execution. 2: This is the hard part about simply replacing the BKL with another fs-specific lock. Other lock types aren't recursive.
I wrote the original email proposing that SUSE switch from reiserfs to ext3. At the risk of triggering responses of "The lady doth protest too much," I'll restate a few statements I've made elsewhere in response to common questions:
1) The decision has *nothing* to do with Hans' situation. The email was released on the same day as the initial story broke, but it was pointed out to me after I had sent the email. I was concerned then, correctly as it turns out, that people would consider the two issues intertwined. They're not. My proposal was based on technical and maintainability reasons alone. The timing is an extremely unfortunate coincidence.
2) SUSE is *not* dropping reiser3 support. This change only affects the default. It doesn't change our support of reiser3 at all. We still support four major file systems: ext3, reiserfs, xfs, and ocfs2. Our installer offers other file systems as well as a convenience, and users are free to use any of them. So, if you're committed to reiser3 or xfs, nothing is stopping you from continuing to deploy systems using them.
3) Many benchmarks show reiser3 as performing better than ext3, and this is mostly true. What isn't shown in those benchmarks is that if you're operating two or more reiser3 file systems in parallel, performance will degrade for both of them due to the use of BKL everywhere. ext3 (and other file systems) will don't degrade in that case. I've also read reports that there is a bit of research going on into making ext3 locking finer grained. I don't have any sources to cite, but any reduction of critical sections without reducing reliability is always a good thing.
People refer to reiser3 as a modern file system, but I'd call it progressive. Reiser3 has served us well for years, but it's showing its age. The basic idea behind reiser3 is still sound, and when extended with integrated integrity checking and better b-tree locking borrowed from years of database research, it would perform extremely well. The problem is that adding the first is a huge disk format change, which means it's no longer reiser3. Adding the second is a hugely invasive change that would throw out a good chunk of the existing code -- again, essentially creating a new file system. It would be like people saying, "I like my ext3 file system, but I don't like the code. Let's start over." Combined with a small development community, it's a recipe for instability and there are more interesting problems out there.
I'm not a fan of this proposal, but I'm curious what the real difference is between this and Internet2 connectivity that get people so incensed? Universities and corporations on Internet2 get higher bandwidth to each other than the rest of the internet, and for that they pay a premium.
It seems to me that the major difference is that it's the telcos coming up with the idea and that end users may actually get to use it. While I'd prefer everyone get access to the higher speed network, what's stopping backbone providers from continuing to upgrade services as they have been?
This seems quite a bit different than previous stories about telcos offering priority on the regular internet to services that pay up. That would definitely be questionable. This is using a completely separate network that they own and charge access to - why shouldn't they be allowed to do this?
Enough with the free speech vs. private university arguments. Free speech applies to government prosecution, not enforcement of the rules of a private organization. As others have noted, if you are a member of a private organization, be it a university, company, or professional sports team, you are subject to their rules if you wish to remain a member.
The problem with this story isn't that the student violated the rules and now doesn't want to deal with the consequences, but that the administration is being accused of interpretting the rules far beyond their intended meaning, employing selective enforcement, and not allowing the student to present his side of the case. As a graduate of a private university, there are usually rules in place to ensure due process for the student as well. I haven't read Marquette's rulebook, nor do I plan to, but the discussion should really be focused around that, not free speech vs. university rules.
Pet peave: why doesn't unlink(2) move stuff to a filesystem-wide deleted area?
UNIX lacks a recycle bin, but so does the Windows NT kernel and the MacOSX kernel. "Recycle Bins" are typically a GUI function, not a kernel function. Try doing an "rm" or "del" using the OSX or Windows command line and see if your files end up in the recycle bin.
KDE and Gnome have a "recycle/trash bin" as well. It's just that a lot of users prefer the command line.
This may be getting too nit picky, but unlink(2) shouldn't do things like that. I've thought about how to implement an automatic undelete cache in a file system, but it just ends up being way too much in-kernel maintenance so that it ultimately detracts from performance. But, flexibility is always there. You're perfectly welcome to override unlink(2) with your own function and LD_PRELOAD, and get exactly the behavior you're asking about - even on the command line. Just make sure you have a "realrm" that uses the stock unlink(2);)
No, that's not the point of my post at all. I wasn't addressing the usefulness of a p2p network in this case - rather on the limited usefulness of the content itself.
There are a few separate issues with how they want to roll this out, and they all revolve around DRM.
1) The limited lifespan. Most people can deal with this, since as someone else mentioned the "On Demand" services and Blockbuster effectively limit the time you're allowed to enjoy the content.
2) Attaching DRM to the content means there is a lack of an open standard. I can't very well write a viewer for the content myself, and any attempt to do so would result in angry lawyers contacting me. While the average user doesn't need to be able to author their own viewing application, it means that the developers who write software for operating systems other than ones from Microsoft or Apple can't either, so everyone else loses too.
These don't seem like big issues to the average home user, but the fact is that most people who are downloading TV shows or movies now aren't average home users. Why would these users give up the freedom and functionality they have now, and pay for the privledge of doing so? This deployment isn't going to meet their demands, and thus the use of unauthorized p2p networks to distribute the content in a format more palatable for those users will continue.
Once again, a big media company comes out with an idea so they can claim to have a legitimate path for viewers to take advantage of -- but yet still totally miss what they are actually looking for.
Until these companies actually meet the demands of the people who are looking to download TV/Movies, unauthorized p2p networks will continue to own the market.
Shortcuts, when viewed from the windowing level, may seem like symlinks. They're not.
Try opening one up with a text editor - you get the contents of the shortcut, not the file contents. Yes, a symlink under UNIX is a file containing the path it points to, but it's interpreted by the kernel -- not userspace.
The links you posted aren't anywhere close to the level of sophistication that this software appears to be. While being able to use ptrace to intercept system calls in this manner is a cool hack, I don't think it's on the same level. The idea of reverse execution doesn't appear to be a new one, but the usability of this package appears to be far beyond that offered by mec.
Maybe I'm misreading the information, but the impression I got is that it has the capability to setup a virtual machine so that you can do kernel-level debugging. It talks of being able to reverse step an OS-boot to the hardware initialization and handling SMP, device I/O, etc.
Yes, the kernel has a kgdb patch available, but it requires a certain level of infrastructure to be available before you can use it. The kernel needs to be able to do serial I/O (potentially after it took a head shot) and errors may occur before then. Having something *completely outside* the machine is quite nice. UML isn't interesting in this aspect either because it's not a true hardware environment and trying to debug a UML instance can be.. interesting at times.
That said, it may be possible to integrate existing open source technologies such as bochs and gdb (a separate child post indicated that gdb has had reverse stepping capability for some time) to create a similar package.
How else might you parallel park, get out and push your car sideways to the curb?
Re:Wow, this article is pure uneducated guesswork.
on
Linux Kernel to Fork?
·
· Score: 1
Well, imagine a Beowulf cluster... How long have those patches existed? There's several ways to build a cluster as long as you patch your kernel.
The "clustering" feature that's in demand isn't Beowulf clustering. That's for distributed computability, and isn't applicable in most enterprises.
What enterprise uses demand is high availabiliy clustering. Sure there are userspace clustering implementations now, but truly advanced HA clustering offers things like shared filesystems which can be fairly invasive in the kernel.
Patching the kernel is not an option for enterprise users. Yes, the GPL absolutely grants them the rights to do so, but distributions will not support homebrew kernels, not matter how small the change. There would be no QA testing on such patches, and support isn't the same as being an auxiliary development staff.
Now, since enterprise kernel feature lists are driven by user demand, once there is a stable implementation, it will likely be included in vendor kernels. Multiple distributions maintaining the same patch sets isn't a situation anyone likes, and at that point it may be considered for mainline.
With the 2.6 kernel, dependencies are handled much better than in 2.4. The amount of time waiting for a compile after changing a file (as long as it's not a core header) is pretty negligible these days.
Re:Which version of 2.6???
on
SUSE 9.2 Released
·
· Score: 2, Informative
No, not that version.
Distribution kernels typically add a number of patches to the vanilla version in order to better meet the needs of their users. This includes features like lkcd or external filesystems, but more importantly, it means that it has critical bug fixes that weren't released into a vanilla release kernel.
The SUSE Linux 9.2 kernel carries the version number of 2.6.8, but is actually based on 2.6.9-rc2, with critical bug fixes beyond that. Since 2.6.9 isn't yet released, it would be inaccurate to call the kernel 2.6.9, but it's hardly vanilla 2.6.8.
You can see just what's in the kernel by checking out SUSE Kernel of the Day, which is built from the CVS tree, and picking the appropriate subdirectory under there.
Honestly, what difference would it make? If the receiver is off, it's recording nothing anyway.
This happens occasionally to me with my digital cable box and tivo, when they do firmware upgrades on the cable box.
The result, you lose programming until you notice that the cable box/satellite reciever has been turned off. This happens so infrequently in practice that it's effectively a non-issue.
If you load everything on the filesystem to memory on boot, you end up wasting a lot of memory, since you typically use only a very small subset of your filesystem at any given time.
The solution would be to load things "on demand," as you've suggested.
Linux already does this, and it does more.
If you've ever looked at the output of free(1) after your system has been running for an hour or so, it will appear as if almost all your memory is in use. See those last two columns, "buffers" and "cached"? That's your "on-demand ramdisk" at work.
Linux will use memory that applications aren't using to cache filesystem data (including executables and metadata) to speed future accesses. If your applications need more memory than is currently free, the kernel will drop cached data rather than swap out application memory to disk. That way, you get the benefits of having your executables on a ramdisk, with the flexibility of not having to sacrifice running application performance in the process.
Slashdot Mirror
It's actually more than that. It means that the drivers should be both GPL *and* in the mainline kernel. The second part is often glossed over, but when a driver is included in the kernel the maintenance load on the original author drops significantly. If a vendor upstreams the driver, they don't need to track every single release anymore. One of the major rules of Linux kernel development is that if you change a global kernel API, then you have to change all of its consumers as well. Vendors would get that upkeep for free.
From the same site as the original PA Wiretap case:
. ssf?/base/news/1182392732222890.xml&coll=1
"He's cleared in police taping DA drops charge stemming from Carlisle traffic stop, declares new county policy"
http://www.pennlive.com/patriotnews/stories/index
Yes, BKL is Big Kernel Lock. If you look at the code, you'll see reiserfs_write_lock(sb) everywhere. This a macro for lock_kernel(), which is the Big Kernel Lock. It's a regular spinlock[1] with two interesting side effects. It's dropped when the thread sleeps and is reacquired when the process reawakens, and you're allowed recursive locks[2]. Since the kernel will schedule when it's waiting on a slow disk, it drops the lock, so that's not a source of contention. The real issue is when it's doing things like balancing the tree or other CPU-intensive task, it's holding the BKL - and that means completely unrelated file systems can't do anything.
To get back to your other question: When you're looking at your top(1) output, since a spinlock is just a tight loop waiting for a value to change, it would appear as sy. When the kernel is waiting on a slow disk, it appears as wa.
1: There is a config option that uses a semaphore instead of a spin lock, but I'm not sure how widely used it is. It would appear as sleeping rather than system execution.
2: This is the hard part about simply replacing the BKL with another fs-specific lock. Other lock types aren't recursive.
I wrote the original email proposing that SUSE switch from reiserfs to ext3. At the risk of triggering responses of "The lady doth protest too much," I'll restate a few statements I've made elsewhere in response to common questions:
t ching-reiserfs-as-it-default-fs/#comment-28534
1) The decision has *nothing* to do with Hans' situation. The email was released on the same day as the initial story broke, but it was pointed out to me after I had sent the email. I was concerned then, correctly as it turns out, that people would consider the two issues intertwined. They're not. My proposal was based on technical and maintainability reasons alone. The timing is an extremely unfortunate coincidence.
2) SUSE is *not* dropping reiser3 support. This change only affects the default. It doesn't change our support of reiser3 at all. We still support four major file systems: ext3, reiserfs, xfs, and ocfs2. Our installer offers other file systems as well as a convenience, and users are free to use any of them. So, if you're committed to reiser3 or xfs, nothing is stopping you from continuing to deploy systems using them.
3) Many benchmarks show reiser3 as performing better than ext3, and this is mostly true. What isn't shown in those benchmarks is that if you're operating two or more reiser3 file systems in parallel, performance will degrade for both of them due to the use of BKL everywhere. ext3 (and other file systems) will don't degrade in that case. I've also read reports that there is a bit of research going on into making ext3 locking finer grained. I don't have any sources to cite, but any reduction of critical sections without reducing reliability is always a good thing.
People refer to reiser3 as a modern file system, but I'd call it progressive. Reiser3 has served us well for years, but it's showing its age. The basic idea behind reiser3 is still sound, and when extended with integrated integrity checking and better b-tree locking borrowed from years of database research, it would perform extremely well. The problem is that adding the first is a huge disk format change, which means it's no longer reiser3. Adding the second is a hugely invasive change that would throw out a good chunk of the existing code -- again, essentially creating a new file system. It would be like people saying, "I like my ext3 file system, but I don't like the code. Let's start over." Combined with a small development community, it's a recipe for instability and there are more interesting problems out there.
I've posted some more lengthy comments here: http://linux.wordpress.com/2006/09/27/suse-102-di
A lot of ISPs have caught on to customers talking with their feet and now lock in subscribers.
I'm not a fan of this proposal, but I'm curious what the real difference is between this and Internet2 connectivity that get people so incensed? Universities and corporations on Internet2 get higher bandwidth to each other than the rest of the internet, and for that they pay a premium.
It seems to me that the major difference is that it's the telcos coming up with the idea and that end users may actually get to use it. While I'd prefer everyone get access to the higher speed network, what's stopping backbone providers from continuing to upgrade services as they have been?
This seems quite a bit different than previous stories about telcos offering priority on the regular internet to services that pay up. That would definitely be questionable. This is using a completely separate network that they own and charge access to - why shouldn't they be allowed to do this?
Enough with the free speech vs. private university arguments. Free speech applies to government prosecution, not enforcement of the rules of a private organization. As others have noted, if you are a member of a private organization, be it a university, company, or professional sports team, you are subject to their rules if you wish to remain a member.
The problem with this story isn't that the student violated the rules and now doesn't want to deal with the consequences, but that the administration is being accused of interpretting the rules far beyond their intended meaning, employing selective enforcement, and not allowing the student to present his side of the case. As a graduate of a private university, there are usually rules in place to ensure due process for the student as well. I haven't read Marquette's rulebook, nor do I plan to, but the discussion should really be focused around that, not free speech vs. university rules.
Pet peave: why doesn't unlink(2) move stuff to a filesystem-wide deleted area?
;)
UNIX lacks a recycle bin, but so does the Windows NT kernel and the MacOSX kernel. "Recycle Bins" are typically a GUI function, not a kernel function. Try doing an "rm" or "del" using the OSX or Windows command line and see if your files end up in the recycle bin.
KDE and Gnome have a "recycle/trash bin" as well. It's just that a lot of users prefer the command line.
This may be getting too nit picky, but unlink(2) shouldn't do things like that. I've thought about how to implement an automatic undelete cache in a file system, but it just ends up being way too much in-kernel maintenance so that it ultimately detracts from performance. But, flexibility is always there. You're perfectly welcome to override unlink(2) with your own function and LD_PRELOAD, and get exactly the behavior you're asking about - even on the command line. Just make sure you have a "realrm" that uses the stock unlink(2)
No, that's not the point of my post at all. I wasn't addressing the usefulness of a p2p network in this case - rather on the limited usefulness of the content itself.
There are a few separate issues with how they want to roll this out, and they all revolve around DRM.
1) The limited lifespan. Most people can deal with this, since as someone else mentioned the "On Demand" services and Blockbuster effectively limit the time you're allowed to enjoy the content.
2) Attaching DRM to the content means there is a lack of an open standard. I can't very well write a viewer for the content myself, and any attempt to do so would result in angry lawyers contacting me. While the average user doesn't need to be able to author their own viewing application, it means that the developers who write software for operating systems other than ones from Microsoft or Apple can't either, so everyone else loses too.
These don't seem like big issues to the average home user, but the fact is that most people who are downloading TV shows or movies now aren't average home users. Why would these users give up the freedom and functionality they have now, and pay for the privledge of doing so? This deployment isn't going to meet their demands, and thus the use of unauthorized p2p networks to distribute the content in a format more palatable for those users will continue.
Once again, a big media company comes out with an idea so they can claim to have a legitimate path for viewers to take advantage of -- but yet still totally miss what they are actually looking for.
Until these companies actually meet the demands of the people who are looking to download TV/Movies, unauthorized p2p networks will continue to own the market.
So are all their customers now in for some ill communication?
Shortcuts, when viewed from the windowing level, may seem like symlinks. They're not.
Try opening one up with a text editor - you get the contents of the shortcut, not the file contents. Yes, a symlink under UNIX is a file containing the path it points to, but it's interpreted by the kernel -- not userspace.
The links you posted aren't anywhere close to the level of sophistication that this software appears to be. While being able to use ptrace to intercept system calls in this manner is a cool hack, I don't think it's on the same level. The idea of reverse execution doesn't appear to be a new one, but the usability of this package appears to be far beyond that offered by mec.
Maybe I'm misreading the information, but the impression I got is that it has the capability to setup a virtual machine so that you can do kernel-level debugging. It talks of being able to reverse step an OS-boot to the hardware initialization and handling SMP, device I/O, etc.
Yes, the kernel has a kgdb patch available, but it requires a certain level of infrastructure to be available before you can use it. The kernel needs to be able to do serial I/O (potentially after it took a head shot) and errors may occur before then. Having something *completely outside* the machine is quite nice. UML isn't interesting in this aspect either because it's not a true hardware environment and trying to debug a UML instance can be.. interesting at times.
That said, it may be possible to integrate existing open source technologies such as bochs and gdb (a separate child post indicated that gdb has had reverse stepping capability for some time) to create a similar package.
Someone decides to write a PC emulator in the same "oh neat" fashion of the c-64 in a joystick.
or
"In The Future, Everything Will Work" - Schwa Corporation
Take your pick.
The motherboard were faulty with respect to network connectivity on boot, not stability.
The end result of replacing them before a major outage? Having another outage.
How else might you parallel park, get out and push your car sideways to the curb?
Well, imagine a Beowulf cluster... How long have those patches existed? There's several ways to build a cluster as long as you patch your kernel.
The "clustering" feature that's in demand isn't Beowulf clustering. That's for distributed computability, and isn't applicable in most enterprises.
What enterprise uses demand is high availabiliy clustering. Sure there are userspace clustering implementations now, but truly advanced HA clustering offers things like shared filesystems which can be fairly invasive in the kernel.
Patching the kernel is not an option for enterprise users.
Yes, the GPL absolutely grants them the rights to do so, but distributions will not support homebrew kernels, not matter how small the change. There would be no QA testing on such patches, and support isn't the same as being an auxiliary development staff.
Now, since enterprise kernel feature lists are driven by user demand, once there is a stable implementation, it will likely be included in vendor kernels. Multiple distributions maintaining the same patch sets isn't a situation anyone likes, and at that point it may be considered for mainline.
With the 2.6 kernel, dependencies are handled much better than in 2.4. The amount of time waiting for a compile after changing a file (as long as it's not a core header) is pretty negligible these days.
No, not that version.
Distribution kernels typically add a number of patches to the vanilla version in order to better meet the needs of their users. This includes features like lkcd or external filesystems, but more importantly, it means that it has critical bug fixes that weren't released into a vanilla release kernel.
The SUSE Linux 9.2 kernel carries the version number of 2.6.8, but is actually based on 2.6.9-rc2, with critical bug fixes beyond that. Since 2.6.9 isn't yet released, it would be inaccurate to call the kernel 2.6.9, but it's hardly vanilla 2.6.8.
You can see just what's in the kernel by checking out SUSE Kernel of the Day, which is built from the CVS tree, and picking the appropriate subdirectory under there.
Right, we must not waffle or send mexed missages even if presented with new facts.
What, are you running as an incumbent US president?
Aww, Mozilla, the touchy feely browser. It hugs my resources!
Honestly, what difference would it make? If the receiver is off, it's recording nothing anyway.
This happens occasionally to me with my digital cable box and tivo, when they do firmware upgrades on the cable box.
The result, you lose programming until you notice that the cable box/satellite reciever has been turned off. This happens so infrequently in practice that it's effectively a non-issue.
If it works, don't fix it.
If you load everything on the filesystem to memory on boot, you end up wasting a lot of memory, since you typically use only a very small subset of your filesystem at any given time.
The solution would be to load things "on demand," as you've suggested.
Linux already does this, and it does more.
If you've ever looked at the output of free(1) after your system has been running for an hour or so, it will appear as if almost all your memory is in use. See those last two columns, "buffers" and "cached"? That's your "on-demand ramdisk" at work.
Linux will use memory that applications aren't using to cache filesystem data (including executables and metadata) to speed future accesses. If your applications need more memory than is currently free, the kernel will drop cached data rather than swap out application memory to disk. That way, you get the benefits of having your executables on a ramdisk, with the flexibility of not having to sacrifice running application performance in the process.
Actually, both have journaling filesystems.
hfs+ supports a journal (starting with macos 10.2.2 server and 10.3 panther), and ntfs5 supports a journal (starting with win2k)