How do you get people/organisations to share their disk space, clock cycles, and bandwidth for other people's data?
That's easy. Let them make a profit from doing so. Let's say that I have a bunch of data that I need to distribute to a hundred sites. Doing that via standard means is pretty inefficient, sucking up a lot of costly bandwidth. Doing it via something like OceanStore might be much more efficient, and that fact creates a potential market niche. One of the most interesting ideas in OceanStore is non-technical; it's the idea that it allows "data access providers" (my term, not theirs) to offer a new service that is more efficient than what it replaces. If I want to distribute big mounds of data between a hundred sites, I might well do better to engage the services of such a data access provider using OceanStore than paying a mere bandwidth provider for all that unnecessary traffic.
The long term goal IMHO is to keep every version of every document ever made, forever.
That may be your long-term goal, but it's not an explicit goal of OceanStore. As it turns out, though, it might happen anyway. One of OceanStore's central goals is to make data pretty much indestructible, and if there were a version-retirement system it could potentially be subverted to destroy data. Last I heard, this was still an unresolved issue.
There is a project named Elephant - they never forget - that does have permanent maintenance of all versions as an explicit goal. I don't have a URL handy, but it shouldn't be hard to find via the standard search methods.
However, this ignores, that when data is being stored in far flung places, the potential for interception by both domestic and international friendly and hostile entities is possible
I don't think the problem is being ignored. I have in fact discussed security with the OceanStore folks face to face, and I can assure you that they understand the problems. One aspect of the problem is that there's no perfect solution: you either want your data everywhere you go, or you want it to be totally secure in one location. No matter how many levels of attack and countermeasure you go through, you keep coming back to that. OceanStore will do the best they can to keep data secure, and they have a formidable bag of tricks at their disposal (e.g. the partial-key stuff), but at some level of paranoia no such data-distribution facility could be considered a good fit for secrecy needs. Those people can and should use something else, which does not reflect at all on OceanStore or the value it provides.
If my network connection is down, how do I work offline?
Distributed operation and disconnected operation are IMO separable problems. There is a certain appeal in the idea of using the same approach to handle both, and some decent systems - e.g. Coda - have been based on that idea, but I believe it's a mistake.
My answer to your question is that you use some separate mechanism such as Palm conduits or the Windows Briefcase to handle the disconnected-operation part. Whenever and wherever you happen to be connected, you'll get to sync vs. the closest replica of your data in the distributed data store.
Software projects are among the HARDEST to place an estimate on.>
Absolutely. It's hard, in fact it's very hard, and personally I'm not all that good at it, but I've had the good fortune to work with people who do it well and I've seen some pretty good results. Just because something's hard doesn't mean it shouldn't be attempted or done as well as possible...heck, writing kernel code is hard too.;-)
Worse, most companies don't use the successes/failures of past projects to adjust the prediction of new projects.
Also very true, and very unfortunate. This is only one of several areas in which it seems that companies are slow to learn from their mistakes.
willing to lay money that they're no where near the 3% of construction projects.
Right again. I happen to believe that such accuracy is actually achievable, but it requires a strong commitment to adopting, learning, and using the tools. In actuality, too few organizations seem willing to make that commitment.
That said, I still think that Linux development needs to grow in this direction. For one thing, people are watching. Some of those people are not our friends, and I'd rather fix a chink in the armor than try to hide it. For another thing, good processes have been developed for a reason. Bugfests and slipped schedules and incompatibilities and lack of documentation frustrate nobody as much as they do the developers themselves. In the long run, even the people who hated doing these things will later be glad they did.
While it is indeed unreasonable to expect that products will ship exactly on announced dates, and that pressuring people to do so might result in the release of still-buggy code, I think there's room for more discipline than is in fact being exhibited by the Linux kernel gurus. Scheduling software projects is not totally a black art. People experienced in a particular kind of programming can often come up with remarkably good estimates of how long things will take, how much extra time to allow for bugs that fall out during testing, etc. Nobody's perfect, but it is entirely possible to come up with a date whose percentage probability of being met is in the high nineties.
Why doesn't this happen in Linux? Two reasons: optimism and lack of discipline. There's no significant penalty for missing a date in open source, so there's no incentive to be pessimistic. When people aren't afraid of the consequences for a date being wrong, they'll usually give you a "best guess" - 51% confidence - date. People who hold themselves to a higher standard of diligence might give you a 90% number, but the project as a whole invariably ends up delayed by the people who couldn't be bothered coming up with a solid number when the project started.
Lack of discipline bites us in several ways:
There's no project plan to speak of. Features and patches get added even in the latest stages of development - something that should make any professionally-minded developer cringe. Nobody knows until it ships what Linux 2.x is actually going to be.
There's practically no design phase to speak of, so people just plain don't know what they're getting into when they start a release, so of course they don't know how long it will take.
There's no specific review process, rarely any serious unit tests, and never any regression tests. There's no decent bug-tracking system. Even well into a project, it's impossible to guess where we stand on the "bug curve".
All of this adds up to create an extremely unpredictable development environment. It's only because of hard work and raw talent that Linux kernel release dates aren't ten times more of a joke than Microsoft's have ever been. With talent and work and just a tiny bit of engineering discipline, we could do a hell of a lot better than we're doing wrt release dates.
A long time ago, in an almost but not quite pre-email era, I read an article suggesting that we need several new kinds of punctuation to augment the familiar exclamation point, question mark, etc. I don't remember most of them, but the one that stuck in my mind is the "irol" (a play on "irony" and "eye-roll") to indicate sarcasm. The author even proposed a glyph for it, but I can't quite remember what it looked like.
If anyone knows anything about the article, and particularly if they know of a copy online, please send me email.
A color LCD of usable brightness (another huge drain on battery life) is going to output a certain amount of energy
True for LCD, but why limit yourself to one technology?. There's no reason a screen has to emit light at all. After looking at several flavors of "electronic paper" it doesn't seem particularly fanciful to imagine a display which consumes zero power if the image isn't changing and which is readable under the same wide variety of conditions as regular paper. It may well be that such displays will always lag behind more conventional technologies in areas such as transition time or color depth, but for a very wide variety of devices and applications that would still be a big win.
Even within the realm of light-emitting display technology, there's plenty of room to reduce power consumption. For example, the Light Emitting Polymer work at CDT could lead to displays that consume a lot less power than CRT or LCD displays, in addition to being extremely thin, light and flexible.
I'm not trying to argue with you here. I completely agree with your main point that power consumption needs to be addressed beyond the CPU. Displays and rotating media in particular are at least as deserving of attention. This is all just FYI.
One problem I see with this is that it means that both branches have to be ready at the same time in order to make the descision.
I don't think that's necessarily true. At a certain point it's perfectly reasonable to say "too late, you lose, better luck next time" and yank the slow group's approved-fork status.
The problem I see with your proposal is that it doesn't solve the basic problem of placing unreasonable - and non-technical - barriers before people who want to try something different. In my example two posts ago of how things work now, the red team not only has to do their own original work but faces the formidable extra obstacle of tracking changes made by the white team. The white team, by contrast, is given carte blanche (heh) to proceed without worrying about the red team's changes, or even whether their own changes will adversely affect the red team's efforts. This is not a small issue. With all of the interpendencies in the kernel (see my comments on modularity or lack thereof), this makes it extremely difficult for red to keep up unless they have many more developers than white - and if they did have such a majority they'd be the white team.
This is actually very similar to a current political debate about winner-take-all vs. proportional representation. What we have now is very close to winner-take-all; those in the minority are effectively shut out of the process. Approved forks are like a loyal opposition; they give the dissenters at least some chance to get their point across. What's important is that we find some way to turn dissent and competition into productive forces. Are approved forks the best or only way to achieve that? Do they achieve it at all? Maybe; maybe not. What's certain is that continuing to let good ideas get "killed in committee" (this political analogy works better than I thought) is not healthy for Linux in the long term.
P2P and Heirarchy both have their strengths and weakenesses, and particularly clever developers can pool strengths without amplifying weakenesses and get some pretty neat systems...
I agree completely. It was not my intent to imply that P2P is uselss, merely that it's a poor fit for problems where scalability is a major design issue. "Hybrid vigor" is a very real phenomenon in computing.
This is a bandwagon that just won't roll very far, and the reason - as usual - is obvious to people who've studied the field for a while. Naively implemented, a P2P protocol tends to generate O(n^2) messages for a given workload, where N is the number of nodes. This can often be brought down to O(n) but only with absolutely top-notch developers and a lot of effort. Better than O(n) is usually impossible.
By contrast, hierarchical systems tend to hover between O(n) and O(log(n)) depending on the particular problem. This does not necessarily apply only to single-rooted hierarchies, either. A multi-rooted hierarchy tends to exhibit the same scaling behavior, though of course the more roots you have the more you start to look like P2P and share its scaling characteristics.
The long and the short of it is that P2P just doesn't scale well. Even the best-implemented P2P protocol can merely approach the message efficiency of a naively implemented hierarchical protocol. For large numbers of nodes this results in the P2P implementation simply getting swamped. The only question is how large and how swamped it has to be before it becomes unusable.
I thought that a somewhat-concrete example might help clarify what I'm suggesting.
Let's say that I, and a significant number of other folks, want to rework the Linux SCSI mid-layer using the CAM-based BSD code as a model. We'll call this group "red" in reference to 1917. The "white" group, which includes some individuals with more clout than any of the reds, also wants to rework the SCSI mid-layer but wants to reinvent the wheel and do it a whole new way unlike any other platform. In addition to the project-specific "core code" for each version, a certain number of tweaks would be necessary to common code to make the core code "fit". Here's what would happen now:
The white group's changes, to both project-specific and common code, will be accepted as soon as they're ready and put into the latest official development kernel.
The red group's changes will remain in an unofficial patch. The patch will not appear in the usual places people go to for Linux updates, so the reds will have to develop their own separate distribution network.
Eventually, both projects will be near completion. The white code will already be in the official Linux kernel, and will be accepted by default. The red code, will only get in if several conditions are met:
The red code is not only better, but quantifiably and significantly better, than the white code.
The red code must be a patch not to the kernel as it was when both projects started, but to the current kernel loaded with white-specific changes.
The reds can somehow overcome the objection that their code cannot be as well tested as the white code which has had the advantages of the official-Linux distribution system for several months.
Not surprisingly, given the much higher standard applied to the reds than to the whites, the red code never makes it into Linux. If red was in fact better, but not by enough to overcome these obstacles, what the Linux community gets is inferior code.
Now, here's how I think things would work with an approved fork.
Two separate development kernel branches are created, for red and white. Both are widely distributed on all the usual Linux sites.
Both red and white are required to track the others' changes to common code. When incompatible changes occur, Linus steps in and decides which change takes precedence.
Eventually, both red and white code are ready. Both have been tested equally. Both are fully self-contained and up-to-date kernel versions ready to go to the next stage.
One version is accepted as the next official Linux version. The other is relegated to the status of an unapproved fork not subject to the rules that governed the approved fork before.
The members of the "losing" team join the "winners" and try to push for applicable elements of their approach to be rolled into the official version. Eventually, the Linux community gets the best code, perhaps even better than either team's individual efforts.
Chances are, a large number of voters(I think they should be developers) would vote to keep good desktop performance; after all, that's what most of them will be programming for.
That seems like a reasonable outcome. Some of us - myself included in this example - might not like it, but if that's what "the people" want...
Next thing you know, someone is in charge for a year or two, making horrible technical decisions
Whoa, wait a minute. I'm suggesting that the community be allowed to decide between work products, not between people. "Who's in charge" wouldn't be affected; it would still be the same not-quite-meritocracy it is now. Those same people would be making a promise to give other people's ideas a fair chance, not to give up their leadership positions.
In any case, even if our leadership were elected, if they started making bad technical decisions the same remedy would apply that always applies in a democracy: vote them out. The scenario you describe, of people getting "into power" and remaining there despite inferior technical decision-making, just seems impossible to me.
Issues that are very big(ie: Big Iron support) should be resolved through compromise(if a compromise is technically feasible), as opposed to "this way" or "that way".
Yes, if possible. However, as you seem aware, sometimes it's not. Sometimes there really is just "this way" and "that way" and they're fundamentally incompatible. Furthermore, sometimes it's not clear even to the most highly informed people which will turn out better. All I'm saying is: do the experiment. Right now, when such an impasse is reached, only one option is pursued - the one that Linus and/or other senior developers happen to favor, for reasons that may or may not be entirely supportable from a technical standpoint. I think we have enough developers now that, in at least a few of the hardest cases, we can try both and see what happens, but I don't think that can work if one group is "real Linux" and the other group is "a bunch of renegades who are trying to destroy real Linux".
As I pointed out in an earlier post, one critical part of making this work is an agreement on both sides to share information about common problems and solutions. I would expect that, any time there's a fork, a "winner" would eventually be declared and the "loser" would then abandon the fork in favor of trying to apply lessons learned during the process to the new official Linux. Maybe that's too idealistic, maybe people are too stubborn and territorial and ego- or profit-driven to allow things to happen that way. *shrug* It's just an idea that I think deserves at least a moment's consideration, even if it's later rejected, and in general it doesn't seem to get considered at all.
Here's another point I forgot to mention. If, as in your example, A has better drivers and B has a better scheduler but code cannot be readily shared between the two, I'd say it might be because the code isn't modular enough. In an ideal world one could mix and match "best of breed" kernel components - even things like drivers and schedulers - in the same way that we already do for applications. There's nothing fundamentally different about kernels that precludes this, and if you'll look back at my original post you'll see that modularity was the very first item on my wish list. Maybe the idea of a fork will be more palatable when things have been made more modular...which IMO makes it all more important that modularity be improved.
The Linux model of having a bunch of stuff out there competing does a better job of getting the winners into widespread use than a system where one version may (e.g.) have better drivers but a worse scheduler than another version, because they forked and can no longer just take each other's code when one of them is clearly better
I agree, but with a caveat. The alternative you present is not what I'm suggesting, and it wouldn't be allowed to happen anyway. The idea here is just an extension of the odd-number/even-number idea we already have. There would always be exactly one official Linux, compared to which any forks have a somewhat inferior status. It's like after 2.4 we allow both 2.5red and 2.5blue to go ahead, with a promise that when the time comes to make 2.5 into 2.6 fair consideration will be given to basing 2.6 on 2.5blue instead of 2.5red.
Looked at another way, it's like Linus (used here for convenience as a personification of Linux authority) saying, "I don't believe in this enough to devote my own time and effort to it, but I recognize that I'm not infallible. I'm willing to let the community decide which they prefer, instead of branding anyone who disagrees with me as a renegade splinter group unworthy of attention or support." Wouldn't that be nice? What I'm suggesting is that the dictator - however benevolent he may be - allow elections to be held once in a while, with real candidates given a real opportunity to present their own views to the electorate. The Linux "brand" has become too important to be the exclusive property of one person, even Linus. Linus does other things besides Linux, and Linux can be done by other people besides Linus.
But I think that *BSD is the fork that you were wishing for
I knew someone would say that. I don't strongly disagree, but I do disagree. Anyone capable of doing meaningful work knows of the *BSD option. If they have foregone that option, there's probably a reason. Most often it's because they're more familiar with Linux. It could also be because they want to address a perceived deficiency in Linux, and working on *BSD doesn't do that. Their idea might not even apply to *BSD. In any case, I think they should have that option.
In order for this to work, there has to be at least lukewarm support from the regular Linux cabal. For example, the following:
An agreement, from both sides, to share information on common problems and common solutions.
An agreement from the "in crowd" not to hamper the forkers' ability to contribute by ostracizing them - e.g. by refusing to answer questions that would be answered for anyone else, by leaving the forkers "out of the loop" when discussing changes that affect them, or by plain old badmouthing.
The right to use the name "Linux", so long as the versions remain clearly distinguishable.
I don't think that's too much to ask. It's not like asking for the insiders to devote gobs of their own time to further a project they don't believe in. If the people involved can be mature enough to make and keep these kinds of promises, and let the community decide in a semi-democratic way which set of changes produced the best result[1], a fork could be a very good thing. On the other hand, if they decide to be territorial about it, if they decide to put their egos or their prospects for financial gain as full-time paid "Linux gurus" ahead of technical progress, then it could be really ugly. I could not recommend an "unsanctioned" fork; that would only hurt everyone. The only way a fork can be beneficial is if the cabal members can be persuaded of the benefits. Quite honestly, I don't think they're up to it, so a productive fork is an extremely remote possibility.
[1] Of course, the community that really matters is the community of distribution providers, who get to decide which version to use as the basis for what they provide, and in many cases they'd be "voting" for what their own members/employees happened to be involved in. It's hardly in Red Hat's interest, for example, to say that someone else's kernel mods worked out better than the ones they paid their own highly-touted employees to produce. This is hardly what I'd call an example of democracy in action, but it's at least a little bit closer than the current autocracy.
Here are some of the ways I'd like to see Linux evolve in the near future:
Better modularity. There's still way too much interdependency between what should be separate subsystems. Sometimes this is in the form of explicit references to each others' data or functions, but just as often it's a more subtle but still undocumented reliance on "X never does this so I won't even check for it, and God help the guy who comes after me if X ever changes".
A better VFS layer. Just having one isn't enough because, to be blunt, some folks did a pretty piss-poor job of specifying, implementing, or testing it. This fact is the primary reason 2.4 isn't out yet.
A mature persistent-module-data interface. We don't need anything as overburdened as the Windows registry or AIX ODM, but we do need something and recent steps in this direction are a very good sign.
Journaled, soft-update, and atomic-update filesystems. Not one, but several. Let them compete. This is an area where Linux can be the foundation for significant improvements in the state of the art.
Better testing. We need a serious test suite for each of the major kernel components, and one for the whole system as well. We should be beyond the point where the current near-total lack of unit or regression tests is acceptable.
Better performance management and monitoring tools. How many of you have used PerfMon on NT? The way it's implemented is kind of crufty, but the flexibility and functionality it provides makes Linux look really bad by comparison. It should be a standard part of Linux kernel or major-subsystem (e.g. database, webserver) development to define and export counters for a general tool like PerfMon to use. A lot of the bickering on linux-kernel about where the bottlenecks are would be neatly settled by a five-minute session with such a tool.
ACLs? I'm still not 100% convinced we need them, but they are more powerful than the current system and there seems to be a demand. In any case they're likely to become a checklist item for a lot of folks soon.
Lastly, there's one more thing I think Linux needs, but explaining why takes more than should go into a single list item. Linux needs a good forking. Seriously. Competition is good. The cabal - yes, there is one in effect, even if its exact membership is debatable - generally has good ideas and provides incredible value in bringing order to chaos. On the other hand, the Powers That Be sometimes suffer from severe Not Invented Here syndrome, and sometimes they use their bully pulpit to shout down perfectly good ideas that conflict with their own biases (or even projects that would compete with what they want to work on). Several have recently seemed to start believing that they're omniscient, as though merely being a genius wasn't good enough. Linus and the others deserve our gratitude, and our respect, but not worship or unquestioning obedience. They need a wakeup call, in the form of someone defying their wishes and achieving superior results in the process.
This will happen, sooner or later, one way or another. We have two choices:
Embrace the possibility as a generator of innovation and healthy competition, and as a way to keep everyone honest and humble.
Let it become a source of chaos and dissension, until someone else eats our lunch for us.
That's it. There are no other choices. I know this will probably not "resonate" with the younger members of the audience, but I would compare the situation to a divorce. The most amicable divorces, where people still remain friends, occur when the people accept the reality and work together on making necessary change go smoothly. The messiest, most destructive divorces happen when people have stayed together long after their interests diverged and they've spent years learning how to hate each other. I don't want Linux to turn into War of the Roses.
Unfortunately, the previous poster picked an example that was too simple to highlight the weaknesses of the rwxrwxrwx permission system. Nonetheless, you still managed to get it wrong on two counts. In the first place, as other posters have since pointed out, the problem specification did not say that the file should be writable by bob or readable by carol, both of which your "solution" allows. Here's where you have a fundamental problem: you have three different kinds of permission you need to specify (read-only for bob, write-only for carol, and none for anyone else) and only two places (group and world) to specify them. It just doesn't work. The model simply isn't up to the task.
Your second error is in assuming that alex is able to create/modify/delete groups on the fly to suit his needs which may differ for every single file he owns. This would not be the case on most systems - almost never on a well-run one. Even on a system that allowed any user to edit/etc/group, the rapid proliferation of groups that would result from this attempt to use a weak model to simulate a stronger one would quickly run into problems with scalability and manageability.
In short, you just can't make rwxrwxrwx do ACLs' job. The current permissions system is an artifact of a very time- and environment-specific tradeoff between functionality and resources (space, processor time).
You can simulate ACL's through users/groups/ownership/permissions just fine.
Simply wrong. Read/write/execute gives eight combinations, and user/group/world only allows three to be expressed simultaneously, and expressiveness is further limited by the fact that most users can't add/delete/modify groups to suit their purposes. ACLs, by contrast, can be used to express all eight possibilities at once, and apply each to arbitrary sets of users.
Anyone who ever studied even one minute of logic can see that the two systems are not equivalent.
integrating C and -Script- becomes a project in itself. I dare you to figure out tools like SWIG, or even to use the python.h files manually. You're in there for some major undertaking.
And how, exactly, would you make it easier? In particular, how would you make it easier without sacrificing the full power and flexibility of extensions?
I think the real problem is with terminology. It's not a matter of "inner loops" vs. "outer loops". It's about performance-critical parts vs. non-performance-critical parts. For most scripts and tools and anything else short of full-blown industrial strength applications, coding any part of your Python application as an extension module is just stupid. In those cases where it might make sense, it's really not that hard to apply standard software-engineering approaches to the design of one or more extension modules which neatly encapsulate the performance-critical parts of your app behind a fairly simple interface. I've done it several times, and been quite satisfied with the results...and it's not because I'm particularly easy to satisfy.;-)
I'm looking for a high-level language, regardless of its syntax that: (1) understands seemlessly C/C++ header files and the APIs that it exposes. No SWIGgin! (But also, no IDL or stuff like that, just direct invocation!)
That's a pretty tall order. C/C++ interfaces aren't exactly self-describing, even as far as number of arguments in many cases, let alone type and meaning etc. There's actually a Python extension module - forgot the name - that lets you do pretty much the sort of direct invocation you're talking about, putting all of the burden on the programmer to make sure they're passing the right number of arguments etc. It turns out not to be all that useful, though, because it really only provides part of a real extension interface - the "interpreter calls X" part. A full extension interface also allows X to call the interpreter, or X to interpret or manipulate or even create complex structures - objects, lists, dictionaries, aggregates of same - comprehensible to the interpreted code, and so on. Without all of that, which is not available in the model you propose, what you have is a crippled extension interface lacking many benefits inherent in the real thing.
2) grabs its language run-time...and nicely links it into one, single clean executable
This sounds pretty much like what "freeze" does, or perhaps a minimally-enhanced version of it. Could you explain what exactly you want that goes beyond that?
BTW, keep in mind that creation of a single monolithic executable is not always the goal. Many ways of doing what you describe would preclude component models or interactive use. It's just one more example of how Python represents a certain set of tradeoffs. It's not perfect for every usage, but the omission of certain features you might want might well be a conscious decision based on the fact that their inclusion might have precluded some other feature that is more generally useful.
most of the people coming in here going on about how ugly perl is typically haven't spent more than an hour using it
And I suppose this is in contrast to the Perl weenies who come into Python articles to bash Python, based on their many many hours spent studying the pros and cons of each? Yeah right. Most Perl folks spend about three minutes looking at Python, notice maybe one thing about the language itself - usually the indentation thing - but mostly just notice that it doesn't look like Perl, and then spend the rest of their lives making the illogical claim that Perl must be better because more people use it.
Python has its weaknesses, which I have gone out of my way to describe in my last post on the subject (a copy of which pretty easy for any non-moron to find on my website), but overall it's a fine language. As with Ruby, and Pike, and several other languages, on technical merit it deserves at least equal consideration to Perl.
I feel pretty much the same way. Ruby seems to be a step forward compared to Python in some respects, and a step backward in others. Overall it's a wash, and since Python already works very well for me I don't think I'd gain much by learning Ruby. That doesn't mean, though, that I would discourage anyone who doesn't already have a scripting language under their belt from choosing Ruby over Python. From what I can tell, they would learn better habits from Ruby than from Perl or Tcl.
TCL, Python, and Perl were not the products of someone who knew anything about language design. They were more like on-the-job training exercises for less than qualified individuals
Wrong. John Ousterhout, creator of Tcl, is a CS professor. Guido van Rossum, creator of Python, is well versed in language design. You may not like their decisions - I personally detest Tcl - but I would bet either of them is about 10x more qualified to design a language than anyone here.
Larry Wall, on the other hand, has no discernible background or ability in language design, and it shows in Perl.
Slashdot Mirror
That's easy. Let them make a profit from doing so. Let's say that I have a bunch of data that I need to distribute to a hundred sites. Doing that via standard means is pretty inefficient, sucking up a lot of costly bandwidth. Doing it via something like OceanStore might be much more efficient, and that fact creates a potential market niche. One of the most interesting ideas in OceanStore is non-technical; it's the idea that it allows "data access providers" (my term, not theirs) to offer a new service that is more efficient than what it replaces. If I want to distribute big mounds of data between a hundred sites, I might well do better to engage the services of such a data access provider using OceanStore than paying a mere bandwidth provider for all that unnecessary traffic.
Yes, it does.
That may be your long-term goal, but it's not an explicit goal of OceanStore. As it turns out, though, it might happen anyway. One of OceanStore's central goals is to make data pretty much indestructible, and if there were a version-retirement system it could potentially be subverted to destroy data. Last I heard, this was still an unresolved issue.
There is a project named Elephant - they never forget - that does have permanent maintenance of all versions as an explicit goal. I don't have a URL handy, but it shouldn't be hard to find via the standard search methods.
I don't think the problem is being ignored. I have in fact discussed security with the OceanStore folks face to face, and I can assure you that they understand the problems. One aspect of the problem is that there's no perfect solution: you either want your data everywhere you go, or you want it to be totally secure in one location. No matter how many levels of attack and countermeasure you go through, you keep coming back to that. OceanStore will do the best they can to keep data secure, and they have a formidable bag of tricks at their disposal (e.g. the partial-key stuff), but at some level of paranoia no such data-distribution facility could be considered a good fit for secrecy needs. Those people can and should use something else, which does not reflect at all on OceanStore or the value it provides.
Distributed operation and disconnected operation are IMO separable problems. There is a certain appeal in the idea of using the same approach to handle both, and some decent systems - e.g. Coda - have been based on that idea, but I believe it's a mistake.
My answer to your question is that you use some separate mechanism such as Palm conduits or the Windows Briefcase to handle the disconnected-operation part. Whenever and wherever you happen to be connected, you'll get to sync vs. the closest replica of your data in the distributed data store.
Absolutely. It's hard, in fact it's very hard, and personally I'm not all that good at it, but I've had the good fortune to work with people who do it well and I've seen some pretty good results. Just because something's hard doesn't mean it shouldn't be attempted or done as well as possible...heck, writing kernel code is hard too. ;-)
Also very true, and very unfortunate. This is only one of several areas in which it seems that companies are slow to learn from their mistakes.
Right again. I happen to believe that such accuracy is actually achievable, but it requires a strong commitment to adopting, learning, and using the tools. In actuality, too few organizations seem willing to make that commitment.
That said, I still think that Linux development needs to grow in this direction. For one thing, people are watching. Some of those people are not our friends, and I'd rather fix a chink in the armor than try to hide it. For another thing, good processes have been developed for a reason. Bugfests and slipped schedules and incompatibilities and lack of documentation frustrate nobody as much as they do the developers themselves. In the long run, even the people who hated doing these things will later be glad they did.
While it is indeed unreasonable to expect that products will ship exactly on announced dates, and that pressuring people to do so might result in the release of still-buggy code, I think there's room for more discipline than is in fact being exhibited by the Linux kernel gurus. Scheduling software projects is not totally a black art. People experienced in a particular kind of programming can often come up with remarkably good estimates of how long things will take, how much extra time to allow for bugs that fall out during testing, etc. Nobody's perfect, but it is entirely possible to come up with a date whose percentage probability of being met is in the high nineties.
Why doesn't this happen in Linux? Two reasons: optimism and lack of discipline. There's no significant penalty for missing a date in open source, so there's no incentive to be pessimistic. When people aren't afraid of the consequences for a date being wrong, they'll usually give you a "best guess" - 51% confidence - date. People who hold themselves to a higher standard of diligence might give you a 90% number, but the project as a whole invariably ends up delayed by the people who couldn't be bothered coming up with a solid number when the project started.
Lack of discipline bites us in several ways:
All of this adds up to create an extremely unpredictable development environment. It's only because of hard work and raw talent that Linux kernel release dates aren't ten times more of a joke than Microsoft's have ever been. With talent and work and just a tiny bit of engineering discipline, we could do a hell of a lot better than we're doing wrt release dates.
A long time ago, in an almost but not quite pre-email era, I read an article suggesting that we need several new kinds of punctuation to augment the familiar exclamation point, question mark, etc. I don't remember most of them, but the one that stuck in my mind is the "irol" (a play on "irony" and "eye-roll") to indicate sarcasm. The author even proposed a glyph for it, but I can't quite remember what it looked like.
If anyone knows anything about the article, and particularly if they know of a copy online, please send me email.
I wasn't being sarcastic at all; I meant that. Compliments are rare enough around here, you should try to accept them well when they're given.
Brilliant! That's a very astute observation, well put.
True for LCD, but why limit yourself to one technology?. There's no reason a screen has to emit light at all. After looking at several flavors of "electronic paper" it doesn't seem particularly fanciful to imagine a display which consumes zero power if the image isn't changing and which is readable under the same wide variety of conditions as regular paper. It may well be that such displays will always lag behind more conventional technologies in areas such as transition time or color depth, but for a very wide variety of devices and applications that would still be a big win.
Even within the realm of light-emitting display technology, there's plenty of room to reduce power consumption. For example, the Light Emitting Polymer work at CDT could lead to displays that consume a lot less power than CRT or LCD displays, in addition to being extremely thin, light and flexible.
I'm not trying to argue with you here. I completely agree with your main point that power consumption needs to be addressed beyond the CPU. Displays and rotating media in particular are at least as deserving of attention. This is all just FYI.
I don't think that's necessarily true. At a certain point it's perfectly reasonable to say "too late, you lose, better luck next time" and yank the slow group's approved-fork status.
The problem I see with your proposal is that it doesn't solve the basic problem of placing unreasonable - and non-technical - barriers before people who want to try something different. In my example two posts ago of how things work now, the red team not only has to do their own original work but faces the formidable extra obstacle of tracking changes made by the white team. The white team, by contrast, is given carte blanche (heh) to proceed without worrying about the red team's changes, or even whether their own changes will adversely affect the red team's efforts. This is not a small issue. With all of the interpendencies in the kernel (see my comments on modularity or lack thereof), this makes it extremely difficult for red to keep up unless they have many more developers than white - and if they did have such a majority they'd be the white team.
This is actually very similar to a current political debate about winner-take-all vs. proportional representation. What we have now is very close to winner-take-all; those in the minority are effectively shut out of the process. Approved forks are like a loyal opposition; they give the dissenters at least some chance to get their point across. What's important is that we find some way to turn dissent and competition into productive forces. Are approved forks the best or only way to achieve that? Do they achieve it at all? Maybe; maybe not. What's certain is that continuing to let good ideas get "killed in committee" (this political analogy works better than I thought) is not healthy for Linux in the long term.
I agree completely. It was not my intent to imply that P2P is uselss, merely that it's a poor fit for problems where scalability is a major design issue. "Hybrid vigor" is a very real phenomenon in computing.
This is a bandwagon that just won't roll very far, and the reason - as usual - is obvious to people who've studied the field for a while. Naively implemented, a P2P protocol tends to generate O(n^2) messages for a given workload, where N is the number of nodes. This can often be brought down to O(n) but only with absolutely top-notch developers and a lot of effort. Better than O(n) is usually impossible.
By contrast, hierarchical systems tend to hover between O(n) and O(log(n)) depending on the particular problem. This does not necessarily apply only to single-rooted hierarchies, either. A multi-rooted hierarchy tends to exhibit the same scaling behavior, though of course the more roots you have the more you start to look like P2P and share its scaling characteristics.
The long and the short of it is that P2P just doesn't scale well. Even the best-implemented P2P protocol can merely approach the message efficiency of a naively implemented hierarchical protocol. For large numbers of nodes this results in the P2P implementation simply getting swamped. The only question is how large and how swamped it has to be before it becomes unusable.
I thought that a somewhat-concrete example might help clarify what I'm suggesting.
Let's say that I, and a significant number of other folks, want to rework the Linux SCSI mid-layer using the CAM-based BSD code as a model. We'll call this group "red" in reference to 1917. The "white" group, which includes some individuals with more clout than any of the reds, also wants to rework the SCSI mid-layer but wants to reinvent the wheel and do it a whole new way unlike any other platform. In addition to the project-specific "core code" for each version, a certain number of tweaks would be necessary to common code to make the core code "fit". Here's what would happen now:
Now, here's how I think things would work with an approved fork.
Doesn't that just seem a whole lot better?
That seems like a reasonable outcome. Some of us - myself included in this example - might not like it, but if that's what "the people" want...
Whoa, wait a minute. I'm suggesting that the community be allowed to decide between work products, not between people. "Who's in charge" wouldn't be affected; it would still be the same not-quite-meritocracy it is now. Those same people would be making a promise to give other people's ideas a fair chance, not to give up their leadership positions.
In any case, even if our leadership were elected, if they started making bad technical decisions the same remedy would apply that always applies in a democracy: vote them out. The scenario you describe, of people getting "into power" and remaining there despite inferior technical decision-making, just seems impossible to me.
Yes, if possible. However, as you seem aware, sometimes it's not. Sometimes there really is just "this way" and "that way" and they're fundamentally incompatible. Furthermore, sometimes it's not clear even to the most highly informed people which will turn out better. All I'm saying is: do the experiment. Right now, when such an impasse is reached, only one option is pursued - the one that Linus and/or other senior developers happen to favor, for reasons that may or may not be entirely supportable from a technical standpoint. I think we have enough developers now that, in at least a few of the hardest cases, we can try both and see what happens, but I don't think that can work if one group is "real Linux" and the other group is "a bunch of renegades who are trying to destroy real Linux".
As I pointed out in an earlier post, one critical part of making this work is an agreement on both sides to share information about common problems and solutions. I would expect that, any time there's a fork, a "winner" would eventually be declared and the "loser" would then abandon the fork in favor of trying to apply lessons learned during the process to the new official Linux. Maybe that's too idealistic, maybe people are too stubborn and territorial and ego- or profit-driven to allow things to happen that way. *shrug* It's just an idea that I think deserves at least a moment's consideration, even if it's later rejected, and in general it doesn't seem to get considered at all.
Here's another point I forgot to mention. If, as in your example, A has better drivers and B has a better scheduler but code cannot be readily shared between the two, I'd say it might be because the code isn't modular enough. In an ideal world one could mix and match "best of breed" kernel components - even things like drivers and schedulers - in the same way that we already do for applications. There's nothing fundamentally different about kernels that precludes this, and if you'll look back at my original post you'll see that modularity was the very first item on my wish list. Maybe the idea of a fork will be more palatable when things have been made more modular...which IMO makes it all more important that modularity be improved.
I agree, but with a caveat. The alternative you present is not what I'm suggesting, and it wouldn't be allowed to happen anyway. The idea here is just an extension of the odd-number/even-number idea we already have. There would always be exactly one official Linux, compared to which any forks have a somewhat inferior status. It's like after 2.4 we allow both 2.5red and 2.5blue to go ahead, with a promise that when the time comes to make 2.5 into 2.6 fair consideration will be given to basing 2.6 on 2.5blue instead of 2.5red.
Looked at another way, it's like Linus (used here for convenience as a personification of Linux authority) saying, "I don't believe in this enough to devote my own time and effort to it, but I recognize that I'm not infallible. I'm willing to let the community decide which they prefer, instead of branding anyone who disagrees with me as a renegade splinter group unworthy of attention or support." Wouldn't that be nice? What I'm suggesting is that the dictator - however benevolent he may be - allow elections to be held once in a while, with real candidates given a real opportunity to present their own views to the electorate. The Linux "brand" has become too important to be the exclusive property of one person, even Linus. Linus does other things besides Linux, and Linux can be done by other people besides Linus.
I knew someone would say that. I don't strongly disagree, but I do disagree. Anyone capable of doing meaningful work knows of the *BSD option. If they have foregone that option, there's probably a reason. Most often it's because they're more familiar with Linux. It could also be because they want to address a perceived deficiency in Linux, and working on *BSD doesn't do that. Their idea might not even apply to *BSD. In any case, I think they should have that option.
In order for this to work, there has to be at least lukewarm support from the regular Linux cabal. For example, the following:
I don't think that's too much to ask. It's not like asking for the insiders to devote gobs of their own time to further a project they don't believe in. If the people involved can be mature enough to make and keep these kinds of promises, and let the community decide in a semi-democratic way which set of changes produced the best result[1], a fork could be a very good thing. On the other hand, if they decide to be territorial about it, if they decide to put their egos or their prospects for financial gain as full-time paid "Linux gurus" ahead of technical progress, then it could be really ugly. I could not recommend an "unsanctioned" fork; that would only hurt everyone. The only way a fork can be beneficial is if the cabal members can be persuaded of the benefits. Quite honestly, I don't think they're up to it, so a productive fork is an extremely remote possibility.
[1] Of course, the community that really matters is the community of distribution providers, who get to decide which version to use as the basis for what they provide, and in many cases they'd be "voting" for what their own members/employees happened to be involved in. It's hardly in Red Hat's interest, for example, to say that someone else's kernel mods worked out better than the ones they paid their own highly-touted employees to produce. This is hardly what I'd call an example of democracy in action, but it's at least a little bit closer than the current autocracy.
Here are some of the ways I'd like to see Linux evolve in the near future:
Lastly, there's one more thing I think Linux needs, but explaining why takes more than should go into a single list item. Linux needs a good forking. Seriously. Competition is good. The cabal - yes, there is one in effect, even if its exact membership is debatable - generally has good ideas and provides incredible value in bringing order to chaos. On the other hand, the Powers That Be sometimes suffer from severe Not Invented Here syndrome, and sometimes they use their bully pulpit to shout down perfectly good ideas that conflict with their own biases (or even projects that would compete with what they want to work on). Several have recently seemed to start believing that they're omniscient, as though merely being a genius wasn't good enough. Linus and the others deserve our gratitude, and our respect, but not worship or unquestioning obedience. They need a wakeup call, in the form of someone defying their wishes and achieving superior results in the process.
This will happen, sooner or later, one way or another. We have two choices:
That's it. There are no other choices. I know this will probably not "resonate" with the younger members of the audience, but I would compare the situation to a divorce. The most amicable divorces, where people still remain friends, occur when the people accept the reality and work together on making necessary change go smoothly. The messiest, most destructive divorces happen when people have stayed together long after their interests diverged and they've spent years learning how to hate each other. I don't want Linux to turn into War of the Roses.
Unfortunately, the previous poster picked an example that was too simple to highlight the weaknesses of the rwxrwxrwx permission system. Nonetheless, you still managed to get it wrong on two counts. In the first place, as other posters have since pointed out, the problem specification did not say that the file should be writable by bob or readable by carol, both of which your "solution" allows. Here's where you have a fundamental problem: you have three different kinds of permission you need to specify (read-only for bob, write-only for carol, and none for anyone else) and only two places (group and world) to specify them. It just doesn't work. The model simply isn't up to the task.
Your second error is in assuming that alex is able to create/modify/delete groups on the fly to suit his needs which may differ for every single file he owns. This would not be the case on most systems - almost never on a well-run one. Even on a system that allowed any user to edit /etc/group, the rapid proliferation of groups that would result from this attempt to use a weak model to simulate a stronger one would quickly run into problems with scalability and manageability.
In short, you just can't make rwxrwxrwx do ACLs' job. The current permissions system is an artifact of a very time- and environment-specific tradeoff between functionality and resources (space, processor time).
Simply wrong. Read/write/execute gives eight combinations, and user/group/world only allows three to be expressed simultaneously, and expressiveness is further limited by the fact that most users can't add/delete/modify groups to suit their purposes. ACLs, by contrast, can be used to express all eight possibilities at once, and apply each to arbitrary sets of users.
Anyone who ever studied even one minute of logic can see that the two systems are not equivalent.
And how, exactly, would you make it easier? In particular, how would you make it easier without sacrificing the full power and flexibility of extensions?
I think the real problem is with terminology. It's not a matter of "inner loops" vs. "outer loops". It's about performance-critical parts vs. non-performance-critical parts. For most scripts and tools and anything else short of full-blown industrial strength applications, coding any part of your Python application as an extension module is just stupid. In those cases where it might make sense, it's really not that hard to apply standard software-engineering approaches to the design of one or more extension modules which neatly encapsulate the performance-critical parts of your app behind a fairly simple interface. I've done it several times, and been quite satisfied with the results...and it's not because I'm particularly easy to satisfy. ;-)
That's a pretty tall order. C/C++ interfaces aren't exactly self-describing, even as far as number of arguments in many cases, let alone type and meaning etc. There's actually a Python extension module - forgot the name - that lets you do pretty much the sort of direct invocation you're talking about, putting all of the burden on the programmer to make sure they're passing the right number of arguments etc. It turns out not to be all that useful, though, because it really only provides part of a real extension interface - the "interpreter calls X" part. A full extension interface also allows X to call the interpreter, or X to interpret or manipulate or even create complex structures - objects, lists, dictionaries, aggregates of same - comprehensible to the interpreted code, and so on. Without all of that, which is not available in the model you propose, what you have is a crippled extension interface lacking many benefits inherent in the real thing.
This sounds pretty much like what "freeze" does, or perhaps a minimally-enhanced version of it. Could you explain what exactly you want that goes beyond that?
BTW, keep in mind that creation of a single monolithic executable is not always the goal. Many ways of doing what you describe would preclude component models or interactive use. It's just one more example of how Python represents a certain set of tradeoffs. It's not perfect for every usage, but the omission of certain features you might want might well be a conscious decision based on the fact that their inclusion might have precluded some other feature that is more generally useful.
And I suppose this is in contrast to the Perl weenies who come into Python articles to bash Python, based on their many many hours spent studying the pros and cons of each? Yeah right. Most Perl folks spend about three minutes looking at Python, notice maybe one thing about the language itself - usually the indentation thing - but mostly just notice that it doesn't look like Perl, and then spend the rest of their lives making the illogical claim that Perl must be better because more people use it.
Python has its weaknesses, which I have gone out of my way to describe in my last post on the subject (a copy of which pretty easy for any non-moron to find on my website), but overall it's a fine language. As with Ruby, and Pike, and several other languages, on technical merit it deserves at least equal consideration to Perl.
I feel pretty much the same way. Ruby seems to be a step forward compared to Python in some respects, and a step backward in others. Overall it's a wash, and since Python already works very well for me I don't think I'd gain much by learning Ruby. That doesn't mean, though, that I would discourage anyone who doesn't already have a scripting language under their belt from choosing Ruby over Python. From what I can tell, they would learn better habits from Ruby than from Perl or Tcl.
Wrong. John Ousterhout, creator of Tcl, is a CS professor. Guido van Rossum, creator of Python, is well versed in language design. You may not like their decisions - I personally detest Tcl - but I would bet either of them is about 10x more qualified to design a language than anyone here.
Larry Wall, on the other hand, has no discernible background or ability in language design, and it shows in Perl.