CentOS Administrator Reappears

str8edge sends word that Lance Davis, the CentOS project administrator who had mysteriously gone absent, has now returned and is working with the development team to get things back on track. From their announcement: "The CentOS Development team had a routine meeting today with Lance Davis in attendance. During the meeting a majority of issues were resolved immediately and a working agreement was reached with deadlines for remaining unresolved issues. There should be no impact to any CentOS users going forward. The CentOS project is now in control of the CentOS.org and CentOS.info domains and owns all trademarks, materials, and artwork in the CentOS distributions. We look forward to working with Lance to quickly complete all the agreed upon issues. More information will follow soon."

Comments

Appalachian Trail

[yoghurt]

Hiking that Appalachian trail can be tricky. I hear it goes all the way to Argentina.

Re:Appalachian Trail

[Anonymous Coward]

2. Kidnapped by space aliens; managed to escape when they neglected to secure the Dilithium Crystal Hatch.

3. My grandmother died. No, not the one that died six months ago, or the one that died a year before that; this was my *biological* grandmother.

4. Didn't realize the batteries on my beeper died.

5. Met an old classmate from Yale, who gave me GHB and tried to induct me into the Skulls organization. Managed to escape by commandeering a single shell and out-rowing their eight-man shell.

6. Just came back from the Tour de France; found it tough to compete when you refuse doping.

7. Dog ate the Post-It notes.

8. Just came back from central Africa, where I was building makeshift shelters and administering medical supplies in the fight against AIDS, tuberculosis, and malaria. How was the last couple weeks for y'all?

More likely

[Ritz_Just_Ritz]

Lance realized this very public oops wasn't going to do anything for his future employment prospects. A shame it had to come to that, but sometimes you need to upgrade from a feather to a cattle prod to get results.

Re:More likely

[Anonymous Coward]

Me neither! My mission critical servers are too important to trust to open source. I am switching everything to Windows.

Re:More likely

[PixelSmack]

I totally agree, unfortunately i administer a few centOS boxes at work. This will be bought up and i will argue for moving to a more open community distro which is a shame because i quite like centOS - however if it can not be relied upon like that it just looks bad.

Re:More likely

[Vellmont]

And CentOS relying on one person for as much as seemingly their very existence (by their own tone over this issue) has absolutely guaranteed that I will never use CentOS for anything important.

If by "CentOS" you're talking about the Centos.org domain and some IRC channels, you're right. If by "CentOS" you mean updating and developing the operating system, you're wrong. Any open source project is always about the developers behind it. There are many developers involved in this project, and the project itself isn't dependent any any one of them.

My guess is the thing you care about is the OS and not a domain name. Drawing conclusions from tone and not facts is just a bad practice in general.

Re:More likely

[sabernet]

"relying on one person for as much as seemingly their very existence"

Ssssh! Do you want to start a flamewar with the Apple fans too?

....sorry, had to :)

Re:More likely

[operator_error]

The philosophy that has been applied to Debian development has served it well over the years. Consider using either it, or a derivative like Ubuntu. Since I have chosen this path, I've had no regrets.

This is a complete debacle for CentOS.

Re:More likely

[Attila Dimedici]

And CentOS relying on one person for as much as seemingly their very existence (by their own tone over this issue) has absolutely guaranteed that I will never use CentOS for anything important. This entire thing should never have been news in the first place for two reasons: 1. If the health of the company and their product is absolutely dependent on the well being of Lance, then they should have done everything they could to keep this story quiet, as it is embarrassing. 2. A cranky engineer screwing off for a few days is common enough that it was a non-story to begin with.

Ok, I'm not real familiar with everything that is going on. However, it appears that this happened as a result of the rest of the CentOS development team pushing Lance to work with them in setting things up so that they weren't absolutely dependent on him. He appears to have been resisting this step.
Rather than saying, "Too bad, CentOS is my baby and I'm not giving up control" he appears to have said, "Yeah, you're right. We need to have backups and I'll get you an accounting of the money we've raised." Then he never took any of the steps that would have allowed that to actually happen. When the phone calls and emails from the rest of the development team got insistent, he just stopped answering the phone or his emails.

Re:More likely

[Linker3000]

Do you not think that the issues at the heart of your (very valid) concerns are now being addressed - albeit a little later than they should have been?

I think the situation with CentOS's command and control structure merits monitoring for a short while to see how things settle down.

FWIW I have around 10 servers running various versions of CentOS and am keeping an eye on developments.

Re:More likely

[Anonymous Coward]

You know, RedHat ES is only $349 a year. You could just migrate to RedHat ES and enjoy full support while still having the same features and environment as CentOS...

Re:More likely

[Macka]

1. If the health of the company and their product is absolutely dependent on the well being of Lance, then they should have done everything they could to keep this story quiet, as it is embarrassing.

They did. Washing your dirty laundry in public is never pleasant, but in this case they needed to find a way to get Lance to engage and had run out of options. Shining a public spotlight on him seems to have done the trick, so it was the correct move.

2. A cranky engineer screwing off for a few days is common enough that it was a non-story to begin with

They've been trying to resolve this quietly for about a year and they were getting no where.

And CentOS relying on one person for as much as seemingly their very existence (by their own tone over this issue) has absolutely guaranteed that I will never use CentOS for anything important.

Hm, I smell the fresh scent of manure in the air. From your tone I'd bet that you never have used CentOS for anything important, or you wouldn't be so quick to give it up. Not that this is going to be an issue for much longer, which makes your objection pointless.

Re:More likely

[jjohnson]

What's not to rely on? The distribution itself was never in danger. The only thing Lance controlled was the domain name, some IRC channels, and the PayPal account. Now Lance has handed those things over, and they'll move forward with a foundation to control the project.

Re:More likely

[Anonymous Coward]

you should know that when /.'ers say things like "I'd never use that for anything important" they mean either they wouldn't install it on the slack box they have set up with 6 years uptime on it or they would never use it to find/download/store pr0n.

It's been my experience that most people who say things about not trusting this or that to mission critical production environments are not actually in any position to chose what happens in a production environment or are in fact not in need of any sort of production environment being as they don't actually *do* anything.

Re:More likely

[MoralHazard]

OK, how the hell did the parent poster get modded to +5, informative? He has the wrong facts on virtually EVERY important point. He could have read yesterday's Slashdot. article, or just Googled the damn story, but I guess he didn't have the time.

Let's correct his factual problems, shall we?

  1) The CENTOS organization is not a "company", nor is the distro a "product". It's an informally-organized open source project, and the Linux distro they produce isn't sold or supported for profit by the project, itself. (There are many other companies that do provide CENTOS support contracts, though, and some of the developers may own/work for some of those companies.)

  2) The health of the CENTOS distro and organization were never "absolutely dependent" on Lance Davis. He controlled the project's domain name registration, the Google AdWords account, and a few other important resources. But these were inconveniences, at worst: Had Lance not responded to the open letter, the rest of the developers would simply have registered a new domain name, set up new repos/wikis/blogs, and copied the project data over. Lance would have been forgotten as the speed bump that he was. (And if Lance breaks his promises AGAIN and fails meet the latest deadlines, this is what we'll see happen.)

  3) Lance Davis didn't "screw off for a few days"--over the course of a year, he repeatedly made and broke promises, and failed to either provide accounting for the project's finances or to turn the relevent logins over to other group members. Then, he just stopped returning phone calls and emails, and he quit attending real-life and IRC meetings. Meanwhile, the Google AdWords account was raking in a few thousand dollars per month, and to all outward appearances, it looked a lot like Lance was just taking it for himself.

THE REAL STORY:

The lack of a formal structure (a la nonprofit incorporation, like Fedora or Debian) seems to be CENTOS's biggest problem, and the community's perception of this dispute does cloud the project's future. But like any open-source project, it's impossible for one person to be anything more than an inconvenience.

  * The source code repos and packages are globally mirrored by dozens of independent organizations, and Lance Davis never had control over any of them.

  * Domain registration, hosting, and such are cheap--even if some of the AdWords money were misappropriated, the developers could still pass the hat and/or offload bandwidth to the mirror providers. Hell, they could always move to SourceForge for free, if they were really desperate.

  * As long as the CENTOS core community learns its lesson, here, they can recover and grow stronger than before. They need to incorporate as a not-for-profit foundation, establish a board of directors, executive roles, accounting practices, and all the other structural crap that goes with it. It's not a trivial amount of work (in the US, at least--I don't know about the UK), but this episode demonstrates why successful, influential, long-lived F/OSS groups like the GNU, Debian, etc. have all decided to go this route.

Finally, on a personal note, I would like to ask yttrstein why he feels compelled to burden the rest of us with his un-informed opinions on this topic. He could have easily researched the issue, in about 5 or 10 minutes, and perhaps contributed something worth reading.

Re:More likely

[FreakyGreenLeaky]

Finally, on a personal note, I would like to ask yttrstein why he feels compelled to burden the rest of us with his un-informed opinions on this topic. He could have easily researched the issue, in about 5 or 10 minutes, and perhaps contributed something worth reading.

He's probably 12 years old or 45 and lives in his mom's basement. His mindless gurgle resulted in your interesting post, so perhaps he'll learn something before his 8pm scheduled online/pr0n wank session...

Re:More likely

[jez9999]

Finally, on a personal note, I would like to ask yttrstein why he feels compelled to burden the rest of us with his un-informed opinions on this topic. He could have easily researched the issue, in about 5 or 10 minutes, and perhaps contributed something worth reading.

You must be new here.

Re:More likely

[jfp51]

Yes because the switch-over to whatever the 'new' distribution would have been called would have been so complicated, I mean the developers even stated in the open letter that: "(...)hot machines exist to allow for a cutover with a simple one time installation of one RPM package." Debacle indeed...

Re:More likely

[rohan972]

Amen to that. I now feel have no choice but to spurn CentOS as I would spurn a rabid dog.

With a load of buckshot? That's a bit harsh!

Did he...

[jsse]

reappear in front of the team one day, with bloodstain and mud all over his body, and yelled "I'm single, AGAIN!".

Two weeks

[MaizeMan]

He'd been invisible for more than two weeks. Once you're in a position of responsibility like that the longest you can disappear without making prior plans is maybe a long weekend. Which sucks because sometimes you're going to want to crawl into a hole and ignore what has gone wrong with the world but you don't have that freedom when people are counting on you.

Re:Two weeks

[Macka]

No. According to the Open Letter from Ralph Angenendt that kick started all this Lance dropped off the CentOS radar sometime in 2008.

Re:Two weeks

[GaryOlson]

I don't think the CentOS revenue sources or computing facilities are significant enough to finance a small CIA democracy promotional action.

I needed something to cheer me up

[Linker3000]

Good news.

Here I am in my sickbed writing rsync scripts for cross-site backups between CentOS-based servers, and seeing the headline made me smile, in-between fits of coughing.

If by some amazing chance Karanbir Singh see this - I promise to rack up the dual Itanium server for IA64 testing and dev as soon as I get back to work and clean up a few other outstanding issues.

Re:I needed something to cheer me up

[kbsingh]

Hey,

Awrite then, let me know when that IA64 machine comes online :D)

and hope coughing isnt too manic.

Re:I needed something to cheer me up

[Linker3000]

Ha - thanks for that. I'll live!!

Signed off at the moment so lots of time to 'do stuff' at home.

Two other 'urgent' projects crashed my plans for the Dell server but I'll let you know when I get back on track.

Glad to see things seem to be moving in the right direction for you and the rest of the core team.

Great way to piss off LTS userbase.

[rikrebel]

This whole story is unnerving.

CentOS is widely used in datacenters due to it's red-hattyness, it's Long Term Support, and conservative adoption of whizbang.

It's by far my favorite distrobution for important servers.

I have already had two meetings over this and had my team start their proposals for alternate LTS distros and a migration plan. I am sure I am not the only one.

If the CentOS project manages to remove this single point of failure I think confidence will return. But I think I'll keep my projects going for a while just in case.

Re:Great way to piss off LTS userbase.

[jjohnson]

Isn't that just what this article is about? Lance Davis is AWOL for almost a year, the rest of the project publishes an open letter, Davis shows up and hands over the keys. What more resolution is needed?

In a way, this gives me some more confidence in CentOS, insofar as the rest of the admins were willing to "break glass in case of emergency" and deal with Davis' erratic leadership. They spent a long time trying to deal with it quietly and internally, but when it came down to it, they basically removed him the way all OSS projects end up doing it, with public pressure.

CentOS, FOSS, and leadership problems.

[cenc]

Everyone will jump on this as proof that open source projects can not be trusted or relied on. Now, that may or may not be true. This instance really is not a poster child for problems with FOSS projects. We are talking about a project based on repackaging and rebranding a commercial distro. The heavy lifting is done by RH and other projects.

This should be food for thought however about other projects, which there are many many instances of FOSS project management issues leaving users high and dry because of political issues.

We really need some better organizational standards for FOSS project management, not just high quality code. Remember the segment of society we are talking about. They might be great at programing or whatever, but they rarely have the leadership and organization skills to handle a project once it reaches a critical mass of popularity or use.

One of the first things I have to do, after years of using FOSS, is look at the project and see how healthy it is before deciding to implement it in my biz. I have to do things like look at how many projects have derived work from it, who is contributing to it, how alive is the forum community both for developers and users, development cycles, and so on.

What we really need is some sort of organizational certification. Something that an end user of FOSS or other FOSS project can with one glance determine what is the status of the organization and the project. Especially the large important ones. Are there for example policies in place to handle the death of the head of the project? Is there a formal system for order of succession? Is there policy for archiving legacy code and related information?

The worse thing that can happen to a FOSS project is a cult of personality forming around just one person ( that is more than just PR).

Re:CentOS, FOSS, and leadership problems.

[cenc]

Really?

Yes, corporations have these problems also. When they don't deal with them, they go under. There is a reason why corporations sink so much time and money in to insuring they don't happen.

These problems however are not so much similar to the problems you find in companies, but problems you find in none-profit organizations of any stripe. Places where ego is basis for much of the personal incentive for getting involved. Spend some time on your average neighborhood NGO board of directors, and you will see the very similar things happen to their projects.

Re:CentOS, FOSS, and leadership problems.

[QuarkofNature]

At my company, we vet the software we use, both proprietary and FOSS, prior to using it in our systems. I think you raise some good points here...one thing I have observed is that our "standards" for doing this analysis are very commercial-company-centric.

The folks who do trade studies "get" how to look at company financials, strength, size, etc., to ensure that we aren't going down a bad path with a piece of proprietary software. Yet, in most cases, I see people at a loss of how to do equivalent analysis for FOSS products. It might be surprising to some people around here, but many still don't grasp just how different a developer-and-user community for a product is, compared to a corporation that produces software. And even for those of us who do understand the differences, it's still sometimes tricky to do a fair comparative analysis.

Just as the OSI has tried to formalize what open source means, and helps vet licenses to make it easier for people wanting to use FOSS software, it might be very useful to come up with some standard measures of the health of FOSS projects, and start gathering that data in one place for popular ones.

Re:They better watch out

[FlyingBishop]

That's absurd, C# wasn't released until 2000.

Everyone knows the Reagan-bot's software was written in Lisp.

Re:I'm really glad to hear this!

[farrellj]

Because if you are supporting CentOS systems in the field, it is easier to do so with a system that is running the same OS, as it, at very least, provides a system you can experiment with. It also means you will have the roughly same software load, and you won't be used to running apps that are not on the server. As well, replicating your server on your laptop also means having a system you can replicate a problem with, even if you are travelling. Of course, now that a 4 Gig laptop is possible, you could be running that replica system in a Virtual machine...I've been running Slackware 64 on my Laptop, and it is running very nicely. My main server also runs Slackware, but 32 bit, and has been running for a few years, with a few updates...it also has an AMD64, so I may move to that as it's next update.

ttyl
          Farrell

Where's the money?

[WombatDeath]

People have been donating money to centos.org, presumably wishing to further the goals of the project. Is this money (plus the advertising revenue) still available for its intended purpose?

Not accusing anyone of anything, but this question is quite important and doesn't seem to be addressed in the update.

Re:Maybe they should update the frontpage....

[jroysdon]

The developers we and are available and that never changed. Lance hadn't contributed in some time, and was really just wearing the "founder hat" and keys to the centos.org domain, irc found status, and paypal account.

Re:nobody worries about Slackware like CentOS

[iggymanz]

but slackware will probably die when Patrick does

Re:I'm really glad to hear this!

[Ex-Linux-Fanboy]

You're hardly a software developer - you aren't willing to find solutions yourself

You know, I used to have this kind of attitude. Then I grew up.

Did you know Dennis Ritchie uses Microsoft Windows and Microsoft Outlook to read email and post to Usenet? Have you every thought about why?

The thing is this: everything works out of the box in Windows XP (well, except for the sound card, but the workaround is posted online and it about 15 minutes of bother to get going). I, at this point in my life, have better things to do with my time than to get things to work in Linux when they already work in Windows. Such as actually develop software.

This is the problem with the Linux community at Slashdot. It's a very immature and insecure community; when people mention they have problems and are using Windows instead because of those problems, people react with denial and attack the messenger instead of being mature and acknowledging the problems.

Excuse me, but I tried compiling various ALPS drivers in CentOS. I spent, oh, about 2 hours on it and, to make a long story short, it didn't work. If the Linux community wants to flame me instead of trying to help me (or, at least being civil), that's fine. Your message is clear: You don't want people using Linux. You want people using Windows XP. You do not want to make Linux a viable desktop operating system.

And, oh, about Ubuntu: It was very unstable for me, with constant crashes. I blogged all about it.

Thanks for playing.

Linux zealots piss me off.

Re:Argentina

[thejynxed]

Ahhhh, young grasshoppa'. You shall be enlightened!

A certain Republican Governor of South Carolina, Mr. Mark Sanford, claimed he was hiking the Appalachian Trail when in actuality, he was in Argentina getting his groove on with his Latina hoochie-mama. He's married, and was quite vocal about being "Pro-Family".

Hence all of the jokes, and why 9/11 Repugs holding public office are hypocrites of the first magnitude.

Re:Hate to break it to you...

[csartanis]

Did you stab your face yet? Inquiring minds want to know.