But there isn't just raw computing power in play here. There is also the IO requirements, memory requirements, etc. That's the beauty of grid computing--by distributing the load you can increase the the throughput of the entire system, not just an individual component.
Like you, I've seen a lot of people freak out when one mentions using NFS. I've come to the conclusion that either a) these people didn't know what they were doing or b) were using an incredibly broken/fragile NFS implementation.
What annoys me in particular about the latter is that sometimes these implementations evolve from fragile to usable or good enough or whatever. But some admins seem to be unable to comprehend that things improve and stick to some home grown process without looking to see what has changed outside their world. They could spend less time re-applying bandages and more time doing fun things. [Like log processing with Hadoop.:) ]
Of course, there are the usual complaints about NFS security from folks who didn't know that NFS had a spec for Kerberos support or their OS-of-choice didn't support it. All hail NFSv4 for making Kerberos support a requirement...
That is a very good point. I'm used to dealing with scales beyond a single node;) where you have access to such things.
In any case, I'm considering borrowing the idea and using it to 'watch' blocks on HDFS. I think it would be interesting to have a visual of blocks/files getting read/written/replicated. It might show patterns that we're otherwise not seeing.
Why use ssh + tail -f when one can send the output to a centralized syslog server? There isn't any need to setup an account, keys, etc. when you can have the individual servers consolidate the data for you.
Having worked with AIX 5L (as well as 1.x on PS/2, 3.x on RS6k and 4.x on PPC), my impression of the 'L' part was mainly "Look! We have rpm! Now you can install OpenSSH and a bunch of other freeware from this companion cd! See how Linux compatible we are!" Junior admins are still "stuck" with SMIT until they get enough experience to learn all of the funky commands that AIX requires you to learn in order to actually manage the system from the shell.
While it is nice to see IBM (slowly) moving away from the ODM, AIX is hardly setting the world on fire with innovation. Just looking at the AIX Beta 6 page posted in this thread. Comparing the listed features is... laughable. IBM is *clearly* playing catch up.
Workload Partitions sounds like Zones
RBAC==RBAC in Trusted Solaris, moved to Solaris 8 in 2000
probevuew==dtrace (but is probevuew as good?)
Is AIX the only OS that doesn't have a working NSCD or graphical install at this point?
Trusted AIX==Trusted Solaris (15 years?) and its replacement Trusted Extensions for Solaris
Secure By Default==Secure By Default as added in Solaris 10 Update 3
SMIT has always been superior to the equiv. Sun utilities, but putting a web interface on it should have happened in 1995. Now it just looks like they are copying Sun, which has had some form of a web console for a while now. The "AIX Security Expert" LDAP template thing sounds interesting. I wonder just how comprehensive it really is though, especially considering how glued on Kerberos always felt. [Why did it take NFSv4 for IBM to add RPCSEC to NFS? That should have happened *years* ago!]
Of the list, I'd love to have some IBM gear to play with the installation revamp. [I've since changed jobs and no longer do AIX... almost pure Linux shop now.] Doing a fresh install of AIX was always painful and NIM is probably the worst P.O.S. ever compared to pretty much every other network installation system. I always wanted IBM to open source that so it could be replaced with something decent and cross platform, rather than the seemingly black box that it is.
I thought Sun paid Pervasive for their distribution in Solaris 10. Has this changed or by "core developers" do we really mean "people who commit the pkgs to the WOS" or "people who actually work at Sun who also just happen to work on Pg in their free time"?
I'm surprised your list doesn't include the inability to evacuate a drive. This is one of the biggest problems with ZFS right now. Makes it a real pain to upgrade drives.
Implementing MapReduce is much easier these days: just install and contribute to the Hadoop project. This is an open source, Java-based MapReduce implementation, including a distrbuted filesystem called HDFS.
Even though it is implemented in Java, you can use just about anything with it, using the Hadoop streaming functionality.
... and in the mean time, they are losing old and potentially new customers to Linux because they are too busy screwing around with things that "someone might use someday" (as the person who gave the presentation about raising the limits on the number of nics said) vs. revamping and upgrading other critical toolsets. [C'mon: This is 2007. Why isn't rsync being bundled yet? Jumpstart is great and all, but why can't I mirror *and* patch without having to worry about blowing up my root filesystem or doing crazy things in finish scripts? Etc, etc, etc.]
As I stated, I was thinking primarily of home directories. DB's are a slightly different beast though.;)
I'd hope as a former DBA he's aware of the impact of just adding storage to system... because it is rarely "free" after you take into consideration the costs of things like hardware maintenance, backup/recovery, redundancy, etc. But I'm sure he was looking at it from the application's perspective. [Where, yes, using the space to store more tables/indices/whatever is generally preferred to conserving space if there is a big application payoff.]
I was also thinking of an instance when a fairly well known kernel engineer was complaining about IT's inability to deploy some rather complicated service (Kerberos? something like that) to all of the large company we worked for (300 or so server rooms globally, many with paper cups and string for connectivity). After all, he was able to do it on his home network... Argh.
A long, long time ago I was the primary SA for a building that housed one of the top projects at a well known software company. As part of my routine, I would walk the hallways, talking to the admin. assistants and a handful of Important Users to get an idea of what was happening in their world. [After all, if I knew what they were doing, I could make sure that I didn't schedule an outage right in the middle of them doing major work. So, unlike a lot of my brethren, I don't mind mingling with the masses.] One day on this walk, a person called me into his office and said:
"I need for you to do task XYZ. This is for project ABC so it has the highest priority."
I reminded said user that he needs to file a ticket ASAP, but I'll get to it as soon as I can.
I continue my stroll down the hallway (my office was in another building, so I couldn't run to my desk if I wanted to) and was stopped by another user. He said:
"I need for you to do task LMN. This is for project ABC so it has the highest priority."
Same thing: reminded said user that he needs to file a ticket ASAP, but I'll get to it as soon as I can....
As a bit of an epilogue: Months and months of walking this hallway had taught me one thing--this set of users in that hallway was particularly abusive of the SA team. Anything and everything was an emergency and they were highly abusive to junior SAs. I had had it. So one day when nearly the same scenario happened, I responded a bit differently. I went down the hall, grabbed user 123 and took him to the last user's (let's say 567) office.
User 123 has task XYZ for project ABC. User 567 has task LMN for project ABC. You are both on project ABC. You both say that your tasks have the highest priorities. I'm going to request that you sort this out and let me know which one is the actual highest priority. If it makes any difference, I'll be having a chat with the big kahuna for this project on my way out.... and I plan on asking her opinion as well. Give kahuna's admin a call to let her know your decision.
... and left. As everyone knows, the admin. assistants are the real power brokers in large companies. This one was no different.
By the time I got to the head kahuna's admin. assistant, she told me that both users didn't actually need those things done after all. I also noticed that the users were a bit more realistic in setting priorities for tasks as well....
I think Woz's post tells us what a lot of us already know: Just because you're "technical" doesn't mean you can be a "high-end" administrator or understand the difficulties/nuances of "scaling up".
It reminds me of many, many, many conversations I've had with programmers, qa, etc, over the years where they tell me what they perceive to be the solution to the problem without really understanding either the long term impact or other factors. [I'm sure we've all heard the "disks are cheap" line when someone has filled their home directory with crap.]
Age of Mythology is five years old, yet I'm still playing several games of it a week. I was a bit disappointed that they decided to do AoE III instead of a follow-on to AoM. I love the fact that it has Norse and Egyptian mythologies in addition to your studied-to-death-in-American-schools Greek. Hopefully an AoM II will appear and have Sumerian or American Indian or Chinese or some other non-typical pantheons.
I suspect the UK version of "The Apprentice" is based off the US show of the same name that has been running for several years now. The "boss" is Donald Trump in our version.
... which is rather ironic considering that you had issues explaining why a US design wouldn't work in the European market. That doesn't appear to be true for television, at least.:)
While I realize you specifically asked about Linux, it is probably worth pointing out that NetBSD has been used as an embedded OS on ARM for quite a while. See NetBSD's embedded page for more information.
Sun's history and reputation on the Mac with things not related to the JVM is pretty awful. This has as much of a chance of seeing the light of day in a usable format as their version of Watson. Give it a year and by then Sun will have 'realigned their priorities' via reorg or a RIF will have wiped out the group that is working on this.
I parsed that as intending to mean that the Evans Data report splits the various Oracle DB offerings (Enterprise Edition, Standard Edition, etc) into separate products for purposes of calculating market share. Of course, that begs the question of how they are calculating the MySQL installations. Does usage under the commercial/enterprise license count differently than the open source version?
Sun employees generally get Solaris as a desktop via SunRay. The folks who install Linux, in general, are a) working on cross-platform products and b) do it outside IT's support services.
Slashdot Mirror
Large scale log processing isn't hard if you have the right tools. :)
But there isn't just raw computing power in play here. There is also the IO requirements, memory requirements, etc. That's the beauty of grid computing--by distributing the load you can increase the the throughput of the entire system, not just an individual component.
What annoys me in particular about the latter is that sometimes these implementations evolve from fragile to usable or good enough or whatever. But some admins seem to be unable to comprehend that things improve and stick to some home grown process without looking to see what has changed outside their world. They could spend less time re-applying bandages and more time doing fun things. [Like log processing with Hadoop. :) ]
Of course, there are the usual complaints about NFS security from folks who didn't know that NFS had a spec for Kerberos support or their OS-of-choice didn't support it. All hail NFSv4 for making Kerberos support a requirement...
In any case, I'm considering borrowing the idea and using it to 'watch' blocks on HDFS. I think it would be interesting to have a visual of blocks/files getting read/written/replicated. It might show patterns that we're otherwise not seeing.
Why use ssh + tail -f when one can send the output to a centralized syslog server? There isn't any need to setup an account, keys, etc. when you can have the individual servers consolidate the data for you.
Hopefully those of us at Yahoo! working on Hadoop (and its related project Pig) can help change this perception.
Linux is already running and certified for Niagara.
Hardly.
Having worked with AIX 5L (as well as 1.x on PS/2, 3.x on RS6k and 4.x on PPC), my impression of the 'L' part was mainly "Look! We have rpm! Now you can install OpenSSH and a bunch of other freeware from this companion cd! See how Linux compatible we are!" Junior admins are still "stuck" with SMIT until they get enough experience to learn all of the funky commands that AIX requires you to learn in order to actually manage the system from the shell.
While it is nice to see IBM (slowly) moving away from the ODM, AIX is hardly setting the world on fire with innovation. Just looking at the AIX Beta 6 page posted in this thread. Comparing the listed features is... laughable. IBM is *clearly* playing catch up.
Workload Partitions sounds like Zones
RBAC==RBAC in Trusted Solaris, moved to Solaris 8 in 2000
probevuew==dtrace (but is probevuew as good?)
Is AIX the only OS that doesn't have a working NSCD or graphical install at this point?
Trusted AIX==Trusted Solaris (15 years?) and its replacement Trusted Extensions for Solaris
Secure By Default==Secure By Default as added in Solaris 10 Update 3
SMIT has always been superior to the equiv. Sun utilities, but putting a web interface on it should have happened in 1995. Now it just looks like they are copying Sun, which has had some form of a web console for a while now. The "AIX Security Expert" LDAP template thing sounds interesting. I wonder just how comprehensive it really is though, especially considering how glued on Kerberos always felt. [Why did it take NFSv4 for IBM to add RPCSEC to NFS? That should have happened *years* ago!]
Of the list, I'd love to have some IBM gear to play with the installation revamp. [I've since changed jobs and no longer do AIX... almost pure Linux shop now.] Doing a fresh install of AIX was always painful and NIM is probably the worst P.O.S. ever compared to pretty much every other network installation system. I always wanted IBM to open source that so it could be replaced with something decent and cross platform, rather than the seemingly black box that it is.
If Sun is copying IBM, where is "OpenAIX"?
I thought Sun paid Pervasive for their distribution in Solaris 10. Has this changed or by "core developers" do we really mean "people who commit the pkgs to the WOS" or "people who actually work at Sun who also just happen to work on Pg in their free time"?
I'm surprised your list doesn't include the inability to evacuate a drive. This is one of the biggest problems with ZFS right now. Makes it a real pain to upgrade drives.
Even though it is implemented in Java, you can use just about anything with it, using the Hadoop streaming functionality.
Everyone knows Cory Doctorow wears a red cape and goggles.
... and in the mean time, they are losing old and potentially new customers to Linux because they are too busy screwing around with things that "someone might use someday" (as the person who gave the presentation about raising the limits on the number of nics said) vs. revamping and upgrading other critical toolsets. [C'mon: This is 2007. Why isn't rsync being bundled yet? Jumpstart is great and all, but why can't I mirror *and* patch without having to worry about blowing up my root filesystem or doing crazy things in finish scripts? Etc, etc, etc.]
As I stated, I was thinking primarily of home directories. DB's are a slightly different beast though. ;)
I'd hope as a former DBA he's aware of the impact of just adding storage to system... because it is rarely "free" after you take into consideration the costs of things like hardware maintenance, backup/recovery, redundancy, etc. But I'm sure he was looking at it from the application's perspective. [Where, yes, using the space to store more tables/indices/whatever is generally preferred to conserving space if there is a big application payoff.]
I was also thinking of an instance when a fairly well known kernel engineer was complaining about IT's inability to deploy some rather complicated service (Kerberos? something like that) to all of the large company we worked for (300 or so server rooms globally, many with paper cups and string for connectivity). After all, he was able to do it on his home network... Argh.
While we're sharing anecdotes:
... and left. As everyone knows, the admin. assistants are the real power brokers in large companies. This one was no different.
A long, long time ago I was the primary SA for a building that housed one of the top projects at a well known software company. As part of my routine, I would walk the hallways, talking to the admin. assistants and a handful of Important Users to get an idea of what was happening in their world. [After all, if I knew what they were doing, I could make sure that I didn't schedule an outage right in the middle of them doing major work. So, unlike a lot of my brethren, I don't mind mingling with the masses.] One day on this walk, a person called me into his office and said:
"I need for you to do task XYZ. This is for project ABC so it has the highest priority."
I reminded said user that he needs to file a ticket ASAP, but I'll get to it as soon as I can.
I continue my stroll down the hallway (my office was in another building, so I couldn't run to my desk if I wanted to) and was stopped by another user. He said:
"I need for you to do task LMN. This is for project ABC so it has the highest priority."
Same thing: reminded said user that he needs to file a ticket ASAP, but I'll get to it as soon as I can....
As a bit of an epilogue: Months and months of walking this hallway had taught me one thing--this set of users in that hallway was particularly abusive of the SA team. Anything and everything was an emergency and they were highly abusive to junior SAs. I had had it. So one day when nearly the same scenario happened, I responded a bit differently. I went down the hall, grabbed user 123 and took him to the last user's (let's say 567) office.
User 123 has task XYZ for project ABC. User 567 has task LMN for project ABC. You are both on project ABC. You both say that your tasks have the highest priorities. I'm going to request that you sort this out and let me know which one is the actual highest priority. If it makes any difference, I'll be having a chat with the big kahuna for this project on my way out.... and I plan on asking her opinion as well. Give kahuna's admin a call to let her know your decision.
By the time I got to the head kahuna's admin. assistant, she told me that both users didn't actually need those things done after all. I also noticed that the users were a bit more realistic in setting priorities for tasks as well....
I think Woz's post tells us what a lot of us already know: Just because you're "technical" doesn't mean you can be a "high-end" administrator or understand the difficulties/nuances of "scaling up".
It reminds me of many, many, many conversations I've had with programmers, qa, etc, over the years where they tell me what they perceive to be the solution to the problem without really understanding either the long term impact or other factors. [I'm sure we've all heard the "disks are cheap" line when someone has filled their home directory with crap.]
Age of Mythology is five years old, yet I'm still playing several games of it a week. I was a bit disappointed that they decided to do AoE III instead of a follow-on to AoM. I love the fact that it has Norse and Egyptian mythologies in addition to your studied-to-death-in-American-schools Greek. Hopefully an AoM II will appear and have Sumerian or American Indian or Chinese or some other non-typical pantheons.
I suspect the UK version of "The Apprentice" is based off the US show of the same name that has been running for several years now. The "boss" is Donald Trump in our version.
... which is rather ironic considering that you had issues explaining why a US design wouldn't work in the European market. That doesn't appear to be true for television, at least. :)
While I realize you specifically asked about Linux, it is probably worth pointing out that NetBSD has been used as an embedded OS on ARM for quite a while. See NetBSD's embedded page for more information.
Sun's history and reputation on the Mac with things not related to the JVM is pretty awful. This has as much of a chance of seeing the light of day in a usable format as their version of Watson. Give it a year and by then Sun will have 'realigned their priorities' via reorg or a RIF will have wiped out the group that is working on this.
I parsed that as intending to mean that the Evans Data report splits the various Oracle DB offerings (Enterprise Edition, Standard Edition, etc) into separate products for purposes of calculating market share. Of course, that begs the question of how they are calculating the MySQL installations. Does usage under the commercial/enterprise license count differently than the open source version?
It should probably be noted that Grant has a BS in EE from USC.
... they are the same people that don't allow plus signs (+) in email addresses, despite being completely and totally valid.
Sun employees generally get Solaris as a desktop via SunRay. The folks who install Linux, in general, are a) working on cross-platform products and b) do it outside IT's support services.