This is a lot of fun doing, and I don't really care if no one else ever uses it, we'll use it for our servers and workstations, so that's really all that matters. It's important for everything to work right, for me, not for a mysterious user base of anonymous people on the Internet. If they want to use it too, cool.
Users will probably spot differences, for better or worse.. Probably most people will spot that I have an insanely simplistic view of startup files (BSD style). I fully believe that if you're going to be using a *nix machine, you need to know how to at least edit a config file or two, so my config files are easy to read and edit, and I don't feel there's a real *NEED* to make fuzzy happy GUI's for every admin function.
We love upgrading hardware on our Linux servers at work. Pretty much, we yank the hard drive out of an old server, stick it in the new server, and turn it on. As long as we're going to newer hardware, it works fine. If the kernel we're running was compiled for a really wrong kernel, it won't boot, but that's easy enough to fix.
We've had the occasional server go whacky with a bad motherboard, and just moved the drives to a new machine, and away it goes.
One server we did recently, we upgraded the machine first, moving the drives to a new machine, and turning it back on. Then we upgraded the RAID on it, which consisted of copying all the files over to the RAID, and rebooting. Painless. The biggest headache with it was having the admin who was local to it wait for all the staff to go home, so they wouldn't bitch that it was down for the 10 minutes it took.
Hmm, what did the copy consist of?
While people were working we did:
(leaving the active mounts in place) mkdir/old mount/dev/hda1/old mount/dev/sda1/old/files mkdir/dev/sdb1/new/ cp -RPp/old/new
Then when we were ready, we locked all the users out, and to sync up all the changes for the last day or two we did:
rsync -av/old/new
(or something tremendously close to that)
The/dev/sda1 and/dev/sdb1 were the external arrays. They're seen by Linux (or whatever OS) as a single SCSI hard drive. Gotta love it.
Yes, the OS was on a single IDE drive. That system had grown since it was first implemented, with two IDE drives, and no concept of what it would eventually be used for.
What would it take to do the same thing on a large (hundreds of Gb) Windows fileserver? We didn't touch anything in the process, it all just worked. No redoing user permissions, no headaches. I believe the biggest problem was moving cabling, and changing the terminator on the SCSI chain.
Actually, the disk was WinXP home which included SP1. I don't have the disk or license handy, I think it got filed away with all the other disks and manuals that came with it.
But yes, it did take about 4 hours. And no, I'm not on a slow connection. That day I was on a 768Kb/3Mb DSL from PacBell/SBC. It was 4 hours of download, reboot, lather, rinse, repeat.
The technique for including 3rd party drivers is anything but new. It's exactly the same way I made an Adaptec AAA-130 (I believe) RAID card work on a machine years ago with WinNT 4.0. Far from improved.
Maybe I'm being obnoxious, but I fully expect when they release "better" versions of their software, I expect them to actually improve it. If they're releasing a newer version of XP, they could at least include the current drivers with the CD. It's not like I dug it out of a file drawer where I tossed it back in 2001. I bought it new at the store with the parts for the machine. Right along side it in the store were current copies of RedHat and Slackware (oddly enough), and either of those would have installed flawlessly. Actually, Slackware did.
Actually, that was a port completely done by TransGaming. They have mention of it on their site. It's only the original "The Sims" game, not all the expansion packs. They clearly say that the expansion packs won't work with it.
The distro was a special gaming edition of Mandrake, which cost $60. It would be well worth it, if she could use her expansion packs. She has all of them, 6 I believe, which at $50/ea makes for a substantial investment in a single game, which she isn't willing to give up.:(
TransGaming says that you *CAN* use the version of TheSims included with Mandrake on other versions of Linux, as long as you have the right libraries installed, which I can't imagine being very hard, since they're right on the Mandrake disks.:)
I like Slackware. It's not pretty, but on a brand new machine, it took just about 30 minutes to install, and I was done. Even with adding programs in that I wanted, it was about 45 minutes total.
With WinXP home (same machine), I had to find a floppy disk to make a driver disk to allow me to install onto the SATA hard drive (the only drive), then it took roughly two hours to do the full install, and then another 4 hours to do the service packs to get it to a stable position. Then there was adding in programs to get it to basic functionality.
And before anyone throws cost of ownership, ummm, Slackware was free. Well, I paid for the CD's, but that was nothing compared to buying WinXP.
The beginning of the WinXP is anything but pretty. It's the same text (with some colors) interface that WinNT used.. Amazing. State of the f***in' art..
I've installed plenty of other distros too. Very pretty installers, that all work. I've only hit the rare error, but nothing compared to the last WinXP install that I did. 5 minutes in, for the first couple tries, and then it stops because it doesn't believe there's a hard drive in the machine..
Freakin' SATA. It's been out for a few days now, maybe someone at Microsoft has heard of it. {sigh} Ok, I'll be willing to say that maybe it was the controller, but hey, it's a name brand controller, built into a name brand motherboard, and hey, Linux saw it. Linux, you know, the one Microsoft bashes for being just a bunch of hobbiests doing it. Well, why can the hobbiests do what the multi-billion company can't??
I tried to install the same copy of XP on the *SAME* machine, into a VMWare virtual machine (booted to Linux). Nope, no-go.. I know, it looks like a different machine.. But, why? Because I may be an evil software pirate? Or I may be Joe-user who changed his mind about the hardware I was installing, and swapped it all around.. I've been known to build a machine, and when I'm done, swap video and sound cards, or even motherboards. Oh, no, that's not permissible in the wonderful world of Microsoft. I have to call and ask for permission if I do that. I opted to *NOT* call Microsoft, and beg for permission to use software I paid for in a perfectly legitimate fashion. I installed an old copy of Win98, which is no longer used on any PC's here (*MY* copy, that *I* paid for), which doesn't bitch, and threaten to not work if I don't pay for extra licenses.
This particular machine happens to be my girlfriend's machine. She wanted Windows, so she could play "The Sims". I have to honestly say, Slackware ran a whole lot faster than XP, and this isn't a slow machine. AMD 2800+, 1GB ram, 200GB SATA drive. If only we could get game companies to recognize Linux is better, faster, and more stable, she wouldn't have even wanted Windows.
And, no, "The Sims" doesn't work under wine.:( But, it does crash very well after a bit of game play. I suspect problems with the software, not Windows, in this case.
That's what I run. If I want something, I usually go to the source, and install what they have to offer, rather than waiting for anyone to do anything for me. Why should I wait for a mainstream distro to patch the hell out of something, and distribute it to me with convoluted configuration options, when I can get it exactly like the author intended it? Slack makes this fairly easy. One of the first things I do when I install a Slackware system, is to download the new kernel sources, and compile it myself. Not that there's anything technically wrong with Slackware's kernel, I just like to know that my system is running exactly the way it should, without any extra drivers, or fluff.
We're building up a distro like this. It's going to be called "LMLinux". There's a really brief overview at http://lmlinux.com . We almost have the first alpha release done, but not quite. Our package management backend is just about complete, which looks rather nice.. Everythings stored in a MySQL database, for easy reference. It will make for very easy work, for anyone to browse to a few lookup pages, or for a package manager to find things.
I'm all about doing things from source, and doing it the way the author intended. But hey, if people like patchwork systems, let them get another distro with a kernel that has various patches reverse-engineered to work in the wrong version, etc, etc..
In our ideal world, you have the option of installing the package from our server, or grabbing it from the author yourself, and installing it yourself, without worrying that the next automagic update will hose whatever you've done.
I've done work for RedHat and Debian people, who absolutely freak out at the idea that I'd even think of downloading something myself, and installing it myself.
With all the complaints I see about x distro not having this, or not doing that, I kinda giggle. My Slackware installs do everything. Farther up in the thread is someone complaining about a lack of NTFS support by whatever distro. NTFS? Mine does it, I compiled it into the kernel, if there will be NTFS drives to read. I watch any video type I want, I've installed xine. It's not rocket science, read the readme.:)
And, you're absolutely right, you get all the free support you need in the mailing lists for the program, or checking on dejanews (eerr, groups.google.com)
Found the story on why they were shut down. Basically the insurance company wanted them to have a driver with two years experience and a class c commercial drivers license. The Coast Guard was requiring that they had a certified sea captain and deck hand on board. So every tour would require 3 extra people, and their salary would probably exceed the profit of the tour.
The fatalities were with another DUKW tour, in Arkansas.
I've seen one driving around Tampa Florida. It's run by Duck Tours . It's kind of wierd looking up, and seeing a bright yellow amphibious thing driving down the road.
For something as old as it is, I've only seen it broken down once. It's normal route went by my office in Tampa (when I worked there), and to their credit, there were always cute girls on it when it drove by.:)
I'm surprised no one corrected me on this. I was just writing something else and realized you can't fit 80 servers per rack.:)
1u = 1.75"
A standard rack is 70" tall.
You can fit 40 machines per rack. Knock 1u off for the switch, so 39 per rack, not 78. The price for servers and routers comes out the same, just twice as much floor space, which to us non-multi-billion-dollar companies costs money.:)
I'm fairly sure Google outright owns their facilities, so they don't have to worry about that square footage. That's just a little less space for the foosball room.
You'd be really surprised to see what we have. Ya, it's far less than a single leased machine in someone elses room. I've been tempted to put pictures online of the colo's and our equipment, but every time I have a decent set of photos together, we've already made a change at one of the colo's, and alas, no camera.
I haven't been back to New York, since another admin was up replacing machines, which is sad, it probably looks much prettier there now, than when I first set it up.:) I found a way to fit 12 4u machines in a cabinet, and still get the doors to shut.:)
So let me see if I got this right. You bothered to take the time to write your message, for one character in the wrong case? You probably read through the whole thread looking for someone to make that mistake too, didn't you?
gigabyte. happy?
"They are very smart offering the one gigabyte storage."
So, what's it feel like, being so right? Been looking for it for a while, haven't you? Feel good?
I'd have to assume that the smallest drives they have in operation are 20Gb, but I'd suspect they probably be smarter not to use extra storage across all those servers, they'd probably go with some large arrays.
Just looking at raw space, you could put 14 of those on a SCSI chain, and multiple cards in one machine (say 4 to keep things simple). For just $422,000, you could have 196Tb of space on a single machine. I'd strongly suspect they'd distribute it over a few more servers than just one.:)
They're very smart offering the 1Gb storage. It'll get users in, who don't like the limits. I've offered my users "unlimited" storage for years. They were on a 45Gb drive for a long time, but are now on a 330Gb array. With 246 users, they're only using 7Gb, and honestly, most of that space is my usage. I archive several busy mailing lists, for my own usage, and plenty of accounts that exist, "just in case" we need a message from it from years ago.:)
Google will see a very small number of it's users getting anywhere close to 1Gb. I'm sure there'll be plenty of/. readers hitting them in the first months, trying to fill their own boxes, but outside of that, it will only be very occasionally that.
One account we have, that simply collects spam sent to other accounts, took 7 months to collect 1.1Gb of mail. Hmmm, maybe I should purge that one. Nah, we still have 300+Gb free.:)
Every article I've read about Google's servers says they use "commodity" parts, which means they buy pretty much the same stuff we buy. They also indicate that they use as much memory as possible, and don't use hard drives, or use the drives as little as possible. From my interview with Google, they asked quite a few questions about RAID0, RAID1 (and combinations of those), I'd believe they stick in two drives to ensure data doesn't get lost due to power outages.
We get good name brand parts wholesale, which I'd expect is what they do too. So, assuming 1u Asus, Tyan, or SuperMicro machines stuffed full of memory, with hard drives big enough to hold the OS plus an image of whatever they store in memory (ramdrives?), they'd require at most 3Gb (OS) + 4Gb (ramdrive backup). I don't recall seeing dual CPU's, but we'll go with that assumption.
The nice base machine we had settled on for quite a while was the Asus 1400r, which consisted of dual 1.4Ghz PIII's, 2Gb RAM, and 20Gb and 200Gb hard drives. Our cost was roughly $1500. They'd lower the drive cost, but incrase the memory cost, so they'd probably cost about $1700, but I'm sure Google got better pricing, buying the quantity they were getting.
The count of 88 machines per rack is a bit high. You get 80u's per standard rack, but you can't stuff it full of machines, unless you get very creative. I'd suspect they have 2 switches, and a few power management units per rack. The APC's we use take 8 machines per unit, and are 1u tall. There are other power management units, that don't take up rack space, which they may be using, but only the folks at Google really know.
Assuming the maximum density, and equipment that was available as "commodity" equipment at the time, they'd have 2 Cisco 2948's and 78 servers per rack.
Lets not forget core networking equipment. That's worth a few bucks.:)
Each set of 39 servers would probably be connected to their routers via GigE fiber (I couldn't imageine them using 100baseT for this) Right now we're guestimating 1700 racks. They have locations in 3 cities, so we'll assume they have at least 9 routers. They'd probably use Cisco 12000's, or something along that line. Checking eBay, you can get a nice Cisco 12008 for just $27,000, but that's the smaller one. I've toured a few places who had them, and pointed at them citing them to be just over $1,000,000.
Google has a couple thousand employees, but we've found that our servers make *VERY* nice workstations too.:) Well, not the Asus 1400r, those are built into a 1u case, but other machines we've built for servers are very easy to build into midtowers instead. Those machines don't get gobs of memory, but do get extras like nice sound cards and CD/DVD players. The price would be the same, as they'd probably still be attaching them to the same networking equipment. 132,000 servers, and 2,682 workstations and dev machines is probably fairly close to what they have.
I believe this to be a more fair estimate, than the story gave. They're quoting pricing for a nice fast *CURRENT* machine, but Google has said before that they buy commodity machines. They do like we do. We buy cheap (relatively) and lots of them, just like Google does. We didn't pattern ourselves after Google, we made this decision long before Google even existed.
When *WE* decided to go this router, we looked at many options. The "provider" we had, before we went on our own, leasing space and bandwidth directly from Tier 1 providers, opted for the monolythic sy
Write one? How hard is it to put together a few pages to insert and read a table? Then it'll do exactly what you want it to do, without a bunch of fluff.
I don't recommend Slashcode. It's really big, and does lots and lots of great stuff, but literally you'll spend the first month trying to figure the whole thing out. If you want a site like Slashdot, then go for it, but it didn't appear that's what you were trying to do.
I'm replying now, with 777 ahead of me, so you've probably seen quite a few tales of how bad your job isn't.
Programming can be stressful, especially when the customer doesn't exactly know what they want, and you want to make the project perfect. There isn't much worse than getting the specs from a customer (i.e. boss), and putting together something beautiful, just for them to come back and say it doesn't do what they wanted. Of course it does do exactly what they wanted, that's why you spent a good bit of time with them before you started, asking lots of questions.
As senior sysadmin where I work, where the majority of my job should be really high-end technical stuff, plenty of web programming comes to me. "Can you do this?" Of course I can. Does it put priority over problem X? Of course it does, whatever the bosses thing of now takes priority over anything they told you to do previously, until they realize that the last thing isn't done yet, and even if you tell them the last thing isn't done because they said the new thing is priority, it doesn't matter.
This isn't a problem being a programmer or sysadmin, it's a problem with working. Bosses always want everything from you, and don't understand creativity or time constraints. Like right now, I should probably be working on a half dozen other things, but I'm anything but inspired (and it's the middle of the weekend), so even if I sat down and forced myself to write something, it would suck. Inspiration is everything for creative work.
No matter how much stress you're under, it will never be as much as someone else. I'm on call 24/7, and answer directly two 3 people. Anything and everything comes across my desk eventually, even stuff I don't want any part of.
Friday, one of our developers had a computer problem. He was using Windows XP, and it crashed. Hard.. That was it, he didn't want Windows any more, he wanted Linux. So I gave it to him. I felt this was a reasonable use of my time, if it would mean that he wouldn't be dealing with system crashes any more. He did ask me, how often does Linux crash? I had to be honest. The only times I've "crashed" linux machines, is when I'm doing things I really shouldn't have been doing.:) The last memorable crash I had was me kinda replacing/lib/ with something that shouldn't have been there.:)
Your responsibility is less than someone elses. For example, your boss either is depending on you to do your work, and possibly answering to other people (investors, partners, shareholders). If your job doesn't get done, he's going to be in shit over it.
Just imagine if you were a programmer for Microsoft. Not only would you have the stress of making sure your program works well (ha!), but all of your friends will be calling you every time their computers crash. "Hey Bob, you work at Microsoft, right? Can you fix my computer?" That's stress.:)
Just find a way to relax and unwind after work. When you're not working, don't worry about work, or at least try. Have you hit the point where you start dreaming about programming, debugging a large project all night, just to wake up in the morning to find that you were sleeping, and not a single line was written, and you don't remember any of it, but you know it was perfect in your dream? Aparently that's a common one. I used to have them all the time, but then I started drinking more. Alcohol fixes everything.:)
The only programmers with no-stress jobs were programmers during the dot-bomb days, who got hired with huge salaries, that didn't actually do anything. Those days are gone.
You get plenty of blowjobs before you're married. They're leading you in.
Once you're married, that's the end of it. It's all "you bastard, do this" "you bastard, do that" "give me this" "give me that"
I knew a guy once, had a great perception of women. He was a bit nuts, but was absolutely correct about this. Everything a woman says is "I need" or "I want", which translates to "give me", and it almost always translates to money.
Try this observation sometime. Don't say tell her you're doing it, just listen to what she's saying.
"Hi honey welcome home. How was work today." = "I want..."
"Do you think we can go out tonight" = "to be entertained. Spend money on me."
"Honey, I saw this great dress at the mall" = "Give me money, I want a dress"
"I love you" = "pull out your wallet schmuck, I want something"
Damn, my girlfriend saw me typing this. Guess I'm going to the strip club tonight. They're blatently honest about it. "$?? for a lapdance". "Want me to sit and talk to you? Buy me a drink and pay me $??"
I did have a few strange opportunities through my life. I've worked, known, and dated a few strippers (not all apply to any of them), and I've gone to strip clubs and would just sit there, and when they had nothing else to do (slow night), they'd just sit down and talk to me. One girl I was dating, I'd give her $10 in $1 bills while she was dancing, and she'd give them back to me at the end of the dance. The perception is, if someone else is paying she must be good, so they'll pay too. It worked. Guys, you're all suckers. It takes a lot to be on the winning side with a dancer, and the most important part is, never be a customer at a strip club.
Try being the senior admin for a relatively large network, and being responsible to make sure everything works.
When someone abuses our free hosting servers ("hey lets put a bunch of TGP's up, and see how much bandwidth we can use."), or whatever they decide to try today, you may have been working til 4am on regular work and side projects, and then have the pager give a quick down&up page at 5am, followed by a wonderful succession of down and up pages, until they manage to send so much traffic, it completely overwhelms the 100Mb/s connection that machine has, and you finally get that serious down page at 6am, which says that it's staying down until someone fixes it. If you're lucky, you can manage to get through all the traffic and get on, or if you have a management interface (second NIC, so you aren't fighting the http requests to get to the machine). If you're not lucky, the phone calls begin. Maybe you can get someone at the facility to get on, but more than likely it's going to take someone who has the authority to suspend the abusers account.
It doesn't always get that bad. Usually on the first page, when things aren't all that rough, we'll have found and suspended the abuser. But if it had been a particulary hard night, some pagers aren't working, or whatever, you may find yourself in a situation where you're exhausted, can't see straight, and typing with one eye opened just so everything on the screen won't be double.
You may be cursing under your breath, but probably you're so wiped out that anything coming out of your mouth is incoherent grumbles, even to yourself.
Lather, rinse repeat.
I can't complain much, we've added extra safeguards in lately, and the pager has been fairly quiet. (knock on wood) It happens like this. A few weeks of quiet, followed by some major emergency.
A high-priority machine in another location had a no-name external RAID5 array fail. The box iteself decided to shut off. 5am they start calling me "What are we going to do!" Luckly, the admin at that location knows his shit too, and got it back together with minimal pain. A few "try this, I'm going back to sleep" calls were all it took from me. That was 5am to 8am, so I was bright and shiny at 10am for the start of the work day.
Who knows what it will be tommorrow. No matter how much we try to predict the problems, it will always be something we didn't expect. Someone with admin rights accidently deletes something essential ('rm -rf/'). The boss is getting all bent out of shape (with good reason) about Acacia (current unjustified lawsuit against us). Rumor got out that someone did something bad?
One morning the 6am phone call was about DoS attacks originating on our network. Some other provider somehow found my bosses phone number, and reported it to us. 20 minutes of screaming later, I get on the phone with the provider. They had a user who determined that we were attacking from port 80 to some high port on the customers machine. mysite.com:80 -> client:>1024 is the response to a web request. So I spent the next half hour educating them in how the Internet works, and how the guy had his firewall rules too strict if it was reporting that. Really, that's the last thing I want to be doing first thing in the morning.
Morning sex. Rolling out of bed after a full 8 hours sleep. Not rolling out of bed after a full 8 hours sleep. Those are things *I* want to be doing in the morning.
That does sound like a good idea. I wonder how they'd implement it? What's the difference between rebooting because someone upgraded the kernel and failed, and someone smacking the machine with their APC MasterSwitch because someone did something stupid.
Something I would like is if there was an "only next boot" option in Lili. Maybe there is, but I've never seen it. Do something like this:
shutdown -r now --try-image=NewKernel
If it works, you change lilo.conf to use this new kernel, if it doesn't, next time it boots, it comes back with the last kernel.
I'd think it would be a pain to implement the other way. I wouldn't think there's a good way for lilo/grub to know if everything came up successfully. I've seen network drivers that allow assigning IP's, but they don't actually work for connectivity. I can't remember which one it was off-hand, but I've seen it happen. Or more likely, in a multiple NIC environment, the wrong card comes up as eth0, which doesn't have a network cable on it.
Well, with our own machines. We have a standing rule, the person that hoses a kernel upgrade is the one that gets to drive down to the colo and fix it. Needless to say, we practice on the machine that are in the nearest colo first, before we do the distant remote ones. No one wants to foot the bill of flying from Florida to New York to fix a mistake they made.:)
(fingers crossed) we've never hosed a machine too terribly far away. A few times we've forgotten to put in the network driver for particular cards, especially on odd-ball machines.
On a late-night "we have to have this server up *NOW*" install, I built out the server, threw the kernel together (on the console), and drove the machine to the colo. I plugged it in, and turned it on, with the assumption everything was right (usually at about 2am) to get home and find it isn't on the network. An hour and another kernel compile later, it's working.:)
Pretty much, our distant remote machines are very redundant, so if we hose one, it's not a big deal. If we hose 5, well someone is in for a plane ride.
It's usually worth taking the plane ride anyways, there's usually something non-urgent that's waiting to be done that can be done while we're there.
We did have an urgent one-task plane ride once. One of the facilites we're in had a brown-out. When the power came back, our connectivity didn't. The switch was unhappy. The colo's site tech tried resetting it, but that did nothing for us, so someone (me) took a plane ride carrying a switch and laptop. 20 minutes in the colo, 2 hours in taxi's, and 8 hours on planes before I go t home. That was a long night. Exhausted, I did get to have breakfast in the McDonalds in Times Square though (stopped by to see someone before they went to work).
Slashdot Mirror
Hmm.. I did it? :)
This is a lot of fun doing, and I don't really care if no one else ever uses it, we'll use it for our servers and workstations, so that's really all that matters. It's important for everything to work right, for me, not for a mysterious user base of anonymous people on the Internet. If they want to use it too, cool.
Users will probably spot differences, for better or worse.. Probably most people will spot that I have an insanely simplistic view of startup files (BSD style). I fully believe that if you're going to be using a *nix machine, you need to know how to at least edit a config file or two, so my config files are easy to read and edit, and I don't feel there's a real *NEED* to make fuzzy happy GUI's for every admin function.
We love upgrading hardware on our Linux servers at work. Pretty much, we yank the hard drive out of an old server, stick it in the new server, and turn it on. As long as we're going to newer hardware, it works fine. If the kernel we're running was compiled for a really wrong kernel, it won't boot, but that's easy enough to fix.
/old /dev/hda1 /old /dev/sda1 /old/files /dev/sdb1 /new/ /old /new
/old /new
/dev/sda1 and /dev/sdb1 were the external arrays. They're seen by Linux (or whatever OS) as a single SCSI hard drive. Gotta love it.
We've had the occasional server go whacky with a bad motherboard, and just moved the drives to a new machine, and away it goes.
One server we did recently, we upgraded the machine first, moving the drives to a new machine, and turning it back on. Then we upgraded the RAID on it, which consisted of copying all the files over to the RAID, and rebooting. Painless. The biggest headache with it was having the admin who was local to it wait for all the staff to go home, so they wouldn't bitch that it was down for the 10 minutes it took.
Hmm, what did the copy consist of?
While people were working we did:
(leaving the active mounts in place)
mkdir
mount
mount
mkdir
cp -RPp
Then when we were ready, we locked all the users out, and to sync up all the changes for the last day or two we did:
rsync -av
(or something tremendously close to that)
The
Yes, the OS was on a single IDE drive. That system had grown since it was first implemented, with two IDE drives, and no concept of what it would eventually be used for.
What would it take to do the same thing on a large (hundreds of Gb) Windows fileserver? We didn't touch anything in the process, it all just worked. No redoing user permissions, no headaches. I believe the biggest problem was moving cabling, and changing the terminator on the SCSI chain.
Actually, the disk was WinXP home which included SP1. I don't have the disk or license handy, I think it got filed away with all the other disks and manuals that came with it.
But yes, it did take about 4 hours. And no, I'm not on a slow connection. That day I was on a 768Kb/3Mb DSL from PacBell/SBC. It was 4 hours of download, reboot, lather, rinse, repeat.
The technique for including 3rd party drivers is anything but new. It's exactly the same way I made an Adaptec AAA-130 (I believe) RAID card work on a machine years ago with WinNT 4.0. Far from improved.
Maybe I'm being obnoxious, but I fully expect when they release "better" versions of their software, I expect them to actually improve it. If they're releasing a newer version of XP, they could at least include the current drivers with the CD. It's not like I dug it out of a file drawer where I tossed it back in 2001. I bought it new at the store with the parts for the machine. Right along side it in the store were current copies of RedHat and Slackware (oddly enough), and either of those would have installed flawlessly. Actually, Slackware did.
Actually, that was a port completely done by TransGaming. They have mention of it on their site. It's only the original "The Sims" game, not all the expansion packs. They clearly say that the expansion packs won't work with it.
:(
:)
The distro was a special gaming edition of Mandrake, which cost $60. It would be well worth it, if she could use her expansion packs. She has all of them, 6 I believe, which at $50/ea makes for a substantial investment in a single game, which she isn't willing to give up.
TransGaming says that you *CAN* use the version of TheSims included with Mandrake on other versions of Linux, as long as you have the right libraries installed, which I can't imagine being very hard, since they're right on the Mandrake disks.
I like Slackware. It's not pretty, but on a brand new machine, it took just about 30 minutes to install, and I was done. Even with adding programs in that I wanted, it was about 45 minutes total.
:( But, it does crash very well after a bit of game play. I suspect problems with the software, not Windows, in this case.
With WinXP home (same machine), I had to find a floppy disk to make a driver disk to allow me to install onto the SATA hard drive (the only drive), then it took roughly two hours to do the full install, and then another 4 hours to do the service packs to get it to a stable position. Then there was adding in programs to get it to basic functionality.
And before anyone throws cost of ownership, ummm, Slackware was free. Well, I paid for the CD's, but that was nothing compared to buying WinXP.
The beginning of the WinXP is anything but pretty. It's the same text (with some colors) interface that WinNT used.. Amazing. State of the f***in' art..
I've installed plenty of other distros too. Very pretty installers, that all work. I've only hit the rare error, but nothing compared to the last WinXP install that I did. 5 minutes in, for the first couple tries, and then it stops because it doesn't believe there's a hard drive in the machine..
Freakin' SATA. It's been out for a few days now, maybe someone at Microsoft has heard of it. {sigh} Ok, I'll be willing to say that maybe it was the controller, but hey, it's a name brand controller, built into a name brand motherboard, and hey, Linux saw it. Linux, you know, the one Microsoft bashes for being just a bunch of hobbiests doing it. Well, why can the hobbiests do what the multi-billion company can't??
I tried to install the same copy of XP on the *SAME* machine, into a VMWare virtual machine (booted to Linux). Nope, no-go.. I know, it looks like a different machine.. But, why? Because I may be an evil software pirate? Or I may be Joe-user who changed his mind about the hardware I was installing, and swapped it all around.. I've been known to build a machine, and when I'm done, swap video and sound cards, or even motherboards. Oh, no, that's not permissible in the wonderful world of Microsoft. I have to call and ask for permission if I do that. I opted to *NOT* call Microsoft, and beg for permission to use software I paid for in a perfectly legitimate fashion. I installed an old copy of Win98, which is no longer used on any PC's here (*MY* copy, that *I* paid for), which doesn't bitch, and threaten to not work if I don't pay for extra licenses.
This particular machine happens to be my girlfriend's machine. She wanted Windows, so she could play "The Sims". I have to honestly say, Slackware ran a whole lot faster than XP, and this isn't a slow machine. AMD 2800+, 1GB ram, 200GB SATA drive. If only we could get game companies to recognize Linux is better, faster, and more stable, she wouldn't have even wanted Windows.
And, no, "The Sims" doesn't work under wine.
I'd have to bet Slackware.
:)
That's what I run. If I want something, I usually go to the source, and install what they have to offer, rather than waiting for anyone to do anything for me. Why should I wait for a mainstream distro to patch the hell out of something, and distribute it to me with convoluted configuration options, when I can get it exactly like the author intended it? Slack makes this fairly easy. One of the first things I do when I install a Slackware system, is to download the new kernel sources, and compile it myself. Not that there's anything technically wrong with Slackware's kernel, I just like to know that my system is running exactly the way it should, without any extra drivers, or fluff.
We're building up a distro like this. It's going to be called "LMLinux". There's a really brief overview at http://lmlinux.com . We almost have the first alpha release done, but not quite. Our package management backend is just about complete, which looks rather nice.. Everythings stored in a MySQL database, for easy reference. It will make for very easy work, for anyone to browse to a few lookup pages, or for a package manager to find things.
I'm all about doing things from source, and doing it the way the author intended. But hey, if people like patchwork systems, let them get another distro with a kernel that has various patches reverse-engineered to work in the wrong version, etc, etc..
In our ideal world, you have the option of installing the package from our server, or grabbing it from the author yourself, and installing it yourself, without worrying that the next automagic update will hose whatever you've done.
I've done work for RedHat and Debian people, who absolutely freak out at the idea that I'd even think of downloading something myself, and installing it myself.
With all the complaints I see about x distro not having this, or not doing that, I kinda giggle. My Slackware installs do everything. Farther up in the thread is someone complaining about a lack of NTFS support by whatever distro. NTFS? Mine does it, I compiled it into the kernel, if there will be NTFS drives to read. I watch any video type I want, I've installed xine. It's not rocket science, read the readme.
And, you're absolutely right, you get all the free support you need in the mailing lists for the program, or checking on dejanews (eerr, groups.google.com)
Found the story on why they were shut down. Basically the insurance company wanted them to have a driver with two years experience and a class c commercial drivers license. The Coast Guard was requiring that they had a certified sea captain and deck hand on board. So every tour would require 3 extra people, and their salary would probably exceed the profit of the tour.
The fatalities were with another DUKW tour, in Arkansas.
I've seen one driving around Tampa Florida. It's run by Duck Tours . It's kind of wierd looking up, and seeing a bright yellow amphibious thing driving down the road.
:)
For something as old as it is, I've only seen it broken down once. It's normal route went by my office in Tampa (when I worked there), and to their credit, there were always cute girls on it when it drove by.
I'm surprised no one corrected me on this. I was just writing something else and realized you can't fit 80 servers per rack.
1u = 1.75"
A standard rack is 70" tall.
You can fit 40 machines per rack. Knock 1u off for the switch, so 39 per rack, not 78. The price for servers and routers comes out the same, just twice as much floor space, which to us non-multi-billion-dollar companies costs money.
I'm fairly sure Google outright owns their facilities, so they don't have to worry about that square footage. That's just a little less space for the foosball room.
You'd be really surprised to see what we have. Ya, it's far less than a single leased machine in someone elses room. I've been tempted to put pictures online of the colo's and our equipment, but every time I have a decent set of photos together, we've already made a change at one of the colo's, and alas, no camera.
:) I found a way to fit 12 4u machines in a cabinet, and still get the doors to shut. :)
I haven't been back to New York, since another admin was up replacing machines, which is sad, it probably looks much prettier there now, than when I first set it up.
So let me see if I got this right. You bothered to take the time to write your message, for one character in the wrong case? You probably read through the whole thread looking for someone to make that mistake too, didn't you?
gigabyte. happy?
"They are very smart offering the one gigabyte storage."
So, what's it feel like, being so right? Been looking for it for a while, haven't you? Feel good?
I'd have to assume that the smallest drives they have in operation are 20Gb, but I'd suspect they probably be smarter not to use extra storage across all those servers, they'd probably go with some large arrays.
:)
:)
/. readers hitting them in the first months, trying to fill their own boxes, but outside of that, it will only be very occasionally that.
:)
We can buy this for about
Promise VTrak 15100 $4075
15 WD 250Gb SATA $230/ea $3450
---------------
3.5Tb (single RAID5) $7525
Just looking at raw space, you could put 14 of those on a SCSI chain, and multiple cards in one machine (say 4 to keep things simple). For just $422,000, you could have 196Tb of space on a single machine. I'd strongly suspect they'd distribute it over a few more servers than just one.
They're very smart offering the 1Gb storage. It'll get users in, who don't like the limits. I've offered my users "unlimited" storage for years. They were on a 45Gb drive for a long time, but are now on a 330Gb array. With 246 users, they're only using 7Gb, and honestly, most of that space is my usage. I archive several busy mailing lists, for my own usage, and plenty of accounts that exist, "just in case" we need a message from it from years ago.
Google will see a very small number of it's users getting anywhere close to 1Gb. I'm sure there'll be plenty of
One account we have, that simply collects spam sent to other accounts, took 7 months to collect 1.1Gb of mail. Hmmm, maybe I should purge that one. Nah, we still have 300+Gb free.
Score us for 200 servers, 180 running Linux, all in the US. Oh, and just two admins running the show. :)
His pricing in the summary may be a bit off.
:)
:) Well, not the Asus 1400r, those are built into a 1u case, but other machines we've built for servers are very easy to build into midtowers instead. Those machines don't get gobs of memory, but do get extras like nice sound cards and CD/DVD players. The price would be the same, as they'd probably still be attaching them to the same networking equipment. 132,000 servers, and 2,682 workstations and dev machines is probably fairly close to what they have.
Every article I've read about Google's servers says they use "commodity" parts, which means they buy pretty much the same stuff we buy. They also indicate that they use as much memory as possible, and don't use hard drives, or use the drives as little as possible. From my interview with Google, they asked quite a few questions about RAID0, RAID1 (and combinations of those), I'd believe they stick in two drives to ensure data doesn't get lost due to power outages.
We get good name brand parts wholesale, which I'd expect is what they do too. So, assuming 1u Asus, Tyan, or SuperMicro machines stuffed full of memory, with hard drives big enough to hold the OS plus an image of whatever they store in memory (ramdrives?), they'd require at most 3Gb (OS) + 4Gb (ramdrive backup). I don't recall seeing dual CPU's, but we'll go with that assumption.
The nice base machine we had settled on for quite a while was the Asus 1400r, which consisted of dual 1.4Ghz PIII's, 2Gb RAM, and 20Gb and 200Gb hard drives. Our cost was roughly $1500. They'd lower the drive cost, but incrase the memory cost, so they'd probably cost about $1700, but I'm sure Google got better pricing, buying the quantity they were getting.
The count of 88 machines per rack is a bit high. You get 80u's per standard rack, but you can't stuff it full of machines, unless you get very creative. I'd suspect they have 2 switches, and a few power management units per rack. The APC's we use take 8 machines per unit, and are 1u tall. There are other power management units, that don't take up rack space, which they may be using, but only the folks at Google really know.
Assuming the maximum density, and equipment that was available as "commodity" equipment at the time, they'd have 2 Cisco 2948's and 78 servers per rack.
$1700 * 78 (servers)
+
$3000 * 2 (switches)
+
$1000 (power management)
--------
$139,600 per rack (78 servers)
Lets not forget core networking equipment. That's worth a few bucks.
Each set of 39 servers would probably be connected to their routers via GigE fiber (I couldn't imageine them using 100baseT for this) Right now we're guestimating 1700 racks. They have locations in 3 cities, so we'll assume they have at least 9 routers. They'd probably use Cisco 12000's, or something along that line. Checking eBay, you can get a nice Cisco 12008 for just $27,000, but that's the smaller one. I've toured a few places who had them, and pointed at them citing them to be just over $1,000,000.
So....
$250,000,000 (ttl expenses)
- $ 9,000,000 (routers)
------
$241,000,000
/ $ 139,600
------
1726 racks
* 78 (machines per rack)
------
134,682 machines
Google has a couple thousand employees, but we've found that our servers make *VERY* nice workstations too.
I believe this to be a more fair estimate, than the story gave. They're quoting pricing for a nice fast *CURRENT* machine, but Google has said before that they buy commodity machines. They do like we do. We buy cheap (relatively) and lots of them, just like Google does. We didn't pattern ourselves after Google, we made this decision long before Google even existed.
When *WE* decided to go this router, we looked at many options. The "provider" we had, before we went on our own, leasing space and bandwidth directly from Tier 1 providers, opted for the monolythic sy
s/Slashdot/Internet/g
Write one? How hard is it to put together a few pages to insert and read a table? Then it'll do exactly what you want it to do, without a bunch of fluff.
I don't recommend Slashcode. It's really big, and does lots and lots of great stuff, but literally you'll spend the first month trying to figure the whole thing out. If you want a site like Slashdot, then go for it, but it didn't appear that's what you were trying to do.
I've heard it in American law enforcement. I believe it has the same meaning.
I'm replying now, with 777 ahead of me, so you've probably seen quite a few tales of how bad your job isn't.
:) The last memorable crash I had was me kinda replacing /lib/ with something that shouldn't have been there. :)
:)
:)
Programming can be stressful, especially when the customer doesn't exactly know what they want, and you want to make the project perfect. There isn't much worse than getting the specs from a customer (i.e. boss), and putting together something beautiful, just for them to come back and say it doesn't do what they wanted. Of course it does do exactly what they wanted, that's why you spent a good bit of time with them before you started, asking lots of questions.
As senior sysadmin where I work, where the majority of my job should be really high-end technical stuff, plenty of web programming comes to me. "Can you do this?" Of course I can. Does it put priority over problem X? Of course it does, whatever the bosses thing of now takes priority over anything they told you to do previously, until they realize that the last thing isn't done yet, and even if you tell them the last thing isn't done because they said the new thing is priority, it doesn't matter.
This isn't a problem being a programmer or sysadmin, it's a problem with working. Bosses always want everything from you, and don't understand creativity or time constraints. Like right now, I should probably be working on a half dozen other things, but I'm anything but inspired (and it's the middle of the weekend), so even if I sat down and forced myself to write something, it would suck. Inspiration is everything for creative work.
No matter how much stress you're under, it will never be as much as someone else. I'm on call 24/7, and answer directly two 3 people. Anything and everything comes across my desk eventually, even stuff I don't want any part of.
Friday, one of our developers had a computer problem. He was using Windows XP, and it crashed. Hard.. That was it, he didn't want Windows any more, he wanted Linux. So I gave it to him. I felt this was a reasonable use of my time, if it would mean that he wouldn't be dealing with system crashes any more. He did ask me, how often does Linux crash? I had to be honest. The only times I've "crashed" linux machines, is when I'm doing things I really shouldn't have been doing.
Your responsibility is less than someone elses. For example, your boss either is depending on you to do your work, and possibly answering to other people (investors, partners, shareholders). If your job doesn't get done, he's going to be in shit over it.
Just imagine if you were a programmer for Microsoft. Not only would you have the stress of making sure your program works well (ha!), but all of your friends will be calling you every time their computers crash. "Hey Bob, you work at Microsoft, right? Can you fix my computer?" That's stress.
Just find a way to relax and unwind after work. When you're not working, don't worry about work, or at least try. Have you hit the point where you start dreaming about programming, debugging a large project all night, just to wake up in the morning to find that you were sleeping, and not a single line was written, and you don't remember any of it, but you know it was perfect in your dream? Aparently that's a common one. I used to have them all the time, but then I started drinking more. Alcohol fixes everything.
The only programmers with no-stress jobs were programmers during the dot-bomb days, who got hired with huge salaries, that didn't actually do anything. Those days are gone.
Wives go from giving oral to being oral.
You get plenty of blowjobs before you're married. They're leading you in.
Once you're married, that's the end of it. It's all "you bastard, do this" "you bastard, do that" "give me this" "give me that"
I knew a guy once, had a great perception of women. He was a bit nuts, but was absolutely correct about this. Everything a woman says is "I need" or "I want", which translates to "give me", and it almost always translates to money.
Try this observation sometime. Don't say tell her you're doing it, just listen to what she's saying.
"Hi honey welcome home. How was work today." = "I want..."
"Do you think we can go out tonight" = "to be entertained. Spend money on me."
"Honey, I saw this great dress at the mall" = "Give me money, I want a dress"
"I love you" = "pull out your wallet schmuck, I want something"
Damn, my girlfriend saw me typing this. Guess I'm going to the strip club tonight. They're blatently honest about it. "$?? for a lapdance". "Want me to sit and talk to you? Buy me a drink and pay me $??"
I did have a few strange opportunities through my life. I've worked, known, and dated a few strippers (not all apply to any of them), and I've gone to strip clubs and would just sit there, and when they had nothing else to do (slow night), they'd just sit down and talk to me. One girl I was dating, I'd give her $10 in $1 bills while she was dancing, and she'd give them back to me at the end of the dance. The perception is, if someone else is paying she must be good, so they'll pay too. It worked. Guys, you're all suckers. It takes a lot to be on the winning side with a dancer, and the most important part is, never be a customer at a strip club.
6 hours? That must be nice..
/'). The boss is getting all bent out of shape (with good reason) about Acacia (current unjustified lawsuit against us). Rumor got out that someone did something bad?
Try being the senior admin for a relatively large network, and being responsible to make sure everything works.
When someone abuses our free hosting servers ("hey lets put a bunch of TGP's up, and see how much bandwidth we can use."), or whatever they decide to try today, you may have been working til 4am on regular work and side projects, and then have the pager give a quick down&up page at 5am, followed by a wonderful succession of down and up pages, until they manage to send so much traffic, it completely overwhelms the 100Mb/s connection that machine has, and you finally get that serious down page at 6am, which says that it's staying down until someone fixes it. If you're lucky, you can manage to get through all the traffic and get on, or if you have a management interface (second NIC, so you aren't fighting the http requests to get to the machine). If you're not lucky, the phone calls begin. Maybe you can get someone at the facility to get on, but more than likely it's going to take someone who has the authority to suspend the abusers account.
It doesn't always get that bad. Usually on the first page, when things aren't all that rough, we'll have found and suspended the abuser. But if it had been a particulary hard night, some pagers aren't working, or whatever, you may find yourself in a situation where you're exhausted, can't see straight, and typing with one eye opened just so everything on the screen won't be double.
You may be cursing under your breath, but probably you're so wiped out that anything coming out of your mouth is incoherent grumbles, even to yourself.
Lather, rinse repeat.
I can't complain much, we've added extra safeguards in lately, and the pager has been fairly quiet. (knock on wood) It happens like this. A few weeks of quiet, followed by some major emergency.
A high-priority machine in another location had a no-name external RAID5 array fail. The box iteself decided to shut off. 5am they start calling me "What are we going to do!" Luckly, the admin at that location knows his shit too, and got it back together with minimal pain. A few "try this, I'm going back to sleep" calls were all it took from me. That was 5am to 8am, so I was bright and shiny at 10am for the start of the work day.
Who knows what it will be tommorrow. No matter how much we try to predict the problems, it will always be something we didn't expect. Someone with admin rights accidently deletes something essential ('rm -rf
One morning the 6am phone call was about DoS attacks originating on our network. Some other provider somehow found my bosses phone number, and reported it to us. 20 minutes of screaming later, I get on the phone with the provider. They had a user who determined that we were attacking from port 80 to some high port on the customers machine. mysite.com:80 -> client:>1024 is the response to a web request. So I spent the next half hour educating them in how the Internet works, and how the guy had his firewall rules too strict if it was reporting that. Really, that's the last thing I want to be doing first thing in the morning.
Morning sex. Rolling out of bed after a full 8 hours sleep. Not rolling out of bed after a full 8 hours sleep. Those are things *I* want to be doing in the morning.
Nah, just set it to verbose.
Don't worry, I wash my hands frequently. But years of thumbing through a book, and it'll show wear.
Shut up and get back to work. I don't care if there's a time difference between our offices, you're suppose to be working, not reading
That does sound like a good idea. I wonder how they'd implement it? What's the difference between rebooting because someone upgraded the kernel and failed, and someone smacking the machine with their APC MasterSwitch because someone did something stupid.
Something I would like is if there was an "only next boot" option in Lili. Maybe there is, but I've never seen it. Do something like this:
shutdown -r now --try-image=NewKernel
If it works, you change lilo.conf to use this new kernel, if it doesn't, next time it boots, it comes back with the last kernel.
I'd think it would be a pain to implement the other way. I wouldn't think there's a good way for lilo/grub to know if everything came up successfully. I've seen network drivers that allow assigning IP's, but they don't actually work for connectivity. I can't remember which one it was off-hand, but I've seen it happen. Or more likely, in a multiple NIC environment, the wrong card comes up as eth0, which doesn't have a network cable on it.
Been there, done that. :)
:)
:)
Well, with our own machines. We have a standing rule, the person that hoses a kernel upgrade is the one that gets to drive down to the colo and fix it. Needless to say, we practice on the machine that are in the nearest colo first, before we do the distant remote ones. No one wants to foot the bill of flying from Florida to New York to fix a mistake they made.
(fingers crossed) we've never hosed a machine too terribly far away. A few times we've forgotten to put in the network driver for particular cards, especially on odd-ball machines.
On a late-night "we have to have this server up *NOW*" install, I built out the server, threw the kernel together (on the console), and drove the machine to the colo. I plugged it in, and turned it on, with the assumption everything was right (usually at about 2am) to get home and find it isn't on the network. An hour and another kernel compile later, it's working.
Pretty much, our distant remote machines are very redundant, so if we hose one, it's not a big deal. If we hose 5, well someone is in for a plane ride.
It's usually worth taking the plane ride anyways, there's usually something non-urgent that's waiting to be done that can be done while we're there.
We did have an urgent one-task plane ride once. One of the facilites we're in had a brown-out. When the power came back, our connectivity didn't. The switch was unhappy. The colo's site tech tried resetting it, but that did nothing for us, so someone (me) took a plane ride carrying a switch and laptop. 20 minutes in the colo, 2 hours in taxi's, and 8 hours on planes before I go t home. That was a long night. Exhausted, I did get to have breakfast in the McDonalds in Times Square though (stopped by to see someone before they went to work).