Slashdot Mirror

• In every step building the EV1 business, I've had to make decisions that I believed in my heart were in the best interests...[blah blah blah]

Actually, it's more likely RackShack changed it's name to EV1Servers due to the fact that their #1 competitor RackSPACE was rather unhappy about the name confusion that was going on.

... but there are some other things to be said.

First off, you don't tell us a lot about your business. Do you have well-stated service level agreements? If not, get the hell off slashdot and do SLAs ... you've got noplace to stand without them.

Second, once you know what your service level committments really are, ask yourself "do I have half a clue what the fuck I'm doing?" If the honest answer is "no", do what has been suggested and move your services off to a hosted-service firm. (I have heard good things about Rackspace but there are a bunh of these firms around.) Even if the honest answer is "yes", you should seriously consider a hosting firm, because a lot of the things you need to do have a big fixed cost up front; you get better payoff by buying a part of the up-front costs someone else alerady paid.

Third, one topic you've left off the list is disaster recover -- what they now call "business continuity planning" on the grounds that "disaster" sounds so scary. You need to make sure that you have (1) off-site backups sufficiently far offsite to be pretty certain that a local disaster won't compromise them. (I had several clients and acquaintances who were knocked off the air on 9/11/01. Some of them had backups outside Manhattan, and they got working again quickly. Some of them didn't, and some of those firms are gone now.) You also need (2) to know where you're going to go with those backups if the disaster comes. A number of firms will let you pay them a small fee that guarantees access to backup servers; the other end of the spectrum is to have a secondary site that's geographically well separated, eg, New York City and Princeton, NJ or southern Connecticut for Wall Street firms, or San Francisco and Salt Lake City for the west coast. Once you've set this up, test it. Cut over to the backup every so often.

How do you know what level of this stuff you need? That's why you need a well-stated SLA.

I always see ads for Rackspace.com, and looking at their site, they just announced 100% uptime for 12 months. Apparently they let you purchase (and/or rent?) a rack-mounted box with which you can do whatever. I'm not sure about bandwidth limits/DNS/etc.

Has anyone had any experience with these guys? I was thinking of getting a dedicated server box for myself sometime but I haven't had the time/money to do so.

As I cited in a reply farther up the page, Rackspace has few local customers here in Cowtown, USA (aka San Antonio TX) but that hasn't stopped them from prospering.

A solid company with a good product will have no problem making deals happen. Example: Rackspace is located here in San Antonio.

• interesting overall view
  
• neat to look at, shows you that Linux users include www.thawte.com, Rackspace, www.dialtoneinternet.com and www.cihost.com
  
• a neat look at sites with linux in their name
  

E.g., suppose 300 users with a common interest (let's use Everquest for this example) get together and agree to each donate $1.50 a month. That's $450/month. That's enough for a dedicated server with 10 GB/month bandwidth from rackspace.com.

These 300 users then find Everquest sites that they like, and offer to give those sites free hosting on their server, in exchange for premium access (e.g., no ads for the server owners).

• 1. LinuxNewbie -- They've got tutorials on just about everything.
  1. The Linux Security Portal
  1. The raQ3 download site -- security patches here
  1. CommuniTech's satisfaction ratings with EOpinions - looks like you're not the only dissatisfied customer
  1. A bunch of Linux security stuff here
  
  
  Hope this helps...

IT COSTS VERY LITTLE TO GET WEBHOSTING???? I disagree as hell. Webhosting is cheap if you don't get a bigger site, that needs a dedicated server and 57 GB of bandwidth every month. Check out the prices at RackSpace and see what such a server costs.

It's when your site grows big and popular that it ends being a work of love. Capitalism is practially stuffed down your throat in this situation. Sad but true.

Having lived there for 4 years, I can say that they have the best sushi (The best sushi restraunt? Sushi Expo in San Jose where Hillsdale and Camden meet, a little north of I85).

On the flipside, I can say the housing situation sucks. My apartment, while I worked at TurboLinux, was in Pacifica. It was a very cool town, just 10 minutes from San Fran. 25 minutes from Brisbane, where TL was. Right near the ocean, easy walk to a organic store where I could get the best veggies and fruits.

However, it leaked like crazy all rainy season (winter) and cost about $2000/mo. and was a 2 bedroom 800+ sq/ft. apartment.

I decided to move to San Antonio after leaving TL to work for RackSpace. My apartment is now 1600sq/ft for only $800/mo.

Double the space for half the price. :)

Some data points for those who don't know:

• A Housing survey when I left, 6 months ago said the average 1 bedroom apartment was $1700/mo.
• Most apartments are rented, sight-unseen, within hours of their being put in papers.
• If you make only $50k a year, you probably are homeless, in massive debt, or living with a large number of roommates, or some combination.

Ciao!

do not want my local net admin/nazi to control *MY* e-mail

Aww, poor little snookums. Here's a suggestion - run up rackspace.com, order a server, and configure it howyou please. Run your open relays, accept your spam, do whatever you damn well please (assuming you're actually capable of running a system). The only freedom you get on someone else's system is the freedom they decide to give you.

I once had a friend at a start-up ask me whether they should colo or do it themselves. I have an edited reply here: http://www.nspf.net/colo.txt. In it, I talk about several things that they or a colo provider would have to think about and plan for if they were building and managing a data center. As you look at a data center, you might want to think of some of these issues as well.

One of the older reasons that people started using colocation was that local loop charges from an office into the ISPs was expensive. By putting a server at an ISP, you avoided those charges. The data centers at the colo were better than one's typical office. Once you get past DS3 speeds, the LEC charges don't look as expensive as they used to. Smaller sites (<=10Mbps) are still better served at a colo facility.

Recently, the reasons for picking a colo provider have morphed into: "What am I willing to outsource?" When you pick a colo site, you're trusting your machines, networking, facilities management and physical security to another company. Are they better at doing it than you?

Some colo providers distinguish themselves in various ways. One might be better-connected into AOL or broadband networks. Some focus on connectivity. Some might be better aquainted with NT than others. Some might have a great daytime Sun or Linux staff. One might offer database outsourcing/management. One might offer backups. One might rent you EMC/SRDF disk space. One might offer managed servers instead of using your servers. These features could good reasons to select one over another, but only if they complement your operation. Make sure, though, that you don't get attached too much to any colo provider's value-adds. For example, if they have network problems every week and your data is trapped in their managed database, you'll have to live with it until you can duplicate the functionality that you'll have to leave behind.

Initially, in an R&D and trial/rollout phase, many companies can get away with outsourcing as much as they can to focus on their site's development, but as the site becomes more popular and as users depend on it more, the uptime and reliability of the site becomes much more important. You can't just let your colo or network provider screw up anymore. It's unacceptable. Decrease your dependence on any colo provider as your site becomes more important.

A very annoying feature of running a colocation site is working remotely. If you need to do anything more than hit a power button on a machine, you need to figure out a way to not have to do that. Don't change tapes, buy an autoloader with lots of tape capacity. Don't use machines that need keyboards/monitors - use serial console servers to access the serial consoles of your devices or use something like Citrix to manage your NT boxes. Buy reliable machines that don't crash as often. Inside a server, a 10000 RPM or 15000 RPM disk tends to fail more than a 7200 RPM disk. A beefier power supply running at half capacity runs longer than a cheaper one running at capacity. Buy more servers than you need for everything so that you can migrate your service from failed servers to standby ones. Don't run the latest version of an operating system. Run the most stable/patched version of the OS. Eliminate all single points of failure from the networking side (including having more than one upstream ISP if possible). Routers and layer4 gear need to reboot sometimes. Always buy more than one of each.

The best way to avoid failure is to have multiple data centers. You can care less about the reliability/availability of one data center because you can always direct traffic to another one. Many web sites make a mistake early on of building a single dependency into their site, whether it's a database or a filesystem, something keeps the site from running in parallel with a similarly-configured. Plan from the start by running your web site from 3 locations, and you'll be able to scale your site very well. You'll also be able to pick and choose from cheaper colo providers that don't do N+1 redundancy to help reduce your costs.

At some point, every colo provider will let you down. It's inevitable that something bad will happen. Picking a colo provider that learns from its mistakes can be better than one that strives to make none. We would expect the same of our employees if we ran a data center ourselves.

If you're good at managing servers remotely, the location of your site(s) becomes less important:

• I have a friend who colocated his servers in San Diego while his team worked in the Bay Area. One of his justifications was that it was just as fast to drive from his office in SF to Oakland and fly a plane to San Diego than it was to drive from San Francisco to San Jose during rush hour. Becasue he had skilled remote hands, he didn't have to fly down often.

Some thoughts on networking for colo providers:

• I used to think that I wanted to have servers closer to the exchange points. Exchange points are crowded places for traffic. If something happens elsewhere on a network, the traffic that is rerouted near your upstream routers might saturate your connectivity to anywhere. I'd now rather put my servers in a less-crowded area (St Louis? Denver? Dallas?) with good connectivity and let my bits get to most of the country/world without the congestion at the peering points.
• A Tier 1 ISP may not be as good as a Tier 2 ISP with multiple transit paths into Tier 1 ISPs. I use SimpleNet in SanDiego as an example. They have six or more DS3/OC3 circuits out of their data center to different providers. Most other colo providers in our area have only one or two outbound paths. If connectivity out through one ISP starts to suck (fiber cut, BGP flapping), an ISP with multiple transit links can just shut off the bad ISP until it recovers.
• If XX% of my traffic goes to a particular online service provider (AOL, CompuServe), a broadband network (@Home, RoadRunner), a WAP network (Sprint, others), through a portal (Yahoo, Excite), or to a dial-up audience (UUNET, AT&T), I'd want my servers to have really good conectivity to their network. I might choose a data center that has special peering or connectivity to my target network.
• It really sucks when a colo provider oversells their capacity. Find out how much capacity a colo provider advertises. Then figure out how much 40% of that capacity is. Thell them you have BIG plans for a quick rollout. If they say they can handle it and don't suggest that they to order more capacity first, then they are likely to bump against their physical outbound limit which could mean packet loss for your site and every site withing the data center. You, already being a customer, would lose. If a colo puts you and everyone else behind bandwith limiters (like Xedia), that's good. They can rate-limit bandwidth hogs to keep everyone else running smoothly.

Random musings:

• One set of colo providers that I'm gaining respect for are the ones that try to do as little as possible. Imagine a climate-controlled U-Haul storage facility with padlocks in front of 10x10 rooms. Add conduits for cables that go to other cages that contain LEC fiber or ISPs' routers. Even a high school dropout can reliably run such a facility. I can take care of my own racks and networking. I can deal directly with my upstream ISP's without the colo middle man. Equinix follows this model.
• Another type of colo is gaining my interest as well. Look at rackspace.com. Dell and Intel and perhaps others have the rent-a-server model. Sure, it's managed-server like others offer, but they keep themselves on the physical/parts side and can custom-build and install servers to my specs within a couple days of lead time. They don't try to manage the server at all - just rent me assembled server parts and bandwidth. I would no longer have to buy servers, just rent them. If there's a problem with a server, I can have them repair the server or build me a new one right away. I don't have to wait for my vendor to ship me a spare part to install - it's their server, and they damned well better have spare parts.
• Distributing static content through a CDP like SandPiper or Akamai can significantly help the scalability of your web site. The only machines that you'd have to colocate would be site-critical back-end servers and masters for your site's content. A serious mistake that sites make is pumping too much bandwidth out of their colo provider. The CDPs have gigabits of good bandwidth all around the world to rent you. Try not to use more than, say, 20% of your colo provider's available bandwidth. As your site grows, your colo provider (with 4-6 months lead time on new fiber uplinks) might not be able to keep up with you if you depend too much on them for bandwidth.
• If you have servers in a remote colo facility, like on another coast, make friends with a local geek or have a really smart consultant available near that facility to save yourself 5-hour plane trips. Frequent flyer miles don't make up for the time you waste on airplanes, and it can get old fast.
• You can never have enough tools or spare parts in your colo facility.

Just stuff to think about.
--
Eric Ziegast

(PS: I used to work at ISPs and colo providers just like synx. I currently help run a very popular web site at several different colo facilities.)

Yeah,
I work at a dedicated server hosting company called Rackspace.com (shameless plug), and we have a whole frieking wall of Cobalt Raq's (1's,2's and 3's). My advice is that you get a Raq if you want to resell web space. If not, a dedicated Linux or FreeBSD box would be better. Especially if you want to customize your system. Cobalt voids warranties if you even recompile the kernel... and you can't upgrade the CPU's on them. With a Linux box, you can go to SMP and get a RAID-5 w/ a GB of RAM if you want.

• Fuck Censorship!
• ONDCP: Oppressive Network Drug Content Propaganda
• Elian Raid
• DARE Sucks