Slashdot Mirror
Managing Bandwidth and Bandwidth Costs?
"I'd like to illustrate the second concept. When you have your (for example) T1 and you're not really using it, you are still paying for all that bandwidth. It's like the car that sits in your garage, you're still paying insurance and car payments on it even though you're not using it. But then you put up a new game, serve new media or suddenly become the 'Site of the Day' and your bandwidth is flooded and maxed out. For that case, it's like you've bought a car that only goes 40 miles an hour but while the demand exists and only while that demand exists, you need a car that goes 150 miles an hour. You don't want to pay the money for a car that goes 150 because you only need it occasionally. Later, you know you'll need that car to go 220 but you're not there yet.
So if this makes sense with regards to bandwidth, it is like you'd want burst-bandwidth depending on need. Do any of you face this problem? If you do and have solved it, I'd love to hear about your strategy. Once this is solved, we get back to the first question, how do you manage that cost, put a number on it and either fit it in to your business model or pass it on to your customers?"
We can do it the slackware way - tar it up and let someone else do it.
Or we can do it the debian way - let someone else tar it up and then do it.
But we like the Slashdot way most. Post it up and watch the traffic floooooooooooooooow....
Too many to spell out here.. Search google, LOTS of people sell bandwidth at X mbits/s with a BURST to x mbit/s.
You figure our how much you will estimate you will use as a total per month, and you pay for that. You cruise at 2mbit/s mostly, then you explode to 145mbit/s and at the end of the month you average out to 6mbit/s.
The idea is you BUY your peak speed, but pay for a low average..
EVERYONE does this now, call around... It was kind of silly for you to ask Slashdot how to do this. UGH.
- Voxel
Modesty is one of life's greatest attributes
..and let the kids distribute it for you.
Where possible BitTorrent is your friend. If you can use it, let your customers help out with the bandwidth usage. They will probably get the file(s) quicker and you won't have servers getting torched.
how can we get our company's bandwidth setup so we can survive a slashdotting?
maybe if he got some comments from the peeps at osnews or cnet...
vodka, straight up, thank you!
Maybe?
Check it out: http://bitconjurer.org/BitTorrent/
cygnuhchur
If you want to pay for data by the gig, rather than by the pipe size, just sign up with an ISP which allows that.
At the ISP where I work we offer fiber connections that allow for increased bandwidth for certain periods of time. For example, our burstable connections are usually around 1-3 meg for normal times, then burstable up to 10 megs. I'm sure you can find something suitable to you.
:-)
*blatant sales pitch*
If your buisness is near Southern Ontario, check out our website at www.sentex.net. We rock
Here's a free tip --- don't post your site to slashdot.
Congratulations - you passed this one.
Anyway the answer to all your prayers is obviously BitTorrent.
Whenever Savvis has end of quarter sales you can negotiate the price a ton.
Right now they're about 700/month for a managed T1 with local loop included.
1) Agree with your ISP on a standard data rate, burstable to X as needed.
2) Use RTG to monitor traffic in and out, making sure that you know what switch/ports/etc. that client is using.
3) Charge the client (this is usally done based on 95th percentile).
4) Profit!!!
libertarianswag.com
It seems like an aboveous place to use some Peer2Peer method. (Unless these X meg downloads are pay per download or members only). BitTorrent is very good at this but stick links for people to download it off other systems like gnutella or kazaa... (whatever is the latest best). These people who download your errr... whatever it is, will pass it on.
All your company needs to do is to seed the files on the networks. Much lower bandwidth than serving each one.
Mouse powered Chips, Open source Processors and Lego
This is what Akamai and the like are sold for.
It's almost certainly going to be cheaper than just buying bandwidth.
Or you could go for the approach of colocating your own box somewhere central for the heavily hit stuff.
Even this will be a whole lot cheaper and won't impact on your normal traffic to your organisation.
ALTQ
This guy is way out there
I don't know what solution you'll eventually decide on, but to test it you can post a link to the file on AskSlashdot.
:)
My blog
How bout paying for a full time high bandwidth connection, and then whenever Slashdot posts a story about the release of Mandrake, Red Hat, etc., you post a mirror of it?
Then this whole "unused bandwidth" issue would be kinda moot. Everyone ones (well, except the person paying the bill).
"People that quote themselves in their signatures bother me" - athakur999
If you need proof that Kast is better than BT in this situation look at http://konspire.sourceforge.net/BitTorrent.shtml .
Hope it helps.
As soon as we all have 50mpbs 802X connections and, anonymous P2P software. This kind of question will never be asked.
With 100 gig hds, and reliable high speed connectivity 24/7. It's pretty easy to see what could be built from that.
Someone wants to download the new 150mb CS or BF1942 upgrade? Just enter the name of the file, select it...and the P2P software does the rest. Initiating multiple downloads from about 20 or so nodes in parallel. Then you just glue the program together once you have downloaded all the pieces.
Mirroring is such a hack, and dynamic bandwidth is the last gasp of the client-server paradigm. Let's move on.
The government has a defect: it's potentially democratic. Corporations have no defect: they're pure tyrannies. -Chomsky
May end up costing us on an "as-used" basis. Wouldn't it be nice, actually, for the occasional e-mail reader, web-browsing, non-bandwidth-hogging person, to only pay $5/month for the bandwidth they consume? Of course for most of us, it'd end up costing us $80/monthly for our DSL/Cable connections, as we download gigabytes a month, at blindingly fast speeds. Sort of like cellphones are used now-a-days; on an as-used basis, with a chunk of pre-paid megabytes, or whatever. Of course, if you're an ISP, there's the issue of counting those megabytes, which could be solved by bandwidth-accounting software (which I'm sure there are plenty of, but here's one I know of for Windows-based systems.
Plain and simple...let others provide the bandwidth for you.
-R
that offers "pay by the GB". The hosting company I work for has GiGE links but only pays for the exact amount of traffic we push through the link. The monthly cost on the line is minimal, or nonexistent depending on the provider.
Since we have more then one link as well, it gives us redundancy and the upper hand to negotiate the best price per GB, so we can send 90% of our traffic out that link. If the next month a different provider comes back with a cheaper price, we switch it around and send the 90% out their link. Within days of cutting the traffic off for a link, we can usually expect a phone call from the sales rep with lower price offer.
Any idle links we have don't cost us anything extra, since we _will not_ deal with any provider that doesn't offer pay by the GB. Paying for the raw link speed, regardless of how much traffic you push through, or paying 95th percentile prices are all mostly a rip off.
Open Source Time and Attendance, Job Costing a
Get a job at an ISP that doesnt monitor bandwidth across its core network then sneak a server into the machine-room on a 100Mbps connection that has 2x155Mbps links that are 3 hops to the biggest Internet Exchange point in the country.
:)
Then you just fire up BitTorrent and Overnet.
Very Sweet.
PS - Dont normally AC my posts but on this one I dont want to waste the Karma just to instill Bandwidth Envy
Couple of different options. First, you could talk to the Content Delivery Networks (CDN's) like Akamai or Digital Island. They can probably help you (for a price).
Another option is colocation. In particular if you have short traffic spikes. Many colocation places charge your for at a '95 Percentile'. This will cut out about 3 days worth of 'peek traffic' and you only pay for the maximum bandwidth you use after removing the top 5%. Just make sure the colocation place has enough bandwidth to handle the spikes.
Some ISPs (e.g. Yipes) offer flexible contracts that allow fast (daily?) bandwidth changes. So if you announce a new version of your product, you can increase your bandwidth until the rush is over.
One hint: Try to move the large file/content away from your 'importants' networks, so other things like e-mail keep flowing even if the content site is running into issues due to load.
---- join dshield.org Distributed Intrusion Detec
It seems to me that this is exactly the type of problem that places like akamai and cable and wireless (was digital island) are trying to solve. Pay only for the bandwidth you use, leverage their existing distributed architecture, profit. You can try to get bustable bandwidth etc, but in the past I've found it to be more expensive. Things may have changed since them (a year ago) but you should still look into a content delivery network.
Based on recent research at the University of Waterloo, you may well be able to treat the bandwidth usage as a risk factor and treat the option to buy more bandwidth as exactly that: an option on a real commodity. You would likely be able, then, to price the value of waiting to invest versus the value of investing now with a given expected return. Basically the cost of holding off on investing would then be quantifiable and you could choose the best time for investment.
There has been some good research done on this lately which you can read up on at the U. Waterloo Scientific Computation Group which did the work in co-operation with telecoms and the Finance department. The math is perhaps a little heavy going, but the results may put you on a firmer footing than doing the same computation with NPV or similar methods.
Disclaimer: I'm currently doing research with this group, though not exactly on this topic.
Let's face it. Most of the suggestions above are useless. Since when is a company going to officially distribute stuff via Kazaa or BitTorrent? Sorry, but when Microsoft says 'To download our latest Service Pack, use Kazaa' then pigs will be flying. It's so unprofessional.
The easiest solution is not to host it yourself, but to use specialized file hosting ISPs. There are lots of these around, and it's a trivial task on Google to find one at the price you want. These are ISPs that entirely focus on hosting large files for download, with servers optimized for that job.
There's no point in lagging out your regular servers which are probably optimized for something else.. and a dedicated file host can scale as you go.. which would usually cost you a packet.
How much will be web-content? Files? Ftp? Small or large files? What about your customers? Are the "tech savy"? US-only? Old? Young?
Maybe there are other distribution-modells that are better for you than the tradidional client-server approach?
Some suggestions:
1.I heard about a game company that saved *thousands* of dollars each month when they actively started to supply game magazines with demos and paches so they could include them on their CD's. (+extra addvertising)
2.How about getting a deal with a similar company that about sharing bandwith? You create a ftp-mirror for them and vica-versa.
3.Maybe if you are distributing large files (25MB+)to many users BitTorrent could help you.
Melius mori in libertate quam vivere in servitute.
If that was *my* real problem with a car... i guess i would rent a car... why not put the content on a remote server with enough bandwidht all the time?
rm -rf /home/leia
Start with a best guess for amount of bandwidth based on number of users and type of usage. Then monitor and adjust. If you pay for uneeded bandwidth you may be losing money. If underestimate bandwith needs and lose business, you may lose money...
Some things to consider checking into:
Use SNMP and RMON to manage and monitor your wide area connections. You will be able to do trending on your traffic to see what percentage of used is used during any predetermined interval. Free tools like MRTG are a great place to start.
Good network hardware, maybe most specifically your edge router(s) and firewall(s), that allows you to configure some sort of priority queuing or traffic shaping. Cisco's QOS features in its IOS based routers are known to be particularly strong.
A service or dedicated person to make sense of the traffic reports. It is an ongoing process.
You will also likely want to consider measuring "latency" as well as bandwidth consumption.
Also note that during a one minute average, 50% use can be 30 seconds of 100% and 30 seconds of 0%. Be aware that many of the bandwidth utilization numbers that are reported can be averages. You will want to look for peaks as well.
Good Luck.
~8^]
Welcome to the realities of hosting, why it doesn't always pay to be the most popular site on the Internet.
Finding a good host, be it co-locating or rackspace, is the first part in finding something moderately economical for bandwidth charges in the case you run over the limit. Some of the ones I've dealt with keep track and will contact you if they think it's in your best interest to upgrade to a better package. Then there's others that didn't want to offer a competitive price...
A trick I know a few sites employ with Apache is to turn on Zlib compression - pages are served faster, and less bandwidth is used to serve them which leaves a little more headroom.
Find a way to pass on the cost of serving to your customers. Not unheard of, but transition can be rough...
If you have the control, packet shaping would be the next step to throttle how much bandwidth is allocated to a service/etc. Linux recently had a kernel patch for this released, where before you had to pay lot$ for hardware based solutions...
The basic solution is to move your servers -- at least the ones that will be handling high bursts of traffic -- to an offsite, dedicated hosting facility. A facility like this will have big pipes coming in with the cost split among all of their customers, and typically offer you pricing based on burstable bandwidth use.
It's just not economical for a single company with occasional high-bandwidth requirements to bring in a pipe that sits idle 90% of the time. A co-location facility will serve many different customers who may have spikes of bandwidth usage, but their spikes will typically not be simultaneous, making overall bandwidth usage more efficient.
Lots of hosting providers offer this, including Verio, Dynamic Hosting, Superb Hosting and many others.
Streamload has the lowest bandwidth rates for hosting files (they are not a "web" host) and charge you for only per GB transferred not the peak.
www.streamload.com
Slashdot this site mercilessly! http://www.borahbands.com/
Most ISP charge by the 95% utilization, this means that they sample (8640 samples)your usage every 5 mins for the entire month. Then they remove the top 5% of your peak (432 samples) and the highest value is you billed usage for the month.
So you can see that for 432 samples (each same represents 5 mins) you can have very very high usage for free. Be very careful because if you have a 50 Meg/sec burst for the 433 sample you will be charged the entire month at 50 meg/sec usage (very big bill).
We use Packeteer Packet shaper to help us cap our usage and bill.
You said that bandwidth spikes now when 30000 users need a few meg file. Do they all need the same file? If so, can you send it via multicast instead of unicast? That way will save you to the point of (n*30000 - n*1) bandwidth right now...
and you could ignore the hassle of my previous post...
btw-just gotta ignore others negative comments, this was a fine place to post your query...
~8^]
Depending on what exactly you are doing you may consider using multicast and joining the "Mbone". I don't know if it would suit your needs or not, but it wouldn't hurt you to research it and see if it applies.
I'm running into the problem of hosting several different sites on shared servers, and need to be able to tell each customer what their useage is - what are you all using to do this?
"Going to war without France is like going deer hunting without your accordion." - Jed Babbin
Slashdot this site mercilessly! http://www.borahbands.com/
It may not work for you but for most people you can colocate on a nice 100mb pipe and or get dedicated server on a 100mb pipe for less then the cost of a t1 . Personally I would check out theplanet and canaca . But ask around and make sure you get a good uptime gaurentee . If you want you can try and get a commitment out of the people your colocating with.
Try and optimize the thingy [I presume its code?] for size?
Funny thing that... make things smaller they are easier to send.
Yours crapfully,
Capt. Obvious
Someday, I'll have a real sig.
A year ago I started a new job, as CTO for a company who needed to distribute files, randomly sized, and never with the ability to predict what day any give file would be available. So the first thing I did was look at the Akami type companies. I ultimately went with a company called SpeedEra. They offered a similar service, but cheaper. They handle the east and west coast distribution of our files, and we have them set up as two "distinct" points of presence. I then went with a local NYC ISP, and locked in at a very good rate, for 25Mbit/CIR. Knowing that my 95th percentile is about 23mbit for the web servers and our "primary" file distribution servers. So combine this, with our subscription level of service, which includes a file download tool (swarming tool, with p2p capabilities like bitconjurer) we have a nice robust infrastructure. We have continued to use more and more bandwidth, but knowing our 95%, I can up our CIR before the end of any month, and save on overage charges, etc.
There are other options too, like throttling bandwidth from within Apache, or at the switch port level. Aa for monitoring, I use JFFNMS (jffnms.sourceforge.net), and my ISP offers similar monitoring, so I can double check my data.
Just my 2 cents.
ftp.cdrom.com was able to fully saturate it's gigabit network uplink years ago, and it's just an x86 ftp server. 145mbit is like pissing in a river.
--sdem
Post your Web site or whatever on /.
/. effect will let you know right away what your needs will be. Many a poor bastard has found that out only TOO well.
The
All your bandwidth are belong to Slashdot.
It is easier to build strong children than to repair broken men. -Frederick Douglass
I manage a web hosting service, and, before you can try to figure-out what's your best data solution, you need to understand your usage.
I solved our bandwidth issue with the developpement of a SNMP bandwidth collector that calculates every 5 minutes, the running rate, and tabulates the sum of data. With that, I was able to calculate, on my own, my total usage per month, per day, per hour, per 5 minutes.
It turns out, after 1 month of data, going with a rate that is calculated at the 95th percentile, was the most efficient method, because it allowed me to feed my bursts and not get penalized, and it lowered significantly my $$ / gig (50% cheaper).
So, like I say, good analysis tools, a traffic shaper, and good relationship with your ISP saves you a lot of $$ on bandwidth charges
Most places are willing to sell "burstable" bandwidth -- they could run, say, an OC-3, but you might normally only use 3 Mbps of it. You'll pay more than if you just bought 3 Mbps of plain old traffic, but if you get hit with a deluge of traffic, you can use all 155 Mbps of it. (But then you get charged for it.) If you bandwidth usage is irregular, buying "burstable" bandwidth is definitely the best way to go. "Burstable" pricing is usually more expensive per Mbps, but it's cheaper than getting a super-big line and using a tiny fraction of it. A second thing, that's probably a much 'messier' solution, would be to try to resell some hosting space or something. (But then most people will want support, and you risk someone spamming or whatnot... But it's an alternative to keep in the back of your mind.)
________________________________________________
suwain_2
Also, if your business can get away with it, try and spread the downloads out among time. For example: split your customer base by some hashing algorithm, and notify them evenly spaced out over a few days. Or even tie bandwidth monitoring to notification, so that as the usage peak from the last batch of notifications begins to drop, start notifying the next batch. Again, this depends on your business model of course.
_______
2B1ASK1
They are anti free speech.
There are numerous slashdot articles about them.
A local dialup ISP just sets the router to stop forwarding for 45 seconds at a random interval, somewhere between 5 and 45 minutes after the last stall.
You wouldn't believe how many dialup customers use KazaaStein or other music stealers. Really ruins their downloads, but the text people don't seem to mind.
The music stealers go elsewhere the first time a competitor offers a deal for new customers. We've found 5% of our customers use 95% of bancwidth, give them to the competition.
On the other hand you may want to rent hosting.
30.000 * 5 MB = 150 GB. Several hosting sites gives more than 200 GB/ month at economic prives (400 $ )
You may want to rent several mirrors so that it is redundant and they don't get colapsed by the trafic.
As an aditionall advantage your firm does keep al their resourses free of all the problems asociated to the peaks. But it is important to be aware of the condition of the hosting.
Just for internal distribution it is easy to use distribution list so that only one copy has to travell across MAN and WAN. LAN could use Gygabit ethernet.
You know what why don't you go read a god damn book you corporate Fachist.
There are plenty of books on managing bandwidth and there are plenty of sites on the web.
And no I'm not gonna look them up for you!
If your engineers can't figure this out then you should get out of the business.
I for one resent anyone looking for free labor in slashdot?
If you need someone to solve this problem for you why don't you hire a slashdotter!
Later..Geez
Several carriers will move multicast data - Sprint comes to mind, I think UUNet does, and I hear there are some other tier 1 providers out there, but I don't use them
Only works as long as all the clients are on multicast aware networks
I am very easy to get along with, but I don't have time to waste being nice to people who are being stupid. -Theo
How to handle this situation primarily depends on where your site is hosted.
If you have your servers hosted with co-location provider like AT&T, Globix, Cable and Wireless (ha ha), Verio ot the like then you'll almost always have the option to "burst" above your monthly allotment of transfer per month - problem solved. Just do some simple math and figure out what is the max of the LAN connection of the provider, how much your server(s) can handle with respect to transfer and you'll be able to figure out if you need to add hardware or NIC's to handle the load. Generally speaking the "bursting" is usually calculated fairly. Also, in months that you use less bandwidth you won't have to pay for a higher class of service above the already agreed upon monthly transfer rates.
However, It doesn't sound like this is the case. A t-1 has a physical capacity to transfer 1.544 Mb/sec. You will never be able to "burst" above this rate. As long as your servers are at the end of this pipe they will never be able to transfer more than ~ 180K/sec to the internet at large. Your options at this point are:
* Add more capacity to your hosting site (expensive).
* "Partner" with a company like Akamai for content delivery. It'll be a few thousand dollars plus the cost of bandwith to set up the content redistribution. Not really a bad deal if you are serious about delivering your content to your users. They also have great reporting about who and where your users are coming from.
* Your last option is to set up a shared/dedicated hosting account with a provider that charges you by the GB. That way you only pay for what you use + the monthly cost of the server. Try interland they have some good deals.
The bottom line is that if your site is hosted at the end of a physical connection you own - it's not going to be enough. You'll need a datacenter environment that has a PhatPipe to the internet and a machine or machine(s) that can handle the throughput. If you have the cash go with Akamai - they are good at what they do.
P2P realizes the two facts that you obviously don't:
1) Not everyone uses their Internet connection 24 hours per da
2) Most people don't need hard drive space these days (i.e. storage is cheap)
Combine these two developments, and you have a lot of upload bandwidth sitting idle. I would argue that P2P becomes MORE effective, not less, as you move to legitimate files, because people are more likely to leave it running when they aren't afraid of the RIAA/MPAA tracing their connection down. Since ISPs have been reluctant heretofore to ban P2P traffic (after all, it is a driving factor for adoption of high-speed Internet access), all those hours people are sleeping with a P2P program open is free downloads for the rest of us.
There are valid complaints against P2P (untrusted incoming executable data, very high latency, no centralized validation of data, etc.) Use them instead of the bandwidth non-issue if you don't like P2P.
I can't The Planet any more highly. They have some very competitive rates, and they are very proffesional. I have some web space for a client there, and just yesterday bought a hosted server off of them. I remomend them to all my clients as well.
Pretty Pictures!
Remember the gollum MTV awards discussion? The .torrent files actually made it into "too many connections" stage. Behold the power of slashdot! Not even your torrents are safe =)
gollum mtv awards
30869: a torrent slashdotted?
543308: too many connections...
Seriously, this the way to do it because it allows you to quickly scale from your tiny little webserver out to a very fast worldwide network of servers on fast networks, yet you don't have to invest in any extra hardware, software, or manpower. With virtually no upfront costs you can have a network just like the big boys on a momemnt's notice, and then turn it back off again just as quickly.
First of all, you need to know what's normal for your network, and how your bandwidth is actually being used:
h no logies_white_paper09186a008017f93b.shtml
d ex .pl?i=Technologies&f=1387
http://www.linuxgeek.org/netflow-howto.php
http://wwwstats.net.wisc.edu/
http://www.arbornetworks.com/products_sp.php
Then you need to make use of technologies which allow you to provision accordingly:
http://www.cisco.com/en/US/tech/tk543/tk757/tec
http://www.cisco.com/pcgi-bin/Support/browse/in
Juniper and other vendors have equivalent capabilities, I'm just not very familiar with them. But the concepts are the same.
Put apache on it. Try to get a contract based on bandwidth used. Configure it to accept connections only when you have the situations when you need it. Have robots.txt turned on to help keep away bots. Make your primary site issue redirects to the ISP apache box when usage gets too high.
It's like a download /. effect problem.
.5$-1$ for another Gig)
There are 2 limitations for common people:
1) ISPâ(TM)s will cap your burst download since they donâ(TM)t want other sites / servers hosted by them to be effected by your burst of bandwidth needs.
2) Your server is caped by the concurrent users, which will pile up since the file served is several MB.
Building a mirror network or using akamai is expensive⦠using kazza or Bit Torrent is too âoegarageâ in thinking and no corporate will use that â¦
I was also thinking about this problem and thought to use file-hosting service:
1) It does not affect my servers
2) You pay by the Gig (15-20$ for 30Gig per month
3) They say they are built for this kind of problem
Using google it looks like FileBurst, Filekicker, Topdownloads and Swmirror are the leading file-hosting companies. Any experiences with these kinds of services?
I don't like the fact that your looking for free labor.
But here if you can read? can you?
Then this item might help you get a clue on how to better use your bandwidth!
http://www.internetwk.com/news/news0420-3.htm
Its called a Xedia Unit and we used it quite efficiently at an old company I worked for back in the late 90's.
Notice the date on the article.
Be sure to tell your colo or file hosting provider what your projected usage is, and how many megabits you may want access to, to assure that they can handle it. You may also want to make a courtesy call a day or so prior to each launch to let them know what to expect.
Remember when Eddy Van Halen got tounge cancer a couple years ago? THAT was a busy weekend for their website, which we host. Of course, they didn't have any warning, but boy-o, that was bigger than any slashdot effect that I've ever seen. We also host O'Reilly (the computer book folks), so we certainly see plenty of slashdotting.
We're at: http://www.sonic.net/sales/colo/
Shop around - but keep in mind that buying from someone near your intended downloader may help you with both latency and costs. The SF Bay Area has the best pricing for bandwidth, and the lowest latency connections to the highest number of users - that said, if your target market is on the east coast, you should be in Hearndon, VA or NY or Boston.
-Dane Jasper (Sonic.net)
-- Dane Jasper Sonic.net, Inc.
A web host that charges ONLY for bytes transferred, and disk space used..
/. readers to use up all my bandwidth.)
Nearly Free Speech is great for that. I host my business' web site with them, and I've gone for more than a year on only my initial $50 deposit. (My company's name can be determined through other means, I'm not going to invite
Another non-functioning site was "uncertainty.microsoft.com."
The purpose of that site was not known.
Well,
:-)
since you are talking about FUTURE, and not now, I would first and foremost use Linux to manage the bandwidth. I assume you haven't made any buying decisions yet?
You can buy propritary heavy lifting equipment such as Cisco gear, and collapse the entire management tier into one switch, but I wouldn't.
It will work fine, BUT you can't reuse Cisco gear as desktop PC's, or even donate them to a school when they get too old.
My way of building things, you can reuse the equipment when its all done.
With that said, I would get a rack of i860 Motherboards, such as SuperMicro's with 6-8 Gigabit Ethernet cards in them and build a BGP based solution.
You get:
1) PC Hardware, commodity items. Easy to replace, easy to repair should something go wrong.
In fact, for some customers I have designed such solutions for, I usually put a spare unit in each rack, on the floor, and just shove it in when I need it.
In any case (4U), each rack would have about 10 machines, with one serving as a switching backbone. I would connect them all with Gigabit Fiber cards into one box. This box forms the QOS SLAVE. We had 6 racks of these, so I had 6 QOS slaves, that then were connected to a Backbone machine that monitored each racks incomming T3 Line, and rerouted the traffic from one rack to another based on how much was comming into each slaves backplane. Each of the 6 slaves was connected to the master, which had Nagios, and the master BGP/ASN lists.
The application was a video feed, from 8 different racks inside the same building, going out AND comming in from various sites. Voice over IP was also used over the same T3 lines.
Insuring enough bandwidth and prioritizing the traffic was key.
Most i860 PCI-Xpress (64Bit PCI) i860 Motherboards, especially the ones from www.supermicro.com can handle 6-8 cards in a chassis. Some manufactures sell external PCI-Xpress chassis and can accomodate up to 16 cards externally.
2) Linux Kernel 2.4.21 coupled with BGP, QOS additions to the kernel and a decent management program like Nagios, will give you all the bells and whistles you get with the big boys.
Except you don't pay millions for such a setup.
3) With such a system, you shold be able to build QOS ratios between BGP ASN's where you can alternatively route to a given pipe through a different rack that has the LEAST amount of activity on it. (i.e. in my case, that was, the least amount of T3 activity, easily monitored of course with Nagios and some simple SNMP scripts which updated the QOS tables, and if required BGP ASN lists on the master machine for the rack.)
You can adaptively route based on QOS signals and change your BGP asn lists to reflect traffic levels in almost realtime.
Been there, done that.
Ironically, my biggest problem with all this, was heat and backup power. 400-500Watt power supplies in a colo, that was originally designed for 4 U chassis with 200-300 watt power supplies, made the place pretty toasty. (78-80 degrees).
It is amazing how much heat the new Xeon processors and motherboards throw off, especially if you MAX OUT all the PCI-Xpress slots in them!
New air conditioning system, and an extra 5K main had to be shunted to the building. Total cost was about 32 Thousand. About 1/4 the cost of the entire project so that was a bummer.
After that the average tempreture dropped to 60 degrees inside!
I loved it, I could watch my pretty Nagios 3D graphs and eat my Diary Queen (one accross the street from the colo) Peanut Buster Parfie while watching billions of bytes go by, at least when I the customer called me to do work at the site.
-Hack
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
Is what they do just what you want? I do not have a clue what they cost to the customers but i've seen alot of akamai hardware spread around different hosting facilities around the europe.
I think their basic product is "x amount of space and bandwidth very close to the end user".
For example, Ximian used akamai at somepoint when their first gnome 1 desktop came out. I downloaded it and installer was *blazing* fast. I was wondering what was going on as i never get +500kb/s speeds from us and i was pretty suprised when the trace stopped at 2 hops into our own datacenter =)
yush
Have the timeout on a file reasonable, though. FTP files generally don't change that frequently, so hold those for 48 hours. HTTP pages can change a lot more often. Make sure the caching server is configured to respect the HTTP flag to not cache, and don't cache unmarked web pages for more than a few minutes.
This should be sufficient to spread out the load some. I prefer to have a few "extras" in there, though. If I know there's going to be a popular file released, it makes sense to have a CRON job pre-load that file into the cache during off-peak hours. That way, it's in the cache and ready for users by the time anyone gets round to requesting it.
If there are a few web sites that are VERY popular but largely static, set up a "neighbor" cache, which ONLY caches those web sites, with a very long time-out. Use a program like Harvest to grab the entire site, via the cache, and you'll then have everything ready for the users. (It'll also be searchable, via Harvest, which'll be a bonus.)
The second option is at the network layer, and should be used only if the above is not sufficient. Enable "diff-services" and "Quality of Service" in the kernel. How to do this depends on the OS you use. Linux and all the *BSDs support these options, but how you set it up varies with each.
Once you've done that, enable "HQF" or "CBQ" queueing discipline, and attach those to the FTP and HTTP services. Configure them such that each user is guaranteed a certain level of bandwidth for that service, if they request that or more. They get more only of nobody else needs it. (This is usually described as a "soft ceiling".)
You also want to enable the "RED" networking option.
This isn't as hard to do as it sounds, and it can massively ease network congestion.
(CBQ = Class Based Queueing. HQF = Heirarchical Queueing Function. RED = Random Early Detection.)
Once you've applied both the "high level" and "low level" solutions, your network congestion should be massively eased. Again, though, use pre-loading for the cache as extensively as you can to ease those peak-time burdens.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Post a link on Slashdot that says: Irrefutable proof that SCO took code from linux instead of the other way round.
Put your server in a colocation center. You can normally get deals where you pay for a low bandwidth regularly, but have the ability to burst quite high. As an example, you can be connected to a 100 megabit port on their router, but pay a base rate for, say, 1 or 2 megabits.
If that's not an option, you can always look into per-IP bandwidth limitting and QOS.
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
Here's the easiest solution yet: Distribute your file via BitTorrent. Problem solved!
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
an excellent hosting firm with access to lots of bandwidth with pay-by-the download pricing is rackspace.com
if you are looking at huge volumes get a rack or two in co-los from a good backbone provider like level three
both options are cheaper than the rolls-royce option, a CDN like akamei or digital island
He'd be happy to make money customizing it for people...
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Try ww.nearlyfreespeech.net. They offer web hosting for $1/gigabyte downloaded. Then just put the problem file there and link your downloaders to it. The trouble with most billing methods is that they are generally non-linear, with a huge increase in the per byte cost once you exceed some contracted amount. Burstable billing is the worst - if someone comes along with a big pipe and downloads for 6% of the month, you can have a huge bill for a modest amount of bytes.
You didn't specifiy the pattern of irregularity...
Once a month? Every 6 months? Once a week? What?
Anyway, why wouldn't you colocate a distribution server, or lease a dedicated box, whatever... Most datacenters will accomodate all kinds of bandwidth patterns, from per byte to burstable to dedicated rates. IMO, thats the most versitile solution. Furthermore, all the p2p (bitorrent) suggestions may work to reduce some costs, but you'll still need a solid service to get the file out to those first p2p clients, and to get it to those unwilling to use p2p type services.
I dont' remember that game in the warez releases... hmmm. Walnut Creek .. Sounds kinda boring.
Modesty is one of life's greatest attributes
Disclaimer: This particular site is run by friends of mine, but they're not the only ones who do this...
You think you're going to throw 20 rackshack servers at the problem? *giggle-snort-giggle*
Rackshack and the other bottom-of-barrel hosters might be okay for giving away porn and hosting hit-n-run spammer/scammer fraud sites, but other than that it's crap bandwidth on a crap network with crap hardware designed and maintained by crap adminstrators. I don't think anyone that is serious about quality would bother using rackhsack.
Oh, and about P2P? The fact that so many posters even suggest it proves that Slashdot is a haven of "power users" and real professionals don't even post here anymore except to troll.
You won the weekly /. bad humor award! Take a bow.
Karma: Dyn-o-mite!(mostly affected by Jimmy Walker reading your comments)
Avoiding Network Capacity Collapse
What I would do is colocate all my servers in a datacenter.
Take a burstable bandwidth, let's say that can burst to 100mbs, but to control your bandwidth in most time to ensure you do not go over the cost, you configure your router to not allow more than let's say 1mb of bandwitdh or whatever you want as a maximum and willing to pay for in normal time.
You should then monitor your bandwidth usage in real time, as well as the logs on the machines, and adjust the traffic shaping to the amount of traffic you want to allow.
For example, you know what on that day, you will do a marketing operation, and you are willing to spend $xxx more for the bandwidth, you then change your setting right before your marketing plan to the maximum of bandwidth you are willing to pay.
my 2c...
not to mention the problem of "have you ever gotten a file from bit torrent that was invalid?" I have.
Really? I thought BitTorrent used MD5 or SHA1 hashing to make sure the file transfer had no errors.
Will I retire or break 10K?
Who in their right mind modded the parent to +5???
"Bandwidth trading" escapades have bancrupted Williams and contributed to Enron having to go ever more aggressive with their "creative accounting." It reeks of the same foul smell as the "real options" nonsense. (Note: the article referenced even has "real options" in the keywords).
Basically you've got a whole bunch of not-so-smart people who can fake expertise in nearly anything as long as they don't get their hands dirty (aka academics) colluding with some not-so-smart MBA's in executive positions - and then you've got a problem for everyone around them.
This is a problem that needs to be priced based on actuarial type methods (a'la insurance). Just because you can write a (usually bad) price process for "bandwidth" doesn't mean that you can logically apply all the consequent financial mathematics.
And, here's a party-pooper - real options are not options at all - no matter what tenured clowns like Pindyck and Dixit want you to believe.
Spike-free billing is available by purchasing file hosting a per gigabyte transferred basis. Do not pay more than $1 per GB and only pay for completed downloads and insist on real-time reporting. To compute your budget in advance get the file size in bytes and multiply by the number of downloads and divide by 1073741824 (bytes/GB). Make sure the provider has redundant gig lines and an adequate server farm to handle your anticipated number of simultaneous connections.
Why not set up some sort of cache between the users and the internet, that way new requests can be served over the usual means and if other users want the same content its cached. It would have to be a large cache I know but if you used some sort of database and flushed content after 24 hours or something you may get away with a couple of hdds. This way the excess transfers are limited to your LAN rather than T1 connections.
Unfortunately, no one can be told what my sig is...
Colocate a system or two at an ISP that charges by transfer for the large traffic content, and upload updates to that over a standard Internet link using rsync or ftp. Then, you only need to pay for actual data transferred.
If your company is adamant about having everything in-house, get a Point-Point link to the nearest Internet Nexus (probably not far if you're in any sort of city), and terminate its far end in an ISP that will charge by transfer instead of by allocation.
Ideally, lease dark fiber for this link so that you don't need to pay a telco to carry your traffic over their network, although since you're not buying transit from the telco, it will drop the price of the link significantly.
I work for a company that hosts 40+ websites and back office applications on its Internet infrastructure.
I would recommend what we have rolled out since it is working well. We have multiple T3 connections (each to a different ISP and diverse local loop routing for redundancy). You can order burstable T3 connections from the ISPs. They bill a certain rate for your contracted bandwidth (95th percentile calculation) and then you pay extra for extra bandwidth used. This works well since with the 95th percentile billing we still get several hours at peak utilization before we go over our limit.
You must keep close track of your utilization to keep the ISPs honest and to make sure your base level is set correctly and that you are not exceeding it too often (a plug for MRTG should go here although I have no affiliation with the project, just an appreciator). If you are exceeding the burst level frequently, at some point it may become more cost effective to increase the base than to pay the burst fees.
Brush up on your BGP also to help you balance your traffic.
Joe
For corporate downloads of legal content, there's no reason a P2P system need be anonymous. See BitTorrent for an example of that. Having your P2P software handle searching as well as downloads -- well, you could, but why not use a specialized engine for each rather than trying to have one piece of software that Does It All?
Based upon my experience, ZeroLag Communications is what I would recommend. ZeroLag offers flexible contract and base bandwidth usage upon 95th percentile, which allow me to burst-bandwidth from time to time and will never get bill for. Infrastructure within ZeroLag is incredible, they are tier 1 ISP with Gigabit backbone, extremely low ping time from just anywhere. SoundBlvd.com, which is my site that I colo within ZeroLag network, usually use very low bandwith. Last month, my product features in Total BMW Magazine. which cause the traffic to spike from time to time, ZeroLag was able to handle the traffic with ease. My actual contract was only for 1 Mb/s but burstable up to 100Mb/s, which I have no doubt that their network can. Anyway, I am sure that ZeroLag can sure come up with plan that will suite your need best just like mine.
As mentioned above by Zav, P2P will not work as a viable business solution for managed bandwidth. Companies need to absolutely guarantee that their data delivery will be intact and unaltered. P2P systems allow for Companies data to be altered in transit to the end person. This could have nasty side effects such as virus and Trojan horses being planted in the data.
The Company I work for has been providing highly burstable solutions for many years. We hosted all of the 20th Century Fox and Paramount Picture movies web sites for three years each. For example, we hosted the Titanic Movie, the most trafficked movie web site ever. Everyone had expected that movie to be a flop and when it was a huge hit, we had to have many extra megabits available to satisfy the millions of hits the site was getting.
The experience of hosting two previous Star Trek movie web sites was also very valuable. IBM would be running TV commercials featuring the Star Trek Movie for their Ecommerce solutions and the web site would be hammered with 10,000 accesses per second. As an interesting side note, this is when we found out that Apache and Linux on a $3,000 PC would outperform IRIX and Netscape Web Server on a $50,000 Silicon Graphics Server. We have been a primarily Linux shop ever since.
The knowledge and experience gained by hosting these web sites that needed to be ultra scalable is applied to all our current clients.
We offer both GB per month solutions and Megabit per second solutions. The comments posted above that GB per month is a better value is actually not always true. It will really depend upon your needs. ZeroLag Communications can provide solutions either way and we will show you which way your dollar would go further.
One specific technology we would offer to Companies like Zavâ(TM)s is compression. This has the effect of speeding up a web transfer and at the same time reducing the bandwidth. This is a win/win for both parties. Compression may not work in every situation, but when it does work, the effect is quite amazing on the users experience.
Another technology is an overly engineered network. Our capacity to deliver bandwidth far exceeds the needs of our clients. Most other companies offer the opposite solution or an oversubscribed network model. An oversubscribed network has many dropped packets which often adds extra retransmissions when you least want them, when you have bursting traffic. The person who posted above with ADSL described the problem perfectly. When not being able to send back an ACK successfully, this causes a retransmission. This means MORE bandwidth must be used to send the data and therefore further slows the download times and increases the Companies bandwidth costs!
ZeroLag on the other hand, never drops packets because our internal and edge networks are capable of transfering multiple Gigabits. Our top of the line Cisco Routers, Firewall and Switches are idle even during heavy traffic surges. This scalability is ideal for companies who are concerned about being able to burst.
One other advantage of going with a Company like ZeroLag is we can help your Company build out the servers it will need. We can help you build out a mission critical, fault tolerant, scaleable and secure server infrastructure. This way you can be driving 40 Mph today, jump to 150 Mph hour occasionally when needed and be able to go 220 Mph next year as the business builds.
Please feel free to contact me at ZeroLag Communications if you would like more information.
Thanks,
Greg Strelzoff
Greg at ZeroLag dot com
As the first local court system in the US that streamed their own court proceedings via the Internet, we were inundated with hits to our media server. By using the bandwidth throttling in Real Server we were able to maximize the amount of clients until our bandwidth was completely exhausted. Granted we only had a T1 at the time, but we did not expect the world wide, traditional. media exposure that we received. At the peak we were getting over one thousand hits per minute.
What have we done since? Currently we have added an additional T1, and have a 20Mb Ethernet connection on order.
With the costs of dedicated "high-speed" 'net connections at very reasonable rates (less than $2500 per month for 20Mb) and the intelligent use of client bandwidth throttling it would be very easy to do what you are wanting to accomplish.
Phear The Phat Penguin
hmmm... pot kettle black.
I think you're being a bit closed minded about this ... mxs didn't say to tell your customers to get it off eDonkey. He said to consider building your own P2P into the installer. And I think he meant to consider colocating somewhere and to shop around.
Just for clarification, P2P is not limited to crap like BearShare, eDonkey, or Kazaa. P2P is a term that implies peer-to-peer information sharing, that could usefully be built into many applications... like program installers, internet telephony (H323 -- NetMeeting & GnomeMeeting & GnomePhone (all P2P already)), encrypted|(voice&video) IM programs, cpu load balancing, etc.
The UK's largest supplier (Energis) takes a unique approach to co-location... they charge a flat ate per server/rack, regardless of the bandwidth you use (>100Mbps).
;)
Prices start at $400/month fora single server and $800/month for a full rack. $800/month for 100Mbps line. Now that's a deal.
Allows you to scale too
Your best bet is to use a web host. Companies such as Server Beach(advertising on slashdot) or Rack Shack offer you ridiculous amounts of bandwidth for $99 a month. If you need a less "dirtier" approach to hosting you can go with companies such as RackSpace. These companies allow you to change your bandwidth requirements monthly. So if you know your going to be releasing a new version of your software next week you can get them to up the bandwidth for that month. If you go over your allocation most hosts will charge you at a set rate which is decided before you join the host. The later sort of hosts can offer you load balancing so that you can manage your CPU time etc. You can find hosts like these on some of the many web host directory sites such as http://www.webhostdir.com or http://www.tophosts.com. To find out what others have to say about them, you can look through some of the many forums such as http://forums.webhostdir.com or http://www.webhostingtalk.com. The forums often give you an idea of any problems customers faced with hosts and would be able to provide you with more options than you would get here. Getting an expert to manage your hosting is often better. They have multiple data lines in to their data centers, power generators if the power goes down, and so much more to help keep your site serving your customers. At the end of the day the benefits offered by professional hosts gives you much more value for money than buying and managing your own boxes but if you must do it yourself, I think the general rule of thumb is to keep your line uner 75% of capacity at all times. There must be a network nerd who can confirm that for me.
Emule let's you throttle your upstream and it's been doing so since I know it.
I agree... Burstable pricing will kill you. think about it, 2.5 days of burst, can kill your 95%!! You should also be wary of the quality of the bw you are getting. If its cheap, it most likely is Cogent. We had a contract with them for 1 year, could not use it, because some of our customers are VOIP providers, and the latency would spike @ 10Mb. I work for a small sooftware company, and all we are about it Bandwidth pricing and trading. We liscecne our market technology to other large Carriers, as well as run our colo business, that utlizes our own technology, to maximize customers needs. Invisible Hand Networks (IHN), developed an IP pricing and provisioning platform called Merkato, at Columbia University. Currently, we operate and participate in multiple global markets. We also operate a service business, (Invisible Hand Services, formerly StreamingHand) based on this technology, that provides "on demand" IP capacity for content delivery purposes. The service allows our customers to purchase bandwidth as needed in as little as five minute increments or as long as they need. These customers simply configure an intelligent software "agent" which then purchases bandwidth on the customer's behalf in response to real-time traffic monitoring for real demand. Because of the efficiencies realized under this model, cost savings for the customer are significant. For instance, the price per Mbps of bandwidth has averaged approximately $125.00 per Mbps per Month in recent months. And because it is purchased in 5 minute increments it mimics a megabyte delivered model where the customer is only really paying for what they need. For Example: a customer hosting a game demo or streaming media file would configure the agent to respond to demand for these objects. If the traffic were to pick up to 70 Mbps for 2 hours they would pay $23.52 ($125.00*70Mbps/31days/24hours *2 hours)). Because of the "Free Market" nature of our technology, we do not utilize term contracts or monthly cost minimums. In short, if you don't use the bandwidth, then you don't pay anything. Customers in your line of business, usually start out by hosting your burstable traffic with us and allowing the system to buy your optimal bandwidth corresponding to your demand, not your contract. If you want to pay for only what you use, then check us out: www.invisiblehand.net or you contact us at sales@invisiblehand.net Good luck..
Si serÃn pelotudos, que pusieron un link a una "banda" pero de mÃsica, je je
bueno, quiero mandarle un saludo a todos los que estÃn escribiendo
y leyendo
en español
vamo arriba el español !
Los latinos estamos siendo mayorÃa !
Find a content-delivery provider. Something like cw.com or akamai.
You should buy a ram jet powered locomotive... never a problem with speed or overloading your hauling capacity... be sure to lease it.
Getting a burstable line to the net is not hard at all; most ISPs work that way (you pay for 6 mbps, for instance, but have a 100 mbps ethernet line, so you just pay when you burst above 6).
One thing you guys might want to think about is using a bandwidth arbitrator for when you do have a busy day. There's one good project I know of: the Linux Bandwidth Arbitrator. It's easy and free, and it'll keep individual users from hogging bandwidth -- and meter all users to whatever rate you choose. It's based on the Linux bridging code, so the arbitrator will be invisible above Layer 2 (thereby being a little more secure).
I, for one, welcome our new Antichrist overlord.
I agree... Burstable pricing will kill you. think about it, 2.5 days of burst, can kill your 95%!!
You should also be wary of the quality of the bw you are getting. If its cheap, it most likely a very low quality provider. We had a contract with one "C" named provider for 1 year, and could not use it, because some of our customers are VOIP providers, and the latency would spike @ 10Mb.
I work for a small software company, and all we are about it Bandwidth pricing and trading. We liscecne our market technology to other large Carriers, as well as run our colo business, that utilizes our own technology, to maximize customers needs.
Invisible Hand Networks (IHN), developed an IP pricing and provisioning platform called Merkato, at Columbia University. Currently, we operate and participate in multiple global markets. We also operate a service business, (Invisible Hand Services, formerly StreamingHand) based on this technology, that provides "on demand" IP capacity for content delivery purposes. The service allows our customers to purchase bandwidth as needed in as little as five minute increments or as long as they need. These customers simply configure an intelligent software "agent" which then purchases bandwidth on the customer's behalf in response to real-time traffic monitoring for real demand.
Because of the efficiencies realized under this model, cost savings for the customer are significant. For instance, the price per Mbps of bandwidth has averaged approximately $125.00 per Mbps per Month in recent months. And because it is purchased in 5 minute increments it mimics a megabyte delivered model where the customer is only really paying for what they need.
For Example: a customer hosting a game demo or streaming media file would configure the agent to respond to demand for these objects. If the traffic were to pick up to 70 Mbps for 2 hours they would pay $23.52 ($125.00*70Mbps/31days/24hours *2 hours)).
Because of the "Free Market" nature of our technology, we do not utilize term contracts or monthly cost minimums. In short, if you don't use the bandwidth, then you don't pay anything.
Customers in your line of business, usually start out by hosting your burstable traffic with us and allowing the system to buy your optimal bandwidth corresponding to your demand, not your contract.
If you want to pay for only what you use, then check us out: www.invisiblehand.net or you contact us at sales@invisiblehand.net Good luck..