Are Web Pages Getting Larger?

An anonymous reader asks: "I work for a large multinational in a remote part of world. Our connectivity to the outside world (the Internet as well as company communications) is all done via a single E1 line - that's 2Mbps. Thousands of users. The company keeps access pretty well screwed down for security reasons, and the fact that our link to the outside world costs almost $300K/year! Our growing problem is Internet traffic. While policing of non-business use is very active, Internet traffic continues to grow. I'm becoming convinced that one of our problems is that average web page size is growing. As more of the world enjoys broadband access, I think web developers have less reason to limit the size of their web pages. Large images, flash animations and other size-increasing content seem increasingly common. Am I right? Can anyone point to a recent study that would support my theory, and help me convince my management that we just plain need more bandwidth?"

Comments

the answer is...

[yagu]

I think the answer to your question lies within the technology itself, and the obvious answer is "yes", web pages are getting larger. Consider that:

• processor speeds have increased nearly 80x since the approximate time of the "web".
• disk and memory have increased even more dramatically than processor speeds
• average bandwidth (more optical networks, more 100M LAN's, etc.) has increased dramatically.
• features and functionality of browsers has expanded dramatically (sorry, not going to go into the laundry list here)
• the number of web sites has grown exponentially
• the ways of finding these sites has become easier if not just plain simple

So, yes, the web universe is "expanding" in very nearly every dimension. To your specific question, will you need to petition for more bandwidth? Undoubtedly. And, I can't imagine it isn't doable at today's rates. It sounds like a balky bureaucracy, not a question of need. Good luck.

I think maybe the better question to ask, is what has happened to the general psyche of the average employee, and how do you address it? If I had to guess (see, I'm not proving anything with this post!) I'd guess the technology has easily stepped up to the task of underpinning the network use but people still have not learned how to modulate and attenuate the siren that is the internet. (Maybe that would help decelerate your need to upgrade and expand bandwidth.)

Caching.

[slashkitty]

You could add a local caching proxy server and/or set browsers to cache longer to reduce bandwidth. Have you done an analysis on how much of the traffic is people just pulling up the same pages?

Re:Caching.

[duffbeer703]

I'm sure that a company that can afford a $300k E1 circuit in some third world shithole knows about caching proxy servers.

Content-Encoding: gzip

[DrSkwid]

The un-adoption of mod_gzip and whatever IIS *should* use is also prevalent.
Ticking the box used to crash IIS but these days it actually works, not that you'd notice :

Response Headers - http://www.microsoft.com/

Date: Thu, 08 Dec 2005 10:30:40 GMT
Content-Length: 23186
Content-Type: text/html; charset=utf-8
Cache-Control: private
Server: Microsoft-IIS/6.0
P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727

200 OK

Response Headers - http://slashdot.org/

Transfer-Encoding: chunked
Date: Thu, 08 Dec 2005 10:40:11 GMT
Content-Type: text/html; charset=iso-8859-1
Cache-Control: no-cache
Server: Apache/1.3.33 (Unix) mod_gzip/1.3.26.1a mod_perl/1.29
SLASH_LOG_DATA: mainpage
X-Powered-By: Slash 2.005000090
X-Fry: Where's Captain Bender? Off catastrophizing some other planet?
Pragma: no-cache
Vary: User-Agent,Accept-Encoding
Content-Encoding: gzip

200 OK

Re:An E1 costs $300k/yr?

[anticypher]

An E1 data circuit via a satellite channel to Africa or the Middle East will run about US$125k to US$200k/year, in satellite costs, uplink and downlink station maintenance, and the actual internet connection in Europe or NYC.

Compressors, TCP (packet shaping) optimisers, proxy caches, DNS/email caching, webvertising blocks, QoS and agressive firewall rules are pretty much a given for any kind of expensive satellite connection. On the luser end, to really make use of the web they can set their browsers to not automatically load images, change their TCP window to something huge, and a bunch of other tricks to keep themselves happy. Remote stations with large numbers of geeks have NNTP servers locally to keep up on the non-web world. IRC/IM is quite widely used, because they don't use much bandwidth at all (although I've heard of remote stations banning MSN messenger because it won't work without constantly loading advertising images)

But really, US$300k per year for an E1 circuit? There isn't any place on earth still that expensive. Drop me an email, we'll do lunch.

the AC