Slashdot Mirror
Are Web Pages Getting Larger?
An anonymous reader asks: "I work for a large multinational in a remote part of world. Our connectivity to the outside world (the Internet as well as company communications) is all done via a single E1 line - that's 2Mbps. Thousands of users. The company keeps access pretty well screwed down for security reasons, and the fact that our link to the outside world costs almost $300K/year! Our growing problem is Internet traffic. While policing of non-business use is very active, Internet traffic continues to grow. I'm becoming convinced that one of our problems is that average web page size is growing. As more of the world enjoys broadband access, I think web developers have less reason to limit the size of their web pages. Large images, flash animations and other size-increasing content seem increasingly common. Am I right? Can anyone point to a recent study that would support my theory, and help me convince my management that we just plain need more bandwidth?"
I think the answer to your question lies within the technology itself, and the obvious answer is "yes", web pages are getting larger. Consider that:
So, yes, the web universe is "expanding" in very nearly every dimension. To your specific question, will you need to petition for more bandwidth? Undoubtedly. And, I can't imagine it isn't doable at today's rates. It sounds like a balky bureaucracy, not a question of need. Good luck.
I think maybe the better question to ask, is what has happened to the general psyche of the average employee, and how do you address it? If I had to guess (see, I'm not proving anything with this post!) I'd guess the technology has easily stepped up to the task of underpinning the network use but people still have not learned how to modulate and attenuate the siren that is the internet. (Maybe that would help decelerate your need to upgrade and expand bandwidth.)
You could add a local caching proxy server and/or set browsers to cache longer to reduce bandwidth. Have you done an analysis on how much of the traffic is people just pulling up the same pages?
-- these are only opinions and they might not be mine.
Lynx
A website and all of its pages can be expected to grow over its lifetime, but a lot of newer sites are lot smaller than previous generations. The wide adoption of CSS, and all the user friendliness tech evangalism emphasizing simplicity over noise has been paying off those who listen. There are still a lot of sites, such as web forums, where the attitude seems to be to make have really complex themes with almost no CSS and let mod_gzip/deflate deal with the task of making it small.
You need to change policy, not spend more money. Change the cache settings on clients. Insert caching proxy servers. Make sure mail, DNS, etc. is local. Et cetera. You should find a solution that does not have a linear (at best) relationship with the number of users.
Web pages are getting larger. It might be what is causing your increase in utilization, but to me it's hard to believe (although if your users are viewing a lot of embedded videos, that's another story). And, if it's hard for me to believe, it's probably going to be hard for your PHB to believe, too.
"Duh! Here's more content"
With the broadband market now including a minimum of 25% of home users, and up to maybe 40%, though I haven't looked at those numbers in some time, would be a contributing factor to the fact that yes, web pages are getting bigger.
One way to see proof of this is using the wayback machine.
http://www.waybackmachine.org/
I took a quick sampling of the NYTimes homepage, and noticed that the number has increased by a few kilobytes per year, from 56K in 2001, to 67K in 2003, to 83K in 2005. That's not even counting images. They've added more ad banners since the old days. If you google search, I'm sure you will find stuff.
Ad banners have increased in size, and complexity over time. Streaming content, is another addition, as well as more services running over the network.
You probably have a number of contributing factors happening to your bandwidth, in addition to web pages.
- Unless you have an internal instant messenging environment, you may have many ppl chatting away on services having to use your bandwidth.
- Email for personal use. Jokes, funny attachments, and worms clogging up things.
Here are a couple of suggestions to try and improve traffic:
- block services that shouldn't be run at the office like streaming music content.
- block websites that you see can have an impact on traffic, that you believe users should not be visiting. ie: quicktime movies.
- block your daytrading slacker coworkers.
- block ad servers entirely! this should drastically improve your situation, and be the easiest to implement.
- switch to an internal instant messenging service, if you haven't done so already.
- disable unnecessary services.
- ensure that you have an internet policy that prohibits the users from using their work companies for personal use.
- cache often used content.
according to archive.org/waybackmachine:
html size (doesn't include images/dependencies)
slashdot.org yahoo.com microsoft.com
1996 - 7k 11k
1997 - 9k -
1998 23k 10k 20k
1999 35k 10k 20k
2000 36k 12k 17k
2001 41k 16k 21k
2002 39k 17k 28k
2003 39k 32k 31k
2004 51k 33k 38k
Today 19k 14k 22k
the trend has certainly been up, but lately big sites' main pages seem to be slimming down, due to CSS as well as a tendency to store style and javascript in separate file
Web developers (and programmers in general) don't care about optimizing anymore, they just want it to be done so they can get paid. Worrying about such trivial things as a few kbytes or making valid and accessible HTML is asking too much of them.
From a web-designer standpoint, a lot of size can be reduced without altering the content.
Are you serving up nicely formatted HTML with indentations? That's wasteful. Strip whitespace and carriage returns.
Are you using HTML comments? Why? Does the customer really need to see them? Do you need to waste that bandwidth? Delete them or use comments in your server-side scripting language of choice.
Are you using GIF's where PNG's would be smaller? Or PNG's where GIF's would be smaller?
Have you optmized your PNGs, JPEGs and GIFs? (I don't remember a GIF optimizer, but there are plenty of non-destructive ones).
A 50x50 JPEG preview of an item does not need embedded comments, thumbnails, or EXIF data.
If you must use animated GIF's, be sure they are optimized and not full-frame!
Are you using pictures of words, when actual stylized text could convey the same message?
Are you using inline JavaScript or CSS, rather than calling it from a cacheable external file?
Are you using PDF, Flash or Java when it's not ABSOLUTELY necessary?
From a user's standpoint, the best solution, short of getting more bandwith: use less bandwidth. Turn off image loading or use a text-based browser. Don't browse the web as much. If you have a choice of sites to use, use the one that is smallest. Use a proxy. blah blah.