Slashdot Mirror
Are Web Pages Getting Larger?
An anonymous reader asks: "I work for a large multinational in a remote part of world. Our connectivity to the outside world (the Internet as well as company communications) is all done via a single E1 line - that's 2Mbps. Thousands of users. The company keeps access pretty well screwed down for security reasons, and the fact that our link to the outside world costs almost $300K/year! Our growing problem is Internet traffic. While policing of non-business use is very active, Internet traffic continues to grow. I'm becoming convinced that one of our problems is that average web page size is growing. As more of the world enjoys broadband access, I think web developers have less reason to limit the size of their web pages. Large images, flash animations and other size-increasing content seem increasingly common. Am I right? Can anyone point to a recent study that would support my theory, and help me convince my management that we just plain need more bandwidth?"
I think the answer to your question lies within the technology itself, and the obvious answer is "yes", web pages are getting larger. Consider that:
So, yes, the web universe is "expanding" in very nearly every dimension. To your specific question, will you need to petition for more bandwidth? Undoubtedly. And, I can't imagine it isn't doable at today's rates. It sounds like a balky bureaucracy, not a question of need. Good luck.
I think maybe the better question to ask, is what has happened to the general psyche of the average employee, and how do you address it? If I had to guess (see, I'm not proving anything with this post!) I'd guess the technology has easily stepped up to the task of underpinning the network use but people still have not learned how to modulate and attenuate the siren that is the internet. (Maybe that would help decelerate your need to upgrade and expand bandwidth.)
You could add a local caching proxy server and/or set browsers to cache longer to reduce bandwidth. Have you done an analysis on how much of the traffic is people just pulling up the same pages?
-- these are only opinions and they might not be mine.
Yes, it's like how many software developers write lazy code these days since memory is cheap. Bandwidth is cheap now so many web developers are writing lazy pages.
Or it could be the dreaded Skype sucking up all your bandwidth as it has done at my company.
Lynx
A website and all of its pages can be expected to grow over its lifetime, but a lot of newer sites are lot smaller than previous generations. The wide adoption of CSS, and all the user friendliness tech evangalism emphasizing simplicity over noise has been paying off those who listen. There are still a lot of sites, such as web forums, where the attitude seems to be to make have really complex themes with almost no CSS and let mod_gzip/deflate deal with the task of making it small.
Ouch. I assume it's the European counterpart to the T1, T1s in the US are a lot cheaper, like 2% of that cost. I suppose you really mean the "remote part of the word" part.
But yes, I wouldn't be surprised if web pages are getting larger. On average, web pages are a lot more complex than in the past, with lots of links and sections and borders done with lots of little images. There are more ways to post images now, probably less care in compressing them, rather than compress them with care to not ruin the image too much and yet keep the file small, they might be less compressed or simply PNG-ed which can result in pretty large files.
You need to change policy, not spend more money. Change the cache settings on clients. Insert caching proxy servers. Make sure mail, DNS, etc. is local. Et cetera. You should find a solution that does not have a linear (at best) relationship with the number of users.
Web pages are getting larger. It might be what is causing your increase in utilization, but to me it's hard to believe (although if your users are viewing a lot of embedded videos, that's another story). And, if it's hard for me to believe, it's probably going to be hard for your PHB to believe, too.
If you've been on the web for a long time, AND still use dialup (that would be moi), you can distinctly detect that webpages in general are a lot more busy and "larger". They just cram more stuff per URL now. I remember surfing happily, including images, on a 25 megaherz system using 64 megs of RAM with a 9.600 modem. Yes, some sites were slow, especially those that used a ton of images on the same page,but it was most doable for most pages.. but now??? sheesh...
.com addy.
disclaimer: conversational and "average" claims, exceptions abound, no facts, just anecdotal SWAG, YMMV, of course "your" experience is different, and etc.
I run slashdot "lite" precisely for this reason in fact. Loads much faster than full bloat, all the same articles and comments, less "stuff" you don't really need. I wish more sites had it, in fact, I wish there was a way to always get the "lite" or "print this page" or "text only" version of sites directly from the search engines, like a preferences selection. FF/moz plugin??? Would be nice. You see it all the time with internet vids, they have broadband or dialup choose one. No reason web pages in general couldn't have that now, and have it in the tags so that the search engines could automagically put those URLS as the main "clicky" with your searches or going to the generic
"Duh! Here's more content"
With the broadband market now including a minimum of 25% of home users, and up to maybe 40%, though I haven't looked at those numbers in some time, would be a contributing factor to the fact that yes, web pages are getting bigger.
One way to see proof of this is using the wayback machine.
http://www.waybackmachine.org/
I took a quick sampling of the NYTimes homepage, and noticed that the number has increased by a few kilobytes per year, from 56K in 2001, to 67K in 2003, to 83K in 2005. That's not even counting images. They've added more ad banners since the old days. If you google search, I'm sure you will find stuff.
Ad banners have increased in size, and complexity over time. Streaming content, is another addition, as well as more services running over the network.
You probably have a number of contributing factors happening to your bandwidth, in addition to web pages.
- Unless you have an internal instant messenging environment, you may have many ppl chatting away on services having to use your bandwidth.
- Email for personal use. Jokes, funny attachments, and worms clogging up things.
Here are a couple of suggestions to try and improve traffic:
- block services that shouldn't be run at the office like streaming music content.
- block websites that you see can have an impact on traffic, that you believe users should not be visiting. ie: quicktime movies.
- block your daytrading slacker coworkers.
- block ad servers entirely! this should drastically improve your situation, and be the easiest to implement.
- switch to an internal instant messenging service, if you haven't done so already.
- disable unnecessary services.
- ensure that you have an internet policy that prohibits the users from using their work companies for personal use.
- cache often used content.
There may be more unique arrangements of bits to collect, but as for content, most of the Internet's rather slim. Mostly it's personal websites, livejournal and its analogues, sales, and aggregators. On the side, you get some regular news sites.
I've found some informative and interesting personal websites, including nuwen.net and a fair bit about amateur linguistics. Other than that, if you want more content, the best place to look is universities.
It's not more content, generally, just more media.
this is one of the dumbest /. stories ever
I mean wtf....are they getting bigger? you mean...you mean as more and more people attach databases? and...and...java? etc...
I mean really...how stupid is this
We seldom regret saying too little but often regret saying too much.
Welcome to the club.
It isn't just low-economy bandwidth ISP that is suffering this onslaught of dazzling eye-candies of HTML pages.
Blackberries and java-enabled cell phones also face the same problem. But, there is a way as other postings have shown -- web proxy cache.
One can defer uploading of images with a like-sized null (one-dot) image placeholder. These images, if the end-user desires, can be retrieved selectively. This is the BIGGEST bandwidth saver. Other is filtering of ad-contents.
Otherwise, it's more sucking raw eggs through tiny straws for ya!
according to archive.org/waybackmachine:
html size (doesn't include images/dependencies)
slashdot.org yahoo.com microsoft.com
1996 - 7k 11k
1997 - 9k -
1998 23k 10k 20k
1999 35k 10k 20k
2000 36k 12k 17k
2001 41k 16k 21k
2002 39k 17k 28k
2003 39k 32k 31k
2004 51k 33k 38k
Today 19k 14k 22k
the trend has certainly been up, but lately big sites' main pages seem to be slimming down, due to CSS as well as a tendency to store style and javascript in separate file
http://www.google.com/search?hl=en&lr=&safe=off&q= average+web+page+size
w -much-info/
w -much-info/internet.htmlw -much-info-2003/internet.htm
has some good results
http://www.sims.berkeley.edu/research/projects/ho
has info from 2000 and a link to the same info from 2003
specific internet 2000
http://www.sims.berkeley.edu/research/projects/ho
and 2003
http://www.sims.berkeley.edu/research/projects/ho
it's worth noting, these types of statistics can take a year or more to compile..
every day http://en.wikipedia.org/wiki/Special:Random
With low bandwidth connections, the quality of the experience of accessing information over the web is lower, and so people access less.
It is true that broadband has allowed pages to be bigger than before, but trends in UI design have also resulted in more text and fewer images, and with mod_gzip a mostly text/css site ends up being a lot lighter than its image/table predecessor.
I suspect that if you decreased the quality of the surfing experience by introducing about 1.5 seconds of latency into each web request (this would add up as most pages require multiple requests), you'd see the bandwidth utilization drop as people thought more carefully before clicking on links, etc.
Amazing magic tricks
Web developers (and programmers in general) don't care about optimizing anymore, they just want it to be done so they can get paid. Worrying about such trivial things as a few kbytes or making valid and accessible HTML is asking too much of them.
From a web-designer standpoint, a lot of size can be reduced without altering the content.
Are you serving up nicely formatted HTML with indentations? That's wasteful. Strip whitespace and carriage returns.
Are you using HTML comments? Why? Does the customer really need to see them? Do you need to waste that bandwidth? Delete them or use comments in your server-side scripting language of choice.
Are you using GIF's where PNG's would be smaller? Or PNG's where GIF's would be smaller?
Have you optmized your PNGs, JPEGs and GIFs? (I don't remember a GIF optimizer, but there are plenty of non-destructive ones).
A 50x50 JPEG preview of an item does not need embedded comments, thumbnails, or EXIF data.
If you must use animated GIF's, be sure they are optimized and not full-frame!
Are you using pictures of words, when actual stylized text could convey the same message?
Are you using inline JavaScript or CSS, rather than calling it from a cacheable external file?
Are you using PDF, Flash or Java when it's not ABSOLUTELY necessary?
From a user's standpoint, the best solution, short of getting more bandwith: use less bandwidth. Turn off image loading or use a text-based browser. Don't browse the web as much. If you have a choice of sites to use, use the one that is smallest. Use a proxy. blah blah.
Anyway, I'm just kidding... or am I?
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
You need a smart gateway. Your E1's border router, or a gateway immediately behind it, needs traffic shaping and queueing. Pretty much any circuit anywhere needs traffic queueing. Either side of your E1 could probably benefit from a compressed virtual circuit such as maybe a VPN. Compress all traffic that way. If you locally host your web servers, you can use a reverse proxy that includes mod_gzip and other stuff to strip whitespace from their content. You can also control your users' behaviors with caching proxies like squid and with a layer 7 packet filter. The layer 7 filter will protect against p2p and such. If you think the network is being abused but you want to encourage self-censorship, make the squid logs public. :)
For one, you can use a good proxy to cache webpages that are constantly brought back up. For two, you can use it to block out animated GIFs and flash game/ads that take so much bandwidth. I hope this helps.
Youre an employee there. Youre part of the company. You should be working to SAVE them money.
.exe files, flash, jar files etc. I wouldnt say you NEED more bandwidth, I say you COULD USE more bandwidth. I could always use more bandwidth. Actually a T3 right here now wouldnt be so bad.. I wouldnt have to WAIT for the damn DVD ISOs to download.
So just asking for $$$ for bandwidth is failing them, in a way. If things arent working out AT ALL right now and you have to fix it, they might pay. But if youre making do with things, they wont slap down another $100K.
So try other ideas like giving them quotas. They'll learn to fit their browsing in the quotas. Cache the pages, try to disable some things like maybe
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
They are getting bigger and bigger and bigger...
Sure web pages are getting larger but telecoms costs are decreasing too (at least in my part of the world), albeit not in direct proportion.
Invest in a proxy adblocker, such as http://www.privoxy.org/. This is easy to justify to management, it really doesn't interfere with legitimate web usage in my experience.
Create a group policy for Outlook that disables the ability to change the automatic picture downloading in Outlook2003.
Adjust you user's Exchange delivery restrictions to something sensible. Does anyone need to send something larger than 5Mb? If so then they can use the FTP server that you've kindly provided, they may decide they didn't need to send it after all when its not as simple as sending a mail.
This will drop your bandwith usage by a significant margin.
--
Freedom in a rivalrous commons brings ruin to all.
Lawrence Lessig
The un-adoption of mod_gzip and whatever IIS *should* use is also prevalent.
Ticking the box used to crash IIS but these days it actually works, not that you'd notice :
Response Headers - http://www.microsoft.com/
Date: Thu, 08 Dec 2005 10:30:40 GMT
Content-Length: 23186
Content-Type: text/html; charset=utf-8
Cache-Control: private
Server: Microsoft-IIS/6.0
P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
200 OK
Response Headers - http://slashdot.org/
Transfer-Encoding: chunked
Date: Thu, 08 Dec 2005 10:40:11 GMT
Content-Type: text/html; charset=iso-8859-1
Cache-Control: no-cache
Server: Apache/1.3.33 (Unix) mod_gzip/1.3.26.1a mod_perl/1.29
SLASH_LOG_DATA: mainpage
X-Powered-By: Slash 2.005000090
X-Fry: Where's Captain Bender? Off catastrophizing some other planet?
Pragma: no-cache
Vary: User-Agent,Accept-Encoding
Content-Encoding: gzip
200 OK
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
First get strong management backing. If not just back off and forget the whole thing.
;).
Then:
Option A: Fascist
All outbound nonSSL web traffic (80-83, 8000-8999) goes through transparent proxy.
All outbound SSL webtraffic must be via nontransparent proxy.
All users get their own PCs and IP addresses.
All users don't share PCs where possible.
No other outbound or inbound traffic allowed to client networks.
DNS, mail via the corporate servers.
Then go through the web logs and get the top 10 users and the top 10 sites visited.
Option B: Give people enough rope to hang themselves
Announce that statistics of all internet usage will be kept.
Make sure the clueless know what that really means, especially the powerful clueless ones...
Allow whatever traffic the users want with some safety in mind.
Produce a list of the top 10 users and the top 10 sites.
Careful though - often you'd find the bosses are the top culprits (that's why you need strong support from management). But even if that's the case, unless you are unlucky or do things badly, they'd just get more bandwidth for everyone overall... Just so they can listen to internet radio the whole day long...
If I were boss, I'd just make a _public_ list of the top 10 users and their top 10 sites every month. After a while I'd know a bit more about my staff
There's no need to guess when you can check. A quick gander at wikipedia or google define says that an E1 offers 2.048 Mbit/s of bandwidth whereas a T1 offers 1.544 Mbit/s. However your cost point is taken since an E1 obviously isn't 50 times the speed T1...
As others have suggested, you need to have local servers for caching web traffic, dns, etc. You can also block content such as advertisements and flash files.
However, assuming that has all failed, you can still setup an extranet proxy. The advantage of this is that the machine in your extranet can be run at much lower monthly rates in an area of cheap bandwidth (such as the US), and then COMPRESS the data before it hits your own pipe.
Essentially, use the following steps:
1. Purchase a dedicated server with adequate bandwidth. You can easily get this for less than $200/mo.
2. Install caching proxy and dns software on the dedicated server
3. Configure your local proxy and dns servers to pull from the extranet proxy.
(use SSL for encryption, gzip for compression... or build a tunnel)
You will need to make sure that the overhead of encryption doesn't outway the benefit of compression.
A quick way of doing this might be to block all (or most) Flash presentations. It is my experience that any site that heavily employs Flash displays tends to be very thin on content, despite the fact that they usually chew through a fair amount of bandwidth.
"When the moon is in the Seventh House
and Jupiter aligns with Mars
Then peace will guide the planets
And love will steer the stars
This is the dawning of the age of Web 2.0
The age of Web 2.0
Web 2.0! Web 2.0!"
With apologies to the writer of the song "Aquarius" from the musical "Hair".
Sure webpages get more and more crap, but there are still solutions, I saw the first post mentioned caches, but you can also block all ads, flash, embeded video, maybe there exists proxys which removes unneccesary stuff, and so on.
;)
Or you could move to Sweden or south Korea
Yes.
1) Transparent cacheing via Squid
2) Packet shaping to throttle or block certain protocols (Kill off P2P from your corporate network, for example)
3) Offsite proxying to compress text (HTML, JS, CSS, etc) to save bandwidth. Also might want to think about recompressing JPEG/GIF/etc offsite to save bandwidth at the sacrifice of quality
4) REQUIRE FLASHBLOCK ON ALL COMPUTERS. Seriously, the good thing about FlashBlock is that it doesn't prevent flash, only that flash will only load if you click on them. The benefit of this is that all the flash advertisements don't load, saving a LOT of bandwidth.
5) Look into Google Web Accelerator, or something similar. It will save a lot more bandwidth than just a web cache, and as I understand it, it also takes care of number 3 above by compressinve and DIFFing files for you before transfering them to you.
6) Find a cheaper connection. Seriously, there has got to be something better out there. Get some satellite internet out there for web traffic, since the 500ms latency doesn't matter for the web.
... and software expands to fill all available space. So I think that answers the question of whether or not Web pages are getting bigger.
The other problem is that the value of software is inversely proportional to the quantity of its output. So in that respect, while Web pages are getting bigger they are simultaneously becoming less useful.
The higher the technology, the sharper that two-edged sword.
Yes webpage sizes are getting larger. This is due to many things, but mostly due to people not wanting to know how things work, only that they do work. Frontpage is still widely used to create websites of various sizes. GoLive is used to create many websites. These two programs alone tend to default to using tables for designs that don't need them, they also add more and more extra coding each time someone changes something.
I worked a few months at a company doing their websites. I was told I had to use GoLive (version 6 on mac) in order to work there. I had never used GoLive before but it's an Adobe product so the basics are there, plus it is a piece of software, those are easy enough to figure out, at least for me, and I did learn it pretty thoroughly.
I looked over the previous sites that I was not allowed to make changes to. These were sites that needed updating every few days with new items. As I looked through the code on these pages, they were simple layouts that were burdened with multiple nested tables, some even blank as the content had been removed due to someone simply copying it in the gui and pasting it in other locations. There were multiple font tags within a number of td elements. Some had spans nested within p elements nested within div elements all within that td element. This page was running over 70kB in size just for the html, and then they laid so many images on the page, images for almost every textable item they had outside of the paragraphs. Even images for blank sections of color, not those little 1px clear gif's but for large areas of black.
To waste time on a slow day I went through and removed a bunch of that unnecessary nesting, but left the main tables alone and reduced the size to 35kB for the html. Everything still laid out correctly and of course the page loaded a bit quicker for the layout, the images still had to load in. Note: this was on a renamed page, not the live one.
Sorry, I got off on a tangent. Anyway, as more people rely on WYSIWYG editors and do not learn at least the basics of html then you are going to get bloated pages. I'm not saying that WYSIWYG editors are bad, but if people don't know when to use a DIV layout over a Table layout, then they probably don't know that they should change their preferences on each design to prefer those, plus changing it to use CSS instead of FONT tags. That will clear up a lot of excess baggage.
When bidding out my current jobs, I make sure to let them know the pages will be fast to load, and easy to navigate. I talk them out of unnecessary flash, and unnecessary images. Granted I have lost a few jobs due to my refusal to use flash, but that is my choice. I believe flash has it's place, but it should not be used for trivial things on most sites.
Just out of curiosity, does anyone know of an easy-to-setup Squid distro? Something that runs maybe as a live CD, with minimal or easy configuration? (Like Smoothwall, basically...)
There are a few places I know of that are really dying for web cache servers, because even without looking at the logs I can tell that they are probably spending GB a month downloading the same banner ads and Google splashscreen to every user on the network. It's a situation where there are hundreds of users and they're all basically doing the same thing as each other.
Last time I took a look at Squid -- granted this was a few years ago -- it was still a bit intimidating to set up for most people, especially if you wanted something that was transparent to the users (i.e. works without having to point your browser to it as a proxy).
Thoughts? I'm hoping there's some sort of CD that you can pop in and turn a disused box into an instant cache server. Or if not, maybe somebody will give the idea some thought. There is a lot of traffic being created at schools and small networks that's entirely redundant, but where the admins don't have (or think they don't have) the resources to set up a cache.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Try browsing for a while with a modem. Normal browsing. You will find out everything takes ages to load; if you remember, web pages were fine with dialup...